Would a replacement/update to MPQ files be benefial to DevilutionX?

[malvarenga123]

Note: This discussion is aimed mainly at devices that have limited RAM and use (micro)SD cards.

After seeing the work that was done to support ZSO files into OPL and considering @glebm is playing with the idea of recompressing files inside MPQs using an external tool, I was wondering if it wouldn't be better for devices with limited resources if the MPQ files were replaced with another format or updated to use faster compression schemes (like zstd or LZ4).

I think this might lead to faster loading times and less wear to memory cards on devices like the 3DS or PSVita.

PS: @glebm wouldn't a lossless audio format be more favorable to CPU and RAM usage instead of MP3?

[AJenbo]

We are never going to drop support for MPQ. Alone the fact that all most Windows users have to do is extract the zip and double click the exe (assuming they have Diablo installed via GOG) makes this a required feature.

So any additional formats we would add support for would have to make up for it's addition, rather then as a replacement. But if ZSO can do so then I'm not against adding support for it.

I do prefer that the formates are well supported, easy to create and inspect.

PS: @glebm wouldn't a lossless audio format be more favorable to CPU and RAM usage instead of MP3?

I don't think lossless equates fast and light on ressources.

[StephenCWills]

MP3 can achieve incredibly small file sizes without significant degradation in sound quality because it is tailored specifically for compressing audio data by removing frequencies that the human ear is typically unable to hear. You won't get the same level of performance from general-purpose compression algorithms like zstd or LZ4 because they are unaware of the type of data they are compressing.

You might want to have a look at this discussion.
facebook/zstd#361

As for CPU usage, I can't say how MP3 compares to zstd or LZ4, but AJenbo is right that lossy/lossless is basically irrelevant. We did test the change to MP3 to make sure it worked okay on 3DS and PSVita specifically. It doesn't seem to significantly degrade performance. If you're looking to improve performance on these devices, efforts would be better spent offloading the rendering onto the GPU. By far, graphics rendering takes up the vast majority of the CPU resources in this game.

As a final note, it helps that MP3 is supported by SDL_audiolib.

[glebm]

I think it may be beneficial for some devices to unpack the MPQ and just use regular files. We could even mmap these files, which can help very RAM-constrained devices.

As for the lossless audio format, it depends on the device. DevilutionX keeps some sounds load in-memory at all times, while streaming the others. For sounds that are in-memory MP3 could help by reducing memory usage.