Support horizontally flipped TGA images #68
Conversation
The RLE TGA images were converted from raw TGA ones using GIMP and some tweaks as the GIMP exporter only support bottom-left and top-left: - Left ones are unmodified GIMP export. - Right ones are in-GIMP horizontally flipped images exported as left ones then the TGA X flip bit was modified by hand in a hex editor.
The TGA format is a bottom-left format by default: the first byte of the first column is expected to be displayed at the bottom-left of the screen, this behavior is inherited from devices painting the screen from the bottom left to the top right. The TGA format provides two bits in the image descriptor byte to paint the data from other sides. The 4th bit tells the device to paint the screen from right-to left and the 5th bit tells the device to paint the screen from top to bottom. So basically: - 00: from bottom-left to top-right - 01: from top-left to bottom-right - 10: from bottom-right to top-left - 11: from top-right to bottom-left Previously stb_image only read the 5th bit and then only supported the loading of vertically flipped images, stb_image was ignoring the 4th bit coding the horizontal flip. Now both flipping directions are supported for both raw and RLE storage.
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
from
a276c16
to
bcb7c34
Compare
|
This is tested against existing raw TGA test files already provided in the repository, and additional RLE TGA test files provided with the PR. The raw TGA ones are from https://gitlab.com/illwieckz/investigating-tga-orientation-in-imagemagick They are generated with a script written for the only purpose of generating those reference images for debugging image libraries and tools. The RLE TGA images were converted from the raw TGA ones using GIMP and some tweaks, as the GIMP exporter only supports bottom-left and top-left:
|
|
We are in need for a reliable tool we can recommend that is able to convert TGA to PNG, as TGA is the de-facto legacy idTech3 lossless image format and we prefer to store PNG or WebP images in repository (and cwebp expects PNG as input), so we need to reliably convert TGA files to PNG. ImageMagick convert is not reliable, it is known to be buggy for years, it is also well known that there is no way to know if a given ImageMagick convert version is affected or not: The bug is considered to not be a bug by the maintainer despite countless reports and a complete study having been published with an alternate from-scratch implementation written against the specifications to compare: For reference, here are the specifications: The crunch repository being a required dependency of the engine because of the |
|
I plan to submit the stb_image changes to upstream. |
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
7 times, most recently
from
71e63b3
to
a5680a3
Compare
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
from
2bf5fc4
to
b733cbd
Compare
|
I added some checks of what crunch produces, it confirmed i686 without SSE2 doesn't produce same file than all other tested platforms, as discussed there: When passing the |
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
10 times, most recently
from
9e879a6
to
4385e86
Compare
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
from
4385e86
to
078bdb5
Compare
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
3 times, most recently
from
f74086e
to
504b4aa
Compare
|
So, I implemented the idea of dumping a checksum file and check against it. It happened that by doing so I was able to test the checksum for all the files, and then I discovered that “jpeg to dds” is not reproducible. When running a GCC build on Ubuntu 24.04 or on 22.04 chroot on 24.04 on my own Zen 2 amd64, I don't get the same result for jpg to dds that I get with GCC on Ubuntu 22.04 on Azure amd64… I don't know why yet but I give up on that for now. So for now the exhaustive file check is disabled, what is still done is the clone check: for every command that should produce the same file (like png to crn and bmp to crn, or bottom-left tga to crn and top-right tga to crn), it is checked that all the clones produce the same checksum. This is enough to verify that image orientation is properly implemented. |
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
4 times, most recently
from
525a02b
to
4f3de1c
Compare
|
So I discovered that GCC defaults on some Now CMake also disables FP contraction whith I enabled the file checksuming, it now works everywhere. If needed (for example when committing a change changing some format on purpose), the checksum database can be re-generated this way: test/test.py --record |
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
from
a934e25
to
e48b33b
Compare
|
Hmm, I still have a mistmatch on jpg-to-dds, converting
But on my end I got this whatever the compiler:
What's weird is that it's always one on CI, and always another one on my end. |
|
The only
|
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
from
4cfe27f
to
d1332ec
Compare
|
So I disabled exhaustive file checking again, I only left clone checking enabled. More research may be done in another time in another PR, there is no need to postpone merging what is already working. |
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
5 times, most recently
from
25cc8c7
to
711a7be
Compare
- appveyor: do not build the shared library (it is unusable yet), - azure-pipelines: make all binaries use the same library, - codeql: use the shared library.
|
I finally reproduced the jpg-to-dds mismatch! I thought that such mismatch was insane:
The only difference I could identify was that it was running on my end, and the hypothesis of my home being subject to a spell was not realistic, so I started to check the differences between the CI servers and my own system, looking at environment variables, etc. Then I found the difference… My computer has 32 threads. Those CI systems provide no more than 4 threads (If I'm right: 2 on AppVeyor and 4 on Azure). As soon as I do |
illwieckz
force-pushed
the
illwieckz/tga-x-flip
branch
2 times, most recently
from
fc21439
to
0974162
Compare
|
So, it looks like all the options to make image conversion reproducible in a CI are there, I then enabled the exhaustive file check. |
|
LGTM |
Support horizontally flipped TGA images.
The TGA format is a bottom-left format by default: the first byte of the first column is expected to be displayed at the bottom-left of the screen, this behavior is inherited from devices painting the screen from the bottom left to the top right.
The TGA format provides two bits in the image descriptor byte to paint the data from other sides. The 4th bit tells the device to paint the screen from right-to left and the 5th bit tells the device to paint the screen from top to bottom.
So basically:
Previously stb_image only read the 5th bit and then only supported the loading of vertically flipped images, stb_image was ignoring the 4th bit coding the horizontal flip. Now both flipping directions are supported for both raw and RLE storage.