gz failing to process

[M-Gonzalo]

(0.00%) Possible zLib-Stream in GZIP found at position 0
Compressed size: 6406001
Can be decompressed to 18424752 bytes
No matches
New size: 17661029 instead of 17661001     

Done.
Time: 5 second(s), 893 millisecond(s)

Recompressed streams: 0/1
GZip streams: 0/1

cloudflared.tar.gz

[schnaader]

Very interesting, seems to be a gzip and preflate related bug. Precomp 0.4.6 (the last version not using preflate) showed the correct compressed and decompressed size (17,660,983 -> 37,623,808), but couldn't proceed further (as expected). These sizes are correct and can be confirmed by looking at the file with 7-Zip.

Precomp 0.4.7 and 0.4.8dev (using preflate) show incorrect sizes (6,406,001 -> 18,424,752) like you posted - so it seems to detect only a part of the data. As the gzip header doesn't contain size information, this most likely is some bug related to the deflate parsing in preflate.

As a workaround, "-brute" manages to process some other parts and gives a slightly better result ("-cn -brute" -> 31,158,148 bytes, "-cl -brute" -> 16,748,698 bytes).

I will investigate this further and depending on the outcome (if it really is a bug in preflate), I might create an additional issue in the preflate repository.

[as-com]

I have another valid gzip file that precomp cannot process. It appears to be produced by Go's gzip implementation.

Here's the gzip file: https://drive.google.com/file/d/1VAK8e9aYpTREjYIvaQ9dDu8oLdcaNmoK/view?usp=sharing

And here's the raw deflate stream: https://drive.google.com/file/d/1RXQdupGnuDpfbPEw5t_MGa4AkwE69v8P/view?usp=sharing

I attempted to get preflate itself to process the (valid!) deflate stream, but it simply says "failed." I believe there is an issue with preflate.

[as-com]

I've narrowed down the issue to preflate's home-grown Huffman coding implementation not liking at least one of the Huffman trees that are included in the deflate stream.

The problem exists in this function: https://github.com/deus-libri/preflate/blob/609eefaa96ac6c51d7b1a3fb29e0ed94d0f3623e/support/huffman_helper.cpp#L20

It runs all the way to the bottom of the function, where the return statement evaluates to false. The expression seems suspect to me, but I don't have time right now to compare it with the spec/debug it further.

[as-com]

I found a fix, PR in #137