v0 FSST implementation #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
a10y
commented
Aug 12, 2024
| counts1: Vec<usize>, | ||
|
|
||
| /// Frequency count for each code-pair. | ||
| counts2: Vec<Vec<usize>>, |
There was a problem hiding this comment.
I wanted to make this a [[usize; 512]; 512] but that exceeds Rust's default stack size, and I thought it'd be annoying to force every consumer to have to raise their stack limit when they use this crate
|
Benchmark results. Note that decompression speed is great, compression speed, which should be 2-4x slower than decompression, is currently 1000x slower. Will need to implement the vectorization stuff. |
a10y
commented
Aug 12, 2024
| @@ -0,0 +1 @@ | |||
| How these papers have been placed in sequence will be made manifest in the reading of them. All needless matters have been eliminated, so that a history almost at variance with the possibilities of later-day belief may stand forth as simple fact. There is throughout no statement of past things wherein memory may err, for all the records chosen are exactly contemporary, given from the standpoints and within the range of knowledge of those who made them. We left in pretty good time, and came after nightfall to Klausenburgh. Here I stopped for the night at the Hotel Royale. I had for dinner, or rather supper, a chicken done up some way with red pepper, which was very good but thirsty. (Mem., get recipe for Mina.) I asked the waiter, and he said it was called “paprika hendl,” and that, as it was a national dish, I should be able to get it anywhere along the Carpathians. I found my smattering of German very useful here; indeed, I don’t know how I should be able to get on without it. | |||
There was a problem hiding this comment.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial implementation of FSST.
We implement symbol table building, compression and decompression routines.
decompress-singlebenchmark is able to decompress at ~1-2 cycles/byte. Note that to get there, I had to implement the unaligned writes pointer arithmetic stuff that you see here in thedecompressmethod