-
Notifications
You must be signed in to change notification settings - Fork 105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use Rust implementation instead? #91
Comments
xref: dask/fastparquet#488 I think that in general this is a good idea in general. To geta good response here, it would need to show that
|
Seems reasonable. I see in To my understanding, there is the raw, used for streaming, and the framed (entire in-memory streams like you mentioned) formats of snappy. Can I assume the hadoop format reference is a reference to the raw format? Those are the only two formats in the Rust implementation. If this isn't the case, then I guess there is no point in starting. Also, could you specify what is considered a "large" install size, is ~1MB too big? |
Even though I may be a maintainer here, I don't actually follow the snappy specs... So long as the existing de/compresss functions and their stream counterparts srtill product identical output, I would be happy! |
Hi there, working from home has left me with less time than expected. I've added a new commit to >>> import io
>>> import snappy
>>> import cramjam
>>> data = b'hi, hello there'
>>> raw = io.BytesIO(data)
>>> output = io.BytesIO()
>>> snappy.stream_compress(raw, output)
>>> output.seek(0)
0
>>> output.read()
b'\xff\x06\x00\x00sNaPpY\x01\x13\x00\x00\x82\x8f\x01\xb8hi, hello there'
>>> cramjam.snappy_compress(data)
b'\xff\x06\x00\x00sNaPpY\x01\x13\x00\x00\x82\x8f\x01\xb8hi, hello there'
>>> snappy.compress(data)
b'\x0f8hi, hello there'
>>> cramjam.snappy_compress_raw(data)
b'\x0f8hi, hello there'
>>> One of my concerns with making a PR to |
Hi there!
I was curious if you'd be open to having the Rust implementation of snappy instead of the C dependency which can lead to troubles, especially when using
python-snappy
in environments like AWS Lambda?I've made cramjam which does just this but would be willing to attempt migrating this project in a similar way. It wouldn't require any system dependencies and packing them in a wheel (OSX, Linux & Windows supported), of course, wouldn't require any compiler for the user. As it is right now,
cramjam
which includes snappy results in about ~1.5MB for linux wheels.Anyway, let me know what you think and I'd be willing to start messing around with it. 👍
The text was updated successfully, but these errors were encountered: