Parquet writing examples/macro/guidance

[lyuben-todorov]

Hi, I have a long-standing puzzle with parquet's presence in rust.

My end-goal is to be able to write parquet files containing my data in rust in an efficient manner (no json, senseless conversions, etc.). For this my logical approach would be to look for/make my own derive macro for Parquet writers for structs. However, the parquet_derive crate is lacking a lot of features (nested structures) and according to ASF slack isn't really actively developed at the moment. I tried implementing a derive macro for an Arrow RecordBatch writer (from the arrow crate) but I quickly ran into problems with using the arrow crate itself. And because of that experience I'm really starting to think that writing parquet in anything other than Java was not meant to be, but that is still not the theoretical case.

I'm asking the maintainer as a person with more experience with the parquet format and ecosystem, are my goals possible? If yes, could you please provide me some guidance on what would need to be done, what's the best way to approach it and maybe a code example of implementing the conversion of data (Vec) into a parquet file.

[jorgecarleitao]

Thanks a lot for reaching out! Some questions:

1. Is the data columnar in nature (or is it e.g. a stream of rows)?
2. can you lay out your data according to the arrow format?
3. is the data flat (i.e. no nested structures)?

If all are yes, then I would try using arrow2 directly.

If 1 or 2 is negative, I would try out arrow2-derive to build a RecordBatch

If 3 is negative, we still do not support it in arrow2 (in the roadmap, see e.g. jorgecarleitao/arrow2#504).

[jorgecarleitao]

Writing nested parquet is hard due to dremel, and java has (10+?) years of bagage on it. It will take a bit to get this through. I know more or less what I need to do but it is still a non-trivial exercise :)

[lyuben-todorov]

Thanks for the timely reply!

1. The data itself is a stream of rows but I've planned for splitting it up in batches for a columnar store. I currently achieve my goals through AWS-manages services and such, however they're way too limiting and I'd rather have some control over the conversion process.
2. Yes, actually I find the arrow format really expressive and it does sound like something my data could use, such as the FixedSizeBinary (we have a lot of 'custom' types that could benefit from zero-copy-ness if they were encoded as fixed size byte arrays instead of Vec's).
   My only question is why a signed integer is used to denote size in FixedSizeBinary(i32) and a couple more places in the arrow/parquet/apache domain (guessing it's a java thing).
3. Sadly not, however in reality I'm already convinced that it's simpler to implement some code generation tools that provide conversion to/from nested types than it is to implement the actual arrow/parquet/derive support for that.

I'll be keeping track of these crates and especially that issue. Unfortunately I don't think I have the expertise to contribute. I can't get myself to sit down and fully understand dremel encoding :)

P.S. Didn't see the new discussion feature, sorry!