Crossover

[v1gnesh]

Hey,

You may know about the proj called serde_arrow, which allows Rust data types to be converted to Arrow representation, for playing nice with the Rust analytics ecosystem (DataFusion, Polars).

Wondering if you can weigh in on this post I made there about a crossover... allowing binrw to directly use serde_arrow to convert to arrow (as an option), as opposed to going via Rust data types. Or at least easing the pathway through Rust data types somehow.

Keen to know what you think..

[csnover]

Hi! Thanks for your feedback. I don’t understand what you are asking for binrw to do here? You can derive both binrw and serde at the same time on the same struct to convert to some format that uses serde.

[v1gnesh]

You can derive both binrw and serde at the same time on the same struct to convert to some format that uses serde.

My brain can't compute 😅.
If you have time, could you show a basic example.
My question is whether it's a desirable thing to have (in your perspective) for binrw to convert directly to arrow spec, hiding the work of deriving to Rust types in the middle.
That is... use binrw to derive arrow types (as an option, wherein it abstracts the setup necessary for deriving to serde, then deriving arrow from serde).

[csnover]

My apologies, I am struggling to achieve a mutual understanding here. Maybe answering these questions would help:

1. What’s the design goal of binrw, in your mind?
2. How does that goal fit with the idea of doing something special for one particular data format (in this case, arrow format)?
3. Since binrw generates parsers by authors decorating Rust types, how would it be possible for binrw to convert between data formats without having any Rust type in the middle?

With regards to an example of using multiple traits, it is not special, it is the way traits work in Rust:

use binrw::BinRead;
use serde::Serialize;
use serde_arrow::arrow2::{serialize_into_fields, serialize_into_arrays};

#[derive(BinRead, Serialize)]
struct Foo { /* ... */ }

fn convert_to_arrow<R: binrw::io::Read + binrw::io::Seek>(stream: &mut R) -> /* ... */ {
  let items = vec! [ Foo::read_options(stream, binrw::Endian::Little, ())? ];
  let fields = serialize_into_fields(&items, TracingOptions::default())?;
  let arrays = serialize_into_arrays(&fields, &items)?;
  /* ... */
}

[v1gnesh]

Do correct me if I'm wrong :)

What’s the design goal of binrw, in your mind?

Easier parsing of binary data.

How does that goal fit with the idea of doing something special for one particular data format?

Is it right to say that there are at least 2 types of things people parse with binrw

• static content like PNG files or whatever
• content like logs or packet capture that spans over some time window

In the second case, I'm guessing it's desirable (not just for me) to convert the source binary data to a format/spec that's becoming the standard in data analytics.
I do understand that binrw's goal is just the parsing part and not necessarily the second part (convert to another format).
Just interested in knowing what you think about it.

I do admit, I'm a forever newbie w.r.t programming. The example you've shown is very helpful, thank you!
There's also arrow2-convert that can help with the second part.
So from my perspective, I wanted to ask if you see this being valuable, as this depends on how your userbase currently use binrw (to simplify -- one-time file conversions or "streaming" data used to do some analysis later).

[csnover]

Thank you for your thoughts! It is always helpful to receive feedback to better understand the needs of users and whether we need to do something different.

Is it right to say that there are at least 2 types of things people parse with binrw

I understand why you are making a distinction between these things. From the perspective of binrw, though, they are equivalent, since the goal of binrw is as you state: to make it easier to parse (and write) binary data. Once data has been parsed, it’s up to the implementer to decide what they want to do with it, not binrw.

I do understand that binrw's goal is just the parsing part and not necessarily the second part (convert to another format). Just interested in knowing what you think about it.

One could certainly use binrw as the basis for a conversion library, and proposals like #98 are intended to make it easier for third-party crates to provide some standard serialisations of other types. Conversions require knowledge of how to transform between specific data formats, though, which isn’t something that can be done generically within the core binrw library.

My sense is that the right way to handle your specific situation is to do what is shown in the example and leverage the other existing library to transform the specific data source format to target arrow2 format.