Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raw JSON writer (~10x faster) (#5314) #5318

Merged
merged 4 commits into from
Jan 24, 2024
Merged

Raw JSON writer (~10x faster) (#5314) #5318

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e8f914c
Raw JSON writer (#5314)
tustvold Jan 20, 2024
9a8e46c
Fix bench name
tustvold Jan 24, 2024
afaf192
Review feedback
tustvold Jan 24, 2024
008b086
Review feedback
tustvold Jan 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions arrow-json/src/writer/encoder.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,13 @@ pub struct EncoderOptions {
pub explicit_nulls: bool,
}

/// A trait to format array values as JSON values
///
/// Nullability is handled by the caller to allow encoding nulls implicitly, i.e. `{}` instead of `{"a": null}`
pub trait Encoder {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please document the expectations on nullability here? Specifically, it seems like this code assumes that this is invoked with idx for non-null entries, which was not clear to me on my first read of this code

/// Encode the non-null value at index `idx` to `out`
///
/// The behaviour is unspecified if `idx` corresponds to a null index
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

fn encode(&mut self, idx: usize, out: &mut Vec<u8>);
}

Expand Down Expand Up @@ -173,6 +179,9 @@ trait PrimitiveEncode: ArrowNativeType {
// Workaround https://github.com/rust-lang/rust/issues/61415
fn init_buffer() -> Self::Buffer;

/// Encode the primitive value as bytes, returning a reference to that slice.
///
/// `buf` is temporary space that may be used
fn encode(self, buf: &mut Self::Buffer) -> &[u8];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would help to document what encode does here

Suggested change
fn encode(self, buf: &mut Self::Buffer) -> &[u8];
/// Encode the primitive value as bytes, returning a reference to that slice.
/// `buf` is temporary space that may be used
fn encode(self, buf: &mut Self::Buffer) -> &[u8];

}

Expand Down Expand Up @@ -219,14 +228,14 @@ macro_rules! float_encode {
float_encode!(f32, f64);

impl PrimitiveEncode for f16 {
type Buffer = <f64 as PrimitiveEncode>::Buffer;
type Buffer = <f32 as PrimitiveEncode>::Buffer;

fn init_buffer() -> Self::Buffer {
f64::init_buffer()
f32::init_buffer()
}

fn encode(self, buf: &mut Self::Buffer) -> &[u8] {
self.to_f64().encode(buf)
self.to_f32().encode(buf)
}
}

Expand Down
4 changes: 2 additions & 2 deletions arrow-json/test/data/basic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{"a":1, "b":2.0, "c":false, "d":"4", "e":"1970-1-2", "f": "1.02", "g": "2012-04-23T18:25:43.511", "h": 1.2802734375}
{"a":-10, "b":-3.5, "c":true, "d":"4", "e": "1969-12-31", "f": "-0.3", "g": "2016-04-23T18:25:43.511", "h": 3.140625}
{"a":1, "b":2.0, "c":false, "d":"4", "e":"1970-1-2", "f": "1.02", "g": "2012-04-23T18:25:43.511", "h": 1.125}
{"a":-10, "b":-3.5, "c":true, "d":"4", "e": "1969-12-31", "f": "-0.3", "g": "2016-04-23T18:25:43.511", "h": 3.5}
{"a":2, "b":0.6, "c":false, "d":"text", "e": "1970-01-02 11:11:11", "f": "1377.223"}
{"a":1, "b":2.0, "c":false, "d":"4", "f": "1337.009"}
{"a":7, "b":-3.5, "c":true, "d":"4", "f": "1"}
Expand Down
Loading