WIP Upgrade to arrow-rs/parquet 54.0.0
#13663
Conversation
github-actions
bot
added
common
alamb
changed the title
alamb
changed the title
alamb
force-pushed
the
alamb/upgrade_to_arrow
branch
from
bd6517c to
44bb39d
Compare
alamb
changed the title
54.0.0
alamb
force-pushed
the
alamb/upgrade_to_arrow
branch
from
3c5f100 to
51fdc1e
Compare
github-actions
bot
added
physical-expr
| @@ -1777,6 +1777,8 @@ fn round_trip_datatype() { | |||
| } | |||
| } | |||
|
|
|||
| // TODO file a ticket about handling deprecated dict_id attributes | |||
| #[allow(deprecated)] | |||
There was a problem hiding this comment.
I don't know what to do here:
- Should we remove the
field_idfrom DataFusion serialization code (as it is planned for removal eventually in DataFusion)? - Should we un-deprecate
field_idin arrow-rs? - Something else?
@avantgardnerio, @thinkharderdev and @Dandandan , this test was added in
However, @brancz's upstream PR in arrow-rs has deprecated dict_id on Field and all related APIs with the idea it will be eventually removed (and become a detail of the IPC transport rather than visible in the Schema)
- Add deprecation warnings for everything related to
dict_idarrow-rs#6873 upstream in arrow-rs
@thinkharderdev mentioned apache/arrow-rs#6873 (comment)
If people are using the dictionary ID it's probably for something that should properly be done with kv metadata instead.
So one possibility could be to provide some more documentation / writeup as an upgrade guide for arrow explaining how to use field metadata instead ?
Please advise
There was a problem hiding this comment.
If people are using the dictionary ID it's probably for something that should properly be done with kv metadata instead
I mostly meant that if you give people 32 bits of space someone, somewhere will use it as metadata if they don't need it for anything else. I don't really have any particular use case in mind.
I don't have any objection to removing the field from datafusion serde. When we added that test we were still fighting to try and ensure unique dict IDs but we've long since moved on and found better ways around it.
There was a problem hiding this comment.
Or maybe on second thought we should keep it around until it is fully removed from upstream?
There was a problem hiding this comment.
Yeah, that sounds like a reasonable approach. I'll update the comments to match if no one else has a usecase
There was a problem hiding this comment.
Agreed. Also, #8457 (comment) suggests it was never important to set the dict id directly, just that it wasn't sent correctly on the wire and already called out what to do once we remove the dict id. There were in fact a number of dict fixes that were necessary for us to correctly communicate the dicts correctly over the wire as well.
alamb
force-pushed
the
alamb/upgrade_to_arrow
branch
from
51fdc1e to
fb080a2
Compare
alamb
force-pushed
the
alamb/upgrade_to_arrow
branch
from
fb080a2 to
2fe785b
Compare
| @@ -459,10 +459,6 @@ config_namespace! { | |||
| /// default parquet writer setting | |||
| pub statistics_enabled: Option<String>, transform = str::to_lowercase, default = Some("page".into()) | |||
|
|
|||
| /// (writing) Sets max statistics size for any column. If NULL, uses | |||
| /// default parquet writer setting | |||
| pub max_statistics_size: Option<usize>, default = Some(4096) | |||
There was a problem hiding this comment.
I think simply removing the config option is pretty rough as you end up with errors liek this
External error: query failed: DataFusion error: Invalid or Unsupported Configuration: Config value "max_statistics_size" not found on ParquetOptions
The best thing to do would be to mark the config setting deprecated, but there wasn't an obvious way to do this with the existing macros. I will futz with the macros and see if I can make it deprecated somehow
| @@ -321,6 +321,8 @@ impl TryFrom<&protobuf::Field> for Field { | |||
| fn try_from(field: &protobuf::Field) -> Result<Self, Self::Error> { | |||
| let datatype = field.arrow_type.as_deref().required("arrow_type")?; | |||
| let field = if field.dict_id != 0 { | |||
| // TODO file a ticket about handling deprecated dict_id attributes | |||
There was a problem hiding this comment.
Need to update per apache/arrow-rs#6885 I think
Which issue does this PR close?
Related to
54.0.0(December 2024) arrow-rs#6342Rationale for this change
Prepare DataFusion for the next arrow-rs release
Also, help test to ensure that the next arrow-rs release has no regressions by testing with DataFusion before we release it
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?