Add optional column_order in JSON reader #17029
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lamarrr
reviewed
Oct 22, 2024
|
@ttnghia This PR is ready for testing. This will enforce the column order and also insert empty all-null columns if not present in the json data. |
ttnghia
reviewed
Oct 24, 2024
ttnghia
reviewed
Oct 24, 2024
ttnghia
reviewed
Oct 24, 2024
ttnghia
reviewed
Oct 24, 2024
ttnghia
reviewed
Oct 24, 2024
ttnghia
reviewed
Oct 24, 2024
cpp/src/io/json/json_column.cu
Outdated
| child_columns.emplace_back(std::move(all_null_column)); | ||
| continue; | ||
| } | ||
| column_names.emplace_back(found_col->first); |
There was a problem hiding this comment.
Above we have if (prune_columns and found_col == std::end thus here if !prune_columns then we still have found_col == std::end.
There was a problem hiding this comment.
I added prune_columns condition to col_order decision. This case won't happen.
ttnghia
reviewed
Oct 24, 2024
ttnghia
reviewed
Oct 24, 2024
ttnghia
reviewed
Oct 24, 2024
karthikeyann
added
improvement
ttnghia
approved these changes
Nov 4, 2024
karthikeyann
changed the title
karthikeyann
changed the title
karthikeyann
added
the
5 - Ready to Merge
vuule
reviewed
Nov 7, 2024
vuule
reviewed
Nov 7, 2024
changed logic of has_column_order used std::invalid_argument update auto to references at few places
ttnghia
approved these changes
Nov 7, 2024
vuule
approved these changes
Nov 7, 2024
|
/merge |
karthikeyann
added
Spark
rapids-bot bot
pushed a commit
that referenced
this pull request
Nov 8, 2024
…read_json` directly (#17180) With this PR, `Table.readJSON` will return the output from libcudf `read_json` directly without the need of reordering the columns to match with the input schema, as well as generating all-nulls columns for the ones in the input schema that do not exist in the JSON data. This is because libcudf `read_json` already does these thus we no longer have to do it. Depends on: * #17029 Partially contributes to NVIDIA/spark-rapids#11560. Closes #17002 Authors: - Nghia Truong (https://github.com/ttnghia) Approvers: - Robert (Bobby) Evans (https://github.com/revans2) URL: #17180
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds optional column order to enforce column order in the output. This feature is required by spark from_json.
Optional
column_orderis added toschema_element, and it is validated during reader_option creation. The column order can be specified at root level and for any struct in any level.• For root level, the dtypes should be schema_element with type STRUCT. (schema_element is added to variant dtypes)
• For nested level, column_order can be specified for any STRUCT type. (could be a map of schema_element , or schema_element)
If the column order is not specified, the order of columns is same as the order of columns that appear in json file.
Closes #17240 (metadata updated)
Closes #17091 (will return all nulls column if not present in input json)
Closes #17090 (fixed with new schema_element as dtype)
Closes #16799 (output columns are created from column_order if present)
Checklist