Fix array of objects serialization #727
Conversation
cchandurkar
requested review from
sfc-gh-japatel,
sfc-gh-tzhang,
sfc-gh-tjones,
sfc-gh-rcheng and
a team
as code owners
| @@ -287,14 +287,7 @@ private Map<String, Object> getMapFromJsonNodeForStreamingIngest(JsonNode node) | |||
| String columnName = columnNames.next(); | |||
| JsonNode columnNode = node.get(columnName); | |||
| Object columnValue; | |||
| if (columnNode.isArray()) { | |||
There was a problem hiding this comment.
This is an interesting observation, I assume this was added for a reason, we will evaluate it to see if it will break anything, thanks!
There was a problem hiding this comment.
Looks like it was introduced in PR that enabled schematization and haven't been updated since.
There was a problem hiding this comment.
ok, I tired some quick experiments and I think it makes sense to make the change you suggested, it allows the array to be ingested with its original data type (like we should ingest [1,2] but not ["1","2"]. But for json, I believe you still need to do parse_json in order to use it since it will be ingested as a string?
There was a problem hiding this comment.
If you serialize an entire array node, its JSON elements won't be stringified unless they are stringified in original data. They will appear as "expected" in description. And that won't require to use parse_json() as array elements are variants.
|
Closing as #730 has been merged |
…r Schematization (snowflakedb#730) Before this change, every element in the array will be added as a STRING, this change preserves the old data type in the source, for example when the input is [1, 2], the ingested value will be [1, 2] now instead of ["1", "2"] Forked from snowflakedb#727, with additional tests
…r Schematization (snowflakedb#730) Before this change, every element in the array will be added as a STRING, this change preserves the old data type in the source, for example when the input is [1, 2], the ingested value will be [1, 2] now instead of ["1", "2"] Forked from snowflakedb#727, with additional tests
…r Schematization (snowflakedb#730) Before this change, every element in the array will be added as a STRING, this change preserves the old data type in the source, for example when the input is [1, 2], the ingested value will be [1, 2] now instead of ["1", "2"] Forked from snowflakedb#727, with additional tests
…r Schematization (snowflakedb#730) Before this change, every element in the array will be added as a STRING, this change preserves the old data type in the source, for example when the input is [1, 2], the ingested value will be [1, 2] now instead of ["1", "2"] Forked from snowflakedb#727, with additional tests
…r Schematization (snowflakedb#730) Before this change, every element in the array will be added as a STRING, this change preserves the old data type in the source, for example when the input is [1, 2], the ingested value will be [1, 2] now instead of ["1", "2"] Forked from snowflakedb#727, with additional tests
Currently, array of structs/objects are being written as an array of stringified json objects.
Here's a quick comparison:
Expected value in ARRAY column:
Actual value:
I think it's happening because those objects/structs are getting serialized twice. A simple fix here would be to serialize an entire ArrayNode as is instead of creating a
List<String>out of it. This way, the value (Stringified array) is a valid JSON that is accepted bynet.snowflake.ingest.streaming.SnowflakeStreamingIngestChannel.insertRow()Issue: #724