fix: Fix incorrect primitive type detection

[JasonYeMSFT]

Problem

typeLength, and potentially precision, with value "null" causes incorrect primitive type detection result.

Solution

We should handle the null values such that when the typeLength or precisions field is of value "null", its primitive type are detected as "INT64".

Steps to Verify:
The bug reproduces when the parquet file consists of a Dictionary_Page with a INT64 field whose typeLength is null upon read. Unfortunately, I don't have such a test file for now. My debugging was based on a piece of privately shared data from our customer.

When the bug reproduces, the primitive type parsed from the schema (Fixed_Length_Byte_Array) won't match the primitive type discovered from the column data (Int64). Due to a discrepancy on how the library decodes data pages, when the data is in a Dictionary_Page, the decoding logic will hit the check for typeLength and fail. For Data_Page and Data_Page_V2, decoding ignores the schema and privileges the primitive type inferred from the column data. However, for Dictionary_Page, decoding uses the primitive type specified in the schema.

decodeDataPageV2

parquetjs/lib/reader.ts

Line 1104 in 91fc71f

let values = decodeValues(

decodeDictionaryPage

parquetjs/lib/reader.ts

Line 947 in 91fc71f

return decodeValues(opts.column!.primitiveType!, opts.column!.encoding!, dictCursor, (header.dictionary_page_header!).num_values, opts)

Notice that one uses "opts.type" while the other uses "opts.column.primitiveType".

[JasonYeMSFT]

I tried to create a test file using the library but it seems like it doesn't support writing Dictionary_Page for now?

[wilwade]

There are other updates that are needed in this area (Issue #121 is about this as well), but this is a good, clean fix for the specific problem.

[wilwade]

I tried to create a test file using the library but it seems like it doesn't support writing Dictionary_Page for now?

I'm not sure. I don't think we've really had to mess with that code since the fork.