Currently parquet don't reads some specific bytes, why?

[Sach1nAgarwal]

for this parquet file :-
bigint_tbl.txt
This parquet files has 874 bytes. And has 2 columns, it has 10 rows.

parquet reader will read :-
first last 8 bytes, range 866 to 873 bytes
secondly then metadata, range 273-865 bytes
third then first column, range 4 - 99 bytes
fourth then second column, range 100 - 189 bytes

Start 4 bytes not read because it contains 'PAR1'.
Why 190 to 272 bytes not read ? @tustvold

[tustvold]

Could you provide a code sample of how you are reading the file?

[Sach1nAgarwal]

First creating ArrowReaderBuilder, then ParquetRecordBatchStream from ArrowReaderBuilder,

let builder:ArrowReaderBuilder  = ParquetRecordBatchStreamBuilder::new(reader)
                                                             .await
                                                             .map_err(|err| inputerror!("{:?}", err))?;

let parquet_column_indices = vec![0,1];
let mask = ProjectionMask::leaves(builder.parquet_schema(), parquet_column_indices);

let parquet: ParquetRecordBatchStream =  builder
                                            .with_projection(mask)
                                            .with_batch_size(batch_size)
                                            .build()
                                            .map_err(|err| invalid!("{:?}", err))?,

if let Some(batch) = parquet.as_mut().unwrap().next().await {
               return Some(batch.map_err(|e| {
                   ArrowError::ParquetError(String::from(format!("Parquet error: {}", e)))
               }));
           }

then reading columns from batch.

[tustvold]

The remaining bytes are most likely index information which you aren't telling it to read (as there is no point if you aren't doing predicate pushdown).

It can be enabled with https://docs.rs/parquet/latest/parquet/file/serialized_reader/struct.ReadOptionsBuilder.html#method.with_page_index

https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/ may also be informative

[Sach1nAgarwal]

@tustvold It reads first all columns sequentially, after all columns read, then decoding starts, can reading each columns and decoding can be parallelized?

[Sach1nAgarwal]

The remaining bytes are most likely index information which you aren't telling it to read (as there is no point if you aren't doing predicate pushdown).

It can be enabled with https://docs.rs/parquet/latest/parquet/file/serialized_reader/struct.ReadOptionsBuilder.html#method.with_page_index

https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/ may also be informative

Thanks

[tustvold]

It reads first all columns sequentially

It reads a row group at a time, this is inline with the formulation of a ColumnChunk as the unit of IO. See #1473 for more context. This approach is somewhat optimised for object storage / HDFS, where it is really important to fetch large blocks at once for cost and performance reasons.

Amusingly it also shows benefits for async file IO as tokio currently calls spawn_blocking for each blocking IO call, which is remarkably expensive. If optimising for local filesystem, the synchronous reader may be more performant, async brings limited benefits unless using something like glomio.

That being said there is potential to fetch the next row group whilst decoding the previous, this is tracked in #3635. In general the assumption of arrow-rs is that parallelism is delegated to a higher level query engine that can reason more about it, see apache/datafusion#5057 for example