Unable to read parquet data without _metadata/_common_metadata

[porscheme]

We are not able to read parquet file(s) in a folder.

• Although, we can read file by file not by using folder name
• This folder doesn't have either of _metadata or_common_metadata file

Below is how we write parquet files

            _columns = new Column[]
            {
                new Column<string>("studentId"),
                new Column<byte[]>("Data"),
                new Column<string>("changeType"),
                new Column<DateTime>("UpdatedDateTime")
            };

            ConcurrentDictionary<string, Tuple<byte[], string, DateTimeOffset>> myResult = new ConcurrentDictionary<string, Tuple<byte[], string, DateTimeOffset>>();
                DataLakeFileClient file = resultClient.GetFileClient(fileName));

                using (Stream us = file.OpenWrite(true))
                {
                    using (var writer = new ParquetSharp.IO.ManagedOutputStream(us))
                    {
                        using (var fw = new ParquetFileWriter(writer, _columns))
                        {
                            // Write each column (in the order of schema) as a row group
                            using RowGroupWriter rowGroup = fw.AppendRowGroup();

                            // Write StudentId
                            using (var pIdWriter = rowGroup.NextColumn().LogicalWriter<string>())
                            {
                                pIdWriter.WriteBatch(myResult.Keys.ToArray<string>());
                            }

                            // Write Data
                            using (var dataWriter = rowGroup.NextColumn().LogicalWriter<byte[]>())
                            {
                                List<byte[]> ba = new List<byte[]>();
                                foreach (var key in myResult.Keys)
                                {
                                    ba.Add(myResult[key].Item1);
                                }
                                dataWriter.WriteBatch(ba.ToArray<byte[]>());
                            }

                            // Write changeType
                            using (var cw = rowGroup.NextColumn().LogicalWriter<string>())
                            {
                                List<string> ca = new List<string>();
                                foreach (var key in myResult.Keys)
                                {
                                    ca.Add(myResult[key].Item2);
                                }
                                cw.WriteBatch(ca.ToArray<string>());
                            }

                            // Write UpdatedDateTime
                            using (var dtw = rowGroup.NextColumn().LogicalWriter<DateTime>())
                            {
                                List<DateTime> dta = new List<DateTime>();
                                foreach (var key in myResult.Keys)
                                {
                                    dta.Add(myResult[key].Item3.DateTime);
                                }
                                dtw.WriteBatch(dta.ToArray<DateTime>());
                            }

                            fw.Close();
                        }
                    }
                }

[adamreeve]

Hi @porscheme, that sounds quite odd, I don't know what would be causing that behaviour.

What do you mean exactly by "not able to read", can you provide the error message you get?

And how are you reading the files? Using new ParquetFileReader(filePath) where filePath is a string path to the file? As another workaround you could possibly open the file as a .NET Stream and then pass that to the ParquetFileReader constructor.

[adamreeve]

I see you also created an issue at #362 with some more details on how you're reading the data. I've left a response there, but this looks like an issue with how you're reading the files rather than a problem with ParquetSharp.