Is there a way to track progress with CsvDataReader?

[0xced]

Currently, I’m counting the number of lines in my input stream (by counting the number of \n characters), then I rewind the stream and track progress by number of lines read. It works for a FileStream but would not work on a non-seekable stream (and is not ideal from a performance point of view).

I would prefer to track progress with the number of bytes read by the CsvDataReader but I haven’t found how. Is this actually possible? If not, would you accept a pull request?

[MarkPflug]

The way that I've done this before is to use a wrapper stream that counts the bytes passing through it. It works well, but only works for streams where the length is known. If the length of the stream is unknown, then I don't see a way to report progress without buffering the entire file somewhere.

Here is an example:
https://github.com/dataplat/dbatools/pull/8393/files
That example is using the lumenworks CSV parser, but the construction would be the same for my library. Specifically, see the ProgressStream.cs and Import-DbaCsv.ps1 files.

[0xced]

Thanks for your extremely quick reply! I will try this approach, it looks fine.

And indeed I was thinking about a scenario where the total length is known upfront such as a file stream, an HTTP response with a known content-length or a zip file (through ZipArchiveEntry.Length).

[MarkPflug]

And don't forget to dispose the wrapped stream (like I did). dataplat/dbatools#8596
Amusing to me that this bug was reported the same day I pointed you to this solution.

[yunhyejoung]

And don't forget to dispose the wrapped stream (like I did). dataplat/dbatools#8596 Amusing to me that this bug was reported the same day I pointed you to this solution.

using Sylvan.Data;
using Sylvan.Data.Csv;
using System.Linq;

using var dr = CsvDataReader.Create("data.csv");
IEnumerable records = dr.GetRecords();

IDataReader dr = records.AsDataReader().AsDbDataReader();

while( dr.Read() ) //<===== have to do dr.Read()
{
.............
}

There is no error

But, return type : sylvan.data.syncObjectDataReader

I want to :
Sylvan.data.transformDataReader
Or
Sylvan.Data.CsvDataReader