Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for reading with BatchClient #89

Open
maddisondavid opened this issue Jan 10, 2020 · 3 comments · May be fixed by #92
Open

Add support for reading with BatchClient #89

maddisondavid opened this issue Jan 10, 2020 · 3 comments · May be fixed by #92
Assignees

Comments

@maddisondavid
Copy link
Contributor

Currently the PravegaReader uses the streaming API to read the stream however for Historical read testing we also need to test with the BatchClient.

Proposal
Add an extra -batchreader option, that if specified, tells the reader to consume the stream with the BatchClient. The number of consumers will still be specified and the reader will allocate the segments provided by the BatchClient to the available consumers.

Latency and Throughput metrics would be recorded as they would for any test using the streaming reader.

@claudiofahey
Copy link

What should happen when the batch reader finishes reading the stream? Does the benchmark end or does the batch reader process the next batch? Should we parallelize the reads from each segment or just use a single reader worker? If using multiple workers, we would need to synchronize them after each batch.

@maddisondavid maddisondavid linked a pull request Jan 12, 2020 that will close this issue
@maddisondavid
Copy link
Contributor Author

At the moment the idea is that we simply use the BatchClient instead of the streaming reader to read the entire stream. Once it get to the end of the segment ranges provided by the BatchClient when the benchmark started the benchmark will end.

The number of consumers will be honored with the test partitioning the number of SegmentRanges supplied by the BatchClient amongst the number of consumers.

The emphasis here is for Historical Stream processing testing to see if the BatchClient provides a better throughput than the streaming reader.

@maddisondavid
Copy link
Contributor Author

A PR for this has been opened here #92

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants