You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the PravegaReader uses the streaming API to read the stream however for Historical read testing we also need to test with the BatchClient.
Proposal
Add an extra -batchreader option, that if specified, tells the reader to consume the stream with the BatchClient. The number of consumers will still be specified and the reader will allocate the segments provided by the BatchClient to the available consumers.
Latency and Throughput metrics would be recorded as they would for any test using the streaming reader.
The text was updated successfully, but these errors were encountered:
What should happen when the batch reader finishes reading the stream? Does the benchmark end or does the batch reader process the next batch? Should we parallelize the reads from each segment or just use a single reader worker? If using multiple workers, we would need to synchronize them after each batch.
At the moment the idea is that we simply use the BatchClient instead of the streaming reader to read the entire stream. Once it get to the end of the segment ranges provided by the BatchClient when the benchmark started the benchmark will end.
The number of consumers will be honored with the test partitioning the number of SegmentRanges supplied by the BatchClient amongst the number of consumers.
The emphasis here is for Historical Stream processing testing to see if the BatchClient provides a better throughput than the streaming reader.
Currently the PravegaReader uses the streaming API to read the stream however for Historical read testing we also need to test with the BatchClient.
Proposal
Add an extra
-batchreader
option, that if specified, tells the reader to consume the stream with the BatchClient. The number of consumers will still be specified and the reader will allocate the segments provided by the BatchClient to the available consumers.Latency and Throughput metrics would be recorded as they would for any test using the streaming reader.
The text was updated successfully, but these errors were encountered: