-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Managed streaming - better estimation for decision for streaming vs queuing #368
Conversation
This is a very big PR, can you summarize the changes in more detail? Also, for disableRetries, maybe we want to fold this in to a more robust Policy object like @yogilad suggested and worked on. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Basically done, just go over the rest of the comments. Most of them are nothing
|
||
if (properties.isDisableRetries()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you?
data/src/main/java/com/microsoft/azure/kusto/data/ClientFactory.java
Outdated
Show resolved
Hide resolved
ingest/src/main/java/com/microsoft/azure/kusto/ingest/source/AbstractSourceInfo.java
Show resolved
Hide resolved
ingest/src/main/java/com/microsoft/azure/kusto/ingest/IngestClientBase.java
Show resolved
Hide resolved
ingest/src/main/java/com/microsoft/azure/kusto/ingest/ManagedStreamingIngestClient.java
Show resolved
Hide resolved
* If the size of the stream is bigger than {@value MAX_STREAMING_SIZE_BYTES}, it will fall back to the queued streaming client. | ||
* By default the policy for choosing a queued ingestion on the first try is the checking of weather the size of the estimated | ||
* raw stream size (a conversion to compressed CSV) is bigger than 4MB, it will fall back to the queued streaming client. | ||
* Use SourceInfo.size to override size estimations, alternatively - use setQueuingPolicy to override the predicate heuristics. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- the name is still wrong (it's SourceInfo.ge/settRawSizeInBytes)
- I think just putting it right next to the previous line with this wording is confusing, since it makes it seem that they are related.
...src/main/java/com/microsoft/azure/kusto/data/auth/endpoints/WellKnownKustoEndpointsData.java
Show resolved
Hide resolved
data/src/main/java/com/microsoft/azure/kusto/data/ClientFactory.java
Outdated
Show resolved
Hide resolved
ingest/src/main/java/com/microsoft/azure/kusto/ingest/ResourceAlgorithms.java
Show resolved
Hide resolved
ingest/src/main/java/com/microsoft/azure/kusto/ingest/source/FileSourceInfo.java
Show resolved
Hide resolved
Go over the pending comments, but approved in principle |
Added
an estimation against the 4mb limit after dividing or multiplying by some factor described by the consts:
// Regardless of the format, we don't want to stream more than 10mb
int MAX_STREAMING_STREAM_SIZE_BYTES = 10 * 1024 * 1024;
// Used against the users input of raw data size
int MAX_STREAMING_RAW_SIZE_BYTES = 6 * 1024 * 1024;
double JSON_UNCOMPRESSED_FACTOR = 1.5d;
int NON_BINARY_FACTOR = 2;
double BINARY_COMPRESSED_FACTOR = 2d;
double BINARY_UNCOMPRESSED_FACTOR = 1.5d;
This will also allow users to stream bigger than 4mb non-compressed data