Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transient IOError #331

Merged
merged 35 commits into from
Nov 27, 2023
Merged

Transient IOError #331

merged 35 commits into from
Nov 27, 2023

Conversation

ohadbitt
Copy link
Collaborator

@ohadbitt ohadbitt commented Nov 21, 2023

Fixed

  • IOException is now checked against known subclasses that should be retried or checked for string "time out" to decide for permanency
    Following exception was raised and considered permanent
    shaded.msdataflow.com.microsoft.azure.kusto.data.exceptions.DataServiceException: IOError when trying to retrieve CloudInfo
    at shaded.msdataflow.com.microsoft.azure.kusto.data.auth.CloudInfo.retrieveCloudInfoForCluster(CloudInfo.java:122)
    ..
    Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
    ..
    Caused by: org.apache.http.conn.HttpHostConnectException: Connect to ingest-pfclientquality.centralus.kusto.windows.net:443 [ingest-pfclientquality.centralus.kusto.windows.net/52.185.106.107] failed: Connection timed out (Connection timed out)

Aadded

  • Retry cloud info fetch

  • Split Utils to HttpPostUtils and Utils, so it now makes sense this class has the post() methods...
    Wanted to move all http related classes to a new http package as i found it too crowded - but couldnt move HttpClientProperties as its used by client and therefore a breaking change, so its half way now and added todo to finish on next major

@ohadbitt ohadbitt requested review from AsafMah and yihezkel November 21, 2023 06:46
Copy link

github-actions bot commented Nov 21, 2023

Test Results

0 tests   - 348   0 ✔️  - 348   0s ⏱️ -12s
0 suites  -   23   0 💤 ±    0 
0 files    -   23   0 ±    0 

Results for commit ca158ad. ± Comparison against base commit 938123c.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Nov 21, 2023

Test Results

309 tests   - 63   303 ✔️  - 63   3h 34m 24s ⏱️ + 1h 55m 0s
  23 suites  -   1       6 💤 ±  0 
  23 files    -   1       0 ±  0 

Results for commit cc4890e. ± Comparison against base commit 484c34b.

This pull request removes 73 and adds 10 tests. Note that renamed tests count towards both.
com.microsoft.azure.kusto.data.HttpClientFactoryTest ‑ testNullProperties
com.microsoft.azure.kusto.data.HttpClientFactoryTest ‑ testProperties
com.microsoft.azure.kusto.data.UncloseableStreamTest ‑ testMarkSupported_MatchesInnerMarkSupported
com.microsoft.azure.kusto.data.UncloseableStreamTest ‑ whenAvailableThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.UncloseableStreamTest ‑ whenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.UncloseableStreamTest ‑ whenMarkThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.UncloseableStreamTest ‑ whenReadThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.UncloseableStreamTest ‑ whenResetThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.UncloseableStreamTest ‑ whenSkipThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClientTest ‑ IngestFromStream_Success(boolean, boolean)[1]
…
com.microsoft.azure.kusto.data.UtilitiesTest ‑ isRetrieable
com.microsoft.azure.kusto.data.http.HttpClientFactoryTest ‑ testNullProperties
com.microsoft.azure.kusto.data.http.HttpClientFactoryTest ‑ testProperties
com.microsoft.azure.kusto.data.http.UncloseableStreamTest ‑ testMarkSupported_MatchesInnerMarkSupported
com.microsoft.azure.kusto.data.http.UncloseableStreamTest ‑ whenAvailableThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.http.UncloseableStreamTest ‑ whenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.http.UncloseableStreamTest ‑ whenMarkThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.http.UncloseableStreamTest ‑ whenReadThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.http.UncloseableStreamTest ‑ whenResetThenClose_ThenInnerStreamIsNotClosed
com.microsoft.azure.kusto.data.http.UncloseableStreamTest ‑ whenSkipThenClose_ThenInnerStreamIsNotClosed

♻️ This comment has been updated with latest results.

@ohadbitt ohadbitt requested a review from yogilad November 21, 2023 19:04
Copy link

codecov bot commented Nov 22, 2023

Codecov Report

Attention: 135 lines in your changes are missing coverage. Please review.

Comparison is base (484c34b) 58.84% compared to head (cc4890e) 56.00%.

Files Patch % Lines
...microsoft/azure/kusto/data/http/HttpPostUtils.java 23.85% 82 Missing and 1 partial ⚠️
...com/microsoft/azure/kusto/data/auth/CloudInfo.java 2.56% 38 Missing ⚠️
...ain/java/com/microsoft/azure/kusto/data/Utils.java 45.00% 10 Missing and 1 partial ⚠️
...ava/com/microsoft/azure/kusto/data/ClientImpl.java 0.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master     #331      +/-   ##
============================================
- Coverage     58.84%   56.00%   -2.85%     
+ Complexity      822      797      -25     
============================================
  Files           108      109       +1     
  Lines          4046     4071      +25     
  Branches        424      429       +5     
============================================
- Hits           2381     2280     -101     
- Misses         1476     1600     +124     
- Partials        189      191       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@AsafMah AsafMah self-requested a review November 26, 2023 12:03
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;

class ManagedStreamingIngestClientTest {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you delete this whole file (if you moved it I can't see)

@@ -272,11 +272,6 @@
<artifactId>resilience4j-retry</artifactId>
<version>${resilience4j.version}</version>
</dependency>
<dependency>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we know it's ok to remove?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was here only because the verify phase complained

@@ -90,49 +92,69 @@ public static CloudInfo retrieveCloudInfoForCluster(String clusterUrl,
return cloudInfo;
}

CloudInfo result;
for (int i = 0;; i++) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "ExponentialRetry" class did not fit here?

@AsafMah AsafMah self-requested a review November 27, 2023 06:50
@ohadbitt ohadbitt merged commit 88ddc35 into master Nov 27, 2023
8 checks passed
@ohadbitt ohadbitt deleted the IoErrorsAreTransient branch November 27, 2023 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants