Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration tests are flaky and sometimes time out #649

Open
carreter opened this issue Aug 21, 2024 · 1 comment
Open

Integration tests are flaky and sometimes time out #649

carreter opened this issue Aug 21, 2024 · 1 comment

Comments

@carreter
Copy link
Contributor

The tests under ./tests seem to take about as long as the configured test timeout, so they will flake pretty often.

Failures look like this:

--- FAIL: TestProxyDial_RequestCancelled_Concurrent_GRPC (61.89s)
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:426: Agent connections leaked: 
        
        Diff:
        --- metric output does not match expectation; want
        +++ got:
        @@ -2,3 +2,3 @@
         # TYPE konnectivity_network_proxy_agent_open_endpoint_connections gauge
        -konnectivity_network_proxy_agent_open_endpoint_connections 0
        +konnectivity_network_proxy_agent_open_endpoint_connections 4

I think this is related to a behavior I've noticed where the agent will continue attempting to establish connections even after receiving a SIGTERM. I'm having trouble reproducing it right now, will elaborate further in a comment.

@carreter
Copy link
Contributor Author

Here's the issue I was mentioning of connections being attempted even after a SIGTERM:

I0821 17:21:18.808036       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:24.111147       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:24.111373       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:31.262424       1 server.go:124] "Received first signal" signal="terminated"                                                                                                                                                                                                                                                                                                                                                                                                     
I0821 17:21:31.719962       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:31.720167       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:42.039194       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:42.039478       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:52.514095       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:52.514299       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
Stream closed EOF for kube-system/konnectivity-agent-668fc5b9dd-ztzzp (konnectivity-agent-container)                                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant