Using --sync-forever results in agents calling Connect() in an infinite loop

[andrewsykim]

#285 introduced a new flag called --sync-forever on the agent, which allows for dynamically updating the server count returned from the server to the client. However, enabling --sync-forever results in the agent calling Connect() repeatedly, which is evident from the "Connect to" messages in the following logs:

I0331 03:24:05.232705       1 options.go:114] Keepalive time set to 1m0s.
I0331 03:24:05.232723       1 options.go:116] AgentIdentifiers set to .
I0331 03:24:05.232731       1 options.go:117] WarnOnChannelLimit set to false.
I0331 03:24:05.232738       1 options.go:118] SyncForever set to true.
I0331 03:24:05.259984       1 client.go:253] "Connect to" server="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
I0331 03:24:05.260258       1 clientset.go:222] "sync added client connecting to proxy server" serverID="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
I0331 03:24:05.260458       1 client.go:355] "Start serving" serverID="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
E0331 03:24:08.970173       1 client.go:505] "connection read failure" err="read tcp 10.24.1.10:53782->10.24.1.6:10250: use of closed network connection" connectionID=1
I0331 03:24:08.970541       1 client.go:486] "Exiting remoteToProxy" connectionID=1
E0331 03:24:08.970612       1 client.go:541] "conn write failure" err="write tcp 10.24.1.10:53782->10.24.1.6:10250: use of closed network connection" connectionID=1
I0331 03:24:08.970635       1 client.go:524] "Exiting proxyToRemote" connectionID=1
I0331 03:24:10.574573       1 client.go:253] "Connect to" server="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
I0331 03:24:16.061383       1 client.go:253] "Connect to" server="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
I0331 03:24:24.076367       1 client.go:253] "Connect to" server="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
E0331 03:24:30.071545       1 client.go:505] "connection read failure" err="read tcp 10.24.1.10:53816->10.24.1.6:10250: use of closed network connection" connectionID=4
I0331 03:24:30.071801       1 client.go:486] "Exiting remoteToProxy" connectionID=4
E0331 03:24:30.071919       1 client.go:541] "conn write failure" err="write tcp 10.24.1.10:53816->10.24.1.6:10250: use of closed network connection" connectionID=4
I0331 03:24:30.071940       1 client.go:524] "Exiting proxyToRemote" connectionID=4
E0331 03:24:30.170689       1 client.go:541] "conn write failure" err="write tcp 10.24.1.10:53812->10.24.1.6:10250: use of closed network connection" connectionID=3
E0331 03:24:30.171054       1 client.go:505] "connection read failure" err="read tcp 10.24.1.10:53812->10.24.1.6:10250: use of closed network connection" connectionID=3
I0331 03:24:30.171501       1 client.go:524] "Exiting proxyToRemote" connectionID=3
I0331 03:24:30.171566       1 client.go:486] "Exiting remoteToProxy" connectionID=3
I0331 03:24:35.843569       1 client.go:253] "Connect to" server="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
I0331 03:24:53.447789       1 client.go:253] "Connect to" server="77b12a12-5b6f-48fa-b087-d2c15d8607ed"
I0331 03:25:20.554573       1 client.go:253] "Connect to" server="77b12a12-5b6f-48fa-b087-d2c15d8607ed"

It seems unexpected that --sync-forever would force a redial of the stream even though there are no server count updates. More importantly, it seems possible that calling Connect() repeatedly could be disruptive to existing connections since the stream for an agent as seen by the server could change mid-flight (although I'm mostly speculating about this and don't have evidence this is true).

I also noticed that after enabling --sync-forever, I see the following messages from the agent occasionally which could be related:

E0331 07:32:58.612213       1 client.go:371] "could not read stream" err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\"

From the gRPC docs, this seems related to keepalives.

[andrewsykim]

@cheftako any ideas if this is expected behavior when using --sync-forever?

[cheftako]

The agent calling Connect() repeatedly (forever) is expected. However it should not effect the existing streams/tunnels. It should not be possible to change the stream to a server mid-flight. The handshake is supposed to always cancel the new connection/stream when it realizes this is a connection/stream to an already connected server.

[andrewsykim]

Closing this issue since this seems to be working as expected.