Cluster members don't discover each other on initial boot, but all works well after one of the containers is restarted

[laszlomiklosik]

I am using a cluster in "standalone mode" with 2 Keycloak nodes running on different AWS EC2 machines (from different availability zones). The 2 Keycloak instances can reach each other on port 7600 via the Docker host IPs).
I was able to mount the latest TCPPING.cli script to the latest official Keycloak image (11.0.2) instead of using this custom Keycloak image (which is very great inspiration and which I also used to experiment with TCPPING and JDBCPING!). The cluster works as expected.

I encounter 1 problem though with the initial pairing of the cluster members: my experience is that in order for the cluster nodes to initially discover each other, a restart of one of the containers is necessary. More exactly: 2 newly created Keycloak containers won't discover each other until I restart one of them. After this initial pairing all works as expected, but this is a bit annoying for the initial run of the Keycloak cluster in all new environments and will also require extra restarts after each Keycloak upgrade.

I tried to use a lighter alternative to restarting Keycloak: executing a Wildfly reload using /opt/jboss/keycloak/bin/jboss-cli.sh --connect --command=":reload"(which would have been easy to add to TCPPING.cli), but this is not sufficient.

I experienced the same with your image as well. Let me know if you have a solution to this problem or in case you didn't encounter it at all.

[ivangfr]

Hi @laszlomiklosik , thanks for the comment and sorry for the late response.

Nowadays, I don't have resources to simulate 2 Keycloak nodes on any cloud provider using TCPPING. So, I didn't try it extensively. On the other hand, JDBCPING was easier for me to test and run it locally in my machine.

Whenever I have time, I will create 2 machines using Vagrant and try to reproduce what you have perfectly described locally.

I will let you know about my finds!

[ivangfr]

Hey @laszlomiklosik , I've created some virtual machines using Vagrant to test the discovery protocols TCPPING and JDBC_PING. Btw, could you please share the TCPPING.cli that you are using? Thanks

[laszlomiklosik]

I used this exact version https://github.com/ivangfr/keycloak-clustered/blob/master/11.0.2/cli/TCPPING.cli and reproduce the described problem. I am now using JDBC_PING and I did not focus on finding a solution/workaround to get TCPPING work.