keep getting ssh handshake errors notifications

[arunalakmal]

Hi Everyone

I have been using Flux with Multi cluster setup with single repository. My requirement was to make sure all the environments have the similar configurations as everything identical in each and every cluster. I need to mention that everything is perfectly working and I don't see any reconcile issues or failure to update the changes made to the repo to the system.

But I have enabled the Slack Notification and I'm keep getting this ssh authentication error and I can't find the issue behind that, but no issues or failures reported and everything is working as expected.

Frequency, between these notifications is around 15 minutes.

Why I'm keep getting this notification? I really appreciate if you can help me to understand the issue or fix any issues I have.

here is the snap of my Slack Channel. I'm pretty confused 🤔🤯

[stefanprodan]

Can you please post here the output of:

• flux check
• flux get sources git
• flux logs --level=error
• kubectl describe gitrepository flux-system -n flux-system

[arunalakmal]

Thank you for the response @stefanprodan

Actually, I checked the error logs of the source controllers in each and every environments and found that the dev environment source controller is having the issue. This dev environment was deployed using a separate branch in my main repo while all other prod environments deployed through the master branch. Could that be a problem?

Here is the output you requested

flux check:

► checking prerequisites
✗ flux 0.11.0 <0.13.3 (new version is available, please upgrade)
✔ kubectl 1.20.2 >=1.18.0-0
✔ Kubernetes 1.18.9-eks-d1db3c >=1.16.0-0
► checking controllers
✗ helm-controller: deployment not ready
► ghcr.io/fluxcd/helm-controller:v0.9.0
✗ image-automation-controller: deployment not ready
► ghcr.io/fluxcd/image-automation-controller:v0.7.0
✗ image-reflector-controller: deployment not ready
► ghcr.io/fluxcd/image-reflector-controller:v0.7.1
✗ kustomize-controller: deployment not ready
► ghcr.io/fluxcd/kustomize-controller:v0.10.0
✗ notification-controller: deployment not ready
► ghcr.io/fluxcd/notification-controller:v0.11.0
✗ source-controller: deployment not ready
► ghcr.io/fluxcd/source-controller:v0.10.0
✔ all checks passed

flux get sources git : (removed the repo name)

NAME       	READY	MESSAGE                                                                                                                                                                                                                 	REVISION                                            	SUSPENDED
flux-system	False	unable to clone 'ssh://git@gitlab.com/xxxxxxxxx/xxxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain	dev-cluster/6bd44e1ecdc3df5516129a0c761495e2c04ae245	False

flux logs --level=error : (Entire log was filed with the below error)

2021-05-09T01:48:57.034Z error GitRepository/flux-system.flux-system - Reconciler error unable to clone 'ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-05-09T02:05:37.679Z error GitRepository/flux-system.flux-system - Reconciler error unable to clone 'ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-05-09T02:22:18.219Z error GitRepository/flux-system.flux-system - Reconciler error unable to clone 'ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-05-09T02:38:58.839Z error GitRepository/flux-system.flux-system - Reconciler error unable to clone 'ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-05-09T02:55:39.440Z error GitRepository/flux-system.flux-system - Reconciler error unable to clone 'ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-05-09T03:12:20.020Z error GitRepository/flux-system.flux-system - Reconciler error unable to clone 'ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-05-09T03:29:00.618Z error GitRepository/flux-system.flux-system - Reconciler error unable to clone 'ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

kubectl describe gitrepository flux-system -n flux-system:

Name:         flux-system
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/checksum=2eb67099584d601d26334a45562d25bf4ab1829d
              kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>
API Version:  source.toolkit.fluxcd.io/v1beta1
Kind:         GitRepository
Metadata:
  Creation Timestamp:  2021-04-09T12:09:33Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:  1
  Managed Fields:
    API Version:  source.toolkit.fluxcd.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
      f:spec:
        .:
        f:gitImplementation:
        f:interval:
        f:ref:
          .:
          f:branch:
        f:secretRef:
          .:
          f:name:
        f:timeout:
        f:url:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-04-09T12:15:33Z
    API Version:  source.toolkit.fluxcd.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:kustomize.toolkit.fluxcd.io/checksum:
          f:kustomize.toolkit.fluxcd.io/name:
          f:kustomize.toolkit.fluxcd.io/namespace:
    Manager:      kustomize-controller
    Operation:    Update
    Time:         2021-05-07T01:07:58Z
    API Version:  source.toolkit.fluxcd.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizers.fluxcd.io":
      f:status:
        .:
        f:artifact:
          .:
          f:checksum:
          f:lastUpdateTime:
          f:path:
          f:revision:
          f:url:
        f:conditions:
        f:observedGeneration:
        f:url:
    Manager:         source-controller
    Operation:       Update
    Time:            2021-05-08T10:48:18Z
  Resource Version:  53099311
  Self Link:         /apis/source.toolkit.fluxcd.io/v1beta1/namespaces/flux-system/gitrepositories/flux-system
  UID:               5d01db1b-0cb2-4d10-b73e-038f97b25893
Spec:
  Git Implementation:  go-git
  Interval:            1m0s
  Ref:
    Branch:  dev-cluster
  Secret Ref:
    Name:   flux-system
  Timeout:  20s
  URL:      ssh://git@gitlab.com/xxxxxxx/xxxxxxx-gitops-administrator
Status:
  Artifact:
    Checksum:          03bbff486b987f4970c9b8e967cfd34f772c0ba1
    Last Update Time:  2021-05-07T01:07:42Z
    Path:              gitrepository/flux-system/flux-system/6bd44e1ecdc3df5516129a0c761495e2c04ae245.tar.gz
    Revision:          dev-cluster/6bd44e1ecdc3df5516129a0c761495e2c04ae245
    URL:               http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/6bd44e1ecdc3df5516129a0c761495e2c04ae245.tar.gz
  Conditions:
    Last Transition Time:  2021-05-07T20:14:06Z
    Message:               unable to clone 'ssh://git@gitlab.com/xxxxxxxxx/xxxxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
    Reason:                GitOperationFailed
    Status:                False
    Type:                  Ready
  Observed Generation:     1
  URL:                     http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Events:
  Type    Reason  Age                    From               Message
  ----    ------  ----                   ----               -------
  Normal  error   7m21s (x125 over 31h)  source-controller  unable to clone 'ssh://git@gitlab.com/xxxxxxxxx/xxxxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

BTW, I retried to deploy it in to the cluster and did see the below output and the error:

► connecting to gitlab.com
✔ repository cloned
✚ generating manifests
✔ components manifests pushed
► generating sync manifests
✔ sync manifests pushed
► applying sync manifests
◎ waiting for cluster sync
✗ unable to clone 'ssh://git@gitlab.com/xxxxxxxxx/xxxxxxxxxxx-gitops-administrator', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Definitely, there is an error and what could be the breaking point of this. We didn't change anything in the Flux deployments other than the application manifest updates which needed in the dev environment. Thanks.

[arunalakmal]

@stefanprodan I completely, uninstalled the Flux and reinstalled. it worked this time and now I can see that it is fetching the repo perfectly and no ssh handshake errors. Not sure what could be the reason, any thoughts? what could go wrong? I'm not sure this is an environment specific issue or not as there were no similar cases on the internet. 🤔

Let me monitor this for couple of days and share the output. Then, we can close this off. Thanks for the response again!

[stefanprodan]

You need to update the CLI and the controllers to v0.13. In earlier versions, bootstrapping on the same path for different branches would result in the same deploy key name. So now the other cluster will lose auth, its deploy key was overwritten.

[arunalakmal]

Ohh.... So, at this point, can you recommend the steps that, I need to follow.

In my current production clusters:

1. uninstall the current Flux deployments
2. update the CLI to v0.13
3. reinstall the controllers

would work?

I have checked the flux get sources git and still see it fetched the latest commit and "READY".

NAME       	READY	MESSAGE                                                          	REVISION                                       	SUSPENDED
flux-system	True 	Fetched revision: master/5ae46deffb89363c69458e0ef84fd14600cd35c1	master/5ae46deffb89363c69458e0ef84fd14600cd35c1	False

Appreciate your recommendations. Thanks.

[arunalakmal]

As per the documentation, I have upgraded the flux CLI and using bootstrap command (I used it to deploy the controllers). But flux check command says my deployments are not ready. But all the controllers are healthy. Is this expected?

► checking prerequisites
✔ kubectl 1.20.2 >=1.18.0-0
✔ Kubernetes 1.18.9-eks-d1db3c >=1.16.0-0
► checking controllers
✗ helm-controller: deployment not ready
► ghcr.io/fluxcd/helm-controller:v0.10.0
✗ image-automation-controller: deployment not ready
► ghcr.io/fluxcd/image-automation-controller:v0.9.1
✗ image-reflector-controller: deployment not ready
► ghcr.io/fluxcd/image-reflector-controller:v0.9.1
✗ kustomize-controller: deployment not ready
► ghcr.io/fluxcd/kustomize-controller:v0.12.0
✗ notification-controller: deployment not ready
► ghcr.io/fluxcd/notification-controller:v0.13.0
✗ source-controller: deployment not ready
► ghcr.io/fluxcd/source-controller:v0.12.1
✔ all checks passed

This is the controllers state:

kubectl get pods -n flux-system                                                                                                               ✔  5391  14:49:02
NAME                                           READY   STATUS    RESTARTS   AGE
helm-controller-848675645f-5l8vl               1/1     Running   0          94s
image-automation-controller-788d84f47d-wnckh   1/1     Running   0          94s
image-reflector-controller-578b9589c7-szwb2    1/1     Running   0          94s
kustomize-controller-549ddf9686-nx4b7          1/1     Running   0          94s
notification-controller-75f97596cb-59cjm       1/1     Running   0          94s
source-controller-6845477565-5jnpm             1/1     Running   0          92s

kubectl get deploy -n flux-system                                                                                                             ✔  5394  14:54:49
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
helm-controller               1/1     1            1           5h24m
image-automation-controller   1/1     1            1           5h24m
image-reflector-controller    1/1     1            1           5h24m
kustomize-controller          1/1     1            1           5h24m
notification-controller       1/1     1            1           5h24m
source-controller             1/1     1            1           5h24m

[arunalakmal]

I think now my clusters are pretty much stable. I have upgraded the CLI to v0.13 as informed by @stefanprodan and tried to upgrade the controllers using the same bootstrap command I used to deploy. The controllers which were having issues didn't work and failed the upgrade throwing the same ssl handshake error while others successful the bootstrapping command. For the failed controller upgrades (Which were having issues), I had to uninstall completely and redeployed with the same bootstrap commands. It's been a while now and my entire environment is stable. I think we are all good and @stefanprodan thank you for the response.