Failed to download artifact, i/o timeout error using iptables-nft and Calico

[Eagleman7]

Hello!

I recently build a new cluster with packer, terraform and ansible (kubespray) on Red Hat Enterprise Linux 8. This uses iptables-nft and Calico. All the pods are running on my new cluster, however when I deploy flux2 I get a timeout error:

I tried both with and without --network-policy=false.

[root@control]# flux bootstrap github   --owner="${GITHUB_USER}"   --repository="myrepo-k8s-refined"   --path=kubernetes/clusters/staging   --branch=main   --personal --network-policy=false
► connecting to github.com
✔ repository cloned
✚ generating manifests
✔ components are up to date
► installing components in flux-system namespace
namespace/flux-system unchanged
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io configured
serviceaccount/helm-controller unchanged
serviceaccount/kustomize-controller unchanged
serviceaccount/notification-controller unchanged
serviceaccount/source-controller unchanged
clusterrole.rbac.authorization.k8s.io/crd-controller-flux-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-flux-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/crd-controller-flux-system unchanged
service/notification-controller unchanged
service/source-controller unchanged
service/webhook-receiver unchanged
deployment.apps/helm-controller configured
deployment.apps/kustomize-controller configured
deployment.apps/notification-controller configured
deployment.apps/source-controller configured
◎ verifying installation
✔ install completed
► generating sync manifests
► applying sync manifests
◎ waiting for cluster sync
✗ failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/5414ac00f39698d89400294fa928602d3379d6f4.tar.gz giving up after 10 attempt(s): Get "http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/5414ac00f39698d89400294fa928602d3379d6f4.tar.gz": dial tcp 10.233.54.75:80: i/o timeout

[root@control]# flux check
► checking prerequisites
✔ kubectl 1.20.4 >=1.18.0
✔ Kubernetes 1.19.9 >=1.16.0
► checking controllers
✔ source-controller: healthy
► ghcr.io/fluxcd/source-controller:v0.11.0
✔ kustomize-controller: healthy
► ghcr.io/fluxcd/kustomize-controller:v0.11.0
✔ helm-controller: healthy
► ghcr.io/fluxcd/helm-controller:v0.9.0
✔ notification-controller: healthy
► ghcr.io/fluxcd/notification-controller:v0.12.0
✔ all checks passed

I wonder what I am doing wrong, I am also curious who 10.233.54.75 is in this case, I cannot find that IP using this command on any pods:

[root@control]# kubectl get pod -o wide -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE    IP             NODE    NOMINATED NODE   READINESS GATES
flux-system   helm-controller-5bbd94c75-wsmkq            1/1     Running   0          16m    10.233.90.4    node1   <none>           <none>
flux-system   kustomize-controller-97dbdcb99-jggjg       1/1     Running   0          16m    10.233.96.3    node2   <none>           <none>
flux-system   notification-controller-d66bdc787-7lxvr    1/1     Running   0          16m    10.233.90.5    node1   <none>           <none>
flux-system   source-controller-86d84856cf-bzccp         1/1     Running   0          16m    10.233.92.3    node3   <none>           <none>
kube-system   calico-kube-controllers-7fbf9b4bbb-gngbv   1/1     Running   0          111m   192.168.6.22   node3   <none>           <none>
kube-system   calico-node-29s6h                          1/1     Running   0          112m   192.168.6.20   node1   <none>           <none>
kube-system   calico-node-d5ntm                          1/1     Running   0          112m   192.168.6.21   node2   <none>           <none>
kube-system   calico-node-rprj6                          1/1     Running   0          112m   192.168.6.22   node3   <none>           <none>
kube-system   coredns-7677f9bb54-lbpvm                   1/1     Running   0          111m   10.233.96.1    node2   <none>           <none>
kube-system   coredns-7677f9bb54-m4788                   1/1     Running   0          111m   10.233.92.1    node3   <none>           <none>
kube-system   dns-autoscaler-5b7b5c9b6f-x9k26            1/1     Running   0          111m   10.233.90.1    node1   <none>           <none>
kube-system   kube-apiserver-node1                       1/1     Running   0          113m   192.168.6.20   node1   <none>           <none>
kube-system   kube-apiserver-node2                       1/1     Running   0          113m   192.168.6.21   node2   <none>           <none>
kube-system   kube-apiserver-node3                       1/1     Running   0          113m   192.168.6.22   node3   <none>           <none>
kube-system   kube-controller-manager-node1              1/1     Running   0          113m   192.168.6.20   node1   <none>           <none>
kube-system   kube-controller-manager-node2              1/1     Running   0          113m   192.168.6.21   node2   <none>           <none>
kube-system   kube-controller-manager-node3              1/1     Running   0          113m   192.168.6.22   node3   <none>           <none>
kube-system   kube-proxy-2x824                           1/1     Running   0          112m   192.168.6.20   node1   <none>           <none>
kube-system   kube-proxy-67xr8                           1/1     Running   0          112m   192.168.6.21   node2   <none>           <none>
kube-system   kube-proxy-whc6w                           1/1     Running   0          112m   192.168.6.22   node3   <none>           <none>
kube-system   kube-scheduler-node1                       1/1     Running   0          113m   192.168.6.20   node1   <none>           <none>
kube-system   kube-scheduler-node2                       1/1     Running   0          113m   192.168.6.21   node2   <none>           <none>
kube-system   kube-scheduler-node3                       1/1     Running   0          113m   192.168.6.22   node3   <none>           <none>
kube-system   nodelocaldns-f9clr                         1/1     Running   0          111m   192.168.6.22   node3   <none>           <none>
kube-system   nodelocaldns-xjsvm                         1/1     Running   0          111m   192.168.6.20   node1   <none>           <none>
kube-system   nodelocaldns-zrvzv                         1/1     Running   0          111m   192.168.6.21   node2   <none>           <none>

Any help is appreciated!

[stefanprodan]

My guess is that your Kubernetes network is broken, before deploying Flux make sure HTTP calls work inside your cluster.

I am also curious who 10.233.54.75 is in this case, I cannot find that IP using this command on any pods

It's the source-controller.flux-system.svc.cluster.local. ClusterIP.

[Eagleman7]

Thanks @stefanprodan,

I went on to the calico slack support channel, and together with 2 other people we figured out it was caused by vSphere blocking encapsulated trafffic between nodes. We disabled encapsulation and now it seems to work!