DNS resolution unreliable in v0.24.0 when using network policies #3713
Comments
|
/assign @aojea |
|
(Just filing this early since 0.24.0 just came out. Will try to create a self-contained test case. I have not yet checked whether the same problem affects early Kind releases using the built-in kindnet; and whether the problem affects clusters not using |
|
I wonder if this only applies to external DNS, or in some specific environment (kernel / loaded modules maybe?) We are making a lot of in-cluster DNS queries in CI and I haven't noticed an increase in flakiness, but we're also not actually running network policy tests yet and the majority of tests don't make egress requests (excluding pulling images at the node level). |
|
hmm, it seems related to this kubernetes-sigs/kube-network-policies#12 @jglick appreciate if you give me a reproducer |
|
The problem affects Java processes, not just Not reproducible just running In my full reproducer, the problem is fixed simply by declining to create any I will try to start creating a minimal reproducer I can share. |
hmm, then it maybe related to the way the java resolver performs the dns resolution , IIRC it uses a stub resolver and not libc https://biriukov.dev/docs/resolver-dual-stack-application/8-stub-resolvers-in-languages/#84-java-and-netty once we have the reproducer we can get to the bottom, appreciate the effort |
|
@aojea java in addition to curl, presumably curl is using libc (though we don't know which build and which libc) |
|
Might be glibc vs muslc |
|
OK, I have a reproducer. Simple apiVersion: apps/v1
kind: Deployment
metadata:
name: x
labels:
app: x
spec:
replicas: 1
selector:
matchLabels:
app: x
template:
metadata:
labels:
app: x
spec:
containers:
- name: x
image: curlimages/curl
command:
- sleep
args:
- infinity
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: x
spec:
podSelector:
matchLabels:
app: x
ingress:
- ports:
- port: 9999
protocol: TCP
from:
- podSelector:
matchLabels:
app: 'y'And then repeatedly run kubectl exec deploy/x -- time nslookup www.google.comSometimes this will succeed instantly; most times it will wait precisely 2.5s before succeeding. If you kubectl delete networkpolicies.networking.k8s.io xthen the lookup begins reliably completing immediately. Note that the policy is controlling ingress to the test pod while the affected action should involve only outbound connections. |
jglick
changed the title
|
@jglick can you check if at least one of the coredns pods is in the same node than the affected pod? |
WDYM? There is only one node here. From a freshly created cluster, $ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6f6b679f8f-4kzpv 1/1 Running 0 29s
kube-system coredns-6f6b679f8f-ddrwm 1/1 Running 0 29s
kube-system etcd-kind-control-plane 1/1 Running 0 36s
kube-system kindnet-bw8b7 1/1 Running 0 29s
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 35s
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 35s
kube-system kube-proxy-kxdmq 1/1 Running 0 29s
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 35s
local-path-storage local-path-provisioner-57c5987fd4-5dm47 1/1 Running 0 29sor $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane 33s v1.31.0after a few seconds as expected. |
|
I see, I think the problem happen when the destination is in the same node ... smells as a kernel bug |
|
can you edit the coredns deployment to set only one replica and try again?
|
|
Indeed after kubectl -n kube-system scale deployment coredns --replicas 1the problem seems to go away. |
|
It seems a kernel bug, I will track it in the original project kubernetes-sigs/kube-network-policies#12 (comment) , in the meantime as a workaround
sorry for not having a better answer yet and thanks for reporting feedback |
Not at all! The workaround seems quite straightforward. For that matter, why not set this deployment to one replica by default in Kind, or at least single-node Kind? It does not seem like it really needs to be highly available or scalable in a test cluster. |
|
Edit: Not related to this issue. See comment below. I'm running into the same issue but scaling coredns down to a single replica does not fix the issue. In my case the issue comes from I'm running a single node Kind cluster and some, but not all, |
there are a lot of subtle parameters that we need to validate to confirm is the same issue. Can you paste your applied network policies manifests and confirm your kind version and kind image? |
|
Sorry about that - this is our fault. We didn't realize that OOTB NetworkPolicy enforcement was added in v0.24.0 and some of our existing NetworkPolicies started blocking access to DNS once we upgraded to v0.24.0 from v0.23.0. PEBKAC. |
|
So in summary we have some network policies that are now taking effect, and there's no bug? If not, please re-open. |
|
@BenTheElder: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
There is a bug, as stated in the original description. There is a workaround but users should not need to dig for it. Can Kind just switch to one replica by default, as in #3713 (comment)? |
I'm talking with the kernel maintainers to see if we can fix it in the root, if it takes time we'll try to figure out other workarounds, but I prefer to not use kind to compensate other components bugs, so in order of preference is :)
We have a reproducer so I expect this or the next week we can have more progress https://bugzilla.netfilter.org/show_bug.cgi?id=1766 Sorry for the inconvenience |
|
Identified the problem in the kernel, a fix will be sent soon https://lore.kernel.org/netfilter-devel/20240918095404.GA11988@breakpoint.cc/T/#m0d5934b598507bcd0b25c04cdb62a964880dd751 |
|
I guess that will take a while to roll out. I think this merits a mention in https://github.com/kubernetes-sigs/kind/releases/tag/v0.24.0 with the known workaround? |
we are progressing, slowly .... just updating with the developments, let's see if we can have the problem solved by kind or kindnet without relying on the workaround |
|
Ok, got something that solves the problem without cutting functionality kubernetes-sigs/kube-network-policies#95, once it merges I need to cut a new release and then update kindnet |
aojea
added
the
priority/important-soon
Even though we have a work-around is specific to DNS traffic, it is still possible for the underlying kernel bug to be triggered with rules applied to other UDP traffic. Until Kind moves to a kernel version that incorporates the patch for the kernel bug, it might be a good idea to add a known issue regarding the potential for problematic behavior with other UDP-based network policy rules. |
kind doesn't control the kernel version, the user does (kind only creates containers). If we think we have an unmitigated issue depending on the kernel patch, we'll need to at least add a known issues entry for this (see https://kind.sigs.k8s.io/docs/contributing/development/#documentation if anyone wants to help update https://kind.sigs.k8s.io/docs/user/known-issues/). I haven't looked at the workaround myself yet so I won't speculate on that part. Can folks here confirm the workaround at least solves the originally reported DNS issue for them? The best way is:
|
|
Rechecked #3713 (comment). Still reproduces (intermittently) in 0.24.0; does not reproduce in local build of 10e058c. So I think the effectiveness of #3752 is confirmed. |
|
I tested it thoroughly but the more people acking the better, thanks all |
|
Any plans for an 0.25 with this fix? |
|
I'm working on that currently :-) |
|
Annnd it's out |
I tried using the new built-in
NetworkPolicysupport in v0.24.0 (#842 / #3612) instead of Calico. It did not work well, apparently due to DNS problems.What happened:
Requests to either pods inside the cluster by service name, or external hosts, were often delayed or failed due to DNS failures.
What you expected to happen:
Requests should succeed immediately, up to the responsiveness of the service.
How to reproduce it (as minimally and precisely as possible):
Will try to put together a minimal test case. In the meantime:
In my original setup a cluster was created with
and I then installed Calico using
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/calico.yaml kubectl -n kube-system set env daemonset/calico-node FELIX_IGNORELOOSERPF=trueand defined some network policies. This worked fine, including after creating a fresh cluster using v0.24.0; for example from a pod I could access another pod in the same namespace using a service name
provided of course that
othersvcpermitted that incoming connection; the command prints content essentially instantly. Contacting external (Internet) services was also as reliable as the site itself.After deleting the Calico setup and rerunning the exact same scenario, sometimes the
curlcommand works instantly as before; sometimes it fails and claims theothersvchost could not be found; other times it works, but only after a delay of several seconds. Contacting hosts on the Internet is also unreliable, sometimes giving a hostname resolution error. For examplesometimes works and sometimes does not
whereas
works reliably.
Environment:
Kind v024.0, Docker 27.1.2, Ubuntu Noble
The text was updated successfully, but these errors were encountered: