Kind CNI Does Not Support Default Deny Network Policy

[network-charles]

What happened:
After applying a default deny policy, the pod could still communicate. I had to install Calico for it to work.

What you expected to happen:
Pods shouldn't be able to communicate between themselves.

How to reproduce it (as minimally and precisely as possible):

# Create cluster 
kind create cluster
# Create 2 pods and expose them
kubectl run frontend --image=nginx
kubectl run backend --image=nginx
kubectl expose pod frontend --port 80
kubectl expose pod backend --port 80

# Create default-deny network policy
kubectl apply -f https://raw.githubusercontent.com/killer-sh/cks-course-environment/master/course-content/cluster-setup/network-policies/default-deny/default-deny.yaml

# Try to curl from frontend to backend and vice-versa
kubectl exec frontend -- curl backend
kubectl exec backend -- curl frontend

Anything else we need to know?:
No

Environment:

• kind version: (use kind version): kind v0.23.0 go1.21.10 linux/amd64
• Runtime info: (use docker info):

Client: Docker Engine - Community
 Version:    27.1.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 11
 Server Version: 27.1.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1023-aws
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.738GiB
 Name: ip-172-31-23-63
 ID: 639e9164-0580-4734-a476-1ff138f00013
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release):

                                                                                                  PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kubernetes version: (use kubectl version):

Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

• Any proxies or other special environment settings?: No

[aojea]

This has been implemented recently and is only available on the main branch

[BenTheElder]

NetworkPolicy isn't part of Kubernetes conformance and current kind releases did not implement NetworkPolicy at all

That's not a bug.

However, there will be a network policy implementation feature in the next release, which will happen around when Kubernetes 1.31 releases if all goes well.

/kind feature
/remove-kind bug

You can try this in advance if you install the latest kind sources https://kind.sigs.k8s.io/docs/user/quick-start/#installation
This should already be implemented in main.

[network-charles]

Ah, I see. Thanks for the clarification.

[thepaulmacca]

I've just tried this with the latest release, and the policy in the example above still isn't being applied

Can anyone else confirm?

[BenTheElder]

I can't repro, I see that if I try to curl it hangs, but if I kubectl delete -f https://raw.githubusercontent.com/killer-sh/cks-course-environment/master/course-content/cluster-setup/network-policies/default-deny/default-deny.yaml then it returns the nginx welcome page.

[BenTheElder]

I installed kind v0.24.0 then ran kind create cluster with no options as in the snippet from #3705 (comment)

Are you doing anything different? It's totally possible that we have a bug, but I seem to see the expected behavior when using kind v0.24.0 and the default node image (which is Kubernetes v1.31.0, with the https://github.com/kubernetes-sigs/kube-network-policies integration in kindnetd)

[BenTheElder]

Also @network-charles sorry for the confusion, this has been a longstanding request / gap and you're far from the first to see this as a bug / surprise.

It was an intentional choice previously to focus on shipping only what is required for fully conformant APIs and not anything less standardized as a way to help prevent scope creep and encourage developing with portable APIs and avoid ... competition over whose third party implementation would be provided in Kubernetes's upstream test tools.

... but network policy is part of the core standards, and now we have the minimal upstream implementation at https://github.com/kubernetes-sigs/kube-network-policies, so as of this morning https://github.com/kubernetes-sigs/kind/releases/tag/v0.24.0 is finally out with network policy support out of the box.

[BenTheElder]

Previously tracked in: #842

[network-charles]

I can confirm that this has been fixed in v0.24.0.

[thepaulmacca]

Does the fact that I'm using WSL2 matter? As this still isn't working for me

Ive had to go back to using calico for now

[network-charles]

When you execute “kind --version” you ought to see v0.24.0. If you did and you still experience this error, it could be a WSL2 issue.

[thepaulmacca]

Yeah that's the version I'm seeing. Must be a WSL2 specific issue then

[aojea]

Yeah that's the version I'm seeing. Must be a WSL2 specific issue then

I can't see why, you need to use one of the images linked here https://github.com/kubernetes-sigs/kind/releases/tag/v0.24.0

if it still fails just do a tarball from the kind export logs and attach it here

[thepaulmacca]

I'm using the latest image. I'm on mobile atm, so will grab some logs when I get the chance and add here

[FerencTothBARB]

I have the same problem on WSL2 (Ubuntu 20.04) using Kind v0.24.0 with 1.31 image: the network policy rules have no effect.

In the kindnet log I see many errors like these:

I0823 08:20:05.161339       1 controller.go:374] Syncing nftables rules
I0823 08:20:06.664521       1 controller.go:761] error syncing nftables rules /dev/stdin:12:49-56: Error: Could not process rule: No such file or directory
add rule inet kube-network-policies postrouting ct state established,related accept
                                                ^^^^^^^^
/dev/stdin:13:69-73: Error: Could not process rule: No such file or directory
add rule inet kube-network-policies postrouting ip saddr @podips-v4 queue num 100 bypass comment "process IPv4 traffic with network policy enforcement"
                                                                    ^^^^^
/dev/stdin:14:69-73: Error: Could not process rule: No such file or directory
add rule inet kube-network-policies postrouting ip daddr @podips-v4 queue num 100 bypass comment "process IPv4 traffic with network policy enforcement"
                                                                    ^^^^^
/dev/stdin:15:70-74: Error: Could not process rule: No such file or directory
add rule inet kube-network-policies postrouting ip6 saddr @podips-v6 queue num 100 bypass comment "process IPv6 traffic with network policy enforcement"
                                                                     ^^^^^
/dev/stdin:16:70-74: Error: Could not process rule: No such file or directory
add rule inet kube-network-policies postrouting ip6 daddr @podips-v6 queue num 100 bypass comment "process IPv6 traffic with network policy enforcement"
                                                                     ^^^^^

Both iptables and nftables packages are installed:

$ dpkg -l | grep "  ..tables"
ii  iptables                       1.8.4-3ubuntu2.1                  amd64        administration tools for packet filtering and NAT
ii  nftables                       0.9.3-2                           amd64        Program to control packet filtering rules by Netfilter project

This Ubuntu version is still using iptables. So I guess we need to configure kindnet to fall back to use iptables instead of nftables.

Anyway, can we reopen this issue?

[aojea]

add rule inet kube-network-policies postrouting ip6 saddr @podips-v6 queue num 100 bypass comment "process IPv6 traffic with network policy enforcement"

you need the nft-queue kernel module

[FerencTothBARB]

add rule inet kube-network-policies postrouting ip6 saddr @podips-v6 queue num 100 bypass comment "process IPv6 traffic with network policy enforcement"

you need the nft-queue kernel module

Oh, thanks.
Unfortunately it is not available on WSL2:

$ zcat /proc/config.gz | grep NFT_QUEUE
# CONFIG_NFT_QUEUE is not set

Is it possible to configure kindnet to use the legacy iptables?

[aojea]

we already had similar problems with WSL2 kernels see https://kind.sigs.k8s.io/docs/user/using-wsl2/#kubernetes-service-with-session-affinity

It seems there are two options, run a custom kernel https://learn.microsoft.com/en-us/answers/questions/1426263/how-to-enable-modules-in-wsl2 or request the wsl2 kernel to enable this module by default https://github.com/microsoft/WSL2-Linux-Kernel?tab=readme-ov-file

Is it possible to configure kindnet to use the legacy iptables?

it is more nuanced like that, in a first stage we have iptables support but the iptables infrastructure lack some features that made that solution not sustainable . The kernel and kubernetes communities are also moving to nftables , so it is better for WSL2 to support nftables , since today is this problem but tomorrow it will be other