NetworkPolicy support

[phcollignon]

Hello,
Is there any support for network policies in kind ?
(calico CNI plugin or any other network plugin with NetworkPolicy support ?)
Does kindnet support it ? is it the default network plugin ?
Thanks

[aojea]

@phcollignon I think that some users reported in the channel that they had network policies working in kind using calico.
kindnet is very simple and doesn't have Network Policy support and is not really something in scope.
You can replace kindnet with any of the other CNI plugins, just use DisableDefaultCNI: true in your kind configuration and install your own plugin following their instructions once kind finish to create the cluster.

[phcollignon]

Thanks, I'll give it a try. And if I succeed I'll document it ..

[kfyharukz]

Related to the above discussion...

Calico can deploy only to use network policy (i.e. policy-only mode).
However, combining PTP with Calico has a problem. Calico expects veth names of Pods are generated with the following algorithm:

func generateHostVethName(prefix, namespace, podname string) string {
		h := sha1.New()
		h.Write([]byte(fmt.Sprintf("%s.%s", namespace, podname)))
		return fmt.Sprintf("%s%s", prefix, hex.EncodeToString(h.Sum(nil))[:11])
}

We have been successfully using Calico NetworkPolicy with a modified plugin. Our example is here.
So, how about applying a modified PTP CNI plugin to kindnet?

[phcollignon]

Thanks .. for advice. I'll try it. My main usage is Kubernetes for DEV on local laptop.
I was used to kubeadm-dind-cluster project (K8s docker in docker) which is retired project. So I came to kind but I am wondering if it is a good option for local dev with all features (storage, networkpolicies, ...)
I want to avoid minikube because it's too heavy

[aojea]

Related to the above discussion...

Calico can deploy only to use network policy (i.e. policy-only mode).
However, combining PTP with Calico has a problem. Calico expects veth names of Pods are generated with the following algorithm:

func generateHostVethName(prefix, namespace, podname string) string {
		h := sha1.New()
		h.Write([]byte(fmt.Sprintf("%s.%s", namespace, podname)))
		return fmt.Sprintf("%s%s", prefix, hex.EncodeToString(h.Sum(nil))[:11])
}

We have been successfully using Calico NetworkPolicy with a modified plugin. Our example is here.
So, how about applying a modified PTP CNI plugin to kindnet?

Kindnet is using the standard CNI plugins https://github.com/containernetworking/plugins

[BenTheElder]

I was used to kubeadm-dind-cluster project (K8s docker in docker) which is retired project. So I came to kind but I am wondering if it is a good option for local dev with all features (storage, networkpolicies, ...)

Storage has a tracking issue and is a bit thorny, we're working with sig-storage upstream. It's possible to deploy something today, but multi-node local storage options for testing are pretty limited at the moment for the entire ecosystem, we aim to fix that. for single node the CSI local hostpath driver should work, or the rancher local storage system.

currently what we ship is roughly the same as what kubeadm-dind-shipped is the same as what hack/local-up-cluster.sh ships...

Network policies can be used also by simply disabling the built in CNI and deploying your preferred CNI. we don't currently ship them as they don't seem to be widely depended on including for kubernetes's own testing, but we might in the future.

So, how about applying a modified PTP CNI plugin to kindnet?

AFAIK GKE also supports containerd nodes using upstream ptp, and has a calico network policy only option ... perhaps we only need to tweak some config.

I'd prefer to get this working with the upstream ptp plugin than need to ship our own.

[kfyharukz]

Indeed, modifying CNI PTP plugin may not be the best way.
Let me describe considered options to use Calico network policy:

1. Send a patch to the upstream CNI PTP
2. Edit veth name in Calico's WorkloadEndpoint resource from outside Calico's control
3. Patch PTP locally for kind (I've proposed above)

I did not think option 1 would be accepted by CNI because CNI does not define how to obtain Kubernetes Pod NamespacedName information. It is merely a k8s convention for CNI (shown in this issue).

I think option 3 is a reasonable choice.

[aojea]

Indeed, modifying CNI PTP plugin may not be the best way.
Let me describe considered options to use Calico network policy:

1. Send a patch to the upstream CNI PTP
2. Edit veth name in Calico's WorkloadEndpoint resource from outside Calico's control
3. Patch PTP locally for kind (I've proposed above)

I did not think option 1 would be accepted by CNI because CNI does not define how to obtain Kubernetes Pod NamespacedName information. It is merely a k8s convention for CNI (shown in this issue).

I think option 3 is a reasonable choice.

@kfyharukz I think that the problem that you are describing is a Calico feature, not a Kubernetes feature.

From Calico's documentations

Endpoints
Calico network policies apply to endpoints. In Kubernetes, each pod is a Calico endpoint. However, Calico can support other kinds of endpoints. There are two types of Calico endpoints: workload endpoints (such as a Kubernetes pod or OpenStack VM) and host endpoints (an interface or group of interfaces on a host).

Calico's Kubernetes network policies work with kindnet, however, Calico has more features, like WorkloadEndpoint that doesn't :).

In order to use Calico's only network policies you should replace kindnet and use Calico as the CNI., disabling kindnet in the kind config file and installing Calico once kinds create the cluster.

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  disableDefaultCNI: true

[phcollignon]

Little feedback to confirm that NetworkPolicy is supported with Calico CNI.
I disabled kindnet and configured Calico by adding this to the config file :

networking:
  disableDefaultCNI: true 
  podSubnet: 192.168.0.0/16 

Then install Calico and disable rpf check (not sure it is still needed):

kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
kubectl -n kube-system set env daemonset/calico-node FELIX_IGNORELOOSERPF=true

Then I started the cluster again and checked some NetworkPolicies which seem to work as excpeted.

kind create cluster --config configfile

Thanks to all of you for the hints

[aojea]

/close

seems this issue was already solved
#842 (comment)

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

/close

seems this issue was already solved
#842 (comment)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

we may or may not want to target testing the network policy API in kubernetes e2e testing anyhow. this is still something to think about.

[BenTheElder]

I'll be revisiting this this quarter, looking at increasing kubernetes test coverage.

[aojea]

this topic is interesting for sig-network, there is also a group to work in a possible network policy v2.
let me ask @jayunit100 if he is using KIND, he performed a nice demo the other day in the sig-networking meeting

[aojea]

You can use this in the meantime https://github.com/aojea/kube-netpol/blob/main/README.md#install

[aojea]

Ok, this is the official one https://github.com/kubernetes-sigs/kube-network-policies

[jayunit100]

I guess this is resolved ?

Sounds like the solution suggested here is "for kind you can install a cni after the fact like calico or antrea or ovn.... or you can run the kube-network-policies daemonset", but kind itself won't bootstrap a netpol implementation for you, it's outside the scope of the project.

[aojea]

but kind itself won't bootstrap a netpol implementation for you, it's outside the scope of the project.

it is the same with the loadbalancer, you can use cloud-provider-kind or metallb, or for ingress you can choose any of them ... we try to provide the very minimum and more demanded, NetworkPolicies inside a kind cluster can only be used for testing, is not that you pretend to create a security environment for Pods inside a kind cluster 😄

[BenTheElder]

We can consider default installing this one, currently we only default install binaries/components that either:

• are used by kind itself
• are needed to pass conformance

But that line is semi-arbitrary and testing Kubernetes is important to us.
Partially we want to avoid bloat (k8s installs / kind node images are already pretty large), partially to avoid "picking favorites", partially to promote portable Kubernetes APIs over e.g. kind specific behaviors/APIs wherever possible.

IF we started including network policy we would need to either bundle it with our "CNI" or add another similar knob for not-installing it.

[...] we try to provide the very minimum and more demanded, NetworkPolicies inside a kind cluster can only be used for testing, is not that you pretend to create a security environment for Pods inside a kind cluster 😄

Yes, but testing security policies is a thing even as a cluster end user, even if KIND clusters have poor security you're going to install RBAC rules etc. I think the question is:

• Do we consider these APIs to be sufficiently portable?
• If so, will we reconsider them for conformance?
• If not, we probably shouldn't bundle by default.
• How much size will this add to the cluster?

.... Re-evaluating this is on my TODO list next time we get a chance to sync :-)

[BenTheElder]

I think cloud-provider-kind is a bit different .... we'll have to think about this one as well.

[aojea]

• If so, will we reconsider them for conformance?

Conformance APIs sounds like a good bar

[aojea]

It seems feedback from users is that having network policies in kind by default is a good feature to have

/priority important-soon

Talked with @neolit123 and kubeadm policies will not accept them, so this implies we have to implement similar to the local-storage addon.
So we only need to decide if network policies is opt-in or opt-out

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableNetworkPolicies: true

or

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
 networkPolicies: true

/assign @aojea @BenTheElder

[aojea]

here you are , if someone wants to do beta testing #3611

[aojea]

Implemented

[BenTheElder]

This is available now, in https://github.com/kubernetes-sigs/kind/releases/tag/v0.24.0