Egress policies don't work for egress traffic outside of the cluster

[ConorPKeegan]

For IPv6 clusters, all IPv4 egress traffic cannot filtered with policies (and likely vice versa for IPv4 clusters with IPv6 link-local addresses).

Example (on a clean IPv6 EKS cluster):

Using this policy to deny all egress traffic (except DNS):

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - port: 53
      protocol: UDP

Using this pod to test with:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: netshoot
  name: netshoot
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot
    command: ["/bin/bash"]
    args: ["-c", "sleep inf"]

All IPv6 traffic is blocked from accessing external resources, but IPv4 is not:

$ kubectl exec -it netshoot -- curl --max-time 3 -I -v example.com
*   Trying [2606:2800:220:1:248:1893:25c8:1946]:80...
*   Trying 93.184.216.34:80...
* Connected to example.com (93.184.216.34) port 80 (#0)
> HEAD / HTTP/1.1
> Host: example.com
> User-Agent: curl/8.0.1
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Age: 192079
Age: 192079
< Cache-Control: max-age=604800
Cache-Control: max-age=604800
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Date: Fri, 08 Sep 2023 15:18:21 GMT
Date: Fri, 08 Sep 2023 15:18:21 GMT
< Etag: "3147526947"
Etag: "3147526947"
< Expires: Fri, 15 Sep 2023 15:18:21 GMT
Expires: Fri, 15 Sep 2023 15:18:21 GMT
< Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
< Server: ECS (nyb/1D06)
Server: ECS (nyb/1D06)
< X-Cache: HIT
X-Cache: HIT
< Content-Length: 1256
Content-Length: 1256

< 
* Connection #0 to host example.com left intact

[vaskozl]

This is especially prudent given that you can't disable the virtual v4 interfaces creation by the CNI on v6 clusters.

On v4 clusters you have the option of not enable the veths via ENABLE_V6_EGRESS, but there is no such alternative on v6 clusters meaning there is no reliable way to selectively block public egress.

If that were possible this issue would be somewhat mitigated.

[achevuru]

Network policy controller can only operate in either IPv4 or IPv6 mode. Agent will rely on the resolved endpoints provided by the NP controller (via the custom CRD) to enforce policies against the Pod's primary interface created by the VPC CNI plugin (i.e.,) eth0. Policy enforcement will not extend to any secondary interfaces pod might have (via chained plugins or Multus) and/or IPs that belong to different IP family than the cluster.

@vaskozl Same extends to egress-v4 or egress-cni plugin that is chained to VPC CNI. We can try to provide an option to disable that on IPv6 clusters instead.

https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/network-policy-faq.md

[ConorPKeegan]

Policy enforcement will not extend to any secondary interfaces pod might have (via chained plugins or Multus) and/or IPs that belong to different IP family than the cluster.

The issue here is that in this case we're not talking about a cluster that has intentionally added an additional interface with chained plugins or Multus, this is just using the base VPC CNI with only the network policies enabled. This surely means that anybody using this on a cluster will run into exactly the same issue, without being able to find any documentation to explain this absolutely huge caveat to the network policies.

Are there any recommended workarounds for this currently, as it cannot be used in it's current form?

[achevuru]

@ConorPKeegan EKS doesn't support dual stack clusters, and egress-v4 support that EKS currently provides (by default) is via a chained plugin and so the same caveat called out in the above FAQ extends to egress v4 plugin as well. As you might already know, this interface is an egress only interface and the IPv4 IP attached to this interface is not routable beyond the node and is exclusively for egress only support. So, this caveat applies to pods on an IPv6 only cluster trying to reach an IPv4 endpoint via the secondary interface created by the chained plugin. From VPC CNI point of view, we are considering providing an option to disable the chained plugin if there is no need for such an use case in a particular cluster.

Pods on IPv4 clusters will not have any secondary interfaces created by VPC CNI by default and IPv6 Link local addresses are not routable.

[htyagi-aws]

I can confirm the behavior reported on this thread -

The behavior on ipv6 cluster is a bit different than ipv4 cluster

Network Policy that blocks all outgoing traffic

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: client-one-deny-egress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: nginx
  egress: []
  policyTypes:
  - Egress

Its behavior On Ipv6 cluster :

root@nginx:/# curl --max-time 3 -I -v example.com
* Resolving timed out after 3000 milliseconds
* Closing connection 0
curl: (28) Resolving timed out after 3000 milliseconds

As can be seen above in the first attempt we try to curl using 'name' and since connectivity with kube-dns is blocked via policy the name resolution does not happen and connection fails there. We might think here our use-case is achieved but under the hood ipv4 is still allowed.

To prove this theory we by-pass the DNS name resolution process and hit the ipv4 address directly while policy is still applied as shown in second attempt show below, it is able to establish connection with destination.

root@nginx:/# curl --max-time 3 -I -v 93.184.216.34:80
*   Trying 93.184.216.34:80...
* Connected to 93.184.216.34 (93.184.216.34) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 93.184.216.34
> User-Agent: curl/7.88.1
> Accept: */*
> 
* Connection #0 to host 93.184.216.34 left intact

Note : 93.184.216.34 is the ipv4 address for example.com.

If we put ipv6 address in the curl request it is still blocked as show below -

root@nginx:/# curl --max-time 3 -I -v [2606:2800:220:1:248:1893:25c8:1946]:80
*   Trying [2606:2800:220:1:248:1893:25c8:1946]:80...
* Connection timed out after 3001 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 3001 milliseconds

This proves that ipv4 traffic isn't blocked on ipv6 cluster even with the simplest of network policies that is supposed to block everything.

Its behavior on ipv4 cluster :

htxxx@c8xxxxxxxxx-policies % kubectl exec -it nginx-77b4fdf86c-nm8vz  -- curl --max-time 3 -I -v 93.184.216.34:80
*   Trying 93.184.216.34:80...
* Connection timed out after 3001 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 3001 milliseconds

As can be seen the ipv4 traffic is blocked on ipv4 cluster with the same network policy even if you directly hit the IP address.

[achevuru]

@htyagi-aws As discussed offline, this is expected behavior on EKS IPv6 clusters. NP solution right now is single stack only (i.e.,) either IPv4 or IPv6 (EKS doesn't have dual stack support). VPC CNI chains a light weight CNI plugin that creates a secondary interface for IPv6 pods to facilitate egress IPv4 communication - Our motivation behind this is to help with migration if one of the dependencies is not yet IPv6 ready. IP addresses assigned to these v4 only interfaces are from the 169.254 range and are not routable beyond the node and are also not advertised back to the kubelet. These interfaces are purely for egress only IPv4 support (i.e.,) no one can reach these pods using an IPv4 address and so these are purely IPv6 only pods. As called out in the FAQ, NP is only enforced on interfaces created by VPC CNI (i.e.,) primary interface of the pod and are not enforced on any interfaces created by chained plugins.

We will provide an option to disable the secondary egress v4 plugin to avoid any IPv4 fall back concerns. Will keep this thread updated w.r.t that.

[med-94]

@achevuru Thanks for confirming you'll work to provide an option to disable the egress v4 plugin, this should be sufficient for our use case, as we can just use DNS64 / NAT64 to provide connectivity to the v4 internet.

I think it would be worth explicitly calling out this caveat in the FAQ, the most relevant point in there currently seem to be:

Q) Is Multus compatible with Network Policies? Can I use multiple CNIs or network interfaces with Network Policies?
No, Network Policies only work with the eth0 interface.

As someone not familiar with the internals of VPC CNI it's not obvious to me that using VPC CNI in its default configuration will result in a chained plugin and second interface that the Network policies will not be applied to. I think it would be worth adding extra entries to the FAQ along the lines of:

Q) Are Network Policies applied to egress IPv6 traffic in an IPv4 cluster?
By default VPC CNI in an IPv4 cluster cannot reach IPv6 IPs. If ENABLE_V6_EGRESS is set then VPC CNI will add a second interface with a link local IPv6 address used for egress only. Network policies will not apply to this link local interface, and therefore Network Policies filtering outbound IPv6 traffic will not apply.

Q) Are Network Policies applied to egress IPv4 traffic in an IPv6 cluster?
No. VPC CNI's current behaviour is to add a second interface with a link local IPv4 address to allow IPv6 pods to reach the v4 internet. Network policies will not apply to this link local interface, and therefore Network Policies filtering outbound IPv4 traffic will not apply.

[ConorPKeegan]

Having the ability to turn off the v4 egress plugin in a v6 cluster is quite important to us being able to move our project into production (currently aiming for the 16th October), is this likely to be completed by then or is there a date (or estimated date) when this is likely to be completed?

[jdn5126]

@ConorPKeegan we are still determining internally when we can provide a new VPC CNI release that allows disabling the egress-cni plugin in an IPv6 cluster. In the meantime, you can manually disable the chained plugin by editing the conflist, or you can override the default conflist and provide your own once aws/amazon-vpc-cni-k8s#2551 ships in the next VPC CNI release

[jdn5126]

@ConorPKeegan a new environment variable that allows disabling the egress-cni chained plugin in IPv6 cluster, ENABLE_V4_EGRESS, has merged to the VPC CNI. This will ship in the v1.15.1 release, which is currently planned for early October

[jdn5126]

Updating this issue because of the dates discussed earlier. VPC CNI v1.15.1 release has been slightly delayed in order to get more PRs in, but still targeting release within the next two weeks. Tagging @ConorPKeegan since you had mentioned a dependency on 10/16

[jdn5126]

Closing now that VPC CNI v1.15.1 is released on GitHub: https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.15.1. This release contains the Network Policy agent v1.0.4 image tag

[ConorPKeegan]

Thank you, is the v1.0.4 tag planned to be pushed to this repo soon?