failed to assign an IP address to container after enabling custom networking

[barsilver]

What happened:

Every once in a while since I have set the vpc-cni add-on to use custom networking we're facing the following errors, even though there are enough IP addresses in the secondary subnets:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c3b3c54fed84a015bf5a71d5f78ba19ad1ca7f2dcf73eb25fd3ff0dc7ff7c5ed": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

When running the following command on this node:
kubectl describe node ip-172-32-1-101.ec2.internal | grep 'pods\|PrivateIPv4Address'

We see that the node's capacity shows:

pods:                    58

However, when running:
./max-pods-calculator.sh --instance-type c6i.2xlarge --cni-version "1.19.2-eksbuild.1" --cni-custom-networking-enabled
The result shows that the maximum number of pods that should be able to run on this instance type is 44.

In /var/log/user_data.log of the affected node, the following configurations are visible:

INFO: --dns-cluster-ip='10.100.0.10'
2025-03-17T10:51:50+0000 [eks-bootstrap] INFO: --use-max-pods='false'
2025-03-17T10:51:50+0000 [eks-bootstrap] INFO: --kubelet-extra-args='--node-labels="component=entitle,karpenter.k8s.aws/ec2nodeclass=default,karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=spot" --register-with-taints="karpenter.sh/unregistered:NoExecute" --max-pods=58'
2025-03-17T10:51:50+0000 [eks-bootstrap] INFO: Using kubelet version 1.31.5
2025-03-17T10:51:50+0000 [eks-bootstrap] INFO: Using containerd as the container runtime
Created symlink from /etc/systemd/system/multi-user.target.wants/sys-fs-bpf.mount to /etc/systemd/system/sys-fs-bpf.mount.
2025-03-17T10:51:51+0000 [eks-bootstrap] INFO: Using IP family: ipv4

The ipamd.log show errors like these:

{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: IP address pool stats: total 42, assigned 42"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: ENI eni-04ddad5f92297dbb6 does not have available addresses"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 100.64.220.11/ffffffff"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 100.64.220.11/ffffffff"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 100.64.130.44/ffffffff"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 100.64.130.44/ffffffff"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 100.64.139.212/ffffffff"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 100.64.139.212/ffffffff"}
{"level":"debug","ts":"2025-03-18T07:15:21.077Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 100.64.216.129/ffffffff"}

These errors started after attempting to resolve an issue where IP addresses were exhausted in the subnets the cluster was using. The subnets were too small, and both nodes and pods were using the same CIDR range.
Secondary subnets were added using Terraform with the following configuration:

module "custom-netwotking" {
  source         = "../../modules/vpc-cni-custom-networking"
  cluster_name   = module.eks_cluster.eks_cluster_id
  secondary_cidr = "100.64.0.0/16"
  secondary_subnets = {
    us-east-1a = "100.64.0.0/17"
    us-east-1b = "100.64.128.0/17"
  }
}

The ENIConfig for the newly created subnets looks like this:

apiVersion: v1
items:
- apiVersion: crd.k8s.amazonaws.com/v1alpha1
  kind: ENIConfig
  metadata:
    annotations:
    generation: 1
    name: us-east-1a
  spec:
    securityGroups:
    - sg-07029
    subnet: subnet-075de
- apiVersion: crd.k8s.amazonaws.com/v1alpha1
  kind: ENIConfig
  metadata:
    annotations:
    generation: 1
    name: us-east-1b
  spec:
    securityGroups:
    - sg-07029
    subnet: subnet-095f8
kind: List
metadata:
  resourceVersion: ""

When I tried setting ENABLE_POD_ENI=true and DISABLE_TCP_EARLY_DEMUX=true and then restarted the nodes, I received the following errors while the nodes failed to remove:

  Normal  ControllerVersionNotice  7s (x14 over 49m)    vpc-resource-controller  The node is managed by VPC resource controller version v1.6.3
  Normal  NodeTrunkFailedInit      6s (x14 over 49m)    vpc-resource-controller  The node failed initializing trunk interface

Environment:

• Node Kubernetes version: v1.31.4-eks-aeac579
• CNI Version: 1.19.2-eksbuild.1
• Node AMI: amazon-eks-arm64-node-1.31-v20250123

[FeLvi-zzz]

I have same problem.

vpc-cni sets max pod to the node where it is.
https://github.com/aws/amazon-vpc-cni-k8s/blob/7b288b8e1860d2d3a5f286e29aa576011f91a7c6/misc/eni-max-pods.txt

but when custom networking(including security groups for pods) is enabling, 1 ENI is reserved for trunk-eni.
https://aws.amazon.com/jp/blogs/containers/leveraging-cni-custom-networking-alongside-security-groups-for-pods-in-amazon-eks/

Pods on a node cannot use trunk-eni IPs, so I believe vpc-cni should not refer eni-max-pods.txt when using custom networking.

I'd like maintainers to create something like eni-max-pods-for-custom-networking.txt.

related awslabs/amazon-eks-ami#2051

[haouc]

Hey @FeLvi-zzz, we are tracking the optimization including #3094 . We can look into the calculation. @barsilver , when you enable custom networking, the primary ENI will not be used to assign IPs. Thanks for the requests on optimizing the calculation.

We are also working on collecting data on supporting this #3067 . Please feel free to submit your use case there if you see fits.

[barsilver]

Hi @haouc,

My issue at first was that I don’t have enough IP addresses in the node subnets. Initially, both pods and nodes were using the same CIDR range, which was too small. To address this, I created secondary subnets with a broader CIDR range. These subnets now have 32,556 and 32,507 available IPv4 addresses - far more than needed.

However, I still frequently see failed to assign an IP address to container errors on the pods. It seems that at some point, the node reaches its limit in ipamd.log (IP address pool stats: total 42, assigned 42). Despite this, pods continue trying to be scheduled on the node, causing errors. The max pods per node should be 58.

Should I set --use-max-pods=true?
How can I resolve this issue in the meantime? Enabling custom networking didn’t help and introduced new errors.
Am I missing any environment variable configurations in aws-node?

- env:
        - name: ADDITIONAL_ENI_TAGS
          value: '{}'
        - name: ANNOTATE_POD_IP
          value: "false"
        - name: AWS_VPC_CNI_NODE_PORT_SUPPORT
          value: "true"
        - name: AWS_VPC_ENI_MTU
          value: "9001"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
        - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
          value: "false"
        - name: AWS_VPC_K8S_CNI_LOGLEVEL
          value: DEBUG
        - name: AWS_VPC_K8S_CNI_LOG_FILE
          value: /host/var/log/aws-routed-eni/ipamd.log
        - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
          value: prng
        - name: AWS_VPC_K8S_CNI_VETHPREFIX
          value: eni
        - name: AWS_VPC_K8S_PLUGIN_LOG_FILE
          value: /var/log/aws-routed-eni/plugin.log
        - name: AWS_VPC_K8S_PLUGIN_LOG_LEVEL
          value: DEBUG
        - name: CLUSTER_NAME
          value: prod-cluster
        - name: DISABLE_INTROSPECTION
          value: "false"
        - name: DISABLE_METRICS
          value: "false"
        - name: DISABLE_NETWORK_RESOURCE_PROVISIONING
          value: "false"
        - name: ENABLE_IPv4
          value: "true"
        - name: ENABLE_IPv6
          value: "false"
        - name: ENABLE_POD_ENI
          value: "true"
        - name: ENABLE_PREFIX_DELEGATION
          value: "false"
        - name: ENABLE_SUBNET_DISCOVERY
          value: "true"
        - name: ENI_CONFIG_LABEL_DEF
          value: failure-domain.beta.kubernetes.io/zone
        - name: NETWORK_POLICY_ENFORCING_MODE
          value: standard
        - name: VPC_CNI_VERSION
          value: v1.19.2
        - name: VPC_ID
          value: vpc-1234
        - name: WARM_ENI_TARGET
          value: "1"
        - name: WARM_PREFIX_TARGET
          value: "1"

[haouc]

@barsilver looks like you have Security Group for Pods enabled in your shared case

        - name: ENABLE_POD_ENI
          value: "true"

This will take one spot from ENIs on the instance. If you are not using the feature, I would suggest turn it off. Regarding you were seeing the error in node event, I guess you were not using the feature supported instance types.

[barsilver]

Hi @haouc,

As I mentioned earlier, I enabled ENABLE_POD_ENI=true only because I was encountering failed to assign an IP address to container errors. If I disable it, these errors persist.

Since the pods are using security groups, I assumed this var needed to be enabled. The security group specified in the ENIConfig resource is the same one used by the nodes in my cluster. I'm not entirely sure if this is necessary, but when creating the ENIConfig, setting a security group is mandatory, so I used the same one as the nodes.

I don’t mind disabling this setting, but I’m concerned that I might need it due to my current configuration. Could you clarify if there's a way to proceed without it, or if there's an alternative approach?

[haouc]

@barsilver sorry for late response.

Since the pods are using security groups, I assumed this var needed to be enabled.

Yes, if you are using the feature, this need to be set as true. Custom networking and Security Group for Pods are two features and not mutually depending on each other. Please clarify if you are using this feature?

[haouc]

If you are using AL2023 AMI and facing max-pod calculation issue, please refer to this issue opened recently.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[TebogoTS]

I'm experiencing the same error as reported in this issue,

Environment
EKS Cluster: v1.30.
Instance Type: t3.2xlarge (4 ENI limit, 15 IPs per ENI)
Karpenter: Used for node provisioning
AWS VPC CNI:
Custom Networking: Enabled
Custom subnets: Non-routable 100.x
ENV:
            - name: AWS_VPC_K8S_PLUGIN_LOG_LEVEL
              value: DEBUG
            - name: ENABLE_POD_ENI
              value: 'false'
            - name: ENABLE_PREFIX_DELEGATION
              value: 'false'
            - name: ENABLE_SUBNET_DISCOVERY
              value: 'true'
            - name: ENI_CONFIG_LABEL_DEF
              value: topology.kubernetes.io/zone
            - name: NETWORK_POLICY_ENFORCING_MODE
              value: standard
            - name: VPC_CNI_VERSION
              value: v1.19.5
            - name: VPC_ID
              value: vpc-0f7a7b45057e52d53
            - name: WARM_ENI_TARGET
              value: '0'
            - name: WARM_PREFIX_TARGET
              value: '1'
            - name: WARM_IP_TARGET
              value: '4'
            - name: MINIMUM_IP_TARGET
              value: '8'

Error Details
Error Log (2025-05-30T12:37:41.636Z):
{"level":"error","ts":"2025-05-30T12:37:41.636Z","caller":"aws-k8s-agent/main.go:42","msg":"Initialization failure: Failed to attach any ENIs for custom networking"}
{"level":"debug","ts":"2025-05-30T12:37:41.636Z","caller":"ipamd/ipamd.go:567","msg":"Skipping ENI allocation as the max ENI limit is already reached"}

@barsilver are you using managed nodegroups or karpenter? I have not tried to reproduce this on managed nodes.

[TebogoTS]

Workaround:
We set a max-pods flag for the kubelet in the Karpenter EC2NodeClass for t3.2xlarge to prevent over-scheduling. Kubelet sets max-pods to 58 for t3.2xlarge, which is too high, as custom networking excludes the primary ENI (15 IPs per ENI, 14 usable after primary IP reservation), causing ENI/IP exhaustion. See: https://github.com/awslabs/amazon-eks-ami/blob/main/nodeadm/internal/kubelet/eni-max-pods.txt#L37

This workaround has limitations, such as handling different instance types. Prefix delegation isn’t viable for us, as our subnet lacks a /28 range, which could’ve been an alternative.

IMO, the VPC CNI should automatically calculate the correct pod IP limits with custom networking to allow kubelet to annotate accurate max-pods values.

[jayanthvn]

IMO, the VPC CNI should automatically calculate the correct pod IP limits with custom networking to allow kubelet to annotate accurate max-pods values.

Yes that would have been ideal but once we update kubelet max pods we would need to restart kubelet. There was a proposal upstream kubernetes on dynamic max pod update but it wasn't taken forward.