feature request: when using prefixes, assign primary IP of ENI within prefix

[adammw]

What would you like to be added:
When using IP Prefix delegation with AWS CNI plugin and Custom Network Config (ie ENIConfig CRDs), all pod IPs are attached to secondary ENIs. The ENI has an auto-attached primary IP address which (as far as I can tell) is unused and takes up space that would otherwise be available for other CIDR prefixes. If instead the ENI has a custom primary IP that is within the prefix being attached, that would 'free up' that CIDR block for use by another node.

Before

After

Why is this needed:
Approaching the network limits in some of our clusters. We can see that without CIDR reservations, 9 CIDR blocks are used up with only 6 prefixes as the primary IPs are randomly allocated and prevent those additional 3 blocks from being used. CIDR reservations can help in preventing the primary IPs from being allocated in that space, but would be easier/nicer if we didn't allocate them at all, which while a primary IP does seem to be required for an ENI, if it can overlap the prefix then it wouldn't take up any space that isn't already taken up by the prefix delegation.

[jdn5126]

@adammw the primary IP for an ENI is assigned by EC2, and not something that we have control over. I assume it is used by some internal EC2 logic. For networking limits, are you using 10.0.0.0/8 subnet for your VPC? And since you are using custom networking, are you attaching 100.64.0.0/10 CIDR for more space?

[adammw]

@jdn5126 the CreateNetworkInterface API call supports specifying a PrivateIpAddress and exact Ipv4Prefix (instead of just a count of prefixes to attach) - it's just that they're not used in favour of auto-allocation by EC2 at the moment.

Thinking about how to do this, we'd have to either select a prefix ourselves in the plugin and then pass it to AWS in that call, or use the AllocateIpamPoolCidr call to get a prefix from an IPAM pool (but that also requires you to use the AWS IPAM) - so either option would have to be opt-in as there is obviously downsides to both approaches and other setup required.

Alternatively, if you have any sway over the EC2 service team responsible for the CreateNetworkInterface call, to add a flag to the API that tells it to set the private IP address within one of the prefixes it allocates rather than a random allocation, since that would require no real overhead from the plugin's POV.

As for networking, we're using both the 10.0.0.0/8 and 172.16.0.0/12 address space internally but the problem is just a specific subnet we made slightly too small to run both blue/green nodes during deployment, and trying to squeeze the most out of the existing subnets we have created.

[jdn5126]

A flag to the API would be the ideal approach. I will reach out to EC2 team internally and see what they think

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[adammw]

@jdn5126 any updates from the EC2 service team?

[jdn5126]

@adammw sorry for the delay, I created a new ticket for the EC2 team to follow up on the initial request. I will track the ticket now that it is in their queue.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

Issue closed due to inactivity.