Adding support in CNI for managing multiple network interface card on the instance

[jaydeokar]

What type of PR is this?
feature

Which issue does this PR fix?:

N/A

What does this PR do / Why do we need it?:
Amazon VPC CNI only manages Network Card 0 on all instances (including multicard supported instances). This restricts the bandwidth usage to only network card 0 as pod interfaces are only connected to NIC 0. With this change, CNI now starts to manage all the available network cards.
A pod can now request access to these network cards via an annotation to make use of all the available bandwidth. A pod which requires this support can do so via annotation. This will create interfaces in the pod namespace equal to the number of network cards available for use on the instance. The pods can then use these interfaces for their egress traffic which has certain BW requirements.

Describing major changes as the change log is significant

CNI

Add flow

• CNI looks for annotations on the pod using the capabilities option. If the pod contains annotation k8s.amazonaws.com/nicConfig: multi-nic-attachment, CNI will ask for multiple IPs from network cards available on the instance.
• IPAMD now returns a list of IPs (IPAddress field in GRPC response). The number of IPs returned equals the number of interfaces created by CNI inside the Pod.
• Host veth name for multi-nic interfaces is generated by <pod-namespace>.<pod-name>.<index>. However we still retain the original naming convention for the first interface of the pod <pod-namespace>.<pod-name>
• Container veth name for multi-nic interfaces has a prefix mNicIf eg mNicf1, mNicIf2 ...
• IPAMD now returns route table number instead of just the device number. The Route table number is calculated as (network card index * max ENIs per NIC) + device-number + 1. Note Route table number 1 (device 0, network card 0) is the main route table for CNI
• CNI now stores the Route table number of the interface in the container interface struct PciID, which is used to cleanup pod networking when IPAMD is down (del with prev result)
• CNI runs a loop over the IP address and attaches to the Pod network.

Delete flow

• CNI relies on IPAMD to return the number of IPs and interface to delete
• If IPAMD is down, CNI uses the prev result to delete the Pod and Host Network

IPAMD

Node Init Flow

1. IPAMD now initializes all the network card on the instance and creates datastore for each network card
2. For IPv4 and IPv6, it initializes the host network for the primary interface
   a. For IPv6, it adds an additional rule in NAT table to exlude SNAT for link local traffic. This is required for DHCPv6 address
3. IPAMD now includes checks for UpdatingCIDRRules when the VPC CIDR or the Security Group associated with the ENIs changes for IPv6 (which already existed for IPv4)
4. On node init, IPAMD initializes all the available datastores and if the datastore is low, it creates interfaces for each network card. It has a new function for IPv6 ENIs where we create 1 ENI for each network card with IPv6 address and then assign an Prefix of size /80. If the creation of ENI fails (due to missing permission), we don't fail

Testing done on this change:
Yes, ran all the test suites on a single card instance and ran manual tests on a multicard- instance

Will this PR introduce any new dependencies?:
No

Will this break upgrades or downgrades? Has updating a running cluster been tested?:
Upgrades should be fine. Downgrade requires to delete the pods using multi-nic annotation and then downgrade otherwise the Pod IPs/Host Networking setup can leak

Does this change require updates to the CNI daemonset config files to work?:
No

Does this PR introduce any user-facing change?:
Yes, customers will now see interfaces attached to NIC > 0 on supported instances

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.