AWS ENA driver not reporting all statistics #1197
Description
Description
The ENA driver installed with flatcar does not produce all of the metrics that AWS advertise as being available. The documentation states that metrics like conntrack_allowance_available should be available, but they are not present.
Output of ethtool -S eth0:
ethtool -S eth0
NIC statistics:
tx_timeout: 0
suspend: 0
resume: 0
wd_expired: 0
interface_up: 1
interface_down: 0
admin_q_pause: 0
bw_in_allowance_exceeded: 77193
bw_out_allowance_exceeded: 8487
pps_allowance_exceeded: 0
conntrack_allowance_exceeded: 0
linklocal_allowance_exceeded: 0
..continues..
Output of modinfo ena:
modinfo ena
filename: /lib/modules/5.15.106-flatcar/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz
license: GPL
description: Elastic Network Adapter (ENA)
author: Amazon.com, Inc. or its affiliates
alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:
retpoline: Y
intree: Y
name: ena
vermagic: 5.15.106-flatcar SMP mod_unload
sig_id: PKCS#7
signer:
sig_key:
sig_hashalgo: unknown
signature:
The instance and the AMI supports ENA metrics:
aws ec2 describe-instances --instance-ids i-03caf708273b6aced --query "Reservations[].Instances[].EnaSupport"
[
true
]
aws ec2 describe-images --image-id ami-041894d5bf5ede441 --query "Images[].EnaSupport"
[
true
]
OS Information:
cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3510.2.1
VERSION_ID=3510.2.1
BUILD_ID=2023-04-24-2100
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3510.2.1 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3510.2.1:*:*:*:*:*:*:*"
This is likely due to the ENA driver being version older than 2.8.1.
Impact
The impact is that we are unable to export all the metrics that should be available to us limiting our ability to detect and resolve issues.
Environment and steps to reproduce
Flatcar running on an m6i.2xlarge EC2 instance within AWS. When querying ethtool -S eth0 | grep conntrack_allowance_available you should get a result.
Expected behavior
The full set of ENA metrics should be reported.
Additional information
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html#network-performance-metrics
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances
This works as expected on Amazon Linux 2 on the same instance type.