update branch ENI operation metrics & dev guide

[sushrk]

Description of changes:
Updated the metrics to measure branch ENI operations- create branch ENI, annotate pod, and initialize trunk. The summary metric also measures 0.5, 0.9, 0.99 quantiles (50th, 90th, and 99th percentiles), and quantile 0(min value) & 1(max value).

Test
Running scale tests to deploy 1K pods repeatedly:

# HELP branch_provider_operation_latency Branch Provider operations latency in ms
# TYPE branch_provider_operation_latency summary
branch_provider_operation_latency{branch_provider_operation="annotate_branch_eni",resource_count="1",quantile="0"} 9
branch_provider_operation_latency{branch_provider_operation="annotate_branch_eni",resource_count="1",quantile="0.5"} 15
branch_provider_operation_latency{branch_provider_operation="annotate_branch_eni",resource_count="1",quantile="0.9"} 35
branch_provider_operation_latency{branch_provider_operation="annotate_branch_eni",resource_count="1",quantile="0.99"} 84
branch_provider_operation_latency{branch_provider_operation="annotate_branch_eni",resource_count="1",quantile="1"} 110
branch_provider_operation_latency_sum{branch_provider_operation="annotate_branch_eni",resource_count="1"} 32414
branch_provider_operation_latency_count{branch_provider_operation="annotate_branch_eni",resource_count="1"} 1612
branch_provider_operation_latency{branch_provider_operation="create_branch_eni",resource_count="1",quantile="0"} 817
branch_provider_operation_latency{branch_provider_operation="create_branch_eni",resource_count="1",quantile="0.5"} 4928
branch_provider_operation_latency{branch_provider_operation="create_branch_eni",resource_count="1",quantile="0.9"} 5453
branch_provider_operation_latency{branch_provider_operation="create_branch_eni",resource_count="1",quantile="0.99"} 5854
branch_provider_operation_latency{branch_provider_operation="create_branch_eni",resource_count="1",quantile="1"} 15032
branch_provider_operation_latency_sum{branch_provider_operation="create_branch_eni",resource_count="1"} 7.565694e+06
branch_provider_operation_latency_count{branch_provider_operation="create_branch_eni",resource_count="1"} 1613
branch_provider_operation_latency{branch_provider_operation="init_trunk",resource_count="1",quantile="0"} 3112
branch_provider_operation_latency{branch_provider_operation="init_trunk",resource_count="1",quantile="0.5"} 3186
branch_provider_operation_latency{branch_provider_operation="init_trunk",resource_count="1",quantile="0.9"} 15665
branch_provider_operation_latency{branch_provider_operation="init_trunk",resource_count="1",quantile="0.99"} 15869
branch_provider_operation_latency{branch_provider_operation="init_trunk",resource_count="1",quantile="1"} 15869
branch_provider_operation_latency_sum{branch_provider_operation="init_trunk",resource_count="1"} 276597
branch_provider_operation_latency_count{branch_provider_operation="init_trunk",resource_count="1"} 30

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[haouc]

lgtm

[barryrobison]

Just curious - will these metrics be exposed somewhere for the customer?

[yash97]

lgtm 🚀

[sushrk]

Just curious - will these metrics be exposed somewhere for the customer?

@barryrobison yes, control plane metrics can be viewed via this command

kubectl get --raw /metrics

EDIT: Sorry only APIServer and etcd metrics are exposed at the moment. The controller metrics is not exposed.

More info on monitoring the metrics can be found here https://aws.github.io/aws-eks-best-practices/reliability/docs/controlplane/#monitor-control-plane-metrics