Proposal: Support multiple custom/external metrics servers in the cluster

[zroubalik]

Currently only one custom or external metrics server in the cluster is supported (because there can be only one APIService resouce specified), as already discussed in #3 in #68. This way users are not able use multiple projects that are consuming this framework, eg. KEDA, k8s-prometheus-adapter or Datadog etc.

I was thinking about options how to solve this and I'd like to propose, that we will introduce aggregator and child components.

• aggregator - that's basically the adapter that we have today (ie. it is responsible for communication with k8s api server, APIService registration etc). + additional functionality: it registers child components as they are added in the cluster, transferring the requests for particular metrics to the child components and will maintain a cache of metrics that are provided by individual childs (which could always fall back to calling ListMetrics() on each child).
  Could either use:
  • a new controller + CRD to register new child components and forward the REST call
  • use grpc for communication between aggregator & child, this way the child components could register themselves in the aggregator (by initializing the connection) and we won't need a controller and CRD for this part
• child - very lightweight component, that's transferring the metrics to the aggregator, user will just have to implement the interfaces

With this approach, in the cluster there will be one aggregator component which will transfer requests to the individual child. Individual projects, that are consuming this framework, will have to just implement the child. If grpc is used, the whole communication could be pretty simple and we can even include this change into the existing codebase and allow some mixed scenario, where aggregator could provide some metrics as well, but I'd like to avoid this scenario.

What do you think about this proposal? Do you think that this is something that could be acceptable by this project? I can elaborate more (and formalize the proposal), if I see an interest in this.

And yeah, I can contribute this 😄

[dgrisonnet]

Hey @zroubalik, thank you for coming up with this proposal, I think this would be an amazing addition to this project.

I would personally lean towards the CRD approach, as this would not require mutating existing adapters. Moreover, in the long run, this would allow more configuration possibilities than with grpc. If at some point, we need to select metrics from a specific producer, I think, it would be easier to do so with CRDs.

Does your definition of a child component require an actual modification of existing adapters? To me, child components are the current adapters, and the aggregator is a whole new component dispatching API call to its children.

Out of curiosity, with this approach, is your goal to have one component being both an aggregator and a child or have a standalone aggregator?

Do you think that this is something that could be acceptable by this project?

I think that it is worth adding a boilerplate to create metrics aggregators to this project as having multiple metric providers falls into the overall experience of custom metrics in Kubernetes.

[s-urbaniak]

This proposal sounds good to me too! 🎉 cc'ing @serathius as I am curious about his thoughts too. Generally, from my side I'd love to see i.e. namespace based support for aggregated APIs in Kubernetes itself, which would solve that problem maybe in another way, but supporting it natively here definitely sounds also like a viable approach.

[zroubalik]

@s-urbaniak great, I'll follow with some more details later.

wrt the namespace based support in k8s itself, if I understand this correctly, this wouldn't help in this case if user would like to consume metrics from multiple providers from the same namespace, or am I wrong?

[s-urbaniak]

@zroubalik correct and this is actually the edge case that I am actually worried about as there doesn't seem to be a clear boundary of responsibilities which provider is responsible for what namespace.

[mwerickso]

We are in need of this exact use case. We need our subscribers to use a Stackdriver metric directly from GCP to scale off of. However, in the same namespace we have flask services that need to use the DD trace metrics to scale off request count.

In the interim, we have our subscribers using the GCP metric from Datadog, but this adds an 8-10 minute delay in the external metrics server getting the metric to scale off (that is how much time it takes for DD to get metrics from GCP).

Is it possible to get this on the roadmap? Is there any information that I can contribute to help architect how this would be solved?

[zroubalik]

@mwerickso sorry for the delay. I am planning to start working on a POC, any help is definitely more than welcome. I will add some details later and will let you know then.

[ellistarn]

This proposal is awesome. In Karpenter, we decided against the k8s metrics api due to this exact single-slot problem: https://github.com/awslabs/karpenter/blob/main/docs/DESIGN.md#prometheus-vs-kubernetes-metrics-api.

I'm a little nervous about the idea of separating by namespace. Namespace is increasingly seen as a security boundary. Similarly, prometheus has moved towards multiple prometheuses deployed into different namespaces. By dividing metrics providers along namespaces, we lose the security properties of namespaces (i.e. metric RBAC) as well as enforce a cluster global semantic for that metrics provider.

Instead, I was thinking about a model where a metrics provider could advertise the metrics paths it supports, and effectively have a universal metrics api router that detects these paths and routes to the right metrics provider. This could be modeled with CRDs, as you're suggesting.

Strawman workflow:

1. Install universal metrics server as the external metrics provider
2. Install a specific metrics provider server and CRD. CRD would encapsulate paths and metric translation definition
3. Universal metrics server would watch CRDs and detect ones that include a well known struct (i.e that encapsulates metrics path). This struct must be a subfield of the metrics provider's CRD.
4. Users create CRD specific to their metrics provider (e.g. prom/cloudwatch/stackdriver metrics), UMS detects the CRDs and routes to the configured metrics provider server.

[Pluies]

Having ran into the same issue, I'm looking forward to this proposal gaining traction! 🙌

In the meantime, one way of getting around this issue is to reverse-proxy to several metrics provider; I've written a PoC/example at: https://github.com/Pluies/external-metrics-multiplexer

[serathius]

Overall issue touches problem of extending kube aggregation to support multiple apiservers either by namespace or by other means. I think this should be first brought to apimachinery who has more broad view on extending K8s APIs.

[arjunrn]

Based on @DirectXMan12 's comment 2 years ago I looked into what would be required to implement a service which discovers metrics from multiple metrics apiservers and then routes metric requests to the correct server. It would look something like what @ellistarn described. I've written a proposal on how this would work and implemented a prototype which discovers both custom and external metrics. The advantage of this solution is that it doesn't require any changes to Kubernetes APIs or existing custom metrics servers. Also there's no extra configuration for metrics. There's still the open question of how to deal with metrics which are provided by multiple metrics apiservers but this can be handled in the current design or in later iterations. Before I actually put more work into turning the prototype to something that's production ready I would appreciate any feedback particularly from sig-instrumentation.

[serathius]

Awesome, Thanks for doing this!
Could you create a KEP from this document? https://github.com/kubernetes/enhancements/tree/master/keps#kubernetes-enhancement-proposals-keps
Making official enhancement proposal will allow to increase visibility of this proposal and get some input from SIG apimachinery ensuring that work can move forward.

I would be happy to help you with KEP process.