|
| 1 | +# Proposal-376: Gateway Metric Aggregator |
| 2 | + |
| 3 | +<!-- |
| 4 | +This is the title of your Proposal. Keep it short, simple, and descriptive. A good |
| 5 | +title can help communicate what the Proposal is and should be considered as part of |
| 6 | +any review. |
| 7 | +--> |
| 8 | + |
| 9 | +<!-- |
| 10 | +A table of contents is helpful for quickly jumping to sections of a Proposal and for |
| 11 | +highlighting any additional information provided beyond the standard Proposal |
| 12 | +template. |
| 13 | +
|
| 14 | +Ensure the TOC is wrapped with |
| 15 | + <code><!-- toc --&rt;<!-- /toc --&rt;</code> |
| 16 | +tags, and then generate with `hack/update-toc.sh`. |
| 17 | +--> |
| 18 | + |
| 19 | +<!-- toc --> |
| 20 | +- [Summary](#summary) |
| 21 | +- [Motivation](#motivation) |
| 22 | + - [Goals](#goals) |
| 23 | + - [Non-Goals](#non-goals) |
| 24 | +- [Proposal](#proposal) |
| 25 | + - [User Stories (Optional)](#user-stories-optional) |
| 26 | + - [Story 1](#story-1) |
| 27 | + - [Story 2](#story-2) |
| 28 | + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) |
| 29 | + - [Risks and Mitigations](#risks-and-mitigations) |
| 30 | +- [Design Details](#design-details) |
| 31 | + - [Test Plan](#test-plan) |
| 32 | + - [Prerequisite testing updates](#prerequisite-testing-updates) |
| 33 | + - [Unit tests](#unit-tests) |
| 34 | + - [Integration tests](#integration-tests) |
| 35 | + - [e2e tests](#e2e-tests) |
| 36 | + - [Graduation Criteria](#graduation-criteria) |
| 37 | +- [Implementation History](#implementation-history) |
| 38 | +- [Drawbacks](#drawbacks) |
| 39 | +- [Alternatives](#alternatives) |
| 40 | +<!-- /toc --> |
| 41 | + |
| 42 | +## Summary |
| 43 | + |
| 44 | +<!-- |
| 45 | +This section is incredibly important for producing high-quality, user-focused |
| 46 | +documentation such as release notes or a development roadmap. It should be |
| 47 | +possible to collect this information before implementation begins, in order to |
| 48 | +avoid requiring implementors to split their attention between writing release |
| 49 | +notes and implementing the feature itself. Proposal editors and SIG Docs |
| 50 | +should help to ensure that the tone and content of the `Summary` section is |
| 51 | +useful for a wide audience. |
| 52 | +
|
| 53 | +A good summary is probably at least a paragraph in length. |
| 54 | +
|
| 55 | +Both in this section and below, follow the guidelines of the [documentation |
| 56 | +style guide]. In particular, wrap lines to a reasonable length, to make it |
| 57 | +easier for reviewers to cite specific portions, and to minimize diff churn on |
| 58 | +updates. |
| 59 | +
|
| 60 | +--> |
| 61 | + |
| 62 | +Metric-based scheduling is common in many systems, including Kubernetes. For GenAI, this becomes more complex because of the heavy computational requirements of models. This proposal outlines a design for a metric aggregator that can efficiently handle the unique challenges posed by GenAI workloads. |
| 63 | + |
| 64 | +## Motivation |
| 65 | + |
| 66 | +<!-- |
| 67 | +This section is for explicitly listing the motivation, goals, and non-goals of |
| 68 | +this Proposal. Describe why the change is important and the benefits to users. The |
| 69 | +motivation section can optionally provide links to [experience reports] to |
| 70 | +demonstrate the interest in a Proposal within the wider InftyAI community. |
| 71 | +
|
| 72 | +[experience reports]: https://github.com/golang/go/wiki/ExperienceReports |
| 73 | +--> |
| 74 | + |
| 75 | +With traditional services, because the final results will be generated in a very short time, common algorithms like round-robin or least-connection are enough. |
| 76 | + |
| 77 | +However, in inference services, because of the heavy computations of the matrix multiplication, the result generation is often very slow, which is an essential difference with the traditional services. Therefore, we need more advanced algorithms to help us make wise scheduling decisions. For example, based on the inference engine's queue size, kv cache size, or combined metrics. |
| 78 | + |
| 79 | +All these indicators should be collected from the inference engines for further analysis, that's why a metric aggregator is needed. |
| 80 | + |
| 81 | +### Goals |
| 82 | + |
| 83 | +<!-- |
| 84 | +List the specific goals of the Proposal. What is it trying to achieve? How will we |
| 85 | +know that this has succeeded? |
| 86 | +--> |
| 87 | + |
| 88 | +- A simple implementation with least-latency scheduling algorithm |
| 89 | +- Extensible with different consumers in the cluster, like the Lora autoscaler or the ai gateway |
| 90 | +- Metrics visualization support, like Grafana |
| 91 | + |
| 92 | +### Non-Goals |
| 93 | + |
| 94 | +<!-- |
| 95 | +What is out of scope for this Proposal? Listing non-goals helps to focus discussion |
| 96 | +and make progress. |
| 97 | +--> |
| 98 | + |
| 99 | +- Different scheduling algorithm implementations in ai gateway, like prefix-cache aware |
| 100 | +- LoRA aware scheduling implementation, will be left to another KEP |
| 101 | +- Performance consideration in big clusters should be left to the Beta level |
| 102 | + |
| 103 | +## Proposal |
| 104 | + |
| 105 | +<!-- |
| 106 | +This is where we get down to the specifics of what the proposal actually is. |
| 107 | +This should have enough detail that reviewers can understand exactly what |
| 108 | +you're proposing, but should not include things like API designs or |
| 109 | +implementation. What is the desired outcome and how do we measure success?. |
| 110 | +The "Design Details" section below is for the real |
| 111 | +nitty-gritty. |
| 112 | +--> |
| 113 | + |
| 114 | +### User Stories (Optional) |
| 115 | + |
| 116 | +<!-- |
| 117 | +Detail the things that people will be able to do if this Proposal is implemented. |
| 118 | +Include as much detail as possible so that people can understand the "how" of |
| 119 | +the system. The goal here is to make this feel real for users without getting |
| 120 | +bogged down. |
| 121 | +--> |
| 122 | + |
| 123 | +#### Story 1 |
| 124 | + |
| 125 | +As a user, I hope my LLM request could be routed to the least-latency instance, so that I can get the result as soon as possible. |
| 126 | + |
| 127 | +#### Story 2 |
| 128 | + |
| 129 | +As a RAG user, when retrieving documents, sometime they'are the same, so I hope my request could be routed to the instance with the most available kv cache to avoid the repetitive calculation, which is know as the prefix cache aware scheduling. |
| 130 | + |
| 131 | +### Notes/Constraints/Caveats (Optional) |
| 132 | + |
| 133 | +<!-- |
| 134 | +What are the caveats to the proposal? |
| 135 | +What are some important details that didn't come across above? |
| 136 | +Go in to as much detail as necessary here. |
| 137 | +This might be a good place to talk about core concepts and how they relate. |
| 138 | +--> |
| 139 | + |
| 140 | +- Metrics-based routing should meet the baseline requirements: even the metrics are unavailable or outdated, the system should still be able to work, despite the fact that the request response may be slower. For example, metrics-based lora scheduling maybe unfit here because once the metric indicates the wrong instance, we may hit 500 server error, it's unacceptable. Unless the inference engine will fetch the models dynamically. |
| 141 | +- Once the gateway picks the Pod for scheduling, it could happen that the Pod suddenly becomes unavailable, we should support fallback mechanism to default service routing. |
| 142 | + |
| 143 | +### Risks and Mitigations |
| 144 | + |
| 145 | +<!-- |
| 146 | +What are the risks of this proposal, and how do we mitigate? Think broadly. |
| 147 | +For example, consider both security and how this will impact the larger |
| 148 | +InftyAI ecosystem. |
| 149 | +
|
| 150 | +How will security be reviewed, and by whom? |
| 151 | +
|
| 152 | +How will UX be reviewed, and by whom? |
| 153 | +
|
| 154 | +Consider including folks who also work outside the SIG or subproject. |
| 155 | +--> |
| 156 | + |
| 157 | +The metrics might be outdated or even unable to fetch, the router then may make suboptimal decisions, but as mentioned above, the system can still work with a slow response. |
| 158 | + |
| 159 | +## Design Details |
| 160 | + |
| 161 | +<!-- |
| 162 | +This section should contain enough information that the specifics of your |
| 163 | +change are understandable. This may include API specs (though not always |
| 164 | +required) or even code snippets. If there's any ambiguity about HOW your |
| 165 | +proposal will be implemented, this is the place to discuss them. |
| 166 | +--> |
| 167 | + |
| 168 | +The overall flow looks like: |
| 169 | + |
| 170 | + |
| 171 | + |
| 172 | + |
| 173 | +### Steps |
| 174 | + |
| 175 | +Let's break down the flow into several steps: |
| 176 | + |
| 177 | +- Step 1: we'll collect the metrics from the inference workloads in metrics aggregator. |
| 178 | +- Step 2: the aggregator will parse the metrics and store them in the redis, this is for HA consideration and cache sharing. Once the instance is down, we can still retrieve the metrics from redis. And if we have multiple instances, we can share the metrics with each other via redis. Considering Envoy AI gateway already uses Redis for limit rating, we'll reuse the Redis here. |
| 179 | +- Step 3 & 4: Traffic comes, the gateway plugin (we'll call it router later) will retrieve the metrics from Redis and make routing decisions based on different algorithms, like queue size aware scheduling. |
| 180 | +- Step 5: The router will send the request to the selected instance, and the instance will return the result to the router, return to the user finally. |
| 181 | + |
| 182 | +### Additional components introduced: |
| 183 | + |
| 184 | +- Metrics Aggregator (MA): MA is working as the controller plane to sync the metrics, this is also one of the reason why we want to decouple it from the router, which working as a data plane. MA has several components: |
| 185 | + - A Pod controller to manage the Pod lifecycle, for example, once a Pod is ready, it will add it to the internal store, and each Pod will fork a background goroutine to sync the metrics continuously, 50ms interval by default. Once the Pod is deleted, the goroutine will be stopped and removed from the store. |
| 186 | + - A internal store to parse the metric results, and store it in the backend storage, like Redis. |
| 187 | +- Redis: a Redis instance is necessary for the metrics storage and sharing, we can use the existing Redis instance in the cluster, or deploy a new one if necessary. We should have storage interface to support different backends in the future. |
| 188 | +- Router: a new router or [DynamicLoadBalancingBackend](https://github.com/envoyproxy/ai-gateway/blob/be2b479b04bc7a219b0c8239143bfbabebdcd615/filterapi/filterconfig.go#L199-L208) specifically in Envoy AI gateway to pick the best-fit Pod endpoints. However, we may block by the upstream issue [here](https://github.com/envoyproxy/ai-gateway/issues/604), we'll work with the Envoy AI Gateway team to resolve it ASAP. Maybe the final design will impact our implementation a bit but not much I think. |
| 189 | + |
| 190 | +### Data Structure |
| 191 | + |
| 192 | +The data structure could be varied based on the metrics we want to collect, let's take the queue size as an example: |
| 193 | + |
| 194 | +Because redis is a kv store, we'll use the ZSET to store the results, `LeastLatency::ModelName` as the key, Pod name as the member and the (runningQueueSize * 0.3 + waitingQueueSize * 0.7) as the score, the factor of waitingQueueSize is higher because metric is a delayed indicator. RunningQueueSize and WaitingQueueSize are two metrics most of the inference engines support. |
| 195 | + |
| 196 | +We'll also have another key to record the update timestamp. For example, a Pod named "default/fake-pod" with the score = 0.5, the set commands look like: |
| 197 | + |
| 198 | +```bash |
| 199 | +# set the update timestamp |
| 200 | +SET default/fake-pod "2025-05-12T06:16:27Z" |
| 201 | + |
| 202 | +# set the score |
| 203 | +ZADD LeastLatency::ModelName 0.5 default/fake-pod |
| 204 | +``` |
| 205 | + |
| 206 | +When collecting, we'll update the timestamp and score together. Setting the top 5 is enough for us to help reduce the storage pressure since it's a memory-based database. We don't use the expiration key here is just because most of the time, the metrics should be updated at a regular interval. |
| 207 | + |
| 208 | +When retrieving, we'll first query the ZSET to get the top 5 records, and iterate them one by one to verify that `currentTimestamp - recordTimestamp < 5s`, if not, skipping to the next one. This is to avoid outdated metrics. Once picked the exact endpoint, we'll reset the score with waitingQueueSize + 1 to avoid hotspot issues, especially when metrics update is blocked by some reasons. |
| 209 | + |
| 210 | +If all metrics are outdated, we'll fallback to the default service. |
| 211 | + |
| 212 | +Note: the algorithm is not the final one, we'll have more discussions with the community to find the best one. |
| 213 | + |
| 214 | +### Test Plan |
| 215 | + |
| 216 | +<!-- |
| 217 | +**Note:** *Not required until targeted at a release.* |
| 218 | +The goal is to ensure that we don't accept enhancements with inadequate testing. |
| 219 | +
|
| 220 | +All code is expected to have adequate tests (eventually with coverage |
| 221 | +expectations). |
| 222 | +
|
| 223 | +[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md |
| 224 | +--> |
| 225 | + |
| 226 | +[x] I/we understand the owners of the involved components may require updates to |
| 227 | +existing tests to make this code solid enough prior to committing the changes necessary |
| 228 | +to implement this enhancement. |
| 229 | + |
| 230 | +##### Prerequisite testing updates |
| 231 | + |
| 232 | +<!-- |
| 233 | +Based on reviewers feedback describe what additional tests need to be added prior |
| 234 | +implementing this enhancement to ensure the enhancements have also solid foundations. |
| 235 | +--> |
| 236 | + |
| 237 | +##### Unit tests |
| 238 | + |
| 239 | +<!-- |
| 240 | +In principle every added code should have complete unit test coverage, so providing |
| 241 | +the exact set of tests will not bring additional value. |
| 242 | +However, if complete unit test coverage is not possible, explain the reason of it |
| 243 | +together with explanation why this is acceptable. |
| 244 | +--> |
| 245 | + |
| 246 | +<!-- |
| 247 | +Additionally, for Alpha try to enumerate the core package you will be touching |
| 248 | +to implement this enhancement and provide the current unit coverage for those |
| 249 | +in the form of: |
| 250 | +- <package>: <date> - <current test coverage> |
| 251 | +
|
| 252 | +This can inform certain test coverage improvements that we want to do before |
| 253 | +extending the production code to implement this enhancement. |
| 254 | +--> |
| 255 | + |
| 256 | +- Hard to predict now since it's a new component, but try the best to make sure all the functionalities are covered. |
| 257 | + |
| 258 | +##### Integration tests |
| 259 | + |
| 260 | +<!-- |
| 261 | +Integration tests allow control of the configuration parameters used to start the binaries under test. |
| 262 | +This is different from e2e tests which do not allow configuration of parameters. |
| 263 | +Doing this allows testing non-default options and multiple different and potentially conflicting command line options. |
| 264 | +--> |
| 265 | + |
| 266 | +<!-- |
| 267 | +This question should be filled when targeting a release. |
| 268 | +For Alpha, describe what tests will be added to ensure proper quality of the enhancement. |
| 269 | +
|
| 270 | +For Beta and GA, add links to added tests together with links to k8s-triage for those tests: |
| 271 | +https://storage.googleapis.com/k8s-triage/index.html |
| 272 | +--> |
| 273 | + |
| 274 | +- By faking the metrics to make sure the router can pick the right instance. |
| 275 | + |
| 276 | +##### e2e tests |
| 277 | + |
| 278 | +<!-- |
| 279 | +This question should be filled when targeting a release. |
| 280 | +For Alpha, describe what tests will be added to ensure proper quality of the enhancement. |
| 281 | +
|
| 282 | +For Beta and GA, add links to added tests together with links to k8s-triage for those tests: |
| 283 | +https://storage.googleapis.com/k8s-triage/index.html |
| 284 | +
|
| 285 | +We expect no non-infra related flakes in the last month as a GA graduation criteria. |
| 286 | +--> |
| 287 | + |
| 288 | +- Add one e2e test to make sure the whole system can be launched via helm chart. |
| 289 | +- For performance, we'll have benchmarks rather than e2e tests. |
| 290 | + |
| 291 | +### Graduation Criteria |
| 292 | + |
| 293 | +<!-- |
| 294 | +
|
| 295 | +Clearly define what it means for the feature to be implemented and |
| 296 | +considered stable. |
| 297 | +
|
| 298 | +If the feature you are introducing has high complexity, consider adding graduation |
| 299 | +milestones with these graduation criteria: |
| 300 | +- [Maturity levels (`alpha`, `beta`, `stable`)][maturity-levels] |
| 301 | +- [Feature gate][feature gate] lifecycle |
| 302 | +- [Deprecation policy][deprecation-policy] |
| 303 | +
|
| 304 | +[feature gate]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md |
| 305 | +[maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions |
| 306 | +[deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/ |
| 307 | +--> |
| 308 | + |
| 309 | +Beta: |
| 310 | + |
| 311 | +- Other storages rather than KV store who supports only key-value pairs which might be not enough for more complex scenarios, like the prefix-cache aware scenario. |
| 312 | +- HA support, once the metrics aggregator is down, the system should still work. |
| 313 | +- No performance issues in big clusters, we may use daemonsets to report metrics. |
| 314 | +- Once the picked Pod is down after the routing decision, router will fallback to the default service. Fallback mode is already supported in Envoy AI gateway. |
| 315 | + |
| 316 | +## Implementation History |
| 317 | + |
| 318 | +<!-- |
| 319 | +Major milestones in the lifecycle of a Proposal should be tracked in this section. |
| 320 | +Major milestones might include: |
| 321 | +- the `Summary` and `Motivation` sections being merged, signaling SIG acceptance |
| 322 | +- the `Proposal` section being merged, signaling agreement on a proposed design |
| 323 | +- the date implementation started |
| 324 | +- the first llmaz release where an initial version of the Proposal was available |
| 325 | +- the version of llmaz where the Proposal graduated to general availability |
| 326 | +- when the Proposal was retired or superseded |
| 327 | +--> |
| 328 | + |
| 329 | +- 2025-05-08: Proposal initialized and submitted for review |
| 330 | + |
| 331 | +## Drawbacks |
| 332 | + |
| 333 | +<!-- |
| 334 | +Why should this Proposal _not_ be implemented? |
| 335 | +--> |
| 336 | + |
| 337 | +## Alternatives |
| 338 | + |
| 339 | +<!-- |
| 340 | +What other approaches did you consider, and why did you rule them out? These do |
| 341 | +not need to be as detailed as the proposal, but should include enough |
| 342 | +information to express the idea and why it was not acceptable. |
| 343 | +--> |
| 344 | + |
| 345 | +- When collecting metrics from the inference workloads, `PUSH` mode will put less pressure on the gateway side, or the gateway will have iterate all the Pods which obviously will lead to performance issues. We didn't pick the approach because it will either add additional load to the inference workload and introduces more complexity to the system. The current approach will fork as much goroutines as the number of inference workloads to sync the metrics in parallel, this is feasible because goroutine is lightweight. Once the metrics aggregator becomes the bottleneck, we can consider to use `PUSH` mode at node level. |
0 commit comments