Rotating a node pool can result in broken graphs

[felder]

Bug description

After rotating a node pool, the "Running Users", "Memory Commitment %", and "CPU Commitment %" graphs fail to render because the promql queries backing them result in illegal many -> one relationships.

For example, with "Running Users":

sum(kube_pod_status_phase{phase="Running"} * on (namespace, pod) group_right() kube_pod_labels{label_app="jupyterhub", label_component="singleuser-server"}) by (namespace)

In order for that to work, kube_pod_status_phase needs to have unique rows when keyed on namespace and pod. Unfortunately, in this case due to the node pool rotation multiple rows with different "kubernetes_node" labels can be returned for the same namespace and pod.

Expected behaviour

Graphs render.

Actual behaviour

Graphs show an error.

How to reproduce

Not easily reproduced probably, but a user may encounter it if they rotate node pools for some reason.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[felder]

The fix for this is use the group() aggregator in promql to ensure uniqueness.

For "Running Users" this modification to the promql does the trick:

sum(group(kube_pod_status_phase{phase="Running"}) by (pod,namespace) * on (namespace, pod) group_right() kube_pod_labels{label_app="jupyterhub", label_component="singleuser-server"}) by (namespace)