Expose a panel that combines user activity from all prometheus datasources

[choldgraf]

The current global users dashboard breaks it down by cluster. However sometimes you want to be able to just have a single plot that shows all users across all clusters. For example if you wanted to make a single point to a service stakeholder that "there are XX users of this service in general over time".

It'd be useful if the "Global Usage Dashboard" had a second panel that included a single aggregate plot, probably separate from the one that breaks down plots by cluster because the scales of the y-axis scale is likely different.

[consideRatio]

Captured from #111 (comment):

I don't understand the implementation side of this, so I will trust your + @GeorgianaElena's judgment when it comes to what is possible. I agree that any time we have to hard-code something, it becomes brittle, and I like the idea of finding ways to make things less brittle.

My main question is how I could answer the same question that (for me) the global dashboard answers:

• What is the trend-line of monthly, weekly, daily active users aggregated across all clusters/hubs/etc being deployed?
• Secondarily: Any other metric that might be important across all clusters (e.g. "What % downtime do we experience aggregated across all clusters?")

It sounds like perhaps the answer is: we shouldn't answer this with a single Grafana dashboard. If that's the case, perhaps we can provide guidance for how somebody could aggregate this data on their own, and then generate the aggregate graphs in a custom way? (e.g. that's similar to what we do in 2i2c at https://2i2c.org/kpis/cloud/

[consideRatio]

I think https://community.grafana.com/t/dashboard-datasource-include-all-not-working-to-query-multiple-dbs/91103/9?u=consideratio clarifies that combining a varliable amount of queries - one per datasource, isn't supported. I've seen it be suggested that could be made as a feature request in https://community.grafana.com/t/repeat-queries-for-all-datasources/43490 - but I've not seen such feature request be made.

I think practically it may be a big hurdle for grafana to support this, so I doubt it will be done.

---

I think given the complexity to support this, its not in scope for this project to go for it - but it could be in scope for 2i2c to provide such dashboard, and in scope for this project to facilitate generating such dashboard as it already is to some degree by allowing --dashboards-dir to be specified.