Instrument the Ruler Notifier

[gotjosh]

What this PR does:

The ruler notifier is not instrumented at the moment. This commit wraps
around the Ruler registry with the userID as a label and cortex_ as a
prefix and passes it to the notifier options.

The result? We have metrics.

Which issue(s) this PR fixes:
Fixes #2193

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[jtlisi]

My main concern here is that we are adding a histogram metric with a user label. We already are reporting a lot of metrics for each user in the ruler and we should attempt to be a bit more judicious about what metrics we expose. When we encountered this situation previously we solved it by creating a shim metrics reporter that registers the registry we pass to the vendored Prometheus code. We can likely use the same pattern here.

Look at this file for reference as to what I'm referring to:
https://github.com/cortexproject/cortex/blob/master/pkg/alertmanager/alertmanager_metrics.go

[gouthamve]

LGTM pending rebase. Slightly concerning but should be okay. prometheus_notifications_latency_seconds is a summary adding 5 metrics per AM. And given people typically run 3 AMs, it will be an additional 15 metrics per user. I think it should be fine. If its not, we can revert later.

[pracucci]

LGTM! (modulo a couple of nits)

[pstibrany]

Since we are just reexporting metrics from Prometheus without any other processing or aggregation, I don't think we should prefix this. That said, we already have other exported Prometheus metrics with prefix, so approving in the current state.

[gotjosh]

I can see that the changelog is having conflicts again, just let me know if we're ready to merge and I'll fix it 😄

[pracucci]

@gotjosh I took care of CHANGELOG.

[gotjosh]

Thanks @pracucci !