Add remote write protocol 2.0 support

[dimitarvdimitrov]

What should we do?

Prometheus added remote write protocol 2.0 experimental support in v2.54.0 (released on 2024-08-09). We should add the support in Mimir too.

How will we do it (roughly)?

• Add remote write 2.0 support in Mimir distributors
• Backport all (applicable) optimizations we did to remote write 1.0 (un)marshalling
• Compare performance between remote write 1.0 and 2.0
• Allow to enable 2.0 support on a per-tenant basis

Private design doc: https://docs.google.com/document/d/1JSwhdWRODOeGlNIRpYvnEHK6aH7d42n4ZJ_rfFvt-Lo/edit?tab=t.0#heading=h.5sybau7waq2q

Out of the scope of this work:

• Change data format between distributors and ingesters / Kafka (keep using protocol 1.0 format). This should be a follow up deliverable.

Size?

Between Medium (= ~1 month) and Large (= ~3 month).

What will we deliver?

• Add remote write 2.0 experimental support in Mimir, fully merged in Mimir but disabled by default
• Test remote write 2.0 in dev

What are the documentation dependencies?

• No documentation changes to docs until enabled by default

Urgency?

Not urgent yet, but we can't lag too much behind Prometheus

[jmichalek132]

FYI otel collector contrib has lfx mentorship project starting in September to add support in the remote write exporter for remote write 2.0. Tracking issue: open-telemetry/opentelemetry-collector-contrib#33661.

[krajorama]

This is becoming a bit more urgent now due to two projects depending on it: native histograms with custom buckets (NHCB) and proper handling of OTLP created timestamp (current solution we don't want to upstream because it's a bit of a workaround).