Native Histogram Support

[f1ko]

Is your feature request related to a problem? Please describe

Prometheus v2.40.0 introduces native histograms (as a new experimental feature).
Applications that already want to push native histogram metrics via remote write require an endpoint that support it.
To remain compatibility with Prometheus VictoriaMetrics shall support native histograms (once stable).

Although changes are still likely since it is not marked as stable yet some initial thought regarding the future implementation into VictoriaMetrics may be done already.
Tracking native histogram implementation on Prometheus can be done here.

Describe the solution you'd like

VictoriaMetrics shall support native histograms.

Describe alternatives you've considered

Using Prometheus with native histograms enabled as a remote write endpoint is a valid alternative for applications that require it although a bit expensive if VictoriaMetrics is already in use.

Additional information

No response

[valyala]

@f1ko , thanks for filing the feature request! Native histograms in Prometheus look promising. Let's wait until they are stabilized and become widely used and then think on how to integrate them into VictoriaMetrics.

In the mean time it is recommended using VictoriaMetrics histograms instead.

[gautamg795]

Hi! Looks like most of the format of native histograms has been stabilized at this point, and it's largely implementation details that remain. As Mimir now supports ingesting native histograms, and a few of the Prometheus projects now export them, would you consider implementing this prior to full stabilization on the Prometheus side?

For context, we're exporting Prometheus metrics from a Rust app, and there aren't yet any ways to export either VictoriaMetrics histograms or Prometheus native histograms in Rust. We'll need to build support ourselves and would prefer to implement the latter for maximum flexibility long term. If VictoriaMetrics might support native histograms within a few months, we'd probably just deal with traditional histograms for a little bit longer.

[7olstoy]

Actively using it with k6, waiting the VM supporting.

[orloffv]

Hello @valyala,

In your previous comment on this thread, dated January 31, 2023, you mentioned that the proposal for native histograms in Prometheus seemed promising and it was worth waiting for their stabilization and widespread adoption. Given that a significant part of the native histogram format has now been stabilized, as noted in later comments, I am wondering if there are any plans to implement this functionality in VictoriaMetrics soon? This would be especially beneficial for users eager to leverage VictoriaMetrics with Prometheus's latest features.

[zorek187]

@valyala, really waiting for prometheus native histograms support in Victoria Metrics to ship k6 results, any updates here?

[valyala]

More than two years passed since the introduction of native histograms in Prometheus, and they are still experimental:

Native histograms were introduced as an experimental feature in November 2022. They are a concept that touches almost every part of the Prometheus stack. The first version of the Prometheus server supporting native histograms was v2.40.0. The support had to be enabled via a feature flag --enable-feature=native-histograms. (TODO: This is still the case with the current release v2.55 and v3.00. Update this section with the stable release, once it has happened.)

The introduction of native histograms in Prometheus resulted in numerous issues and bugs at Prometheus source code. Some of them aren't fixed yet.

Due to the facts above, our decision at VictoriaMetrics is to wait until native histograms become non-experimental feature in Prometheus. After that we are going to implement the following features:

• To add support for Prometheus native histograms received via Prometheus remote write protocol, so external services such as Prometheus could send native histograms to VictoriaMetrics and vmagent. Internally, the Prometheus native histograms must be converted into VictoriaMetrics histograms before being stored in VictoriaMetrics.
• To add support for scraping of Prometheus native histograms. Currently Prometheus exporters do not support exporting native histograms via plaintext exposition format at /metrics page. Instead, they must be configured for exporting metrics in binary form, and the scraper (such as vmagent or Prometheus) must be configured to scrape metrics in binary form from such exporters. See these docs. This significantly complicates practical usage of Prometheus native histograms. This also lower the priority of adding support for scraping of Prometheus native histograms at VictoriaMetrics and vmagent side.

We aren't going to add support for Prometheus native histograms storage at VictoriaMetrics, since we believe that VictoriaMetrics histograms are better on every aspect than Prometheus native histograms - they are compatible with Prometheus data model, they are stable since 2019, they can be exposed in plaintext exposition format at /metrics page. That's why VictoriaMetrics and vmagent will automatically convert Prometheus native histograms to VictoriaMetrics histograms at data ingestion stage.

[tylerbenson]

add support for Prometheus native histograms received via Prometheus remote write protocol

@valyala what is the expected timeframe for this?