Optimize distributor push on error

[pracucci]

What this PR does:
Following up the work done in ingesters to optimize the unhappy path (#3969 #3971 #3973), in this PR I'm introducing a benchmark for the Distributor.Push() on validation error and optimizing it.

Contrary to the ingester, for the distributor I didn't pick the path of returning static errors from ValidateLabels() and ValidateSample() and then offer a function to format them because the formatting function ended up to replicate the validation logic too. What I'm proposing in this PR is returning a ValidationError which can be converted to HTTPGRPC error. We still have 1 allocation for each series/sample but, as you can see from the benchmark, the impact is minimal: the real optimization is not having to format an httpgrpc error each time.

Benchmark:

Distributor_PushOnError/max_label_name_length_limit_reached-12     15.8ms ± 0%     5.0ms ± 1%  -68.46%  (p=0.016 n=4+5)
Distributor_PushOnError/max_label_value_length_limit_reached-12    35.9ms ± 1%     3.0ms ± 0%  -91.75%  (p=0.016 n=5+4)
Distributor_PushOnError/timestamp_too_old-12                       2.37ms ± 0%    1.03ms ± 1%  -56.72%  (p=0.016 n=4+5)
Distributor_PushOnError/timestamp_too_new-12                       2.37ms ± 1%    1.03ms ± 2%  -56.60%  (p=0.016 n=4+5)
Distributor_PushOnError/ingestion_rate_limit_reached-12             877µs ± 0%     869µs ± 0%   -0.91%  (p=0.029 n=4+4)
Distributor_PushOnError/too_many_labels_limit_reached-12           19.6ms ± 0%     1.3ms ± 1%  -93.44%  (p=0.008 n=5+5)

name                                                             old alloc/op   new alloc/op   delta
Distributor_PushOnError/max_label_name_length_limit_reached-12     11.2MB ± 0%     0.1MB ± 0%  -98.85%  (p=0.008 n=5+5)
Distributor_PushOnError/max_label_value_length_limit_reached-12    13.3MB ± 0%     0.1MB ± 0%  -99.03%  (p=0.008 n=5+5)
Distributor_PushOnError/timestamp_too_old-12                        824kB ± 0%     118kB ± 0%  -85.71%  (p=0.008 n=5+5)
Distributor_PushOnError/timestamp_too_new-12                        824kB ± 0%     118kB ± 0%  -85.72%  (p=0.008 n=5+5)
Distributor_PushOnError/ingestion_rate_limit_reached-12            85.2kB ± 0%    85.3kB ± 0%     ~     (p=0.056 n=5+5)
Distributor_PushOnError/too_many_labels_limit_reached-12           10.1MB ± 0%     0.1MB ± 0%  -99.07%  (p=0.008 n=5+5)

name                                                             old allocs/op  new allocs/op  delta
Distributor_PushOnError/max_label_name_length_limit_reached-12      58.1k ± 0%      3.1k ± 0%  -94.69%  (p=0.008 n=5+5)
Distributor_PushOnError/max_label_value_length_limit_reached-12     59.1k ± 0%      3.1k ± 0%  -94.79%  (p=0.008 n=5+5)
Distributor_PushOnError/timestamp_too_old-12                        19.0k ± 0%      5.0k ± 0%  -73.56%  (p=0.008 n=5+5)
Distributor_PushOnError/timestamp_too_new-12                        19.0k ± 0%      5.0k ± 0%  -73.56%  (p=0.008 n=5+5)
Distributor_PushOnError/ingestion_rate_limit_reached-12             4.03k ± 0%     4.03k ± 0%     ~     (p=0.167 n=5+5)
Distributor_PushOnError/too_many_labels_limit_reached-12             114k ± 0%        3k ± 0%  -97.26%  (p=0.008 n=5+5)

No optimization to ingestion rate limit but added to the benchmark to measure it too.

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

LGTM (with few non-blocking nits), nice improvement!

[pstibrany]

Nit: It would be nicer if ValidationError interface was not defined in terms of specific transport mechanism. Instead it could simply have a method IsValidationError() bool [or some other that makes sense]. Call to httpgrpc.Errorf could then be done in distributor, by using message from err.Error() function, which would do the formatting. Duplicity in that all validation errors return same StatusBadRequest code would be removed.

[pracucci]

Sounds a good idea. Done!

[pstibrany]

This naming confused me... cause is simply the first formatting-argument for message.

[pracucci]

I see your point but I can't get a better idea for naming, so I just used cause which was the variable name already used in ValidateLabels().

[pstibrany]

Conflicts with change in #3982.