Skip to content

Improve OtlpMetricsSender API #5994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 11, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/modules/ROOT/pages/implementations/otlp.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ management:
key1: value1
----

1. `url` - The URL to which data is reported. Environment variables `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_ENDPOINT` are also supported in the default implementation. If a value is not provided, it defaults to `http://localhost:4318/v1/metrics`
1. `url` - The address to which metrics are published. Environment variables `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_ENDPOINT` are also supported in the default implementation. If a value is not provided, it defaults to `http://localhost:4318/v1/metrics`
2. `batchSize` - number of ``Meter``s to include in a single payload sent to the backend. The default is 10,000.
3. `aggregationTemporality` - https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality[Aggregation temporality, window=_blank] determines how the additive quantities are expressed, in relation to time. The environment variable `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` is supported by the default implementation. The supported values are `cumulative` or `delta`. Defaults to `cumulative`.
4. `headers` - Additional headers to send with exported metrics. This can be used for authorization headers. By default, headers are loaded from the config. If that is not set, they can be taken from the environment variables `OTEL_EXPORTER_OTLP_HEADERS` and `OTEL_EXPORTER_OTLP_METRICS_HEADERS`. If a header is set in both the environmental variables, the header in the latter overrides the former.
Expand Down Expand Up @@ -78,7 +78,7 @@ You may use a different `HttpSender` implementation by creating and configuring
include::{include-java}/metrics/OtlpMeterRegistryCustomizationTest.java[tags=customizeHttpSender, indent=0]
-----

You can also provide a custom implementation of `OtlpMetricsSender` that does not use HTTP at all.
You can also provide a custom implementation of `OtlpMetricsSender` that does not use `HttpSender` at all. `OtlpConfig#url` will be used as the address when the sender is called in the `OtlpMeterRegistry` `publish` method.
For instance, if you made a gRPC implementation, you could configure it in the following way.
Micrometer does not currently provide a gRPC implementation of `OtlpMetricsSender`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ class OtlpMeterRegistryCustomizationTest {
void customizeHttpSender() {
// tag::customizeHttpSender[]
OtlpConfig config = OtlpConfig.DEFAULT;
OtlpHttpMetricsSender httpMetricsSender = new OtlpHttpMetricsSender(new OkHttpSender(), config);
OtlpHttpMetricsSender httpMetricsSender = new OtlpHttpMetricsSender(new OkHttpSender());
OtlpMeterRegistry meterRegistry = OtlpMeterRegistry.builder(config).metricsSender(httpMetricsSender).build();
// end::customizeHttpSender[]
}
Expand All @@ -49,7 +49,7 @@ void customizeOtlpSender() {
private static class OtlpGrpcMetricsSender implements OtlpMetricsSender {

@Override
public void send(byte[] metricsData, Map<String, String> headers) {
public void send(String address, byte[] metricsData, Map<String, String> headers) {
}

}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,12 @@ interface MetricsSender {
/**
* Send encoded metrics data from a {@link io.micrometer.core.instrument.MeterRegistry
* MeterRegistry}.
* @param address where to send the metrics
* @param metricsData encoded batch of metrics
* @param headers metadata to send as headers with the metrics data
* @throws Throwable when there is an exception in sending the metrics; the caller
* should handle this in some way such as logging the exception
*/
void send(byte[] metricsData, Map<String, String> headers);
void send(String address, byte[] metricsData, Map<String, String> headers) throws Throwable;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't mention it in the PR description but throws Throwable was added here and elsewhere. The point was to ensure that the caller (MeterRegistry publish implementation) handles the exception case, usually with logging. HttpSender similarly declares throws Throwable. That said, it may be worth looking at doing things differently here because catching Throwable includes things like OOME, as pointed out by @marcingrzejszczak in an offline review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can catch Exception instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be the address, headers, data order more logical?

Suggested change
void send(String address, byte[] metricsData, Map<String, String> headers) throws Throwable;
void send(String address, Map<String, String> headers, byte[] metricsData) throws Throwable;

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure. I think arguments could be made for a variety of orders. Switching to a builder pattern may help avoid the matter of order entirely.


}
Original file line number Diff line number Diff line change
Expand Up @@ -32,32 +32,31 @@ public class OtlpHttpMetricsSender implements OtlpMetricsSender {

private final HttpSender httpSender;

private final OtlpConfig config;

private final String userAgentHeader;

public OtlpHttpMetricsSender(HttpSender httpSender, OtlpConfig config) {
public OtlpHttpMetricsSender(HttpSender httpSender) {
this.httpSender = httpSender;
this.config = config;
this.userAgentHeader = getUserAgentHeader();
}

/**
* Send a batch of OTLP Protobuf format metrics to an OTLP HTTP receiver.
* @param address address of the OTLP HTTP receiver to which metrics will be sent
* @param metricsData OTLP protobuf encoded batch of metrics
* @param headers metadata to send as headers with the metrics data
* @throws Throwable when there is an exception in sending the metrics; the caller
* should handle this in some way such as logging the exception
*/
@Override
public void send(byte[] metricsData, Map<String, String> headers) {
HttpSender.Request.Builder httpRequest = this.httpSender.post(config.url())
public void send(String address, byte[] metricsData, Map<String, String> headers) throws Throwable {
HttpSender.Request.Builder httpRequest = this.httpSender.post(address)
.withHeader("User-Agent", userAgentHeader)
.withContent("application/x-protobuf", metricsData);
headers.forEach(httpRequest::withHeader);
try {
HttpSender.Response response = httpRequest.send();
if (!response.isSuccessful()) {
logger.warn(
"Failed to publish metrics (context: {}). Server responded with HTTP status code {} and body {}",
getConfigurationContext(), response.code(), response.body());
}
}
catch (Throwable e) {
logger.warn("Failed to publish metrics (context: {}) ", getConfigurationContext(), e);
HttpSender.Response response = httpRequest.send();
if (!response.isSuccessful()) {
throw new OtlpHttpMetricsSendUnsuccessfulException(String
.format("Server responded with HTTP status code %d and body %s", response.code(), response.body()));
}
}

Expand All @@ -70,14 +69,12 @@ private String getUserAgentHeader() {
return userAgent;
}

/**
* Get the configuration context.
* @return A message containing enough information for the log reader to figure out
* what configuration details may have contributed to the failure.
*/
private String getConfigurationContext() {
// While other values may contribute to failures, these two are most common
return "url=" + config.url() + ", resource-attributes=" + config.resourceAttributes();
private static class OtlpHttpMetricsSendUnsuccessfulException extends RuntimeException {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking if we should make this more general and public so that other implementations can reuse it and users can get the same exception class for the different implementation.


public OtlpHttpMetricsSendUnsuccessfulException(String message) {
super(message);
}

}

}
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ public OtlpMeterRegistry(OtlpConfig config, Clock clock) {
* @since 1.14.0
*/
public OtlpMeterRegistry(OtlpConfig config, Clock clock, ThreadFactory threadFactory) {
this(config, clock, threadFactory, new OtlpHttpMetricsSender(new HttpUrlConnectionSender(), config));
this(config, clock, threadFactory, new OtlpHttpMetricsSender(new HttpUrlConnectionSender()));
}

private OtlpMeterRegistry(OtlpConfig config, Clock clock, ThreadFactory threadFactory,
Expand Down Expand Up @@ -181,14 +181,25 @@ protected void publish() {
.build())
.build();

metricsSender.send(request.toByteArray(), this.config.headers());
metricsSender.send(config.url(), request.toByteArray(), config.headers());
}
catch (Throwable e) {
logger.warn("Failed to publish metrics to OTLP receiver", e);
logger.warn(String.format("Failed to publish metrics to OTLP receiver (context: %s)",
getConfigurationContext()), e);
}
}
}

/**
* Get the configuration context.
* @return A message containing enough information for the log reader to figure out
* what configuration details may have contributed to the failure.
*/
private String getConfigurationContext() {
// While other values may contribute to failures, these two are most common
return "url=" + config.url() + ", resource-attributes=" + config.resourceAttributes();
}

@Override
protected <T> Gauge newGauge(Meter.Id id, @Nullable T obj, ToDoubleFunction<T> valueFunction) {
return new DefaultGauge<>(id, obj, valueFunction);
Expand Down Expand Up @@ -492,7 +503,7 @@ public static class Builder {

private Builder(OtlpConfig otlpConfig) {
this.otlpConfig = otlpConfig;
this.metricsSender = new OtlpHttpMetricsSender(new HttpUrlConnectionSender(), otlpConfig);
this.metricsSender = new OtlpHttpMetricsSender(new HttpUrlConnectionSender());
}

/** Override the default clock. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,14 @@
public interface OtlpMetricsSender extends MetricsSender {

/**
* Send a batch of OTLP Protobuf format metrics to a receiver.
* Send a batch of OTLP Protobuf format metrics to an OTLP receiver.
* @param address address of the OTLP receiver to which metrics will be sent
* @param metricsData OTLP protobuf encoded batch of metrics
* @param headers metadata to send as headers with the metrics data
* @throws Throwable when there is an exception in sending the metrics; the caller
* should handle this in some way such as logging the exception
*/
@Override
void send(byte[] metricsData, Map<String, String> headers);
void send(String address, byte[] metricsData, Map<String, String> headers) throws Throwable;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think taking the address as a String is generic enough to work for other implementations than the HTTP one, but let me know if anyone has a concern about this. The implementation is responsible for parsing the address String. The OtlpMeterRegistry will always send the OtlpConfig#url as the address, so while it may semantically be weird if the address isn't actually a URL but the configured sender implementation can handle it (since it is a String that isn't necessarily a URL), this should work nonetheless.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm always cautious of adding extra parameters to methods. Would the API be cleaner if send method's parameters were consolidated to a value type?

Something like this (pseudocode incoming)

interface MetricsSender {

    /**
     * Send encoded metrics data from a {@link io.micrometer.core.instrument.MeterRegistry
     * MeterRegistry}.
     * @param metricsData encoded batch of metrics
     * @param headers metadata to send as headers with the metrics data
     */
    void send(MetricsSenderPayload payload);

    class MetricsSenderPayload {
        private final String address;
        private final byte[] payload;
        private final Map<String, String> headers;

        // constructor and accessors omitted
    }

and then MeterRegistry can just do this

MetricsSender.MetricsSenderPayload payload = new MetricsSender.MetricsSenderPayload(
    this.config.url(),
    request.toByteArray(),
    this.config.headers()
);
metricsSender.send(payload);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review. We could do something like that, and there's precedent in the codebase with HttpSender taking a Request object. I can take a closer look at it tomorrow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've played around with this a bit locally, but there are a number of choices, especially given the package-private MetricsSender (I would probably get rid of MetricsSender entirely when adding a Request type for simplicity). I've suggested to the team a pragmatic way of moving forward would be to merge this as is, since it fixes the issue described in the PR description. Then in a follow-up PR, probably after the milestone release set to go out today, we can iterate on the feedback given.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm totally fine calling it address but if you are interested in an alternative: endpoint.


}
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ void setUp() {
this.clock = new MockClock();
OtlpConfig config = otlpConfig();
this.mockHttpSender = mock(HttpSender.class);
OtlpMetricsSender metricsSender = new OtlpHttpMetricsSender(mockHttpSender, config);
OtlpMetricsSender metricsSender = new OtlpHttpMetricsSender(mockHttpSender);
this.registry = OtlpMeterRegistry.builder(config).clock(clock).metricsSender(metricsSender).build();
this.registryWithExponentialHistogram = new OtlpMeterRegistry(exponentialHistogramOtlpConfig(), clock);
}
Expand Down