[processor/transform] OTTL aggregation function support for Logs

[drewrelmas]

Component(s)

processor/transform

Describe the issue you're reporting

In reading up more on supported OTTL functions, I noticed that aggregate logic like aggregate_on_attributes are only compatible with metrics.

I didn't open this issue as a feature request as there could be background knowledge in OTEL I am lacking about why this isn't a good idea but was envisioning a case where a customer is only interested in collecting aggregated data about log events with the main goal of reducing event volume. This isn't the same as using filter processor to drop records on certain conditions, but more on condensing data without loss.

One example could be grouping related logs into a single exemplar record with key statistics (count, etc...). Another use case is conditional rate-limiting of exported data - sending all log records by default and only aggregating once a threshold limit is breached.

This type of information could be better represented by a metric signal in the original instrumentation, but it could be convenient for a customer to avoid refactoring and be able to manipulate log records to get the same result in the Collector.

[github-actions]

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley @edmocosta

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[axw]

@drewrelmas apologies for the delay.

I can think of a couple of ways you could go about some of these things, depending on needs:

• You could use https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/logdedupprocessor to de-duplicate logs.
• You could use https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/signaltometricsconnector to derive metrics from logs (and then aggregate those metrics).
• You could use https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor apply probabilistic sampling of logs. I think to make it conditional you would either need to make it adaptive or reservoir-sampling. Per https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor#sampling-threshold-information, the sampling threshold information is recorded as attributes on the sampled log records. You could use this to estimate the original count. I know this is information is honoured by signaltometricsconnector for spans, but I'm not sure off hand if it is for logs too.

Anyway, let me know if that helps at all, and whether you still think an OTTL aggregation function would be needed.

[drewrelmas]

Thanks for the response, I'll check into these sources and read up a little more.

[axw]

@drewrelmas I'll close this for now just to avoid it lingering indefinitely. If after you've researched those options you still feel there's a gap, could you please open a new issue explaining how the existing options are insufficient? Thank you!