[exporterhelper] Add an ability to batch by a key from pdata or context

[dmitryax]

Introduce a new exporterhelper.BatcherOption with the following signature:

func WithBatchIdentifier(f IdentifyBatchFunc, batchIdentifiersLimit int) BatcherOption`

where

// IdentifyBatchFunc returns an identifier for a batch. This function can be used to separate particular
// Requests into different batches. Batches with different identifiers will not be merged together.
// Provided context can be used to extract information from the context and use it as a part of the identifier as well.
type IdentifyBatchFunc func(ctx context.Context, r Request) string

batchIdentifiersLimit defines a maximum number of allowed identifiers. Requests producing batch identifiers that exceed this limit will be dropped. If this value is zero, then there is no limit on the number of identifiers.

The naming can be changed. The idea is that we need to provide exporters with a way to produce different batches based on some custom string key taken from the Request of context. This API would allow us to replicate the metadata_keys option in the batch processor, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md#batching-and-client-metadata.

Initial version of the exporter batch that includes this functionality can be found in 5d662b6

[sfc-gh-sili]

I can take a look at this issue. @dmitryax Would you mind assigning to me?

[jmacd]

@bogdandrutu FYI

[jmacd]

As the author of the batchprocessor metadata-keys feature, I would report that we eventually found it necessary to add metadata-key support in the OTel-Arrow streaming exporter itself. This is because if you want to use metadata as authorization at the stream level (e.g., so a load balancer can see it), then individual export streams have to be pinned to the metadata values. I am not sure if this will be possible in the exporterhelper, but thought I should mention it.

See this: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/otelarrowexporter/README.md#matching-metadata-per-stream

[sfc-gh-sili]

Checked in with @jmacd offline to have a better understanding of the use case of metadata-based key batching in otel-arrow exporter.

OpenTelemetry Arrow exporter and receiver was developed to optimize network usage. Otel-arrow exporter converts OTLP data to Apache Arrow format to save bandwidth and Otel-Arrow receiver reverses the process. The graph below illustrates this:

Each stream begins with a "stream head," and the connection remains open for some time. The OTAP layer expects requests within the same stream to correspond to the same tenant and uses stream-level metadata in the header for rate-limiting, load balancing, and access control purpose. Therefore, the Otel-Arrow exporter batches based on the stream

[axw]

Possibly of interest: in open-telemetry/opentelemetry-collector-contrib#38888 (comment) I have implemented a PoC of routing & partitioning batches using OTTL. That enables arbitrary partitioning, e.g. by trace ID, by service.name resource attribute, by X-Tenant-Id client metadata, etc.

This wouldn't work in core unless we were to move OTTL here, but it should be possible to use in a contrib processor whose only job is partitioning. The processor would split requests and include the partition key in each request's metadata. This key could then be used in the batch helper.

So, with that in mind, I wonder if we should only support batching by request metadata like in the batch processor. Then if someone wants to partition/batch by some property of the data they can use the partitioning processor.

[dmitryax]

Can be tracked separately as part of #12473