Error indexing in elasticsearchexporter when duplicate attributes recieved for log records

[stevejgordon]

Component(s)

exporter/elasticsearch

What happened?

Description

The OTel spec states that attributes with the same key should not be exported. However, duplicates can occur with the .NET SDK when using the IncludeScopes option. I'll raise an issue for that separately. When these are sent to a collector using the Elasticsearch exporter, the indexing fails with:

2025-04-10T13:56:00.069+0100    error   [email protected]/bulkindexer.go:343       bulk indexer flush error        {"otelcol.component.id": "elasticsearch/otel", "otelcol.component.kind": "Exporter", "otelcol.signal": "logs", "error": "flush failed (400): {\"error\":{\"type\":\"x_content_parse_exception\",\"caused_by\":{\"type\":\"json_parse_exception\"}}}"}

Steps to Reproduce

Send a log record with duplicate attributes to the collector. In an ASP.NET Core app, this can occur when using this registration, which enables IncludeScopes.

See open-telemetry/opentelemetry-dotnet#4324, which acknowledges this behaviour and non-compliance with the spec.

Expected Result

The log is exported to Elasticsearch

Actual Result

An error occurs, and the log (batch) is lost.

Collector version

0.120.1

Environment information

OpenTelemetry Collector configuration

Log output

Additional context

No response

[carsonip]

/label exporter/elasticsearch -needs-triage

[github-actions]

Pinging code owners for exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.