Problem creating the index in elastic cloud for traces received via otlp

[Kaua3045]

Component(s)

exporter/elasticsearch

What happened?

Description

When sending data to the elastic cloud, it ends up not creating the dynamic index automatically for traces, only for logs.

Steps to Reproduce

Configure the dynamic index for traces using set(attributes["elasticsearch.index"], Concat(["traces", attributes["service.name"], attributes["environment"]], "-"))

Expected Result

The index is created in elasticsearch based on the service.name and environment

Actual Result

It doesn't create, it only works if you remove everything and leave the index creation to default traces

Collector version

0.127.0 (latest)

Environment information

Environment

OS: EC2 with Debian 20
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    exporters:
      debug:
        verbosity: detailed
      elasticsearch:
        endpoint: url
        tls:
          insecure_skip_verify: true
        api_key: ****

      prometheus:
        endpoint: 0.0.0.0:9464

    processors:
      transform/logs:
         log_statements:
         - context: log
           statements:
             - delete_key(attributes, "code.function")
             - delete_key(attributes, "code.lineno")
         - context: resource
           statements:
             - set(attributes["elasticsearch.index"], Concat(["logs", attributes["service.name"], attributes["environment"]], "-"))
      transform/traces:
         trace_statements:
         - context: resource
           statements:
             - set(attributes["elasticsearch.index"], Concat(["traces", attributes["service.name"], attributes["environment"]], "-"))
      resource:
        attributes:
          - key: "process.pid"
            action: delete
          - key: "process.executable.path"
            action: delete
          - key: "process.command_line"
            action: delete
          - key: "process.runtime.name"
            action: delete
          - key: "telemetry.sdk.language"
            action: delete
          - key: "telemetry.sdk.name"
            action: delete
          - key: "telemetry.sdk.version"
            action: delete

      batch: {}

    extensions:
      health_check: {}

    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [transform/traces, resource]
          exporters: [elasticsearch,debug]
        metrics:
          receivers: [otlp]
          exporters: [prometheus]
        logs:
          receivers: [otlp]
          processors: [transform/logs, resource]
          exporters: [elasticsearch]

Log output

2025-06-03T00:39:16.198Z	error	elasticsearchexporter@v0.127.0/bulkindexer.go:379	failed to index document	{"resource": {}, "otelcol.component.id": "elasticsearch", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "index": "traces-template-dev", "error.type": "index_not_found_exception", "error.reason": ""}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.127.0/bulkindexer.go:379
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.127.0/bulkindexer.go:350
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.127.0/bulkindexer.go:343
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.127.0/bulkindexer.go:235

Additional context

spring application.yml config

otel:
  propagators:
    - b3
  resource:
    attributes:
      environment: dev
      service:
        name: invoices

At some point I even noticed that elasticsearch.index was appearing in attributes in the debug logs and there it was like this elasticsearch.index: traces-nil-nil, but then in the next traces it no longer showed, the resources are arriving as I will show below, but only the log index is actually created.

2025-06-03T01:21:46.076Z	info	Traces	{"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 7}
2025-06-03T01:21:46.076Z	info	ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.24.0
Resource attributes:
     -> environment: Str(dev)
     -> host.arch: Str(amd64)
     -> host.name: Str(invoices-dev-56d798cdf-chvsj)
     -> os.description: Str(Linux 6.1.0-37-cloud-amd64)
     -> os.type: Str(linux)
     -> process.command_args: Slice(["/opt/java/openjdk/bin/java","-jar","/opt/app/application.jar"])
     -> telemetry.distro.name: Str(opentelemetry-spring-boot-starter)
     -> elasticsearch.index: Str(traces-template-dev)
     -> process.runtime.description: Str(Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.7+6-LTS)
     -> telemetry.distro.version: Str(2.11.0)
     -> process.runtime.version: Str(21.0.7+6-LTS)
     -> service.instance.id: Str(7e3251f1-92a2-462d-80b2-43e58af940e7)
     -> service.name: Str(template)
     -> service.version: Str(0.1.0)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope invoices 
Span #0
    Trace ID       : 79f0b4191af413a0890329c5a7fcd40f
    Parent ID      : 10bc2714159d38c8
    ID             : 3ad14e61d3f31230
    Name           : http.filter.idempotency_key_filter
    Kind           : Internal
    Start time     : 2025-06-03 01:21:45.789409565 +0000 UTC
    End time       : 2025-06-03 01:21:45.791822205 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> http.user_agent: Str(Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36)
     -> http.method: Str(GET)
     -> http.path: Str(/api/ping)
     -> http.remote_address: Str(10.42.0.31)
ScopeSpans #1
ScopeSpans SchemaURL: 
InstrumentationScope org.springframework.boot 3.4.2
Span #0
    Trace ID       : 79f0b4191af413a0890329c5a7fcd40f
    Parent ID      : 788d7c1b479669e4
    ID             : 1f0142be25b86e25
    Name           : authorize request
    Kind           : Internal
    Start time     : 2025-06-03 01:21:45.788284715 +0000 UTC
    End time       : 2025-06-03 01:21:45.78877837 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> spring.security.authentication.authorities: Str(n/a)
     -> spring.security.authentication.type: Str(n/a)
     -> spring.security.object: Str(request)
     -> spring.security.authorization.decision.details: Str(AuthorizationDecision [granted=true])
     -> spring.security.authorization.decision: Str(true)
Span #1
    Trace ID       : 79f0b4191af413a0890329c5a7fcd40f
    Parent ID      : 1a4a9f1a250afd76
    ID             : 788d7c1b479669e4
    Name           : security filterchain before
    Kind           : Internal
    Start time     : 2025-06-03 01:21:45.787926733 +0000 UTC
    End time       : 2025-06-03 01:21:45.788866303 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> spring.security.filterchain.position: Str(13)
     -> spring.security.filterchain.size: Str(13)
     -> spring.security.reached.filter.name: Str(AuthorizationFilter)
     -> security.security.reached.filter.section: Str(before)

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[axw]

I suspect the issue here is that you don't have an Elasticsearch index template that matches traces-template-dev. By default, Elasticsearch ships with index templates that match logs-* and metrics-*, but not traces-*.

It does ship with an index template that matches traces-apm-*, and another that matches traces-*.otel-*. Since you're using the default mapping mode ("otel") you should be sending traces to a data stream the matches the second pattern I mentioned.

Try setting the index to traces-template.otel-dev, and see if that works. You may be better off setting data_stream.dataset and data_stream.namespace. When you do that, the exporter will automatically add this ".otel" suffix to the dataset.

[carsonip]

/label -needs-triage

[carsonip]

data_stream.dataset and data_stream.namespace. When you do that, the exporter will automatically add this ".otel" suffix to the dataset.

++ on setting attributes data_stream.dataset and data_stream.namespace. Sounds like the best path forward for this use case.

[Kaua3045]

Ok, using traces-.otel- it worked, correctly the index was created automatically, I will test as recommended later using the data stream

[axw]

Thanks. Closing as there's no bug in the exporter.