Propose using a different schema to represent Events in a span

[awangc]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

When storing Span Events in elasticsearch, the event name becomes the key in the default mapping mode, under which different attributes are stored, e.g. if we have events with name "my-event-1", "my-event-2", then in Elasticsearch we'll have Events.my-event-1.time, Events.my-event-2.time, etc. This does not seem to follow the data format for events for a Span from opentelemetry collector, which are modeled as an array of Span_Event, in which a Span_Event will contain fields like time, name and array of attributes.

The issue I see with this approach is that if name is given arbitrary values (e.g. random UUIDs), then we could see an arbitrary increase in the number of keys, leading to mapping field explosion in Elasticsearch.

Describe the solution you'd like

Store span events as an array in elasticsearch, in which each element is an object with fields with time, name and array of attribute (with dropped attribute counts as another possible field - like the Span_Event class)

Admittedly this format may require nested objects which may bring its own performance issues, but it resembles more the data layout from opentelemetry pdata.

Describe alternatives you've considered

No response

Additional context

The schema proposed above would follow the same format for spans, e.g., we have Span.Name and Span.Attributes, and we'd have Event.Name and Event.Attributes, and more closely represents the Event as defined in opentelemetry

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[gregkalapos]

Hey 👏

with the default mapping mode (which is the mode none), this is indeed the case, but for OTel traces, I'm not sure how useful that mode is at all. If you look at how data is stored, most things won't follow the shape of the data suggested by OTel - e.g. resource attributes are also stored under Resource.* directly and Kibana is not showing those traces either on the APM UI and the service isn't visible anywhere under Observability - and that's because the none mapping mode.

So for traces, the otel mapping mode would be the way to go.

For that mapping mode there was already a lot of thinking on how data is stored.

When it comes to how OTel data is stored in Elasticsearch, with the otel mapping mode, we have 3 main data streams where data ends up - 1) traces 2) logs, and 3) metrics.

All I'm saying here is meant for the otel mapping mode.

The idea is that events are modelled as log records, therefore they'll end up in the logs data stream - that's very natural for log events, may not be for span events, but I think so far, the idea is that all events will end up in the same data stream. You can of course connect back those span events to the original span via the span id.

This is aligned with what's stated here in the docs.

It's interesting to see that in OTLP, there is a specific type for span events which is totally different from log events.

In any case - with the otel mapping mode, there should not be a cardinality problem, because the event name is a top level field and other fields - e.g. attributes are not embedded under the event name in that case.

So, I'd say the otel mapping mode is the way to go.

Having said that, a few throughs on the default mapping mode (none):

The issue I see with this approach is that if name is given arbitrary values (e.g. random UUIDs), then we could see an arbitrary increase in the number of keys, leading to mapping field explosion in Elasticsearch.

That's technically clearly possible, and would be indeed an issue. Honestly I'd have expected to spec to say something about event name cardinality, all I find is this part here:

It is also recommended that the event names have low-cardinality, so care must be taken to use fields that identify the class of Events but not the instance of the Event.

With that, I'd say, if the user follows the spec, there should not be cardinality explosion. On that other hand, the events API easily allows this, so I think you raise a good point here.

[awangc]

@gregkalapos it seems the otel mapping mode constructs indices based on different attributes, and span events are sent to a logs- index. Is there a way to change this integration? Like having the elasticsearch.index.prefix attributes that can change the prefix for the trace index? This is so that we can group all related data with the same prefix.

EDIT: I removed a section of this comment that said I could not get this mapping mode to work, that was because I was on an old version of Elasticsearch. I'm currently on 8.15.5 and it is working.

[carsonip]

Looking at the code, elasticsearch.index.prefix should work, but documents will then be sent to a fallback index name which is a concatenation of prefix, configured index name, and suffix.

OTel mode is designed to send to ES 8.16+, and with the default setup, a lot of the curated UI in Kibana e.g. APM UI will work out of the box. Changing the index name will certainly break that functionality.

Hence my question, what's the use case of sending span events to have a different data stream type? Do you usually query the documents using an API, or in Discover? How does sending span events to a logs- data stream as opposed to something else affect your workflow?

Valid data stream types in ECS are logs, metrics and traces. Since span events aren't metrics and traces, they could only be logs, so that they work well in APM UI out of the box.

[awangc]

@carsonip Actually, I'm trying a custom solution that is not based on Elastic APM 😄 , thus my desire to be able to configure the index prefix. In our logging solution we do not use logs- prefix, but have our custom prefix, partly so that we don't interfere with any solutions predefined by Elastic.

If otel mode is set to support APM, how about changing the schema of default mode for Span Events? Since these are currently stored in the index where traces are being stored?

[carsonip]

@awangc sorry for the delayed response. Have you tried using otel mode with traces_dynamic_index::enabled: true and setting elasticsearch.index.prefix in a processor?

processors:
  transform:
    trace_statements:
      - context: spanevent
        statements:
          - set(attributes["elasticsearch.index.prefix"], "foo")

and es exporter config:

exporters:
  elasticsearch:
    traces_dynamic_index:
      enabled: true
    traces_index: bar

I'm getting

2025-02-06T18:52:51.643Z        error   elasticsearchexporter@v0.115.0/bulkindexer.go:346       failed to index document        {"kind": "exporter", "data_type": "traces", "name": "elasticsearch", "index": "foobar", "error.type": "index_not_found_exception", "error.reason": "no such index [foobar] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.115.0/bulkindexer.go:346
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.115.0/bulkindexer.go:318
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.115.0/bulkindexer.go:294

Notice the index is now called foobar, which is from prefix "foo" and traces_index "bar". which means with the right index templates, they should index properly and just work. However, the index template should be similar to the otel index templates (especially the mappings) in 8.16 Elasticsearch for passthrough fields to work correctly.