[Bug]: Elasticsearch default compression breaks OpenSearch index creation

[dutchhaag]

What happened?

Hello,

I am using Jaeger with the OpenSearch storage backend. I am using AWS Opensearch version 2.19.

I recently attempted to upgrade from Jaeger 2.5.0 to 2.6.0, but I have ran into a bug with Index Creation using the opensearch storage backend which is causing my Jaeger service to fail.

This commit 1b1cfe2 by @timonegk sets the gzip encryption by default, but I do not believe we want this on the call to create the index, rather just on calls which submit traces.

Rolling back to 2.5.0 successfully creates my index.

Steps to reproduce

Setup latest jaeger 2.6.0 to use an AWS Opensearch storage backend
Attempt to start Jaeger, see failure in OS Index creation.
Roll back to 2.5.0 and see OS Index creation succeed.

Expected behavior

Index creates successfully and my Jaeger instance turns healthy.

Relevant log output

`Error: cannot start pipelines: failed to start "jaeger_storage_exporter" exporter: cannot create trace write: failed to create template "traces-jaeger-span": elastic: Error 500 (Internal Server Error): Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes [type=not_x_content_exception]`

Screenshot

No response

Additional context

No response

Jaeger backend version

2.6.0

SDK

Latest jaeger version attempting to store traces in 2.19 Opensearch AWS.
https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/config-opensearch.yaml Use this configuration.

Pipeline

No response

Stogage backend

AWS OpenSearch

Operating system

Linux

Deployment model

ECS

Deployment configs

[yurishkuro]

Thanks for reporting

[suvaidkhan]

Hey @yurishkuro new to Jaeger, would you mind providing a bit more context on this?
Thanks!

[yurishkuro]

I think the issue description is already quite detailed and well researched. What is not clear?

[suvaidkhan]

@yurishkuro do I need to default the--es.http-compression flag to false or override this flag to false when creating indexes?
If its the latter I'm not sure where I should add this check?

Thanks.

[timonegk]

I am actually using Jaeger with AWS OpenSearch as well and index creation works. Could you verify that the compression is at fault by running Jaeger with --es.http-compression=false in 2.6.0?

[dutchhaag]

That flag does not appear to be working on 2.6.0

[yurishkuro]

@dutchhaag v2 is not configured via flags, there is a corresponding config property

[dutchhaag]

Thank you @yurishkuro . I attempted to set the http compression to false with this block in my config. But still receive the same error.

jaeger_storage:
    backends:
      some_storage:
        opensearch:
          server_urls:
            - http://localhost:9200
          http_compression: false #disable compression
          indices:
            index_prefix: "traces"
            spans:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1
            services:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1
            dependencies:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1
            sampling:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1