Skip to content

Failure to disable Resource provider logs. #6772

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AntonioLyubchev opened this issue Oct 9, 2024 · 8 comments
Closed

Failure to disable Resource provider logs. #6772

AntonioLyubchev opened this issue Oct 9, 2024 · 8 comments
Labels
Bug Something isn't working

Comments

@AntonioLyubchev
Copy link

AntonioLyubchev commented Oct 9, 2024

Describe the bug
Failure to disable resource properties from a structured log message.

I want to log only resource attributes I set in the configuration YAML, so I added

otel:
  java:
    enabled:
      resource:
        providers: io.opentelemetry.sdk.autoconfigure.internal.EnvironmentResourceProvider

Which almost did the job, I still get:
image

"telemetry": {
  "sdk": {
    "name": "opentelemetry",
    "language": "java",
    "version": "1.42.1"
  }
}

I tried using:

otel:
  experimental:
    resource:
      disabled:
        keys: [ telemetry.sdk.language, telemetry.sdk.name, telemetry.sdk.version ] not working

Steps to reproduce
Create a project using Spring Boot Starter and configure otlp export to collector through grpc.
Produce a log message.
Observe the log message in elastic through the elastic exporter.

What did you expect to see?
When specifying io.opentelemetry.sdk.autoconfigure.internal.EnvironmentResourceProvider I expected to see only

"Resource": {
      "environment": "dev",
      "service": {
        "name": "PoC-OpenTelemetry-SP3-Webflux-REST"
      }
}

since this is what I have set in the application yaml.

What did you see instead?
I saw an additional resource in the structured log message:

"telemetry": {
  "sdk": {
    "name": "opentelemetry",
    "language": "java",
    "version": "1.42.1"
  }
}

Full YAML config:

# Simulated environment variable injection
APP_ENV: dev
LOGBACK_EXPERIMENTAL_LOG_ATTRIBUTES: false # Enable the capture of experimental log attributes thread.name and thread.id
LOGBACK_CAPTURE_MDC_ATTRIBUTES: "*" # Comma separated list of MDC attributes to capture. Use the wildcard character * to capture all attributes.
LOGBACK_CAPTURE_CODE_ATTRIBUTES: false # Enable the capture of source code attributes. Note that capturing source code attributes at logging sites might add a performance overhead.
LOGBACK_CAPTURE_MARKER_ATTRIBUTE: false # Enable the capture of Logback markers as attributes.
LOGBACK_CAPTURE_LOGGER_CONTEXT_ATTRIBUTES: false  # Enable the capture of Logback logger context properties as attributes. (This feature adds MDC properties as top level attributes instead of nested in _source)
LOGBACK_CAPTURE_KEY_VALUE_PAIR_ATTRIBUTES: false  # Enable the capture of Logback key value pairs as attributes.
OTLP_PROTOCOL: grpc
OTLP_ENDPOINT: http://localhost:4317

otel:
  exporter:
    otlp:
      endpoint: ${OTLP_ENDPOINT}
      protocol: ${OTLP_PROTOCOL}
  metrics:
    exporter: [ otlp ]
  traces:
    exporter: [ otlp ]
  propagators: [ tracecontext ]
  resource:
    attributes:
      environment: ${APP_ENV}
  instrumentation:
    logback-appender:
      experimental-log-attributes: ${LOGBACK_EXPERIMENTAL_LOG_ATTRIBUTES}
      experimental:
        capture-mdc-attributes: ${LOGBACK_CAPTURE_MDC_ATTRIBUTES}
        capture-code-attributes: ${LOGBACK_CAPTURE_CODE_ATTRIBUTES}
        capture-marker-attribute: ${LOGBACK_CAPTURE_MARKER_ATTRIBUTE}
        capture-logger-context-attributes: ${LOGBACK_CAPTURE_LOGGER_CONTEXT_ATTRIBUTES}
        capture-key-value-pair-attributes: ${LOGBACK_CAPTURE_KEY_VALUE_PAIR_ATTRIBUTES}
  java:
    enabled:
      resource:
        providers: io.opentelemetry.sdk.autoconfigure.internal.EnvironmentResourceProvider
  experimental:
    resource:
      disabled:
        keys: [ telemetry.sdk.language, telemetry.sdk.name, telemetry.sdk.version ]
  service:
    name: ${spring.application.name}

What version and what artifacts are you using?
How did you reference these artifacts? pom.xml

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-spring-boot-starter</artifactId>
</dependency>

...

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-instrumentation-bom</artifactId>
            <version>2.8.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Environment
Compiler: coretto-21.0.4
OS: Windows 11

Additional context
Just to note: io.opentelemetry.sdk.autoconfigure.internal.EnvironmentResourceProvider is not described here: https://opentelemetry.io/docs/languages/java/configuration/#resourceprovider

Also, can you please tell me how can I toggle all the other attributes present in the structured log:
image
For example, I don't want:

  • TraceFlags
  • SeverityText
  • SeverityNumber
  • data_stream.type
@AntonioLyubchev AntonioLyubchev added the Bug Something isn't working label Oct 9, 2024
@jack-berg
Copy link
Member

Oops replied here because I saw that comment before this issue:

Let's use this as an issue to track fixing otel.experimental.resource.disabled-keys to apply to the default attributes included in Resource.getDefault()

@AntonioLyubchev
Copy link
Author

@jack-berg I see that the disabled keys option is labeled:
"This option is experimental and subject to change or removal."
But I think this is incredibly important to be present since we need to control what's added to the structured log.

@AntonioLyubchev
Copy link
Author

AntonioLyubchev commented Oct 10, 2024

@jack-berg ahh.... found it

The yaml format is restricted to comma separated list. Found it after some debugging and digging:
image

This solves the issue with the SDK attributes:

otel:
  experimental:
    resource:
      disabled:
        keys: "telemetry.sdk.language,telemetry.sdk.name,telemetry.sdk.version,data_stream.type,SeverityNumber"

However, SeverityNumber and data_stream.type are still present in the final log.

I assume it's because the functionality removes only "Resources", but in that case how do we remove other unnecessary fields?

On a side note, can I log somehow the JSON message to the console for debugging purposes? Since the collector's "debug" exporter doesn't show the literal JSON message in the output.

@jack-berg
Copy link
Member

But I think this is incredibly important to be present since we need to control what's added to the structured log.

I agree. All of our properties go through a maturation cycle where they start as experimental for a period of time before being promoted to stable, with all the strong backwards compatibility guarantees. otel.experimental.resource.disabled-keys has been around for a while and its probably time to consider stabilizing it.

However, SeverityNumber and data_stream.type are still present in the final log. I assume it's because the functionality removes only "Resources", but in that case how do we remove other unnecessary fields?

SeverityNumber is a top level field in the log data model, so resource attribute configuration has no affect. If you're using the collector, you may be able to remove it with the transformprocessor (haven't confirmed this but its what I would try).

I'm not sure what data_stream.type is. It may be some artifact of the serializer used to print the data.

On a side note, can I log somehow the JSON message to the console for debugging purposes? Since the collector's "debug" exporter doesn't show the literal JSON message in the output.

You may be interested in the collector's fileexporter, which prints the OTLP JSON encoding of messages to files.

In opentelemetry-java we publish OtlpJsonLogging{Signal}Exporters out of io.opentelemetry:opentelemetry-exporter-logging-otlp. These are configured by setting OTEL_LOGS_EXPORTER=logging-otlp, as disusssed here.

@AntonioLyubchev
Copy link
Author

AntonioLyubchev commented Oct 10, 2024

Closing as there is no more work here.

A good idea would be documenting that most yaml properties strictly rely on comma separated list.

@jack-berg
Copy link
Member

A good idea would be documenting that most yaml properties strictly rely on comma separated list.

The YAML format you reference isn't defined as part of this repo. I believe its specific to the spring boot autoconfigure module. Maybe @zeitlinger, @jeanbisutti can comment.

@zeitlinger
Copy link
Member

the spring starter does support native lists - if proper support is added here - I simply forgot that one

@zeitlinger
Copy link
Member

Here's the PR to fix this open-telemetry/opentelemetry-java-instrumentation#12434

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants