Inject component-identifying scope attributes #12617
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
requested review from
andrzej-stencel,
bogdandrutu,
dmathieu,
dmitryax and
evan-bradley
jmacd
approved these changes
Mar 18, 2025
mx-psi
reviewed
Mar 19, 2025
|
We discussed offline adding a feature gate for this and all other internal telemetry related changes, I intend to merge this once the comments are addressed and the feature gate has been added |
mx-psi
reviewed
Mar 27, 2025
mx-psi
approved these changes
Mar 28, 2025
Merged
via the queue into
open-telemetry:main
with commit Mar 28, 2025
54c13a9
52 of 56 checks passed
3 tasks
github-merge-queue bot
pushed a commit
that referenced
this pull request
Apr 28, 2025
…behind feature gate (#12933) #### Context PR #12617 introduced logic to inject new instrumentation scope attributes in all internal telemetry to identify which Collector component it came from. These attributes had already been added to internal logs as regular log attributes, and this PR switched them to scope attributes for consistency. The new logic was placed behind an Alpha stage feature gate, `telemetry.newPipelineTelemetry`. Unfortunately, the default "off" state of the feature gate disabled the injection of component-identifying attributes entirely, which was a regression since they had been present in internal logs in previous releases. See issue #12870 for an in-depth discussion of this issue. To correct this, PR #12856 was filed, which stabilized the feature gate, making it on by default, with no way to disable it, and removed the logic that the feature gate used to toggle. This was thought to be the simplest way to mitigate the regression in the "off" state, since we planned to stabilize the feature eventually anyways. Unfortunately, it was found that the "on" state of the feature gate causes a different issue: [the Prometheus exporter](https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/prometheus) is the default way of exporting the Collector's internal metrics, accessible at `collector:8888/metrics`. This exporter does not currently have any support for instrumentation scope attributes, meaning that metric streams differentiated by said attributes but not by any other identifying property will appear as aliases to Prometheus, which causes an error. This completely breaks the export of Collector metrics through Prometheus under some simple configurations, which is a release blocker. #### Description To fix this issue, this PR sets the `telemetry.newPipelineTelemetry` feature gate back to "Alpha" (off by default), and reintroduces logic to disable the injection of the new instrumentation scope attributes when the gate is off, but only in internal metrics. Note that the new logic is still used unconditionally for logs and traces, to avoid reintroducing the logs issue (#12870). This should avoid breaking the Collector in its default configuration while we try to get a fix in the Prometheus exporter. #### Link to tracking issue No tracking issue currently, will probably file one later. #### Testing I performed some simple manual testing with a config file like the following: ```yaml receivers: otlp: [...] processors: batch: exporters: debug: [...] service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [debug] traces: receivers: [otlp] processors: [batch] exporters: [debug] telemetry: metrics: level: detailed traces: [...] logs: [...] ``` The two batch processors create aliased metric streams, which are only differentiated by the new component attributes. I checked that: 1. this config causes an error in the Prometheus exporter on main; 2. the error is resolved by default after applying this PR; 3. the error reappears when enabling the feature gate (this is expected) 4. scope attributes are added on the traces and logs no matter the state of the gate.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fork of #12384 to showcase how component attributes can be injected into scope attributes instead of log/metric/span attributes. See that PR for more context.
To see the diff from the previous PR, filter changes starting from the "Prototype using scope attributes" commit.
Link to tracking issue
Resolves #12217
Also incidentally resolves #12213 and resolves #12117
Testing
I updated the existing tests to check for scope attributes, and did some manual testing with a debug exporter to check that the scope attributes are added/removed properly.