AWS Lambda Runtime internal handlers need to be ignored from being instrumented and so traced. #10736
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
serkan-ozal
force-pushed
the
fix/instrumentation/aws-lambda/ignore-aws-lambda-internal-handlers
branch
from
b931fa8 to
a4ed1bf
Compare
…strumented and so traced. Otherwise, there might be double root spans.
serkan-ozal
force-pushed
the
fix/instrumentation/aws-lambda/ignore-aws-lambda-internal-handlers
branch
from
a4ed1bf to
8621e84
Compare
laurit
approved these changes
Mar 4, 2024
|
@serkan-ozal please, it looks like we'll need same check here. |
|
@cleverchuk I had not added it there because of SQSEvent check here: Since |
|
@serkan-ozal Gotcha. It looks sufficient to not add anymore code. Thanks for pointing that out. |
|
cc @tylerbenson |
6 tasks
srprash
added a commit
to aws-observability/aws-otel-java-instrumentation
that referenced
this pull request
Jan 15, 2025
…ed spans (#1000) ### Issue 1 - Duplicate lambda root spans The instrumentation produces 2 lambda root spans upon each invocation of the function. This is a known issue in OTel: open-telemetry/opentelemetry-java-instrumentation#7808 #### Fix - Otel has fixed it in [this PR](open-telemetry/opentelemetry-java-instrumentation#10736) but the change wasn't ported to the 1.x versions. So I took the diff from the upstream commit and made it into a patch that we apply when building the ADOT Java Lambda Layer for v1.33.x. ### Issue 2 - Unsampled spans do not produce Application Signals metrics On lambda environment, we export 100% of the spans to X-Ray to ensure we are able to provide 100% Application Signals metrics. However, currently only the sampled spans show up on the "Services" page and the unsampled spans do not. #### Fix - Upon comparing the sampled vs unsampled spans, I noticed that the unsampled spans are missing the attributes like `aws.local.service` and `aws.local.operation` which are required to generate Application Signals metrics. - The fix is to wrap the `OtlpUdpSpanExporter` instance for unsampled spans with the `AwsMetricAttributesSpanExporter` so that the exported spans have the desired attributes. #### Testing - After creating a layer with the fix, I set the `OTEL_TRACES_SAMPLER` to `always_off`. Then I invoked the function once. - The metrics appeared on the Application Signals console. - See the following screenshots <img width="1722" alt="image" src="https://github.com/user-attachments/assets/fa10e09f-ae24-4ab3-989f-838aacfb7e50" /> <img width="1722" alt="image" src="https://github.com/user-attachments/assets/d777304f-2cd9-4348-8e29-74f4a8b6b917" /> *Description of changes:* By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
lukeina2z
added a commit
to lukeina2z/aws-otel-java-instrumentation
that referenced
this pull request
Mar 13, 2025
V2 PR: open-telemetry/opentelemetry-java-instrumentation#10736 V2 Issue: Non-top level server span created by lambda instrumentation open-telemetry/opentelemetry-java-instrumentation#7808 The fix has been ported to v1 as a patch for non-stream handlers. This change extends the fix to the stream handler in the v1 repository. Testing: All unit tests pass. End-to-end tests pass. Backward Compatibility: No risk of breaking existing functionality. The change only adds instrumentation for RequestStreamHandler without modifying existing behavior for RequestHandler. Existing users who do not use RequestStreamHandler remain unaffected.
lukeina2z
added a commit
to aws-observability/aws-otel-java-instrumentation
that referenced
this pull request
Mar 14, 2025
…1047) - **V2 Issue:** [Non-top-level server span created by Lambda instrumentation](open-telemetry/opentelemetry-java-instrumentation#7808) - **V2 PR:** [#10736](open-telemetry/opentelemetry-java-instrumentation#10736) The fix has been ported to v1 as a patch for non-stream handlers. This change extends the fix to the stream handler in the v1 repository. ### Testing - All unit tests pass. - End-to-end tests pass. (with Java11, 17, and 21 runtime on Lambda) ### Backward Compatibility - No risk of breaking existing functionality. - The change only adds instrumentation for `RequestStreamHandler` without modifying existing behavior for `RequestHandler`. - Existing users who do not use `RequestStreamHandler` remain unaffected. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… Otherwise, there might be double root spans.
Fixes #7808