Tags set with SetTag aren't showing up in APM

[vesquen]

Expected Behavior

Using SetTag() on a span after getting it from the context with tracer.SpanFromContext() should add a tag to the span.

Actual Behavior

All default tags are there, but the added tag(s) aren't showing up.

Steps to Reproduce the Problem

1. Have an AWS API Gateway + Lamba setup (with Datadog Lambda Extension)
2. Wrap the handler with lambda.StartWithOptions(ddlambda.WrapFunction(handler.Handler(), nil))
3. Set a tag in the handler with:

return func(ctx context.Context, event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
		if span, ok := tracer.SpanFromContext(ctx); ok {
			span.SetTag("hello", "world")
		}
        
                ...
}

4. Check APM. The traces appear (inferred API Gateway span + cold start span + lambda handler span), but the "hello world"-tag is missing.

Specifications

• Datadog Go Lambda package version: v1.19.0
• Go version: 1.23.2
• gopkg.in/DataDog/dd-trace-go.v1 v1.68.0
• Lambda Extension: v65

ENV Vars:

• DD_ENHANCED_METRICS: true
• DD_FETCH_LAMBDA_TAGS: true
• DD_FLUSH_TO_LOG: true
• DD_LOG_LEVEL: ERROR
• DD_SERVERLESS_LOGS_ENABLED: true
• DD_SITE: datadoghq.com
• DD_TRACE_ENABLED: true

A similar issue was reported a while ago: #124

[purple4reina]

Hi @vesquen, this is sadly a known issue with how our lambda instrumentation works with golang and other languages that use what we call "universal instrumentation" (ie java, dotnet, ruby, go). For these languages we defer a lot of instrumentation work to the Datadog Extension, including the creation of the root aws.lambda span. Thus, it's not currently possible to add tags to this root span.

The work around would be to create another span and add your tags there.

return func(ctx context.Context, event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	child := tracer.StartSpan("child-span", tracer.ChildOf(ctx))
	child.SetTag("hello", "world")
	defer child.Finish()        

	...
}

[vesquen]

Hi @purple4reina

Thank you for your input (and the workaround). Given the nature of this “bug”, I suppose this won’t change in the short term?

[purple4reina]

@vesquen Unfortunately not in the short term, though we are actively considering alternative approaches. However, none of them would land in the coming quarters.

Will the work around I offered help you? If not, what exactly is it you are attempting to do. There should be a way for us to get you what you need, albeit with a little more creativity than otherwise necessary.

[vesquen]

Hi @purple4reina

The workaround works as expected.