Skip to content

Duplicated traces when API Gateway AWS_PROXY enabled and using XRAY #175

Open
@xrn

Description

@xrn

Hey team,

My project is using

github.com/DataDog/datadog-lambda-go v1.17.0
gopkg.in/DataDog/dd-trace-go.v1 v1.64.1
https://github.com/DataDog/datadog-lambda-extension/releases/tag/v58

Lambda config

DD_API_KEY ....
DD_APM_DD_URL http://xxx.xxx.xxx.xxx:3835
DD_CAPTURE_LAMBDA_PAYLOAD true
DD_ENV PROD
DD_LOGS_CONFIG_LOGS_DD_URL http://xxx.xxx.xxx.xxx:3835
DD_LOGS_CONFIG_LOGS_NO_SSL true
DD_LOGS_CONFIG_USE_COMPRESSION true
DD_LOGS_CONFIG_USE_HTTP true
DD_LOGS_ENABLED true
DD_MERGE_XRAY_TRACES true
DD_SITE datadoghq.eu
DD_TRACE_ENABLED true
DD_TRACE_STARTUP_LOGS false
DD_UNIVERSAL_INSTRUMENTATION true
DD_URL http://xxx.xxx.xxx.xxx:3835

We are using API Gateway and Lambdas, And I found an incorrect behavior from end user perspective. When API Gateway integrates with Lambda using AWS intregration type all work well - I have nice tree of connected spans all is beautiful for over 50 endpoints which we have. Worth to note that API is represented as https://domain.com and properties like

{
   "api":{
      "endpoint":{
         "method":"GET",
         "registered":"true",
         "route":"/v0/.....",
         "route_id":"/v0/...."
      }
   },
   "api_gateway":{
      "account_id":"...",
      "request_id":"....",
      "rest_api_id":".....",
      "stage":"v0"
   },
   "aws_api_id":"....",
   "aws_api_stage":"v0",
   "aws_base_path":"v0",
   "aws_domain":"....",

But there are 2 endpoint which are using AWS_PROXY integration type as they are returning binary files and such integration works better for us. And now traces are not so perfect - we are getting new service domain.com instead of https://domain.com with completly different set of properties which do not includes "api" json object. In some cases we see only connected that span + lambda in other we see that + full trace like in regular execution combined together.

I am not sure if this is right repo, or maybe I should report this at Agent repo - but sth is wrong here - switch to AWS_PROXY integration type from AWS should not have that impact.

At the end this leads to:

  1. In traces I have 2 services https://domian/com and domain.com
  2. In API Catalog I have endpoint without assigned API - which is also incorrect and is happening only for AWS_PROXY endpoints

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions