Skip to content

Commit 4014347

Browse files
xiuxiu
authored
[receiver/datadog] Address semconv noncompliance (#39678)
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description **Note: this PR addresses several things, happy to split it in multiple PRs if it helps.** In #36924, a number of noncompliances were identified. This PR addresses most of them: > Misalignments applying to all span types > > * A resource attribute deployment.resource.name should be defined instead of the span attribute deployment.environment: Str(production) > * A resource attribute process.pid should be defined instead of the attribute process.id Addressed by bumping semconv to latest in the collector (1.30.0) > Misalignments applying to HTTP server span > > * Adopt new OTel Semantic Conventions http.request.method and http.response.status_code instead of http.method and http.status_code > * The span name should be ${http.request.method} ${http.route}, the attribute dd.span.Resource is right for this span type > * OTTL transformation to fix the pb: set (name, attributes["dd.span.Resource"]) where name == "servlet.request" Added in 89e2bff > Misalignment applying to internal Spring Framework spans(span name spring.handler) > > Span should be kind = SpanKind.SPAN_KIND_INTERNAL > * OTTL transformation to fix the pb: set (kind, SPAN_KIND_INTERNAL) where kind == SPAN_KIND_SERVER and name == "spring.handler" > * Span name should be the Java method nam hat is carried over by the dd.span.Resource attribute > * OTTL transformation to fix the pb: `set (name, attributes["dd.span.Resource"]) where name I didn't add the span kind transformation for now as it seemed too involved for the receiver, happy to get feedback on that. Rest in 89e2bff > Misalignments applying to Database client spans > > * The resource attribute service.name should be set to the value of the span attribute _dd.base_service Done in d067d37. This might be a big change though. > * Rename the following span attributes > * db.type -> db.system > * db.operation -> db.operation.name > * db.instance -> db.name > * The span name should be set to ${db.operation.name} ${} Done in f78caac <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue #36924 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit tests were updated. <!--Please delete paragraphs that you did not use before submitting.-->
1 parent d4c2c6b commit 4014347

File tree

6 files changed

+484
-26
lines changed

6 files changed

+484
-26
lines changed

‎.chloggen/datadog_receiver_address_semconv_noncompliance.yaml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: enhancement
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
7+
component: datadogreceiver
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: Address semantic conventions noncompliance and add support for http/db
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues: [36924]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext: |
19+
* Bump semantic conventions to v1.30.0
20+
* Add support for http and db attributes
21+
* Use datadog's base service as service.name when available
22+
* Set `server.address` on client/producer/consumer spans
23+
* Properly name postgresql/redis/servlet/spring spans
24+
25+
# If your change doesn't affect end users or the exported elements of any package,
26+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
27+
# Optional: The change log or logs in which this entry should be included.
28+
# e.g. '[user]' or '[user, api]'
29+
# Include 'user' if the change is relevant to end users.
30+
# Include 'api' if there is a change to a library API.
31+
# Default: '[user]'
32+
change_logs: [user]

‎receiver/datadogreceiver/internal/translator/series_test.go

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import (
1414
"github.com/stretchr/testify/require"
1515
"go.opentelemetry.io/collector/component"
1616
"go.opentelemetry.io/collector/pdata/pmetric"
17+
semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
1718
)
1819

1920
func strPtr(s string) *string { return &s }
@@ -400,13 +401,13 @@ func TestTranslateSeriesV2(t *testing.T) {
400401
requireMetricAndDataPointCounts(t, result, 1, 0)
401402

402403
require.Equal(t, 1, result.ResourceMetrics().Len())
403-
v, exists := result.ResourceMetrics().At(0).Resource().Attributes().Get("host.name")
404+
v, exists := result.ResourceMetrics().At(0).Resource().Attributes().Get(string(semconv.HostNameKey))
404405
require.True(t, exists)
405406
require.Equal(t, "Host1", v.AsString())
406-
v, exists = result.ResourceMetrics().At(0).Resource().Attributes().Get("deployment.environment")
407+
v, exists = result.ResourceMetrics().At(0).Resource().Attributes().Get(string(semconv.DeploymentEnvironmentNameKey))
407408
require.True(t, exists)
408409
require.Equal(t, "tag1", v.AsString())
409-
v, exists = result.ResourceMetrics().At(0).Resource().Attributes().Get("service.version")
410+
v, exists = result.ResourceMetrics().At(0).Resource().Attributes().Get(string(semconv.ServiceVersionKey))
410411
require.True(t, exists)
411412
require.Equal(t, "tag2", v.AsString())
412413

‎receiver/datadogreceiver/internal/translator/tags.go

Lines changed: 43 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,22 @@ import (
88
"sync"
99

1010
"go.opentelemetry.io/collector/pdata/pcommon"
11-
semconv "go.opentelemetry.io/otel/semconv/v1.16.0"
11+
semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
1212
)
1313

1414
// See:
1515
// https://docs.datadoghq.com/opentelemetry/schema_semantics/semantic_mapping/
1616
// https://github.com/DataDog/opentelemetry-mapping-go/blob/main/pkg/otlp/attributes/attributes.go
1717
var datadogKnownResourceAttributes = map[string]string{
18-
"env": string(semconv.DeploymentEnvironmentKey),
18+
"env": string(semconv.DeploymentEnvironmentNameKey),
1919
"service": string(semconv.ServiceNameKey),
2020
"version": string(semconv.ServiceVersionKey),
2121

2222
// Container-related attributes
2323
"container_id": string(semconv.ContainerIDKey),
2424
"container_name": string(semconv.ContainerNameKey),
2525
"image_name": string(semconv.ContainerImageNameKey),
26-
"image_tag": string(semconv.ContainerImageTagKey),
26+
"image_tag": string(semconv.ContainerImageTagsKey),
2727
"runtime": string(semconv.ContainerRuntimeKey),
2828

2929
// Cloud-related attributes
@@ -50,6 +50,25 @@ var datadogKnownResourceAttributes = map[string]string{
5050
"kube_namespace": string(semconv.K8SNamespaceNameKey),
5151
"pod_name": string(semconv.K8SPodNameKey),
5252

53+
// HTTP
54+
"http.client_ip": string(semconv.ClientAddressKey),
55+
"http.response.content_length": string(semconv.HTTPResponseBodySizeKey),
56+
"http.status_code": string(semconv.HTTPResponseStatusCodeKey),
57+
"http.request.content_length": string(semconv.HTTPRequestBodySizeKey),
58+
"http.referer": "http.request.header.referer",
59+
"http.method": string(semconv.HTTPRequestMethodKey),
60+
"http.route": string(semconv.HTTPRouteKey),
61+
"http.version": string(semconv.NetworkProtocolVersionKey),
62+
"http.server_name": string(semconv.ServerAddressKey),
63+
"http.url": string(semconv.URLFullKey),
64+
"http.useragent": string(semconv.UserAgentOriginalKey),
65+
66+
// DB
67+
"db.type": string(semconv.DBSystemNameKey),
68+
"db.operation": string(semconv.DBOperationNameKey),
69+
"db.instance": string(semconv.DBCollectionNameKey),
70+
"db.pool.name": string(semconv.DBClientConnectionPoolNameKey),
71+
5372
// Other
5473
"process_id": string(semconv.ProcessPIDKey),
5574
"error.stacktrace": string(semconv.ExceptionStacktraceKey),
@@ -80,6 +99,15 @@ func translateDatadogKeyToOTel(k string) string {
8099
if otelKey, ok := datadogKnownResourceAttributes[strings.ToLower(k)]; ok {
81100
return otelKey
82101
}
102+
103+
// HTTP dynamic attributes
104+
if strings.HasPrefix(k, "http.response.headers.") { // type: string[]
105+
header := strings.TrimPrefix(k, "http.response.headers.")
106+
return "http.response.header." + header
107+
} else if strings.HasPrefix(k, "http.request.headers.") { // type: string[]
108+
header := strings.TrimPrefix(k, "http.request.headers.")
109+
return "http.request.header." + header
110+
}
83111
return k
84112
}
85113

@@ -136,12 +164,21 @@ func tagsToAttributes(tags []string, host string, stringPool *StringPool) attrib
136164
for _, tag := range tags {
137165
key, val = translateDatadogTagToKeyValuePair(tag)
138166
if attr, ok := datadogKnownResourceAttributes[key]; ok {
139-
val = stringPool.Intern(val) // No need to intern the key if we already have it
140-
attrs.resource.PutStr(attr, val)
167+
val = stringPool.Intern(val) // No need to intern the key if we already have it
168+
if attr == string(semconv.ContainerImageTagsKey) { // type: string[]
169+
attrs.resource.PutEmptySlice(attr).AppendEmpty().SetStr(val)
170+
} else {
171+
attrs.resource.PutStr(attr, val)
172+
}
141173
} else {
142174
key = stringPool.Intern(translateDatadogKeyToOTel(key))
143175
val = stringPool.Intern(val)
144-
attrs.dp.PutStr(key, val)
176+
if strings.HasPrefix(key, "http.request.header.") || strings.HasPrefix(key, "http.response.header.") {
177+
// type string[]
178+
attrs.resource.PutEmptySlice(key).AppendEmpty().SetStr(val)
179+
} else {
180+
attrs.dp.PutStr(key, val)
181+
}
145182
}
146183
}
147184

‎receiver/datadogreceiver/internal/translator/tags_test.go

Lines changed: 42 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import (
88

99
"github.com/stretchr/testify/assert"
1010
"go.opentelemetry.io/collector/pdata/pcommon"
11+
semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
1112
)
1213

1314
func TestGetMetricAttributes(t *testing.T) {
@@ -32,7 +33,7 @@ func TestGetMetricAttributes(t *testing.T) {
3233
tags: []string{},
3334
host: "host",
3435
expectedResourceAttrs: newMapFromKV(t, map[string]any{
35-
"host.name": "host",
36+
string(semconv.HostNameKey): "host",
3637
}),
3738
expectedScopeAttrs: pcommon.NewMap(),
3839
expectedDpAttrs: pcommon.NewMap(),
@@ -42,10 +43,10 @@ func TestGetMetricAttributes(t *testing.T) {
4243
tags: []string{"env:prod", "service:my-service", "version:1.0"},
4344
host: "host",
4445
expectedResourceAttrs: newMapFromKV(t, map[string]any{
45-
"host.name": "host",
46-
"deployment.environment": "prod",
47-
"service.name": "my-service",
48-
"service.version": "1.0",
46+
string(semconv.HostNameKey): "host",
47+
string(semconv.DeploymentEnvironmentNameKey): "prod",
48+
string(semconv.ServiceNameKey): "my-service",
49+
string(semconv.ServiceVersionKey): "1.0",
4950
}),
5051
expectedScopeAttrs: pcommon.NewMap(),
5152
expectedDpAttrs: pcommon.NewMap(),
@@ -55,8 +56,8 @@ func TestGetMetricAttributes(t *testing.T) {
5556
tags: []string{"env:prod", "foo"},
5657
host: "host",
5758
expectedResourceAttrs: newMapFromKV(t, map[string]any{
58-
"host.name": "host",
59-
"deployment.environment": "prod",
59+
string(semconv.HostNameKey): "host",
60+
string(semconv.DeploymentEnvironmentNameKey): "prod",
6061
}),
6162
expectedScopeAttrs: pcommon.NewMap(),
6263
expectedDpAttrs: newMapFromKV(t, map[string]any{
@@ -147,4 +148,38 @@ func TestTranslateDataDogKeyToOtel(t *testing.T) {
147148
assert.Equal(t, v, translateDatadogKeyToOTel(k))
148149
})
149150
}
151+
152+
// test dynamic attributes:
153+
// * http.request.header.<header_name>
154+
// * http.response.header.<header_name>
155+
assert.Equal(t, "http.request.header.referer", translateDatadogKeyToOTel("http.request.headers.referer"))
156+
assert.Equal(t, "http.response.header.content-type", translateDatadogKeyToOTel("http.response.headers.content-type"))
157+
}
158+
159+
func TestImageTags(t *testing.T) {
160+
// make sure container.image.tags is a string[]
161+
expected := "[\"tag1\"]"
162+
tags := []string{"env:prod", "foo", "image_tag:tag1"}
163+
host := "host"
164+
pool := newStringPool()
165+
166+
attrs := tagsToAttributes(tags, host, pool)
167+
imageTags, _ := attrs.resource.Get(string(semconv.ContainerImageTagsKey))
168+
assert.Equal(t, expected, imageTags.AsString())
169+
}
170+
171+
func TestHTTPHeaders(t *testing.T) {
172+
// make sure container.image.tags is a string[]
173+
expected := "[\"value\"]"
174+
tags := []string{"env:prod", "foo", "http.request.headers.header:value", "http.response.headers.header:value"}
175+
host := "host"
176+
pool := newStringPool()
177+
178+
attrs := tagsToAttributes(tags, host, pool)
179+
header, found := attrs.resource.Get("http.request.header.header")
180+
assert.True(t, found)
181+
assert.Equal(t, expected, header.AsString())
182+
header, found = attrs.resource.Get("http.response.header.header")
183+
assert.True(t, found)
184+
assert.Equal(t, expected, header.AsString())
150185
}

‎receiver/datadogreceiver/internal/translator/traces_translator.go

Lines changed: 82 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ package translator // import "github.com/open-telemetry/opentelemetry-collector-
55

66
import (
77
"bytes"
8+
"cmp"
89
"encoding/binary"
910
"encoding/json"
1011
"errors"
@@ -20,7 +21,7 @@ import (
2021
"github.com/hashicorp/golang-lru/v2/simplelru"
2122
"go.opentelemetry.io/collector/pdata/pcommon"
2223
"go.opentelemetry.io/collector/pdata/ptrace"
23-
semconv "go.opentelemetry.io/otel/semconv/v1.16.0"
24+
semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
2425
oteltrace "go.opentelemetry.io/otel/trace"
2526
"go.uber.org/zap"
2627
"google.golang.org/protobuf/proto"
@@ -44,6 +45,18 @@ const (
4445
attributeDatadogSpanID = "datadog.span.id"
4546
)
4647

48+
var spanProcessor = map[string]func(*pb.Span, *ptrace.Span){
49+
// HTTP
50+
"servlet.request": processHTTPSpan,
51+
52+
// Internal
53+
"spring.handler": processInternalSpan,
54+
55+
// Database
56+
"postgresql.query": processDBSpan,
57+
"redis.query": processDBSpan,
58+
}
59+
4760
func upsertHeadersAttributes(req *http.Request, attrs pcommon.Map) {
4861
if ddTracerVersion := req.Header.Get(header.TracerVersion); ddTracerVersion != "" {
4962
attrs.PutStr(string(semconv.TelemetrySDKVersionKey), "Datadog-"+ddTracerVersion)
@@ -88,21 +101,61 @@ func traceID64to128(span *pb.Span, traceIDCache *simplelru.LRU[uint64, pcommon.T
88101
return pcommon.TraceID{}, nil
89102
}
90103

104+
func processInternalSpan(span *pb.Span, newSpan *ptrace.Span) {
105+
newSpan.SetName(span.Resource)
106+
newSpan.SetKind(ptrace.SpanKindInternal)
107+
}
108+
109+
func processHTTPSpan(span *pb.Span, newSpan *ptrace.Span) {
110+
// https://opentelemetry.io/docs/specs/semconv/http/http-spans/#name
111+
// We assume that http.route coming from datadog is low cardinality
112+
if val, ok := span.Meta["http.method"]; ok {
113+
if suffix, ok := span.Meta["http.route"]; ok {
114+
newSpan.SetName(val + " " + suffix)
115+
} else {
116+
newSpan.SetName(val)
117+
}
118+
}
119+
}
120+
121+
func processDBSpan(span *pb.Span, newSpan *ptrace.Span) {
122+
// https://opentelemetry.io/docs/specs/semconv/database/database-spans/#name
123+
if val, ok := span.Meta["db.query.summary"]; ok {
124+
newSpan.SetName(val)
125+
} else {
126+
if val, ok = span.Meta["db.operation"]; ok {
127+
newSpan.SetName(val)
128+
suffix := cmp.Or(span.Meta["db.instance"], span.Meta["db.namespace"], span.Meta["peer.hostname"])
129+
if suffix != "" {
130+
newSpan.SetName(val + " " + suffix)
131+
}
132+
} else if val, ok = span.Meta["db.type"]; ok {
133+
newSpan.SetName(val)
134+
}
135+
}
136+
}
137+
138+
func processSpanByName(span *pb.Span, newSpan *ptrace.Span) {
139+
if processor, ok := spanProcessor[span.Name]; ok {
140+
processor(span, newSpan)
141+
}
142+
}
143+
91144
func ToTraces(logger *zap.Logger, payload *pb.TracerPayload, req *http.Request, traceIDCache *simplelru.LRU[uint64, pcommon.TraceID]) (ptrace.Traces, error) {
92145
var traces pb.Traces
93146
for _, p := range payload.GetChunks() {
94147
traces = append(traces, p.GetSpans())
95148
}
96149
sharedAttributes := pcommon.NewMap()
97150
for k, v := range map[string]string{
98-
string(semconv.ContainerIDKey): payload.ContainerID,
99-
string(semconv.TelemetrySDKLanguageKey): payload.LanguageName,
100-
string(semconv.ProcessRuntimeVersionKey): payload.LanguageVersion,
101-
string(semconv.DeploymentEnvironmentKey): payload.Env,
102-
string(semconv.HostNameKey): payload.Hostname,
103-
string(semconv.ServiceVersionKey): payload.AppVersion,
104-
string(semconv.TelemetrySDKNameKey): "Datadog",
105-
string(semconv.TelemetrySDKVersionKey): payload.TracerVersion,
151+
string(semconv.ContainerIDKey): payload.ContainerID,
152+
string(semconv.TelemetrySDKLanguageKey): payload.LanguageName,
153+
string(semconv.ProcessRuntimeVersionKey): payload.LanguageVersion,
154+
string(semconv.DeploymentEnvironmentNameKey): payload.Env,
155+
string(semconv.HostNameKey): payload.Hostname,
156+
string(semconv.ServiceVersionKey): payload.AppVersion,
157+
string(semconv.TelemetrySDKNameKey): "Datadog",
158+
string(semconv.TelemetrySDKVersionKey): payload.TracerVersion,
106159
} {
107160
if v != "" {
108161
sharedAttributes.PutStr(k, v)
@@ -125,6 +178,11 @@ func ToTraces(logger *zap.Logger, payload *pb.TracerPayload, req *http.Request,
125178

126179
for _, trace := range traces {
127180
for _, span := range trace {
181+
// Restore base service name as the service name.
182+
// Without this, internal spans such as postgresql queries have a service.name set to postgresql
183+
if val, ok := span.Meta["_dd.base_service"]; ok {
184+
span.Service = val
185+
}
128186
slice, exist := groupByService[span.Service]
129187
if !exist {
130188
slice = ptrace.NewSpanSlice()
@@ -193,6 +251,21 @@ func ToTraces(logger *zap.Logger, payload *pb.TracerPayload, req *http.Request,
193251
newSpan.SetKind(ptrace.SpanKindUnspecified)
194252
}
195253
}
254+
255+
// For client/producer/consumer spans, if we have `peer.hostname`, and `server.address` is unset, set
256+
// `server.address` to `peer.hostname`.
257+
if newSpan.Kind() == ptrace.SpanKindClient ||
258+
newSpan.Kind() == ptrace.SpanKindProducer ||
259+
newSpan.Kind() == ptrace.SpanKindConsumer {
260+
if _, ok := newSpan.Attributes().Get("server.address"); !ok {
261+
if val, ok := span.Meta["peer.hostname"]; ok {
262+
newSpan.Attributes().PutStr("server.address", val)
263+
}
264+
}
265+
}
266+
267+
// Some spans need specific processing (http, db, ...)
268+
processSpanByName(span, &newSpan)
196269
}
197270
}
198271

0 commit comments

Comments
 (0)