Skip to content

Commit 6395ff8

Browse files
ChrsMarkChrsMark
authored
[pkg/stanza] container: move k8s metadata into resource attributes (#33353)
**Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Container parser should add k8s metadata as resource attributes and not as log record attributes. **Link to tracking Issue:** <Issue number if applicable> Fixes #33341 **Testing:** <Describe what testing was performed and which tests were added.> Manual testing on local k8s cluster: ```console 2024-06-04T06:40:08.219Z info ResourceLog #0 Resource SchemaURL: Resource attributes: -> k8s.pod.uid: Str(d5ecc924-e255-4525-b5be-6437939b1e4d) -> k8s.container.name: Str(busybox) -> k8s.namespace.name: Str(default) -> k8s.pod.name: Str(daemonset-logs-dhzcq) -> k8s.container.restart_count: Str(0) ScopeLogs #0 ScopeLogs SchemaURL: InstrumentationScope LogRecord #0 ObservedTimestamp: 2024-06-04 06:40:08.007370503 +0000 UTC Timestamp: 2024-06-04 06:40:07.855932421 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str(otel logs at 06:40:07) Attributes: -> logtag: Str(F) -> key2: Map({"key_in":"val2"}) -> log.file.path: Str(/var/log/pods/default_daemonset-logs-dhzcq_d5ecc924-e255-4525-b5be-6437939b1e4d/busybox/0.log) -> time: Str(2024-06-04T06:40:07.855932421Z) -> log.iostream: Str(stdout) Trace ID: Span ID: Flags: 0 LogRecord #1 ObservedTimestamp: 2024-06-04 06:40:08.007451031 +0000 UTC Timestamp: 2024-06-04 06:40:07.957875321 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str(otel logs at 06:40:07) Attributes: -> log.file.path: Str(/var/log/pods/default_daemonset-logs-dhzcq_d5ecc924-e255-4525-b5be-6437939b1e4d/busybox/0.log) -> log.iostream: Str(stdout) -> time: Str(2024-06-04T06:40:07.957875321Z) -> key2: Map({"key_in":"val2"}) -> logtag: Str(F) Trace ID: Span ID: Flags: 0 ``` **Documentation:** <Describe the documentation added.> ~ --------- Signed-off-by: ChrsMark <[email protected]>
1 parent 982c728 commit 6395ff8

File tree

4 files changed

+104
-54
lines changed

4 files changed

+104
-54
lines changed

‎.chloggen/container_parser_resource_attributes.yaml

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: bug_fix
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
7+
component: filelogreceiver
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: "Container parser should add k8s metadata as resource attributes and not as log record attributes"
11+
12+
13+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
14+
issues: [33341]
15+
16+
# (Optional) One or more lines of additional information to render under the primary note.
17+
# These lines will be padded with 2 spaces and then inserted directly into the document.
18+
# Use pipe (|) for multiline entries.
19+
subtext:
20+
21+
# If your change doesn't affect end users or the exported elements of any package,
22+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
23+
# Optional: The change log or logs in which this entry should be included.
24+
# e.g. '[user]' or '[user, api]'
25+
# Include 'user' if the change is relevant to end users.
26+
# Include 'api' if there is a change to a library API.
27+
# Default: '[user]'
28+
change_logs: []

‎pkg/stanza/docs/operators/container.md

+43-31
Original file line numberDiff line numberDiff line change
@@ -31,17 +31,13 @@ will produce the following k8s metadata:
3131

3232
```json
3333
{
34-
"attributes": {
35-
"k8s": {
36-
"container": {
37-
"name": "kube-controller",
38-
"restart_count": "1"
39-
}, "pod": {
40-
"uid": "49cc7c1fd3702c40b2686ea7486091d6",
41-
"name": "kube-controller-kind-control-plane"
42-
}, "namespace": {
43-
"name": "some-ns"
44-
}
34+
"resource": {
35+
"attributes": {
36+
"k8s.pod.name": "kube-controller-kind-control-plane",
37+
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
38+
"k8s.container.name": "kube-controller",
39+
"k8s.container.restart_count": "1",
40+
"k8s.namespace.name": "some"
4541
}
4642
}
4743
}
@@ -84,12 +80,16 @@ Note: in this example the `format: docker` is optional since formats can be auto
8480
"attributes": {
8581
"time": "2024-03-30T08:31:20.545192187Z",
8682
"log.iostream": "stdout",
87-
"k8s.pod.name": "kube-controller-kind-control-plane",
88-
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
89-
"k8s.container.name": "kube-controller",
90-
"k8s.container.restart_count": "1",
91-
"k8s.namespace.name": "some",
9283
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
84+
},
85+
"resource": {
86+
"attributes": {
87+
"k8s.pod.name": "kube-controller-kind-control-plane",
88+
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
89+
"k8s.container.name": "kube-controller",
90+
"k8s.container.restart_count": "1",
91+
"k8s.namespace.name": "some"
92+
}
9393
}
9494
}
9595
```
@@ -129,12 +129,16 @@ Configuration:
129129
"time": "2024-04-13T07:59:37.505201169-05:00",
130130
"logtag": "F",
131131
"log.iostream": "stdout",
132-
"k8s.pod.name": "kube-controller-kind-control-plane",
133-
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
134-
"k8s.container.name": "kube-controller",
135-
"k8s.container.restart_count": "1",
136-
"k8s.namespace.name": "some",
137132
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
133+
},
134+
"resource": {
135+
"attributes": {
136+
"k8s.pod.name": "kube-controller-kind-control-plane",
137+
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
138+
"k8s.container.name": "kube-controller",
139+
"k8s.container.restart_count": "1",
140+
"k8s.namespace.name": "some"
141+
}
138142
}
139143
}
140144
```
@@ -174,12 +178,16 @@ Configuration:
174178
"time": "2023-06-22T10:27:25.813799277Z",
175179
"logtag": "F",
176180
"log.iostream": "stdout",
177-
"k8s.pod.name": "kube-controller-kind-control-plane",
178-
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
179-
"k8s.container.name": "kube-controller",
180-
"k8s.container.restart_count": "1",
181-
"k8s.namespace.name": "some",
182181
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
182+
},
183+
"resource": {
184+
"attributes": {
185+
"k8s.pod.name": "kube-controller-kind-control-plane",
186+
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
187+
"k8s.container.name": "kube-controller",
188+
"k8s.container.restart_count": "1",
189+
"k8s.namespace.name": "some"
190+
}
183191
}
184192
}
185193
```
@@ -227,12 +235,16 @@ Configuration:
227235
"time": "2023-06-22T10:27:25.813799277Z",
228236
"logtag": "F",
229237
"log.iostream": "stdout",
230-
"k8s.pod.name": "kube-controller-kind-control-plane",
231-
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
232-
"k8s.container.name": "kube-controller",
233-
"k8s.container.restart_count": "1",
234-
"k8s.namespace.name": "some",
235238
"log.file.path": "/var/log/pods/some_kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log"
239+
},
240+
"resource": {
241+
"attributes": {
242+
"k8s.pod.name": "kube-controller-kind-control-plane",
243+
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d6",
244+
"k8s.container.name": "kube-controller",
245+
"k8s.container.restart_count": "1",
246+
"k8s.namespace.name": "some"
247+
}
236248
}
237249
}
238250
```

‎pkg/stanza/operator/parser/container/parser.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -300,7 +300,7 @@ func (p *Parser) extractk8sMetaFromFilePath(e *entry.Entry) error {
300300
}
301301

302302
for originalKey, attributeKey := range k8sMetadataMapping {
303-
newField := entry.NewAttributeField(attributeKey)
303+
newField := entry.NewResourceField(attributeKey)
304304
if err := newField.Set(e, parsedValues[originalKey]); err != nil {
305305
return fmt.Errorf("failed to set %v as metadata at %v", originalKey, attributeKey)
306306
}

‎pkg/stanza/operator/parser/container/parser_test.go

+32-22
Original file line numberDiff line numberDiff line change
@@ -157,16 +157,18 @@ func TestProcess(t *testing.T) {
157157
},
158158
&entry.Entry{
159159
Attributes: map[string]any{
160-
"time": "2029-03-30T08:31:20.545192187Z",
161-
"log.iostream": "stdout",
160+
"time": "2029-03-30T08:31:20.545192187Z",
161+
"log.iostream": "stdout",
162+
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
163+
},
164+
Body: "INFO: log line here",
165+
Resource: map[string]any{
162166
"k8s.pod.name": "kube-scheduler-kind-control-plane",
163167
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
164168
"k8s.container.name": "kube-scheduler44",
165169
"k8s.container.restart_count": "1",
166170
"k8s.namespace.name": "some",
167-
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
168171
},
169-
Body: "INFO: log line here",
170172
Timestamp: time.Date(2029, time.March, 30, 8, 31, 20, 545192187, time.UTC),
171173
},
172174
},
@@ -212,17 +214,19 @@ func TestRecombineProcess(t *testing.T) {
212214
[]*entry.Entry{
213215
{
214216
Attributes: map[string]any{
215-
"time": "2024-04-13T07:59:37.505201169-10:00",
216-
"log.iostream": "stdout",
217-
"logtag": "F",
217+
"time": "2024-04-13T07:59:37.505201169-10:00",
218+
"log.iostream": "stdout",
219+
"logtag": "F",
220+
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
221+
},
222+
Body: "standalone crio line which is awesome!",
223+
Resource: map[string]any{
218224
"k8s.pod.name": "kube-scheduler-kind-control-plane",
219225
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
220226
"k8s.container.name": "kube-scheduler44",
221227
"k8s.container.restart_count": "1",
222228
"k8s.namespace.name": "some",
223-
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
224229
},
225-
Body: "standalone crio line which is awesome!",
226230
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.FixedZone("", -10*60*60)),
227231
},
228232
},
@@ -246,15 +250,17 @@ func TestRecombineProcess(t *testing.T) {
246250
[]*entry.Entry{
247251
{
248252
Attributes: map[string]any{
249-
"time": "2024-04-13T07:59:37.505201169Z",
250-
"log.iostream": "stdout",
251-
"logtag": "F",
253+
"time": "2024-04-13T07:59:37.505201169Z",
254+
"log.iostream": "stdout",
255+
"logtag": "F",
256+
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
257+
},
258+
Resource: map[string]any{
252259
"k8s.pod.name": "kube-scheduler-kind-control-plane",
253260
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
254261
"k8s.container.name": "kube-scheduler44",
255262
"k8s.container.restart_count": "1",
256263
"k8s.namespace.name": "some",
257-
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
258264
},
259265
Body: "standalone containerd line which is awesome!",
260266
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.UTC),
@@ -286,15 +292,17 @@ func TestRecombineProcess(t *testing.T) {
286292
[]*entry.Entry{
287293
{
288294
Attributes: map[string]any{
289-
"time": "2024-04-13T07:59:37.505201169-10:00",
290-
"log.iostream": "stdout",
291-
"logtag": "P",
295+
"time": "2024-04-13T07:59:37.505201169-10:00",
296+
"log.iostream": "stdout",
297+
"logtag": "P",
298+
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
299+
},
300+
Resource: map[string]any{
292301
"k8s.pod.name": "kube-scheduler-kind-control-plane",
293302
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
294303
"k8s.container.name": "kube-scheduler44",
295304
"k8s.container.restart_count": "1",
296305
"k8s.namespace.name": "some",
297-
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
298306
},
299307
Body: "standalone crio line which is awesome!",
300308
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.FixedZone("", -10*60*60)),
@@ -326,17 +334,19 @@ func TestRecombineProcess(t *testing.T) {
326334
[]*entry.Entry{
327335
{
328336
Attributes: map[string]any{
329-
"time": "2024-04-13T07:59:37.505201169Z",
330-
"log.iostream": "stdout",
331-
"logtag": "P",
337+
"time": "2024-04-13T07:59:37.505201169Z",
338+
"log.iostream": "stdout",
339+
"logtag": "P",
340+
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
341+
},
342+
Body: "standalone containerd line which is awesome!",
343+
Resource: map[string]any{
332344
"k8s.pod.name": "kube-scheduler-kind-control-plane",
333345
"k8s.pod.uid": "49cc7c1fd3702c40b2686ea7486091d3",
334346
"k8s.container.name": "kube-scheduler44",
335347
"k8s.container.restart_count": "1",
336348
"k8s.namespace.name": "some",
337-
"log.file.path": "/var/log/pods/some_kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log",
338349
},
339-
Body: "standalone containerd line which is awesome!",
340350
Timestamp: time.Date(2024, time.April, 13, 7, 59, 37, 505201169, time.UTC),
341351
},
342352
},

0 commit comments

Comments
 (0)