Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify context path of OTEL transformers #4134

Merged
merged 4 commits into from
Mar 20, 2025
Merged

Modify context path of OTEL transformers #4134

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1763044
Modify context path of OTEL transformers
benjaminjb Mar 17, 2025
3fab41d
Update for pgbouncer
benjaminjb Mar 20, 2025
4f80c64
remove context: log
benjaminjb Mar 20, 2025
af7bb9c
add info around pgbouncer columns avoided
benjaminjb Mar 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion internal/collector/generated/pgbackrest_logs_transforms.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion internal/collector/generated/postgres_logs_transforms.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 10 additions & 10 deletions internal/collector/patroni.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,23 +69,23 @@ func EnablePatroniLogging(ctx context.Context,
`set(instrumentation_scope.name, "patroni")`,

// https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/-/pkg/ottl/ottlfuncs#parsejson
`set(cache, ParseJSON(body["original"]))`,
`set(log.cache, ParseJSON(log.body["original"]))`,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 Do we still have to set context=log? If so, what a strange change they made.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not. I'm not sure of the benefit of this change.

That said, is it expected for almost every emission to be LogRecord #0?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like an index, maybe? The batch time is quite small by default, so one-at-a-time sounds plausible.


// The log severity is in the "levelname" field.
// https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitytext
`set(severity_text, cache["levelname"])`,
`set(log.severity_text, log.cache["levelname"])`,

// Map Patroni (python) "logging levels" to OpenTelemetry severity levels.
//
// https://docs.python.org/3.6/library/logging.html#logging-levels
// https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitynumber
// https://github.com/open-telemetry/opentelemetry-python/blob/v1.29.0/opentelemetry-api/src/opentelemetry/_logs/severity/__init__.py
// https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/-/pkg/ottl/contexts/ottllog#enums
`set(severity_number, SEVERITY_NUMBER_DEBUG) where severity_text == "DEBUG"`,
`set(severity_number, SEVERITY_NUMBER_INFO) where severity_text == "INFO"`,
`set(severity_number, SEVERITY_NUMBER_WARN) where severity_text == "WARNING"`,
`set(severity_number, SEVERITY_NUMBER_ERROR) where severity_text == "ERROR"`,
`set(severity_number, SEVERITY_NUMBER_FATAL) where severity_text == "CRITICAL"`,
`set(log.severity_number, SEVERITY_NUMBER_DEBUG) where log.severity_text == "DEBUG"`,
`set(log.severity_number, SEVERITY_NUMBER_INFO) where log.severity_text == "INFO"`,
`set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_text == "WARNING"`,
`set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.severity_text == "ERROR"`,
`set(log.severity_number, SEVERITY_NUMBER_FATAL) where log.severity_text == "CRITICAL"`,

// Parse the "asctime" field into the record timestamp.
// The format is neither RFC 3339 nor ISO 8601:
Expand All @@ -95,14 +95,14 @@ func EnablePatroniLogging(ctx context.Context,
//
// https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/-/pkg/stanza/docs/types/timestamp.md
// https://docs.python.org/3.6/library/logging.html#logging.LogRecord
`set(time, Time(cache["asctime"], "%F %T,%L"))`,
`set(log.time, Time(log.cache["asctime"], "%F %T,%L")) where IsString(log.cache["asctime"])`,

// Keep the unparsed log record in a standard attribute, and replace
// the log record body with the message field.
//
// https://github.com/open-telemetry/semantic-conventions/blob/v1.29.0/docs/general/logs.md
`set(attributes["log.record.original"], body["original"])`,
`set(body, cache["message"])`,
`set(log.attributes["log.record.original"], log.body["original"])`,
`set(log.body, log.cache["message"])`,
},
}},
}
Expand Down
50 changes: 30 additions & 20 deletions internal/collector/patroni_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,16 +74,21 @@ processors:
- context: log
statements:
- set(instrumentation_scope.name, "patroni")
- set(cache, ParseJSON(body["original"]))
- set(severity_text, cache["levelname"])
- set(severity_number, SEVERITY_NUMBER_DEBUG) where severity_text == "DEBUG"
- set(severity_number, SEVERITY_NUMBER_INFO) where severity_text == "INFO"
- set(severity_number, SEVERITY_NUMBER_WARN) where severity_text == "WARNING"
- set(severity_number, SEVERITY_NUMBER_ERROR) where severity_text == "ERROR"
- set(severity_number, SEVERITY_NUMBER_FATAL) where severity_text == "CRITICAL"
- set(time, Time(cache["asctime"], "%F %T,%L"))
- set(attributes["log.record.original"], body["original"])
- set(body, cache["message"])
- set(log.cache, ParseJSON(log.body["original"]))
- set(log.severity_text, log.cache["levelname"])
- set(log.severity_number, SEVERITY_NUMBER_DEBUG) where log.severity_text ==
"DEBUG"
- set(log.severity_number, SEVERITY_NUMBER_INFO) where log.severity_text ==
"INFO"
- set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_text ==
"WARNING"
- set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.severity_text ==
"ERROR"
- set(log.severity_number, SEVERITY_NUMBER_FATAL) where log.severity_text ==
"CRITICAL"
- set(log.time, Time(log.cache["asctime"], "%F %T,%L")) where IsString(log.cache["asctime"])
- set(log.attributes["log.record.original"], log.body["original"])
- set(log.body, log.cache["message"])
receivers:
filelog/patroni_jsonlog:
include:
Expand Down Expand Up @@ -169,16 +174,21 @@ processors:
- context: log
statements:
- set(instrumentation_scope.name, "patroni")
- set(cache, ParseJSON(body["original"]))
- set(severity_text, cache["levelname"])
- set(severity_number, SEVERITY_NUMBER_DEBUG) where severity_text == "DEBUG"
- set(severity_number, SEVERITY_NUMBER_INFO) where severity_text == "INFO"
- set(severity_number, SEVERITY_NUMBER_WARN) where severity_text == "WARNING"
- set(severity_number, SEVERITY_NUMBER_ERROR) where severity_text == "ERROR"
- set(severity_number, SEVERITY_NUMBER_FATAL) where severity_text == "CRITICAL"
- set(time, Time(cache["asctime"], "%F %T,%L"))
- set(attributes["log.record.original"], body["original"])
- set(body, cache["message"])
- set(log.cache, ParseJSON(log.body["original"]))
- set(log.severity_text, log.cache["levelname"])
- set(log.severity_number, SEVERITY_NUMBER_DEBUG) where log.severity_text ==
"DEBUG"
- set(log.severity_number, SEVERITY_NUMBER_INFO) where log.severity_text ==
"INFO"
- set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_text ==
"WARNING"
- set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.severity_text ==
"ERROR"
- set(log.severity_number, SEVERITY_NUMBER_FATAL) where log.severity_text ==
"CRITICAL"
- set(log.time, Time(log.cache["asctime"], "%F %T,%L")) where IsString(log.cache["asctime"])
- set(log.attributes["log.record.original"], log.body["original"])
- set(log.body, log.cache["message"])
receivers:
filelog/patroni_jsonlog:
include:
Expand Down
Loading