Skip to content

Commit 96d9b71

Browse files
alexvanboxelalexvanboxel
committed
Clarify the mapping between Google Cloud Logging and OpenTelemetry
* Clarify that the JSON representation is used for the LogEntry * Change to Pascal-case for representing the field in LogEntry * Map MonitoredResource to resource attributes with gcp prefix * Clarify the type of the Body for jsonPayload, protoPayload, and textPayload * Mapping section for Severity from Cloud Logging to OpenTelemetry * Mapping section for Severity from OpenTelemetry to Cloud Logging * Semantic Mapping section (only includes log.record.uid) * Clarified that the JSON representation is used for the LogEntry * Map LogEntryOperation to resource attribute with gcp.operation (KVList) * Map LogEntrySourceLocation to resource attribute with gcp.source_location (KVList) * Map LogSplit to resource attribute with gcp.log_split (KVList) With this [draft collector PR](open-telemetry/opentelemetry-collector-contrib#29299), the PubsubReceiver will be able to handle the native translation of Google Cloud LogEntry's. But translations are opinionated. I prefer that we agree on the target mapping before merging the PR.
1 parent cf79597 commit 96d9b71

File tree

1 file changed

+66
-15
lines changed

1 file changed

+66
-15
lines changed

‎specification/logs/data-model-appendix.md

+66-15
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ the respective exporter documentation if exact details are required.
1717
* [Apache HTTP Server access log](#apache-http-server-access-log)
1818
* [CloudTrail Log Event](#cloudtrail-log-event)
1919
* [Google Cloud Logging](#google-cloud-logging)
20+
+ [Semantic Mapping](#semantic-mapping)
21+
+ [Severity Mapping](#severity-mapping)
22+
+ [Miscellaneous](#miscellaneous)
2023
* [Elastic Common Schema](#elastic-common-schema)
2124
- [Appendix B: `SeverityNumber` example mappings](#appendix-b-severitynumber-example-mappings)
2225

@@ -482,21 +485,69 @@ When mapping from the unified model to HEC, we apply this additional mapping:
482485

483486
### Google Cloud Logging
484487

485-
Field | Type | Description | Maps to Unified Model Field
486-
-----------------|--------------------| ------------------------------------------------------- | ---------------------------
487-
timestamp | string | The time the event described by the log entry occurred. | Timestamp
488-
resource | MonitoredResource | The monitored resource that produced this log entry. | Resource
489-
log_name | string | The URL-encoded LOG_ID suffix of the log_name field identifies which log stream this entry belongs to. | Attributes["gcp.log_name"]
490-
json_payload | google.protobuf.Struct | The log entry payload, represented as a structure that is expressed as a JSON object. | Body
491-
proto_payload | google.protobuf.Any | The log entry payload, represented as a protocol buffer. | Body
492-
text_payload | string | The log entry payload, represented as a Unicode string (UTF-8). | Body
493-
severity | LogSeverity | The severity of the log entry. | Severity
494-
trace | string | The trace associated with the log entry, if any. | TraceId
495-
span_id | string | The span ID within the trace associated with the log entry. | SpanId
496-
labels | map<string,string> | A set of user-defined (key, value) data that provides additional information about the log entry. | Attributes
497-
http_request | HttpRequest | The HTTP request associated with the log entry, if any. | Attributes["gcp.http_request"]
498-
trace_sampled | boolean | The sampling decision of the trace associated with the log entry. | TraceFlags.SAMPLED
499-
All other fields | | | Attributes["gcp.*"]
488+
Google Cloud logging uses the [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) to
489+
cary log information. In this section the JSON representation is used for mapping of the fields to OpenTelemetry fields
490+
and attributes.
491+
492+
#### Semantic Mapping
493+
494+
Some of the attributes can be moved to OpenTelemetry Semantic Conventions.
495+
496+
| Field | Type | Description | Maps to Unified Model Field |
497+
|-------------|-----------|----------------------------------------|--------------------------------|
498+
| `insert_id` | `boolean` | A unique identifier for the log entry. | `Attributes["log.record.uid"]` |
499+
500+
#### Severity Mapping
501+
502+
Mapping from [Google Cloud Log Severity](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity)
503+
to the OpenTelemetry Severity Number
504+
505+
| CloudLog | Severity Number | CloudLog Description |
506+
|------------------|------------------|----------------------------------------------------------------------------------------|
507+
| `DEFAULT`(0) | `UNSPECIFIED`(0) | The log entry has no assigned severity level. |
508+
| `DEBUG`(100) | `DEBUG`(5) | Debug or trace information. |
509+
| `INFO`(200) | `INFO`(9) | Routine information, such as ongoing status or performance. |
510+
| `NOTICE`(300) | `INFO2`(10) | Normal but significant events, such as start up, shut down, or a configuration change. |
511+
| `WARNING`(400) | `WARN`(13) | Warning events might cause problems. |
512+
| `ERROR`(500) | `ERROR`(17) | Error events are likely to cause problems. |
513+
| `CRITICAL`(600) | `FATAL`(21) | Critical events cause more severe problems or outages. |
514+
| `ALERT`(700) | `FATAL2`(22) | A person must take an action immediately. |
515+
| `EMERGENCY`(800) | `FATAL4`(24) | One or more systems are unusable. |
516+
517+
Mapping from OpenTelemetry Severity Number to a
518+
[Google Cloud Log Severity](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity)
519+
520+
| Severity Number | CloudLog |
521+
|-----------------------------|------------------|
522+
| `UNSPECIFIED`(0) | `DEFAULT`(0) |
523+
| `TRACE`(1) - `DEBUG4`(8) | `DEBUG`(100) |
524+
| `INFO`(9) | `INFO`(200) |
525+
| `INFO2`(10) - `INFO4`(12) | `NOTICE`(300) |
526+
| `WARN`(13) - `WARN4`(16) | `WARNING`(400) |
527+
| `ERROR`(17) - `ERROR4`(20 | `ERROR`(500) |
528+
| `FATAL`(21) | `CRITICAL`(600) |
529+
| `FATAL2`(22) - `FATAL3`(23) | `ALERT`(700) |
530+
| `FATAL4`(24) | `EMERGENCY`(800) |
531+
532+
#### Miscellaneous
533+
534+
| Field | Type | Description | Maps to Unified Model Field |
535+
|--------------------|--------------------------|------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
536+
| `timestamp` | `string` | The time the event described by the log entry occurred. | `Timestamp` |
537+
| `receiveTimestamp` | `string` | The time the log entry was received. | `ObservedTimestamp` |
538+
| `logName` | `string` | The URL-encoded log ID suffix of the log_name field identifies which log stream this entry belongs to. | `Attributes["gcp.log_name"]` (string) |
539+
| `jsonPayload` | `google.protobuf.Struct` | The log entry payload, represented as a structure that is expressed as a JSON object. | `Body` (KVList) |
540+
| `protoPayload` | `google.protobuf.Any` | The log entry payload, represented as a protocol buffer. | `Body` (KVList, key from JSON representation) |
541+
| `textPayload` | `string` | The log entry payload, represented as a Unicode string (UTF-8). | `Body` (string) |
542+
| `trace` | `string` | The trace associated with the log entry, if any. | `TraceId` |
543+
| `spanId` | `string` | The span ID within the trace associated with the log entry. | `SpanId` |
544+
| `traceSampled` | `boolean` | The sampling decision of the trace associated with the log entry. | `TraceFlags.SAMPLED` |
545+
| `labels` | `map<string,string>` | A set of user-defined (key, value) data that provides additional information about the log entry. | `Attributes` |
546+
| `resource` | `MonitoredResource` | The monitored resource that produced this log entry. | `Resource["gcp.*"]` |
547+
| `httpRequest` | `HttpRequest` | The HTTP request associated with the log entry, if any. | `Attributes["gcp.http_request"]` (KVList) |
548+
| `operation` | `LogEntryOperation` | Information about a operation associated with the log entry. | `Attributes["gcp.operation"]` (KVList) |
549+
| `sourceLocation` | `LogEntrySourceLocation` | Source code location information associated with the log entry. | `Attributes["gcp.source_location"]` (KVList) |
550+
| `split` | `LogSplit` | Information indicating this LogEntry is part of a sequence of multiple log entries split from a single LogEntry. | `Attributes["gcp.log_split"]` (KVList) |
500551

501552
### Elastic Common Schema
502553

0 commit comments

Comments
 (0)