[exporter/loki] Document the migration from the Loki Exporter to the Loki V3 OTLP endpoint

exporter/lokiexporter/README.md

@@ -17,7 +17,201 @@ Exports data via HTTP to [Loki](https://grafana.com/docs/loki/latest/).
 
 This component is **deprecated**: Loki now supports native [OTLP ingestion](https://grafana.com/docs/loki/latest/send-data/otel/) starting from v3. Grafana Cloud also supports [OTLP native ingestion](https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp) for logs. This component will be removed in September 2024.
 
-## Getting Started
+
+### Benefits of the new Loki OpenTelemetry log format
+
+The new format for OpenTelemetry logs introduced in Loki V3 brings the following benefits:
+
+* Native support for the structure of OpenTelemetry logs enabling simpler querying (no more JSON parsing),
+* Simplified client configuration to send OpenTelemetry data using the standard OTLP protocol.
+
+### Loki log message format changes for OpenTelemetry logs
+
+See OpenTelemetry Logs Data Model specification [here](https://opentelemetry.io/docs/specs/otel/logs/data-model/).
+
+| OpenTelemetry log field | Pre Loki V3 | Loki V3 through the Loki OTLP Endpoint |
+| ----- | ----- | ----- |
+| [`Timestamp`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-timestamp) | `timestamp` | `timestamp` |
+| [`ObservedTimestamp`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-observedtimestamp) | Not available | `metadata[observed_timestamp]` |
+| [`TraceId`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-traceid) | `traceid`  field of the Loki JSON log message | `metadata[trace_id]` |
+| [`SpanId`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-spanid) | `spanid` field of the Loki JSON log message | `metadata[span_id]` |
+| [`TraceFlags`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-traceflags) | Not available | `metadata[flags]` |
+| [`SeverityText`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitytext) |  `severity` field of the JSON log message (e.g. `Information`) and `level` label (e.g. `ERROR`, `INFO`...), the `detected_level` label is also available | `metadata[severity_text]`,  the `detected_level` label is also available |
+| [`SeverityNumber`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitynumber) | Not available | `metadata[severity_number]` |
+| [`Body`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-body) | `body`  field of the Loki JSON log message | The Loki log message. `__line__`in LogQL functions (e.g. `line_format`)|
+| [`InstrumentationScope`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-instrumentationscope) | `instrumentation_scope_name` field of the JSON log message | `metadata[scope_name]` |
+| [`Attributes`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-attributes) | JSON fields of the Loki log message | `metadata[xyz]` Where `xyz` is the `_` version of the OTel attribute name (e.g. `thread_name` Loki metadata for the `thread.name` OpenTelemetry attribute)|
+| [`Resource`](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-resource) | `service.name`, `service.namespace`, and `service.instance.id` are promoted as the following labels: `job=[${service.namespace}/]${service.name}`, instance=${service.instance.id}, exporter="OTLP"`.  Other resource attributes are stored as JSON fields of the Loki log message with the prefix `resources_` (e.g. `resources_k8s_namespace_name`) | Default list of resource attributes promoted as Loki labels: `cloud.availability_zone`, `cloud.region`, `container.name`, `deployment.environment`, `k8s.cluster.name`, `k8s.container.name`, `k8s.cronjob.name`, `k8s.daemonset.name`, `k8s.deployment.name`, `k8s.job.name`, `k8s.namespace.name`, `k8s.pod.name`, `k8s.replicaset.name` `k8s.statefulset.name`, `service.instance.id`, `service.name`, `service.namespace`. <br/>Other resource attributes are by default promoted as Loki message metadata.<br/> ℹ️ The list of promoted resource attributes is configurable using Loki’s distributor config parameter `default_resource_attributes_as_index_labels` when using self managed Loki ([here](https://grafana.com/docs/loki/latest/configure/\#distributor)) or opening a support request when using Grafana Cloud |
+
+ℹ️ Additional conversion rules from OpenTelemetry Logs to Loki 
+
+* `.` in attributes and resource attributes are converted into `_` when they are mapped as Loki labels or metadata,  
+* Otel attribute values with complex data types (i.e. arrays, nested structures) are converted into JSON strings
+
+### Migration instructions
+
+### Instrumentation migration: No changes to the instrumentation layer
+
+No changes are needed in the instrumentation layer. OpenTelemetry logs sources like OpenTelemetry SDKs or the OpenTelemetry Collector File Log Receiver don’t have to be modified.
+
+### Logs collection migration: Replace the OpenTelemetry Collector Loki Exporter by the OTLP/HTTP Exporter
+
+OpenTelemetry logs should no longer be exporter to Loki using the OpenTelemetry Collector Loki Exporter and should be now sent to the Loki OTLP endpoint (or the Grafana Cloud OTLP endpoint when on Grafana Cloud) using the [OpenTelemetry Collector OTLP HTTP Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter).
+
+
+OpenTelemetry Collector configuration before migration
+
+```yaml
+========================
+=   BEFORE MIGRATION   =
+========================
+
+extensions:
+  basicauth/loki:
+    client_auth:
+      username: <<username>>
+      password: <<password>>
+
+exporters:
+  loki:
+    auth:
+      authenticator: basicauth/loki
+    endpoint: https://loki.example.com:3100/loki/api/v1/push
+
+service:
+  extensions: [basicauth/loki]
+  pipelines:
+    logs:
+      receivers: [...]
+      processors: [...]
+      exporters: [..., loki]
+
+
+========================
+=   AFTER MIGRATION    =
+========================
+
+extensions:
+  basicauth/loki:
+    client_auth:
+      username: <<username>>
+      password: <<password>>
+
+exporters:
+  otlphttp/loki:
+    auth:
+      authenticator: basicauth/loki
+    endpoint: http://<loki-addr>:3100/otlp/v1/logs
+
+service:
+  extensions: [basicauth/loki]
+  pipelines:
+    logs:
+      receivers: [...]
+      processors: [...]
+      exporters: [..., otlphttp/loki]
+```
+
+* The OTLP HTTP endpoint URL and credentials details for Grafana Cloud users are available using the Grafana Cloud "OpenTelemetry Collector" connection tile.
+* The promotion of OpenTelemetry attributes and resource attributes to Loki labels using the `loki.attribute.labels` and `loki.resource.labels` hints is replaced by the list of promoted attributes managed centrally in Loki. 
+* The default list of resource attributes promoted as labels (see above) should be sufficient for most use cases.  
+* ℹ️ Changes can be made to this list using the Loki distributor configuration parameter `default_resource_attributes_as_index_labels` ([here](https://grafana.com/docs/loki/latest/configure/\#distributor)) for self managed instances and opening a support ticket for Grafana Cloud.
+
+### LogQL queries migration
+
+#### From `job` and `instance` to `service_name`, `service_namespace`, and `service_instance_id`
+
+The Loki labels `job` and `instance` are no longer generated and are replaced by the `service_name`, `service_namespace`, and `service_instance_id` labels.
+
+Example:
+
+```
+BEFORE
+{job="ecommerce/frontend", instance="instance-1234567890"}
+
+AFTER
+{service_name="frontend", service_namespace="ecommerce", service_instance_id="instance-1234567890"}
+```
+
+#### From `| json | an_attribute=...` to `{an_attribute=...}` or `| an_attribute=...`
+
+OTel log attributes, resource attributes, and fields are no longer stored in the JSON message but are stored as 
+* Loki message labels for promoted resource attributes (see list above),
+* Loki message metadata for other resource attributes, log attributes and log fields.
+
+LogQL statements `| json | an_attribute=...` must be converted to:
+
+* Promoted resource attributes: `{an_attribute=...}`
+* For other resource attributes, log attributes, and log fields: `| an_attribute=...`
+
+#### From `| json | traceid=...` and `| json | spanid=...` to `| trace_id=...` and `| span_id=...`
+
+The log fields `SpanID` and `TraceId` where stored as JSON field `spanid` and `traceid`; they are now stored as `metadata[span_id]` and `metadata[trace_id]`, LogQL queries must be changed accordingly.
+
+`TraceID` filters like `| json | traceid=<<traceId>> ...` and `|= <<traceId>> ...` must be converted to `| trace_id=<<traceId>> ...` where `<<traceId>>` and <<spanId>> are the values you search for.  
+Similarly, `SpanID` filters like `| json | spanid=<<spanid>> ...` and `|=<<spanid>> ...` must be converted to `| span_id=<<spanid>> ...`.
+
+Example:
+
+```
+BEFORE
+{exporter="OTLP", job="ecommerce/frontend"} | json | resources_deployment_environment="production" |= "00960a472ea5b87954ca07902d66f914" ...
+
+AFTER
+{service_namespace="ecommerce", service_name="frontend"} | deployment_environment="production" | trace_id="00960a472ea5b87954ca07902d66f914"...
+```
+
+#### From `line_format {{.body}}` to `line_format {{__line__}}`
+
+The `{{.body}}` element of the JSON payload that used to hold the OTel log message body is now the message of the Loki log line and should be referenced as `{{__line__}}` in `line_format` calls.
+
+Example:
+
+```
+AFTER
+{service_namespace="ecommerce", service_name="frontend"} | deployment_environment =~ `production` | line_format `[{{.detected_level}}] {{__line__}}`
+```
+
+#### Using `keep __line__` in direct LogQL queries to prevent stream splits
+
+The number of logstreams return by LogQL queries made directly to Loki (without going through the Grafana GUI) may increase due to the distinct metadata label values and thus impact the sorting of the result as results are sorted per log stream.
+Use the `keep` function to limit the number of log streams return by LogQL queries and have a global doring of the returned  log entries
+
+Example:
+
+```
+AFTER
+... | keep __line__, detected_level
+```
+
+### Grafana visualizations migration
+
+#### Tempo data source: Trace 2 Logs
+
+To enable the "trace to logs" navigation from Tempo to Loki, navigate to the Grafana Tempo data source configuration screen, in the "Trace to logs" section, select "use custom query" and specify the query:
+
+```
+{service_name="${__span.tags["service.name"]}"} | trace_id="${__span.traceId}"
+```
+
+#### Loki data source: Log 2 Trace
+
+To enable the "logs to trace" navigation from Loki to Tempo, navigate to the Grafana Loki data source configuration screen, in the "Derived fields" section, update or create a derived field with:
+* Name: `Trace ID`
+* Type: `Label` (this `Label` name is missleading because it also supports Loki message metadata)
+* Label: `trace_id`
+* Internal link: activated
+* Select the Tempo data source on which "trace to logs" is configured
+
+
+### See Also
+
+* [Loki documentation / Ingesting OpenTelemetry logs](https://grafana.com/docs/loki/latest/send-data/otel/) 
+
+<hr/>
+<hr/>
+
+## Getting Started with the deprecated OpenTelemetry Collector Loki Exporter
 
 The following settings are required: