Add a section for OTel specific values in TraceState. (open-telemetry#1852)

* Add a section for OTel specific values in TraceState.

* Update specification/trace/api.md

Co-authored-by: Yuri Shkuro <[email protected]>

* Update specification/trace/api.md

Co-authored-by: Yuri Shkuro <[email protected]>

* Fix sample.

* Apply feedback.

* Use same notation as TraceContext.

* Remove unused OWS mention.

* Fix link.

* Update specification/trace/sdk.md

Co-authored-by: Yuri Shkuro <[email protected]>

* Clarify values.

* Prevent larger than 256 char values.

* Fix typo.

* Update specification/trace/sdk.md

Co-authored-by: Christian Neumüller <[email protected]>

* Move trace state handling to its own document.

* Update specification/trace/tracestate-handling.md

Co-authored-by: John Watson <[email protected]>

* Fix lint.

* Apply feedback.

* Fix typo.

* s/SHOULD/MUST

* Update specification/trace/tracestate-handling.md

Co-authored-by: Joshua MacDonald <[email protected]>

Co-authored-by: Yuri Shkuro <[email protected]>
Co-authored-by: Christian Neumüller <[email protected]>
Co-authored-by: John Watson <[email protected]>
Co-authored-by: Joshua MacDonald <[email protected]>

Author: 5 people

specification/trace/api.md

@@ -203,7 +203,9 @@ only supports a single flag called [sampled](https://www.w3.org/TR/trace-context
 `TraceState` carries vendor-specific trace identification data, represented as a list
 of key-value pairs. TraceState allows multiple tracing
 systems to participate in the same trace. It is fully described in the [W3C Trace Context
-specification](https://www.w3.org/TR/trace-context/#tracestate-header).
+specification](https://www.w3.org/TR/trace-context/#tracestate-header). For
+specific OTel values in `TraceState`, see the [TraceState Handling](tracestate-handling.md)
+document.
 
 The API MUST implement methods to create a `SpanContext`. These methods SHOULD be the only way to
 create a `SpanContext`. This functionality MUST be fully implemented in the API, and SHOULD NOT be

specification/trace/tracestate-handling.md

@@ -0,0 +1,81 @@
+# TraceState Handling
+
+**Status**: [Experimental](../document-status.md)
+
+In alignment to the [TraceContext](https://www.w3.org/TR/trace-context/) specification, this section uses the
+Augmented Backus-Naur Form (ABNF) notation of [RFC5234](https://www.w3.org/TR/trace-context/#bib-rfc5234),
+including the DIGIT rule in that document.
+
+When setting [TraceState](api.md#tracestate) values that are part of the OTel ecosystem,
+they MUST all be contained in a single entry using the `ot` key, with the value being
+a semicolon separated list of key-value pairs such as:
+
+* `ot=p:8;r:64`
+* `ot=foo:bar;k1:13`
+
+The [TraceContext](https://www.w3.org/TR/trace-context/) specification defines support for multiple "tenants" each to use their own `tracestate` entry by prefixing `tenant@` to tenant-specific values in a mixed tracing environment. OpenTelemetry recognizes this syntax but does not specify an interpretation for multi-tenant `tracestate`.
+The list can be formally defined as:
+
+```
+list        = list-member *( ";" list-member )
+list-member = key ":" value
+```
+
+The complete list length MUST NOT exceed 256 characters, as defined by the
+[TraceState value section](https://www.w3.org/TR/trace-context/#value),
+and the used keys MUST be unique.
+
+Instrumentation libraries and clients MUST NOT use this entry, and they MUST
+instead use their own entry.
+
+## Key
+
+The key is an identifier that describes an OTel concern.
+Simple examples are `p`, `ts`, or `s1`.
+
+The key can be formally defined as:
+
+```
+key        = lcalpha *(lcalpha / DIGIT )
+lcalpha    = %x61-7A ; a-z
+```
+
+Specific keys used by OTel concerns MUST be defined as part as the Specification,
+and hence it is forbidden to use to use any key that has not been defined in
+the Specification itself.
+
+## Value
+
+The value is an opaque string. Although it has no maximum allowed length,
+it is recommended to use short values, as the **entire** list of key-values
+MUST NOT exceed 256 characters.
+
+The value can be formally defined as:
+
+```
+value      = *(chr)
+chr        = ucalpha / lcalpha / DIGIT / "." / "_" / "-"
+ucalpha    = %x41-5A ; A-Z
+lcalpha    = %x61-7A ; a-z
+```
+
+## Setting values
+
+Set values MUST be either updated or added to the `ot` entry in `TraceState`,
+in order to preserve existing values belonging to other OTel concerns. For example,
+if a given concern K wants to set `k1:13`:
+
+* `ot=p:8;r:64` will become `ot=p:8;r:64;k1:13`.
+* `ot=p:8;k1:7;r:64` will become `ot=p8;r:64;k1:13`. Preserving the order is not required.
+
+If setting a value ends up making the entire `ot` entry exceed the 256 characters limit,
+SDKs are advised to abort the operation and signal the user about the error, e.g.
+
+```go
+traceState, ok := SetTraceStateValue(traceState, value)
+if ok {
+  // Successfully set the specified value, traceState was updated.
+} else {
+  // traceState was not updated.
+}
+```