Skip to content
This repository was archived by the owner on Dec 6, 2024. It is now read-only.

Commit 6b59b29

Browse files
deejgregordeejgregor
committed
Inter-process context propagation requirements
1 parent 5531cb2 commit 6b59b29

File tree

1 file changed

+312
-0
lines changed

1 file changed

+312
-0
lines changed

‎text/trace/0000-inter-process-context-propagation.md

Lines changed: 312 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,312 @@
1+
# Inter-process context propagation requirements
2+
3+
Add requirements for inter-process context propagation using
4+
environment variables as the carrier.
5+
6+
This OTEP is based on discussion in [opentelemetry-specification #740](https://github.com/open-telemetry/opentelemetry-specification/issues/740#issue-665588273), builds on the [Context Propagators
7+
API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/api-propagators.md)
8+
and follows [OTEP 0205: Context propagation requirements for messaging
9+
semantic
10+
conventions](https://github.com/open-telemetry/oteps/blob/main/text/trace/0205-messaging-semantic-conventions-context-propagation.md)
11+
as a guide.
12+
13+
## Motivation
14+
15+
There are existing requirements for context propagation over HTTP,
16+
messaging systems, and others, but there are no requirements for
17+
inter-process context propagation, although there is considerable
18+
ad-hoc propagation being done using environment variables today (see [Prior Art](#prior-art)).
19+
This OTEP is to formalize a standard based on the existing informal
20+
concensus.
21+
22+
These are some examples of batch and system use cases that could
23+
benefit from this standardization:
24+
25+
* CI/CD systems
26+
* Build and deployoment tooling
27+
* ETL
28+
* Automation tools
29+
* System-level tooling
30+
* Command-line tools, scripts, etc.
31+
* Tracing helper tools
32+
33+
## Explanation
34+
35+
To propagate context from a parent process to a child process,
36+
instrumentation can use environment variable(s) with upper-cased variable names
37+
as a carrier for the TextMapPropagator:
38+
39+
40+
```
41+
+----------------+
42+
| Parent Process |
43+
+----------------+
44+
|
45+
| Environment variable(s) used as the carrier
46+
| for context propagation, e.g.: TRACEPARENT
47+
v
48+
+----------------+
49+
| Child Process |
50+
+----------------+
51+
```
52+
53+
## Proposed addition to Process and Process Runtime Resources Semantic Conventions
54+
55+
This is proposed to be added to the
56+
[Process and Process Runtime Resources](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/process.md)
57+
semantic conventions
58+
as a new section and sub-section before the exiting "Process" section and at the same level as that section:
59+
60+
### Conventions
61+
62+
> This document does not specify semantic conventions for Spans related to
63+
> process execution. Future versions of these conventions will give clear
64+
> recommendations on process Spans.
65+
66+
#### Inter-Process Context Propagation
67+
68+
To propagate context from a parent process to a child process,
69+
environment variable(s) are used as the TextMapPropagator carrier
70+
with upper-cased key names.
71+
72+
## Internal details
73+
74+
Instrumentation implementations will likely implement support for inter-process context propagation in two pieces:
75+
76+
* Setting environment variable(s) as the carrier when invoking child processes.
77+
* Reading environment variable(s) when a process starts.
78+
79+
### Should this be added to the specification and/or semantic conventions?
80+
81+
Today, we have protocol-specific context propagation requirements in
82+
both opentelemetry-specification and in semantic-conventions. This OTEP
83+
is specifically for context propagation between processes, and the most
84+
appropriate place seems to be to add a context propagation section to
85+
the
86+
[process resource semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/process.md), which is what is proposed above.
87+
88+
Note that the existing semantic conventions that cover context
89+
propagation are focused on semantic conventions for *span attributes*,
90+
whereas the process semantic conventions are for *resource attributes*.
91+
The process resource semantic conventions do not contain any guidance
92+
on when Spans might be created related to processes. Adding guidance
93+
for process Spans is considered out-of-scope for this OTEP, but would
94+
be a good subject for a future OTEP, and is mentioned as an area for
95+
future work in the suggested [Conventions](#conventions) section.
96+
97+
For reference, the existing locations for protocol-specific context
98+
propagation requirements are:
99+
100+
* Specification: [API Propagators, Propagators Distribution section](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/api-propagators.md#propagators-distribution) -- individual propagation mechanisms are referenced from external sources
101+
* Specification: [API Propagators, B3 Requirements section](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/api-propagators.md#b3-requirements) -- specific considerations for mapping B3 to/from OpenTelemetry
102+
* Semantic Conventions: [Messaging Spans, Context Propagation section](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/messaging-spans.md#context-propagation)
103+
* Semantic Conventions: [CloudEvents, Conventions section](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/cloudevents/cloudevents-spans.md#conventions) -- references external specification for the context propagation mechanism
104+
* Semantic Conventions: [AWS Supplemantary Guidelines, Context Propagation section](https://github.com/open-telemetry/semantic-conventions/blob/main/supplementary-guidelines/compatibility/aws.md#context-propagation)
105+
106+
### Environment variable capitalization
107+
108+
The recommendation is to upper-cased keys in the environment variable carrier for the TextMapPropagator to align with prior art already doing this type of propagation, follow common programming conventions, and work on both UNIX and Windows systems.
109+
110+
#### Prior art
111+
112+
There is significant ad-hoc context propagation being done using environment variables today, and it almost soley uses upper-cased environment variable names (in particular, `TRACEPARENT` and `TRACESTATE`; see [Prior Art](#prior-art)).
113+
114+
#### Programming conventions
115+
116+
Common UNIX conventions usually reserve lower-case variable names for use in application programs and upper-case for environment variables:
117+
118+
> **Constants and Environment Variable Names**
119+
>
120+
> All caps, separated with underscores, declared at the top of the file.
121+
>
122+
> Constants and anything exported to the environment should be capitalized.
123+
124+
Source: [Google, Shell Style Guide, Variable Names](https://google.github.io/styleguide/shellguide.html#s7.3-constants-and-environment-variable-names)
125+
126+
> **Variable Names**
127+
>
128+
> As for function names.
129+
>
130+
> ...
131+
>
132+
> **Function Names**
133+
>
134+
> Lower-case, with underscores to separate words. Separate libraries with ::. Parentheses are required after the function name. The keyword function is optional, but must be used consistently throughout a project.
135+
136+
Sources:
137+
138+
* [Google, Shell Style Guide, Variable Names](https://google.github.io/styleguide/shellguide.html#variable-names)
139+
* [Google, Shell Style Guide, Function Names](https://google.github.io/styleguide/shellguide.html#function-names)
140+
141+
#### UNIX
142+
143+
UNIX system utilities use upper-case for environment variables and lower-case are reserved for applications. Using upper-case will prevent conflicts with internal application variables.
144+
145+
> Environment variable names used by the utilities in the XCU specification consist solely of upper-case letters, digits and the "_" (underscore) from the characters defined in Portable Character Set. Other characters may be permitted by an implementation; applications must tolerate the presence of such names. Upper- and lower-case letters retain their unique identities and are not folded together. The name space of environment variable names containing lower-case letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behaviour of the standard utilities.
146+
147+
Source: [The Open Group, The Single UNIX® Specification, Version 2, Environment Variables](https://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html)
148+
149+
#### Windows
150+
151+
Windows appears to be case-insensitive with environment variables, so the case of environment variables is irrelevant for our uses on Windows, and there might be a conflict if OpenTelemetry keys match internal environment variables used in applications.
152+
153+
There does not seem to be clear documentation on case-insensitivity on Windows from Microsoft, however there are consistent third-party references.
154+
155+
Here is a description of an issue in CPython:
156+
157+
> os.environ docs don't mention that the keys get upper-cased automatically on e.g. Windows.
158+
159+
Source: [CPython issue #101754](https://github.com/python/cpython/issues/101754)
160+
161+
Here is a related documentation addition that was made to clarify functionality:
162+
163+
> On Windows, the keys are converted to uppercase. This also applies when getting, setting, or deleting an item. For example, environ['monty'] = 'python' maps the key 'MONTY' to the value 'python'.
164+
165+
Source: [Python 3 library, os.environ](https://docs.python.org/3/library/os.html#os.environ)
166+
167+
Lastly, here is a programming reference that mentions case-insensitivity:
168+
169+
> **(Windows) Environment Variables**
170+
>
171+
> Environment Variables in Windows are NOT case-sensitive (because the legacy DOS is NOT case-sensitive). They are typically named in uppercase, with words joined with underscore (_), e.g., JAVA_HOME.
172+
173+
Source: [yet another insignificant programming notes... Environment Variables in Windows/macOS/Linux](https://www3.ntu.edu.sg/home/ehchua/programming/howto/Environment_Variables.html#zz-2.)
174+
175+
### Allowed characters
176+
177+
The characters allowed in keys by the TextMapPropagator are all allowed as environment variable keys on UNIX and Windows.
178+
179+
#### OpenTelemetry TextMapPropagator
180+
181+
Allowed characters in keys and values, in short:
182+
> "!" / "#" / "$" / "%" / "&" / "'" / "*"
183+
> / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
184+
> / DIGIT / ALPHA
185+
186+
Details below.
187+
188+
> In order to increase compatibility, the key/value pairs MUST only consist of US-ASCII characters that make up valid HTTP header fields as per RFC 7230.
189+
190+
Source: [OpenTelemetry Specification, API Propagators, TextMapPropagator](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/api-propagators.md#textmap-propagator)
191+
192+
> header-field = field-name ":" OWS field-value OWS
193+
>
194+
> field-name = token
195+
196+
Source: [RFC 7230, section 3.2](https://datatracker.ietf.org/doc/html/rfc7230#section-3.2)
197+
198+
> Most HTTP header field values are defined using common syntax
199+
> components (token, quoted-string, and comment) separated by
200+
> whitespace or specific delimiting characters. Delimiters are chosen
201+
> from the set of US-ASCII visual characters not allowed in a token
202+
> (DQUOTE and "(),/:;<=>?@[\]{}").
203+
>
204+
> token = 1*tchar
205+
>
206+
> tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*"
207+
> / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
208+
> / DIGIT / ALPHA
209+
> ; any VCHAR, except delimiters
210+
211+
Source: [RFC 7230, section 3.2.6](https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.6)
212+
213+
#### UNIX
214+
215+
Summary: The [Portable Character Set](https://pubs.opengroup.org/onlinepubs/7908799/xbd/charset.html#tag_001_004) lists all values allowed by `tchar` in RFC 7230.
216+
217+
> These strings have the form name=value; names do not contain the character =. For values to be portable across XSI-conformant systems, the value must be composed of characters from the portable character set (except NUL and as indicated below). There is no meaning associated with the order of strings in the environment. If more than one string in a process' environment has the same name the consequences are undefined.
218+
>
219+
> Environment variable names used by the utilities in the XCU specification consist solely of upper-case letters, digits and the "_" (underscore) from the characters defined in Portable Character Set. Other characters may be permitted by an implementation; applications must tolerate the presence of such names.
220+
221+
Source: [The Open Group, The Single UNIX® Specification, Version 2, Environment Variables](https://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html)
222+
223+
#### Windows
224+
225+
Summary: Windows only disallows the "=" sign, which is not an allowed charater in `tchar` in RFC 7230.
226+
227+
> The name of an environment variable cannot include an equal sign (=).
228+
229+
Source: [Microsoft, Win32, Processes and Threads, Environment Variables](https://learn.microsoft.com/en-us/windows/win32/procthread/environment-variables)
230+
231+
> The maximum size of a user-defined environment variable is 32,767 characters.
232+
233+
Source: [Microsoft, Win32, Processes and Threads, Environment Variables](https://learn.microsoft.com/en-us/windows/win32/procthread/environment-variables)
234+
235+
### Getter and Setter implementation
236+
237+
In our case, environment variables names on Windows are case insensitive, so we will not violate any of the MUSTs by **not** preserving casing in our `Set` and `Get` implementations for environment variable propagation.
238+
239+
> Set
240+
> ...
241+
> The implementation SHOULD preserve casing (e.g. it should not transform Content-Type to content-type) if the used protocol is case insensitive, otherwise it MUST preserve casing.
242+
-- https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/api-propagators.md#set
243+
244+
> Get
245+
> ...
246+
> The Get function is responsible for handling case sensitivity. If the getter is intended to work with a HTTP request object, the getter MUST be case insensitive.
247+
-- https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/api-propagators.md#get
248+
249+
## Trade-offs and mitigations
250+
251+
### Case-sensitivity
252+
253+
On Windows, in particular, because environment variable keys are case insensitive, there is a (low?) chance that automatically instrumented context propagation variables could conflict with existing application environment variables. It will be important that this instrumentation can be disabled should such cases arise.
254+
255+
### Privacy and Security
256+
257+
Environment variables are visible to any process on the system, so it is important to emphasize existing guidance to not place sensitive information that could be propagated.
258+
259+
If an attacker on a local system can send tracing messages, they could view a process' `TRACEPARENT` environment variable and inject traces that appear to be part of the trace they are targeting. This is likely not much of a concern in many deployment scenarios, but if it is, one mitigation would be to protect the trace endpoint by requiring authentication or disabling inter-process context propagation.
260+
261+
## Prior art and alternatives
262+
263+
### Prior art
264+
265+
Many existing users of `TRACEPARENT` and/or
266+
`TRACESTATE` environment variables mentioned in
267+
[opentelemetry-specification #740](https://github.com/open-telemetry/opentelemetry-specification/issues/740):
268+
269+
* [Jenkins OpenTelemetry Plugin](https://github.com/jenkinsci/opentelemetry-plugin)
270+
* [otel-cli generic wrapper](https://github.com/equinix-labs/otel-cli)
271+
* [Maven OpenTelemetry Extension](https://github.com/cyrille-leclerc/opentelemetry-maven-extension)
272+
* [Ansible OpenTelemetry Plugin](https://github.com/ansible-collections/community.general/pull/3091)
273+
* [go-test-trace](https://github.com/rakyll/go-test-trace/commit/22493612be320e0a01c174efe9b2252924f6dda9)
274+
* [Concourse CI](https://github.com/concourse/docs/pull/462)
275+
* [BuildKite agent](https://github.com/buildkite/agent/pull/1548)
276+
* [pytest](https://github.com/chrisguidry/pytest-opentelemetry/issues/20)
277+
* [Kubernetes test-infra Prow](https://github.com/kubernetes/test-infra/issues/30010)
278+
* [hotel-california](https://github.com/parsonsmatt/hotel-california/issues/3)
279+
280+
### Alternatives and why they were not chosen
281+
282+
#### Case-sensitive variable names (not upper-casing keys in the carrier)
283+
284+
Case-sensitive variable names would allow use of the TextMapPropagator
285+
without specifying custom Getter and Setter implementations and would
286+
fully satisfy the case-sensitivity-related SHOULDs and MUSTs in the
287+
[Propagators API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/api-propagators.md)
288+
specification. There are two overriding reasons to use upper-cased
289+
variable names, however:
290+
291+
1. Windows environment variables are case-insensitive.
292+
2. Significant prior art overwhelmingly uses upper-cased environment variables.
293+
294+
#### Using a file for the carrier
295+
296+
Using a JSON file that is stored on the filesystem and referenced
297+
through an environment variable would eliminate the need to workaround
298+
case-insensitivity issues on Windows, however it would introduce a
299+
number of issues:
300+
301+
1. Would introduce an out-of-band file that would need to be created and reliably cleaned up.
302+
2. Managing permissions on the file might be non-trivial in some circumstances (for example, if `sudo` is used).
303+
3. This would deviate from significant prior art that currently uses environment variables.
304+
305+
306+
## Open questions
307+
308+
The author has no open questions at this point.
309+
310+
## Future possibilities
311+
312+
1. Adding guidance on when to create process Spans.

0 commit comments

Comments
 (0)