[EBPF] gpu: handle runtime changes of CUDA_VISIBLE_DEVICES

[gjulianm]

What does this PR do?

This PR adds support for handling runtime updates for the CUDA_VISIBLE_DEVICES environment variable. This consists of the following changes:

• Add an uprobe to the setenv function in libc, that watches for the calls with the CUDA_VISIBLE_DEVICES variable name exclusively. The value will be sent to the GPU event ringbuffer.
• Add a new library set to the shared libraries watcher so that we can watch for processes opening libc.
• Add a rule to the GPU monitoring attacher, to attach to all processes that have setenv. It's necessary to do for all processes, as a process might change its environment variable before calling any CUDA functions. If we probed only processes that have the CUDA library loaded we would miss those events.
• The GPU consumer now detects the events corresponding to changes in this variable, and will update them in the systemContext.
• systemContext will prioritize this value now over the one from /proc/pid/environ when finding the devices visible to a process.
• In order to select the origin of the CUDA_VISIBLE_DEVICES, the code retrieving the environment variable for the process has been moved from the pkg/gpu/cuda package to systemContext.

Motivation

https://datadoghq.atlassian.net/browse/NWL-1275

We have detected an issue in a ray cluster. When deployed in a node with multiple GPUs, ray will spawn one process per GPU. Each process will then update its own CUDA_VISIBLE_DEVICES environment variable. Our current approach will not detect this change, and will consider that all processes are using the same GPU.

Describe how you validated your changes

Unit tests provided, tested also in the GPU load test environment with no significant changes in resource usage.

Finally, manually tested this in an AWS instance with Ray running on multiple GPUs, replicating the environment with the original error.

Possible Drawbacks / Trade-offs

Additional Notes

[Copilot]

Pull Request Overview

This PR adds support for capturing and handling runtime updates to the CUDA_VISIBLE_DEVICES environment variable by introducing a new eBPF uprobe, event type, and propagation through the Go consumer and context.

• Defines a new VisibleDevicesSet event in eBPF C and Go types, tests, and string mappings
• Implements a setenv uprobe to detect changes to CUDA_VISIBLE_DEVICES at runtime
• Propagates and invalidates updated env var values in systemContext and the GPU event consumer

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/network/usm/sharedlibraries/libset.go	Add libc libset constant and suffix mapping
pkg/network/ebpf/c/shared-libraries/probes.h	Introduce match4chars macro and capture libc libset events
pkg/network/ebpf/c/shared-libraries/maps.h	Add libc_shared_libraries perf event array
pkg/gpu/probe.go	Register uprobe__setenv, update shared libs libsets slice
pkg/gpu/ebpf/kprobe_types_string_linux.go	Append new event names to _CudaEventType_name and index
pkg/gpu/ebpf/kprobe_types_linux_test.go	Add alignment test for CudaVisibleDevicesSetEvent
pkg/gpu/ebpf/kprobe_types_linux.go	Define CudaVisibleDevicesSetEvent, update constants & sizes
pkg/gpu/ebpf/kprobe_types.go	Declare Go type and constants for new event
pkg/gpu/ebpf/c/types.h	Define cuda_visible_devices_set_t struct and related macros
pkg/gpu/ebpf/c/runtime/gpu.c	Implement SEC("uprobe/setenv") to read and emit new event
pkg/gpu/cuda/env_test.go	Update tests to call ParseVisibleDevices
pkg/gpu/cuda/env.go	Rename and refactor env var parsing to ParseVisibleDevices
pkg/gpu/context_test.go	Add tests for runtime-updated visible devices cache invalidation
pkg/gpu/context.go	Track and invalidate updated env var values in systemContext
pkg/gpu/consumer.go	Handle VisibleDevicesSet events and update GPU context
pkg/ebpf/uprobes/attacher_test.go	Extend shared-library tests to include libc libset
pkg/ebpf/cgo/genpost.go	Include "Devices" field in generated post-processing

[cit-pr-commenter]

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: d3f1c203-a289-4a9f-83a6-bbc436bbc228

Baseline: 636a14c
Comparison: edcc32a
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+1.96	[-1.17, +5.09]	1	Logs
➖	quality_gate_logs	% cpu utilization	+1.79	[-0.98, +4.56]	1	Logs bounds checks dashboard
➖	file_tree	memory utilization	+0.96	[+0.83, +1.09]	1	Logs
➖	otlp_ingest_logs	memory utilization	+0.81	[+0.68, +0.94]	1	Logs
➖	otlp_ingest_metrics	memory utilization	+0.34	[+0.20, +0.48]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.33	[+0.28, +0.38]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.33	[+0.25, +0.41]	1	Logs bounds checks dashboard
➖	quality_gate_idle	memory utilization	+0.25	[+0.16, +0.34]	1	Logs bounds checks dashboard
➖	ddot_logs	memory utilization	+0.15	[+0.06, +0.24]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	+0.05	[-0.53, +0.63]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	+0.04	[-0.59, +0.67]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	+0.03	[-0.55, +0.61]	1	Logs
➖	file_to_blackhole_1000ms_latency_linear_load	egress throughput	+0.02	[-0.21, +0.26]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.01	[-0.59, +0.62]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.01	[-0.02, +0.04]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.26, +0.24]	1	Logs
➖	docker_containers_memory	memory utilization	-0.01	[-0.07, +0.05]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.01	[-0.60, +0.57]	1	Logs
➖	file_to_blackhole_0ms_latency_http2	egress throughput	-0.04	[-0.62, +0.53]	1	Logs
➖	file_to_blackhole_0ms_latency_http1	egress throughput	-0.11	[-0.67, +0.45]	1	Logs
➖	ddot_metrics	memory utilization	-0.18	[-0.30, -0.06]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.29	[-1.16, +0.59]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-1.00	[-1.09, -0.92]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http1	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http1	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http2	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http2	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

• quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.

[agent-platform-auto-pr]

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 636a14c

Successful checks

Info

	Quality gate	Delta	On disk size (MiB)	Delta	On wire size (MiB)
✅	agent_deb_amd64	$${+0}$$	$${696.11}$$ < $${700.55}$$	$${+0}$$	$${175.87}$$ < $${177.62}$$
✅	agent_deb_amd64_fips	$${+0}$$	$${694.36}$$ < $${699.75}$$	$${-0.02}$$	$${175.35}$$ < $${177.04}$$
✅	agent_heroku_amd64	$${0}$$	$${349.11}$$ < $${362.84}$$	$${-0}$$	$${93.56}$$ < $${98.91}$$
✅	agent_msi	$${+0}$$	$${971.14}$$ < $${974.5}$$	$${-0.03}$$	$${147.22}$$ < $${148.11}$$
✅	agent_rpm_amd64	$${+0}$$	$${696.1}$$ < $${700.53}$$	$${+0.02}$$	$${177.53}$$ < $${179.2}$$
✅	agent_rpm_amd64_fips	$${+0}$$	$${694.35}$$ < $${699.74}$$	$${+0.07}$$	$${177.29}$$ < $${178.75}$$
✅	agent_rpm_arm64	$${+0}$$	$${686.32}$$ < $${690.5}$$	$${+0.03}$$	$${161.12}$$ < $${162.56}$$
✅	agent_rpm_arm64_fips	$${+0}$$	$${684.63}$$ < $${689.8}$$	$${+0.01}$$	$${160.17}$$ < $${161.61}$$
✅	agent_suse_amd64	$${+0}$$	$${696.1}$$ < $${700.54}$$	$${+0.02}$$	$${177.53}$$ < $${179.2}$$
✅	agent_suse_amd64_fips	$${+0}$$	$${694.35}$$ < $${699.74}$$	$${+0.07}$$	$${177.29}$$ < $${178.93}$$
✅	agent_suse_arm64	$${+0}$$	$${686.32}$$ < $${690.43}$$	$${+0.03}$$	$${161.12}$$ < $${162.57}$$
✅	agent_suse_arm64_fips	$${+0}$$	$${684.63}$$ < $${689.78}$$	$${+0.01}$$	$${160.17}$$ < $${161.61}$$
✅	docker_agent_amd64	$${+0.02}$$	$${779.91}$$ < $${784.33}$$	$${+0.01}$$	$${268.64}$$ < $${270.56}$$
✅	docker_agent_arm64	$${+0.02}$$	$${793.59}$$ < $${797.68}$$	$${-0}$$	$${256.02}$$ < $${257.84}$$
✅	docker_agent_jmx_amd64	$${+0.02}$$	$${971.11}$$ < $${975.52}$$	$${+0.01}$$	$${337.6}$$ < $${339.53}$$
✅	docker_agent_jmx_arm64	$${+0.02}$$	$${973.38}$$ < $${977.47}$$	$${+0.01}$$	$${320.97}$$ < $${322.81}$$
✅	docker_agent_windows1809	$${-0.38}$$	$${1183.19}$$ < $${1190.0}$$	$${-0.48}$$	$${416.8}$$ < $${422.66}$$
✅	docker_agent_windows1809_core	$${+0}$$	$${5913.96}$$ < $${5919.41}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows1809_core_jmx	$${+22.06}$$	$${6057.79}$$ < $${6063.56}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows1809_jmx	$${+0}$$	$${1305.63}$$ < $${1311.1}$$	$${-0.07}$$	$${459.78}$$ < $${464.35}$$
✅	docker_agent_windows2022	$${+0}$$	$${1203.29}$$ < $${1208.75}$$	$${+0.01}$$	$${430.27}$$ < $${434.91}$$
✅	docker_agent_windows2022_core	$${-0.16}$$	$${5887.05}$$ < $${5892.66}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows2022_core_jmx	$${+0.05}$$	$${6008.95}$$ < $${6014.08}$$	$${0}$$	$${2048.0}$$ < $${2049.0}$$
✅	docker_agent_windows2022_jmx	$${+0.07}$$	$${1325.06}$$ < $${1330.39}$$	$${-0.02}$$	$${472.53}$$ < $${477.17}$$
✅	docker_cluster_agent_amd64	$${+0.01}$$	$${213.64}$$ < $${214.5}$$	$${-0.01}$$	$${72.61}$$ < $${73.51}$$
✅	docker_cluster_agent_arm64	$${+0.01}$$	$${229.44}$$ < $${230.33}$$	$${+0}$$	$${68.89}$$ < $${69.77}$$
✅	docker_cws_instrumentation_amd64	$${0}$$	$${7.07}$$ < $${7.12}$$	$${-0}$$	$${2.95}$$ < $${3.29}$$
✅	docker_cws_instrumentation_arm64	$${+0}$$	$${6.69}$$ < $${6.92}$$	$${+0}$$	$${2.7}$$ < $${3.07}$$
✅	docker_dogstatsd_amd64	$${+0}$$	$${39.21}$$ < $${39.57}$$	$${-0}$$	$${15.12}$$ < $${15.76}$$
✅	docker_dogstatsd_arm64	$${0}$$	$${37.88}$$ < $${38.2}$$	$${-0}$$	$${14.54}$$ < $${14.83}$$
✅	dogstatsd_deb_amd64	$${0}$$	$${30.42}$$ < $${31.4}$$	$${+0}$$	$${8.0}$$ < $${8.95}$$
✅	dogstatsd_deb_arm64	$${0}$$	$${28.99}$$ < $${29.97}$$	$${-0}$$	$${6.94}$$ < $${7.89}$$
✅	dogstatsd_rpm_amd64	$${0}$$	$${30.42}$$ < $${31.4}$$	$${+0}$$	$${8.01}$$ < $${8.96}$$
✅	dogstatsd_suse_amd64	$${0}$$	$${30.42}$$ < $${31.4}$$	$${+0}$$	$${8.01}$$ < $${8.96}$$
✅	iot_agent_deb_amd64	$${0}$$	$${53.77}$$ < $${54.55}$$	$${-0}$$	$${13.55}$$ < $${14.45}$$
✅	iot_agent_deb_arm64	$${0}$$	$${51.11}$$ < $${51.9}$$	$${-0}$$	$${11.73}$$ < $${12.63}$$
✅	iot_agent_deb_armhf	$${0}$$	$${50.62}$$ < $${51.42}$$	$${-0.02}$$	$${11.79}$$ < $${12.74}$$
✅	iot_agent_rpm_amd64	$${0}$$	$${53.77}$$ < $${54.55}$$	$${-0}$$	$${13.57}$$ < $${14.47}$$
✅	iot_agent_rpm_arm64	$${0}$$	$${51.11}$$ < $${51.91}$$	$${+0}$$	$${11.75}$$ < $${12.65}$$
✅	iot_agent_suse_amd64	$${0}$$	$${53.77}$$ < $${54.55}$$	$${-0}$$	$${13.57}$$ < $${14.47}$$

[agent-platform-auto-pr]

eBPF complexity changes

Summary result: ❔ - needs attention

• Highest complexity change (%): +4.68%
• Highest complexity change (abs.): +305 instructions
• Programs that were above the 85.0% limit of instructions and are now below: 0
• Programs that were below the 85.0% limit of instructions and are now above: 0

shared_libraries details

shared_libraries [programs with changes]

Program	Avg. complexity	Distro with highest complexity	Distro with lowest complexity
do_sys_openat2_exit	🔴 6827.0 (+305.0, +4.68%)	fedora_38/arm64: 🔴 6827.0 (+305.0, +4.68%)	fedora_38/arm64: 🔴 6827.0 (+305.0, +4.68%)
tracepoint__syscalls__sys_exit_open	🟢 3452.2 (-217.6, -5.93%)	fedora_38/arm64: 🟢 3637.0 (-264.0, -6.77%)	debian_10/arm64: 🟢 3406.0 (-206.0, -5.70%)
tracepoint__syscalls__sys_exit_openat	🟢 3452.2 (-217.6, -5.93%)	fedora_38/arm64: 🟢 3637.0 (-264.0, -6.77%)	debian_10/arm64: 🟢 3406.0 (-206.0, -5.70%)
tracepoint__syscalls__sys_exit_openat2	🟢 3452.2 (-217.6, -5.93%)	fedora_38/arm64: 🟢 3637.0 (-264.0, -6.77%)	debian_10/arm64: 🟢 3406.0 (-206.0, -5.70%)

shared_libraries [programs without changes]

Program	Avg. complexity	Distro with highest complexity	Distro with lowest complexity
tracepoint__syscalls__sys_enter_open	⚪ 4549.8 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 5457.0 (+0.0, +0.00%)	debian_10/arm64: ⚪ 4323.0 (+0.0, +0.00%)
tracepoint__syscalls__sys_enter_openat	⚪ 4549.8 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 5457.0 (+0.0, +0.00%)	debian_10/arm64: ⚪ 4323.0 (+0.0, +0.00%)
tracepoint__syscalls__sys_enter_openat2	⚪ 4627.8 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 5543.0 (+0.0, +0.00%)	debian_10/arm64: ⚪ 4399.0 (+0.0, +0.00%)

This report was generated based on the complexity data for the current branch guillermo.julian/setenv-detect (pipeline 70195279, commit edcc32a) and the base branch main (commit 636a14c). Objects without changes are not reported. Contact #ebpf-platform if you have any questions/feedback.

Table complexity legend: 🔵 - new; ⚪ - unchanged; 🟢 - reduced; 🔴 - increased

[guyarb]

LGTM on the USM owned files

[aliciascott]

good for docs

[val06]

few small comments

[gjulianm]

/merge

[dd-devflow-routing-codex]

View all feedbacks in Devflow UI.

2025-07-10 17:00:01 UTC ℹ️ Start processing command /merge

---

2025-07-10 17:00:16 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

---

2025-07-10 17:08:24 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 55m (p90).

---

2025-07-10 18:27:41 UTC ℹ️ MergeQueue: This merge request was merged