[NDINT-290] Versa SLA Metrics w/ Session Auth #37300

ken-schneider · 2025-05-22T19:56:05Z

What does this PR do?

Adds back support for session and CSRF token based authentication to allow calls to Versa analytics APIs which do not support HTTP Basic Auth.

With this change, we can enable collection of the SLA metrics from a previous PR.

Session/Token auth is only used when SLA metrics are being called for, basic auth is still the default for now.

Motivation

Describe how you validated your changes

Ran the agent locally against our own Versa cluster.

Possible Drawbacks / Trade-offs

Additional Notes

zoedt · 2025-05-22T20:15:29Z

pkg/collector/corechecks/network-devices/versa/versa.go

-// generateDeviceNameToIDMap generates a map of device ID to device name to enrich the results from Analytics responses
-func generateDeviceNameToIDMap(deviceMetadata []devicemetadata.DeviceMetadata) map[string]string {
-	deviceNameToIDMap := make(map[string]string)
+// TODO: should we convert the tags map to use ID instead of IP?


woops totally valid, i think device_id probably is better here? 🤦 id/ip really mixed up easily

not blocker for the pr, but agree! thanks for adding this todo

zoedt · 2025-05-22T20:20:24Z

pkg/collector/corechecks/network-devices/versa/client/auth.go

+
+	// Request to /versa/analytics/login to obtain Analytics CSRF prevention token
+	analyticsPayload := url.Values{}
+	analyticsPayload.Set("endpoint", client.analyticsEndpoint) // TODO: WHY? Where can we get this for others?


non blocking question
i forget if during our 1:1 you made a successful call to /versa/login that doesn't require this endpoint parameter but if i recall correctly, returns the token we're looking for?

tbh it's very close to this /versa/analytics/login so maybe i mixed it up 🤔

This flow requires the analytics endpoint parameter to be set. I haven't been able to get that version working, it's something I'm going to look at when I get back.

zoedt · 2025-05-22T20:24:05Z

pkg/collector/corechecks/network-devices/versa/versa.go

@@ -162,6 +152,16 @@ func (v *VersaCheck) Run() error {
 		v.metricsSender.SendDirectorStatus(directorStatus)
 	}

+	// Temporarily commenting out the SLA metrics collection until Versa Analytics auth is sorted out


we can remove this comment if the metrics are flowing in ok!

agent-platform-auto-pr · 2025-05-22T20:25:48Z

Uncompressed package size comparison

Comparison with ancestor c99380382f2ae83bcd454153973f5188637155fe

Diff per package

package	diff	status	size	ancestor	threshold
datadog-iot-agent-x86_64-rpm	0.02MB	⚠️	62.81MB	62.79MB	0.50MB
datadog-iot-agent-x86_64-suse	0.02MB	⚠️	62.81MB	62.79MB	0.50MB
datadog-iot-agent-amd64-deb	0.02MB	⚠️	62.73MB	62.71MB	0.50MB
datadog-agent-x86_64-rpm	0.02MB	⚠️	782.41MB	782.39MB	0.50MB
datadog-agent-x86_64-suse	0.02MB	⚠️	782.41MB	782.39MB	0.50MB
datadog-iot-agent-aarch64-rpm	0.02MB	⚠️	59.39MB	59.37MB	0.50MB
datadog-agent-amd64-deb	0.02MB	⚠️	773.47MB	773.44MB	0.50MB
datadog-heroku-agent-amd64-deb	0.02MB	⚠️	380.18MB	380.16MB	0.50MB
datadog-iot-agent-arm64-deb	0.02MB	⚠️	59.30MB	59.28MB	0.50MB
datadog-agent-aarch64-rpm	0.02MB	⚠️	768.34MB	768.32MB	0.50MB
datadog-agent-arm64-deb	0.02MB	⚠️	759.41MB	759.39MB	0.50MB
datadog-dogstatsd-amd64-deb	0.00MB	✅	31.99MB	31.99MB	0.50MB
datadog-dogstatsd-x86_64-rpm	0.00MB	✅	32.07MB	32.07MB	0.50MB
datadog-dogstatsd-x86_64-suse	0.00MB	✅	32.07MB	32.07MB	0.50MB
datadog-dogstatsd-arm64-deb	0.00MB	✅	30.47MB	30.47MB	0.50MB

Decision

⚠️ Warning

cit-pr-commenter · 2025-05-22T20:51:08Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 31839b5d-3626-4791-9953-02b41b639298

Baseline: e1d209a
Comparison: 0b14520
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	ddot_metrics	memory utilization	+0.19	[+0.07, +0.31]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	+0.13	[-0.48, +0.75]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	+0.06	[-0.55, +0.68]	1	Logs
➖	file_to_blackhole_0ms_latency_http1	egress throughput	+0.05	[-0.53, +0.62]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.04	[-0.57, +0.64]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.25, +0.26]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.02, +0.02]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.00	[-0.58, +0.57]	1	Logs
➖	file_to_blackhole_1000ms_latency_linear_load	egress throughput	-0.03	[-0.27, +0.20]	1	Logs
➖	file_to_blackhole_0ms_latency_http2	egress throughput	-0.05	[-0.66, +0.55]	1	Logs
➖	docker_containers_memory	memory utilization	-0.07	[-0.17, +0.03]	1	Logs
➖	quality_gate_logs	% cpu utilization	-0.12	[-2.83, +2.59]	1	Logs bounds checks dashboard
➖	file_to_blackhole_100ms_latency	egress throughput	-0.15	[-0.78, +0.47]	1	Logs
➖	ddot_logs	memory utilization	-0.27	[-0.42, -0.13]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.40	[-0.53, -0.27]	1	Logs bounds checks dashboard
➖	otlp_ingest_metrics	memory utilization	-0.62	[-0.78, -0.46]	1	Logs
➖	docker_containers_cpu	% cpu utilization	-0.68	[-3.61, +2.25]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	-0.74	[-0.82, -0.66]	1	Logs
➖	otlp_ingest_logs	memory utilization	-0.92	[-1.04, -0.79]	1	Logs
➖	quality_gate_idle	memory utilization	-1.09	[-1.15, -1.02]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	-1.17	[-1.22, -1.11]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-1.49	[-2.32, -0.66]	1	Logs
➖	file_tree	memory utilization	-2.01	[-2.20, -1.83]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http1	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http1	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http2	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http2	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.

agent-platform-auto-pr · 2025-05-22T23:29:13Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor e1d209a

Successful checks

Info

	Quality gate	Delta	On disk size (MiB)	Delta	On wire size (MiB)
✅	agent_deb_amd64	$${+0.03}$$	$${695.74}$$ < $${752.99}$$	$${-0.03}$$	$${175.89}$$ < $${187.44}$$
✅	agent_deb_amd64_fips	$${+0.02}$$	$${693.95}$$ < $${751.36}$$	$${+0.01}$$	$${175.62}$$ < $${187.06}$$
✅	agent_heroku_amd64	$${+0.03}$$	$${358.23}$$ < $${369.68}$$	$${+0}$$	$${96.44}$$ < $${99.55}$$
✅	agent_msi	$${+0.07}$$	$${957.79}$$ < $${987.01}$$	$${+0.04}$$	$${146.17}$$ < $${150.72}$$
✅	agent_rpm_amd64	$${+0.03}$$	$${695.73}$$ < $${752.98}$$	$${+0.04}$$	$${177.5}$$ < $${190.03}$$
✅	agent_rpm_amd64_fips	$${+0.02}$$	$${693.94}$$ < $${751.35}$$	$${-0.04}$$	$${177.32}$$ < $${189.81}$$
✅	agent_rpm_arm64	$${+0.02}$$	$${685.73}$$ < $${739.42}$$	$${+0.05}$$	$${160.91}$$ < $${171.23}$$
✅	agent_rpm_arm64_fips	$${+0.02}$$	$${684.08}$$ < $${737.91}$$	$${-0.09}$$	$${160.0}$$ < $${170.22}$$
✅	agent_suse_amd64	$${+0.03}$$	$${695.73}$$ < $${752.98}$$	$${+0.04}$$	$${177.5}$$ < $${190.03}$$
✅	agent_suse_amd64_fips	$${+0.02}$$	$${693.94}$$ < $${751.35}$$	$${-0.04}$$	$${177.32}$$ < $${189.81}$$
✅	agent_suse_arm64	$${+0.02}$$	$${685.73}$$ < $${739.42}$$	$${+0.05}$$	$${160.91}$$ < $${171.23}$$
✅	agent_suse_arm64_fips	$${+0.02}$$	$${684.08}$$ < $${737.91}$$	$${-0.09}$$	$${160.0}$$ < $${170.22}$$
✅	docker_agent_amd64	$${+0.03}$$	$${779.53}$$ < $${849.39}$$	$${+0.04}$$	$${268.49}$$ < $${288.34}$$
✅	docker_agent_arm64	$${+0.02}$$	$${792.98}$$ < $${858.97}$$	$${+0.01}$$	$${255.85}$$ < $${274.36}$$
✅	docker_agent_jmx_amd64	$${+0.03}$$	$${970.72}$$ < $${1040.59}$$	$${-0.01}$$	$${337.43}$$ < $${357.3}$$
✅	docker_agent_jmx_arm64	$${+0.02}$$	$${972.77}$$ < $${1038.77}$$	$${-0}$$	$${320.79}$$ < $${339.32}$$
✅	docker_agent_windows1809	$${-0.31}$$	$${1485.51}$$ < $${1557.19}$$	$${-0.43}$$	$${487.2}$$ < $${507.33}$$
✅	docker_agent_windows1809_core	$${+0.07}$$	$${6215.85}$$ < $${6287.46}$$	$${0}$$	$${2048.0}$$ < $${2067.76}$$
✅	docker_agent_windows1809_core_jmx	$${+22.2}$$	$${6359.67}$$ < $${6409.04}$$	$${0}$$	$${2048.0}$$ < $${2067.76}$$
✅	docker_agent_windows1809_jmx	$${+0.07}$$	$${1607.52}$$ < $${1679.14}$$	$${-0.02}$$	$${529.89}$$ < $${550.27}$$
✅	docker_agent_windows2022	$${-0.13}$$	$${1505.03}$$ < $${1576.85}$$	$${-0.02}$$	$${500.38}$$ < $${520.76}$$
✅	docker_agent_windows2022_core	$${+0.05}$$	$${6189.1}$$ < $${6260.53}$$	$${0}$$	$${2048.0}$$ < $${2067.76}$$
✅	docker_agent_windows2022_core_jmx	$${+0.06}$$	$${6310.76}$$ < $${6382.17}$$	$${0}$$	$${2048.0}$$ < $${2067.76}$$
✅	docker_agent_windows2022_jmx	$${+0.02}$$	$${1626.9}$$ < $${1698.2}$$	$${+0.01}$$	$${542.65}$$ < $${562.99}$$
✅	docker_cluster_agent_amd64	$${+0.02}$$	$${212.81}$$ < $${259.73}$$	$${-0.01}$$	$${72.36}$$ < $${103.68}$$
✅	docker_cluster_agent_arm64	$${+0}$$	$${228.68}$$ < $${274.24}$$	$${+0}$$	$${68.64}$$ < $${98.45}$$
✅	docker_cws_instrumentation_amd64	$${0}$$	$${7.08}$$ < $${7.12}$$	$${-0}$$	$${2.95}$$ < $${3.29}$$
✅	docker_cws_instrumentation_arm64	$${0}$$	$${6.69}$$ < $${6.92}$$	$${-0}$$	$${2.7}$$ < $${3.07}$$
✅	docker_dogstatsd_amd64	$${-0}$$	$${39.21}$$ < $${39.57}$$	$${+0}$$	$${15.11}$$ < $${15.76}$$
✅	docker_dogstatsd_arm64	$${-0}$$	$${37.88}$$ < $${38.2}$$	$${-0}$$	$${14.53}$$ < $${14.83}$$
✅	dogstatsd_deb_amd64	$${0}$$	$${30.44}$$ < $${31.52}$$	$${+0}$$	$${8.0}$$ < $${8.97}$$
✅	dogstatsd_deb_arm64	$${0}$$	$${29.01}$$ < $${30.08}$$	$${-0}$$	$${6.94}$$ < $${7.92}$$
✅	dogstatsd_rpm_amd64	$${0}$$	$${30.44}$$ < $${31.52}$$	$${+0}$$	$${8.0}$$ < $${8.98}$$
✅	dogstatsd_suse_amd64	$${0}$$	$${30.44}$$ < $${31.52}$$	$${+0}$$	$${8.0}$$ < $${8.98}$$
✅	iot_agent_deb_amd64	$${+0.02}$$	$${50.34}$$ < $${60.17}$$	$${+0.01}$$	$${12.81}$$ < $${15.82}$$
✅	iot_agent_deb_arm64	$${+0.02}$$	$${47.81}$$ < $${56.94}$$	$${+0.01}$$	$${11.12}$$ < $${13.86}$$
✅	iot_agent_deb_armhf	$${+0.03}$$	$${47.38}$$ < $${56.41}$$	$${+0.01}$$	$${11.18}$$ < $${13.86}$$
✅	iot_agent_rpm_amd64	$${+0.02}$$	$${50.34}$$ < $${60.18}$$	$${+0}$$	$${12.83}$$ < $${15.84}$$
✅	iot_agent_rpm_arm64	$${+0.02}$$	$${47.81}$$ < $${56.94}$$	$${+0}$$	$${11.14}$$ < $${13.76}$$
✅	iot_agent_suse_amd64	$${+0.02}$$	$${50.34}$$ < $${60.18}$$	$${+0}$$	$${12.83}$$ < $${15.84}$$

zoedt

lgtmmm ty for re-adding !!

ken-schneider · 2025-06-12T15:31:52Z

/merge

dd-devflow · 2025-06-12T15:32:03Z

View all feedbacks in Devflow UI.

2025-06-12 15:32:02 UTC ℹ️ Start processing command /merge

2025-06-12 15:32:23 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2025-06-12 17:59:22 UTC ⚠️ MergeQueue: This merge request was unqueued

[email protected] unqueued this merge request

ken-schneider · 2025-06-12T17:59:08Z

/merge -c

dd-devflow · 2025-06-12T17:59:14Z

View all feedbacks in Devflow UI.

2025-06-12 17:59:14 UTC ℹ️ Start processing command /merge -c

ken-schneider · 2025-06-12T19:16:28Z

At this point, the logic in auth.go has been updated to avoid making multiple calls to any endpoint. We've added a single GET call to /versa/analytics/auth/user which allows us to make the other two pre-existing calls to log in to the director and analytics once.

ken-schneider · 2025-06-12T19:26:44Z

/merge

dd-devflow · 2025-06-12T19:26:50Z

View all feedbacks in Devflow UI.

2025-06-12 19:26:49 UTC ℹ️ Start processing command /merge

2025-06-12 19:27:03 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2025-06-12 21:03:24 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 53m (p90).

2025-06-12 22:06:21 UTC ℹ️ MergeQueue: This merge request was merged

ken-schneider added 2 commits May 22, 2025 15:48

re-add token auth for analytics, enable SLA collection

47be593

use single wrapper for basic and session calls

625d464

ken-schneider requested a review from a team as a code owner May 22, 2025 19:56

ken-schneider changed the title ~~Ken/versa sla metrics session auth~~ [NDINT-290] Versa SLA Metrics w/ Session Auth May 22, 2025

github-actions bot added medium review PR review might take time team/ndm-integrations labels May 22, 2025

ken-schneider added changelog/no-changelog qa/done QA done before merge and regressions are covered by tests labels May 22, 2025

zoedt reviewed May 22, 2025

View reviewed changes

ken-schneider added 3 commits May 22, 2025 16:54

remove comment

189b2c4

fix test server url parsing

da837ec

fix device ID test

5f3aa85

remove outdated comment

8200280

zoedt approved these changes May 23, 2025

View reviewed changes

ken-schneider and others added 3 commits June 11, 2025 17:05

Merge branch 'main' into ken/versa-sla-metrics-session-auth

a0b49ab

use initial GET for token

4cf720c

update test server routes

b660c8a

use single call for analytics csrf/auth

d8b3a25

remove unnecessary logs

0b14520

dd-mergequeue bot merged commit c7fefaf into main Jun 12, 2025
237 of 241 checks passed

dd-mergequeue bot deleted the ken/versa-sla-metrics-session-auth branch June 12, 2025 22:06

github-actions bot added this to the 7.68.0 milestone Jun 12, 2025

[NDINT-290] Versa SLA Metrics w/ Session Auth #37300

[NDINT-290] Versa SLA Metrics w/ Session Auth #37300

Uh oh!

Conversation

ken-schneider commented May 22, 2025

What does this PR do?

Motivation

Describe how you validated your changes

Possible Drawbacks / Trade-offs

Additional Notes

Uh oh!

zoedt May 22, 2025

Choose a reason for hiding this comment

Uh oh!

zoedt May 22, 2025

Choose a reason for hiding this comment

Uh oh!

zoedt May 22, 2025

Choose a reason for hiding this comment

Uh oh!

ken-schneider May 22, 2025

Choose a reason for hiding this comment

Uh oh!

zoedt May 22, 2025

Choose a reason for hiding this comment

Uh oh!

agent-platform-auto-pr bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uncompressed package size comparison

Decision

Uh oh!

cit-pr-commenter bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

agent-platform-auto-pr bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Static quality checks

Info

Uh oh!

zoedt left a comment

Choose a reason for hiding this comment

Uh oh!

ken-schneider commented Jun 12, 2025

Uh oh!

dd-devflow bot commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ken-schneider commented Jun 12, 2025

Uh oh!

dd-devflow bot commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ken-schneider commented Jun 12, 2025

Uh oh!

ken-schneider commented Jun 12, 2025

Uh oh!

dd-devflow bot commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

agent-platform-auto-pr bot commented May 22, 2025 •

edited

Loading

cit-pr-commenter bot commented May 22, 2025 •

edited

Loading

agent-platform-auto-pr bot commented May 22, 2025 •

edited

Loading

dd-devflow bot commented Jun 12, 2025 •

edited

Loading

dd-devflow bot commented Jun 12, 2025 •

edited

Loading

dd-devflow bot commented Jun 12, 2025 •

edited

Loading