windows_system_processor_queue_length returning incorrect values

[mikhaildudzin]

Current Behavior

After upgrading from version 0.27.2 to 0.30.5
windows_system_processor_queue_length started showing incorrect values in comparison to Perfmon,
Exporter always shows that there's queue even though there's no in perfmon
On the image attached red line indicates upgrade, this behavior is noticed on all servers

Expected Behavior

correct values are gathered

Steps To Reproduce

Environment

• windows_exporter Version: 0.30.5
• Windows Server Version: 2022, 2019

windows_exporter logs

If logs are needed please request and I will provide

Anything else?

No response

[jkroepke]

What is the actual proof that the current values are incorrect and that the values from version 0.27 are the correct ones?

Could you please open perfmon.exe on the affected system and observe the \System\Processor Queue Length metric? It would be very helpful if you could share a screenshot of the live graph over time. Ideally, you can also compare this with what Grafana shows for the same time period. This direct comparison might help confirm if there is indeed a mismatch or if the current metric behaves as expected.

[mikhaildudzin]

Attaching four screenshots, two for each version one with perfmon metrics another with metrics from windows exporter
version 0.27.2 (branch: HEAD, revision: 4e460bc):

version 0.30.5 (branch: HEAD, revision: a56e1ac):

If needed can try to upgrade not straight to 0.30.5, but with smaller steps to identify in which version issue started

[jkroepke]

If needed can try to upgrade not straight to 0.30.5,

I guess v0.29.2 should work while 0.30.0 not.

[jkroepke]

I did some tests on my local system and compared perfmon data with PowerShell using Get-Counter "\System\Processor Queue Length" -Continuous, windows_exporter 0.27, and windows_exporter 0.31 (main branch). All values are different, including the values between PowerShell and perfmon.

I get the feeling it's hard to say which one is actually correct. If an application requests the value of Processor Queue Length, it only reflects the state at that exact moment, and the value can already be different in the next millisecond. The issue is that Windows does not offer an averaged value for this counter.

Starting from version 0.30, windows_exporter uses the modern Windows API (Performance Data Helpers) to fetch values. In contrast, version 0.27 manually scrapes the binary registry values of performance data and tries to decode them.

The main difference is that version 0.30 uses a multi-threaded approach to query Windows, while version 0.27 used a single-threaded one. In your case, it might be possible that Processor Queue Length is always greater than zero, simply because threads from windows_exporter itself are in the processor queue at scrape time.

You could double-check this by running windows_exporter with only the system collector enabled.

[mikhaildudzin]

Yes just doublechecked that with only system collector enabled, queue is not always greater then zero.
For us this metric became less intuitive and less useful, it's harder to say if smth is going on or not since there's always a queue

[jkroepke]

Depends on what you expect. It is less intuitive to check if the value is zero. As I know, if the value divide by CPU Core count is less than 10, it's fine.

https://learn.microsoft.com/en-us/archive/msdn-technet-forums/a29869f1-39c5-4351-ba0e-0b7da321b751