Extend instance state CPU data to include maximum possible CPU usage (ns) per second

[bensmrs]

Currently, only CPU time is reported in the /state API endpoint for instances. While the total CPU time can be interesting to some, I think most users are actually interested in a CPU usage in percents (which we can calculate easily from CPU time) relative to the CPU count (which we don’t have access to).

[stgraber]

I'm not sure how you'd get a percentage value by knowing the current CPU count.

The data we get from the kernel is the total CPU time since start of the instance, this doesn't account for any change in the number of CPUs available to the instance during its lifetime or changes in CPU availability on the host such as isolated CPUs or similar dedicated allocations to other processes on the system.

So knowing that your container used 300s of CPU time, has been running for a week and is on a 64 core system would have your report 0.000775% of CPU usage. That doesn't seem terribly useful to me.

I suspect what most folks are actually after is an instant CPU consumption in percentage but that we definitely don't have the data for.

[bensmrs]

I'm not sure how you'd get a percentage value by knowing the current CPU count.

No, but doing $100\times\frac{dm\left(t\right)}{dt}$ (with $m$ the measure of CPU time), you get the used CPU time as a percentage of elapsed time. Divide it by the CPU count and you get a percentage relative to all the CPUs.

The data we get from the kernel is the total CPU time since start of the instance, this doesn't account for any change in the number of CPUs available to the instance during its lifetime or changes in CPU availability on the host such as isolated CPUs or similar dedicated allocations to other processes on the system.

I’m not sure that invalidates my point.

So knowing that your container used 300s of CPU time, has been running for a week and is on a 64 core system would have your report 0.000775% of CPU usage. That doesn't seem terribly useful to me.

It’s not, so I’m not suggesting exposing the CPU percentage, only CPU time and CPU count. The idea is to allow a WebUI (to be honest, I happen to have started developing one a few months ago) to show immediate (well, after at least 2 measurements) CPU usage in a meter bar, as you currently do on the branded UI with memory and disk, and as most other hypervisor UIs do. On a meter bar, 100% does not mean 100% CPU time, but 100% CPU time on all the CPUs, as 100% CPU time means one CPU being at 100% capacity.

I suspect what most folks are actually after is an instant CPU consumption in percentage but that we definitely don't have the data for.

Well, we do, if we have the CPU count.

[stgraber]

Okay, that makes sense. When using a client which can gather multiple data points, then you can calculate a CPU usage percentage for that time period.

Adding a "count" field to the CPU state struct which reports the number of CPUs based on cpuset should be reasonably straightforward (minus the parsing of the cpuset value but we already have code for that).

How do you want to handle limits.cpu.allowance though?
Say you have a container with limits.cpu=4 limits.cpu.allowance=100ms/200ms on a system with 64 cores.

Just returning the CPU count (4) would be quite incorrect as the container is only allowed to use 50% of one full CPU core worth of time, so should the CPU count be 0.5 then, do we need a separate field to handle the count of CPU threads as opposed to the amount of CPU time that can be used?

Or should we return something like a total of CPU time in nanoseconds that can be used per second of real time?

So you'd get:

• 64000000 on a 64 core system with no CPU limits
• 4000000 for a container that's limited with limits.cpu=4
• 500000 for the case above of a container that's got limits.cpu=4 and limits.cpu.allowance=100ms/200ms

[bensmrs]

Okay, that makes sense. When using a client which can gather multiple data points, then you can calculate a CPU usage percentage for that time period.

Yeah, sorry if the original message was not clear :)

How do you want to handle limits.cpu.allowance though?
Say you have a container with limits.cpu=4 limits.cpu.allowance=100ms/200ms on a system with 64 cores.

Oh that’s a tricky one…

Or should we return something like a total of CPU time in nanoseconds that can be used per second of real time?

I really like this solution

500000 for the case above of a container that's got limits.cpu=4 and limits.cpu.allowance=100ms/200ms

You meant limits.cpu=1, right?

Then it’ll be up to the UI to say “xx% out of yy equivalent CPUs” or stuff like that, or to find a better wording.

[stgraber]

You meant limits.cpu=1, right?

Nope, I meant limits.cpu=4 limits.cpu.allowance=100ms/200ms

This expands to:

• Allowed to be scheduled across 4 CPUs concurrently
• A maximum of CPU time across all CPUs of 100ms can be consumed per 200ms of real time

So basically so long as you have allowance=100ms/200ms, it doesn't matter if limits.cpu=1 or limits.cpu=256. Your maximum CPU time remains 50% of a CPU, but you can burn that 50% of a CPU quite a bit faster by spreading it over multiple CPUs at the same time.

[bensmrs]

Oh right, I had misinterpreted the allowance policy (never used before)

[stgraber]

@bensmrs you want to take this one?

[bensmrs]

Yeah I can do that. But I probably won’t look at it until we’ve finished our mega-PR :)

[bensmrs]

For LXC, shall I take into account limits.cpu.priority (and then use ParseCPU) or are we really only interested in allowances before priority kicks in?
Also, if neither limits.cpu nor limits.cpu.allowance are set for LXC, do I consider all the CPUs available to the host? (I’ll probably do the same as cmd/incusd/devices.go but a common shared function would have been lovely)

[stgraber]

For LXC, shall I take into account limits.cpu.priority (and then use ParseCPU) or are we really only interested in allowances before priority kicks in?

The priority is really just a scheduling priority when under load, so I don't think it would cause a change to the returned value in this case.

Also, if neither limits.cpu nor limits.cpu.allowance are set for LXC, do I consider all the CPUs available to the host? (I’ll probably do the same as cmd/incusd/devices.go but a common shared function would have been lovely)

You should still be able to read the CPU set from the cgroup, even if no limit is applied.
I think for this feature in general, I'd prefer that we pull the cgroup data rather than trusting the configuration as when no limits are applied by Incus, systemd or other systems on the host may still be setting limits.

On cgroup2, you'd basically want to read cpuset.cpus and cpu.max to get the needed values.

We have the cgroup package that helps with that with:

• GetEffectiveCPUset()
• GetCPUCfsLimit() (doesn't exist and would need implementing, following rough logic from SetCPUCfsLimit)

Logic would then basically be:

• Call GetCPUCfsLimit, if a limit is present, calculate the max CPU time from that
• If no CFS limit in place, then call GetEffectiveCPUSet and assume 100% of the CPU count