Articles / INFRASTRUCTURE

CPU limits, CFS throttling and the latency you cannot see in averages

A container can be throttled while its CPU graph shows 20% usage. Here is how CPU limits really work, how to measure throttling, and what it does to a multi-threaded runtime.

InfrastructureKubernetesPerformance
Table of contents
  1. Context
  2. Why averages hide it
  3. Measuring it
  4. The runtime makes it worse
  5. What to do about it
  6. A worked example
  7. Burst, and what changed in newer kernels
  8. Exclusive cores instead
  9. Telling it apart from a runtime problem
  10. Verifying a change
  11. Conclusion
  12. References

There is a classic support conversation in Kubernetes. A service has high tail latency. Its CPU usage graph is flat at 20%. Everyone agrees it cannot be a CPU problem.

It frequently is. The graph is just measuring the wrong thing.

Context

When you set a CPU limit on a container, Kubernetes does not give it a slower processor. It configures the kernel's CFS bandwidth controller with two numbers: a quota and a period.

The period is 100 ms by default. A limit of 500m means a quota of 50 ms of CPU time per 100 ms period, summed across all threads of the container.

When the container has used up its quota inside a period, every one of its threads is paused until the next period starts. This is throttling.

Why averages hide it

The important word in that description is summed. A single-threaded process burns 50 ms of CPU in 50 ms of wall time. A process with five busy threads burns it in 10 ms.

Imagine a request that needs 20 ms of CPU, handled while the quota is almost exhausted. It can run for a few milliseconds, then freeze for the remaining part of the period, and finish up to 90 ms later than it should have.

Now look at the usage graph. Over a 15 second scrape interval, the container consumed well under its limit. The average says 20%. The requests that were frozen say otherwise.

Measuring it

The kernel counts throttling directly. Two cgroup counters, exposed by cAdvisor as Prometheus metrics, are enough:

promql
sum(rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])) by (pod)
/
sum(rate(container_cpu_cfs_periods_total{container!=""}[5m])) by (pod)

This gives the fraction of periods in which the container was throttled. A service that is throttled in more than a few percent of periods is paying latency for it, whatever its utilization says.

On the node itself, the same information lives in the cgroup's cpu.stat, with fields such as nr_throttled and throttled_usec.

The runtime makes it worse

Language runtimes size themselves from the number of CPUs they believe they have. In .NET, Environment.ProcessorCount takes the container's CPU limit into account, rounded up to a whole number.

That value decides things such as how many threads the thread pool starts with and how many heaps the garbage collector uses. So a limit does two things at once: it caps the quota, and it shapes how the runtime parallelizes.

A runtime that spawns work across many threads inside a tight quota is the exact recipe for burning the quota in a few milliseconds and then stalling.

What to do about it

There is no single answer, but the options are well understood:

  • Stop confusing requests and limits. A CPU request is a scheduling guarantee, translated into a relative weight. A limit is a hard cap per period. Many teams only need the first.
  • Remove CPU limits on latency-sensitive services while keeping accurate requests. Under contention, the kernel shares CPU in proportion to requests. Without contention, the service can use idle capacity instead of being throttled while the node sits half empty.
  • If you keep limits, make them generous, and track the throttling ratio as an SLI of its own, not only utilization.
  • Match the runtime to the limit. Tell the runtime how many CPUs it really has, or reduce its parallelism, so it does not spawn more work than the quota can pay for.

Removing limits is controversial, and for good reasons: it makes noisy neighbors possible and removes a safety net against runaway processes. It is a trade-off to make deliberately, with namespace-level quotas and alerts, and not a default to copy blindly.

One more historical note: older kernels had a bug in how unused quota slices expired, which caused throttling even in containers well below their limit. It was fixed in Linux 5.4. If you see unexplained throttling on an old kernel, check that first.

A worked example

A service has a limit of 1 CPU, so the quota is 100 ms per 100 ms period, and it runs 8 busy threads on a node with plenty of idle cores.

All eight threads run in parallel. They consume the 100 ms of quota in 12.5 ms of wall-clock time. Then the container is frozen for the remaining 87.5 ms of the period.

During that window nothing executes: not the request handlers, not the garbage collector, not the health check endpoint. A request that arrived just after the freeze began waits almost the entire period before it is even looked at, and a liveness probe with a short timeout can fail while the service is perfectly healthy.

Across a full second, the container used exactly one core on average, which is what the graph shows. It was also stopped for more than 85% of the time.

Burst, and what changed in newer kernels

Quota that is not used in one period used to be simply lost. Newer kernels (5.14 and later) can let a cgroup carry unused quota forward as a burst allowance (cpu.max.burst on cgroup v2), which absorbs short spikes without throttling. Kubernetes does not expose it as a first-class setting everywhere yet, so check what your distribution and runtime support before relying on it.

Exclusive cores instead

For the most latency-sensitive services, Kubernetes offers the static CPU manager policy. A pod in the Guaranteed QoS class that requests a whole number of CPUs, with requests equal to limits, receives exclusive cores. Nothing else is scheduled on them, and the pod's threads do not compete with neighbors.

It trades flexibility for predictability: those cores cannot be shared when idle, so it suits a small set of critical services and not a general default.

Telling it apart from a runtime problem

Throttling and thread pool starvation look alike from the outside: latency rises, throughput stalls, CPU looks low. The way to separate them is to look at both sides. The cgroup counters say whether the kernel froze the container. The runtime counters, such as the thread pool queue length and the number of busy threads, say whether the application ran out of threads to run work. A service can suffer from both at once, and fixing one makes the other visible.

Verifying a change

After any change to limits, compare the throttled-period ratio and the tail latency before and after, over the same traffic pattern. If removing a limit lowered the ratio to zero and the 99th percentile fell with it, the throttling was real. If nothing moved, the bottleneck is elsewhere, and you have saved yourself a long investigation.

Conclusion

CPU limits are a cap per period, not a share of the machine. Because the quota is consumed by all threads together and enforced in 100 ms windows, a container can stall for most of a window while its average utilization looks perfectly healthy.

The lesson is simple and a bit uncomfortable: the metric we all look at first, CPU usage, is the least informative one here. The throttled-period ratio is the one that tells you whether your latency is being spent waiting for the next window.

References