The first infrastructure problem I ever owned was a queue. In 2015, at a TCS research lab in Pune, the on-prem HPC cluster would back up at peak and researchers would wait hours for a slot. Off-peak, the same machines sat idle. My job was to build a framework that burst jobs to the cloud when the queue got long, across HTCondor, OGS, and Torque schedulers, from a KVM-based OpenNebula private cloud out to AWS.
Peak queue wait went down without buying more nodes. Underneath that was one insight: idle capacity and queue wait are one number, viewed from opposite sides.
I’ve carried that with me through much larger fleets. It’s still the most under-measured thing on most platforms.
Cost is an engineering metric, and it belongs on the same dashboard
I treat cost the way I treat p99 latency: something the platform team owns, measures continuously, and is accountable for. When I built the TCO and cost-comparison models for a logistics-technology company’s executive cloud decision, the useful part wasn’t the total. It was the shape: which workloads were paying for headroom they never used, and which were starved.
Koo and Tokopedia both moved with operational cost coming down alongside performance going up. That’s possible mostly because migration is the one moment everyone agrees to look at utilisation honestly.
Outside a migration, the looking stops. That’s the problem.
Where the idle hides
Static fleets. At a securities brokerage in Indonesia we replaced static VMs under a Kubernetes platform with Omni machine classes: nodes are created with tags and enrol themselves, and the fleet flexes with demand instead of sitting at the size someone guessed a year ago. The old model wasn’t wrong when it was set up. It was wrong six months later and nobody had a graph that said so.
Headroom with no owner. Every team over-requests CPU and memory “to be safe” — and requests, not limits, are what the Kubernetes scheduler bin-packs against, so an inflated request reserves capacity nothing will ever use. Individually reasonable. Across 100 applications it’s a cluster running well under half utilised that looks full to the scheduler. The fix is a utilisation-versus-request graph per namespace and someone whose job it is to look at it.
Expensive accelerators treated like cheap compute. On a fleet where a single node costs what a rack used to, an idle hour is a line item. Utilisation of the scarce resource (GPU, in that world) is an SLI in its own right, and a job that holds one while waiting on I/O is an incident in slow motion.
What I’d put on the capacity dashboard
Four graphs, per pool:
- Utilisation of the scarce resource, real, not requested.
- Queue wait or pending time for work that wants that resource.
- Idle capacity in currency, per day.
- Forecast: at current growth, when does 2 go bad?
If graph 1 is low and graph 2 is high at the same time, you don’t have a capacity problem. You have a scheduling or a bin-packing problem, and buying more will hide it for a quarter and then bring it back.
The uncomfortable part
No one gets paged for idle. That’s why it accumulates. Latency has an owner and an alert; waste has a quarterly review, if it’s lucky.
Give it an owner. Put the currency graph next to the latency graph. Then the queue and the idle stop being two teams’ problems and become one team’s number, which is what they always were.