GPU Economics: Cost Per Inference, Not Cost Per Hour

GPU infrastructure is billed per hour and should be evaluated per unit of work. The gap between those two is where most of the waste lives, and it is invisible unless you measure utilisation.

Start with utilisation, because it is usually the problem

Two different numbers, both needed:

• Allocation utilisation — of the hours you paid for, what fraction had a job running at all? This catches instances left on overnight, which is the single largest source of waste in this category.
• Device utilisation — while a job was running, how busy was the GPU actually? A training loop bottlenecked on data loading can show low device utilisation while the card is fully allocated.

Teams commonly discover allocation utilisation well under half. Fixing that — idle shutdown, separating development from training — is cheaper and faster than any hardware change, and it needs no negotiation with a vendor.

Cost per unit of work

The metric that lets you compare options that are not otherwise comparable:

• Inference: cost per thousand requests, or per million tokens. Whatever unit your business actually counts.
• Training: cost per completed run, or per experiment.

This is what makes a smaller cheaper card versus a larger faster one a decidable question. The larger card costs more per hour and may finish enough faster to be cheaper per run — or may not, if the workload cannot use it. Measure it rather than reasoning about it.

Throughput per rupee, measured on your workload

Vendor benchmarks are run on workloads chosen to look good. Run your own model, at your own batch size and precision, and measure requests or tokens per second. Divide by the hourly cost.

The ranking frequently differs from the specification sheet, because real workloads are constrained by memory bandwidth, data loading or the CPU around the GPU rather than by raw compute.

Rent, reserve or buy

Three regimes with different break-even points:

• On-demand wins at low or unpredictable utilisation. No commitment, highest hourly cost.
• Reserved wins above a utilisation threshold sustained across the commitment term. Calculate the threshold from the actual discount rather than assuming reserved is cheaper.
• Owned hardware becomes competitive at consistently high utilisation over a long horizon — but the comparison must include power, cooling, space, replacement cycle and the staff time to run it. Those are frequently omitted, which makes ownership look better than it is.

The honest test for ownership: if your measured allocation utilisation is not high and steady, renting is cheaper, and buying converts a variable cost into a fixed one you cannot reduce.

Costs that belong in the calculation

• Storage for datasets and checkpoints, which persists between runs
• Data egress, which can rival compute for an inference service returning large outputs
• Engineering time spent on infrastructure rather than on models
• Idle capacity held for latency reasons, which is a real cost of a responsive service

A monthly review worth doing

1. Allocation and device utilisation for the month
2. Cost per unit of work, trended against previous months
3. Largest single line item, and whether it was necessary
4. Any instance running more than a week — justified or forgotten?

Fifteen minutes. In most teams the fourth question alone pays for it.

About the author. Kislay Chandan has worked with schools since 2014. Rather than gathering requirements from the outside, he worked inside a school to see what actually goes wrong day to day, and built what he found into the software. He still works with Shemrock Glitters in Patna — most changes to the product start with something that came up there.