Budgeting for GPU Servers: Where the Money Actually Goes

GPU hosting is priced per GPU-hour, which makes it look simple and leads directly to the mistake that dominates this category: paying for hardware that is switched on and doing nothing.

VRAM decides what you can run at all

Before comparing prices, work out how much video memory your workload needs. A model that does not fit in VRAM does not run slowly — it does not run.

Rough determinants:

• Model parameter count and the precision you serve at. Lower precision reduces memory substantially, usually with acceptable quality loss for inference.
• Batch size and context length for language models — both scale memory use, and context length is the one people underestimate.
• Training needs several times what inference needs for the same model, because of optimiser state and gradients.

Get this number first. It determines which cards are viable, and that determines price far more than anything else.

Idle time is the actual bill

A GPU reserved by the hour bills by the hour whether or not it is computing. Teams provision an instance for a training run, the run finishes overnight, and the instance stays up for three days because nobody was watching.

This — not the hourly rate — is where most GPU budget overruns come from. Practical defences:

• Auto-shutdown on idle, configured before the first run rather than after the first surprise bill
• Separating interactive development from training, so a notebook left open is not holding an expensive card
• Scale-to-zero for inference workloads with gaps in traffic, accepting cold start latency in exchange
• A billing alert at a threshold that would concern you, set on day one

Reserved, on-demand and interruptible

On-demand costs most per hour and commits to nothing. Right for experimentation and unpredictable work.

Reserved or committed capacity costs less per hour in exchange for a term commitment. Worth it only when utilisation is genuinely high and sustained — the break-even is a utilisation percentage, so calculate it rather than assuming.

Interruptible or spot capacity is cheapest and can be reclaimed with little notice. Excellent for training that checkpoints regularly and can resume. Unsuitable for anything user-facing.

Many teams end up with a mix: reserved for baseline inference, on-demand for spikes, interruptible for training.

The costs that are not the GPU

• Storage. Datasets and checkpoints are large and persist. Fast storage attached to the instance is charged separately from cheaper archival storage, and checkpoints accumulate unless someone prunes them.
• Data transfer. Moving data in is usually free; moving it out often is not. An inference service returning large outputs, or a pipeline shuttling data between providers, can generate egress charges comparable to the compute.
• The CPU and RAM around the GPU. Data loading is frequently the bottleneck. An underspecified host starves an expensive card.
• Networking between GPUs for multi-GPU training. Interconnect quality materially affects how well training scales, and it varies between providers at the same nominal card.

Inference and training have different economics

Training is bursty and tolerant of interruption. Optimise for cheap capacity and checkpoint often.

Inference is continuous and latency-sensitive. Optimise for utilisation — batching requests, right-sizing the card to the model rather than buying the largest available, and scaling with traffic.

Serving a small model on a large card because it was what was available is common and expensive.

Before you commit to anything

1. Measure VRAM requirement at your real batch size and context length.
2. Estimate hours per month honestly, including development, not just production.
3. Price the storage and egress separately, not as an afterthought.
4. Run a week on-demand and measure actual utilisation before reserving.
5. Configure idle shutdown and a billing alert before the first job.

About the author. Kislay Chandan has worked with schools since 2014. Rather than gathering requirements from the outside, he worked inside a school to see what actually goes wrong day to day, and built what he found into the software. He still works with Shemrock Glitters in Patna — most changes to the product start with something that came up there.