Use case · inference

Serving a language model.

Capacity decides what you can run at all; bandwidth decides how fast it answers. Here is the cheapest card in the catalogue for each model size, with the price per hour and the template that serves it.

Deploy with vLLM Compare GPUs

8B at FP16: $0.05/h1× A30 on spot
70B at FP8: $0.19/h1× RTX PRO 6000 on spot
Best bandwidth per $: RTX 3080760 GB/s at $0.03/h
Template: vLLMOpenAI-compatible server, pre-pulled on every node

Which GPU

The cheapest node that holds the model.

Computed from the catalogue: weights at the stated precision, 20% of the memory left free, cheapest spot price among all 47 models. A bigger card is never wrong — it is just more expensive.

Model	FP16node · spot price	FP8 / INT8node · spot price	INT4node · spot price
Llama 3.1 8BThe default small assistant · 8B	1× A30 $0.05/h	1× RTX A4000 $0.03/h	1× RTX A4000 $0.03/h
Qwen2.5 32BCode and reasoning · 32B	1× A100 PCIe $0.09/h	1× A40 $0.06/h	1× A30 $0.05/h
Llama 3.3 70BThe production baseline · 70B	4× A40 $0.24/h	2× A40 $0.12/h	1× A40 $0.06/h
Mixtral 8x22BMixture of experts · 141B	8× A40 $0.48/h	4× A40 $0.24/h	2× A40 $0.12/h

These are floors, not recommendations for production: a long context window adds tens of gigabytes of KV cache, and a busy endpoint wants headroom. Each model page shows the same arithmetic for that exact card.

What sets the speed

Token generation is a memory problem.

Every token generated reads the entire model out of GPU memory. That is why two cards with similar FLOPS can differ by a factor of three on the same model, and why HBM cards cost what they cost.

GPU	Memory	Bandwidth	Spot	GB/s per $
RTX 3080Ampere	10 GB	760 GB/s	$0.03	25,333
RTX 3080 TiAmpere	12 GB	912 GB/s	$0.04	22,800
V100 16GBVolta	16 GB	900 GB/s	$0.04	22,500
A100 PCIe 80GBAmpere	80 GB	1,935 GB/s	$0.09	21,500
RTX 5090Blackwell	32 GB	1,792 GB/s	$0.09	19,911
RTX 3090Ampere	24 GB	936 GB/s	$0.05	18,720

Ranked by bandwidth per dollar of spot time. We publish no tokens-per-second figures: they depend on your model, your batch size, your context length and your serving engine, and a number measured on our hardware would not predict yours.

Serve it in three commands

vLLM template

inside the instance

# The model cache lives on scratch, the weights you keep on the volume.
$ export HF_HOME=/mnt/scratch/hf
$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
      --host 0.0.0.0 --port 8000 --max-model-len 8192

# From your laptop, the usual OpenAI client works unchanged.
$ curl http://<instance>:8000/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{"model":"meta-llama/Llama-3.1-8B-Instruct",
           "messages":[{"role":"user","content":"hello"}]}'

vLLM with OpenAI-compatible server on port 8000, Hugging Face cache on the persistent disk. All templates

Which tier

Spot for the batch, on-demand for the endpoint.

Serve on spot

Batch scoring and offline generation, evaluation sweeps, synthetic data, overflow capacity behind a load balancer that can drain a node on the 2-minute notice. You save 30–53% for work that does not mind waiting.

How reclaims work

Serve on-demand

A production API with a latency budget, a customer demo, anything where a cold start in the middle of the afternoon is unacceptable. Never reclaimed, covered by the 99.9% monthly target.

On-demand instances

The hybrid that most teams end up with.

One on-demand replica holds the floor. Spot replicas absorb the peaks and the batch work, and are drained cleanly when the notice fires. The bill looks like on-demand for a fraction of the traffic and like spot for the rest.

FAQ

Serving questions

How much GPU memory does a model need to be served?

The weights, plus room for the KV cache and activations. Weights are about 2 GB per billion parameters at FP16, 1 GB at FP8 and 0.5 GB in 4-bit. We keep 20% of the card free in the tables below; a long context window or a large batch can need more than that, and the serving engine will tell you.

Does a faster GPU mean more tokens per second?

Usually it means more memory bandwidth, not more FLOPS. Generating a token reads the whole model out of memory, so a card with 760 GB/s beats a card with half that on the same model, whatever their compute numbers say.

Should I serve on spot?

A production endpoint with a latency budget belongs on on-demand. Batch scoring, evaluation runs and overflow capacity belong on spot, where the same GPU costs 30–53% less. Many teams run both and let the load balancer drain spot nodes on a reclaim notice.

Do you provide an inference endpoint, or a machine?

A machine. You get a virtual machine with the GPU, the vLLM template already on the node, and an address. What you expose on it — an OpenAI-compatible API, your own gateway, nothing at all — is your decision.

What about quantised models?

They are the cheapest way to serve: a 70B model in 4-bit fits on 1× A40 from $0.06 an hour. Quality depends on the quantisation method and the model; test it on your own prompts before deciding.

Get started

Serve an 8B model for $0.05 an hour.

Pay as you go — no contracts, no minimum commitment. Add credit, launch, stop whenever you want.

Deploy with vLLM Other use cases

Billed per minute from the moment the instance is reachable. Minimum credit $40, no subscription.