Benchmarks

vLLM benchmarks. Cost per GPU.

Name: Ollama Q4 vs vLLM bf16 on a 24 GB A10: Speed, time to first token and GSM8K
Creator: Runcrate
Published: 2026-10-28

Qwen2.5-7B on vLLM, measured on rented GPUs: throughput, latency and cost per 1M output tokens, then FP8, prefix caching and Ollama on a 24 GB card.

Launch an L40S All benchmarks

Measured

$0.12per 1M output tokens

RTX A6000, vLLM, 64 concurrent requests, at the run-day price.

Run 2026-10-26, single runSee the table

On this page

Qwen2.5-7B on vLLM, three GPUsRTX A6000, L40S, H100, run 2026-10-26 FP8 vs bf16 on an H100H100, run 2026-10-27 Prefix caching on vs offL40S, run 2026-10-27 Ollama Q4 vs vLLM bf16 on a 24 GB A10A10, run 2026-10-28

Run 2026-10-26

Qwen2.5-7B on vLLM, three GPUs. Cost per 1M output tokens.

At 64 concurrent requests a million output tokens cost $0.12 on the RTX A6000, $0.17 on the L40S and $0.22 on the H100 at run-day prices. The H100 produced 4,515 tokens per second against 1,727 on the RTX A6000, and every user still saw 79 tokens per second.

Single run: one run per GPU, on one morning. First published 2026-10-26.

The full write-up, with every command

Hardware

GPUs: RTX A6000 48 GB, L40S 48 GB and H100 PCIe 80 GB, one GPU per machine

Software

Model: Qwen/Qwen2.5-7B-Instruct, bf16, no quantization
Engine: vLLM 0.11.0, torch 2.8.0+cu128, transformers 4.57.6

Method

Server: --dtype bfloat16 --max-model-len 4096 --gpu-memory-utilization 0.90, everything else default
Load: vllm bench serve, random prompts, 512 tokens in and exactly 256 out, concurrency 1, 8, 32 and 64 (16, 64, 192 and 320 prompts), seed 42, client on the same machine

Price per hour

RTX A6000$0.74 on 2026-10-26, from $0.57/hr per GPU today
L40S$1.14 on 2026-10-26, from $0.54/hr per GPU on a 2-GPU machine today
H100$3.55 on 2026-10-26, from $3.87/hr per GPU on a 4-GPU machine today

Output tokens per second at 64 concurrent requestsOutput tok/s

Measured on 2026-10-26, from the table below.

Throughput, latency and cost per 1M output tokens

GPU	Concurrency	Output tok/s	Per-user tok/s	Median TTFT	p99 TTFT	$ per 1M, run day	$ per 1M, today
RTX A6000	1	42	43	82 ms	85 ms	4.86	3.74
RTX A6000	8	307	40	289 ms	529 ms	0.67	0.52
RTX A6000	32	1,012	32	479 ms	1,940 ms	0.20	0.15
RTX A6000	64	1,727	32	259 ms	3,347 ms	0.12	0.09
L40S	1	48	49	48 ms	49 ms	6.56	3.11
L40S	8	354	45	161 ms	299 ms	0.89	0.42
L40S	32	1,082	34	279 ms	1,050 ms	0.29	0.14
L40S	64	1,887	33	146 ms	2,169 ms	0.17	0.08
H100	1	104	105	25 ms	26 ms	9.47	10.32
H100	8	776	100	62 ms	125 ms	1.27	1.38
H100	32	2,566	87	233 ms	410 ms	0.38	0.41
H100	64	4,515	79	101 ms	960 ms	0.22	0.24

Per-user tok/s is 1000 / median time per output token: how fast one stream feels to the person reading it.

Run day: the price on 2026-10-26, as the run paid it.

Today: the run-day figure times today's price per GPU-hour over the run-day price per GPU-hour. Today's price is the "from" price per GPU-hour shown on /pricing, which can be on a machine with more GPUs than the run used (the list of GPUs below names it). Prices read 2026-11-04 04:00 UTC.

Time before each machine could serve, measured on the machine

GPU	vLLM install	Model download (15 GB)	Server ready	First chat reply
RTX A6000	29 s	103 s	114 s	1.22 s
L40S	7 s	78 s	67 s	0.85 s
H100	4 s	13 s	54 s	0.56 s

What stood out

The H100 produced 2.6x the RTX A6000's throughput at concurrency 64 for 4.8x the run-day hourly price.
Going from 1 to 64 concurrent requests cut the cost per token by about 40x on all three cards.
p99 time to first token at concurrency 64 on the RTX A6000 was 3.3 s, while the median was 259 ms.
The 15 GB model download took 13 s on one machine and 103 s on another.

Keep in mind

Random prompts, fixed 512/256 shape. Real chat traffic has prefix reuse and varied lengths.
PCIe cards, one per machine. No tensor parallelism, no FP8, no speculative decoding.
Treat differences under about 10% as noise.

Run 2026-10-27

FP8 vs bf16 on an H100. Throughput, memory and GSM8K accuracy.

FP8 gave 1.40x the output throughput at 64 concurrent requests, 6,147 against 4,401 tokens per second. On the first 300 GSM8K questions it answered 270 right against 272 for bf16.

Single run: one run per setting. First published 2026-10-27.

The full write-up, with every command

Hardware

GPU: One H100 PCIe 80 GB

Software

Model: Qwen/Qwen2.5-7B-Instruct, the regular bf16 checkpoint for both modes
Engine: vLLM 0.11.0, torch 2.8.0+cu128, transformers 4.57.6
FP8: vLLM online quantization (--quantization fp8): weights converted at load with a per-tensor scale, activations scaled dynamically, KV cache bf16 in both modes

Method

Throughput: vllm bench serve, random prompts, 512 tokens in and exactly 256 out, concurrency 1, 8, 32 and 64 (16, 64, 192 and 320 prompts), seed 42, client on the same machine, a fresh server per mode
Quality: The first 300 GSM8K test questions, zero-shot, greedy, up to 768 output tokens, 64 requests in flight

Price per hour

H100$3.86 on 2026-10-27, from $3.87/hr per GPU on a 4-GPU machine today

Output tokens per second by concurrencybf16FP8

Measured on 2026-10-27, from the table below.

Throughput, latency and cost per 1M output tokens

Concurrency	Mode	Output tok/s	Per-user tok/s	Median TTFT	p99 TTFT	$ per 1M, run day	$ per 1M, today
1	bf16	104	105	26 ms	28 ms	10.33	10.36
1	FP8	160	162	21 ms	25 ms	6.70	6.72
8	bf16	766	99	119 ms	141 ms	1.40	1.40
8	FP8	1,173	153	88 ms	109 ms	0.91	0.91
32	bf16	2,490	84	163 ms	502 ms	0.43	0.43
32	FP8	3,709	130	221 ms	375 ms	0.29	0.29
64	bf16	4,401	77	106 ms	939 ms	0.24	0.24
64	FP8	6,147	108	98 ms	755 ms	0.17	0.17

Run day: the price on 2026-10-27, as the run paid it.

Memory, as vLLM reported it at startup

Mode	Weights	KV cache	KV cache size	Server ready after
bf16	14.25 GiB	51.27 GiB	959,936 tokens	66.5 s
FP8	8.14 GiB	57.38 GiB	1,074,320 tokens	67.5 s

GSM8K, first 300 test questions

Mode	Correct	Accuracy	Wall time, 64 in flight
bf16	272 / 300	90.7%	15.3 s
FP8	270 / 300	90.0%	12.1 s

What stood out

The two modes gave the same final answer on 272 of 300 questions. bf16 alone was right on 10 questions, FP8 alone on 8.
The online conversion added about a second to a 67 s server start.

Keep in mind

GSM8K with 300 questions has a standard error of about 1.7 points at 90% accuracy, so it cannot show a difference under about 3 points.
Online FP8 (per-tensor). Pre-quantized FP8 checkpoints and an FP8 KV cache were not tested.
Ada and Hopper GPUs have FP8 tensor cores. On older cards vLLM falls back to weight-only FP8 kernels, and the speed math changes.

Run 2026-10-27

Prefix caching on vs off. A shared system prompt, one L40S.

With a 4,000-token shared system prompt and 64 users at once, turning prefix caching off cut output throughput from 1,902 to 254 tokens per second and pushed p99 time to first token from 0.38 s to 23.9 s.

Single run: one run per setting, on one machine, on one day. First published 2026-10-27.

The full write-up, with every command

Hardware

GPU: One L40S 48 GB

Software

Model: Qwen/Qwen2.5-7B-Instruct, bf16, no quantization
Engine: vLLM 0.11.0, torch 2.8.0+cu128, transformers 4.57.6; 14.25 GiB of weights and 24.26 GiB of KV cache (454,272 tokens)

Method

Modes: Default (prefix caching on) and --no-enable-prefix-caching (off), a fresh server per mode, --max-model-len 8192 --gpu-memory-utilization 0.90
Load: vllm bench serve, one shared prefix of 1,000 or 4,000 tokens plus 200 unique tokens, exactly 128 output tokens, concurrency 1, 16 and 64 (24, 160 and 384 requests), client on the same machine
Hit rate: From vLLM's /metrics counters, read before and after every run

Price per hour

L40S$1.14 on 2026-10-27, from $0.54/hr per GPU on a 2-GPU machine today

Throughput and latency, cache on and off

Prefix tokens	Users	Cache	Requests/s	Output tok/s	Median TTFT	p99 TTFT	Median time per token	Hit rate	$ per 1,000 requests, today
1,000	1	on	0.37	48	40 ms	44 ms	20.8 ms	85%	$0.405
1,000	1	off	0.36	47	103 ms	106 ms	20.8 ms	not measured	$0.417
1,000	16	on	5.27	675	163 ms	208 ms	22.5 ms	88%	$0.028
1,000	16	off	3.72	476	314 ms	1,416 ms	31.1 ms	not measured	$0.040
1,000	64	on	14.61	1,870	303 ms	773 ms	31.8 ms	88%	$0.010
1,000	64	off	6.63	849	507 ms	5,940 ms	71.5 ms	not measured	$0.023
4,000	1	on	0.37	47	57 ms	63 ms	21.1 ms	96%	$0.405
4,000	1	off	0.33	42	377 ms	385 ms	21.1 ms	not measured	$0.455
4,000	16	on	5.21	667	98 ms	147 ms	23.4 ms	99%	$0.029
4,000	16	off	1.90	243	850 ms	4,987 ms	59.1 ms	not measured	$0.079
4,000	64	on	14.86	1,902	172 ms	384 ms	32.1 ms	99.6%	$0.010
4,000	64	off	1.99	254	1,936 ms	23,935 ms	240 ms	not measured	$0.075

Today: Price per GPU-hour / (requests per second x 3600) x 1,000, with the GPU kept busy at that concurrency. Today's price is the "from" price per GPU-hour shown on /pricing, which can be on a machine with more GPUs than the run used (the list of GPUs below names it). Prices read 2026-11-04 04:00 UTC.

What it meant for the bill at the run-day price, GPU kept busy at that concurrency

Prefix tokens	Users	$ per 1,000 requests, cache on	Cache off	Ratio
1,000	16	0.060	0.085	1.4x
1,000	64	0.022	0.048	2.2x
4,000	16	0.061	0.167	2.7x
4,000	64	0.021	0.160	7.5x

What stood out

With the cache on, at 64 users the 4,000-token prefix ran at 14.86 requests/s and the 1,000-token prefix at 14.61.
Median time per output token at 64 users went from 32 ms to 240 ms with a 4,000-token prefix and the cache off.
For a single user the cache saves latency but not money: requests per second barely moved (0.37 against 0.33).

Keep in mind

Synthetic prompts: one prefix shared by every request. Real traffic mixes several system prompts and lengths; hit rates will be lower.
The cache could hold the whole prefix easily here (454k tokens of KV cache).
Treat differences under about 10% as noise.

Run 2026-10-28

Ollama Q4 vs vLLM bf16 on a 24 GB A10. Speed, time to first token and GSM8K.

On this card Ollama with the Q4_K_M build was faster than vLLM in bf16 at every load tested, from 1 to 16 users, and cost less per token. Q4 scored 90.0% on 250 GSM8K questions against 91.2% for bf16, and vLLM kept the shorter time to first token once more than one user was waiting.

Single run: one run per setting, on one machine, on one day. First published 2026-10-28.

The full write-up, with every command

Hardware

GPU: One A10, 24 GB (23,028 MiB usable, 150 W limit)

Software

Ollama: Ollama 0.40.12, qwen2.5:7b-instruct-q4_K_M (GGUF, 4-bit) and qwen2.5:7b-instruct-q8_0 (GGUF, 8-bit)
vLLM: vLLM 0.11.0, Qwen/Qwen2.5-7B-Instruct in bf16, torch 2.8.0+cu128, transformers 4.57.6, --max-model-len 4096 --gpu-memory-utilization 0.92

Method

Ollama settings: OLLAMA_NUM_PARALLEL=16 OLLAMA_CONTEXT_LENGTH=4096; all 29 layers on the GPU
Client: 32 fixed prompts, temperature 0, max_tokens 256, streamed through /v1/chat/completions; 1, 4 and 16 users at once (12, 32 and 96 requests), one warm-up request first
Quality: The first 250 GSM8K test questions, zero-shot, greedy, up to 512 tokens, 16 at a time

Price per hour

A10$1.68 on 2026-10-28, from $1.68/hr per GPU today

Throughput, time to first token and cost per 1M output tokens (this test reports p95, not p99)

Server	Users	Output tok/s	Per-user tok/s	Median TTFT	p95 TTFT	$ per 1M, run day	$ per 1M, today
Ollama Q4_K_M	1	89	90	35 ms	40 ms	$5.25	$5.25
Ollama Q4_K_M	4	239	61	110 ms	136 ms	$1.95	$1.95
Ollama Q4_K_M	16	618	41	405 ms	483 ms	$0.75	$0.75
Ollama Q8_0	1	58	58	42 ms	44 ms	$8.09	$8.09
Ollama Q8_0	4	197	50	128 ms	152 ms	$2.37	$2.37
Ollama Q8_0	16	484	32	456 ms	511 ms	$0.96	$0.96
vLLM bf16	1	30	30	47 ms	48 ms	$15.40	$15.40
vLLM bf16	4	119	30	72 ms	100 ms	$3.93	$3.93
vLLM bf16	16	449	28	106 ms	173 ms	$1.04	$1.04

Run day: the price on 2026-10-28, as the run paid it.

Memory and GSM8K

Server	GPU memory in use	GSM8K (250, greedy)	GSM8K wall time
Ollama Q4_K_M	8.4 GiB	90.0% (225/250)	104 s
Ollama Q8_0	11.3 GiB	90.4% (226/250)	140 s
vLLM bf16	21.3 GiB	91.2% (228/250)	125 s

vLLM's memory figure is what it reserves up front, not what it needed: 4.61 GiB of it was KV cache.

What stood out

At 16 users Ollama Q4 still led on total throughput (618 against 449 tokens per second), but the gap had closed from 3x to 1.4x.
At 16 users the median time to first token was 106 ms for vLLM and 405 ms for Ollama Q4.
Q4 and bf16 disagreed on 13 of 250 GSM8K questions: bf16 got 8 of them right and Q4 got 5.

Keep in mind

Not Ollama against vLLM in general: vLLM can also serve 4-bit and 8-bit models, and that comparison would be fairer to it.
256 output tokens and short prompts (about 30 tokens). Long prompts would shift the picture towards prefill speed.
Treat differences under about 10% as noise.

Costs

How the costs are worked out. Run day and today, side by side.

Run day

The cost the write-up printed: the hourly price the machine was launched at that day, against what it measured. Prices move daily, so each run-day figure carries its date. Every run was a single run.

Today

The run-day figure moved by today's price: the "from" price per GPU-hour shown on /pricing, over the run-day rate per GPU-hour. A GPU that is not in stock right now reads "Not in stock" rather than a stale price. Prices read 2026-11-04 04:00 UTC.

GPUs

The GPUs on this page. From price per GPU-hour, today.

RTX A6000from $0.57/hr per GPU
L40Sfrom $0.54/hr per GPU on a 2-GPU machine
H100from $3.87/hr per GPU on a 4-GPU machine
A10from $1.68/hr per GPU

Benchmarks

More benchmarks.

A100 vs H100 vs H200One script, six GPUs.718.46 bf16 TFLOPS. H200, dense bf16 matmul at 8192, run 2026-10-25, single run.Speech to textAudio hours per dollar.315x real time. RTX 5090, faster-whisper large-v3-turbo, batch 16, run 2026-10-28, single run.Text embeddingsCost per passage.$0.032 per 1M passages. L40S, bge-small, 128-token passages, at the run-day price, run 2026-10-29, single run.Training and fine-tuningMemory, speed, cost.14.5 GiB peak memory. LoRA of Qwen2.5-1.5B on an L40S, micro-batch 4, run 2026-10-28, single run.Image generationImages per dollar.$0.91 per 1,000 images. RTX A6000, compiled, batch 1, at the run-day price, run 2026-10-27, single run.GPU startup timeTimed from the API call.186 s to the first CUDA op. L40S, from the create call to a 4096 x 4096 matmul, run 2026-10-26, single run.