Benchmarks

Training benchmarks. Memory, speed, cost.

Name: torch.compile vs eager for training: A 468M model, four GPUs
Creator: Runcrate
Published: 2026-10-29

Full fine-tune, LoRA and QLoRA on one L40S, data parallel training on four PCIe GPUs, and torch.compile on four cards, each with what the training cost.

Launch an L40S All benchmarks

Measured

14.5 GiBpeak memory

LoRA of Qwen2.5-1.5B on an L40S, micro-batch 4.

Run 2026-10-28, single runSee the table

On this page

Full fine-tune vs LoRA vs QLoRA on one L40SL40S, run 2026-10-28 PyTorch DDP on four PCIe RTX A6000sRTX A6000, run 2026-10-26 torch.compile vs eager for trainingRTX 5090, L40S, RTX A6000, H100, run 2026-10-29

Run 2026-10-28

Full fine-tune vs LoRA vs QLoRA on one L40S. Qwen2.5-1.5B.

Full fine-tuning needed 36.9 GiB at micro-batch 4 and ran out of memory at micro-batch 8. LoRA needed 14.5 GiB and trained 1.4x faster, and its held-out loss was a tie with full fine-tuning. QLoRA was slower than both and used more memory than LoRA, not less.

Single run: one run per setting, one seed. First published 2026-10-28.

The full write-up, with every command

Hardware

GPU: One L40S 48 GB (44.4 GiB usable)

Software

Model: Qwen/Qwen2.5-1.5B-Instruct (1.54 billion parameters)
Libraries: transformers 4.57.1, peft 0.17.1, Python 3.12
Methods: Full: fp32 master weights, bf16 autocast. LoRA: bf16 base, rank 16, alpha 32, all attention and MLP projections (18.5 M trainable, 1.2%). QLoRA: the same LoRA on a 4-bit NF4 base

Method

Data: yahma/alpaca-cleaned: 4,000 examples for training (753,486 tokens), the next 300 held out, at most 512 tokens per example
Training: One epoch, 250 optimizer steps of 16 examples, cosine schedule, AdamW, no gradient checkpointing; learning rate 1e-5 (full) and 2e-4 (LoRA, QLoRA)
Measured: Training wall time without model load and evaluation, tokens per second, peak memory from torch.cuda.max_memory_allocated, held-out loss before and after

Price per hour

L40S$1.14 on 2026-10-28, from $0.54/hr per GPU on a 2-GPU machine today

Peak memory at micro-batch 4Peak memory

Measured on 2026-10-28, from the table below.

Memory, speed, held-out loss and the GPU cost of training

Method	Micro-batch x accumulation	Trainable	Train time	Tokens/s	Peak memory	Held-out loss	Training cost, run day	Training cost, today
full	4 x 4	1,543.7 M	210 s	3,595	36.9 GiB	1.103 to 1.065	$0.066	$0.031
LoRA	4 x 4	18.5 M	152 s	4,970	14.5 GiB	1.102 to 1.058	$0.048	$0.023
QLoRA	4 x 4	18.5 M	215 s	3,507	16.5 GiB	1.144 to 1.090	$0.068	$0.032
full	8 x 2	1,543.7 M	out of memory	not measured	not measured	not measured	not measured	not measured
LoRA	8 x 2	18.5 M	161 s	4,680	25.8 GiB	1.102 to 1.056	$0.051	$0.024
QLoRA	8 x 2	18.5 M	237 s	3,184	30.7 GiB	1.144 to 1.089	$0.075	$0.036

Run day: the price on 2026-10-28, as the run paid it.

Today: the run-day figure times today's price per GPU-hour over the run-day price per GPU-hour. Today's price is the "from" price per GPU-hour shown on /pricing, which can be on a machine with more GPUs than the run used (the list of GPUs below names it). Prices read 2026-11-04 04:00 UTC.

What stood out

QLoRA's held-out loss before training was 1.144, against 1.102 for the bf16 model: quantizing the base cost almost as much loss as the fine-tune removed.
LoRA peaked at 14.5 GiB at micro-batch 4 and 25.8 GiB at micro-batch 8. Its train times (152 s and 161 s) come from separate machines, so they are not a speed comparison.
The full fine-tune ran out of memory in the backward pass with 39.6 GiB already allocated.

Keep in mind

Treat differences under about 10% as noise, and the loss differences between full and LoRA as a tie.
The learning rates (1e-5 for full, 2e-4 for LoRA) are common defaults, not tuned.
No gradient checkpointing, no fused or 8-bit optimizers, no FlashAttention 2 (PyTorch SDPA). Each would change the memory numbers.

Run 2026-10-26

PyTorch DDP on four PCIe RTX A6000s. Plain DDP and two fixes.

Plain DDP on four RTX A6000s ran the 124M model at 1.99x one GPU. bf16 gradient compression plus accumulating 4 micro-batches per all-reduce took it to 3.54x, and the 355M model to 3.66x.

Single run: one run per configuration. First published 2026-10-26.

The full write-up, with every command

Hardware

Machine: 4x RTX A6000 48 GB in one VM
Topology: Every GPU pair PHB (through the host bridge), no NVLink, PCIe gen 3 x16 per card

Software

Software: torch 2.11.0+cu128, NCCL as shipped in the wheel
Model: A GPT-2 style decoder in plain PyTorch: 124M parameters (12 layers, 768 wide) and 355M (24 layers, 1024 wide); random tokens, sequence 1024, micro-batch 8 per GPU, bf16 autocast, fused AdamW

Method

Metric: Tokens/s from the median step time over 40 steps after 10 warm-up steps; weak scaling, each GPU always gets its own micro-batch of 8
Fixes: bf16 gradient compression (bf16_compress_hook) and accumulating 4 micro-batches per all-reduce (ddp.no_sync())

Price per hour

4x RTX A6000$2.64 on 2026-10-26 for the machine, from $0.57/hr per GPU today

Speed-up over one GPU, 124M model124M

Measured on 2026-10-26, from the table below.

Training throughput and cost per 1B tokens

Model	Variant	Tokens/s	Speed-up vs 1 GPU, same variant	$ per 1B tokens, run day	$ per 1B tokens, today
124M	1 GPU	64,912	1.00x	2.82 (one GPU's share, $0.66/h)	2.44
124M	4 GPUs, plain DDP	129,032	1.99x	5.68	4.91
124M	4 GPUs, bf16 gradients	172,144	2.65x	4.26	3.68
124M	4 GPUs, accumulate 4	211,834	3.20x	3.46	2.99
124M	4 GPUs, both	234,175	3.54x	3.13	2.70
355M	1 GPU	27,109	1.00x	6.76	5.84
355M	4 GPUs, plain DDP	51,711	1.91x	14.18	12.25
355M	4 GPUs, bf16 gradients	73,653	2.72x	9.96	8.60
355M	4 GPUs, accumulate 4	86,507	3.10x	8.48	7.32
355M	4 GPUs, both	99,152	3.66x	7.40	6.39

1-GPU tokens/s with accumulation 4: 66,189 and 27,893, so the "both" rows are 88% and 89% scaling efficiency against the same recipe on one GPU.

Run day: the price on 2026-10-26, as the run paid it.

What stood out

Even with both fixes, a token cost about 10% more on four GPUs than on one: rent the 4x machine for a shorter run, not to save money.

Keep in mind

Random tokens and a synthetic model: this measures throughput, not convergence. bf16 gradient compression slightly changes numerics.
Machines with NVLink behave differently; this is a PCIe-only result.
Accumulation changes the effective batch size (32 per GPU here), so retune the learning rate if you adopt it.

Run 2026-10-29

torch.compile vs eager for training. A 468M model, four GPUs.

With SDPA attention, torch.compile changed the step time by between 9% slower (RTX 5090) and 3% faster (L40S). With naive attention it made the H100 step 2.6x faster and fit the model in 39 GB instead of 56 GB, and on the 48 GB cards it was the only way the run fit at all.

Single run: one run per setup. First published 2026-10-29.

The full write-up, with every command

Hardware

GPUs: RTX 5090 32 GB, L40S 48 GB, RTX A6000 48 GB and H100 PCIe 80 GB, one per machine

Software

Libraries: transformers 4.57.1, Python 3.12, the same on all four machines; torch.compile in its default mode
Model: LlamaForCausalLM from a config, random init: hidden 1024, 24 layers, 16 heads, FFN 4096, vocab 32,000, 468.2M parameters, weights in bf16

Method

Step: Batch 4 x 2,048 tokens (8,192 tokens per step) of random token ids, fused AdamW (lr 1e-4)
Setups: Eager or torch.compile(model), times attn_implementation sdpa or eager (naive attention), each in its own process
Measured: First step time (includes compilation), 4 more warm-up steps, then the mean of 20 timed steps, tokens per second and peak memory

Price per hour

RTX 5090$1.15 on 2026-10-29, from $0.84/hr per GPU today
L40S$0.54 on 2026-10-29, from $0.54/hr per GPU on a 2-GPU machine today
RTX A6000$0.65 on 2026-10-29, from $0.57/hr per GPU today
H100$3.86 on 2026-10-29, from $3.87/hr per GPU on a 4-GPU machine today

Step time, memory and training tokens per dollar

GPU	Setup	Step (ms)	Tokens/s	Peak memory	First step (s)	M tokens per $, run day	M tokens per $, today
RTX 5090	eager + sdpa attention	196	41,870	17.3 GB	0.6	131	179
RTX 5090	compile + sdpa attention	215	38,040	13.3 GB	21.6	119	163
RTX 5090	eager + eager attention	OOM	not measured	not measured	not measured	not measured	not measured
RTX 5090	compile + eager attention	OOM	not measured	not measured	not measured	not measured	not measured
L40S	eager + sdpa attention	244	33,581	17.3 GB	0.6	224	224
L40S	compile + sdpa attention	236	34,691	13.3 GB	34.2	231	231
L40S	eager + eager attention	OOM	not measured	not measured	not measured	not measured	not measured
L40S	compile + eager attention	391	20,973	39.1 GB	38.9	140	140
RTX A6000	eager + sdpa attention	457	17,942	17.3 GB	0.7	99	113
RTX A6000	compile + sdpa attention	455	18,005	13.3 GB	39.2	100	114
RTX A6000	eager + eager attention	OOM	not measured	not measured	not measured	not measured	not measured
RTX A6000	compile + eager attention	546	15,017	39.1 GB	47.1	83	95
H100	eager + sdpa attention	152	54,022	17.4 GB	0.4	50	50
H100	compile + sdpa attention	160	51,055	13.4 GB	36.4	48	48
H100	eager + eager attention	435	18,813	56.0 GB	0.7	18	18
H100	compile + eager attention	169	48,469	39.1 GB	43.9	45	45

OOM means the run did not fit in the GPU's memory.

Run day: the price on 2026-10-29, as the run paid it.

Today: the run-day figure divided by today's price per GPU-hour over the run-day price per GPU-hour. Today's price is the "from" price per GPU-hour shown on /pricing, which can be on a machine with more GPUs than the run used (the list of GPUs below names it). Prices read 2026-11-04 04:00 UTC.

What stood out

On the H100, eager attention ran at 435 ms per step and 56 GB; compiled, 169 ms and 39 GB.
Eager mode with SDPA was faster than compiled naive attention on every GPU (H100 152 vs 169 ms, L40S 244 vs 391 ms) and used less than half the memory.
On the 32 GB RTX 5090 naive attention did not fit either way.

Keep in mind

One model size (468M), one batch shape, the default compile mode only. Larger models, longer sequences or max-autotune can change the result.
Pure bf16 weights and optimizer state, no mixed precision with fp32 master weights.
The machines differ in CPU and host; small differences may come from the host.

Costs

How the costs are worked out. Run day and today, side by side.

Run day

The cost the write-up printed: the hourly price the machine was launched at that day, against what it measured. Prices move daily, so each run-day figure carries its date. Every run was a single run.

Today

The run-day figure moved by today's price: the "from" price per GPU-hour shown on /pricing, over the run-day rate per GPU-hour. A GPU that is not in stock right now reads "Not in stock" rather than a stale price. Prices read 2026-11-04 04:00 UTC.

GPUs

The GPUs on this page. From price per GPU-hour, today.

L40Sfrom $0.54/hr per GPU on a 2-GPU machine
RTX A6000from $0.57/hr per GPU
RTX 5090from $0.84/hr per GPU
H100from $3.87/hr per GPU on a 4-GPU machine

Benchmarks

More benchmarks.

LLM inferenceCost per GPU.$0.12 per 1M output tokens. RTX A6000, vLLM, 64 concurrent requests, at the run-day price, run 2026-10-26, single run.A100 vs H100 vs H200One script, six GPUs.718.46 bf16 TFLOPS. H200, dense bf16 matmul at 8192, run 2026-10-25, single run.Speech to textAudio hours per dollar.315x real time. RTX 5090, faster-whisper large-v3-turbo, batch 16, run 2026-10-28, single run.Text embeddingsCost per passage.$0.032 per 1M passages. L40S, bge-small, 128-token passages, at the run-day price, run 2026-10-29, single run.Image generationImages per dollar.$0.91 per 1,000 images. RTX A6000, compiled, batch 1, at the run-day price, run 2026-10-27, single run.GPU startup timeTimed from the API call.186 s to the first CUDA op. L40S, from the create call to a 4096 x 4096 matmul, run 2026-10-26, single run.