GPU comparison
H100 vs A40.
The H100 is ahead on memory (80 GB vs 48 GB) and speed (up to 6.6x the FP16 throughput and up to 4.8x the memory bandwidth). The A40 makes sense when your model fits in 48 GB and its hourly price is lower.
Rent either on Runcrate
- H100from $3.15/hr
$0.039 per GB/hrper TFLOPS: varies by card
- A40Cheaperfrom $2.66/hr
$0.055 per GB/hr$1.78 per 100 TFLOPS/hr
Live hourly price, then per GB of memory and per 100 FP16 TFLOPS.
Which is better for
Which to pick for each job. Memory fit and throughput, from the datasheets.
Hold a 70B model in 8-bit weights
70 GB of 8-bit weights
H100: 1 GPU, 10 GB spare
A40: 2 GPUs (96 GB)
H100. Fits on one GPU.
Serve an 8B model in 16-bit weights
16 GB of 16-bit weights
H100: 1 GPU, 64 GB spare
A40: 1 GPU, 32 GB spare
H100. Same GPU count, up to 4.8x the memory bandwidth.
Fine-tune a 70B model with QLoRA
About 46 GB with 4-bit weights
H100: 1 GPU, 34 GB spare
A40: 1 GPU, 2 GB spare
H100. Same GPU count, up to 6.6x the FP16 throughput.
Train on a full node
FP16 tensor throughput per node
H100: 8x, up to 7,912 TFLOPS
A40: 4x, 599 TFLOPS
H100. 8 GPUs per node, not 4.
Throughput per watt
FP16 TFLOPS per 100 W of TDP
H100: Varies by card
A40: 50 TFLOPS per 100 W
Either. Varies by card.
Compare another pair
Hopper
NVIDIA H100
- Memory
- 80 GB HBM3
- FP16 tensor
- up to 989 TFLOPS
- Bandwidth
- up to 3.4 TB/s
Ampere
NVIDIA A40
- Memory
- 48 GB GDDR6
- FP16 tensor
- 149.7 TFLOPS
- Bandwidth
- 696 GB/s
At a glance
Spec by spec. The longer bar is the better number.
Full specs
The complete spec sheet.
| Spec | NVIDIA H100 | NVIDIA A40 |
|---|---|---|
| Architecture | Hopper | Ampere |
| Memory | 80 GB | 48 GB |
| Memory type | HBM3 | GDDR6 |
| Memory bandwidth | up to 3.4 TB/s | 696 GB/s |
| FP32 | up to 67 TFLOPS | 37.4 TFLOPS |
| FP16 tensor | up to 989 TFLOPS | 149.7 TFLOPS |
| INT8 | up to 1979 TOPS | 299 TOPS |
| TDP | up to 700 W | 300 W |
| GPUs per node | 8 | 4 |
| Hourly price (live) | from $3.15/hr | from $2.66/hr |
Questions
H100 or A40. What buyers ask first.
Other comparisons