Skip to content

GPU comparison

H100 vs A40.

The H100 is ahead on memory (80 GB vs 48 GB) and speed (up to 6.6x the FP16 throughput and up to 4.8x the memory bandwidth). The A40 makes sense when your model fits in 48 GB and its hourly price is lower.

Rent either on Runcrate

Live hourly price, then per GB of memory and per 100 FP16 TFLOPS.

Which is better for

Which to pick for each job. Memory fit and throughput, from the datasheets.

  • Hold a 70B model in 8-bit weights

    70 GB of 8-bit weights

    H100: 1 GPU, 10 GB spare

    A40: 2 GPUs (96 GB)

    H100. Fits on one GPU.

  • Serve an 8B model in 16-bit weights

    16 GB of 16-bit weights

    H100: 1 GPU, 64 GB spare

    A40: 1 GPU, 32 GB spare

    H100. Same GPU count, up to 4.8x the memory bandwidth.

  • Fine-tune a 70B model with QLoRA

    About 46 GB with 4-bit weights

    H100: 1 GPU, 34 GB spare

    A40: 1 GPU, 2 GB spare

    H100. Same GPU count, up to 6.6x the FP16 throughput.

  • Train on a full node

    FP16 tensor throughput per node

    H100: 8x, up to 7,912 TFLOPS

    A40: 4x, 599 TFLOPS

    H100. 8 GPUs per node, not 4.

  • Throughput per watt

    FP16 TFLOPS per 100 W of TDP

    H100: Varies by card

    A40: 50 TFLOPS per 100 W

    Either. Varies by card.

Compare another pair

vs

Hopper

NVIDIA H100

from$3.15/hr
Available now · 3 regions
Memory
80 GB HBM3
FP16 tensor
up to 989 TFLOPS
Bandwidth
up to 3.4 TB/s
Full pricing details

Ampere

NVIDIA A40

from$2.66/hrCheaper
Available now · 2 regions
Memory
48 GB GDDR6
FP16 tensor
149.7 TFLOPS
Bandwidth
696 GB/s
Full pricing details

At a glance

Spec by spec. The longer bar is the better number.

Memory

NVIDIA H10080 GB
NVIDIA A4048 GB

Memory bandwidth

NVIDIA H100up to 3.4 TB/s
NVIDIA A40696 GB/s

FP16 tensor

NVIDIA H100up to 989 TFLOPS
NVIDIA A40149.7 TFLOPS

Hourly price, from (live)

NVIDIA H100$3.15/hr
NVIDIA A40$2.66/hr

Full specs

The complete spec sheet.

SpecNVIDIA H100NVIDIA A40
ArchitectureHopperAmpere
Memory80 GB48 GB
Memory typeHBM3GDDR6
Memory bandwidthup to 3.4 TB/s696 GB/s
FP32up to 67 TFLOPS37.4 TFLOPS
FP16 tensorup to 989 TFLOPS149.7 TFLOPS
INT8up to 1979 TOPS299 TOPS
TDPup to 700 W300 W
GPUs per node84
Hourly price (live)from $3.15/hrfrom $2.66/hr

Questions

H100 or A40. What buyers ask first.

Other comparisons

More pairs with the H100 or the A40.