H200 SXM Specifications
Same GH100 compute as H100 SXM. The upgrade is memory: 141 GB HBM3e at 4.8 TB/s, which is what helps inference on large models.
Buy it for
Inference on large models. 141 GB at 4.8 TB/s means a 70B model fits at FP16 on a single card.
Look elsewhere if
Your workload is compute-bound rather than memory-bound — compute is identical to H100 SXM, so you would be paying purely for memory.
Will it fit your system?
This is not a card you can install
SXM5 is a socketed module that mounts to an OEM baseboard, not a PCIe slot. You buy it inside a complete system — it cannot go into an existing desktop or a standard server under any circumstances.
- Form factor
- SXM5
- Cooling
- passive
- Power connectors
- SXM5 socket — no cables; requires an OEM baseboard
- Board power
- 700 W
- Interface
- PCIe Gen 5
Supported formats & features
What the H200 SXM does and does not support in hardware.
Blackwell-generation feature — B200 has it
E4M3 and E5M2 — the standard format for modern inference
Wider exponent than FP16 — fewer overflow issues when training
Drop-in FP32 speedup with no code changes
Quantised inference supported
Not yet confirmed against vendor documentation.
Per-layer automatic FP8 precision management
900 GB/s GPU-to-GPU
Compute-only die — no ray tracing hardware
Corrects single-bit errors — expected for long training runs
What fits in memory
Largest model that fits on one H200 SXM, assuming ~20% of the 141 GB is reserved for KV cache and activations.
| Model | FP16 / BF16 | FP8 |
|---|---|---|
| Llama 8B | Fits | Fits |
| Llama 70B | 2× GPUs | Fits |
| Mixtral 8x22B | 3× GPUs | 2× GPUs |
| Llama 405B | 8× GPUs | 4× GPUs |
Weights only. Long contexts, large batch sizes and optimiser state during training all need substantially more — full fine-tuning typically needs 3–4× the weight memory.
Workload character
Bandwidth and compute are matched closely enough that neither dominates across mixed training and inference.
Cost to run
Electricity for one card at 100% duty, including cooling and PSU overhead (8,585 kWh/year).
At the US average rate, three years of continuous operation costs $3,091 — roughly 10% of the card's $32,000 launch price, before any cooling or hosting fees.
Where it sits in the line
A newer part existing does not make this one a bad buy — it makes it cheaper. What matters is price per unit of the throughput you actually need.
Availability & market
Supply improved through 2025. The default choice for inference fleets that outgrew 80 GB.
Memory
- Capacity
- 141 GB
- Type
- HBM3e
- Bandwidth
- 4,800 GB/s
- Bus width
- 6,144-bit
- GB per TFLOPMemory relative to compute — higher favours large models
- 0.142
Compute
- FP64
- 34 TFLOPS
- FP32
- 67 TFLOPS
- TF32 dense
- 494.5 TFLOPS
- FP16 dense
- 989.5 TFLOPS
- FP16 sparseRequires structured sparsity; rarely reached in practice
- 1,979 TFLOPS
- FP8 dense
- 1,979 TFLOPS
- INT8 dense
- 1,979 TOPS
Silicon
- Codename
- GH100
- Process node
- TSMC 4N
- Transistors
- 80B
- Die size
- 814 mm²
- CUDA cores
- 16,896
- Tensor cores
- 528
- SM count
- 132
- Boost clock
- 1,980 MHz
Platform & power
- TDP
- 700 W
- TFLOPS per wattDense FP16 ÷ TDP — drives your power bill at scale
- 1.41
- Form factor
- SXM5
- Cooling
- passive
- Interconnect
- NVLink 4.0
- Interconnect speed
- 900 GB/s
- PCIe generation
- Gen 5
- MIG
- Yes — up to 7 instances
- Launch
- 2024-Q2
- MSRP at launch
- $32,000
H200 SXM — common questions
Answered from the specifications above. Questions this card's data cannot answer with confidence are left out rather than guessed.
- How much VRAM does the H200 SXM have?
- The H200 SXM has 141 GB of HBM3e memory with 4,800 GB/s of bandwidth. In practice plan on roughly 112 GB for weights: KV cache, activations and fragmentation take the rest, and long contexts or large batches take considerably more.
- What size model can the H200 SXM run?
- On a single H200 SXM, about 56B parameters at FP16, or roughly 112B quantised to FP8. Anything larger has to be split across several GPUs. These are weight-only figures — a long context window can consume tens of gigabytes on its own.
- Can you put the H200 SXM in a desktop PC?
- No. The H200 SXM is a SXM5 module, not a PCIe card. It bolts to an OEM baseboard and is only sold inside a complete system, so there is no way to install one in a desktop or a standard server.
- How much power does the H200 SXM use?
- The H200 SXM is rated at 700 W board power. It draws that from the baseboard rather than from PSU cables, so there is nothing to wire up yourself.
- How much does it cost to run the H200 SXM?
- At the US commercial average of $0.12/kWh, running one H200 SXM continuously costs about $1,030 a year in electricity. That figure includes cooling and power-supply overhead at a factor of 1.4 over board power, and over three years comes to roughly 10% of what the card cost at launch.
- Does the H200 SXM support MIG?
- Yes. The H200 SXM can be partitioned into up to 7 Multi-Instance GPU slices, each with isolated memory and compute. This is what makes it practical to share one card across several users or tenants without them affecting each other's latency.
- Does the H200 SXM support FP8?
- Yes. The H200 SXM has hardware FP8 tensor support, so FP8 weights and activations run at full tensor-core rate — 1,979 dense TFLOPS. This is the format most current inference servers default to for large models.
- Does the H200 SXM support FP4?
- No. The H200 SXM predates hardware FP4. You can run 4-bit quantised weights on it and the memory saving is real, but they are dequantised before compute, so there is no arithmetic speedup.
- Does the H200 SXM support INT8?
- Yes. The H200 SXM supports INT8 acceleration at 1,979 dense TOPS. INT8 remains the most widely supported quantised inference format across frameworks.
- Does the H200 SXM support BF16?
- Yes. The H200 SXM supports bfloat16. Its wider exponent range makes training far less prone to the overflow that plagues FP16, which is why most current training recipes assume it.
- Does the H200 SXM have NVLink?
- Yes. The H200 SXM supports NVLink (NVLink 4.0) at 900 GB/s, so GPU-to-GPU traffic bypasses PCIe. This matters for tensor-parallel serving, where every token crosses the link.
- Does the H200 SXM have RT cores?
- No. The H200 SXM is a compute-only part with no ray-tracing hardware. For rendering or virtual workstations, look at the L40S or an RTX workstation card instead.
- Does the H200 SXM have ECC memory?
- Yes. The H200 SXM has ECC, so single-bit memory errors are corrected rather than silently corrupting a long run.
- What replaced the H200 SXM?
- The B200 (2025) is its successor, with 192 GB of HBM3e at 8,000 GB/s. It is about 2.3x faster in dense FP16. On the used market that gap is what sets a fair price for the H200 SXM.
- Is the H200 SXM worth buying?
- Buy it for: Inference on large models. 141 GB at 4.8 TB/s means a 70B model fits at FP16 on a single card. Look elsewhere if: Your workload is compute-bound rather than memory-bound — compute is identical to H100 SXM, so you would be paying purely for memory.