H100 SXM Specifications
Flagship Hopper. HBM3 and 900 GB/s NVLink make it the default multi-node training part.
Buy it for
Multi-node training of large models, where NVLink at 900 GB/s and HBM3 bandwidth are the binding constraints.
Look elsewhere if
You need more than 80 GB on one card, or you are buying for inference only — H200 or L40S give more memory or more throughput per dollar respectively.
Will it fit your system?
This is not a card you can install
SXM5 is a socketed module that mounts to an OEM baseboard, not a PCIe slot. You buy it inside a complete system — it cannot go into an existing desktop or a standard server under any circumstances.
- Form factor
- SXM5
- Cooling
- passive
- Power connectors
- SXM5 socket — no cables; requires an OEM baseboard
- Board power
- 700 W
- Interface
- PCIe Gen 5
Supported formats & features
What the H100 SXM does and does not support in hardware.
Blackwell-generation feature — B200 has it
E4M3 and E5M2 — the standard format for modern inference
Wider exponent than FP16 — fewer overflow issues when training
Drop-in FP32 speedup with no code changes
Quantised inference supported
Not yet confirmed against vendor documentation.
Per-layer automatic FP8 precision management
900 GB/s GPU-to-GPU
Compute-only die — no ray tracing hardware
Corrects single-bit errors — expected for long training runs
What fits in memory
Largest model that fits on one H100 SXM, assuming ~20% of the 80 GB is reserved for KV cache and activations.
| Model | FP16 / BF16 | FP8 |
|---|---|---|
| Llama 8B | Fits | Fits |
| Llama 70B | 3× GPUs | 2× GPUs |
| Mixtral 8x22B | 5× GPUs | 3× GPUs |
| Llama 405B | 13× GPUs | 7× GPUs |
Weights only. Long contexts, large batch sizes and optimiser state during training all need substantially more — full fine-tuning typically needs 3–4× the weight memory.
Workload character
Bandwidth and compute are matched closely enough that neither dominates across mixed training and inference.
Cost per unit of compute
Breakeven counts hardware cost against rental only. Power, cooling, hosting and depreciation push the real crossover substantially later — see colocation pricing for what hosting actually costs.
Cost to run
Electricity for one card at 100% duty, including cooling and PSU overhead (8,585 kWh/year).
At the US average rate, three years of continuous operation costs $3,091 — roughly 10% of the card's $30,000 launch price, before any cooling or hosting fees.
Where it sits in the line
A newer part existing does not make this one a bad buy — it makes it cheaper. What matters is price per unit of the throughput you actually need.
Availability & market
Widely available through OEM systems and neoclouds. Rarely sold as a bare card — you are buying an HGX baseboard of four or eight.
Memory
- Capacity
- 80 GB
- Type
- HBM3
- Bandwidth
- 3,350 GB/s
- Bus width
- 5,120-bit
- GB per TFLOPMemory relative to compute — higher favours large models
- 0.081
Compute
- FP64
- 34 TFLOPS
- FP32
- 67 TFLOPS
- TF32 dense
- 494.5 TFLOPS
- FP16 dense
- 989.5 TFLOPS
- FP16 sparseRequires structured sparsity; rarely reached in practice
- 1,979 TFLOPS
- FP8 dense
- 1,979 TFLOPS
- INT8 dense
- 1,979 TOPS
Silicon
- Codename
- GH100
- Process node
- TSMC 4N
- Transistors
- 80B
- Die size
- 814 mm²
- CUDA cores
- 16,896
- Tensor cores
- 528
- SM count
- 132
- Boost clock
- 1,980 MHz
Platform & power
- TDP
- 700 W
- TFLOPS per wattDense FP16 ÷ TDP — drives your power bill at scale
- 1.41
- Form factor
- SXM5
- Cooling
- passive
- Interconnect
- NVLink 4.0
- Interconnect speed
- 900 GB/s
- PCIe generation
- Gen 5
- MIG
- Yes — up to 7 instances
- Launch
- 2022-09
- MSRP at launch
- $30,000
H100 SXM — common questions
Answered from the specifications above. Questions this card's data cannot answer with confidence are left out rather than guessed.
- How much VRAM does the H100 SXM have?
- The H100 SXM has 80 GB of HBM3 memory with 3,350 GB/s of bandwidth. In practice plan on roughly 64 GB for weights: KV cache, activations and fragmentation take the rest, and long contexts or large batches take considerably more.
- What size model can the H100 SXM run?
- On a single H100 SXM, about 32B parameters at FP16, or roughly 64B quantised to FP8. Anything larger has to be split across several GPUs. These are weight-only figures — a long context window can consume tens of gigabytes on its own.
- Can you put the H100 SXM in a desktop PC?
- No. The H100 SXM is a SXM5 module, not a PCIe card. It bolts to an OEM baseboard and is only sold inside a complete system, so there is no way to install one in a desktop or a standard server.
- How much power does the H100 SXM use?
- The H100 SXM is rated at 700 W board power. It draws that from the baseboard rather than from PSU cables, so there is nothing to wire up yourself.
- How much does it cost to run the H100 SXM?
- At the US commercial average of $0.12/kWh, running one H100 SXM continuously costs about $1,030 a year in electricity. That figure includes cooling and power-supply overhead at a factor of 1.4 over board power, and over three years comes to roughly 10% of what the card cost at launch.
- Does the H100 SXM support MIG?
- Yes. The H100 SXM can be partitioned into up to 7 Multi-Instance GPU slices, each with isolated memory and compute. This is what makes it practical to share one card across several users or tenants without them affecting each other's latency.
- Does the H100 SXM support FP8?
- Yes. The H100 SXM has hardware FP8 tensor support, so FP8 weights and activations run at full tensor-core rate — 1,979 dense TFLOPS. This is the format most current inference servers default to for large models.
- Does the H100 SXM support FP4?
- No. The H100 SXM predates hardware FP4. You can run 4-bit quantised weights on it and the memory saving is real, but they are dequantised before compute, so there is no arithmetic speedup.
- Does the H100 SXM support INT8?
- Yes. The H100 SXM supports INT8 acceleration at 1,979 dense TOPS. INT8 remains the most widely supported quantised inference format across frameworks.
- Does the H100 SXM support BF16?
- Yes. The H100 SXM supports bfloat16. Its wider exponent range makes training far less prone to the overflow that plagues FP16, which is why most current training recipes assume it.
- Does the H100 SXM have NVLink?
- Yes. The H100 SXM supports NVLink (NVLink 4.0) at 900 GB/s, so GPU-to-GPU traffic bypasses PCIe. This matters for tensor-parallel serving, where every token crosses the link.
- Does the H100 SXM have RT cores?
- No. The H100 SXM is a compute-only part with no ray-tracing hardware. For rendering or virtual workstations, look at the L40S or an RTX workstation card instead.
- Does the H100 SXM have ECC memory?
- Yes. The H100 SXM has ECC, so single-bit memory errors are corrected rather than silently corrupting a long run.
- What replaced the H100 SXM?
- The H200 SXM (2024) is its successor, with 141 GB of HBM3e at 4,800 GB/s. At the same dense FP16 throughput, what changed is memory rather than compute. On the used market that gap is what sets a fair price for the H100 SXM.
- Is the H100 SXM worth buying?
- Buy it for: Multi-node training of large models, where NVLink at 900 GB/s and HBM3 bandwidth are the binding constraints. Look elsewhere if: You need more than 80 GB on one card, or you are buying for inference only — H200 or L40S give more memory or more throughput per dollar respectively.