H100 NVL Specifications
Two PCIe boards bridged and sold as a pair, giving 188 GB across the unit. Built specifically so a 70B model fits without moving to SXM.
Buy it for
LLM inference in PCIe servers. Two bridged boards present 188 GB total with far more bandwidth than a standard H100 PCIe — NVIDIA built it specifically for serving large models without HGX.
Look elsewhere if
You have one free slot or a constrained power budget. It occupies a bridged pair at 400 W per board and cannot be split into two independent cards.
Will it fit your system?
Passively cooled — needs server airflow
This card has no fans. It relies on a chassis forcing air through its heatsink. In a normal desktop it will throttle within minutes and can overheat outright. It needs a rack server with high-static-pressure fans, or an aftermarket shroud and blower.
- Form factor
- PCIe dual-slot
- Cooling
- passive (no fans)
- Power connectors
- 1 × 16-pin (12VHPWR) per board
- Board power
- 400 W
- Suggested PSU
- 900 W or better
- Interface
- PCIe Gen 5
PSU figure assumes one card plus a mainstream CPU and drives, with headroom for transient spikes — modern cards can draw well above their rated TDP for milliseconds, which is what trips an undersized supply.
Supported formats & features
What the H100 NVL does and does not support in hardware.
Blackwell-generation feature — B200 has it
E4M3 and E5M2 — the standard format for modern inference
Wider exponent than FP16 — fewer overflow issues when training
Drop-in FP32 speedup with no code changes
Quantised inference supported
Not yet confirmed against vendor documentation.
Per-layer automatic FP8 precision management
600 GB/s GPU-to-GPU
Compute-only die — no ray tracing hardware
Corrects single-bit errors — expected for long training runs
What fits in memory
Largest model that fits on one H100 NVL, assuming ~20% of the 94 GB is reserved for KV cache and activations.
| Model | FP16 / BF16 | FP8 |
|---|---|---|
| Llama 8B | Fits | Fits |
| Llama 70B | 2× GPUs | Fits |
| Mixtral 8x22B | 4× GPUs | 2× GPUs |
| Llama 405B | 11× GPUs | 6× GPUs |
Weights only. Long contexts, large batch sizes and optimiser state during training all need substantially more — full fine-tuning typically needs 3–4× the weight memory.
Workload character
Bandwidth and compute are matched closely enough that neither dominates across mixed training and inference.
Cost to run
Electricity for one card at 100% duty, including cooling and PSU overhead (4,906 kWh/year).
At the US average rate, three years of continuous operation costs $1,766 — roughly 6% of the card's $30,000 launch price, before any cooling or hosting fees.
Where it sits in the line
Availability & market
Scarce and priced close to SXM H100. Buy it for the specific case where you need Hopper capacity but cannot take an HGX baseboard.
Memory
- Capacity
- 94 GB
- Type
- HBM3
- Bandwidth
- 3,900 GB/s
- Bus width
- 6,016-bit
- GB per TFLOPMemory relative to compute — higher favours large models
- 0.106
Compute
- FP64
- 30 TFLOPS
- FP32
- 60 TFLOPS
- TF32 dense
- 445 TFLOPS
- FP16 dense
- 890 TFLOPS
- FP16 sparseRequires structured sparsity; rarely reached in practice
- 1,780 TFLOPS
- FP8 dense
- 1,780 TFLOPS
- INT8 dense
- 1,780 TOPS
Silicon
- Codename
- GH100
- Process node
- TSMC 4N
- Transistors
- 80B
- Die size
- 814 mm²
- CUDA cores
- 16,896
- Tensor cores
- 528
- SM count
- 132
- Boost clock
- 1,785 MHz
Platform & power
- TDP
- 400 W
- TFLOPS per wattDense FP16 ÷ TDP — drives your power bill at scale
- 2.23
- Form factor
- PCIe dual-slot
- Cooling
- passive
- Interconnect
- NVLink bridge
- Interconnect speed
- 600 GB/s
- PCIe generation
- Gen 5
- MIG
- Yes — up to 7 instances
- Launch
- 2023-03
- MSRP at launch
- $30,000
H100 NVL — common questions
Answered from the specifications above. Questions this card's data cannot answer with confidence are left out rather than guessed.
- How much VRAM does the H100 NVL have?
- The H100 NVL has 94 GB of HBM3 memory with 3,900 GB/s of bandwidth. In practice plan on roughly 75 GB for weights: KV cache, activations and fragmentation take the rest, and long contexts or large batches take considerably more.
- What size model can the H100 NVL run?
- On a single H100 NVL, about 37B parameters at FP16, or roughly 75B quantised to FP8. Anything larger has to be split across several GPUs. These are weight-only figures — a long context window can consume tens of gigabytes on its own.
- Can you put the H100 NVL in a desktop PC?
- Only with added cooling. The H100 NVL is a passively cooled card — it has no fans at all and depends on a rack chassis forcing air through its heatsink. Dropped into an ordinary desktop it will throttle within minutes and can overheat outright. It needs either a rack server with high-static-pressure fans or an aftermarket shroud and blower.
- How much power does the H100 NVL use?
- The H100 NVL is rated at 400 W board power. It is fed through 1 × 16-pin (12VHPWR) per board. For a single-card system, a 900 W supply or better leaves headroom for the CPU, drives and the transient spikes that trip undersized units.
- How much does it cost to run the H100 NVL?
- At the US commercial average of $0.12/kWh, running one H100 NVL continuously costs about $589 a year in electricity. That figure includes cooling and power-supply overhead at a factor of 1.4 over board power, and over three years comes to roughly 6% of what the card cost at launch.
- Does the H100 NVL support MIG?
- Yes. The H100 NVL can be partitioned into up to 7 Multi-Instance GPU slices, each with isolated memory and compute. This is what makes it practical to share one card across several users or tenants without them affecting each other's latency.
- Does the H100 NVL support FP8?
- Yes. The H100 NVL has hardware FP8 tensor support, so FP8 weights and activations run at full tensor-core rate — 1,780 dense TFLOPS. This is the format most current inference servers default to for large models.
- Does the H100 NVL support FP4?
- No. The H100 NVL predates hardware FP4. You can run 4-bit quantised weights on it and the memory saving is real, but they are dequantised before compute, so there is no arithmetic speedup.
- Does the H100 NVL support INT8?
- Yes. The H100 NVL supports INT8 acceleration at 1,780 dense TOPS. INT8 remains the most widely supported quantised inference format across frameworks.
- Does the H100 NVL support BF16?
- Yes. The H100 NVL supports bfloat16. Its wider exponent range makes training far less prone to the overflow that plagues FP16, which is why most current training recipes assume it.
- Does the H100 NVL have NVLink?
- Yes. The H100 NVL supports NVLink (NVLink bridge) at 600 GB/s, so GPU-to-GPU traffic bypasses PCIe. This matters for tensor-parallel serving, where every token crosses the link.
- Does the H100 NVL have RT cores?
- No. The H100 NVL is a compute-only part with no ray-tracing hardware. For rendering or virtual workstations, look at the L40S or an RTX workstation card instead.
- Does the H100 NVL have ECC memory?
- Yes. The H100 NVL has ECC, so single-bit memory errors are corrected rather than silently corrupting a long run.
- Is the H100 NVL worth buying?
- Buy it for: LLM inference in PCIe servers. Two bridged boards present 188 GB total with far more bandwidth than a standard H100 PCIe — NVIDIA built it specifically for serving large models without HGX. Look elsewhere if: You have one free slot or a constrained power budget. It occupies a bridged pair at 400 W per board and cannot be split into two independent cards.