Skip to content
MUSEBOARD

GPU / H100-NVL

NVIDIA H100 NVL 94GB

LLM inference in standard PCIe servers

Plan on this GPU Compare All GPUs
Spec sheet
NVIDIA
Vendor
NVIDIA
Architecture
Hopper
Memory
94 GB HBM3
Bandwidth
3.9 TB/s
Power
400 W
Class
Datacenter
Form factor
PCIe dual-slot
Interconnect
NVLink bridge · 600 GB/s
Released
2023
Workloads
Inference · Fine-tuning
Model fit / Inference
8K ctx · batch 1 · ≥10% headroom
GPUs required per model and precision for NVIDIA H100 NVL 94GB
ModelFP16INT8INT4
Llama 3.2 1B1×3.0 GB1×1.6 GB1×0.95 GB
Llama 3.2 3B1×8.0 GB1×4.5 GB1×2.7 GB
Llama 3.1 8B1×18.7 GB1×9.9 GB1×5.5 GB
Llama 3.1 70B2×158.0 GB1×80.3 GB1×41.5 GB
Llama 3.3 70B2×158.0 GB1×80.3 GB1×41.5 GB
Llama 3.1 405B16×897.2 GB8×450.7 GB4×227.5 GB
Qwen2.5 1.5B1×3.6 GB1×1.9 GB1×1.1 GB
Qwen2.5 3B1×7.1 GB1×3.7 GB1×2.0 GB
Qwen2.5 7B1×17.2 GB1×8.9 GB1×4.7 GB
Qwen2.5 14B1×34.0 GB1×17.8 GB1×9.7 GB
Qwen2.5 32B1×73.6 GB1×37.9 GB1×20.0 GB
Qwen2.5 72B2×162.6 GB1×82.7 GB1×42.7 GB
Mistral 7B1×17.0 GB1×9.0 GB1×5.1 GB
Mistral NeMo 12B1×28.2 GB1×14.8 GB1×8.1 GB
Mistral Small 24B1×53.3 GB1×27.3 GB1×14.3 GB
Mixtral 8x7B2×103.8 GB1×52.4 GB1×26.8 GB
Mixtral 8x22B4×312.1 GB2×157.0 GB1×79.4 GB
Gemma 2 9B1×23.1 GB1×13.0 GB1×7.9 GB
Gemma 2 27B1×62.9 GB1×33.0 GB1×18.0 GB
Phi-3 Mini 3.8B1×11.6 GB1×7.4 GB1×5.3 GB
Phi-3 Medium 14B1×32.5 GB1×17.1 GB1×9.4 GB
DeepSeek R1 Distill Qwen 32B1×74.3 GB1×38.2 GB1×20.2 GB
DeepSeek R1 Distill Llama 70B2×158.0 GB1×80.3 GB1×41.5 GB
DeepSeek V3 671B24×1,477 GB16×738.7 GB8×369.6 GB