Skip to content
MUSEBOARD

GPU / A100-40GB

NVIDIA A100 40GB

Established training and inference clusters

Plan on this GPU Compare All GPUs
Spec sheet
NVIDIA
Vendor
NVIDIA
Architecture
Ampere
Memory
40 GB HBM2
Bandwidth
1.55 TB/s
Power
400 W
Class
Datacenter
Form factor
SXM4
Interconnect
NVLink 3 · 600 GB/s
Released
2020
Workloads
Inference · Fine-tuning · Training
Model fit / Inference
8K ctx · batch 1 · ≥10% headroom
GPUs required per model and precision for NVIDIA A100 40GB
ModelFP16INT8INT4
Llama 3.2 1B1×3.0 GB1×1.6 GB1×0.95 GB
Llama 3.2 3B1×8.0 GB1×4.5 GB1×2.7 GB
Llama 3.1 8B1×18.7 GB1×9.9 GB1×5.5 GB
Llama 3.1 70B8×158.0 GB4×80.3 GB2×41.5 GB
Llama 3.3 70B8×158.0 GB4×80.3 GB2×41.5 GB
Llama 3.1 405B32×897.2 GB16×450.7 GB8×227.5 GB
Qwen2.5 1.5B1×3.6 GB1×1.9 GB1×1.1 GB
Qwen2.5 3B1×7.1 GB1×3.7 GB1×2.0 GB
Qwen2.5 7B1×17.2 GB1×8.9 GB1×4.7 GB
Qwen2.5 14B1×34.0 GB1×17.8 GB1×9.7 GB
Qwen2.5 32B4×73.6 GB2×37.9 GB1×20.0 GB
Qwen2.5 72B8×162.6 GB4×82.7 GB2×42.7 GB
Mistral 7B1×17.0 GB1×9.0 GB1×5.1 GB
Mistral NeMo 12B1×28.2 GB1×14.8 GB1×8.1 GB
Mistral Small 24B2×53.3 GB1×27.3 GB1×14.3 GB
Mixtral 8x7B4×103.8 GB2×52.4 GB1×26.8 GB
Mixtral 8x22B16×312.1 GB8×157.0 GB4×79.4 GB
Gemma 2 9B1×23.1 GB1×13.0 GB1×7.9 GB
Gemma 2 27B2×62.9 GB1×33.0 GB1×18.0 GB
Phi-3 Mini 3.8B1×11.6 GB1×7.4 GB1×5.3 GB
Phi-3 Medium 14B1×32.5 GB1×17.1 GB1×9.4 GB
DeepSeek R1 Distill Qwen 32B4×74.3 GB2×38.2 GB1×20.2 GB
DeepSeek R1 Distill Llama 70B8×158.0 GB4×80.3 GB2×41.5 GB
DeepSeek V3 671B48×1,477 GB24×738.7 GB16×369.6 GB