Models / Catalog
MEMORY BEFORE HARDWARE.
24 model presets with published architecture details — used to estimate weights, KV cache and activation memory.
Weights only
≈140.0GB
70B × 2 B = 140.0 GB
+10% runtime → 154.0 GB
Memory footprint
1 segment = 4 GB
GPU fit
3/15 fit on one GPU · ≥10% headroom
- L424 GB8×
- A1024 GB8×
- RTX 309024 GB8×
- RTX 409024 GB8×
- RTX 509032 GB8×
- A100 40GB40 GB8×
- L40S48 GB4×
- RTX 6000 Ada48 GB4×
- A100 80GB80 GB4×
- H100 SXM80 GB4×
- H100 NVL94 GB2×
- H200141 GB2×
- MI300X192 GB1× FIT
- B200192 GB1× FIT
- MI325X256 GB1× FIT
M1Compute matrix
MODEL × HARDWARE
Estimated inference VRAM (weights + KV cache + overhead) at 8K context, batch 1.
Llama 3.1 8B
8.03B- FP16
- 18.7 GB
- INT8
- 9.9 GB
- INT4
- 5.5 GB
GPU / 1 × RTX 4090
Workbench →Llama 3.1 70B
70.6B- FP16
- 158.0 GB
- INT8
- 80.3 GB
- INT4
- 41.5 GB
GPU / 1 × B200
Workbench →Qwen2.5 14B
14.7B- FP16
- 34.0 GB
- INT8
- 17.8 GB
- INT4
- 9.7 GB
GPU / 1 × A100 40GB
Workbench →Qwen2.5 32B
32.5B- FP16
- 73.6 GB
- INT8
- 37.9 GB
- INT4
- 20.0 GB
GPU / 1 × H100 NVL
Workbench →Qwen2.5 72B
72.7B- FP16
- 162.6 GB
- INT8
- 82.7 GB
- INT4
- 42.7 GB
GPU / 1 × B200
Workbench →Mistral 7B
7.25B- FP16
- 17.0 GB
- INT8
- 9.0 GB
- INT4
- 5.1 GB
GPU / 1 × RTX 4090
Workbench →Mixtral 8x7B
46.7B- FP16
- 103.8 GB
- INT8
- 52.4 GB
- INT4
- 26.8 GB
GPU / 1 × H200
Workbench →DeepSeek V3 671B
671B- FP16
- 1,477 GB
- INT8
- 738.7 GB
- INT4
- 369.6 GB
GPU / 8 × MI325X
Workbench →Custom
- FP16
- 45.8 GB
- INT8
- 23.8 GB
- INT4
- 12.8 GB
GPU / 1 × H100 SXM
Workbench →
M2Catalog
MODEL PRESETS
Meta · Dense
Llama 3.2 1B
- PARAMS
- 1.24B
- FP16
- 2.5 GB
- CTX
- 128K
- LAYERS
- 16
- KV / TOKEN (16-BIT)
- 32 KB
Meta · Dense
Llama 3.2 3B
- PARAMS
- 3.21B
- FP16
- 6.4 GB
- CTX
- 128K
- LAYERS
- 28
- KV / TOKEN (16-BIT)
- 112 KB
Meta · Dense
Llama 3.1 8B
- PARAMS
- 8.03B
- FP16
- 16.1 GB
- CTX
- 128K
- LAYERS
- 32
- KV / TOKEN (16-BIT)
- 128 KB
Meta · Dense
Llama 3.1 70B
- PARAMS
- 70.6B
- FP16
- 141.2 GB
- CTX
- 128K
- LAYERS
- 80
- KV / TOKEN (16-BIT)
- 320 KB
Meta · Dense
Llama 3.3 70B
- PARAMS
- 70.6B
- FP16
- 141.2 GB
- CTX
- 128K
- LAYERS
- 80
- KV / TOKEN (16-BIT)
- 320 KB
Meta · Dense
Llama 3.1 405B
- PARAMS
- 406B
- FP16
- 811.8 GB
- CTX
- 128K
- LAYERS
- 126
- KV / TOKEN (16-BIT)
- 504 KB
Alibaba · Dense
Qwen2.5 1.5B
- PARAMS
- 1.54B
- FP16
- 3.1 GB
- CTX
- 32K
- LAYERS
- 28
- KV / TOKEN (16-BIT)
- 28 KB
Alibaba · Dense
Qwen2.5 3B
- PARAMS
- 3.09B
- FP16
- 6.2 GB
- CTX
- 32K
- LAYERS
- 36
- KV / TOKEN (16-BIT)
- 36 KB
Alibaba · Dense
Qwen2.5 7B
- PARAMS
- 7.62B
- FP16
- 15.2 GB
- CTX
- 128K
- LAYERS
- 28
- KV / TOKEN (16-BIT)
- 56 KB
Alibaba · Dense
Qwen2.5 14B
- PARAMS
- 14.7B
- FP16
- 29.4 GB
- CTX
- 128K
- LAYERS
- 48
- KV / TOKEN (16-BIT)
- 192 KB
Alibaba · Dense
Qwen2.5 32B
- PARAMS
- 32.5B
- FP16
- 65.0 GB
- CTX
- 128K
- LAYERS
- 64
- KV / TOKEN (16-BIT)
- 256 KB
Alibaba · Dense
Qwen2.5 72B
- PARAMS
- 72.7B
- FP16
- 145.4 GB
- CTX
- 128K
- LAYERS
- 80
- KV / TOKEN (16-BIT)
- 320 KB
Mistral AI · Dense
Mistral 7B
- PARAMS
- 7.25B
- FP16
- 14.5 GB
- CTX
- 32K
- LAYERS
- 32
- KV / TOKEN (16-BIT)
- 128 KB
Mistral AI · Dense
Mistral NeMo 12B
- PARAMS
- 12.2B
- FP16
- 24.4 GB
- CTX
- 128K
- LAYERS
- 40
- KV / TOKEN (16-BIT)
- 160 KB
Mistral AI · Dense
Mistral Small 24B
- PARAMS
- 23.6B
- FP16
- 47.2 GB
- CTX
- 32K
- LAYERS
- 40
- KV / TOKEN (16-BIT)
- 160 KB
Mistral AI · MoE
Mixtral 8x7B
- PARAMS
- 46.7B / 12.9B
- FP16
- 93.4 GB
- CTX
- 32K
- LAYERS
- 32
- KV / TOKEN (16-BIT)
- 128 KB
Mistral AI · MoE
Mixtral 8x22B
- PARAMS
- 141B / 39B
- FP16
- 282.0 GB
- CTX
- 64K
- LAYERS
- 56
- KV / TOKEN (16-BIT)
- 224 KB
Google · Dense
Gemma 2 9B
- PARAMS
- 9.24B
- FP16
- 18.5 GB
- CTX
- 8K
- LAYERS
- 42
- KV / TOKEN (16-BIT)
- 336 KB
Google · Dense
Gemma 2 27B
- PARAMS
- 27.2B
- FP16
- 54.4 GB
- CTX
- 8K
- LAYERS
- 46
- KV / TOKEN (16-BIT)
- 368 KB
Microsoft · Dense
Phi-3 Mini 3.8B
- PARAMS
- 3.82B
- FP16
- 7.6 GB
- CTX
- 128K
- LAYERS
- 32
- KV / TOKEN (16-BIT)
- 384 KB
Microsoft · Dense
Phi-3 Medium 14B
- PARAMS
- 14B
- FP16
- 28.0 GB
- CTX
- 128K
- LAYERS
- 40
- KV / TOKEN (16-BIT)
- 200 KB
DeepSeek · Dense
DeepSeek R1 Distill Qwen 32B
- PARAMS
- 32.8B
- FP16
- 65.6 GB
- CTX
- 128K
- LAYERS
- 64
- KV / TOKEN (16-BIT)
- 256 KB
DeepSeek · Dense
DeepSeek R1 Distill Llama 70B
- PARAMS
- 70.6B
- FP16
- 141.2 GB
- CTX
- 128K
- LAYERS
- 80
- KV / TOKEN (16-BIT)
- 320 KB
DeepSeek · MoE
DeepSeek V3 671B
- PARAMS
- 671B / 37B
- FP16
- 1,342 GB
- CTX
- 128K
- LAYERS
- 61
- KV / TOKEN (16-BIT)
- 69 KB