Weight-Only Quantization
Store weights at INT4 but keep activations in FP16 — getting most of the memory win with a fraction of the accuracy cost of quantizing both.
Intent & Description
🎯 Intent
Model weights dominate memory; activations dominate compute precision sensitivity. Quantizing only weights gets most of the memory reduction while keeping compute in FP16.
📋 Context
Quantizing both weights and activations to INT4 is aggressive and hurts accuracy significantly. But most of a large model’s memory is weights, not activations. Dequantizing weights to FP16 just-in-time for each matrix multiplication keeps arithmetic in FP16 while storage stays at INT4.
💡 Solution
Store weight matrices in INT4 (or INT3). At each layer’s forward pass, dequantize the weight matrix from INT4 to FP16, perform the matmul in FP16, then discard the dequantized copy. Activations remain in FP16 throughout the computation. GPTQ, AWQ, and GGUF all use this approach. Calibration determines the optimal quantization grid per weight matrix.
Real-world Use Case
📌 TL;DR
Store weights at INT4, dequantize to FP16 for compute. Gets most of the memory win with a fraction of the accuracy cost of full INT4 quantization.
Advantages
- Approximately 4x memory reduction over FP16 with minimal accuracy loss on most architectures
- Arithmetic remains in FP16 — avoids INT4 matmul precision issues entirely
- Widely supported — GPTQ, AWQ, GGUF are mature, well-maintained ecosystems
Disadvantages
- Dequantization overhead on every forward pass reduces raw throughput vs. native INT4 compute
- Slower than unquantized FP16 on batch inference where memory is not the bottleneck
- Per-matrix calibration required for accurate quantization grid selection