Post-Training Quantization (PTQ)
Reduce model weight precision from FP16 to INT8 or INT4 after training with no retraining — trading a small accuracy loss for large memory reduction and inference speedup.
Intent & Description
🎯 Intent
Cut inference memory footprint and increase throughput by lowering numerical precision without touching the training pipeline.
📋 Context
A 7B parameter model at FP16 requires ~14GB of VRAM. INT8 halves that; INT4 quarters it — making models that were GPU-cluster-only deployable on a single consumer GPU. No retraining needed — this is a post-hoc transformation on any existing checkpoint.
💡 Solution
After training, quantize weights (and optionally activations) from FP16/BF16 to INT8 or INT4 using calibration data to determine per-layer scaling factors that minimize quantization error. Libraries: bitsandbytes (INT8/INT4), GPTQ (INT4 weight quantization), llama.cpp (GGUF). Run calibration on a representative dataset — random calibration data degrades quality.
Real-world Use Case
📌 TL;DR
Shrink the model after training — INT8 halves memory with minimal accuracy loss, INT4 quarters it with more. No retraining required; representative calibration data matters.
Advantages
- No retraining — applies to any existing checkpoint in minutes
- 2-4x memory reduction at INT8/INT4 with minimal quality regression at INT8
- Inference speedup on hardware with native INT8 support (most modern GPUs and NPUs)
Disadvantages
- Accuracy degrades — typically small at INT8, larger at INT4, varies significantly by model and task
- Some layers are more sensitive and may need to stay at higher precision (mixed-precision PTQ)
- Calibration data quality affects quantization grid — poor calibration data → worse accuracy