Category
quantization
3 articles in this category
Practical LLM Quantization in Colab: A Hugging Face Walkthrough
TLDR: This is a practical, notebook-style quantization guide for Google Colab and Hugging Face. You will quantize real models, run inference, compare memory/latency, and learn when to use 4-bit NF4 vs safer INT8 paths. š What You Will Build in Thi...
GPTQ vs AWQ vs NF4: Choosing the Right LLM Quantization Pipeline
TLDR: GPTQ, AWQ, and NF4 all shrink LLMs, but they optimize different constraints. GPTQ focuses on post-training reconstruction error, AWQ protects salient weights for better quality at low bits, and NF4 offers practical 4-bit compression through bit...
LLM Model Quantization: Why, When, and How to Deploy Smaller, Faster Models
TLDR: Quantization converts high-precision model weights and activations (FP16/FP32) into lower-precision formats (INT8 or INT4) so LLMs run with less memory, lower latency, and lower cost. The key is choosing the right quantization method for your a...