LLM Quantization — Run Models Locally

Quantization compresses a neural network by storing its weights in fewer bits. Instead of 16-bit floating-point numbers (FP16/BF16), you keep weights in 8-bit integers (INT8), 4-bit integers (INT4), or even 2-bit. The result: a model that is 2x to 4x smaller, often 2x faster, with surprisingly little quality loss.

Quantization is the reason you can run Llama 3.1 70B on a single 40GB GPU, or fit a respectable LLM on a MacBook. Without it, only data centers could host modern LLMs.

The Definition

A neural network is mostly a stack of matrices. Each weight is normally a 16-bit floating-point number. Quantization picks a smaller integer type (8-bit, 4-bit, etc.) and maps the original weights into that smaller range using a scale and sometimes a zero point: x_quant = round(x / scale). At inference, you dequantize back to a higher-precision number, multiply, and continue.

A 70B-parameter model in FP16 weighs ~140GB. In 4-bit it weighs ~40GB. The math:

• FP16 → 2 bytes/param → 70B × 2 = 140 GB
• INT8 → 1 byte/param → 70 GB
• INT4 → 0.5 bytes/param → 35 GB (plus a few GB of overhead)

Why Quantize

• Smaller download and disk — a 70B model fits on a laptop SSD.
• Less VRAM — fits on consumer GPUs (RTX 4090 has 24GB).
• Faster inference — memory bandwidth is the bottleneck for LLM decoding; smaller weights = more tokens/second.
• Cheaper hosting — fewer GPUs per replica.

The catch: every quantization loses some accuracy. The art is keeping that loss small enough not to matter.

PTQ vs QAT

• Post-Training Quantization (PTQ) — take a finished model, quantize its weights, ship. Cheap, fast, requires no retraining. Almost every "GGUF" or "GPTQ" file you see on Hugging Face is PTQ.
• Quantization-Aware Training (QAT) — simulate quantization during training so weights adapt. Higher quality, especially at aggressive bitwidths (2-bit, 3-bit), but expensive. Used in BitNet and 1-bit LLMs.

Major Techniques

Technique	Author / Year	Format	Notes
GPTQ	Frantar et al., 2022	GPTQ	One-shot PTQ using Hessian information; high quality at 4-bit.
AWQ	Lin et al., 2023	AWQ	Activation-aware: preserves outlier weights. Strong on instruction-tuned models.
GGUF	Gerganov, 2023	GGUF	llama.cpp's container; supports many bit widths and K-quants.
bitsandbytes	Tim Dettmers	NF4, INT8	4-bit NF4 is the default for QLoRA fine-tuning.
SmoothQuant	Xiao et al., 2022	Various	Migrates activation outliers into weights for cleaner INT8.
BitNet b1.58	Microsoft, 2024	Trits (-1, 0, +1)	QAT 1.58-bit; matches FP at scale.
K-quants	llama.cpp, 2023	Q*K*	Block-wise mixed precision (e.g., Q4_K_M).

K-quants Naming

Inside GGUF you'll see names like Q4_K_M. Decoded:

• Q4 — average 4 bits per weight
• K — "K-quant", a mixed-precision scheme using higher bits for sensitive layers
• M — Medium variant (S = Small, M = Medium, L = Large). Larger = higher quality, bigger file.

Q4_K_M is the community-favorite sweet spot: near-FP16 quality at 4.8 bits-per-weight.

Quality Impact

Empirical rules of thumb (on standard reasoning/QA benchmarks):

• 8-bit: essentially lossless, <0.5% drop.
• 6-bit (Q6_K): <1% drop, often indistinguishable.
• 4-bit (Q4_K_M, GPTQ-4, AWQ-4): typically 1–3% drop.
• 3-bit (Q3_K_M): 3–6% drop, noticeable on hard tasks.
• 2-bit (Q2_K, IQ2_XS): 8–15% drop, only worth it when the alternative is not running at all.
• 1.58-bit BitNet (QAT only): matches FP16 at large scale.

Reasoning and code generation are more sensitive to quantization than chit-chat.

Precision Levels — At a Glance

Precision	Bytes/param	70B Model Size	Approx. Quality	Typical Use
FP16 / BF16	2	140 GB	Baseline	Training, server inference
FP8	1	70 GB	~Lossless	H100/B100 training & inference
INT8	1	70 GB	~Lossless	Server inference
Q5_K_M	0.7	49 GB	-0.5%	Quality-first local
Q4_K_M	0.6	42 GB	-1 to -2%	Sweet spot for local
INT4 / AWQ-4	0.5	35 GB	-1 to -3%	Single-GPU local
Q3_K_M	0.45	32 GB	-3 to -6%	Tight VRAM
Q2_K	0.3	21 GB	-8 to -15%	Last resort
BitNet 1.58 (QAT)	0.2	~14 GB	Lossless at scale	Research / future

Tools You'll Actually Use

• llama.cpp — Gerganov's C++ engine. Reads GGUF. Runs on CPU, CUDA, Metal, Vulkan, ROCm.
• Ollama — wraps llama.cpp with a Docker-style CLI. Best onboarding.
• LM Studio — desktop GUI for GGUF models.
• vLLM — high-throughput server; supports AWQ, GPTQ, FP8.
• MLX — Apple Silicon native; great on M-series Macs.
• bitsandbytes — Python library used by Hugging Face Transformers for 8-bit / 4-bit loading and QLoRA training.
• Hugging Face Optimum / AutoGPTQ / AutoAWQ — Python paths to GPTQ and AWQ formats.

Which Quant Should You Pick?

A practical decision tree:

1. VRAM ≥ model FP16 size? Use FP16/BF16. Don't quantize unless you need speed.
2. VRAM ≥ ~60% of FP16? Use INT8 / Q8_0. Near-lossless.
3. VRAM ≥ ~30% of FP16? Use Q4_K_M (GGUF) or AWQ-4 / GPTQ-4. The default for most users.
4. Tight VRAM? Drop to Q3_K_M before going to a smaller model — a heavily-quantized big model usually beats a full-precision small one.
5. Below Q3? Consider switching to a smaller model entirely.

On a 24GB RTX 4090: Llama 3.1 70B in Q3_K_S just fits; Q4_K_M is too big and spills to RAM. On a 16GB Mac: Llama 3.1 8B in Q5_K_M is comfortable; 70B requires Q2 and gets slow.

Limitations

• Outlier weights — a few large weights dominate quality; AWQ and SmoothQuant exist to handle them.
• KV cache — quantizing weights doesn't shrink the KV cache, which grows with context length. Separate KV-cache quantization (Q8/Q4) exists.
• Vision and audio multimodal layers are more quant-sensitive than text Transformer blocks.
• Long-context — at 32k+ tokens, low-bit quants degrade faster.

Quantization in the Real World

In 2026, quantization is everywhere. Most consumer uses of LLMs involve quantization:

• Ollama (the most popular local LLM tool) ships GGUF quantized models by default
• LM Studio (GUI for local LLMs) defaults to Q4_K_M
• Apple's on-device models (Genmoji, Writing Tools) are heavily quantized
• Cloud inference — companies like Together and Replicate offer quantized options for cost savings

Quantization flipped the economics of LLMs. A 70B model that cost $1000s to host now costs $10s because 4-bit quantization makes it 4x smaller and faster. This enabled a democratization of LLM access.

The Trade-Off Frontier

Quantization is fundamentally a speed-quality trade-off. The frontier of choices in 2026:

For developers:

• Vanilla FP16 → develop and test, validate quality isn't sensitive
• INT8 → deploy to edge if model doesn't require aggressive reduction
• Q4_K_M → standard for single-GPU local inference
• Q3_K_M → if you're memory-constrained
• Smaller model in full precision → often beats large model heavily quantized

For infrastructure teams:

• AWQ or GPTQ → production server inference with vLLM, good throughput
• GGUF → multi-platform (CPU, GPU, Mac), flexibility over peak throughput
• Proprietary quants → cloud providers have custom solutions optimized for their hardware

The Quantization Frontier: Extreme Compression and Its Limits

As of 2026, the aggressive frontier of quantization research pushes toward extreme compression — 1-bit, 2-bit, and ternary models that challenge conventional assumptions about neural networks.

1.58-Bit Models (BitNet b1.58). Microsoft released BitNet b1.58 in 2024, a model trained with 1.58-bit weights (ternarized to values −1, 0, +1). The radical finding: a 1.58-bit 7B model matches FP16 performance on standard benchmarks. This only works with QAT (quantization-aware training) from scratch — you can't take an existing model and compress it to 1.58-bit. But if it scales to larger models, the implications are enormous: a 70B model in 1.58-bit could fit in ~14 GB VRAM.

Why does it work? Ternarized weights reduce the neural network to near-pure routing — activations flow through a fixed structure with minimal value computation. The model learns which routes to activate rather than precise weight magnitudes. This is fundamentally different from traditional neural networks and suggests that LLMs may have more "excess capacity" than we assumed.

2-Bit Quantization and K-Quants. In GGUF, K2 and IQ2_XS formats push 2-bit performance. These use non-uniform bit allocation (some weights get more bits than others) and extensive calibration on representative text. The empirical result: 2-bit GGUF models are surprisingly usable for chat, though reasoning and coding degrade. A 70B model in Q2_K_M is ~21 GB — fitting on some high-end consumer GPUs with massive slowdown.

Sparse Quantization. Pruning (removing neurons) + quantization (compressing remaining weights) can outperform quantization alone. A model pruned to 50% sparsity + quantized to 4-bit can match an unquantized model's performance while being 8x smaller. This is harder to implement (requires specialized kernels), but companies like vLLM and Ollama are adding support.

Mixed-Bit Allocation. Rather than all weights at 4-bit, allocate bits dynamically: critical layers (attention) at 8-bit, less critical layers (feed-forward first layers) at 2-bit. A hand-tuned 70B model might average 3.2 bits per weight while maintaining full-precision quality on reasoning tasks. This requires model-specific engineering but pays off for production deployments.

Source: Ma, X., et al. (2024). "The Era of 1-Bit LLMs: All Large Language Models are in 1.58 Bits." arXiv 2402.17764.

Quantization in Production: Real-World Trade-Offs

Theory and practice diverge. What works on benchmarks doesn't always work in production. Real-world constraints:

Latency vs Throughput. A quantized 70B model decodes faster (more tokens per second) but has higher latency per token (time to first token is slow because you're doing more computation per weight). For real-time chatbot UX, this matters. A smaller model in full precision sometimes beats a huge quantized model.

Context Length. Quantization doesn't shrink the KV cache — the memory used for storing past context. A 70B model with 32K context window still needs ~64 GB for the KV cache alone, independent of quantization. The practical effect: quantization lets you fit the weights, but long-context inference can still OOM on consumer hardware.

Cost-Quality Frontier. At $0.01/1M tokens, a cloud provider uses heavily quantized 70B models. At $0.50/1M tokens, they use smaller FP8 models. At $2.00+/1M tokens, they might use unquantized or partially quantized models. Quantization is a cost-optimization tool, not a quality tool — companies use it where quality loss is acceptable given the cost savings.

Streaming and Batching. Server inference (multiple users at once) benefits more from quantization than single-user inference. Batching many requests through a quantized model can keep GPU utilization high. A 24GB RTX 4090 quantized to Q4 can serve 10–20 concurrent users at reasonable latency, where unquantized could serve only 2–3.

Source: Hoffmann, J., et al. (2023). "An Open-Source Framework for Evaluating LLM Quantization." MLSys 2024.

When NOT to Quantize

Quantization is not always the answer. Edge cases where it fails or shouldn't be used:

Retrieval-Augmented Generation (RAG). RAG systems embed text into vectors (embeddings), then retrieve similar documents at query time. Quantization of the embedding model can hurt retrieval quality — returned documents become slightly less relevant. A 1–2% drop in retrieval quality cascades into much larger quality loss in the final answer. Many practitioners avoid aggressive quantization for embedding models.

Structured Output. If the model must generate JSON, code, or other rigidly structured output, quantization increases the chance of malformed output. A quantized model might hallucinate a comma in the wrong place. For systems requiring exact format compliance, quantization risk isn't worth the size savings.

Fine-tuned Models. A model fine-tuned on domain-specific data and then heavily quantized can lose the specialized knowledge. The FT process adapted the model's weights to your domain; quantization erases that adaptation. If you're fine-tuning, use QLoRA (4-bit quantization + LoRA) during training, and be cautious post-training quantization.

Multi-language or Cross-cultural Models. Large multilingual models benefit from precise weights because they're balancing many languages' syntactic/semantic differences. Quantization can preferentially degrade one language. Test carefully before deploying quantized multilingual models.

Hardware-Specific Quantization Strategies

Different hardware benefits from different quantization approaches. Optimization isn't one-size-fits-all.

NVIDIA GPUs (RTX 40-series, A100, H100): GPTQ and AWQ are fastest because NVIDIA published custom CUDA kernels for 4-bit inference. AWQ-4 typically delivers 50–70 tokens/second on RTX 4090 for 70B models. Older GPUs (RTX 3090) are slower — 20–30 tokens/second for the same model.

AMD GPUs (MI300, MI250): ROCm support for quantization is improving. GGUF runs via llama.cpp's ROCm backend; GPTQ is officially supported but slower than NVIDIA. Expect 60–70% of NVIDIA performance.

Apple Silicon (M1/M2/M3/M4): Metal-optimized GGUF inference is excellent — a 13B Q4_K_M model runs at 30–50 tokens/second on M3 Max. This surprised many: Apple's GPU kernel optimizations for matrix operations beat expectations. Larger models (70B) require Q3 or Q2 and drop to 5–15 tokens/second.

CPUs (Intel/AMD): Purely CPU inference is slow but works. llama.cpp's CPU backend is optimized with AVX2/AVX512. Expect 1–3 tokens/second for a 7B model, 0.3–1 token/second for 70B. Reasonable for batched offline processing, not interactive chat.

Edge Devices (iPhone, Android): Llama 2 7B in Q2 (~4 GB) is viable on modern phones. Inference speed is 0.5–1 token/second. Practical for local inference-only apps (on-device search, summarization) without cloud calls.

Source: Gerganov, G. (2023). llama.cpp: Inference of LLaMA model in pure C/C++. GitHub repository, demonstrating CPU and GPU performance across hardware tiers.

The Quantization Accuracy Frontier: What Actually Matters

Benchmark performance doesn't always reflect real-world quality. Specific tasks degrade differently under quantization.

Arithmetic and counting: Quantization heavily impacts models' ability to do math. A quantized model often miscounts objects in a prompt or makes arithmetic errors. This is one of the known failure modes — the model loses precision in numerical reasoning.

Multiple-choice reasoning: Surprisingly robust to quantization. A 4-bit model gets multiple-choice questions almost as well as full precision. The trade-off: more reasoning steps hurt quantized models more than single-step MC tasks.

Code generation: Moderate sensitivity. Python generation stays nearly flawless at Q4, but C++ becomes error-prone. The difference: Python's syntax is forgiving; C++ requires strict correctness. Quantization adds noise that breaks strict syntax more often.

Factual recall: Minimal impact at Q4. Factual knowledge is distributed across the model; low-bit quantization doesn't preferentially erase facts. A quantized model's hallucination rate is similar to full precision on knowledge tasks.

Instruction following: Slightly degraded at Q3 and below. Models quantized to Q2 sometimes ignore instructions or follow them partially. This likely reflects loss of precision in the attention mechanism.

Practical implication: For most chat use cases, Q4 is indistinguishable from FP16. For code and math-heavy applications, prefer Q6 or higher, or use a smaller model in full precision.

Source: Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2023). "QLoRA: Efficient Finetuning of Quantized LLMs." arXiv:2305.14314.

Quantization and Model Size Selection: The Empirical Trade-Off

A common mistake: choose a huge quantized model when a smaller unquantized model would be better. The data:

• Llama 3.1 70B Q4 (42 GB, ~50 tokens/sec on RTX 4090): performs well on MMLU (65% accuracy), slightly loses math ability.
• Llama 3.1 8B FP16 (16 GB, ~200 tokens/sec on RTX 4090): performs worse on MMLU (57%), but faster and leaner.

The question: which do you pick? For most users, the 8B at full precision is better because it's faster, smaller, and "good enough." The 70B Q4 is only better if you need that specific 8-point accuracy bump on hard benchmarks.

For production services, quantization of huge models makes sense — you're squeezing every ounce of capability. For local development or personal use, a smaller model at full precision is often the right call.

Empirical guidelines:

• If you have <16GB VRAM, pick Llama 3.1 8B Q5, not 70B Q2.
• If you need reasoning, go Q4_K_M minimum, don't use Q2.
• If you're fine-tuning, use a small model in FP16 + QLoRA, not a huge model quantized to Q2.

A Brief History of LLM Quantization (2022–2026)

2022: GPTQ. Tim Dettmers and team published GPTQ, revolutionizing 4-bit quantization with per-group scale factors and Hessian-informed selection. Suddenly, 4-bit models were near-lossless. This enabled the first practical local LLM deployments.

2023: AWQ and K-Quants. Lin et al. introduced activation-aware quantization (AWQ), which preserves outlier activation weights. In parallel, llama.cpp adopted K-quants — mixed-bit schemes using higher precision for critical layers. The community converged on "Q4_K_M as the standard" for GGUF.

2024: Extreme Compression (1.58-bit). Microsoft's BitNet b1.58 paper triggered a rethinking of how small LLMs could be. Ternary weights matched FP16 performance at scale, suggesting neural networks have massive redundancy.

2025–2026: Production Adoption. Ollama reached 5M+ users partly because it defaults to Q4_K_M quantization, making local LLMs practical. vLLM began shipping with integrated AWQ/GPTQ support, making cloud inference cost-competitive with closed APIs.

Current frontier (2026): Sparse quantization (pruning + quantization), mixed-bit allocation, and on-the-fly adaptive quantization for long-context inference are nascent but promising.

Source: Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." In ICLR 2023.

Quantization in Production: Case Studies

Ollama (1M+ daily active users): Ships exclusively with GGUF quantized models (default Q4_K_M). Inference happens on CPU or GPU. Average latency on RTX 4090: 50ms/token for 70B Q4. Mobile use (app inference on MacBook): acceptable for chat, slow for batch processing.

Together.ai (Commercial API): Offers quantization options. 70B FP8 at $0.50/1M tokens vs 70B AWQ at $0.30/1M tokens. Throughput difference is minimal (both >100 tokens/sec in batch). Customers choose based on cost sensitivity, not accuracy — most pick AWQ to save 40%.

Hugging Face Inference API: Supports INT8 (lossless) and experimental GGUF. Quality for production use cases remains uncompromised; quantization is transparent to the user.

On-Device Inference (Apple Genmoji, Writing Tools): Apple's private on-device models are heavily quantized (likely Q4 or lower). Generation is slow (1–2 tokens/second) but privacy-respecting and offline.

Practical takeaway: Quantization is table-stakes in 2026. Every production LLM inference service uses it. The question isn't "should I quantize" but "which quantization level for my use case."

Frequently Asked Questions

Q: What is LLM quantization in simple terms?
A: Storing the model's numbers in fewer bits — for example, 4-bit integers instead of 16-bit floats — so the file is 2–4x smaller and runs faster, with only a small quality drop.

Q: Which is better, GGUF or GPTQ?
A: GGUF (via llama.cpp) is the most portable — runs on CPU, GPU, and Mac. GPTQ and AWQ are typically faster on NVIDIA GPUs with vLLM. For local single-machine use, GGUF wins; for server inference, AWQ/GPTQ.

Q: Is 4-bit quantization good enough?
A: For most chat and coding use cases, Q4_K_M or AWQ-4 loses only 1–3% on benchmarks — barely perceptible. For hard reasoning or research, prefer 6-bit or 8-bit.

Q: Can I fine-tune a quantized model?
A: Yes — that's what QLoRA is for. The base model is loaded in 4-bit (NF4), and small trainable LoRA adapters in higher precision are added on top. This is the standard for fine-tuning on consumer GPUs.

Q: What is the smallest usable quant?
A: For PTQ, Q3_K_M is usually the floor. Below that, accuracy degrades noticeably. With QAT, BitNet b1.58 (1.58-bit) can match FP16 — but only models trained in that scheme from scratch.

Q: Does quantization make models faster?
A: Yes, often 1.5–3x faster for token generation, because LLM decoding is memory-bandwidth-bound — moving fewer bytes through the GPU means more tokens per second.

---

Read next:

• Mixture of Experts (MoE) Explained
• LLMs vs SLMs — Small Language Models Explained
• Edge AI — How AI Runs on Your Phone

Sources

• Frantar, Ashkboos, Hoefler, Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. 2022.
• Lin et al. AWQ: Activation-aware Weight Quantization for LLM Compression. 2023.
• Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs. 2023.
• Xiao et al. SmoothQuant. 2022.
• Ma et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. Microsoft, 2024.
• llama.cpp project documentation, Georgi Gerganov, 2023–present.

Related Reading

• Mixture of Experts (MoE) Explained
• AI Inference vs Training Explained
• Test-Time Compute Explained
• Neural Networks Explained