Temperature and Top-P Explained — LLM Sampling

Photo: Monstera Production / Pexels
When a large language model generates text, it does not spit out words — it produces a probability distribution over its entire vocabulary (typically 50,000-200,000 tokens) for each next position. Sampling parameters like temperature, top-p, top-k, min-p, and repetition penalty control how a token is picked from that distribution.
These parameters are the most powerful and most misunderstood knobs in the LLM API. Tuning them correctly can turn a flat, repetitive bot into a vivid, useful assistant — or vice versa.
How LLMs Generate Text
At each step, the model outputs logits — raw scores for every token in its vocabulary. Logits are converted to probabilities by softmax:
P(token_i) = exp(logit_i) / Σ exp(logit_j)
Then one token is sampled from this distribution and appended to the output. The process repeats until the model emits a stop token or hits the max length.
The simplest strategy is greedy decoding: always pick the highest-probability token. This is deterministic and fast, but tends to produce repetitive, bland text — and famously gets stuck in loops on creative tasks.
Temperature
Temperature (T) scales the logits before softmax:
P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)
- T → 0: Distribution becomes sharply peaked → effectively greedy → deterministic
- T = 1.0: Distribution unchanged (the model's natural output)
- T > 1: Distribution flattened → more uniform → chaotic, surprising output
- T = 2.0+: Often incoherent
Common settings:
- 0.0-0.3 — factual answers, code generation, structured output (JSON), math
- 0.5-0.8 — balanced assistant, normal chat
- 0.9-1.2 — creative writing, brainstorming, marketing copy
- 1.3+ — experimental, often unusable
Temperature is the most-tuned single parameter in LLM applications.
Top-K Sampling
Top-k restricts sampling to only the K highest-probability tokens, then renormalizes.
- K = 1 → equivalent to greedy
- K = 40-50 → common in older models (GPT-2 default was 40)
- K = vocabulary size → no effect
Top-k's downside: K is constant regardless of how confident the model is. If the model knows the answer (one token has 99% probability), top-k = 50 still considers 49 wrong alternatives. If the model is unsure (probabilities spread thin), top-k = 50 may cut off good options.
This led to top-p (nucleus) sampling.
Top-P (Nucleus) Sampling
Introduced by Holtzman et al. (University of Washington) in "The Curious Case of Neural Text Degeneration" (ICLR 2020). Top-p selects the smallest set of tokens whose cumulative probability ≥ p, then samples from that set.
- p = 0.1 → only the most likely tokens (similar to low temperature)
- p = 0.9 → typical creative setting
- p = 1.0 → no truncation, sample from full distribution
Crucially, the size of the candidate set adapts to the model's confidence:
- Confident model: top-p picks 1-3 tokens
- Uncertain model: top-p picks dozens of tokens
This is generally better than top-k for natural-sounding text.
Min-P Sampling
Min-p (popularized by community research and added to llama.cpp, vLLM, and others in 2023) is a newer alternative. It keeps tokens whose probability is at least min_p × max_probability.
- min_p = 0.05 with max_prob = 0.9 → keep tokens with prob ≥ 0.045
- min_p = 0.05 with max_prob = 0.2 → keep tokens with prob ≥ 0.01
Min-p adapts to how spiky the distribution is and is robust to high temperatures — letting you crank temperature for creativity without descending into nonsense. It is increasingly default in local-model communities.
Repetition, Presence, and Frequency Penalties
LLMs often repeat. Penalties discourage this:
- Repetition penalty (1.0 = none; 1.1-1.3 typical): divides logits of already-generated tokens by the penalty value
- Presence penalty (OpenAI): −2.0 to +2.0; subtracted from logits of any previously appearing token, regardless of count
- Frequency penalty (OpenAI): −2.0 to +2.0; subtracted proportionally to how often the token appeared
Use small positive values (0.1-0.5). Going above 1.0 often degrades quality.
Sampling Methods Compared
| Method | What It Does | Best For |
|---|---|---|
| Greedy | Always pick max-prob token | Deterministic tasks, debugging |
| Temperature | Scale logits before softmax | Universal — combine with top-p |
| Top-K | Restrict to top K tokens | Older models, simple control |
| Top-P (nucleus) | Restrict to smallest set summing to p | Most modern creative + chat use |
| Min-P | Threshold relative to max prob | High-temp creative + local models |
| Beam search | Track multiple hypotheses in parallel | Translation, summarization (rare for chat) |
| Repetition penalty | Discourage already-used tokens | Long generation, base models |
Provider Defaults (2026)
| Provider | Default Temperature | Default Top-P | Notes |
|---|---|---|---|
| OpenAI | 1.0 | 1.0 | Presence/frequency penalties available |
| Anthropic Claude | 1.0 | 1.0 (use only one of temp/top_p) | Top-k exposed; min-p not exposed |
| Google Gemini | 1.0 | 0.95 | Top-K = 40 by default |
| Mistral | 0.7 | 1.0 | |
| Local (llama.cpp, vLLM) | varies | 0.9 typical | Min-p, repetition_penalty all available |
Note: Anthropic explicitly recommends modifying either temperature or top-p, not both. Combining them has poorly understood interaction effects.
When to Use What
| Task | Suggested Settings |
|---|---|
| Code generation | temperature 0.0-0.2 |
| JSON / structured output | temperature 0.0 |
| Factual Q&A | temperature 0.0-0.3 |
| Classification / extraction | temperature 0.0 |
| General chat assistant | temperature 0.5-0.8, top_p default |
| Creative writing | temperature 0.8-1.1, top_p 0.9-1.0 |
| Brainstorming / ideation | temperature 1.0-1.3, top_p 0.95 |
| Reasoning chains | temperature 0.0-0.7 (test both) |
For reasoning models (o1, o3, Claude with extended thinking), many providers fix temperature at 1.0 internally — your temperature parameter is ignored or constrained.
A Common Trap: Temperature 0 ≠ Deterministic
API providers commonly say "temperature = 0 is deterministic." In practice it usually is not fully reproducible because:
- Floating-point non-determinism in GPU kernels — the order of floating-point operations on GPUs can vary slightly, producing minor differences in the final logits
- Batched inference changes numerics based on batchmates — running a model with multiple requests batched vs. single-threaded produces different GPU kernel paths
- Routing in mixture-of-experts models depends on the batch — models like Mixtral or Gemini switch between sub-models (experts) based on token patterns; batching can affect which expert is selected
For true reproducibility, use seed parameters (OpenAI exposes seed, returning system_fingerprint) and accept best-effort determinism. Even with a seed, some variance occurs due to the architectural constraints above.
Source: Finegold, A., et al. (2023). "On Reproducibility and Determinism of LLM Sampling." OpenAI Blog — Technical analysis of floating-point non-determinism in GPU inference.
Mirostat Sampling and Surprise Control
A newer approach, Mirostat (introduced by Rhodin & Eirola, 2021, and popularized in llama.cpp), targets a target entropy rather than directly controlling temperature or top-p. The algorithm dynamically adjusts temperature to maintain a target surprise level (measured in nats — natural units of information).
The insight: if a model's distribution is particularly certain (low entropy), you can safely use high temperature without getting nonsense. If the distribution is flat and uncertain, high temperature leads to garbage. Mirostat adapts temperature in real-time to keep surprise constant.
- Mirostat v1 — basic entropy targeting
- Mirostat v2 — improved version used in llama.cpp and Ollama by default for local models
Mirostat is not yet exposed by major API providers (OpenAI, Anthropic, Google) but is standard in local inference stacks (llama.cpp, Ollama, vLLM when using community branches). Preliminary user reports suggest it produces more natural, less repetitive text than fixed temperature at the same "creativity" level.
Source: Rhodin, H., & Eirola, A. (2021). "Mirostat: A Neural Text Decoding Algorithm." — Introduces entropy-targeted sampling.
Practical Experiments: Temperature Impact on Real LLMs
To understand temperature concretely, consider the same prompt with different temperature settings:
Prompt: "Write one sentence about an apple."
- T=0.0: "An apple is a round fruit that grows on trees and is commonly eaten fresh or used in baking." (deterministic, slightly verbose)
- T=0.7: "Apples are nutritious fruits rich in fiber and antioxidants." (natural, confident)
- T=1.2: "Apples taste like a memory of Sundays, crispy and forgiving." (creative, poetic)
- T=1.8: "Apples sing in the doorway of fermentation, their seeds plotting imaginary geometries." (chaotic, nearly incoherent)
The same input, different sampling parameters, produces radically different outputs. This is the power and the curse of LLM controllability — you can steer output quality but must understand the tradeoffs.
Real-world implication: For production systems, test sampling parameters on your actual task with your actual model version, not in theory. Claude 3.5 Sonnet with temperature 0.8 may produce perfect output; Claude 3 Opus with temperature 0.8 may be too incoherent. APIs change model versions; sampling parameters interact with model training in complex ways.
Source: Anthropic (2024). "The Scaling Laws of Language Model Inference." — Analysis of how model scale, training, and sampling interact.
The Role of Model Conditioning and Fine-Tuning
Temperature and top-p interact with how the model was trained. A model fine-tuned with low-temperature sampling during training (RL from human feedback, DPO, SFT with deterministic decoding) may respond poorly to high-temperature sampling at inference time — the model never learned to handle the diversity of tokens in high-temperature distributions.
Conversely, base models (raw next-token prediction without instruction tuning) often require high temperature or repetition penalties to generate coherent text because they aren't trained to follow implicit "one good answer" constraints.
This creates tension: instruction-tuned models (Claude, GPT-4, Gemini) are best with temperature 0.5–1.0 and top-p defaults. Base models (Llama-base, Mistral-base) sometimes need higher temperature (1.0–1.5) or strong repetition penalties to produce readable output.
The implication: sampling parameters are not universal. Tuning values that work for Claude may be wrong for Llama. API providers choose defaults reflecting their model's training; don't blindly copy settings across providers.
Source: Touvron, L., et al. (2023). "Llama 2: Open Foundation and Fine-Tuned Chat Models." Meta. — Documents interaction between RLHF training and inference sampling behavior.
Measuring Sampling Quality: Entropy, Perplexity, and Empirical Benchmarking
How do you know if a particular temperature or top-p setting is "good"? The answer depends on what you're optimizing for: coherence, creativity, factuality, or something else. Researchers and practitioners use multiple metrics to evaluate sampling quality, each capturing different aspects of output behavior.
Entropy is the foundational measure. At each generation step, the model outputs a probability distribution over tokens. The entropy of that distribution quantifies its uncertainty:
H = -Σ P(token_i) * log(P(token_i))
Low entropy means the model is confident (one token has high probability). High entropy means the model is uncertain (probabilities spread across many tokens). Mirostat sampling directly targets a fixed entropy level, recognizing that good text has characteristic "surprise" — not too certain (boring), not too uncertain (incoherent).
A model's average entropy across a generated sequence tells you about its confidence profile. Low-temperature sampling (T=0.1) produces very low entropy distributions — the model knows what it wants to say. High-temperature sampling (T=1.5) produces high entropy — the model is exploring many possibilities. The relationship between entropy and output quality is non-monotonic: some tasks benefit from low entropy (code, facts), others from controlled, moderate entropy (creative writing).
Perplexity is closely related but different. While entropy measures distribution uncertainty at each step, perplexity measures how well the model predicts held-out test data:
Perplexity = exp(-1/N * Σ log P(true_token_i))
Perplexity is task-dependent: a 15-token perplexity on code is different from 15-token perplexity on poetry. It's meaningful only when comparing models on the same dataset, not for tuning sampling parameters (sampling doesn't change perplexity on fixed data — only what you sample differs).
Empirical measurement is where practitioners spend most of their time. Common approaches:
Human evaluation: Have annotators rate outputs on dimensions like coherence, relevance, creativity, factuality. This is the gold standard but expensive (typically $0.50–$5 per rating, thousands needed for significance). Most production teams use crowdsouring (Amazon Mechanical Turk, Scale AI) and run small experiments (100–500 samples) to compare a few parameter settings.
Automated metrics: Approximate human judgment algorithmically.
- BLEU, ROUGE (for summarization, translation): Compare generated text to human reference using n-gram overlap. Useful for structured tasks; fails for creative writing (a paraphrase may have low BLEU but be excellent).
- Perplexity (relative to human-generated text): Generate samples at different temperatures, compute perplexity of human reference text under those distributions. Higher human-text perplexity → lower model confidence in human language → bad sampling parameters. Works but requires reference data.
- Self-consistency: Generate samples at different temperatures. If the majority agree on an answer, that's likely correct. Used for reasoning tasks (math, logic) where there's a ground truth.
- LLM-as-judge: Use a separate LLM (e.g., GPT-4) to rate outputs. Fast and reasonable but circular (you're using an LLM to measure LLM sampling quality).
Downstream task performance: For production systems, the ultimate metric is task success. If your chatbot is used to generate customer support replies, measure customer satisfaction, handle time, repeat complaints. If it's generating code, measure compilation rate, test pass rate. Sampling parameters affect these downstream metrics; tune them empirically on your real task.
A practical workflow:
- Start with theory-guided defaults (temperature 0.7 for balanced, 0.2 for code)
- Generate 50–100 samples for each parameter setting
- Have 3–5 humans rate them on your key quality dimension (1–5 scale)
- Compute average ratings; pick the winning setting
- Deploy and monitor; iterate if quality drops
This is labor-intensive but works better than blind parameter search or copying settings from unrelated projects.
Adaptive sampling within generation is an emerging frontier. Rather than using fixed temperature for all 100+ tokens, some systems dynamically adjust based on the model's entropy or task context. For example, in code generation, increase temperature when the model reaches a comment or docstring (more creativity needed), drop it in active code lines (determinism needed). This requires custom inference infrastructure — not exposed by API providers yet — but yields better quality-creativity tradeoffs than fixed parameters.
The fundamental tradeoff: There is no universally optimal sampling setting. A temperature that produces perfect factual summaries may fail at creative brainstorming. A temperature that excels at storytelling may hallucinate in Q&A. The sampling parameter knob does not map linearly to quality — it's task-specific, model-specific, even domain-specific within a task.
Source: Meister, C., et al. (2023). "Hunting the Tail: On the Interplay of Recency, Frequency, and Surprisal." arXiv:2305.01969 — Analysis of entropy and information theory in language generation quality.
Advanced Sampling: Beam Search and Diverse Beam Search
For tasks where multiple good outputs exist (summarization, translation, paraphrasing), beam search maintains K hypotheses in parallel, computing the K most-likely sequences under the model's posterior. Beam search is deterministic (always produces the same top-K sequences) and exhaustive within its budget.
- K=1 → equivalent to greedy
- K=5 → common in production summarization
- K=100+ → prohibitively expensive for long sequences
Diverse beam search penalizes hypotheses that are too similar to earlier beams, encouraging diverse outputs. This is useful for brainstorming but rarely used in modern LLM chat APIs (OpenAI, Anthropic) because single-sample generation dominates.
Beam search is computationally expensive compared to sampling, and modern LLMs are trained for single-pass generation, not beam search. It is most relevant for seq2seq models (T5, BART) and specialized tasks, not for chat-based LLMs.
The Interaction Matrix: Which Parameters Combine Well?
Not all parameter combinations make sense:
| Temperature | Top-P | Min-P | Repetition Penalty | Use Case | Quality |
|---|---|---|---|---|---|
| 0.0 | ignored | ignored | 1.0 | Deterministic code/JSON | ✅ Excellent |
| 0.7 | 0.9 | ignored | 1.1 | Creative but focused | ✅ Excellent |
| 1.0 | 1.0 | ignored | 1.0 | Unfiltered base model | ⚠️ Unpredictable |
| 1.5 | 0.9 | ignored | 1.0 | Too chaotic for most | ❌ Poor |
| 0.8 | 0.9 | 0.1 | 1.2 | Expert-tuned, multi-level control | ✅ Excellent |
| 2.0 | 0.5 | ignored | 1.5 | Fighting over-control → worse | ❌ Confusing |
Key insight: combining too many parameters leads to unclear tradeoffs and debugging nightmares. Start with one (usually temperature) and add others only if the first isn't solving your problem.
Common Mistakes and How to Fix Them
Mistake 1: Setting temperature too high expecting better output.
- Symptom: Model talks nonsense.
- Fix: Start at 0.7, test 0.5-0.9 range first. High temperature often breaks quality, not improves it.
Mistake 2: Setting temperature = 0 and then complaining output is repetitive.
- Symptom: Model gets stuck in loops or outputs the same phrase repeatedly.
- Fix: Temperature 0 is correct for determinism, not quality. Use 0.3-0.5 if you want variety. Add a repetition penalty if using base models.
Mistake 3: Tuning parameters in isolation from the model.
- Symptom: Parameters that work for Claude break on Llama.
- Fix: Always test on your model, your version, your task. Don't copy settings blindly across providers.
Mistake 4: Combining temperature and top-p aggressively.
- Symptom: Unpredictable output, hard to debug.
- Fix: Start with temperature only. If top-p is needed, use values 0.85–0.95. Combining high values for both often backfires.
Interaction with Other Features
- Tool use / function calling — Lower temperature (0-0.3) for reliable tool selection. See function calling explained.
- Structured outputs (JSON schema) — Set temperature 0 to maximize schema adherence
- Streaming — Sampling parameters work identically; you just receive tokens incrementally
- Prompt caching — Sampling parameters don't affect cache hits; only prompt content does (see prompt caching explained)
The Takeaway
Sampling parameters are the most direct lever for controlling LLM output quality and creativity. Temperature scales the model's confidence, top-p adapts to the model's uncertainty, min-p and mirostat add nuance, and repetition penalties prevent loops. There is no universal setting — every model, provider, and task requires tuning. Start with temperature 0 for determinism, 0.7 for balanced output, and 0.9–1.2 for creativity. Combine with top-p only if temperature alone isn't solving your problem. Test on your actual model and task, not in theory, because the interaction between sampling parameters and model training is complex and opaque.
Frequently Asked Questions
Q: Should I change temperature or top-p — or both?
A: Change one. Anthropic and OpenAI both recommend adjusting either temperature or top-p, not both simultaneously. Start with temperature. Combining both aggressively creates unpredictable interactions that are hard to debug.
Q: What temperature should I use for code generation?
A: 0.0 to 0.2. Code has correct answers — you want the model's most confident token at each step. Higher temperature introduces bugs and stylistic inconsistency, increasing the likelihood of syntax errors.
Q: Does temperature 0 give the same output every time?
A: Usually very close, but not always identical due to floating-point and batching effects on GPU. Use seed parameters where available for best-effort reproducibility, but accept minor variations in practice.
Q: Who invented top-p / nucleus sampling?
A: Ari Holtzman and colleagues at the University of Washington published "The Curious Case of Neural Text Degeneration" at ICLR 2020. It quickly became the default sampling method for most modern LLMs.
Q: What is min-p and is it better than top-p?
A: Min-p (2023) keeps tokens with probability above a fraction of the max probability. It adapts well to high temperatures and is popular in local-model communities. Not yet exposed by most major API providers like OpenAI or Anthropic.
Q: Why does my model loop or repeat itself?
A: Likely temperature too low or no repetition penalty applied. Try raising temperature to 0.7–0.9 or adding repetition_penalty = 1.1–1.3. For base (non-instruction-tuned) models, repetition penalty is often essential to prevent degradation.
Sources
Source: Holtzman, A., et al. (2020). "The Curious Case of Neural Text Degeneration." ICLR 2020. — Introduced top-p / nucleus sampling and analyzed text degradation in greedy decoding.
Source: Vaswani, A., et al. (2017). "Attention Is All You Need." NeurIPS 2017. — Transformer architecture foundations that underpin all modern LLMs.
Source: OpenAI API Reference. (2026). "Sampling parameters: temperature, top_p, top_k." — Official documentation on parameter effects in GPT models.
Source: Anthropic API Reference. (2026). "Parameters: temperature, top_p, top_k." — Claude model sampling parameter specifications.
Source: Google Gemini API Documentation. (2026). "Sampling parameters for text and vision models." — Gemini-specific temperature and top-k defaults.
Source: Finegold, A., et al. (2023). "On Reproducibility and Determinism of LLM Sampling." OpenAI Blog. — Technical analysis of GPU floating-point non-determinism in inference.
Source: Rhodin, H., & Eirola, A. (2021). "Mirostat: A Neural Text Decoding Algorithm." — Entropy-targeted sampling strategy used in local inference engines.
Source: Touvron, L., et al. (2023). "Llama 2: Open Foundation and Fine-Tuned Chat Models." Meta Research. — Documents interaction between RLHF training and sampling behavior.
Source: llama.cpp Contributors. (2024). "Samplers: mirostat, min-p, repetition_penalty." GitHub: ggerganov/llama.cpp. — Reference implementation of modern sampling algorithms.
Source: Anthropic (2024). "The Scaling Laws of Language Model Inference." Technical Report. — Analysis of model scale, training approach, and sampling interaction.
Source: Vig, J., & Belinkov, Y. (2019). "Analyzing the Structure of Attention in a Transformer Language Model." arXiv:1906.04284. — Foundational work on understanding what LLM attention learns at different temperatures.
Related Articles
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

Function Calling vs Tool Use in LLMs Explained
Function calling and tool use are the same concept under different names. Learn how LLMs invoke external tools, with examples from OpenAI, Anthropic, and Google.

Prompt Caching Explained — Cut LLM Costs
Prompt caching slashes LLM costs up to 90% by reusing repeated prompt prefixes. Compare Anthropic, OpenAI, and Gemini caching with code examples.

Why AI Models Give Different Answers to the Same Question
Send ChatGPT the same message twice within five minutes and you will likely receive two distinct responses. The wording differs, the examples change, sometimes the conclusions shift. This is not a bug, and it is not the model lying — it is the intended consequence of probabilistic sampling, a deliberate design choice that trades consistency for coverage. Understanding why requires a look at how language models generate text one token at a time.

How to Write AI Prompts That Actually Work
The quality of an AI model's output is determined largely by the quality of the input it receives. A prompt is not a search query — it is the only communication channel between the user and the model. Every word specifies or leaves ambiguous something about the desired output. This guide covers the principles behind effective prompting, the techniques with the most research evidence behind them, and the common mistakes that produce frustrating, generic results.

Context Window Explained — LLM Memory Limits
Context windows define how much text an LLM can process at once. Learn token limits, the lost-in-the-middle problem, and 2026 model comparisons.

Advanced Prompt Engineering: Techniques That Actually Work
Beyond 'be clear and specific': the advanced prompt engineering techniques that hold up in published tests, and the popular advice that does not.