Why AI Models Give Different Answers to the Same Question

Send ChatGPT the same message twice within five minutes and you will likely receive two distinct responses. The wording differs, the examples change, sometimes the conclusions shift. This is not a bug, and it is not the model lying — it is the intended consequence of probabilistic sampling, a deliberate design choice that trades consistency for coverage. Understanding why requires a look at how language models generate text one token at a time.

How Text Generation Actually Works: Token-by-Token Sampling

Language models do not write sentences the way a human types them — producing a complete thought, then committing it to the page. Instead, they generate one token at a time, where each token is a word or subword fragment, and each token choice is drawn from a probability distribution over the entire vocabulary.

Logit — the raw, unnormalized score the model assigns to each possible next token before any probability conversion. A vocabulary of 100,000 tokens means the model computes 100,000 logits at each generation step.

Softmax — the mathematical function that converts logits into a proper probability distribution, where all values are positive and sum to 1.

The model computes logits for all tokens, applies softmax to get probabilities, and then — unless you intervene — samples from that distribution. "The" might have a 32% probability, "A" might have 18%, and "Consider" might have 4%. The model rolls a weighted die and picks one. Then the selected token is appended to the context, new logits are computed, and the die is rolled again.

This is why the same prompt produces different outputs: the die comes up differently each time. A useful analogy is weather forecasting — the model gives you probability estimates, not certainties, and two draws from the same forecast can produce two different days.

Source: Holtzman et al. (2020), "The Curious Case of Neural Text Degeneration," ICLR — the paper that introduced nucleus sampling (top-p) and analyzed why greedy decoding produces repetitive, low-quality text.

Temperature: The Single Most Consequential Setting

Temperature is a scalar value that reshapes the probability distribution before sampling. It is applied by dividing all logits by the temperature value before the softmax step.

• A temperature of 1.0 leaves the distribution unchanged — the model samples from its learned probabilities as-is.
• A temperature below 1.0 (say, 0.2) sharpens the distribution: high-probability tokens become even more dominant, low-probability tokens become negligible. The model becomes conservative, predictable, and repetitive.
• A temperature above 1.0 (say, 1.4) flattens the distribution: unlikely tokens become more competitive. The model becomes more varied, creative — and more prone to errors, non-sequiturs, and hallucination.

Think of temperature like the "confidence dial" on a weather forecast. A low-temperature forecast says "it will definitely rain tomorrow." A high-temperature forecast says "it might rain, or it might be sunny, or it might snow — here are all the possibilities with roughly similar weight."

Temperature	Distribution shape	Practical effect
0.0	Degenerate (always picks max)	Near-deterministic; repetitive on long outputs
0.1–0.3	Very sharp	Consistent, conservative, predictable
0.5–0.7	Moderately sharp	Balanced; good for factual Q&A
0.9–1.0	Natural (default)	Creative, varied; standard for chat
1.2–1.5	Flattened	More diverse; higher error rate
>2.0	Near-uniform	Incoherent; rarely used

Most public-facing AI products default to temperatures between 0.7 and 1.0. OpenAI's API defaults to 1.0; many system prompts for assistants override this to 0.7–0.9 for a balance of naturalness and consistency.

Source: SurePrompts (2026), "LLM Temperature and Sampling: The Complete Reference Guide" — documents temperature effects and practical defaults across major model providers.

Top-P and Top-K: Narrowing the Candidate Pool

Temperature reshapes the distribution, but it does not discard tokens entirely. Top-p and top-k sampling do — they restrict which tokens the model is even allowed to pick from.

Top-k sampling — restricts sampling to the k most probable tokens at each step. If k = 50, only the 50 highest-probability tokens are candidates; all others are zeroed out.

Top-p (nucleus) sampling — restricts sampling to the smallest set of tokens whose cumulative probability exceeds p. If p = 0.9, the model finds the minimal set of highest-probability tokens that together account for 90% of the probability mass, and samples only from that set.

Top-p is generally preferred over top-k because it adapts to the shape of the distribution. When the model is confident (one token has 70% probability), top-p with p=0.9 may include only two or three candidates. When the model is uncertain (probabilities spread evenly across many tokens), top-p casts a wider net. Top-k at k=50 would include 50 candidates in both cases, regardless of whether 48 of them are nearly irrelevant.

Min-p is a newer variant (2024–2025) that sets a minimum probability threshold relative to the most probable token, rather than a fixed cumulative mass. Research in 2025 showed min-p with values of 0.05–0.1 consistently outperforms top-p at higher temperatures, where top-p's flat-distribution flaw causes it to include too many low-quality candidates.

Source: arxiv.org (2025), "Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs" — demonstrates min-p's advantage over top-p at elevated temperatures.

Non-Determinism Beyond Sampling: Hardware and Batching Effects

Even when temperature is set to zero — which in theory means "always pick the most probable token" — AI models in production are not perfectly deterministic. Two identical queries run seconds apart can still return slightly different outputs. The sources of this residual variation are less obvious.

Floating-point non-associativity is the primary culprit. Modern GPUs parallelize the computation of logits by summing contributions across thousands of cores simultaneously. Floating-point addition is not associative: (a + b) + c can differ from a + (b + c) at the precision levels used in GPU arithmetic. The order in which partial sums are computed — which depends on load scheduling and may vary between runs — produces rounding differences that can change which token has the highest logit by a margin smaller than 0.001.

Batching effects compound this. When a model inference server handles many requests simultaneously, it often batches them together for efficiency. Two requests made a millisecond apart may end up in different batches with different batch sizes, changing the internal arithmetic order and producing different logit values even for identical prompts.

Kernel non-determinism — some GPU operations, particularly reduction operations used in attention layers, use non-deterministic parallel algorithms that sacrifice reproducibility for speed. Even with the same random seed and temperature=0, these operations can produce subtly different results.

This means that achieving true determinism requires: fixed random seed, fixed batch size (batch size of 1), deterministic GPU kernels (which reduce throughput by 20–40%), same hardware generation, and same software version. Production deployments rarely satisfy all these constraints.

Source: arxiv.org (2025), "Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference" — documents floating-point non-associativity and batching as the primary sources of temperature-0 variation.

Seeds: When You Need Reproducibility

Most model APIs expose a seed parameter — an integer that initializes the random number generator used during sampling. With the same seed, temperature, top-p, and model version, most API providers will return identical outputs for identical prompts, at least within a single deployment version.

OpenAI's API added seed support in late 2023. Anthropic's Claude API does not expose user-controllable seeds as of mid-2026 — instead, setting temperature to 0 is the closest approximation to determinism available.

Even with seeds, the practical advice from ML engineers in 2026 is to design evaluation pipelines to tolerate semantic equivalence rather than byte-exact match. Two responses that differ in word choice but convey the same factual content should be treated as equivalent for most purposes, rather than as failures of reproducibility.

Why Non-Determinism Is Often Desirable

Deterministic models — those that always pick the most probable next token — suffer from a well-documented failure mode called repetition and generic collapse. Greedy decoding (always picking the max-probability token) tends to produce safe, generic, repetitive text because high-probability tokens are common words and conventional phrasings. The model converges on the statistically average sentence rather than a useful, specific one.

The analogy here is to a coin flip versus a rigged coin. A coin that always comes up heads is perfectly deterministic but useless as a randomness source. An LLM that always picks the most probable token becomes, in a sense, a very expensive autocomplete that writes the same vanilla paragraph every time.

Sampling from the distribution forces the model to explore the probability mass more broadly, which produces more natural, varied, and often more informative responses. A question with a dozen reasonable answers — "what are some good approaches to learning a new language?" — benefits from probabilistic sampling because different runs surface different valid suggestions.

When to Push for Determinism and When to Embrace Variation

The right strategy depends on your use case.

Use low temperature (0.0–0.3) when:

• You need reproducible outputs for regression testing
• The task has a single correct answer (mathematical computation, date lookup, code with verifiable output)
• You are building a pipeline where downstream steps depend on consistent formatting

Use higher temperature (0.7–1.0) when:

• You want creative variation (brainstorming, content drafts, idea generation)
• You are sampling multiple outputs and picking the best (majority vote, self-consistency)
• The task benefits from exploring diverse framings

Self-consistency is a technique that exploits non-determinism productively: generate the same reasoning problem multiple times at moderate temperature, then take the majority answer. For math and logic tasks, this improves accuracy over a single greedy decode because errors are less correlated across samples than correct reasoning chains.

Source: Wang et al. (2023), "Self-Consistency Improves Chain of Thought Reasoning in Language Models," Google Research — demonstrates that majority-voting across multiple sampled reasoning paths outperforms single greedy decoding on arithmetic and symbolic reasoning benchmarks.

Greedy Decoding vs. Sampling — A Worked Example

Walking through a single generation step in slow motion makes the distinction between greedy and sampled outputs concrete. Consider a simple prompt: "The capital of France is". The model computes logits for every token in its vocabulary. After softmax, the top of the distribution might look something like this:

Token	Approximate probability
"Paris"	94.2%
"located"	1.8%
"the"	0.9%
"a"	0.6%
all other tokens	2.5% combined

Under greedy decoding (temperature = 0), the model selects "Paris" every single time without exception. The outcome is deterministic because there is no sampling — the argmax of the logit vector is taken directly. For a factual completion like this one, greedy decoding is ideal: the correct answer dominates the distribution so thoroughly that sampling adds only risk.

Now consider a more open-ended prompt: "A good approach to learning a new language is". The distribution is much flatter. Twenty or thirty different tokens might each have probabilities between 1% and 8%, because many continuations are plausible. Under greedy decoding, the model might select "consistency" or "immersion" as the single most likely token — but "practice," "exposure," "finding," and "speaking" are all nearly as probable. The model committed to the highest-probability token and locked in one trajectory. Every subsequent token is then conditioned on that choice, propagating the initial selection forward.

Under temperature sampling at 0.9, the model rolls the weighted die. On one run it picks "immersion," generating a response about deep immersive practice. On the next run it picks "finding," generating a response about finding a conversation partner. Both are valid, useful responses. The sampling produces diversity that greedy decoding discards.

The divergence compounds across steps. By the fifth or sixth token, two sampled runs have taken different trajectories and will generate sentences with entirely different structures, even though they started from the same prompt. This is why even a small change at the first token — where probabilities are relatively close — can produce responses that read as if they came from different writers.

The practical takeaway from the worked example is that greedy and sampled outputs converge when the model is confident (one token probability above roughly 85%) and diverge when the model is uncertain. For factual queries with a single correct answer, use low temperature. For creative, exploratory, or open-ended tasks, the probability mass is genuinely spread across many good continuations and sampling is the appropriate choice.

Source: Holtzman et al. (2020), "The Curious Case of Neural Text Degeneration," ICLR — analyzes why greedy and beam-search decoding produce degenerate outputs on open-ended generation tasks, and introduces nucleus (top-p) sampling as the preferred alternative.

Why Even "Deterministic" Settings Can Drift

Setting temperature to zero and pinning a seed is not a guarantee of stable outputs over time, even with an identical prompt. Production AI systems introduce several sources of drift that operate above the token-sampling level — changes that can silently alter what a "deterministic" model returns without any change to the user's prompt.

Provider-side model updates are the most significant source. OpenAI, Anthropic, Google, and other API providers update their deployed models continuously. A model labeled "gpt-4o" in January and "gpt-4o" in August is not the same set of weights — it may have received safety patches, fine-tuning updates, or post-training alignment adjustments. The API version string does not change, but the underlying weights do. A seed-pinned, temperature-0 prompt that returned one answer in January can return a different answer in August, not because of sampling variance but because the model has changed. Version-pinning — requesting a specific dated model snapshot like "gpt-4o-2024-08-06" — is the only protection against this, and even then, providers reserve the right to update within a named version for safety reasons.

Silent A/B routing is a related issue. Large inference providers route traffic across multiple model replicas and sometimes across quietly different model variants for testing. Two requests made by the same user thirty seconds apart may be handled by different server instances running subtly different model versions. The user sees the same API endpoint name; the backend is not identical. This is industry-standard practice for production services, not a deliberate deception, but it means that apparent non-determinism at temperature 0 can reflect infrastructure routing rather than sampling randomness.

Context-window truncation affects long prompts. If a prompt plus conversation history exceeds the model's context window, most providers silently truncate the oldest content. A conversation that fit in the context window last month may exceed it after several more turns have been added. The model answering today does so without access to context it had access to previously, producing a different answer not because it "forgot" in a human sense but because the information was not in the input. Applications that manage long conversation histories need explicit strategies — summarization, retrieval, selective pruning — to avoid silent truncation changing behavior.

System-prompt changes at the application layer are often overlooked when users report "the AI changed." ChatGPT's system prompt, which shapes behavior before any user message, has changed multiple times since the product launched. Enterprise deployments set custom system prompts that can be updated by the organization's IT team without notifying end users. If an application's behavior changes but the user's message is identical, the system prompt is the first thing to check.

The combined effect of these four factors — model updates, routing variance, truncation, and system-prompt drift — means that reproducibility in production AI is fundamentally harder than reproducibility in a research notebook. A notebook with a fixed seed, fixed model weights loaded from disk, and batch size of 1 on a single GPU can achieve byte-exact reproducibility. A production API call cannot. Engineering teams that need auditability or regression guarantees must either self-host model weights with strict version control or build evaluation pipelines that compare semantic equivalence rather than string equality.

Source: OpenAI Platform Documentation (2024), "Reproducibility" — describes seed support, model versioning behavior, and the limits of reproducibility guarantees in the OpenAI API.

Frequently Asked Questions

Q: Does setting temperature to 0 make an AI model fully deterministic?
A: Not in practice. Temperature 0 (greedy decoding) dramatically reduces variance, but GPU floating-point non-associativity and batching effects mean identical prompts can still produce slightly different outputs in production. True determinism also requires a fixed seed, consistent batch size, and deterministic kernel settings, which most production deployments do not apply by default.

Q: What temperature does ChatGPT use by default?
A: OpenAI's public API defaults to temperature 1.0. ChatGPT's web product likely uses a system-level configuration in the 0.7–1.0 range, though OpenAI does not publish the exact value. Users can override temperature in API calls but not in the standard chat interface.

Q: Why do I sometimes get very different answers and sometimes nearly identical ones?
A: The variance depends on the probability distribution at each step. If one token is dominant (probability 95%+), temperature has little effect — nearly any temperature setting will pick the same token. Variance appears most when several tokens have similar probabilities, which happens more often on open-ended creative or opinionated questions than on factual queries.

Q: Is top-p or temperature more important to adjust?
A: Temperature is the primary lever — it controls the overall spread of the distribution. Top-p is secondary, acting as a ceiling to prevent very low-probability tokens from being selected even at high temperatures. Most practitioners start by adjusting temperature and leave top-p at 0.9–0.95 unless they have a specific reason to change it.

Q: Can two different AI models give different answers for reasons other than sampling?
A: Yes. Different models have different training data, different fine-tuning, different RLHF preferences, and different architecture sizes. Even at temperature 0, GPT-4 and Claude 3.5 Sonnet will often give different answers to the same question because they learned different statistical associations — not because of sampling randomness.

Q: If I pin a seed, will the model always give exactly the same answer?
A: For most API providers that support seeds (OpenAI, Mistral, some others), yes — with the same seed, temperature, and model version, the output is reproducible within that version. However, model updates can silently change outputs even for pinned seeds, so version-pinning is also necessary for long-term reproducibility.

---

Read next:

• How AI Models Are Trained: The Full Pipeline
• How Large Language Models Work, Explained
• How to Write AI Prompts That Actually Work