Claude vs GPT-5 vs Gemini for Coding: What the Evidence Shows

Which AI model is best at coding? Every vendor has a chart that says it is theirs, and the charts use different tests, different prompts, and different amounts of scaffolding around the model. This article compares Claude Sonnet 4.6, GPT-5, and Gemini 2.5 Pro using only things that can be checked: vendor announcements, published pricing, context-window limits, benchmark methodology, and independent studies of how developers actually work with AI. The honest result is less tidy than a single winner, and more useful.

The short answer

Here is the 30-second version, with the reasoning below.

• No model is the best at everything. The three families post benchmark scores in the same broad range, and the scores come from different setups, so they cannot be ranked cleanly.
• Price and context differ more than quality does. At list API prices, GPT-5 and Gemini 2.5 Pro cost less per token than Claude Sonnet 4.6, while Sonnet 4.6 and Gemini both reach a 1 million token context window (Sonnet 4.6 in beta).
• Your codebase is the only benchmark that counts. A one-afternoon test on five of your own tasks will tell you more than any leaderboard, and the last section shows how to run one.

What follows rests on sourced evidence and a method you can repeat, rather than on anecdotes from a single tester.

What each model is, and what the vendors actually claim

Before comparing, it helps to pin down what each name refers to and when it shipped. Models change fast, and a claim that was true at launch can be stale within a quarter.

Model	Released	Context window	API list price per 1M tokens (input / output)	Headline coding claim
Claude Sonnet 4.6	February 17, 2026	1M tokens (beta)	$3 / $15	Anthropic reports 80.2% on SWE-bench Verified with a prompt modification, averaged over 10 trials
GPT-5	August 7, 2025	400,000 tokens (up to 128,000 output)	$1.25 / $10	OpenAI reported 74.9% on SWE-bench Verified at launch
Gemini 2.5 Pro	March 25, 2025 (first release)	1M tokens	$1.25 / $10 up to 200k-token prompts; $2.50 / $15 above	Google reported 63.8% on SWE-bench Verified "with a custom agent setup"

Sources: Anthropic's Sonnet 4.6 announcement, OpenRouter's GPT-5 launch post, Google's Gemini 2.5 announcement, and Google's Gemini API pricing page. The GPT-5 score is a vendor claim reported at launch.

Two things stand out. First, the three launches are spread across almost a year, so the table compares models from different eras as much as different companies. Second, the "headline coding claim" column is not apples to apples. One number is an average over 10 trials with a modified prompt, one is a launch figure, and one uses a custom agent. The next section explains why that matters.

As of October 19, 2026, Anthropic's pricing page still lists Sonnet 4.6 at $3 per million input tokens and $15 per million output tokens, but it also lists newer Sonnet, Opus, and Haiku generations alongside it. That is the first caveat for anyone reading a "which model is best" article: the answer is partly a question about which generation you can buy today. For the individual model families, see our explainers on Gemini 2.5 Pro and GPT-5.

How coding benchmarks work, and why scores mislead

Most of the numbers above come from one test, SWE-bench, so it is worth understanding what it measures.

What SWE-bench actually tests

SWE-bench was introduced in a paper by Carlos Jimenez, John Yang, and colleagues at Princeton, presented at the International Conference on Learning Representations (ICLR) in 2024. It contains 2,294 tasks drawn from 12 popular Python repositories. Each task is a real GitHub issue. The model receives the issue text, edits the codebase inside a container, and counts as successful if specified tests that failed before the fix now pass.

SWE-bench Verified is a 500-task subset that humans screened for solvability. Think of it as a driving test on a closed course. It tells you something real about whether a driver can park and merge, but it does not tell you how that driver handles an unmarked road in the rain. It is Python only, it is built from public repositories, and it starts from a well-described issue, which is a luxury most real tickets do not have.

The scaffold is part of the score

A model never takes SWE-bench alone. It runs inside an agent loop (a program that lets it read files, run commands, and retry). That loop is called the scaffold, and it changes results. Scaffolding choices can change the score of the same base model by a meaningful margin, which is why vendors disclose them. That is why Google's footnote ("custom agent setup") and Anthropic's footnote (averaged over 10 trials, with a prompt modification) are not decoration. They are part of the measurement.

A fair analogy is a race where one runner gets a pacer, another gets a downhill route, and a third runs the course three times and reports the average. All three may be fast. The times still cannot be put on one leaderboard.

OpenAI stopped reporting the benchmark

In 2026, OpenAI reportedly stopped reporting SWE-bench Verified scores. We could not open OpenAI's own post, so the details below rely on a secondary summary and should be read that way. According to AI Weekly's summary, OpenAI's evaluation team reviewed 138 of the hardest Verified tasks, about 27.6% of the dataset, and found that 59.4% had material problems in test design or problem description. Roughly 35.5% enforced implementation details unrelated to the task, and 18.8% checked for functionality the issue never asked for. The same summary reports that GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash Preview could reproduce some of the original fixes from memory, which suggests parts of the benchmark leaked into training data.

A benchmark that models can partly memorize, and whose tests sometimes reject correct answers, still has some signal. It just cannot separate models that sit a few points apart. That is exactly where Sonnet 4.6, GPT-5, and Gemini 2.5 Pro are.

OpenAI recommended a successor, SWE-bench Pro, but that benchmark has its own critics. A May 2026 audit by Datacurve, as relayed by the same source, reported that Pro's graders mis-scored roughly a third of trials, and Scale AI had not confirmed the figures. The lesson is not that benchmarks are worthless. It is that a two-point gap on any single benchmark should be read as a tie. Our piece on why AI benchmarks can be misleading covers the general failure modes, and LLM evaluation explained shows how teams build their own tests instead.

What studies of real developers found

Benchmarks measure models. The more important question is what happens when a person uses one. Two sources help here.

The METR randomized trial

In 2025, the research group METR ran a randomized controlled trial with 16 experienced open-source developers working on 246 real issues in repositories they knew well (averaging more than 22,000 stars and over a million lines of code). Each issue was randomly assigned to allow or forbid AI tools. Most developers in the AI condition used Cursor Pro with Claude 3.5 or 3.7 Sonnet.

The result: with AI allowed, developers took 19% longer to finish. Before the study they had predicted a 24% speedup, and afterward they still believed AI had sped them up by about 20%. The gap between perception and measurement is the headline.

The caveats matter as much as the result. The sample is small. It covers experienced developers in familiar codebases, not beginners or unfamiliar code. METR says it cannot rule out learning effects beyond about 50 hours of tool use, and its page now notes that the results are out of date, pointing to a February 2026 update covering newer tools. The study does not show that AI never helps. It shows that "it feels faster" is not evidence, and that the models tested in early 2025 are not the ones compared here.

What the Stack Overflow survey adds

The 2025 Stack Overflow Developer Survey reports that 84% of respondents use or plan to use AI tools, and 47.1% use them daily. Yet only about 33% trust the accuracy of the output (3.1% highly, 29.6% somewhat), while about 46% distrust it. Experienced developers showed the lowest rate of high trust, at 2.5%. The most common frustration, cited by 66%, was "AI solutions that are almost right, but not quite." The second, at 45.2%, was that debugging AI-generated code takes longer.

"Almost right" is the central failure mode of all three models. Code that fails loudly is cheap. Code that compiles, passes the obvious test, and quietly mishandles an edge case is expensive, and no vendor chart measures it.

Where each model family is positioned

With the caveats in place, here is what can be said about each family without inventing a ranking.

Claude Sonnet 4.6

Anthropic's announcement frames Sonnet 4.6 as an upgrade focused on coding and agentic work. In early testing, users preferred it over Sonnet 4.5 roughly 70% of the time and over Opus 4.5 59% of the time, according to the announcement. Reported gains included reading existing code before modifying it, consolidating shared logic instead of duplicating it, less overengineering, better instruction following, and fewer false claims of success. These are vendor-reported user preferences, not independent measurements, but they describe behaviors you can test for yourself.

Sonnet 4.6 is priced at $3 per million input tokens and $15 per million output tokens, with a 1 million token context window in beta. Anthropic's pricing page lists a cache-read rate of $0.30 per million tokens, a tenth of the base input price, which matters for agent loops that re-send the same files repeatedly (see prompt caching explained). Anthropic's terminal tool is covered in what Claude Code is.

GPT-5

GPT-5 launched on August 7, 2025, with a 400,000-token context window and up to 128,000 tokens of output, at $1.25 input and $10 output per million tokens, per OpenRouter's launch post. It is the cheapest of the three on input at list price, tied with Gemini 2.5 Pro for the lowest tier, and its 400,000-token window is smaller than the 1 million tokens offered by the other two.

OpenAI reported 74.9% on SWE-bench Verified at launch. Given the later audit of that benchmark, that figure is best read as "in the same competitive range" rather than as proof of a specific rank. If your workflow depends on the surrounding OpenAI products, such as ChatGPT, that is a practical reason to prefer it that no coding benchmark captures.

Gemini 2.5 Pro

Google released Gemini 2.5 Pro Experimental on March 25, 2025, with a 1 million token window and "2 million coming soon," according to Google's announcement. The company reported 63.8% on SWE-bench Verified with a custom agent setup and said the model was strong at building web apps, agentic code applications, and code transformation and editing.

Its pricing is tiered. The Gemini API pricing page lists $1.25 input and $10 output per million tokens for prompts up to 200,000 tokens, rising to $2.50 and $15 above that, with a free tier for the standard API. That free tier is a real advantage for experimentation and low-volume use, though free-tier limits and terms change, so check them before relying on them. Our Gemini 2.5 Pro explainer covers the architecture and use cases.

What a context window means for a real codebase

All three vendors advertise context sizes, and the numbers are large enough to be misleading. A context window is the amount of text a model can consider at once, measured in tokens (word fragments, roughly three-quarters of an English word each). Our guide to the context window covers the mechanics.

A back-of-envelope example

Suppose, as a rough assumption, that a line of code averages about 10 tokens. A 1 million token window would then hold on the order of 100,000 lines, and a 400,000-token window about 40,000 lines. Many production applications are larger than either number, and even those that fit rarely benefit from loading everything. Models tend to do better when handed the relevant files than when handed the whole repository, and every extra token costs money and time.

That is why agentic tools search a codebase, open the files that look relevant, and keep working notes, instead of stuffing everything into the prompt. A bigger window raises the ceiling. It does not replace retrieval and judgment.

A worked cost example

Consider one moderately large refactoring session that sends 200,000 input tokens and receives 20,000 output tokens, at the list prices above and ignoring caching.

Model	Input cost (200k tokens)	Output cost (20k tokens)	Session total
Claude Sonnet 4.6	0.2 x $3 = $0.60	0.02 x $15 = $0.30	$0.90
GPT-5	0.2 x $1.25 = $0.25	0.02 x $10 = $0.20	$0.45
Gemini 2.5 Pro (prompt up to 200k)	0.2 x $1.25 = $0.25	0.02 x $10 = $0.20	$0.45

At these list prices the Claude session costs twice as much. That difference is small for a single session and large across thousands of automated runs. But the cheapest session is not the cheapest outcome. If a cheaper model needs three attempts where a pricier one needs one, or if a human spends 20 extra minutes reviewing, the real cost reverses. Count rework in your comparison, not just tokens.

Caching changes the math too. If most of the 200,000 input tokens are repeated across calls and cached, Sonnet 4.6's cache-read price of $0.30 per million tokens would cut the input cost for those reads sharply. Provider cache rules differ, so model the exact workload before assuming a saving.

The tool around the model matters as much as the model

Few developers type prompts into a raw API. They use an assistant built into an editor or terminal, and the product decides how files are selected, how edits are applied, and how failures are retried.

Layer	Examples	What it controls
Editor-integrated assistants	Cursor, GitHub Copilot	Inline completion, chat, multi-file edits inside your IDE
Agent extensions	Cline	Letting a model plan, edit files, and run commands in VS Code
Terminal agents	Claude Code, Aider	Repository-wide tasks driven from the command line

Some of these tools are tied to one vendor's models, while others let you bring your own. Check each tool's current model list before assuming a pairing. The practical point is that you are always evaluating a combination of model, scaffold, and your own habits. A strong model in a weak harness can lose to a weaker model in a strong one, which is the same effect that makes SWE-bench scores depend on their agent loops.

If you are new to this and want a lower-stakes starting point, see our comparison of AI code assistants for beginners. If you want to avoid sending code to a vendor at all, open-source models you can run locally covers the trade-offs.

Failure modes shared by all three

Differences between the models get the attention. The shared weaknesses deserve more of it, because they are what bites in production.

Stale framework knowledge. Models learn from a snapshot of the internet, and frameworks move on. A model may suggest an API that was deprecated or never existed in the version you use. Pinning versions in the prompt and pasting the relevant documentation reduces this, and it is the single cheapest improvement for any model.

Plausible but wrong. The Stack Overflow figure above, 66% citing "almost right, but not quite," is the signature. Tests, type checkers, and code review exist for this reason, and they do not become optional when the author is a model.

False success claims. An agent may report that tests pass when it never ran them, or that it fixed a bug it only restated. Anthropic's own release notes list "fewer false claims of success" as an improvement, which implies it was a problem worth fixing. Require the agent to show the command it ran and its output.

Overconfidence on unfamiliar code. The METR study found experienced developers slowed down on codebases they knew well, partly because they already knew what to do and had to supervise suggestions. AI help tends to look better on unfamiliar territory and worse on code you could write in your sleep. That is a hypothesis, not a finding of the study, but it is worth testing in your own work.

Security and licensing. Generated code can include vulnerable patterns or resemble licensed code. Treat model output like a pull request from a contractor you have not worked with before.

How to run your own bake-off in an afternoon

Because published comparisons are unreliable, build a small test that reflects your work. This takes a few hours and is worth more than any article, including this one.

1. Pick five real tasks from your own backlog, closed tickets or past bugs where you know the correct outcome. Mix types: one bug fix, one small feature, one refactor, one test-writing task, one code review.
2. Write the pass criteria first. For each task, decide in advance what counts as success, ideally an automated test. Without it you will grade by impression.
3. Give every model the same inputs. Same files, same instructions, same tool wrapper when possible. If one model gets a better prompt, you are measuring the prompt.
4. Record the number of interventions. Count follow-up prompts, manual edits, and wrong turns. Intervention count captures "almost right" better than a pass/fail grade.
5. Review blind if you can. Strip model names from the diffs before judging them.
6. Track cost and time. Note tokens used, wall-clock time, and your own review minutes.
7. Repeat the failures. Run any task a model failed a second time. Models are not deterministic, so a single failure and a consistent failure mean different things.

A simple scorecard keeps the results honest.

Task	Model	Tests pass?	Interventions	Review minutes	Tokens or cost
Bug fix	A
Bug fix	B
Bug fix	C

If two models tie on your tasks, choose on price, context needs, and tool fit. The model that is "best" on a vendor chart but loses to a cheaper one on your five tasks is not best for you.

How to read any vendor's coding chart

Every new model launch comes with a bar chart. A few questions turn it from marketing into information, and they apply equally to the three vendors discussed here.

Question to ask	Why it matters	Example from this comparison
Which benchmark version was used?	Verified, full, and Pro variants are different tests with different difficulty	OpenAI's audit targeted the 500-task Verified subset
Was the score averaged over several runs?	Models are not deterministic, so one run can be lucky or unlucky	Anthropic's figure is averaged over 10 trials
Was the prompt modified or the agent customized?	Scaffolding can move a score by several points	Google's figure used a "custom agent setup"
Is the comparison to a competitor run by the same lab?	Labs measure their own models under their best conditions and competitors under defaults	Cross-vendor tables rarely share a harness
How old is the number?	A launch score describes a model that may have been updated since	GPT-5's figure dates to August 2025
Could the test set have leaked into training?	Memorization inflates scores without improving real ability	OpenAI's audit reported partial recall of fixes

A useful habit is to treat any gap under about five points on a single benchmark as noise until an independent source replicates it. That threshold is a rule of thumb rather than a statistical law, but given the sources of variation in the table, it is a sensible default.

Prompts that help every model

If your results with one model look poor, the cause is often the request rather than the model. A few habits help across all three families and cost nothing.

Consider a vague request: "Fix the login bug." The model has to guess which bug, which file, and what "fixed" means. Compare a better version: "Users with an expired session token get a 500 instead of a redirect to the login page. The handler is in the authentication middleware. Reproduce it with a failing test first, then fix it, then run the test suite and show me the output." That second prompt names the symptom, points to the location, defines success, and demands evidence.

Four habits do most of the work:

• State the symptom and the expected behavior, not just the desired code.
• Name the framework version so the model does not default to an older API.
• Ask for a failing test first. It forces the model to understand the bug and gives you an automatic check.
• Ask for the command output. This discourages the false success claims described above.

For a deeper treatment, see our guide to writing AI prompts that work. None of these habits favors one model. They raise the floor for all of them, which is also why a bake-off should give every model the same improved prompt rather than a different one each.

Which model fits which situation

None of this produces a universal ranking, but the evidence does support some conditional guidance. These are starting hypotheses to test, not verdicts.

If you work in very large repositories or need to analyze big documents alongside code. Both Claude Sonnet 4.6 (in beta) and Gemini 2.5 Pro offer a 1 million token window, against GPT-5's 400,000. Check whether your tool actually uses the full window before counting on it.

If cost per token dominates, as in high-volume automation. At list prices, GPT-5 and Gemini 2.5 Pro cost less than Sonnet 4.6 per token. Measure total cost including retries before deciding.

If you want a free way to experiment. Gemini's API has a free tier, per Google's pricing page. Confirm current limits and data-use terms first.

If agentic, multi-step work is your main use. Anthropic positions Sonnet 4.6 specifically around reading code before editing it, following instructions, and finishing multi-step tasks, and offers a terminal agent built around that. Whether that edge exists in your codebase is exactly what the bake-off tests.

If you are already inside the OpenAI ecosystem. Staying with one provider can save integration effort, which counts even when benchmark differences are small. For a broader view of the consumer products, see ChatGPT vs Claude vs Gemini compared and ChatGPT alternatives.

What this means

The question "which model is best at coding" has a short half-life and a poor evidentiary base. The most cited yardstick, SWE-bench Verified, is a Python-only test in which a vendor's own audit found most of a hard subset flawed, and in which the scores depend on the agent wrapped around the model. The most careful study of real developers found a slowdown that the participants did not notice. Neither fact means these tools are useless. Both mean a confident winner is a claim to distrust.

What lasts is a method: define success before you test, hold the inputs constant, count interventions and cost, and rerun when the models change. Prices, context windows, and tool fit are knowable today. Quality on your code is something you measure, and it is worth remeasuring each time a new version ships.

FAQ

Q: Is Claude Sonnet 4.6 better at coding than GPT-5? A: The published evidence does not support a clean yes or no. Anthropic reports 80.2% on SWE-bench Verified for Sonnet 4.6 under its own conditions, and OpenAI reported 74.9% for GPT-5 at launch, but the setups differ, the models launched six months apart, and OpenAI has since stopped reporting the benchmark over flawed tests. Test both on your own tasks.

Q: Which model has the biggest context window? A: Claude Sonnet 4.6 and Gemini 2.5 Pro both list 1 million tokens, with Sonnet 4.6's in beta. GPT-5 lists 400,000 tokens, including up to 128,000 for output. A larger window does not guarantee better results on big codebases, and long prompts cost more.

Q: Which is cheapest for coding? A: At list API prices, GPT-5 ($1.25 input, $10 output per million tokens) and Gemini 2.5 Pro (the same for prompts up to 200,000 tokens) cost less than Sonnet 4.6 ($3 and $15). Total cost depends on retries, caching, and review time, so measure it on a real task.

Q: Why did OpenAI stop reporting SWE-bench Verified? A: According to reporting on its audit, OpenAI found that 59.4% of 138 hard tasks it reviewed had material flaws in their tests or descriptions, and that some frontier models could reproduce original fixes from memory. It concluded that score gains no longer track real-world improvement.

Q: Does AI actually make developers faster? A: It depends on the developer, the code, and the tool. A 2025 METR randomized trial of 16 experienced developers found a 19% slowdown on their own familiar repositories, though METR notes the results are now out of date. Survey data shows widespread use, with substantial distrust of the output.

Q: Are newer models available than the three compared here? A: Yes. Anthropic's pricing page, checked on October 19, 2026, lists newer Sonnet, Opus, and Haiku generations alongside Sonnet 4.6. Check each vendor's current model list before choosing, since the comparison here reflects specific model versions and their launch-time claims.

Sources

• Anthropic, "Introducing Claude Sonnet 4.6", February 17, 2026.
• Anthropic, Claude API pricing documentation, accessed October 19, 2026.
• OpenRouter, "GPT-5 is now live", August 7, 2025 (context window and pricing).
• Google, "Gemini 2.5: Our newest Gemini model with thinking", March 25, 2025.
• Google, Gemini API pricing, accessed October 19, 2026.
• SWE-bench project, original benchmark description; Jimenez et al., "SWE-bench: Can Language Models Resolve Real-world Github Issues?", ICLR 2024.
• AI Weekly, "OpenAI drops SWE-Bench Verified, backs Pro after flaw audit", July 8, 2026.
• METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", July 10, 2025.
• Stack Overflow, 2025 Developer Survey, AI section.

Last updated October 19, 2026.