Why AI Coding Tools Need Repo Context

Ask an AI coding assistant to "fix the bug in the checkout flow" with no other information, and it will guess based on generic patterns from its training data. Ask the same question inside a tool that has indexed your repository, and it can locate the actual checkout.ts file, trace which function calls which, and match the fix to your project's existing error-handling conventions. The gap between those two answers is almost entirely about what code the model can see when it generates a response, not about how smart the underlying model is. This article walks through why that gap exists, the strategies tools use to close it, and where each one still breaks.

The context window is a hard limit, not a design choice

Every large language model has a context window, the maximum amount of text it can process in a single request. It is measured in tokens, which are chunks of text, often word fragments or short pieces of code. Anthropic's documentation describes the window as the model's "working memory" and notes that it covers everything in the request: the system prompt, every message, tool results, and the output the model generates (Anthropic, context windows). Windows have grown quickly. As of October 2026, that documentation lists many current Claude models at up to 1 million tokens, while older ones sit at 200,000.

A million tokens sounds enormous, and for a small project it is. For a large one it is not. As a rough illustration, if an average line of source code costs about 10 tokens (a ballpark that varies with language and style), a repository of 500,000 lines would come to around 5 million tokens before you add tests, configuration, lockfiles, and documentation. No single request holds that, and even if one did, it would cost more and take longer than the question deserves. For a fuller treatment of the basics, see context windows explained.

There is also an architectural reason that windows do not simply grow forever. Transformers (the architecture behind essentially every current coding assistant) use attention, a mechanism that lets each token weigh every other token, as introduced in Vaswani et al.'s 2017 paper. Standard self-attention compares every token with every other token, so the work grows with the square of the input length. Engineers have found many ways to soften that cost, but longer inputs remain more expensive than shorter ones. For a plain-language walkthrough, see how transformers pay attention to words.

Why a bigger window does not end the problem

Fitting the code in is only half the battle. Models use long inputs unevenly. A 2023 study titled "Lost in the Middle" found that performance is often highest when relevant information sits at the beginning or end of the input and degrades significantly when it sits in the middle, even for models built for long contexts (Liu et al., 2023). Anthropic's own documentation says the same thing in practical terms: more context is not automatically better, because accuracy and recall degrade as token count grows, which it calls context rot, and so curating what is in context matters as much as how much space is available.

So even a generous budget forces a choice: which subset of a codebase goes into the prompt for a given request? That selection problem, more than raw model capability, is what "repo context" engineering exists to solve.

A 2025 arXiv paper on structure-aware code chunking (cAST) reported that cutting code along syntax-tree boundaries instead of naive size limits improved retrieval recall by 4.3 points on the RepoEval benchmark and generation accuracy by 2.67 points on SWE-bench (Zhang et al., 2025). The result suggests that some retrieval failures start before retrieval does, at the moment the code is cut into pieces.

What "repo context" actually includes

When people say a tool "knows your repo," they usually mean a mix of several different things. Treating them separately explains why tools behave so differently.

• Source files. The code itself, including the file you are editing and the files it imports.
• Structure. Directory layout, module boundaries, and which files depend on which.
• Conventions. Naming patterns, error-handling style, test layout, and formatting rules that are visible across many files rather than in any one.
• Dependencies. The versions pinned in a lockfile, which decide what APIs actually exist.
• Instructions. Files that a developer writes for the tool, such as a project notes file loaded at the start of every session.
• History and state. Recent edits, open tabs, failing tests, and sometimes version-control history.

Anthropic's engineering team describes Claude Code as a hybrid of the last two ideas: it loads project instruction files up front for speed, and then uses primitives like glob and grep to retrieve other files just in time (Anthropic, effective context engineering). The upfront part is cheap and predictable. The on-demand part is flexible but depends on the model deciding well what to look for.

Four strategies tools use to work around the limit

There is no single accepted way to give a model repo context, and tools genuinely differ in approach, not just marketing language. Four strategies show up across current products, and most serious tools combine more than one.

Strategy 1: embeddings and retrieval-augmented generation

Retrieval-augmented generation (RAG) converts chunks of code into embeddings, numerical vectors that place semantically similar text near each other in a high-dimensional space. When a developer asks a question, the tool embeds the query, finds the nearest chunks, and inserts them into the prompt. If you want the general idea first, see what RAG is and how embeddings power vector search.

Think of it like a library catalog organized by topic rather than by title. Ask for "something about retry logic" and the catalog points you to a shelf of related books, even if none of them contains the word "retry." That is a strength for vague questions and a weakness for exact ones, because the catalog knows topics, not call graphs.

Cursor's engineering blog describes one production version. The client builds a Merkle tree (a hierarchy of cryptographic hashes) of every file, compares it with the server's copy, and syncs only entries that differ. Changed files are split into syntactic chunks and turned into embeddings in the background, which the post calls "the expensive step," and embeddings are cached by chunk content so unchanged chunks are not re-embedded. The post reports that this design cut median time to the first query from 7.87 seconds to 525 milliseconds when a new user reuses a teammate's index (Cursor, 2026). The same tool's current documentation also describes an exact-match search feature, called Instant Grep, that builds its index on the user's machine (Cursor docs). That pairing, semantic plus exact, is a pattern worth noticing. For more on the editor itself, see what Cursor is.

Strategy 2: symbol graphs and syntax trees

Symbol-graph and AST indexing treats code as structure, not text. A parser turns each file into an abstract syntax tree (a tree showing how the program is grammatically built), and the tool then records definitions, references, imports, and calls. A question like "who calls this function" becomes a graph lookup, which is exact where embeddings are approximate.

One widely used library for building these trees is Tree-sitter, described by its maintainers as "a parser generator tool and an incremental parsing library." Its stated goals include being fast enough to parse on every keystroke and robust enough to give useful results even in the presence of syntax errors (Tree-sitter docs). Those goals matter here, because code in an editor is broken most of the time while someone is typing.

Aider offers a well-documented example. Its "repo map" uses tree-sitter to parse each source file and find definitions and references, then runs a graph ranking algorithm on a graph where each file is a node and dependencies are edges. The map lists the most-referenced identifiers and their signatures, and it is trimmed to a token budget that defaults to 1,000 tokens (Aider blog, 2023; Aider docs). Picture a subway map rather than a street atlas: it leaves out most detail on purpose, so the traveler can see which lines connect. See also what Aider is.

Strategy 3: the working set

Sliding-window or working-set context is the simplest approach and the baseline in nearly every editor-integrated assistant. The tool includes the file you are editing, recently changed files, and open tabs in the prompt regardless of semantic relevance. It requires no indexing step and captures a sensible intuition: code you just touched is usually relevant to what you are about to ask.

Its weakness is the mirror image of its strength. It knows what you are looking at, not what you have forgotten about. If a validation rule lives in a file you have not opened this week, a working-set tool will not surface it.

Strategy 4: agentic search

Agentic search skips pre-built indexes and gives the model tools to explore the repository as a developer would: list directories, run grep or ripgrep for an exact string, open a file, follow an import. The model decides turn by turn what to look at next.

Anthropic's engineers describe this as just-in-time retrieval. Instead of loading all data up front, the agent keeps lightweight references such as file paths and queries, and loads content with tools when needed (Anthropic, effective context engineering). The approach has an obvious advantage for code: names, imports, and paths are explicit and exact, and grep finds them without approximation. It also reads the live files, so there is no stale index. For a product built around this style, see Claude Code explained. Its tradeoff is speed and cost, since each exploration step is another round trip that consumes tokens.

The four strategies side by side

Strategy	How it finds relevant code	Strength	Where it struggles	Example in the wild
Embedding / RAG	Vector similarity over pre-computed embeddings	Good for vague, natural-language questions	Misses exact symbol relationships; index can go stale	Semantic search in editors such as Cursor
Symbol graph / AST	Graph lookups over parsed definitions and references	Exact for "who calls this" and "where is this defined"	Needs a working parser; captures structure, not intent	Aider's repo map
Working set	Open and recently edited files	Zero indexing cost; matches likely intent	Blind to code you have not opened	Most editor assistants as a baseline
Agentic search	Model calls grep, file-read, and list tools in a loop	Exact matches; always reads live files	Slower; can miss code findable only by meaning	Claude Code's glob and grep

No single strategy wins outright, which is why most production tools combine at least two, typically exact search plus either embeddings or a symbol graph.

Why generic suggestions break on real, cross-file codebases

A model working with no repo context tends to invent function signatures, import paths, and variable names that look plausible but do not exist in the project. It is pattern-matching against the public code it saw in training rather than the files in front of the developer. This is the same failure covered in why AI hallucinates facts: when the true answer is not in the available context, the model fills the gap with statistically likely text, and nothing in its basic training objective checks the output against your codebase before the tokens are emitted.

A worked example: the refund function

Suppose a developer asks an assistant to "add a partial refund option." The function processRefund() lives in utils/payments.ts. It is imported under an alias in a controller in another directory, and a decorator defined in a third file enforces a fraud check before any money moves. None of that is visible from the open file.

Here is what each strategy does with the request.

1. No context. The model writes a plausible processRefund(orderId, amount) from scratch, ignoring the real signature and skipping the fraud check.
2. Working set only. If the developer has the controller open, the model sees the call site but not the decorator, so the fraud check is still missed.
3. Embeddings. A query about "refund" probably retrieves payments.ts, which helps, but the decorator's file may not look like a refund at all, so it may not rank.
4. Symbol graph. Starting from processRefund, a graph lookup finds every caller and every decorator applied to it, because those relationships are explicit in the syntax tree.
5. Agentic search. The model greps for processRefund, finds the aliased import, follows it, and reads the decorator, provided it thinks to look.

No strategy is reliably first on every kind of question, which is why the strongest tools use more than one.

How does an AI coding tool decide which files matter for a specific question?

Most tools combine two signals: a working set of recently opened or edited files, and a retrieval step that pulls in files the developer has not touched but that are connected to the query. That retrieval may be a vector similarity search, a graph traversal over parsed symbols, or searches the model runs itself. Increasingly, tools let the model issue its own grep-style searches mid-task instead of relying only on a pre-built index.

How chunking quietly decides what the model can find

Most embedding systems cannot embed a whole file, so they cut code into chunks first. The cutting method matters more than it seems. A naive splitter that breaks every 50 lines will slice functions in half and glue the tail of one function to the head of the next. The embedding of such a chunk blends two unrelated ideas and matches neither query well.

The cAST paper tackled this directly. Its method uses the abstract syntax tree to split large nodes into smaller chunks and merge sibling nodes while staying within a size limit. The authors report gains of 4.3 points in Recall@5 on RepoEval retrieval and 2.67 points in Pass@1 on SWE-bench (Zhang et al., 2025). These are benchmark results on specific datasets, not guarantees for any given repository, but they point to a general lesson: where you draw the boundaries decides what can be found.

Cursor's description of its own pipeline points the same way, since it splits files into syntactic chunks before embedding. The structural and semantic worlds are less separate than the strategy table suggests. Many embedding tools already borrow the syntax tree to decide where to cut.

The benchmark that shows why this matters

How hard is it for a model to work in a real repository? The SWE-bench benchmark was built to measure exactly that. Introduced in 2023, it contains 2,294 software engineering problems drawn from real GitHub issues and pull requests in 12 popular Python repositories. A model receives a codebase and an issue description and must edit the codebase to resolve it, which often requires coordinating changes across multiple functions, classes, and files (Jimenez et al., 2023).

At the time of publication, the best model the authors tested, Claude 2, resolved 1.96 percent of issues. That figure is a snapshot of 2023 and has been far surpassed by later systems, so do not read it as a statement about today's models. What it still shows is why the benchmark remained useful: the task is not "write a function," it is "find the right place in a large codebase and change it correctly." That is a retrieval problem as much as a generation problem.

For a deeper look at how such scores are produced and where they mislead, see why AI benchmarks can be misleading.

The tradeoff nobody puts on the pricing page: indexing cost versus freshness

Building and maintaining retrieval infrastructure costs something. For embedding tools, every change means regenerating embeddings for the affected chunks, and Cursor's engineers call that the expensive step. For AST tools, a file that fails to parse can leave a hole in the graph until it parses cleanly again, although parsers designed for editors, like Tree-sitter, aim to produce useful results even with syntax errors.

Tools face a real choice. Re-index continuously, which costs compute and can slow the editor, or re-index on save or on a schedule, which risks suggesting code from a version of a file a teammate changed minutes ago. This explains a common complaint: an assistant confidently suggesting a function signature that existed before a refactor, because its index has not caught up.

Cursor's hash-tree approach is one answer. Because it can compare file hashes cheaply, it only reprocesses what changed, which keeps the index closer to fresh without re-embedding the whole repository. Agentic search sidesteps staleness entirely by reading the live filesystem on every call, but pays with extra round trips and latency.

Does a bigger context window make repo indexing unnecessary?

Not on its own. A larger window lets a tool include more retrieved code per request, which helps. But filling a window with an entire large codebase is costly, slow, and subject to the middle-of-context weakness described earlier. Retrieval still earns its keep by narrowing thousands of candidate files to the few that matter for the question.

Privacy, cost, and speed: the practical side

Repo context also raises questions that have nothing to do with accuracy.

Concern	Why it matters	What tools do
Privacy	Indexing means code or derived data may leave your machine	Some tools build exact-match indexes locally; Cursor's docs say Instant Grep builds and queries its index on your machine
Cost	Embedding and long prompts consume compute and tokens	Content-hash caching, so unchanged chunks are not re-embedded
Latency	Slow retrieval makes a tool feel broken	Background indexing; index reuse; token budgets such as Aider's 1,000-token default
Access control	Retrieved snippets must not expose code a user should not see	Cursor's post says results are filtered so a client only sees code whose hashes it can prove it has

Anyone choosing a tool for work should read the vendor's own data-handling documentation, because these details differ between products and change over time. The table above reflects what the sources cited here say as of their publication, not a guarantee about any product today.

Where repo context still fails, even when it's working as designed

Three failure modes recur across every strategy.

First, retrieval quality depends on phrasing and chunking. A vague request ("fix the bug") gives an embedding search a weak signal, and a function split awkwardly across a chunk boundary may never surface regardless of how relevant it is.

Second, structural indexes are language-dependent. AST parsing needs a parser for the specific language, and dynamically typed languages, heavy metaprogramming, or unusual build setups can leave gaps, edges the tool cannot resolve because the relationship is not a literal call the parser can trace.

Third, none of this replaces verification. Retrieved context reduces invention by grounding suggestions in real code, but the model is still generating text token by token and not executing the code it proposes. A suggestion that references the right function with the right signature can still use it wrongly. This is why test suites and code review remain the real safety net, and why evaluation habits matter, as discussed in LLM evaluation explained.

A fourth, subtler problem is conflicting context. Large repositories often contain old patterns next to new ones, such as two logging styles after a half-finished migration. A retrieval tool that pulls the old pattern will faithfully reproduce it. The model has no way to know which style the team now prefers unless someone has written that down, which is the job of the instruction files described earlier.

Common misconceptions

"Repo-aware means the model has read my whole codebase." Usually it has not. It has seen a selected slice, and the selection is the whole game.

"Embeddings are the standard, and grep is a hack." Neither is true. Anthropic's engineers describe grep-style retrieval as a deliberate design for Claude Code, and Cursor's current documentation describes an exact-match search alongside semantic search. Different teams have reached different conclusions, and the right answer depends on the codebase and the question.

"A bigger window will make all this obsolete." Larger windows help, but Anthropic's own documentation says more context is not automatically better, and the "Lost in the Middle" results show why.

"Grounded means correct." Grounding reduces one kind of error. It does not test the output.

What this means for choosing a coding tool

If a codebase is small enough to fit comfortably in a window with room to spare, the machinery described here matters less, and a tool that pastes the relevant open files into the prompt will do nearly as well as one running a full index. The gap widens as repositories grow past a few hundred files with real cross-file dependencies, which describes most production code.

For larger projects, the useful question is not whether a tool "has repo context," since almost every serious tool claims some version, but which strategy it uses and how it handles staleness. A tool that only tracks open tabs will miss code you have not opened. One relying only on embeddings can miss an exact relationship that grep or a symbol graph would catch instantly. Tools that combine exact search with semantic or structural retrieval, and that keep their index fresh incrementally, tend to produce fewer confidently wrong suggestions, though "fewer" is the honest bar and not "none."

Developers wiring these tools into automated pipelines often meet the same plumbing that connects agents to other systems, such as function calling and permission-scoped API access, described in how AI agents connect to business apps. Understanding where retrieval can silently fail matters well beyond the editor.

The takeaway

Repo context is not a bolted-on feature. It is a direct engineering response to two constraints: a model's window cannot hold an entire codebase, and a model without grounding in real project files defaults to plausible patterns from its training data. Embeddings, symbol graphs, working sets, and agentic search each solve part of the selection problem at a different cost. The tools that mix them, and that manage the tension between a fresh index and an affordable one, make fewer confident mistakes than any single strategy. The bottleneck in AI-assisted programming is often not the model's intelligence but whether the right fifty lines made it into the prompt.

Sources

• Anthropic. Context windows (accessed October 2026).
• Anthropic Engineering. Effective context engineering for AI agents.
• Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts.
• Vaswani, A., et al. (2017). Attention Is All You Need.
• Zhang, Y., et al. (2025). cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree.
• Jimenez, C. E., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Cursor. How Cursor indexes a codebase (January 27, 2026) and search documentation.
• Aider. Building a better repository map with tree sitter and repo map docs.
• Tree-sitter. Documentation.

---

Read next:

• Why AI hallucinates facts
• How transformers pay attention to words
• How AI agents connect to business apps

FAQ

Q: What is a context window in an AI coding tool? A: It is the maximum amount of text, measured in tokens, that a model can process in one request, including the system prompt, messages, tool results, and its own output. Even windows of up to 1 million tokens cannot hold every large codebase, so tools must select which code to include.

Q: How do coding tools decide which files to show the model? A: Most combine a working set of recently opened or edited files with a retrieval step. That step can be an embedding search for semantic similarity, a symbol-graph lookup for exact relationships such as function calls, or agentic search in which the model runs grep and file-read tools itself.

Q: Why does an AI assistant sometimes suggest a function that does not exist in my project? A: When the relevant code is not in the model's context, it falls back on patterns learned from public code and may produce a plausible but nonexistent function or import path. It is the same failure mode as hallucination in other language model output, and better retrieval reduces it without eliminating it.

Q: Does a bigger context window solve the repo-context problem? A: Not entirely. Larger windows allow more retrieved code per request, but Anthropic's documentation notes that accuracy and recall can degrade as context grows, and research on long contexts found weaker performance for information in the middle. Filling the window is also slower and costlier than retrieving what matters.

Q: What is the difference between embedding retrieval and AST-based symbol indexing? A: Embedding retrieval converts code into vectors and finds chunks semantically similar to a query, which suits vague questions but can miss exact relationships. AST-based indexing parses code into its grammatical structure and records definitions and references, which is exact for questions like "what calls this function" but needs a parser for the language.

Q: Why do some AI coding tools use grep instead of a vector database? A: Code has explicit relationships, such as imports and exact function names, that text search matches precisely, and grep reads the current files rather than a pre-built index that can go stale. Anthropic's engineers describe grep and glob as how Claude Code retrieves files just in time. The cost is extra round trips and tokens.

Q: Do I need to write instructions for the tool to get good results? A: Many tools can load a project notes file at the start of each session, and Anthropic's engineers describe this as part of Claude Code's hybrid approach. Writing down conventions the code does not make obvious, such as which of two patterns is current, gives retrieval something reliable to anchor on.