What Is llms.txt? AI-Friendly Site Files

Photo: Google DeepMind / Pexels
llms.txt is a proposed convention for a markdown file placed at a website's root (/llms.txt) that gives large language models (LLMs) a clean, curated map of the site's most important content. It was introduced by Jeremy Howard, co-founder of Answer.AI and fast.ai, on September 3, 2024 at llmstxt.org.
Think of it as "robots.txt for AI" — but with a critical difference: robots.txt restricts what crawlers may fetch, while llms.txt proactively offers a clean, well-structured version of your site for LLM consumption. Two years on, the file exists on tens of thousands of domains, but the honest 2026 answer to "does it work?" is more complicated than either its advocates or its critics initially claimed. This guide covers the format, the mechanism, the adoption data, and what Google, Anthropic, and OpenAI actually say about it.
Why llms.txt Exists
Modern websites are designed for human browsers, not language models. They include navigation, ads, JavaScript-rendered content, cookie banners, sidebars, and other noise that hurts LLM comprehension.
When an LLM (or an AI agent like ChatGPT, Perplexity, or Claude) fetches a webpage to answer a user's question, it must:
- Render or strip HTML
- Discard chrome and irrelevant blocks
- Identify the core content
- Fit it in a limited context window — the maximum amount of text a model can hold in working memory at once
This is fragile and lossy. A single product page might carry 40 kilobytes of HTML for 400 words of actual content, and a model burning tokens on <nav> markup and cookie-consent scripts has less room left for the answer itself. llms.txt solves this by letting the site owner pre-curate the clean signal instead of leaving extraction to the crawler's best guess.
The Format
llms.txt is plain markdown with a specific structure:
# Project or Site Name
> A short blockquote tagline describing the site or product.
Optional paragraph(s) of background context.
## Docs
- [Quickstart](https://example.com/docs/quickstart.md): Get started in 5 minutes
- [API Reference](https://example.com/docs/api.md): Complete endpoint documentation
## Examples
- [Tutorial: Build a chatbot](https://example.com/tutorials/chatbot.md): Step-by-step guide
## Optional
- [Changelog](https://example.com/changelog.md): Version history
Required structure:
- H1 — site/project name (required, first line)
- Blockquote — one-line summary
- H2 sections — group related links (e.g. Docs, Examples, API, Blog)
- Markdown links with context — each link followed by a colon and description
The ## Optional heading is special: items there can be skipped by an LLM working under tight context limits. This is the same idea behind progressive image loading. The browser (or here, the model) gets the essentials first and pulls in the rest only if it has budget left.
What actually happens when a model fetches it
An agent that supports llms.txt requests the file the same way a browser requests a webpage: a plain HTTP GET to /llms.txt. If the response is 200 and parses as valid markdown, the agent treats the H2 sections as a routing table — it reads link descriptions to decide which URL is worth a follow-up fetch, then requests that specific page (ideally its .md version) instead of the full HTML document. That two-step "read the index, then fetch the specific page" pattern is what saves tokens compared to crawling a whole site tree.
llms.txt vs llms-full.txt
The spec defines two files:
| File | Purpose |
|---|---|
/llms.txt |
Index — links to other resources, like a table of contents |
/llms-full.txt |
Full content of all important pages, concatenated as markdown |
llms-full.txt is useful when you want the LLM to have everything inline without needing to fetch dozens of URLs. Documentation tools like Mintlify auto-generate both. The tradeoff is size: a large docs site's llms-full.txt can run past a million tokens, which defeats the purpose for models with smaller context windows and only makes sense for tools built specifically to ingest the whole thing at once.
llms.txt vs robots.txt vs sitemap.xml
These three files are frequently confused because they all live at the site root and all target automated visitors. They serve different purposes, though, and none of them replaces the others.
Table 1. Root-level site files compared by audience, purpose, and enforcement.
| File | Audience | Purpose | Format | Enforcement |
|---|---|---|---|---|
| robots.txt | Search/AI crawlers | Restrict what may be crawled | Plain text directives | Honored by reputable bots; no legal force |
| sitemap.xml | Search crawlers | List all pages for indexing | XML | Hint for discovery |
| llms.txt | LLMs at inference time | Provide clean, curated content for AI consumption | Markdown | Voluntary convention; no enforcement |
robots.txt is opt-out (block crawling). llms.txt is opt-in (invite ingestion of curated content). A site can use all three at once, and doing so is not redundant: robots.txt governs what training and indexing crawlers may touch, sitemap.xml tells search engines what exists, and llms.txt (where it is actually read) tells an inference-time agent what matters most.
Why It Matters for GEO — in theory
Generative Engine Optimization (GEO) is the practice of optimizing content so AI engines cite it accurately. On paper, llms.txt directly supports GEO in four ways:
- Reducing hallucination risk — the AI reads a clean source instead of guessing from messy HTML
- Improving citation accuracy — links in llms.txt give the AI canonical URLs to cite
- Signaling intent — tells AI which pages the site owner considers authoritative
- Working with limited context — short, structured markdown fits in a small context window without truncation
The theory is sound. The problem, as the 2026 data below shows, is that almost none of the systems this file is meant to help are actually requesting it.
The adoption reality: what the 2026 data actually shows
This is the section most llms.txt explainers skip, and it is the one that matters most if you are deciding whether to spend an afternoon building one.
Ahrefs analyzed 137,210 domains in May 2026 using its Web Analytics and Bot Analytics data. The headline numbers:
- 28% of domains studied (roughly 38,000 sites) publish a valid llms.txt file.
- 97% of those published files received zero requests during the month studied.
- Of the 3% that did get traffic, 96% of the requests came from automated bots, not humans.
- Among bot requests, the largest categories were SEO audit tools (21.7%), unidentified bots (14.9%), general web crawlers (13.1%), and tech-profiling tools (11.6%) — together 60.7% of all traffic to the file.
- AI-related bots (agents, training crawlers, assistants, and retrieval bots combined) accounted for just 19.5% of requests — meaning the file is being fetched far more often by SEO tooling checking whether it exists than by the AI systems it was built for.
Put differently: most llms.txt files are being read by tools built to audit llms.txt adoption, not by the language models the file is supposed to serve.
What Google actually says about it
Google has been unusually direct about this file, which is notable given how rarely the company comments on individual, non-standard proposals. Google's John Mueller addressed the question on June 2, 2026, after SEO consultant Lily Ray pressed him on conflicting guidance. His response: llms.txt is "purely speculative for now," adding that "the file has existed for years, yet none of the AI systems use it — what does it mean?" Mueller has separately compared the enthusiasm around llms.txt to the discredited keywords meta tag from the early 2000s SEO era, a comparison meant to signal that a file's existence does not mean any system is acting on it.
Mueller did leave an opening: "When an AI platform that brings you clients complains that it needs the file for your site, then I'd recommend taking the time to create one." In other words, Google's position is not "never build one," it's "don't build one speculatively — build it if a specific system you care about actually asks for it." Mueller also flagged interest in an emerging alternative, WebMCP, which defines structured, callable actions a page exposes to agents rather than a static descriptive file — see the Alternatives section below.
Who actually reads it — vendor by vendor
The vendor picture in 2026 is uneven, and treating "AI" as one bloc obscures real differences:
- Anthropic publishes its own llms.txt (docs.anthropic.com/llms.txt) and references the convention in its developer-facing "writing for agents" guidance, making it the clearest official adopter among frontier labs.
- Perplexity has said it retrieves llms.txt where present and can use it to help prioritize which pages to pull from a site during a search-answer session.
- OpenAI has not adopted llms.txt as a crawling standard — its documented guidance for controlling its crawlers is robots.txt — but OpenAI does maintain its own llms.txt files for developer-facing surfaces like the Agents SDK, treating it as documentation tooling rather than a search signal.
- Google does not use llms.txt for Search or AI Overviews and has said so explicitly (see Mueller's comments above).
The pattern: llms.txt has real traction as a documentation and agent-tooling convention (docs sites, SDKs, coding agents), and essentially no traction as a search or answer-engine ranking/citation signal. Conflating the two is the single most common mistake in coverage of this file.
Early Adopters (2024-2026)
- Anthropic — docs.anthropic.com/llms.txt
- Cloudflare — developers.cloudflare.com/llms.txt
- fast.ai and Answer.AI — the originating projects
- Mintlify — docs platform that auto-generates llms.txt for hosted sites
- Stripe, Vercel, Supabase, Cursor — dev-tools companies that ship one alongside their docs
- scikit-learn, Hugging Face docs — open-source projects
A community-maintained directory at llmstxt.site tracks adopters and lists thousands of sites as of 2026, though the Ahrefs traffic data above suggests "listed as an adopter" and "actually gets fetched" are two very different things.
How to Create One
- Make a list of your 10-30 most important URLs — docs, key articles, API reference. Do not list everything; the whole point is curation, not a second sitemap.
- Group them under H2 sections that match how an LLM (or a human skimming fast) would categorize them — Docs, Guides, API, Changelog.
- Write one-sentence descriptions for each link. This is the part people skip, and it's the part that actually does the work: the description is the only signal an agent has for deciding whether to fetch that URL before it commits context budget to it.
- Provide markdown versions at each URL (append
.mdor use content negotiation) so a fetching agent gets clean text instead of full HTML with navigation and ads stripped out manually on the fly. - Save to
/llms.txtat the site root and serve it astext/plainortext/markdown. - Keep it current. A stale llms.txt pointing at moved or deleted URLs is worse than no file, because an agent that does trust it will follow dead links.
Minimum viable example for a blog:
# My SEO Blog
> Tactical SEO and AI search guides for indie developers.
## Articles
- [What is GEO?](https://example.com/geo.md): How to rank in ChatGPT and Perplexity
- [Schema markup guide](https://example.com/schema.md): JSON-LD examples that work
## About
- [About the author](https://example.com/about.md): Background and credentials
How long should llms.txt be?
As short as possible while covering the site's key content. Most functional examples run under 100 lines. Anything longer belongs in the separate llms-full.txt file or is better handled by simply linking out — a bloated index defeats the token-saving purpose the file exists for.
Does adding llms.txt cost anything to test?
No. It is a static text file with no server logic, no build step dependency, and no crawl-budget risk the way a misconfigured robots.txt can carry. That low cost is the strongest argument in its favor even amid the weak adoption numbers: publishing one is close to free, so the downside of being wrong about its future usefulness is small.
Criticisms and Limitations
llms.txt has drawn substantial criticism, and by 2026 the skepticism has hardened from "this seems premature" into data-backed pushback. The main objections:
- No enforcement, and now no meaningful usage data behind it either. LLMs aren't required to fetch or honor the file, and as of 2026, none of the major frontier labs' primary crawlers request it in significant volume, per the Ahrefs study above.
- Google has explicitly rejected it as a search signal. Not a lack of comment — an on-the-record statement that it's "purely speculative" and comparable to the deprecated keywords meta tag.
- Redundant with existing infrastructure. If a site already has clean semantic HTML, an accurate sitemap, and structured data, the incremental value of a third index file is unproven.
- Maintenance burden. Every new important page or moved URL means another file to keep in sync, and a stale llms.txt actively misleads any agent that does trust it.
- Unclear or negative ROI for most sites. With 97% of published files getting zero requests, the realistic expected traffic benefit for a typical publisher is close to zero, and the audience that does fetch it is disproportionately SEO tooling rather than end-user-facing AI.
- Risk of becoming a spam vector. Because there's no verification step, sites could stuff llms.txt with unrelated high-value links purely to game future agent-routing decisions, the same failure mode that killed the keywords meta tag two decades ago.
The counter-argument that survives this data: it costs nothing to add, it has genuine traction in one specific niche (developer documentation consumed by coding agents), and if any major AI vendor does formalize support in the future, early adopters lose nothing by having already published one. That is a hedge, not a strategy.
Alternatives and Complements
- Schema.org structured data (JSON-LD) — machine-readable metadata for entities, FAQs, and articles; unlike llms.txt, this is actively used by Google Search and has a decade of documented enforcement behind it.
- OpenGraph and Twitter Cards — already widely used for social previews, also parsed by some AI crawlers for basic page metadata.
- Clean semantic HTML —
<article>,<main>,<nav>tags help every crawler and every future extraction method, not just LLM-specific ones. - Markdown mirrors — serving
.mdversions of pages (the pattern GitHub READMEs and llms.txt links both rely on) works independently of whether llms.txt itself gets fetched. - Model Context Protocol (MCP) — a live, two-way API connection rather than a static file, letting an agent call tools and pull real-time data instead of reading a pre-baked index. See MCP explained for the full architecture.
- WebMCP — an emerging standard, referenced approvingly by Google's John Mueller in the same 2026 comments that dismissed llms.txt, which defines structured, callable actions a webpage exposes directly to browser-based agents rather than a passive descriptive index. It is earlier-stage than MCP but represents the direction Google has signaled interest in.
The throughline across every alternative on this list: the ones with real traction (structured data, MCP) have either years of documented search-engine enforcement or an active two-way protocol behind them. llms.txt has neither yet — it's a passive, unverified convention competing against tooling that vendors have explicit reasons to support.
FAQs
Q: Who created llms.txt? A: Jeremy Howard, co-founder of fast.ai and Answer.AI, proposed it on September 3, 2024 at llmstxt.org.
Q: Do ChatGPT and Claude actually read llms.txt? A: It's vendor-specific. Anthropic maintains its own llms.txt and references the convention in developer guidance, and Perplexity has said it retrieves the file where present. OpenAI's documented crawler-control guidance is robots.txt, not llms.txt, though OpenAI does publish llms.txt for some developer-facing tooling. Google has stated it does not use llms.txt for Search or AI Overviews. No major vendor has made llms.txt a required or guaranteed input.
Q: Does llms.txt actually improve AI search visibility? A: There's no strong evidence that it does for the average site. Ahrefs' May 2026 study of 137,210 domains found 97% of published llms.txt files received zero requests in a month, and of the requests that did happen, only 19.5% came from AI-related bots at all — the rest were SEO audit tools and general crawlers checking for the file's existence.
Q: Where do I put the file?
A: At the site root: https://yourdomain.com/llms.txt. Serve it with content type text/plain or text/markdown.
Q: Is llms.txt the same as ai.txt?
A: No. ai.txt was a separate 2023 proposal by Spawning AI for opting out of AI training data. llms.txt is for inviting in clean inference-time content. Different goal, different mechanism.
Q: Does llms.txt help SEO? A: Not for Google rankings — Google has explicitly said it doesn't use the file and compared the trend to the deprecated keywords meta tag. Its narrower, better-supported use case is documentation and coding-agent tooling, where products like Mintlify, Cursor, and Anthropic's own docs actively consume it.
Q: What does Google actually think of llms.txt? A: Google's John Mueller called it "purely speculative for now" on June 2, 2026, noting the file has existed for years without adoption by AI systems. He suggested building one only if a specific AI platform that sends you traffic explicitly asks for it, and pointed to WebMCP as a standard he finds more promising.
Q: Is there a real cost to publishing an llms.txt file? A: Almost none. It's a static text file with no crawl-budget risk and no build complexity, which is why many sites publish one as a low-cost hedge even without strong evidence of traffic benefit. The real cost is maintenance: a stale file with dead links can actively mislead the small number of agents that do trust it.
Q: What's the difference between llms.txt and llms-full.txt? A: llms.txt is a short index of links with one-line descriptions, meant to help an agent decide what to fetch next. llms-full.txt is the entire concatenated content of a site's key pages in one file, meant for cases where an agent wants everything inline without follow-up requests. Large documentation sites can produce an llms-full.txt running past a million tokens, which limits its usefulness for models with smaller context windows.
Q: Should a small blog or business site bother creating one? A: Given that 97% of published files get zero requests and the file has no confirmed effect on Google Search or AI Overviews, most small sites get more return from structured data (JSON-LD), clean semantic HTML, and an accurate sitemap — all of which have documented enforcement — than from an llms.txt file. It remains a reasonable low-cost addition for a documentation-heavy site or a product with its own SDK/API, where coding agents are the realistic audience.
The takeaway
llms.txt solves a real problem in theory — LLMs choke on messy HTML and limited context windows — but two years after Jeremy Howard proposed it, the systems it was built for still mostly aren't reading it. Ahrefs' 2026 data puts a hard number on what many in the SEO world already suspected: 97% of published files sit unread, and most of the traffic that does arrive comes from SEO tooling auditing adoption rather than from the AI agents the file exists to serve. Its one confirmed niche is developer documentation, where Anthropic, Mintlify, and coding-agent tooling give it a genuine (if narrow) audience. For everyone else, structured data and clean HTML remain the better-supported investment, and llms.txt is best treated as a near-free hedge rather than a visibility strategy.
Sources
- Jeremy Howard, "The /llms.txt file", llmstxt.org (September 3, 2024)
- Answer.AI announcement, "Introducing llms.txt" (2024)
- Ahrefs, "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read" (2026)
- Search Engine Journal, "Google Confirms LLMs.txt Has No Current Implementation" (2026), citing John Mueller's June 2, 2026 comments
- Mintlify documentation, "llms.txt support" (2024)
- Anthropic public llms.txt (docs.anthropic.com/llms.txt)
- llmstxt.site community directory (2024-2026)
Related Articles
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

Prompt Caching Explained — Cut LLM Costs
Prompt caching slashes LLM costs up to 90% by reusing repeated prompt prefixes. Compare Anthropic, OpenAI, and Gemini caching with code examples.

Embeddings Explained — How Vector Search Works
Embeddings turn text and images into vectors that capture meaning. Learn how vector search, RAG, and similarity work — with 2026 models compared.

Generative Engine Optimization Checklist
A practical GEO checklist for making content easier for AI systems to understand, cite, summarize, and connect to related entities.

How to Make Content Citable by AI
AI systems cite pages that make claims easy to verify and quote. This guide explains definitions, tables, sources, author signals, and topical clusters.

Answer Engine Optimization Explained
Answer engine optimization focuses on direct answers, structured sections, FAQ blocks, sourceable claims, and entity coverage for Google, Bing, Perplexity, and AI assistants.

How AI Overviews Choose Sources: What We Know
Google's AI Overviews cite pages from the existing organic index, not a separate AI ranking system. Here's what the data actually shows about source selection.