Picture this: a user asks ChatGPT about your product. The AI visits your site. And instead of content, it gets chaos — HTML tags, JavaScript, cookie banners, duplicate navigation. Out of 500 documentation pages, it manages to read a dozen. The rest falls outside the context window.
Your competitor, who has llms.txt, already handed the AI a clean map: here's what matters, here's where to get the actual content.
Easy to test: ask Perplexity about your product, then ask the same question about a competitor who has llms.txt. The difference in answer quality is immediate. Every day without llms.txt, an AI agent understands your competitor's site better than yours. This isn't theory. Anthropic, HuggingFace, Perplexity, and Cursor have already shipped it. Below: where the file came from, how it works, what's real, and what's still marketing.
One txt file changed SEO in 1994 — the same thing is happening again
September 3, 2024. Jeremy Howard — co-founder of Answer.AI and creator of fast.ai — published the spec for a new standard: llms.txt. The idea is simple. When a user asks an AI agent for help with a tool or product, the agent tries to load information from the site. But the model's context window is too small for thousands of pages. And HTML versions are full of noise — scripts, ads, repeated navigation. Howard proposed the equivalent of robots.txt — not for blocking crawlers, but for navigating AI.
The parallel is exact. robots.txt appeared in 1994. Nobody thought it would reshape the entire web. Now it's absent only on broken sites. llms.txt is following the same path — faster.
What happened next:
- November 2024 — Mintlify adds llms.txt support to all documentation sites. Overnight, thousands of doc sites — including Anthropic and Cursor — get the file
- 2025 — BuiltWith data shows more than 844,000 sites have shipped it
- Google mentions llms.txt in the experimental A2A (Agents-to-Agents) protocol
That adoption speed is rare. robots.txt took years. Here: eighteen months from blog post to 844,000 sites.
robots.txt and llms.txt — two files, opposite logic
| robots.txt (1994) | llms.txt (2024) | |
|---|---|---|
| For | Search crawlers | AI agents, LLMs |
| Job | Block access to pages | Show what and where to read |
| Logic | "Don't go here" | "Go here — here's the map" |
| Compliance | Search engines honor it | Voluntary standard |
Both files are plain text in your site root. No code. robots.txt closes doors. llms.txt opens the right ones — and explains what's behind them.
One Markdown file in your root — and the AI gets a site map
llms.txt is a Markdown file in the root of your site: yourdomain.com/llms.txt. No magic. A plain text file accessible by URL.
Official structure per llmstxt.org spec:
- H1 heading — project name (the only required element)
- Blockquote — short description
- H2 sections — thematic blocks with links and descriptions
The format is intentionally Markdown, not XML or JSON — equally readable by humans, models, and parsers.
File variants
| File | Contents | When to use |
|---|---|---|
| llms.txt | Map: sections + links | Core file, always needed |
| llms-full.txt | All site content in one file | Large docs, SaaS |
| llms-ctx.txt | Extended context | Complex products |
| llms-ctx-full.txt | Maximum context | Advanced use cases |
Bonus: .md versions of pages. If a page lives at /docs/start, the clean version is available at /docs/start.md. The AI reads Markdown directly, without HTML noise.
The file works at request time — not during model training
One key point from the official spec that most people miss: llms.txt is built for inference, not training. The file doesn't change what the model knows "in general." It works in the moment when a specific user asks the AI a question about your product.
How it plays out:
- User asks ChatGPT: "How do I set up authentication in [your product]?"
- The AI agent searches for information
- Instead of crawling 500 pages with 90% HTML noise — reads llms.txt
- Gets the map: here's the authentication section, here's the .md link
- Loads the relevant content and gives a precise answer
That's why the file matters for RAG pipelines and AI agents in Cursor IDE, Perplexity, and Claude Projects.
844,000 sites have shipped it — what that actually means
844,000+ sites is past the experiment stage. Anthropic, HuggingFace, Perplexity, Zapier, Cloudflare, and Cursor have all shipped the file. 1,000+ teams generate it automatically through llmtxtgenerator.com. For documentation and SaaS, it's becoming baseline.
There's also a concrete, unglamorous benefit available right now: AI crawlers read one file instead of walking your entire site. Fewer requests — less server load. No SEO speculation required. A separate signal: Google mentioned llms.txt in the A2A protocol. Experimental so far, but Google rarely cites third-party standards without reason.
Why Google compared llms.txt to meta keywords — and what that means
The honest version of where the data is thin. "+31% AI visibility signals" — a number floating around from some generators. No methodology published. Don't trust it without a verified study.
What to keep in mind:
- SEO impact. John Mueller from Google compared llms.txt to meta keywords in 2025 — the file may simply be ignored. It has no effect on classic Google ranking.
- ChatGPT, Claude, and Gemini haven't confirmed usage. None of the major AI companies has documented that their systems read and respect llms.txt. The standard is voluntary.
- No enforcement. Unlike robots.txt — which search engines honor by convention — llms.txt has no regulator. An AI agent can read the file, or skip it entirely. In practice: Cursor and Perplexity read it reliably. Everyone else: no guarantees.
Three ways to add llms.txt to your site
Choose the approach that fits your site's size and technical setup.
Option 1: Manual (15 minutes)
Create llms.txt in your project root. Upload to the root. Verify: yourdomain.com/llms.txt should open as a page.
# Your Product Name
> Short description: what the product does, who it's for.
## Documentation
- [Quick Start](https://yoursite.com/docs/quickstart.md): Up and running in 5 minutes
- [API Reference](https://yoursite.com/docs/api.md): All methods and parameters
## Guides
- [Webhook Integration](https://yoursite.com/guides/webhooks.md): Setting up notifications
## Optional
- [Changelog](https://yoursite.com/changelog.md): Release historyOption 2: llmtxtgenerator.com (automated)
Best for sites with many pages. Enter your site URL, the tool scans your sitemap and all pages, strips navigation and scripts, converts HTML to AI-optimized Markdown, maps internal links, and generates the final llms.txt. Works with ChatGPT, Claude, and Cursor IDE.
Pricing:
| URLs | Price |
|---|---|
| First 20 URLs | Free |
| Up to 99 URLs | $9 |
| Up to 999 URLs | $19 |
| Up to 1,999 URLs | $49 |
| Up to 4,999 URLs | $79 |
| 5,000+ URLs | $149 |
Option 3: Plugins and packages
VitePress — official plugin. Docusaurus — ready-made module. Drupal — extension available. Python — llms-txt package. JavaScript/Node.js — npm package.
llms-full.txt — when a map isn't enough
llms.txt is a map. Links and sections, but not the content itself. llms-full.txt is all your site content in a single file. A user can copy one URL and load the entire product context directly into an AI chat.
When you need it:
- Large documentation (dozens or hundreds of pages)
- SaaS products with complex APIs
- When users already rely on AI to work with your product
The .md page pattern: /docs/api → publish /docs/api.md in parallel. The AI reads Markdown directly, no parsing required. Your llms.txt should point to .md links, not HTML.
Nobody has to follow it — and that's the standard's real problem
The community is split.
- "Another txt file nobody reads" — the parallel with robots.txt, which aggressive crawlers have ignored for years
- You can't force AI to comply
- Major AI companies are staying quiet
- The standard is forming — better to be in it from the start
- The effect in Cursor IDE and Perplexity is already documented
- Reduced server load is a real, measurable benefit right now
- robots.txt wasn't taken seriously either, at first
Alternatives: schema.org, XML sitemaps with correct priorities, solid OpenGraph tags. All of these help AI too — and their compliance has been proven over time.
Who should add it now vs. who can wait
- SaaS products — users are asking AI about your tool constantly
- Documentation sites — the main use case the standard was built around
- AI tools and platforms — if you're in AI, being AI-friendly is obvious
- Developer tools — developers live in Cursor and GitHub Copilot daily
- Local businesses without documentation
- Simple landing pages
- Sites where AI traffic isn't part of the picture
How to verify the file works:
- Create the file → upload to root: yourdomain.com/llms.txt
- Open Perplexity or Claude, ask a question about your product
- Check whether the AI uses the right information
- Check indexing: llmstxt.site and directory.llmstxt.cloud
In two years, the standard went from a blog post to 844,000 sites
llms.txt appeared less than two years ago. The adoption speed mirrors robots.txt in the mid-90s — only faster. Right now it's an informal proposal with no regulator. The logical next step: W3C or an equivalent body. If that happens, the file moves from "recommended" to de facto required — the same way robots.txt became baseline infrastructure without any enforcement mechanism.
The formats are already evolving: llms-ctx.txt and llms-ctx-full.txt have appeared for richer context. If a significant share of web traffic flows through AI agents by 2027 — and all trends point that way — llms.txt will be as unremarkable as sitemap.xml is today.
One Markdown file in the root. 15 minutes. Anthropic, HuggingFace, and Cursor already shipped it — not because they had to, but because it's obvious. What's confirmed: AI crawlers cut server load, Cursor and Perplexity read the file reliably. SEO impact is an open question — don't trust numbers without methodology.
Tags
About the author
Builds the infrastructure behind LLM.txt Generator. Writes about AI systems, web crawling, and how large language models consume information.
Ready to make your site AI-visible?
Generate your llms.txt file in 10 seconds — free, no account needed.
Get Your llms.txt →