llms.txt: How to Create an AI Crawler File That Actually Works in 2026

What Is llms.txt?

llms.txt is a plain-text Markdown file at your site root (yoursite.com/llms.txt) that gives AI models a curated, machine-readable summary of your site — who you are, what you do, and which pages represent your best content. Instead of making a crawler guess what matters, you hand it the map.

The spec was proposed by Jeremy Howard in September 2024, and it's become a core piece of AEO infrastructure — adoption has been quiet but real — Anthropic publishes one, hundreds of dev-tool and docs sites ship them, and AI crawlers fetch the file when it exists. Erlin's dataset puts the payoff at ~32% AI coverage lift within 14 days — one of the fastest documented levers in AEO, for a file that takes 30 minutes.

It completes the machine-readable trio:

Ours is live if you want the worked example: altyzo.com/llms.txt.

State of play, September 2026: Google's own AI optimization guidance is now explicit — Google Search "doesn't use them," and shipping one will "neither harm nor help" visibility or rankings, because Google ignores the file entirely. The same document dismisses content "chunking" and rewriting pages just for AI systems. What it doesn't cover is the other engines — Claude, Perplexity, and ChatGPT-family crawlers still fetch llms.txt. So the Erlin lift claim is one vendor dataset measuring non-Google coverage; hold it as correlation, not law. The honest read: ship it because it's 30 minutes and makes your site legible to whatever does read it — not because anyone proved it earns citations.

The Format

Simple Markdown, three sections:

1. H1 + one-line description

# Altyzo

> Self-hosted AI software for SEO agencies. It does the research, writing, publishing, and monitoring for SEO, AEO, and GEO work; the agency's team reviews and approves it.

The blockquote is your one-line summary — the first thing a model reads. Make it a sentence worth quoting verbatim, because that's the literal use case.

2. H2 sections grouping your best links

## Platform
- [Pricing](https://altyzo.com/#pricing): $30K one-time license, unlimited clients and seats
- [Altyzo Managed](https://altyzo.com/managed): Done-for-you AEO/GEO service, $8K/month

## Guides
- [AEO checklist](https://altyzo.com/blog/aeo-checklist): 40-point audit, scored on our own site
- [What is AEO](https://altyzo.com/blog/what-is-aeo): The pillar explainer

Format: - [Title](absolute URL): dense description. Group by topic, not URL structure — models understand topics, not your CMS hierarchy.

3. Optional section

## Optional
- [Privacy](https://altyzo.com/privacy): Data handling
- [Contact](https://altyzo.com/contact): Get in touch

Pages models can skip when context runs short. Real pages — just not the ones you want cited.

Hosting Requirements

• 200 status at yoursite.com/llms.txt
• text/plain or text/markdown MIME type
• UTF-8, publicly accessible, absolute URLs only

What Separates Good From Useless

Curate, don't dump. 10–30 authoritative pages, not every URL. The spec's own words: "concise, expert-level information." Brevity is the feature.

Write quote-bait descriptions. Every link description should be a dense sentence a model could cite verbatim. "Self-hosted AI software that does the research, writing, publishing, and monitoring for agencies" survives extraction. "Learn more about our platform" doesn't.

Fix your architecture first. llms.txt reflects your content hierarchy — if your pillar/spoke structure is messy, the file curates noise. Map the structure, then the file.

Stay under ~2,000 words. The file shares context window with the content it points to. A bloated llms.txt defeats its own purpose.

What It Won't Do — Honestly

• No access control. That's robots.txt's job. llms.txt is a pitch, not a gate.
• No guaranteed citations. It improves retrieval accuracy and navigation — whether you get cited still depends on content authority. Entity work first: entity SEO. Then verify what changed: how to track AI citations. Google's own surface is a separate mechanic: how to get cited in AI Overviews.
• No sitemap replacement. Different job, you need both.
• No universal standard. Community-driven spec; engines honor it voluntarily. Given the cost (~30 minutes) vs. the documented lift, it's an asymmetric bet — cheap downside, real upside.

Build It in 8 Steps

1. Pick your 10–30 pages. Pillars, top posts, comparisons, pricing, about. The AEO checklist scores which pages deserve a slot.
2. Write the one-liner. Dense, self-contained, cite-worthy.
3. Group into 3–5 H2 topics. Platform / Guides / Comparisons / Company is a fine skeleton.
4. Write quote-bait descriptions. One sentence each, extractable.
5. Add the Optional section. Privacy, contact, legal.
6. Deploy at root. Next.js: route handler or public/llms.txt. WordPress: upload to root.
7. Verify. curl -I https://yoursite.com/llms.txt → 200 + text MIME.
8. Review quarterly. 15 minutes: new pillars in, stale links out.

Frequently Asked Questions

Is llms.txt the same as robots.txt?

No — robots.txt controls access; llms.txt describes content. Gate vs. map. You need both.

Does Google use llms.txt?

No — and it's not ambiguous anymore. Google's September 2026 guidance states Google Search ignores llms.txt entirely: "neither harm nor help" for visibility or rankings, including AI Overviews and generative features. Claude, Perplexity, and ChatGPT-family crawlers still fetch it — that's who the file is for.

Should every URL go in it?

No — 10–30 authoritative pages. The spec says "concise, expert-level." A bloated file defeats itself.

llms.txt vs llms-full.txt?

llms.txt is the curated index. llms-full.txt concatenates full page content for models that want one big context window. Most sites need only the first; docs-heavy sites may want both.

The Altyzo platform generates and maintains llms.txt automatically — part of the $30K agency license. Or Altyzo Managed builds your whole AI-readable layer — see the GEO agency page for what that work actually covers.