llms.txt: How to Create an AI Crawler File That Actually Works in 2026

What Is llms.txt?

llms.txt is a plain-text Markdown file hosted at the root of your website (yoursite.com/llms.txt) that gives AI language models a curated, machine-readable summary of your site. It's a foundational piece of Answer Engine Optimization (AEO). It tells AI crawlers like ChatGPT, Claude, and Perplexity what your site is about, who you are, and which pages represent your most authoritative content.

The specification was proposed by Jeremy Howard in September 2024 and has since been adopted by Anthropic (Claude), Perplexity, and OpenAI (ChatGPT). Google's official position is still evolving, but Google-Extended (the crawler that powers AI Overviews) respects similar signals.

Think of it as the third file in your site's machine-readable trio:

File	Purpose	Audience
robots.txt	Controls which pages crawlers can access	Search engine bots
sitemap.xml	Lists all pages for search engines to index	Search engine indexers
llms.txt	Summarizes key content for AI understanding	AI language models

robots.txt tells crawlers what they can access. sitemap.xml tells them which URLs exist. llms.txt tells them what your site is about and which pages matter most.

The llms.txt Format

The format is intentionally simple Markdown. Three sections in this order:

1. H1 with site name and one-line description

# Altyzo

> Self-hosted autonomous SEO agents for agencies. Seven agents that research, write, publish, monitor, and optimize for SEO, AEO, and GEO.

The H1 is your brand name. The blockquote (prefixed with >) is a one-line summary of what your site or business does. This is the first thing an AI model reads when it fetches your llms.txt.

2. H2 sections grouping links by topic

## Platform
- [How it works](https://altyzo.com/platform): Seven autonomous agents and how they work together
- [Pricing](https://altyzo.com/pricing): $30K one-time license, unlimited seats and clients
- [Self-hosted deployment](https://altyzo.com/deploy): Docker image with Postgres, ClickHouse, and nginx

## Resources
- [Blog](https://altyzo.com/blog): Guides on AEO, GEO, and AI search for agencies
- [Comparisons](https://altyzo.com/compare): Altyzo vs Ahrefs, SEMrush, Jasper, and Surfer SEO
- [AEO checklist](https://altyzo.com/blog/aeo-checklist): 40-point audit for answer engine optimization

Each H2 groups related links. Each link follows the format: - [Title](URL): Description. The title is the page name, the URL is the absolute URL (including https://), and the description is a brief explanation of what the page contains.

3. Optional section for lower-priority pages

## Optional
- [Privacy policy](https://altyzo.com/privacy): How we handle data
- [Contact](https://altyzo.com/contact): Get in touch

The Optional section is for pages AI can skip if context is limited. These are real pages that exist, but they're not the ones you want AI engines to cite or reference when answering questions about your brand or topic.

Hosting Requirements

The file must meet these technical requirements:

• Return a 200 status code at yoursite.com/llms.txt
• Use text/plain or text/markdown MIME type
• Be encoded in UTF-8
• Be publicly accessible without authentication
• Use absolute URLs (including https://) for all links

What Makes a Good llms.txt File

Most llms.txt files fail because they're either too sparse (just a title and a link) or too bloated (every URL on the site). Here's what actually works:

Curate, don't dump. The spec says "concise, expert-level information gathered in one place." Include your 10 to 30 most authoritative pages, not every URL. Full HTML pages with navigation, ads, and scripts rarely fit cleanly inside a model's context window. The brevity is the whole point.

Write descriptions that are quote-bait. The description after each link should be a dense, self-contained sentence that an AI model could cite verbatim. "Seven autonomous agents that research, write, publish, monitor, and optimize for SEO, AEO, and GEO" is cite-worthy. "Learn more about our platform" is not.

Group by topic, not by URL structure. Group links by what they're about (Platform, Resources, Comparisons, Documentation), not by how they're organized in your CMS. AI models understand topics, not URL hierarchies.

Map a clean hub-and-spoke content structure. If your information architecture is messy, fix that before drafting the file. Otherwise you'll be curating noise. Your llms.txt should reflect a clear content hierarchy where pillar pages link to supporting content.

Keep it under 2,000 words. The file needs to fit inside a model's context window alongside the actual content it's retrieving. A 10,000-word llms.txt defeats the purpose. Curate aggressively.

What llms.txt Does Not Do

Being honest about the limitations is important. llms.txt is not magic.

It doesn't grant or deny crawl permission. That's robots.txt. llms.txt is a structured pitch for context, not an access-control rule. A crawler can index your site without reading llms.txt, and reading llms.txt doesn't prevent a crawler from accessing pages not listed in it.

It doesn't guarantee citations. A 2025 experiment found that llms.txt alone does not boost AI citations without underlying content authority. Crawlers do fetch the file (confirmed in server logs across thousands of sites), but the file improves navigation quality and retrieval accuracy. Content authority determines whether you get cited at all. Building that content authority starts with entity presence. See our entity SEO for AI search guide.

It doesn't replace sitemap.xml. sitemap.xml lists all your URLs for search engine indexers. llms.txt curates your most important pages for AI understanding. They serve different purposes and you need both.

It's not enforced by a central body. The standard is community-driven. There's no guarantee every AI system follows the directives. But the major AI engines (ChatGPT, Claude, Perplexity) do fetch the file, and the format is simple enough that there's no downside to publishing one.

How to Create an llms.txt File: Step by Step

1. Map your top 10 to 30 authoritative pages. These are your pillar pages, highest-traffic blog posts, comparison pages, pricing page, and about page. Skip thin content, tag pages, and administrative pages. For a structured audit of what makes a page citation-ready, see the AEO checklist.

2. Write a one-line brand description. This should be a dense, self-contained sentence that an AI model could cite verbatim. Include what you do, who you serve, and what makes you different.

3. Group your pages by topic. Create 3 to 5 H2 sections that group your links logically. Common groupings: Platform, Resources, Documentation, Comparisons, Company.

4. Write descriptions for each link. Each description should be a dense, self-contained sentence. Not "Learn more." But "Seven autonomous agents that research, write, publish, monitor, and optimize for SEO, AEO, and GEO."

5. Add an Optional section for lower-priority pages. Privacy policy, contact, legal pages. Real pages, but not the ones you want AI engines to cite.

6. Deploy to your site root. The file must be accessible at yoursite.com/llms.txt. In Next.js, this can be a route handler or a static file in the public directory. In WordPress, upload it to your root directory via FTP or a plugin.

7. Verify it returns a 200 status code. Test with curl -I https://yoursite.com/llms.txt. Confirm the content type is text/plain or text/markdown.

8. Update quarterly. Review your llms.txt every quarter. Add new pillar pages, remove outdated links, refresh descriptions. 15 minutes per quarter is enough.

Common llms.txt Questions

Is llms.txt the same as robots.txt?

No. robots.txt controls which pages crawlers can access. llms.txt tells AI models what your site is about and which pages matter most. They serve different purposes and you need both. robots.txt is an access-control rule. llms.txt is a structured pitch for context.

Does Google use llms.txt?

Google's official position is still evolving. Google-Extended (the crawler that powers AI Overviews) respects similar signals. The major AI engines that have adopted llms.txt are Anthropic (Claude), Perplexity, and OpenAI (ChatGPT). There's no downside to publishing one, even if Google doesn't formally use it yet.

Should I include every URL in my llms.txt?

No. The spec says "concise, expert-level information." Include your 10 to 30 most authoritative pages, not every URL. A bloated llms.txt defeats the purpose. Curate aggressively.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a curated index of your most important pages with brief descriptions. llms-full.txt is the complete content of those pages concatenated into a single file for models that prefer one large context window. Most sites only need llms.txt. Documentation-heavy sites may benefit from llms-full.txt as well.

The Bottom Line

Create the file, curate your most authoritative pages, write descriptions that are quote-bait for AI models, and deploy it to your site root. Update it quarterly. That's it. Most sites still don't have one, which makes this a 30-minute fix with a genuine competitive edge. Talk to us about automating llms.txt and content optimization.