What Is llms.txt? An In-Depth Guide to AI-Friendly Website Discovery
Understand the llms.txt v2 proposal, required Markdown format, path-scoped discovery, page-level Markdown, llms-full.txt, limitations, validation, and WordPress implementation.
llms.txt is a proposed Markdown convention that gives AI agents a concise, curated entry point into a website. It is designed for inference-time use: an agent reads the small overview, identifies the most relevant resources, and follows links to cleaner Markdown versions when it needs detail. It complements the existing web stack rather than replacing robots.txt, XML sitemaps, structured data, or strong pages.
Why llms.txt exists
Human-facing web pages carry navigation, advertising, scripts, cookie controls, repeated templates, and visual layout. Converting that interface back into focused text costs tokens and can blur which content is authoritative. Even large context windows are not an invitation to load an entire website indiscriminately.
llms.txt gives the publisher a place to explain what the site is and point to the resources an agent should use. It is closer to a curated map than a database dump. The detail stays on linked pages, so an agent can fetch only what is relevant to the task.
The llms.txt v2 format
The current v2 proposal allows /llms.txt at the origin root and also at subpaths such as /docs/llms.txt. A file covers the pages beneath its path; when several files apply, the most specific path is preferred. This lets a documentation team publish focused guidance even when it does not control the whole domain.
The format is intentionally readable Markdown with a predictable order. An H1 is the only required section, but a production file should normally include enough context and annotated links to be useful.
- Optional byte-order mark.
- One H1 containing the project or site name; this is required.
- An optional blockquote with the short, essential summary.
- Optional non-heading Markdown that explains how to interpret the resources.
- Zero or more H2 sections containing Markdown link lists with optional notes.
- An Optional section, by convention, for lower-priority resources an agent may skip.
llms.txt, robots.txt, sitemap.xml and structured data
These resources solve different problems. robots.txt expresses crawler access preferences. sitemap.xml inventories indexable human-facing URLs. Structured data describes entities and properties within a page. llms.txt curates context and points agents toward the most useful machine-friendly resources.
They work best together. A linked page should be reachable, canonical, allowed by the site’s access policy, and consistent with its structured data. llms.txt should never be used to hide contradictory claims that users cannot see on the public site.
| Resource | Primary job | What it should not be treated as |
|---|---|---|
| robots.txt | Crawler access guidance | A content summary or security boundary |
| sitemap.xml | Broad URL discovery for search engines | A curated context window |
| Structured data | Machine-readable facts about page entities | A site-wide navigation guide |
| llms.txt | Curated overview and links for agents | A ranking guarantee or replacement for good pages |
Why page-level Markdown matters
The strongest implementation does more than list ordinary HTML URLs. The v2 proposal recommends clean Markdown versions at the corresponding URL, either by appending .md, replacing an extension with .md, or using index.md for directory-style pages.
Publishers can advertise the Markdown representation with rel=alternate and type=text/markdown, then point to the covering llms.txt with rel=describedby. These relationships can appear in HTML link elements or HTTP Link headers. That gives an agent a standards-based route from the human page to the clean representation and its site-level context.
Where llms-full.txt fits
Many sites also publish llms-full.txt as a consolidated, higher-context companion. It can be convenient for export, evaluation, or small sites whose important public content fits into one manageable document. It is not the required core of the llms.txt v2 format, so its absence should not invalidate an otherwise correct llms.txt file.
For large sites, blindly concatenating everything creates the problem llms.txt was meant to avoid. Keep the curated overview small, make linked Markdown dependable, and treat any full-content file as an optional delivery format with clear size and freshness controls.
How to write an effective file
Begin with the user tasks an agent should be able to complete. Group authoritative resources by those tasks, write specific link labels, and add a short note explaining what each resource contains. Prefer canonical, stable URLs and remove outdated or duplicate pages.
- Keep the summary factual, concise, and free of marketing superlatives.
- Link to policies, product facts, documentation, and support material that answer real questions.
- Use the Optional section for background or lower-priority material.
- Make generated output update when content is published, removed, or renamed.
- Test linked Markdown for status, content type, structure, and parity with the visible page.
- Give an agent only the llms.txt file, ask representative questions, and observe which links it follows.
Limitations and realistic expectations
llms.txt remains an open proposal, and support differs across tools. A publisher cannot force an AI provider to retrieve the file, follow every link, cite the site, or prefer it over another source. It is also public, so never place secrets, private URLs, unpublished content, or instructions that rely on obscurity inside it.
The practical case for llms.txt is still strong: it is inexpensive, human-readable, testable, and useful to agents and internal retrieval workflows that choose to consume it. Judge success by cleaner discovery and lower ambiguity, not by claims of guaranteed generative-engine rankings.
WordPress implementation checklist
A WordPress implementation should generate from published content, respect password protection and editorial exclusions, expose only public post types, and invalidate caches when relevant content changes. Editors also need per-post controls because not every page belongs in an AI-facing index.
After setup, validate /llms.txt, optional /llms-full.txt, several .md pages, content types, discovery link relations, and behavior after a permalink change. Recheck the output when a plugin, theme, SEO system, or cache layer changes routing.
WordPress implementation
Generate and maintain llms.txt with Clustova
The Clustova LLMs.txt WordPress plugin generates root llms.txt and optional llms-full.txt output locally, serves clean Markdown representations for supported public content, provides per-post inclusion controls, and exports Markdown ZIP archives without sending site content to an external generation API.
Frequently asked questions
Is llms.txt an official web standard?
It is an open proposal and ecosystem convention, not a W3C or IETF standard. Support varies, so it should complement established web fundamentals rather than replace them.
Does llms.txt improve Google rankings?
There is no ranking guarantee. Its purpose is to give compatible AI agents a concise, curated map to machine-friendly content. Search performance still depends on relevance, quality, authority, technical SEO, and many external factors.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the concise discovery map. llms-full.txt is an optional companion convention that can consolidate more content. The core v2 proposal emphasizes a small overview plus links to clean page-level Markdown.
Can llms.txt be used on WordPress?
Yes. It can be generated dynamically from published WordPress content. A robust plugin should respect public visibility, exclusions, post types, caching, and permalink changes.
Primary specifications and further reading
Muhammad Afzal is the founder of Clustova. He builds AI-powered tools that help content marketers, agencies and developers produce high-quality, search-ready content at scale. He writes about SEO, AI content pipelines and the future of content marketing.