What is llms.txt?
llms.txt is a plain-text file placed at the root of a domain (yourdomain.com/llms.txt) that provides AI crawlers and language model systems with a structured, LLM-friendly summary of the site’s content. It was proposed by Jeremy Howard in 2024 as an AI-era parallel to robots.txt — not to block crawlers, but to help them understand a site’s key content efficiently.
The file typically contains Markdown-formatted links to the most important pages on a site, with brief descriptions. When AI systems that build retrieval indexes visit a domain, a well-written llms.txt helps them identify the authoritative pages to index, reducing the chance that thin or duplicate content pollutes the model’s representation of the site.
How to create and use llms.txt
An llms.txt file contains a site name, a brief description, and a curated list of URLs with their purpose. Sections divide the content by type: products, documentation, blog, about. The file should surface the pages that most accurately represent the brand and its offerings — the same pages you want AI systems citing.
While no major search engine has formally declared llms.txt a ranking factor, Perplexity and several AI browser agents read it as a navigation hint. The real value is control: it gives a brand a deliberate channel to direct AI systems to canonical, accurate content rather than leaving retrieval entirely to crawl randomness.
Example
Example
A B2B SaaS company publishes llms.txt listing its product pages, case studies, and pricing page. When Perplexity’s crawler visits, it discovers those pages first, making them more likely to be retrieved and cited in answers about the company’s product category.
Frequently asked questions
Is llms.txt an official standard?
It is a community proposal, not yet an official W3C or search engine standard. However, several AI crawlers and tools already read it, and adoption is growing. Publishing one carries no downside and creates an early-mover channel to AI retrieval systems.
What is the difference between llms.txt and robots.txt?
robots.txt tells crawlers which pages to avoid. llms.txt is a guide to a site’s most important content for AI systems to prioritize. The files serve opposite purposes: exclusion versus curation.