What is llms.txt?
It is a proposal for a Markdown file placed at a website's root, described on its own site, llmstxt.org, as "a proposal to standardise on using an /llms.txt file to provide information to help agents use a website." The proposal comes from Jeremy Howard, first published in September 2024 and revised since. It is not issued by a standards body, and llmstxt.org does not claim that it is: it calls itself a proposal throughout, open for "community input" through a GitHub repository and a discord channel.
What does the proposal actually specify?
A file with a fixed order of Markdown sections, only one of which is required. Per llmstxt.org, a file following the spec contains, in order: an optional byte-order mark, then "An H1 with the name of the project or site. This is the only required section," then "A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file," then optional prose sections, then optional sections under H2 headings that list links. Each link line is "a required markdown hyperlink name, then optionally a : and notes about the file." The proposal also suggests a companion convention: publishing a clean Markdown copy of a page at the same URL with a .md extension, so an agent can fetch text without parsing HTML around it.
How is that different from robots.txt?
robots.txt is defined by an IETF standards-track document, not a voluntary proposal. RFC 9309, published in September 2022, states in its abstract that it "specifies and extends the 'Robots Exclusion Protocol' method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers." Its status note describes the document as one that "represents the consensus of the IETF community" and has been through IESG approval. llms.txt has no equivalent process behind it. llmstxt.org itself draws the line between the two: robots.txt "lets automated tools know what access to a site is considered acceptable," while llms.txt information "is instead used on demand, when an agent needs information about a topic while assisting a user." One is enforceable protocol; the other is a convention some crawlers may choose to read.
Do Perplexity and Google AI Overviews read llms.txt?
We have not found a statement from either source saying so, and we have not found one saying they don't. Neither llmstxt.org nor RFC 9309 makes a claim about which specific AI engines consume an llms.txt file, and we are not aware of a published statement from Perplexity or Google on the question. Anyone asserting a definite answer either way is not citing a source we could locate. That gap, not a hidden preference, is why we treat the file as optional.
What we do on our own site
We publish one, without treating it as something that moves rankings. Our FAQ states our position directly: "It is not an adopted web standard and AI engines do not all read it. We treat it as a cheap, low-risk addition, not a lever." Ours is at /llms.txt. Publishing the file costs little and risks nothing we're aware of, so we did it; we don't report on it as a metric.
What a minimal file contains, and how long it takes
The minimum is an H1 with your business name and a one-line blockquote summary, which already satisfies the spec's only required section plus its recommended second section. From there, a small business can add a short list of links under one or two H2 headings, such as pricing, service pages, or an about page, following the format llmstxt.org shows: a markdown link, an optional colon, and a short note. Writing that file is closer to an hour of work than a project: the format has no build step, no plugin dependency, and no fields to fill beyond what already exists on the site.
What matters more than llms.txt
Two things, both already covered elsewhere: whether crawlers can reach your pages at all, and whether the pages themselves answer a buyer's question in the first sentence. robots.txt controls the first, under the enforceable rules in RFC 9309; page structure controls the second, and no file at the root fixes a page that buries its answer in the third paragraph. If you haven't checked which of your pages already get named or cited by an AI engine, that's the more useful place to start than an llms.txt file.
Our free GEO audit runs 10 buyer questions through Perplexity and Google AI Overviews, three times each, and reports what the 60 answers name and cite for your category. If you want to see how we check what's actually getting cited today, see how we check Perplexity and Google AI Overviews mentions.