robots.txt is a text file crawlers check before fetching your site, and its behavior is defined by RFC 9309, an IETF Standards Track document published in September 2022. The standard is explicit about what the file is not: "These rules are not a form of access authorization." A rule in robots.txt tells a well-behaved crawler what you'd prefer it fetch. It does not stop a crawler that ignores the file, and it does not encrypt or hide anything. The trade-off is plain: allowing a crawler costs you a small, unpredictable amount of server load in exchange for a chance to be named in that crawler's answers; blocking it removes that load and that chance together.
What robots.txt actually controls
robots.txt controls which URLs a cooperating crawler will request, nothing more. Google's own introduction to the file states: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site." The same page warns that "it is not a mechanism for keeping a web page out of Google" — a disallowed page can still appear in results if another site links to it, just without a description. Google also notes that "robots.txt rules may not be supported by all search engines," and that obeying the file is "up to the crawler." For a business, this means robots.txt sets a preference for crawlers that choose to read it. Whether a given AI crawler reads it, and what it does next, differs by company.
Perplexity's two crawlers, and which one matters for citations
Perplexity runs two separate agents, and only one of them is a crawler in the traditional sense. Perplexity's documentation describes PerplexityBot as the one that builds search results: "PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." Perplexity recommends allowing it, along with its published IP ranges. Perplexity also notes that changes take time to apply: "Each setting works independently, and it may take up to 24 hours for our systems to reflect changes." The second agent, Perplexity-User, fetches a page only when a person asks Perplexity a question that sends it to that specific URL, and behaves differently: "This fetcher generally ignores robots.txt rules," because "a user requested the fetch." Blocking PerplexityBot in robots.txt works against your own goal of appearing in Perplexity's results; blocking Perplexity-User in robots.txt mostly does not do anything, since it typically does not check the file.
Google AI Overviews use the regular Search index
Google AI Overviews have no crawler of their own to allow: eligibility is tied to normal Google Search. Google states the requirement directly: "To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements." Blocking Googlebot in robots.txt works against that requirement. Google's robots.txt guide says a blocked page's "URL can still appear in search results, but the search result won't have a description", and a page without a snippet does not meet the AI Overviews condition above. A separate token, Google-Extended, exists for a different purpose: it lets a site limit use of its already-crawled content for training and grounding Google's generative AI models, without touching the crawl that feeds Search itself. Google is explicit about the boundary: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." For a business that wants to appear in AI Overviews, Googlebot is the crawler to keep open. Google-Extended, by Google's own description, does not affect inclusion in Search.
A short example robots.txt for AI visibility
A file aimed at appearing in Perplexity and Google AI Overviews allows the crawlers those surfaces depend on and leaves any other blocking decisions to the rest of the file:
User-agent: Googlebot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /admin/
Perplexity-User does not need a line here since it generally does not consult robots.txt, and there is no separate crawler line for Google AI Overviews to allow or block — Googlebot's access already covers it.
Checklist before you trust robots.txt
A checklist is only useful if it checks the file that crawlers actually fetch, not the one in a repo or a template. Give an edit time to take effect before you re-test it, per Perplexity's own note above on propagation delay. Open yoursite.com/robots.txt in a browser and read the live version; a staging file, a CDN default, or a caching layer can serve something different from what you edited. Then check CDN and firewall rules separately: a Web Application Firewall can block a crawler by IP or user agent regardless of what robots.txt allows, which is why Perplexity publishes IP ranges for both its agents and documents how to whitelist them in Cloudflare and AWS WAF rules. robots.txt is one gate among several; treat it as the first check, not the only one.
Our free GEO audit runs 10 buyer questions through Perplexity and Google AI Overviews, 3 times each, and shows what the 60 answers name and cite for your category.