robots.txt
robots.txt is the fetch permission file at the site root. AI crawlers read it too. Misconfigure it and you vanish from training, search, or both.
Allow and Disallow are still the language. User-agent groups now include GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and whoever launches next week.
Blocking training bots while allowing search bots is a policy choice. Blocking everything and wondering why ChatGPT Search never cites you is a self-own.
robots.txt is not authentication. Clever scrapers ignore it. Real crawlers from big labs usually honor it because they want to stay invited.
Examples
- A publisher Disallows GPTBot and still ranks in Google. ChatGPT’s memory of them ages like milk. Expected.
- A SaaS accidentally Disallows /docs for all bots. Perplexity cites a scraped mirror on a random domain. Fix the file.
Related terms
Checking that a request claiming to be GPTBot really is, via published IP ranges or signed requests.
Bots from model labs and answer engines that fetch pages for training, retrieval, or user-initiated reading.
Proposed robots.txt-style directives for how crawled content may be used: search vs AI input vs training.
llms.txt is a Markdown file at the site root that tells AI crawlers what the site is and which pages to read first. It is not a second robots.txt with legal teeth.
OpenAI runs several bots: GPTBot for training, OAI-SearchBot for ChatGPT Search, and ChatGPT-User for user-triggered fetches.
FAQ
Does robots.txt control training rights legally? +
It is a technical signal, not a contract. Lawyers still argue about the rest. For GEO ops, assume major bots obey it and plan accordingly.
Should I block all AI bots? +
Only if you mean to leave the channel. There is no secret “block training, keep citations” switch that works everywhere.
Track this in Reddex
See Reddit threads and AI answers for your brand in one place. Start with a free analysis.
Get started