OpenAI crawlers
OpenAI runs several bots: GPTBot for training, OAI-SearchBot for ChatGPT Search, and ChatGPT-User for user-triggered fetches.
Read the user-agent. Allowing or blocking “OpenAI” as a blob is how people break Search while trying to opt out of training.
Verify IPs when you can. UA spoofing is a thing. See crawler verification.
Examples
- Logs show ChatGPT-User hitting /pricing during a demo. That is a human in the loop, not a training scrape.
Related terms
Bots from model labs and answer engines that fetch pages for training, retrieval, or user-initiated reading.
ChatGPT Search is OpenAI’s live web retrieval inside ChatGPT. It is a different beast from “what GPT remembers about you.”
Training bots build weights. Search/citation bots fetch for an answer now. User-triggered bots fetch because a human asked.
robots.txt is the fetch permission file at the site root. AI crawlers read it too. Misconfigure it and you vanish from training, search, or both.
FAQ
Which bot should I allow for GEO? +
If you want ChatGPT Search citations, allow the search bot. Training is a separate policy call with legal and competitive angles.
Track this in Reddex
See Reddit threads and AI answers for your brand in one place. Start with a free analysis.
Get started