AI crawler verification
Checking that a request claiming to be GPTBot really is, via published IP ranges or signed requests.
Spoofed UAs exist. Rate-limit unknown “GPTBot” from consumer IPs. Don’t block the real ranges by accident.
Web Bot Auth is the longer-term idea.
Examples
- A scraper uses User-Agent GPTBot from a cloud VM not in OpenAI’s list. You 403 it. Good.
Related terms
Bots from model labs and answer engines that fetch pages for training, retrieval, or user-initiated reading.
robots.txt is the fetch permission file at the site root. AI crawlers read it too. Misconfigure it and you vanish from training, search, or both.
An IETF idea for crawlers to cryptographically sign requests so you can trust who they are.
FAQ
Where are the IP lists? +
Vendors publish them. They change. Automate the fetch; don’t paste a 2024 gist forever.
Track this in Reddex
See Reddit threads and AI answers for your brand in one place. Start with a free analysis.
Get started