AI crawler logs
Server logs that show GPTBot, ClaudeBot, PerplexityBot, and friends hitting your URLs.
If they never fetch the page you “optimized,” you are decorating. If they fetch and you still are not cited, the problem is substance or competition.
Watch 404s they hit. Models love inventing paths; a redirect helps.
Examples
- OAI-SearchBot hammers /blog/old-slug that 404s. You lost a citation to whoever redirected first.
Related terms
Bots from model labs and answer engines that fetch pages for training, retrieval, or user-initiated reading.
How much a bot is willing to fetch. AI bots can be greedier or pickier than Googlebot. Faceted URLs still waste everyone’s time.
OpenAI runs several bots: GPTBot for training, OAI-SearchBot for ChatGPT Search, and ChatGPT-User for user-triggered fetches.
FAQ
CDN logs enough? +
Yes if they keep User-Agent and path. Sample, don’t drown.
Track this in Reddex
See Reddit threads and AI answers for your brand in one place. Start with a free analysis.
Get started