SEO
Crawl budget
How much a bot is willing to fetch. AI bots can be greedier or pickier than Googlebot. Faceted URLs still waste everyone’s time.
Don’t generate infinite calendars. Don’t lock the useful docs behind 30 redirects.
Sitemaps help; they are not a command.
Examples
- A parameter explosion eats GPTBot time. The new /pricing never gets fetched.
Related terms
AI crawler logs
Server logs that show GPTBot, ClaudeBot, PerplexityBot, and friends hitting your URLs.
Crawling and indexing
Discover, fetch, store. Google does it. AI labs do it with other bots and rules. You need both if you want both channels.
XML sitemaps
A list of URLs you want fetched. Still worth having. AI bots may use them, or not. They cannot fix Disallow.
FAQ
Do AI bots respect budget the same way? +
They have their own throttles. Clean URL hygiene helps every bot you actually want.
Track this in Reddex
See Reddit threads and AI answers for your brand in one place. Start with a free analysis.
Get started