Crawling and indexing
Discover, fetch, store. Google does it. AI labs do it with other bots and rules. You need both if you want both channels.
Indexation in GSC ≠ presence in an AI index. Check logs.
llms.txt does not replace a sitemap.
Examples
- Blocked by accident in robots for a new user-agent. Six weeks of “why don’t we get cited.”
Related terms
How AI systems store pages for later retrieval. Not identical to Google’s index. Different bots, different freshness, different junk filters.
robots.txt is the fetch permission file at the site root. AI crawlers read it too. Misconfigure it and you vanish from training, search, or both.
A list of URLs you want fetched. Still worth having. AI bots may use them, or not. They cannot fix Disallow.
FAQ
How to request indexing from OpenAI? +
You mostly can’t. Be linked, fetchable, patient. For Google, use Search Console as usual.
Track this in Reddex
See Reddit threads and AI answers for your brand in one place. Start with a free analysis.
Get started