Web Bot Auth

An IETF idea for crawlers to cryptographically sign requests so you can trust who they are.

Better than UA strings. Adoption will take time. Until then, IP lists.

When it ships widely, use it. Don’t invent your own crypto handshake for GPTBot.

Examples

A CDN supports signed bots. You stop blocking a lab because a scraper stole their UA last year.

Checking that a request claiming to be GPTBot really is, via published IP ranges or signed requests.

Bots from model labs and answer engines that fetch pages for training, retrieval, or user-initiated reading.

robots.txt is the fetch permission file at the site root. AI crawlers read it too. Misconfigure it and you vanish from training, search, or both.

Available today? +

Watch vendor docs. Don’t wait to set robots.txt correctly in the meantime.

See Reddit threads and AI answers for your brand in one place. Start with a free analysis.

Get started