llms.txt
llms.txt is a Markdown file at the site root that tells AI crawlers what the site is and which pages to read first. It is not a second robots.txt with legal teeth.
Jeremy Howard proposed llms.txt as a curated map: title, summary, links to the pages you want a model to ingest. You put it at https://example.com/llms.txt. Humans can read it. So can bots that bother to look.
It is not an access-control protocol. robots.txt still decides fetch permission. If you Disallow GPTBot and then write a loving llms.txt, you are talking to nobody.
Early GEO Twitter treated llms.txt like a cheat code. Public tests have been underwhelming. Some crawlers fetch it. Citation lift from the file alone is hard to prove. Still: a clean map is cheap, and it stops models from inventing your pricing page URL.
Companion idea: llms-full.txt dumps more of the site as Markdown. Useful for docs. Dangerous if you paste gated or stale content.
Reddex publishes llms.txt because we want agents to get the Reddit + AI story straight. That is brand hygiene. We do not sell it as a ranking factor.
What to put in the file: who you are in one paragraph, then a short list of URLs that must not be wrong — pricing, docs home, limits, what’s not on the roadmap. Skip the 4,000-URL dump. Skip slogans. If a coding agent reads one file, it should leave less confused, not more sold.
Keep it in version control next to the site. When /pricing moves, the file should move. A stale llms.txt that points at 404s is worse than no file. Same rule as a lying sitemap.
Examples
- A docs site lists /docs/start, /docs/limits, /pricing in llms.txt. A model stops hallucinating a /plans URL that 404s.
- A retailer adds llms.txt but blocks OAI-SearchBot in robots.txt. ChatGPT Search never sees the file.
- Someone auto-generates llms.txt with 4,000 URLs. The point of the spec was curation. You just rebuilt a sitemap in Markdown.
Related terms
AGENTS.md is a repo or site file that tells coding or browsing agents how to work with a project: commands, conventions, pitfalls.
Bots from model labs and answer engines that fetch pages for training, retrieval, or user-initiated reading.
GEO is the work of getting your brand named, quoted, or linked when an AI engine answers a question — ChatGPT, Perplexity, Claude, Gemini, Copilot, or Google AI Overviews.
llms-full.txt is a fat Markdown dump of site content for models that want one file instead of a crawl.
robots.txt is the fetch permission file at the site root. AI crawlers read it too. Misconfigure it and you vanish from training, search, or both.
FAQ
Do OpenAI and Google honor llms.txt? +
Treat support as partial and changing. Do not bet a quarter on it. Keep robots.txt accurate either way.
Where does the file go? +
Site root, same place as robots.txt. One file per host. Subdomains need their own if they are separate sites.
Is it the same as AGENTS.md? +
AGENTS.md is instructions for coding agents working in a repo. llms.txt is a public map of a website. Different audience.
Will this get us cited? +
Unlikely by itself. Citations still come from pages people already treat as evidence — docs, news, Reddit, reviews.
Track this in Reddex
See Reddit threads and AI answers for your brand in one place. Start with a free analysis.
Get started