Every few months the SEO world gets a new file to argue about. This one is llms.txt. Here is the honest version: it is cheap to add, no major model is confirmed to use it, and your crawler settings matter far more.
What is llms.txt?
llms.txt is a plain-text file you place at yourdomain.com/llms.txt. It lists your most important pages, with short descriptions, in clean Markdown, so an AI tool can find the good stuff without wading through your navigation and cookie banners. Some sites also publish llms-full.txt with the actual content inlined.
It is a community proposal, not an official standard. Google has said it is not using it. OpenAI and Anthropic have not confirmed using it for ranking.
What actually controls AI access
Your robots.txt rules for the AI crawlers do the real work:
| Bot | Purpose | Common choice |
|---|---|---|
Googlebot | Search and AI Overviews | Always allow |
Google-Extended | Gemini grounding and training | Allow for AI visibility |
GPTBot / OAI-SearchBot | ChatGPT training and search | Allow for ChatGPT citations |
PerplexityBot | Perplexity index | Allow for Perplexity citations |
Should you add one?
- If your CMS can generate it automatically, add it. Low effort, no downside
- Do not hand-maintain it for a large site. The cost outweighs any theoretical benefit
- Do not expect a ranking or citation lift from it
- Spend the time on robots.txt, page speed and quotable content instead
Bottom line: llms.txt is a nice-to-have that costs almost nothing if automated. It is not a growth tactic. Anyone selling it as one is ahead of the evidence.
Where it stands right now
Mahdi Hassan · Web Developer & Site Speed
If you want it, wire it into the build so it regenerates from your sitemap and page metadata. A hand-kept llms.txt goes stale in a month, and a stale one is worse than none.
Short case study
A documentation-heavy SaaS
The client wanted llms.txt and llms-full.txt for their docs. We generated both from the existing content pipeline and kept measuring AI citations and assistant referrals for three months.
No measurable change in AI visibility. The real lift that quarter came from restructuring the docs answer-first, which we would recommend doing first, every time.