llms.txt Explained: Should Your Website Have One?

llms.txt Explained: Should Your Website Have One?

If you want the short version of llms.txt explained, here it is. It is a proposed file for helping large language models find the pages you most want them to read. That is interesting. It is also a fair way from being a must-have.

A lot of people are talking about llms.txt as if it is the next robots.txt. I do not reckon that is right. The concept is tidy, the theory is sensible enough, but the real-world evidence is still pretty thin.

That matters because this is exactly the sort of thing marketers and founders want to believe in. One neat file. One simple fix. One technical shortcut for a messy visibility problem. I get the appeal. I have fallen for that kind of thinking before.

If you have already read our take on what answer engine optimization is, this article is the practical filter. It is about whether llms.txt belongs on your actual priority list.

What is llms.txt and what is it actually meant to do?

llms.txt is a proposed Markdown file, usually published at the root of a website, that points AI systems toward the content you consider most useful.

The idea comes from llmstxt.org, which frames it as a cleaner, more curated way to guide LLMs through complex websites. Instead of expecting a model to make sense of every navigation element, JavaScript dependency, archive page, and duplicate document, you give it a simpler map.

That map is not meant to replace the site itself. It is more like a shortlist. A typical llms.txt file includes a project or site title, a short description, and grouped links to key resources. In documentation environments, those links often point to Markdown exports, API references, quickstarts, or other canonical pages.

GitBook makes the strongest case for this on documentation-heavy sites. Its argument is fairly reasonable. If AI assistants are helping people navigate docs, then giving them cleaner inputs may improve answer quality.

What llms.txt is not

This is where the confusion starts.

llms.txt is not a crawl-control file. It does not tell GPTBot, ClaudeBot, or Google-Extended what they may or may not access.

It is not a replacement for canonical URLs, internal linking, Schema.org markup, or a proper sitemap.xml either. It is a proposed convenience layer. Useful in some cases, maybe. Foundational, no.

Should a normal business website add llms.txt right now?

For most normal business websites, no, not as a priority task.

That does not mean you should never test it. It means llms.txt usually sits well behind the basics. If your main service pages are vague, your About page says almost nothing, your structured data is missing, and your site still hides key content behind awkward JavaScript, you have bigger problems to fix first.

Ahrefs took a fairly blunt view on this and I think it is a useful corrective. It noted that no major LLM provider had officially committed to supporting llms.txt, including OpenAI, Anthropic, and Google. Semrush landed in a similar place. Its own write-up included implementation guidance, but still concluded it was probably not worth most people’s time yet.

The server-log evidence is the part I keep coming back to. Semrush reported that Search Engine Land’s llms.txt page received zero visits from Google-Extended, GPTBot, PerplexityBot, or ClaudeBot across part of 2025. Search Engine Journal also quoted John Mueller comparing llms.txt to the keywords meta tag, which was not exactly an endorsement.

So if you are asking should you use llms.txt on a standard marketing site, service business site, or local business website, my view is fairly plain. Not yet, unless the basics are already sorted.

A better use of your time first

If you have three hours available this week, I would spend them here instead:

1. Check robots.txt and bot access rules.
2. Make sure important pages render clean, crawlable HTML.
3. Tighten your service, category, and About pages so they answer obvious questions quickly.
4. Add FAQ and Article schema where it genuinely helps clarity.
5. Improve entity consistency across your site, profiles, and directories.

That work is less fashionable. It is also much more likely to help.

If you want the broader context behind that, we covered it in how to get cited by ChatGPT for your business. The short version is that AI visibility still leans heavily on ordinary website quality.

How is llms.txt different from robots.txt, sitemaps, and structured data?

llms.txt, robots.txt, sitemaps, and structured data solve different problems, which is why mixing them together leads to bad decisions.

Here is the cleaner way to think about it:

Standard	Main job	What it does not do
robots.txt	Controls crawler access	Does not explain which content is most useful
sitemap.xml	Lists indexable URLs	Does not curate or prioritise context for LLMs
Schema.org / JSON-LD	Adds machine-readable meaning to page elements	Does not act as a file-based content map
llms.txt	Curates a shortlist of important pages for LLM consumption	Does not enforce crawling or replace site architecture

This distinction matters because people keep describing llms.txt as the robots.txt for AI. That is sloppy. robots.txt is about access. llms.txt is about guidance.

The llmstxt.org proposal is actually fairly explicit on this point. It is designed to coexist with existing standards, not replace them. In practice, that means you should not treat llms.txt as an excuse to neglect the standards search engines and crawlers already use.

Where structured data still matters more

If you are choosing between implementing llms.txt and cleaning up your structured data, I would still choose structured data on most sites.

Schema.org, JSON-LD, and clear HTML help machines interpret actual entities on the page. They support products, reviews, FAQs, organisations, authors, and locations. That is concrete. llms.txt, by comparison, is more like a signpost.

We explored some of this in our look at markdown vs HTML for AI crawlers. Markdown can be easier for machines to parse in some environments, but that does not mean a business site should suddenly reorganise itself around markdown exports.

When llms.txt probably is worth testing

There are cases where llms.txt makes decent sense.

If you run a documentation portal, an API-heavy product, a large developer knowledge base, or a technical content library with strong Markdown outputs, the file becomes more defensible. GitBook, Vercel, Hugging Face, Zapier, and other documentation-centric examples fit this pattern better than a typical services firm or local business.

The underlying logic is straightforward. Complex docs sites often have versioning issues, archived material, fragmented navigation, and hundreds of URLs. A curated file that points to current, canonical resources may help AI systems reach the right material faster.

That still does not mean support is guaranteed. It just means the cost-benefit trade-off is more favourable.

A simple rule of thumb

If your website already has:

• stable information architecture
• clear canonical docs or knowledge pages
• Markdown or machine-friendly page exports
• a technical team who can maintain the file without fuss

then llms.txt is worth a test.

If your site does not have those things, I would leave it for later. Fair enough.

A better priority order for AI visibility work

The easiest way to make sensible decisions about llms.txt is to put it in the right order.

I would prioritise AI visibility work like this:

1. Access: Confirm important pages are crawlable and not blocked accidentally.
2. Clarity: Make core pages easier for people and machines to understand.
3. Structure: Add useful heading hierarchy, FAQs, Schema.org markup, and internal links.
4. Authority: Improve reviews, citations, directory consistency, LinkedIn presence, and third-party mentions.
5. Canonicality: Remove duplication, fix outdated pages, and make the current version obvious.
6. Optional experiments: Test llms.txt, llms-full.txt, or other protocol-layer ideas once the rest is in good shape.

I like this order because it reflects how visibility usually works in practice. The file that gets attention is rarely the first reason a site becomes useful. More often, it is the page itself.

And to be honest, that is the part people find annoying. It would be much nicer if AI visibility came down to dropping a markdown file into the root folder and calling it sorted. It usually does not.

Frequently Asked Questions

What is llms.txt and what is it actually meant to do?

llms.txt is a proposed Markdown file that gives AI systems a curated list of the pages you most want them to read. It is meant to improve guidance and context, especially on documentation-heavy websites, rather than control crawler access.

Should a normal business website add llms.txt right now?

Most normal business websites should not treat llms.txt as a priority yet. If your crawlability, content clarity, structured data, and core page quality still need work, those will usually matter more than publishing the file.

How is llms.txt different from robots.txt, sitemaps, and structured data?

robots.txt controls crawler access, sitemaps list URLs, and structured data adds meaning to content elements such as products, FAQs, and organisations. llms.txt is different because it is a curated guidance file, not a control or indexing standard.

Which websites are most likely to benefit from llms.txt?

Documentation portals, API products, developer platforms, and large knowledge bases are the strongest candidates because they often have canonical resources, markdown exports, and more complex information architecture. Simpler brochure sites and local business websites are less likely to see meaningful benefit.

Conclusion

llms.txt is interesting. I do not think it is urgent.

If your site is documentation-heavy, technically tidy, and already strong on the basics, it is worth testing. For most other websites, it belongs in the optional-experiments bucket rather than the must-do list.

So before you add another file to your root directory, audit the pages that actually win business. Make them clearer, more crawlable, and easier to verify. Then, if you are still keen, test llms.txt from a position of strength instead of hope.