llms.txt, robots.txt and What AI Actually Reads on Your Website

Every website has two audiences now: people, and the machines that read on their behalf. When a prospect asks ChatGPT or Claude to recommend a provider in your category, the answer is assembled from what those systems were able to read. Generative engine optimisation (GEO, sometimes called LLMO) starts with a dull question. Can they read your site at all?

Two small text files sit at the centre of that question, and they do very different jobs.

robots.txt: the gate

robots.txt has been around since the 1990s. It lives at the root of your domain and tells automated visitors what they may and may not fetch. Reputable AI companies honour it.

What has changed is the list of visitors. These are the ones worth knowing.

User agent	Operator	What it is for
GPTBot	OpenAI	Collecting content to train models
OAI-SearchBot	OpenAI	Indexing for ChatGPT’s search results
ChatGPT-User	OpenAI	Fetching a page because a user asked about it
ClaudeBot	Anthropic	Collecting content to train models
Claude-SearchBot	Anthropic	Indexing for search
Claude-User	Anthropic	Fetching a page on a user’s request
PerplexityBot	Perplexity	Indexing for answers
Perplexity-User	Perplexity	Fetching on a user’s request
Google-Extended	Google	A control token: whether your content may be used for Gemini. It does not affect Google Search

The distinction that matters is between training, search indexing and user-triggered fetching. You can reasonably feel differently about each. A publisher may block training and allow search. A services business usually wants all three, because being absent from the answer costs more than being present in the training data.

Names change. One of the sites we look after was still listing an Anthropic agent that had been retired. Review the list once or twice a year.

Three ways sites block AI without meaning to

1. The host or CDN does it for you. Some platforms now block AI crawlers by default, and Cloudflare made that the default for new sites in 2025. It is a sensible default for publishers and an invisible own goal for a business that wants to be recommended. Check the setting.

2. An old blanket rule. A Disallow: / left over from a staging site, or a security plugin that blocks “unknown bots”, catches AI crawlers along with everything else.

3. The content is not in the HTML. Most AI crawlers do not run JavaScript. If your pages are assembled in the browser, a crawler may receive an empty shell. View the page source. If your headline and body text are not there, neither are you. This is one reason we build on Astro, which sends finished HTML.

llms.txt: the guide

llms.txt is a proposal from 2024. It is a Markdown file at yoursite.com/llms.txt containing a short description of the organisation followed by annotated links to the pages that matter most. Where robots.txt says what may be read, llms.txt says what is worth reading.

A reasonable one looks like this:

# Example Co

> One or two sentences: what you do, for whom, and where.

## Services
- [Service name](https://example.com/service): one line on what it is

## Proof
- [Case study](https://example.com/work/client): the result in a sentence

## Contact
- [Get in touch](https://example.com/contact)

We run one on this site and add one to every site we build.

The honest status report

It would be easy to oversell this, so plainly:

• No major AI provider has committed to using llms.txt when generating answers. Google has said it does not. Adoption so far is mostly by websites, not by the systems reading them.
• It is genuinely used by some tools: coding agents, documentation assistants, and anyone who pastes the file into a conversation to brief an AI about a company quickly.
• It costs almost nothing. Twenty minutes to write, no maintenance beyond updating it when your services change.
• Writing it is clarifying. If you cannot describe your organisation in two sentences and eight links, that is worth knowing.

Our advice: create one, do not expect it to move a number on its own, and spend the real effort on the fundamentals.

What moves the needle instead

1. Be crawlable. Gate open, host not blocking, content in the HTML.
2. Say what you do in plain sentences near the top of each page, in words a stranger would use.
3. Answer real questions. FAQ sections phrased the way people ask, marked up as structured data.
4. Use schema. Organisation, service, local business, article and FAQ markup remove guesswork.
5. Keep a current sitemap and make sure it is the one the search engines hold.
6. Be consistent elsewhere. Assistants cross-check your site against directories, profiles and reviews.

A ten-minute audit

• Open /robots.txt. Any Disallow against the agents above? Any you do not recognise?
• Ask your host whether AI bots are blocked by default.
• View source on your homepage and one service page. Is the main text present?
• Open /llms.txt. Does it exist, and is it current?
• Ask two assistants what your business does and who it suits. Note what they get wrong.

If the answers worry you, send us your URL. We will tell you what an AI can and cannot see.