Nobody's reading your llms.txt

Somebody, somewhere, is selling you an llms.txt right now. It's the hot new file for "AI SEO": put it on your site, and ChatGPT, Claude and friends will understand you, find you and cite you.

So I checked whether they ever pick it up.

I run the Edge, a front door in front of 3,584 websites that logs every visitor that knocks. For 13 days I watched every crawler from OpenAI, Anthropic, Google, Microsoft, Apple, Amazon, Meta and Perplexity. They came 441,515 times. They read robots.txt 192,087 times.

They asked for llms.txt zero times.

First, what is llms.txt?

Every website already has a couple of files that are written for machines, not people:

• robots.txt is the sign on the door. It tells crawlers what they may and may not visit, and the polite ones read it before anything else.
• sitemap.xml is the map. It lists every page so a search engine doesn't miss any.

llms.txt wants to be the third. Jeremy Howard of Answer.AI proposed it in September 2024: a short Markdown file at /llms.txt that sums up your site and links to its most useful pages, so an AI model can skip your menus, cookie banners and scripts and get straight to the good part.

It's a nice idea. The problem is the bit the sales pitch skips: a file at a fixed address only does anything if someone comes to that address and asks for it.

The AI companies never asked

Here is every big AI and search crawler that visited, counted only when it came from its own company's network (more on the fakes in a second):

Crawler	Sites visited	llms.txt
ClaudeBot Anthropic	3,028	0
OAI-SearchBot OpenAI	740	0
GPTBot OpenAI	439	0
bingbot Microsoft	323	0
Googlebot Google	267	0
Bytespider ByteDance	236	0
Amazonbot Amazon	189	0
ChatGPT-User OpenAI	105	0
Applebot Apple	76	0
PerplexityBot Perplexity	37	0
DuckAssistBot DuckDuckGo	32	0
CCBot Common Crawl	18	0
meta-externalagent Meta	17	0
GoogleOther Google	9	0

Fourteen crawlers. 441,515 requests between them, 423,350 from ClaudeBot alone. Not one for /llms.txt, /llms-full.txt, or anything with "llms" in it.

A few rows are worth a second look:

• ClaudeBot visited 3,028 of the sites and read robots.txt 182,628 times. It never once asked for llms.txt. Anthropic publishes an llms.txt for its own developer docs. Its crawler doesn't look for yours.
• GPTBot isn't taking my word for it. I checked every one of its 2,030 requests against the IP list OpenAI publishes. All genuine, 439 sites, zero llms.txt.
• ChatGPT-User is the one that fetches a page when a real person asks ChatGPT about it, which is exactly the moment llms.txt was designed for. It came 622 times. It went for the pages.

Even the fakes don't bother

A user agent is just a name a bot gives itself, and plenty of bots lie. Count every request that claimed to be an AI or search crawler, fakes included, and it's 490,095 requests under 22 different names. Still zero for llms.txt.

That includes one fleet on Google Cloud that wore 20 different crawler costumes (GPTBot, ClaudeBot, PerplexityBot, Applebot, you name it) and cycled through the same 119 sites, about 2,400 requests per costume. It looks like someone testing how websites treat each AI bot. Whoever built it didn't think llms.txt was worth checking either.

Everyone reads robots.txt

Every file at the root of a website gets asked for by somebody. So here's llms.txt next to the files it's trying to sit beside, counting any visitor at all, not just AI crawlers:

Who gets asked for what

Share of the 3,584 sites where anyone requested each file · Oct 3–16

• robots.txt was asked for on 98.7% of the sites, 209,089 times.
• llms.txt was asked for on 6.4% of them, 421 times. That's about 500 times fewer requests.
• ClaudeBot's robots.txt reads on their own outnumber every llms.txt request from everybody, combined, by 420 to 1.
• security.txt, a file most people have never heard of, got asked for on 7 times as many sites as llms.txt.

Out of 5.3 million requests in total, one in 12,283 was for llms.txt. And most of those didn't come from AI.

So who is reading it?

330 different addresses asked for llms.txt, 435 times in all. When I sorted them by who they say they are and how they behave, a pattern jumped out.

Who asked for llms.txt

Share of the 435 llms.txt requests · Oct 8–16

The people reading your llms.txt are mostly the people selling it to you

• Web-data companies (26%). Dataprovider.com and BuiltWith catalogue which websites use which technology. Your llms.txt becomes a row in their "llms.txt adoption" stats.
• SEO tools (22%). SEOJuice, Keywords Everywhere, Semrush's site auditor and a handful of smaller ones. They check whether you have an llms.txt so they can tell you that you need one.
• Audits and an "AI readiness" census (7%). Page-speed tests someone ran, plus a research crawler that also checks for half a dozen other proposed AI files.

Add those up and 55% of all llms.txt requests came from tools whose job is to check whether you have one. The file's biggest audience is its own sales funnel.

The rest is a scraper pretending to be people

The single biggest group, 40%, never says who it is. It's 157 addresses on three Swedish hosting companies, rotating between 16 browser identities: an old Firefox here, an iPhone there. But every one of its requests carries the same technical fingerprint, the kind left by a scripting library, not a browser. Each address visits two or three sites and moves on, which keeps it under rate limits. It grabs llms.txt and the homepage and follows the document links.

Whoever runs it does want a tidy summary of your site. They just don't want you to know who they are, and only 27 of its 157 addresses ever bothered to read robots.txt.

And exactly one visitor looked like a curious human: someone on home broadband, trying /contact/llms.txt, /contact/llms.md and /faqs.md by hand on two sites. Hi. Sorry about the 404.

Why llms.txt won't help your SEO

Put the numbers together and the answer is fairly blunt.

A file nobody fetches can't change anything. SEO is about what search engines and AI assistants actually read. Across 3,584 sites, the companies building those assistants never once tried the address where llms.txt lives. Whatever's in yours, it isn't being weighed, ranked or cited.

Search ranks pages, not your description of them. llms.txt is your site describing itself. Search engines learned long ago not to trust that, which is why Google's John Mueller compared llms.txt to the old keywords meta tag: a list of claims anyone can check against the real pages. Google's Gary Illyes has said Google doesn't support it and has no plans to.

AI answers come from your actual pages. When ChatGPT, Claude or Perplexity cite a site, it's a page their crawler fetched, or one a search index fetched for them. You can see it in the table above: OAI-SearchBot, ClaudeBot and the rest read robots.txt, sitemaps and HTML. That's the whole pipeline.

It's a second copy that goes stale. Even if someone did read it, llms.txt is another place to keep your site's story up to date. The pages already are the story.

Is it ever worth having?

It's not useless everywhere. If you write developer documentation, an llms.txt can be a handy single link for a person to paste into a coding assistant, and some tools will follow it when someone points them at it. That's a convenience for your readers, not a ranking signal.

It also won't hurt, and it isn't a red flag in your logs: just 1.2% of llms.txt askers ever tried a known attack path, against 28.4% of all visitors, and none of them did it after asking.

So if you have one, leave it. Just don't pay for it, and don't expect it to bring anyone.

What to do instead

If you want AI assistants and search engines to find and cite you, spend the time on the files and pages they do read:

• Check your robots.txt. It's the one file every crawler reads first, 192,087 times here. Make sure it isn't blocking the crawlers you want, like OAI-SearchBot, ClaudeBot, PerplexityBot or Googlebot, and decide on purpose which ones you'd rather keep out.
• Have a sitemap.xml and keep it current. Bots asked for it on 88% of these sites.
• Put your words in the HTML. If your content only appears after a pile of JavaScript runs, many crawlers will see an empty page. View the page source; if your text isn't there, they may not see it either.
• Answer the question on the page. A clear page that plainly says what you do, for whom, and how much, is what ends up quoted.
• Make sure the pages load. Fast, reliable, no error pages. A crawler that gets a 500 moves on.

None of it is new, and none of it comes in a shiny new file. That's sort of the point.

How I counted

• The data: access logs from the Edge, covering 3,584 hostnames, 5,343,032 requests from 60,612 addresses, 3 to 16 October 2026. Most sites joined between 8 and 12 October.
• None of these sites publishes an llms.txt. Everyone who asked got a 404. So this measures who goes looking for one, not what a crawler would do with one it found. But looking is the only way to find it: the file lives at one fixed address, and the AI companies' crawlers never tried that address once.
• AI crawlers count only when the name matches and the request came from that company's network (Google's for Googlebot, Microsoft's for bingbot, Amazon's for ClaudeBot and so on). GPTBot was checked against OpenAI's published list. The fakes figure includes everything that only claimed the name.
• "llms.txt requests" means any path with llms in it: /llms.txt, /llms-full.txt, /contact/llms.txt and friends. I post what the bots try each day in the daily reports, and I'm @MrNiceTry on X.