llms.txt, robots.txt and AI Crawlers: What to Allow, What to Block

The decision you are actually making

When people ask "should we block AI bots?", they are usually mixing two questions. One is whether your content should help train future AI models. The other is whether AI assistants should be able to read your pages and cite them when answering users today. Most major AI companies now run separate crawlers for these jobs, which means you can answer the two questions differently.

For a business that wants to be recommended by ChatGPT, Perplexity or Claude, blocking the search crawlers is almost always a mistake. Blocking the training crawlers is a judgment call, and a defensible one.

The main AI user agents

This list reflects what the companies document publicly as of this update. Crawler names and behavior change, so check each vendor's documentation before you ship a robots.txt change.

Three robots.txt setups

Setup A: Maximum visibility

Allow everything. This suits most marketing sites, SaaS companies and service businesses whose public content exists to be found. If you have no robots.txt rules for AI bots, this is effectively what you already have, provided your firewall agrees.

Setup B: Search yes, training no

Allow search and user-triggered crawlers, block training crawlers. This is the most common choice for publishers and brands with original research they want credited but not absorbed.

Setup C: Protect specific sections

Keep marketing pages open to everyone and close paid content, customer areas or proprietary datasets to all AI crawlers. Remember that robots.txt is a request, not a lock. Anything truly private needs authentication.

Weighing the trade-off honestly

Blocking training crawlers has a cost that is hard to measure. Models learn what they know about your brand partly from training data. If your own pages are excluded, the model's background picture of you comes entirely from what other sites say. For a well-known brand with plenty of coverage, that may be fine. For a smaller company, your own site might be the clearest description of what you do.

There is also a practical limit: content excluded from one crawler can still reach models through other routes, such as third-party sites quoting you or archives collected before you added the block. Blocking is a statement of preference that reputable crawlers respect going forward. It is not a guarantee.

Our view: if your public content exists to win customers, Setup A or B is right. Choose B if you have original research, paid content or a strong position on training. Do not block search crawlers unless you have a legal or contractual reason.

What llms.txt is, and what it is not

llms.txt is a proposal published in 2024 by Jeremy Howard of Answer.AI. The idea is a Markdown file at /llms.txt that gives language models a short, curated map of a site: a summary at the top, then links to the most useful pages with one-line descriptions. Some versions also offer an llms-full.txt with the full text of key documentation.

It is not a standard, and it is not an access control. It does not allow or block anything. Adoption by the major answer engines is uneven and mostly undocumented. Google has said publicly that it does not use llms.txt, and other engines have not made clear commitments either way. Where it has clear value today is with developer tools and AI coding assistants that read documentation, which is why many software companies publish one for their docs.

Should you publish one? It takes under an hour and does no harm, so it is a reasonable addition once the basics are done. Just do not expect it to change your visibility in ChatGPT or Google by itself, and be wary of anyone selling it as the key to AI search.

The firewall problem

A surprising number of sites have a carefully written robots.txt and still block every AI crawler, because a CDN or security plugin is doing it upstream. Bot management products often include "block AI bots" toggles, and some have shipped with that setting turned on for new sites. The crawler never gets far enough to read robots.txt.

1. Check your CDN dashboard for AI bot or crawler settings.
2. Search your server or CDN logs for the user agents in the table above and look at the response codes. 403 and 429 responses mean they are being turned away.
3. Test a key page with a user agent switcher or curl, while remembering some providers verify bots by IP as well as user agent.
4. Look for security plugins (on WordPress especially) with bot-blocking rules.

The ZEPREACH AI Visibility Score reads your robots.txt, checks for llms.txt and flags blocked AI crawlers as part of its report, which makes a useful first pass before you dig into logs.

A short maintenance routine

• Review robots.txt every quarter against each vendor's current crawler documentation.
• Keep a comment in robots.txt explaining why each AI rule exists, so the next developer does not "clean it up".
• Watch logs for new AI user agents and decide on each one deliberately.
• Re-test after any CDN, hosting or security plugin change.