Skip to content
ZEPREACH

Notes from the answer engine trenches.

What we have learned getting brands cited by AI. No fluff, no recycled listicles. Updated when the engines change.

Latest · Guide

llms.txt, robots.txt and AI Crawlers: What to Allow, What to Block

Which AI bots should reach your site, what llms.txt can and cannot do, and how to block model training without vanishing from AI answers.

Read the guide
  • robots.txt is the control that matters. Most AI companies run separate crawlers for training and for search, and you can treat them differently.
  • Blocking training crawlers while allowing search crawlers is a reasonable default for brands that want citations but are wary of model training.
  • llms.txt is a community proposal, not a standard. Support across engines is uneven, and Google has said it does not use it. It is cheap to publish but should not be your priority.
  • Check your CDN and firewall too. Many AI bots are blocked there, not in robots.txt.
AEO glossary24 terms, defined in plain English

Free AEO score

Find out what AI says about you.

Run the free AEO score on your site. You will see which engines can read you, what is missing, and what to fix first.

https://

Free. Takes about 20 seconds. No sign-up needed to see your score.

Prefer a human? Book a 30-minute strategy call