What is AI Crawlers?
Key Takeaways
- AI companies run several different crawlers, and each has its own job: training, search indexing or fetching a page for a user request.
- Each crawler identifies itself with a user agent name published in the vendor documentation.
- A site controls access through robots.txt, and some vendors also document other controls.
- Allowing a crawler for search or answers is a different decision from allowing one that collects training material.
Understanding AI Crawlers
The same company can run more than one crawler. Treating them as one blocks too much or too little, so the decision is made per crawler and per purpose.
Because names and purposes change, the vendor documentation is the only reliable source for what each crawler does.
Many site owners choose to allow crawlers that fetch pages to answer users, because those can lead to citations, and to decide separately about training crawlers.
Related Terms
Frequently Asked Questions (FAQs)
Do all AI crawlers obey robots.txt?
The major vendors say their crawlers do. Others may not, and robots.txt cannot force compliance.
Will blocking an AI crawler remove my site from AI answers?
It stops that crawler from fetching pages. How a given AI product finds and uses content depends on that product.
Where do I find the crawler names?
On each vendor documentation page. Names and purposes change, so check them each time you edit your file.