What is an AI crawler?

Three kinds of AI crawler

"AI crawler" covers bots with very different effects on your site:

• Training crawlers collect pages in bulk to train future models. They bring no visits back.
• AI search fetchers fetch pages when a user asks a question, to quote or summarise them in an answer, often with a link.
• AI agents browse on behalf of one user — comparing prices, filling a form, reading documentation.

Most AI companies run separate user agents for these purposes, so you can allow one and block another.

robots.txt and llms.txt

robots.txt is the standard way to tell crawlers what they may fetch. You can address AI crawlers by name — for example disallow a training bot but allow the search fetcher from the same company. It is a request, not an enforcement: well-behaved crawlers follow it, others do not.

llms.txt is a newer, informal convention: a Markdown file at the site root that points AI systems to the pages that best explain your product. It does not grant or deny access; it is a reading list.

Why measure them

AI crawlers matter in two directions. Training crawlers consume bandwidth and content with no direct return; AI search fetchers decide whether your pages are quoted when people ask assistants about your category. Measuring them answers practical questions:

• Which AI companies fetch your site, and how often?
• Which pages do they read — documentation, pricing, the blog?
• Do they respect the rules you set in robots.txt?
• Does AI search fetching turn into visits from AI assistants?

How TapCub shows AI crawlers

TapCub's bot detection labels each AI crawler with its operator and purpose — AI training, AI search or AI agent — and keeps them out of human numbers. They have their own ledger, the AI content ledger, showing which bots fetched which pages and when.

For rules, TapCub takes snapshots of your robots.txt and llms.txt, compares actual fetches with the robots.txt rules and counts violations by day, with the matching rule line as evidence. On the other side, visits that people make from AI assistants are classified as the AI assistant channel, so you can see whether being read turns into being visited. To see what your robots.txt allows each AI crawler to do, paste it into the free AI crawler robots.txt checker.

For the user agents and robots.txt rules that block specific AI crawlers, see how to block AI crawlers; to point them at the pages you do want read, see the llms.txt guide.

Common mistakes

• Blocking everything with "AI" in the name. Blocking search fetchers can remove you from AI answers while barely affecting training.
• Assuming robots.txt is enforced. Measure whether each bot follows it.
• Counting AI fetches as visitors. They inflate traffic and depress conversion rates if they are not separated.
• Writing llms.txt and forgetting it. It should point to pages that are current and accurate.

Related guides: GPTBot, ClaudeBot and robots.txt Disallow All.

See also

• Google-Extended: What It Controls and How to Use It in robots.txt
• Meta-ExternalAgent: What Meta’s AI Crawler Does and How to Control It
• Bytespider: What It Is and How to Control It
• Web crawler
• AI visibility