AI crawlers and bots
Assistants fetch your pages to train, cite and answer. This series shows how to identify them, how robots.txt and llms.txt compare with real hits, and how many visitors they actually send back.
AI crawlers are now a visible share of traffic on most sites. These articles explain how TapCub separates them from search crawlers and people, how to read the crawler ledger, and how to write and verify an llms.txt file. The goal is a decision you can defend: allow, limit or block, based on what each bot takes and what it returns.
Articles in AI crawlers
2 articles
llms.txt: what to write, and how to check whether anyone reads it
llms.txt is an invitation, not a rule. A short guide to writing one that assistants actually use, and the ledger check that tells you whether they did.
Read moreGPTBot, ClaudeBot and the AI content ledger: who takes what
A ledger puts every AI crawler on its own line: fetches, what your robots.txt said, what llms.txt offered, and how many visitors came back. Then the allow-or-block decision is arithmetic.
Read moreOther categories
Each category answers a recurring question we hear from customers. Start with the one closest to your job.
Analytics
Definitions, funnels, retention and the data-quality questions behind every number.
2 articlesGrowth
AARRR, experiments and how to pick the one stage worth fixing first.
1 articleOperations
Running a site day to day: cookieless tracking, migrations, consent and performance.
2 articlesChat
Proactive invites, translation and turning support conversations into conversions.
2 articlesProduct
What shipped, why we built it that way, and how to turn it on.
1 article