TapCub is live — analytics, insights and live chat, free on one platform
Tag

robots.txt

12 articles tagged “robots.txt”. Tags cut across categories, so you may find analytics, chat and product notes side by side.

Tagged robots.txt

12 articles

Sample data: one site, 30 days, server records. TapCub counts SemrushBot, SiteAuditBot and SplitSignalBot in one SEO row (4,800 + 2,300 + 420). After Disallow, OCOB fell 1,900 → 60 (−96.8%); the SEO row stayed at 7,520.
AI SearchNov 16, 2026 · 10 min read

SemrushBot: The 10-Agent Family and How to Control Each One

SemrushBot is a family of ten Semrush crawlers with independent robots.txt tokens. What each does, how to keep SEO data while opting out of SemrushBot-OCOB, how to verify requests, and how to decide per goal.

Read more
Sample data: PerplexityBot read 8,000 pages and sent back 120 visitors (1.5%, ≈67 reads per visitor); the training crawler read 6,000 pages and sent back ≈0 (0%). In TapCub the ledger shows the same row as about 67 reads : 1 visitor, and AI visibility shows Crawl→visit 1.5%.
AI SearchNov 16, 2026 · 10 min read

PerplexityBot: What It Reads, What It Sends Back, and Whether to Allow It

PerplexityBot reads pages so Perplexity answers can cite your site; Perplexity-User fetches on user request. What each does, robots.txt rules, IP verification, and a reads-vs-visitors rule for allowing or blocking.

Read more
Sample data: 30-day window with server records. After the day-15 rule, CCBot only fetched robots.txt (~2 checks/day × 15 days = 30). TapCub runs no DNS check for CCBot, so trust shows Unverified; verify with reverse DNS or ccbot.json.
AI SearchNov 16, 2026 · 9 min read

CCBot: Common Crawl's Dataset Crawler and What Blocking It Changes

CCBot builds Common Crawl's free public dataset — a documented training source for GPT-3, LLaMA and BLOOM. What it fetches, how to verify it, robots.txt rules, and why blocking only shapes future snapshots.

Read more
Sample data: server records. Applebot verified by reverse DNS to applebot.apple.com; Extended never crawls; the wrong rule leaves only robots.txt fetches.
AI SearchNov 16, 2026 · 10 min read

Applebot and Applebot-Extended: Two Agents and How to Control Each One

Applebot crawls for Spotlight, Siri and Safari search; Applebot-Extended is only a robots.txt switch for AI training. What each does, three rule choices, DNS and IP verification, and a worked example of what the wrong block costs.

Read more
Sample data: demo project with server records, last 30 days. After the rule, Amazonbot only fetched robots.txt. TapCub runs no DNS check for Amazon, so trust shows Unverified; check source IPs against the Amazon IP file.
AI SearchNov 16, 2026 · 10 min read

Amazonbot: Amazon's Three Crawlers and How to Control Each One

Amazonbot is Amazon's crawler whose reads may be used to train Amazon AI models — one of three Amazon agents with independent robots.txt settings. What each does, rules per goal, IP verification, and a worked example of reads versus traffic sent back.

Read more
Sample data: TapCub counts both Ahrefs user agents in one AhrefsBot row (38,400 + 3,400 requests). Trust shows Unverified because TapCub runs no DNS check for Ahrefs; verify with reverse DNS or the Ahrefs IP list.
AI SearchNov 16, 2026 · 9 min read

AhrefsBot: What It Does and Whether to Allow or Block It

AhrefsBot reads your site for Ahrefs users — including competitors — and sends back no visitors. What its two crawlers do, robots.txt allow/block/rate-limit rules, verification, and how to decide.

Read more
Illustrative numbers. Block violators at the server, not in robots.txt.
AI SearchUpdated NovNov 3, 2026 · 9 min read

robots.txt Disallow All: What It Blocks, What It Doesn't, and Safer Variants

Two lines in robots.txt ask every crawler to stay off your whole site. What they actually do, what they do not (they do not hide URLs or protect anything), common variants and how Google resolves conflicts, and how to check which crawlers obey.

Read more
Illustrative numbers. Success means a status below 400.
AI SearchNov 3, 2026 · 5 min read

Crawl Budget: Does It Matter for Your Site, and Where Is It Going?

Crawl budget is how much a search engine is willing and able to crawl on your site. When it matters, what wastes it, and how to read server logs to see where search and AI crawlers actually spend their requests.

Read more
Demo workspace with log import. After the rule, GPTBot only fetched robots.txt.
AI SearchNov 3, 2026 · 6 min read

GPTBot: OpenAI's Four Crawlers and How to Control Each One

GPTBot is OpenAI's training crawler, one of four OpenAI agents with independent robots.txt settings. What each does, what opting out changes, rules for common goals, and how to verify and check them in your own data.

Read more
Demo workspace. Requests claiming ClaudeBot from addresses outside Anthropic's list are not verified.
AI SearchNov 3, 2026 · 6 min read

ClaudeBot: What It Crawls, How to Block It and How to Verify It

ClaudeBot is one of three Anthropic web agents. What each one does, what blocking it changes, robots.txt rules for common goals, how to verify requests against Anthropic's published IP list, and how to see ClaudeBot in your own data.

Read more
Sample from the demo project. The compliance page fetches both files and compares robots.txt rules with real fetches; llms.txt fetch counts come from your server logs.
AI SearchUpdated NovSep 17, 2026 · 8 min read

llms.txt: what to write, an example, and how to check whether anyone reads it

What llms.txt is and is not, the structure the proposal defines, a real example, how it should agree with robots.txt, and how to check in your server logs whether AI agents fetch it.

Read more
Sample workspace. Highlighted row: the operator that sends the most people per read.
AI SearchUpdated NovMar 26, 2026 · 9 min read

How to block AI crawlers: user agents, robots.txt rules and how to check them

GPTBot, ClaudeBot, Google-Extended and the rest do different jobs. Which AI crawlers to block, the robots.txt rules to use, what to enforce at the server, and how to check it worked.

Read more
Newsletter

One email a month. Only when there is something worth reading.

New articles, what shipped in the product, and the occasional checklist you can forward to your team.

  • Articles first. Every issue leads with the newest long-form note, summarised in three lines.
  • Product changes that matter. New reports, new SDKs, changed defaults. No marketing filler.
  • Unsubscribe in one click. The link is in every email and it works immediately.

Sent roughly once a month. We never share your address.

Topics you care about

By subscribing you agree to receive the TapCub newsletter. See the privacy policy.

You are on the list.

The next issue goes out at the start of the month. Until then, every article is here on the blog.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics