TapCub is live — analytics, insights and live chat, free on one platform
Tag

Bots

22 articles tagged “Bots”. Tags cut across categories, so you may find analytics, chat and product notes side by side.

Tagged Bots

22 articles

Sample data: one site, 30 days, server records. TapCub counts SemrushBot, SiteAuditBot and SplitSignalBot in one SEO row (4,800 + 2,300 + 420). After Disallow, OCOB fell 1,900 → 60 (−96.8%); the SEO row stayed at 7,520.
AI SearchNov 16, 2026 · 10 min read

SemrushBot: The 10-Agent Family and How to Control Each One

SemrushBot is a family of ten Semrush crawlers with independent robots.txt tokens. What each does, how to keep SEO data while opting out of SemrushBot-OCOB, how to verify requests, and how to decide per goal.

Read more
Sample data: PerplexityBot read 8,000 pages and sent back 120 visitors (1.5%, ≈67 reads per visitor); the training crawler read 6,000 pages and sent back ≈0 (0%). In TapCub the ledger shows the same row as about 67 reads : 1 visitor, and AI visibility shows Crawl→visit 1.5%.
AI SearchNov 16, 2026 · 10 min read

PerplexityBot: What It Reads, What It Sends Back, and Whether to Allow It

PerplexityBot reads pages so Perplexity answers can cite your site; Perplexity-User fetches on user request. What each does, robots.txt rules, IP verification, and a reads-vs-visitors rule for allowing or blocking.

Read more
Sample data: 30-day window with server records. After the day-15 rule, CCBot only fetched robots.txt (~2 checks/day × 15 days = 30). TapCub runs no DNS check for CCBot, so trust shows Unverified; verify with reverse DNS or ccbot.json.
AI SearchNov 16, 2026 · 9 min read

CCBot: Common Crawl's Dataset Crawler and What Blocking It Changes

CCBot builds Common Crawl's free public dataset — a documented training source for GPT-3, LLaMA and BLOOM. What it fetches, how to verify it, robots.txt rules, and why blocking only shapes future snapshots.

Read more
Sample data: server records. Applebot verified by reverse DNS to applebot.apple.com; Extended never crawls; the wrong rule leaves only robots.txt fetches.
AI SearchNov 16, 2026 · 10 min read

Applebot and Applebot-Extended: Two Agents and How to Control Each One

Applebot crawls for Spotlight, Siri and Safari search; Applebot-Extended is only a robots.txt switch for AI training. What each does, three rule choices, DNS and IP verification, and a worked example of what the wrong block costs.

Read more
Sample data: demo project with server records, last 30 days. After the rule, Amazonbot only fetched robots.txt. TapCub runs no DNS check for Amazon, so trust shows Unverified; check source IPs against the Amazon IP file.
AI SearchNov 16, 2026 · 10 min read

Amazonbot: Amazon's Three Crawlers and How to Control Each One

Amazonbot is Amazon's crawler whose reads may be used to train Amazon AI models — one of three Amazon agents with independent robots.txt settings. What each does, rules per goal, IP verification, and a worked example of reads versus traffic sent back.

Read more
Sample data: TapCub counts both Ahrefs user agents in one AhrefsBot row (38,400 + 3,400 requests). Trust shows Unverified because TapCub runs no DNS check for Ahrefs; verify with reverse DNS or the Ahrefs IP list.
AI SearchNov 16, 2026 · 9 min read

AhrefsBot: What It Does and Whether to Allow or Block It

AhrefsBot reads your site for Ahrefs users — including competitors — and sends back no visitors. What its two crawlers do, robots.txt allow/block/rate-limit rules, verification, and how to decide.

Read more
Sample data. One week of server-side crawl data.
AI SearchNov 4, 2026 · 4 min read

Bytespider: What It Is and How to Control It

Bytespider is the web crawler operated by ByteDance, the company behind TikTok. Site owners often notice it because of its request volume. Its user agent, a worked week measuring its load and robots.txt compliance, and the options from robots.txt to a firewall block.

Read more
Sample data and summary of Google’s documentation at the time of writing.
AI SearchNov 4, 2026 · 4 min read

Google-Extended: What It Controls and How to Use It in robots.txt

Google-Extended is a robots.txt token, not a separate crawler. It lets you choose whether content Google crawls may be used for Gemini models and some other AI products, without affecting Google Search. What it controls, what it does not, and how to set it.

Read more
Sample data. One month of server-side crawl data.
AI SearchNov 4, 2026 · 4 min read

Meta-ExternalAgent: What Meta’s AI Crawler Does and How to Control It

Meta-ExternalAgent is the crawler Meta uses to collect web content for AI training and indexing. How it differs from Meta’s link preview crawler, how to verify it, a worked month comparing what it reads with the visitors Meta sends back, and robots.txt rules.

Read more
Sample data. One week of server-side crawl data.
AI SearchNov 4, 2026 · 4 min read

Bingbot: What It Is, How to Verify It and Control Its Crawling

Bingbot is Microsoft’s web crawler for Bing search. Its user agents, how to verify real Bingbot by reverse DNS, a worked week comparing it with Googlebot (with fake Bingbot traffic), how to control its crawl rate and how IndexNow speeds up discovery.

Read more
Sample data. One week of Googlebot fetches by status.
AI SearchNov 4, 2026 · 5 min read

Crawl Errors: Types, Causes and How to Fix Them

Crawl errors are failed requests from search engine bots: DNS and server errors, timeouts, 404s and redirect problems. The main types, what causes them, a worked week of Googlebot data, and the order to fix them in.

Read more
Sample data. One request in eight claiming to be Googlebot was not Google.
AI SearchNov 4, 2026 · 5 min read

Googlebot: What It Is, How It Crawls and How to Verify It

Googlebot is the crawler Google uses to fetch pages for Search. How it crawls and renders, the other Google crawlers you will see in logs, how to tell a real Googlebot from a fake one, and a sample month of log data.

Read more
Sample results. Two failures after a Thursday release: a renamed event and a removed button.
AnalyticsNov 4, 2026 · 6 min read

What Is Data Quality? The Six Dimensions, Applied to Analytics Data

Data quality is how far data can be trusted for the decision you want to make. The six dimensions (accuracy, completeness, consistency, timeliness, validity and uniqueness) with the analytics failure each one catches and a weekly check list.

Read more
Illustrative numbers. Each column adds up to all fetches or all organic visits.
AI SearchNov 3, 2026 · 5 min read

Log File Analysis for SEO: A Tool-Neutral Method in Six Questions

Server logs show what search engine crawlers actually request, not what a crawl tool assumes. How to get the logs, verify that Googlebot is real, and answer six questions that turn raw lines into SEO fixes.

Read more
Illustrative numbers. Block violators at the server, not in robots.txt.
AI SearchUpdated NovNov 3, 2026 · 9 min read

robots.txt Disallow All: What It Blocks, What It Doesn't, and Safer Variants

Two lines in robots.txt ask every crawler to stay off your whole site. What they actually do, what they do not (they do not hide URLs or protect anything), common variants and how Google resolves conflicts, and how to check which crawlers obey.

Read more
Illustrative numbers. Per page per week = fetches ÷ pages ÷ 4.3.
AI SearchNov 3, 2026 · 5 min read

How Often Does Google Crawl a Site? Measure Yours, Then Speed It Up

Google has no fixed crawl schedule: your home page may be fetched many times a day and a deep product page once a month. How to measure crawl frequency on your own site by page type, and what actually gets new pages crawled sooner.

Read more
Illustrative numbers. Success means a status below 400.
AI SearchNov 3, 2026 · 5 min read

Crawl Budget: Does It Matter for Your Site, and Where Is It Going?

Crawl budget is how much a search engine is willing and able to crawl on your site. When it matters, what wastes it, and how to read server logs to see where search and AI crawlers actually spend their requests.

Read more
Made-up domains and demo numbers. Spam rows share one page per session, zero duration and full bounce.
AnalyticsNov 3, 2026 · 5 min read

Referral Spam: Three Kinds, How to Spot Each, and Where to Stop It

Referral spam comes in three kinds: ghost hits that never touch your site, crawler visits and scripted browser visits. The report signals for each and the fix that belongs at each layer, from collection allowlists to filters.

Read more
Demo workspace with log import. After the rule, GPTBot only fetched robots.txt.
AI SearchNov 3, 2026 · 6 min read

GPTBot: OpenAI's Four Crawlers and How to Control Each One

GPTBot is OpenAI's training crawler, one of four OpenAI agents with independent robots.txt settings. What each does, what opting out changes, rules for common goals, and how to verify and check them in your own data.

Read more
Demo workspace. Requests claiming ClaudeBot from addresses outside Anthropic's list are not verified.
AI SearchNov 3, 2026 · 6 min read

ClaudeBot: What It Crawls, How to Block It and How to Verify It

ClaudeBot is one of three Anthropic web agents. What each one does, what blocking it changes, robots.txt rules for common goals, how to verify requests against Anthropic's published IP list, and how to see ClaudeBot in your own data.

Read more
Sample workspace. Highlighted row: the operator that sends the most people per read.
AI SearchUpdated NovMar 26, 2026 · 9 min read

How to block AI crawlers: user agents, robots.txt rules and how to check them

GPTBot, ClaudeBot, Google-Extended and the rest do different jobs. Which AI crawlers to block, the robots.txt rules to use, what to enforce at the server, and how to check it worked.

Read more
Sample workspace. The two spikes in the dashed line are crawler days; the human line does not move.
AnalyticsUpdated NovFeb 12, 2026 · 10 min read

Why analytics tools show different numbers: 8 causes and a reconciliation checklist

Two analytics tools on one site never print the same visitor count. Here are the eight causes, a worked example of where the gap goes, and a checklist to reconcile them.

Read more
Newsletter

One email a month. Only when there is something worth reading.

New articles, what shipped in the product, and the occasional checklist you can forward to your team.

  • Articles first. Every issue leads with the newest long-form note, summarised in three lines.
  • Product changes that matter. New reports, new SDKs, changed defaults. No marketing filler.
  • Unsubscribe in one click. The link is in every email and it works immediately.

Sent roughly once a month. We never share your address.

Topics you care about

By subscribing you agree to receive the TapCub newsletter. See the privacy policy.

You are on the list.

The next issue goes out at the start of the month. Until then, every article is here on the blog.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics