Bots
22 articles tagged “Bots”. Tags cut across categories, so you may find analytics, chat and product notes side by side.
Tagged Bots
22 articles

SemrushBot: The 10-Agent Family and How to Control Each One
SemrushBot is a family of ten Semrush crawlers with independent robots.txt tokens. What each does, how to keep SEO data while opting out of SemrushBot-OCOB, how to verify requests, and how to decide per goal.
Read more
PerplexityBot: What It Reads, What It Sends Back, and Whether to Allow It
PerplexityBot reads pages so Perplexity answers can cite your site; Perplexity-User fetches on user request. What each does, robots.txt rules, IP verification, and a reads-vs-visitors rule for allowing or blocking.
Read more
CCBot: Common Crawl's Dataset Crawler and What Blocking It Changes
CCBot builds Common Crawl's free public dataset — a documented training source for GPT-3, LLaMA and BLOOM. What it fetches, how to verify it, robots.txt rules, and why blocking only shapes future snapshots.
Read more
Applebot and Applebot-Extended: Two Agents and How to Control Each One
Applebot crawls for Spotlight, Siri and Safari search; Applebot-Extended is only a robots.txt switch for AI training. What each does, three rule choices, DNS and IP verification, and a worked example of what the wrong block costs.
Read more
Amazonbot: Amazon's Three Crawlers and How to Control Each One
Amazonbot is Amazon's crawler whose reads may be used to train Amazon AI models — one of three Amazon agents with independent robots.txt settings. What each does, rules per goal, IP verification, and a worked example of reads versus traffic sent back.
Read more
AhrefsBot: What It Does and Whether to Allow or Block It
AhrefsBot reads your site for Ahrefs users — including competitors — and sends back no visitors. What its two crawlers do, robots.txt allow/block/rate-limit rules, verification, and how to decide.
Read more
Bytespider: What It Is and How to Control It
Bytespider is the web crawler operated by ByteDance, the company behind TikTok. Site owners often notice it because of its request volume. Its user agent, a worked week measuring its load and robots.txt compliance, and the options from robots.txt to a firewall block.
Read more
Google-Extended: What It Controls and How to Use It in robots.txt
Google-Extended is a robots.txt token, not a separate crawler. It lets you choose whether content Google crawls may be used for Gemini models and some other AI products, without affecting Google Search. What it controls, what it does not, and how to set it.
Read more
Meta-ExternalAgent: What Meta’s AI Crawler Does and How to Control It
Meta-ExternalAgent is the crawler Meta uses to collect web content for AI training and indexing. How it differs from Meta’s link preview crawler, how to verify it, a worked month comparing what it reads with the visitors Meta sends back, and robots.txt rules.
Read more
Bingbot: What It Is, How to Verify It and Control Its Crawling
Bingbot is Microsoft’s web crawler for Bing search. Its user agents, how to verify real Bingbot by reverse DNS, a worked week comparing it with Googlebot (with fake Bingbot traffic), how to control its crawl rate and how IndexNow speeds up discovery.
Read more
Crawl Errors: Types, Causes and How to Fix Them
Crawl errors are failed requests from search engine bots: DNS and server errors, timeouts, 404s and redirect problems. The main types, what causes them, a worked week of Googlebot data, and the order to fix them in.
Read more
Googlebot: What It Is, How It Crawls and How to Verify It
Googlebot is the crawler Google uses to fetch pages for Search. How it crawls and renders, the other Google crawlers you will see in logs, how to tell a real Googlebot from a fake one, and a sample month of log data.
Read more
What Is Data Quality? The Six Dimensions, Applied to Analytics Data
Data quality is how far data can be trusted for the decision you want to make. The six dimensions (accuracy, completeness, consistency, timeliness, validity and uniqueness) with the analytics failure each one catches and a weekly check list.
Read more
Log File Analysis for SEO: A Tool-Neutral Method in Six Questions
Server logs show what search engine crawlers actually request, not what a crawl tool assumes. How to get the logs, verify that Googlebot is real, and answer six questions that turn raw lines into SEO fixes.
Read more
robots.txt Disallow All: What It Blocks, What It Doesn't, and Safer Variants
Two lines in robots.txt ask every crawler to stay off your whole site. What they actually do, what they do not (they do not hide URLs or protect anything), common variants and how Google resolves conflicts, and how to check which crawlers obey.
Read more
How Often Does Google Crawl a Site? Measure Yours, Then Speed It Up
Google has no fixed crawl schedule: your home page may be fetched many times a day and a deep product page once a month. How to measure crawl frequency on your own site by page type, and what actually gets new pages crawled sooner.
Read more
Crawl Budget: Does It Matter for Your Site, and Where Is It Going?
Crawl budget is how much a search engine is willing and able to crawl on your site. When it matters, what wastes it, and how to read server logs to see where search and AI crawlers actually spend their requests.
Read more
Referral Spam: Three Kinds, How to Spot Each, and Where to Stop It
Referral spam comes in three kinds: ghost hits that never touch your site, crawler visits and scripted browser visits. The report signals for each and the fix that belongs at each layer, from collection allowlists to filters.
Read more
GPTBot: OpenAI's Four Crawlers and How to Control Each One
GPTBot is OpenAI's training crawler, one of four OpenAI agents with independent robots.txt settings. What each does, what opting out changes, rules for common goals, and how to verify and check them in your own data.
Read more
ClaudeBot: What It Crawls, How to Block It and How to Verify It
ClaudeBot is one of three Anthropic web agents. What each one does, what blocking it changes, robots.txt rules for common goals, how to verify requests against Anthropic's published IP list, and how to see ClaudeBot in your own data.
Read more
How to block AI crawlers: user agents, robots.txt rules and how to check them
GPTBot, ClaudeBot, Google-Extended and the rest do different jobs. Which AI crawlers to block, the robots.txt rules to use, what to enforce at the server, and how to check it worked.
Read more
Why analytics tools show different numbers: 8 causes and a reconciliation checklist
Two analytics tools on one site never print the same visitor count. Here are the eight causes, a worked example of where the gap goes, and a checklist to reconcile them.
Read more