robots.txt
12 articles tagged “robots.txt”. Tags cut across categories, so you may find analytics, chat and product notes side by side.
Tagged robots.txt
12 articles

SemrushBot: The 10-Agent Family and How to Control Each One
SemrushBot is a family of ten Semrush crawlers with independent robots.txt tokens. What each does, how to keep SEO data while opting out of SemrushBot-OCOB, how to verify requests, and how to decide per goal.
Read more
PerplexityBot: What It Reads, What It Sends Back, and Whether to Allow It
PerplexityBot reads pages so Perplexity answers can cite your site; Perplexity-User fetches on user request. What each does, robots.txt rules, IP verification, and a reads-vs-visitors rule for allowing or blocking.
Read more
CCBot: Common Crawl's Dataset Crawler and What Blocking It Changes
CCBot builds Common Crawl's free public dataset — a documented training source for GPT-3, LLaMA and BLOOM. What it fetches, how to verify it, robots.txt rules, and why blocking only shapes future snapshots.
Read more
Applebot and Applebot-Extended: Two Agents and How to Control Each One
Applebot crawls for Spotlight, Siri and Safari search; Applebot-Extended is only a robots.txt switch for AI training. What each does, three rule choices, DNS and IP verification, and a worked example of what the wrong block costs.
Read more
Amazonbot: Amazon's Three Crawlers and How to Control Each One
Amazonbot is Amazon's crawler whose reads may be used to train Amazon AI models — one of three Amazon agents with independent robots.txt settings. What each does, rules per goal, IP verification, and a worked example of reads versus traffic sent back.
Read more
AhrefsBot: What It Does and Whether to Allow or Block It
AhrefsBot reads your site for Ahrefs users — including competitors — and sends back no visitors. What its two crawlers do, robots.txt allow/block/rate-limit rules, verification, and how to decide.
Read more
robots.txt Disallow All: What It Blocks, What It Doesn't, and Safer Variants
Two lines in robots.txt ask every crawler to stay off your whole site. What they actually do, what they do not (they do not hide URLs or protect anything), common variants and how Google resolves conflicts, and how to check which crawlers obey.
Read more
Crawl Budget: Does It Matter for Your Site, and Where Is It Going?
Crawl budget is how much a search engine is willing and able to crawl on your site. When it matters, what wastes it, and how to read server logs to see where search and AI crawlers actually spend their requests.
Read more
GPTBot: OpenAI's Four Crawlers and How to Control Each One
GPTBot is OpenAI's training crawler, one of four OpenAI agents with independent robots.txt settings. What each does, what opting out changes, rules for common goals, and how to verify and check them in your own data.
Read more
ClaudeBot: What It Crawls, How to Block It and How to Verify It
ClaudeBot is one of three Anthropic web agents. What each one does, what blocking it changes, robots.txt rules for common goals, how to verify requests against Anthropic's published IP list, and how to see ClaudeBot in your own data.
Read more
llms.txt: what to write, an example, and how to check whether anyone reads it
What llms.txt is and is not, the structure the proposal defines, a real example, how it should agree with robots.txt, and how to check in your server logs whether AI agents fetch it.
Read more
How to block AI crawlers: user agents, robots.txt rules and how to check them
GPTBot, ClaudeBot, Google-Extended and the rest do different jobs. Which AI crawlers to block, the robots.txt rules to use, what to enforce at the server, and how to check it worked.
Read more