Free tool AI bots directory: GPTBot, ClaudeBot, PerplexityBot and more
Which AI crawlers exist, who runs them, what they take your pages for, and how to allow or block each one. Every crawler page quotes the operator’s own documentation and links to it.
- 74 crawlers, same list as TapCub’s bot rules
- 24 pages with sourced facts
- Runs in your browser
Find a crawler, then decide
Search by crawler name, company or robots.txt token, or filter by what the crawler does. Linked names open a page with the operator’s own facts. Tick crawlers to build robots.txt rules below.
| Crawler | Operator | robots.txt token | Add |
|---|---|---|---|
| AI training · 26Collect pages for model training or large datasets. Blocking them does not affect regular search. | |||
| GPTBotOpenAICrawler page | OpenAI | GPTBot | |
| ClaudeBotAnthropicCrawler page | Anthropic | ClaudeBot | |
| anthropic-aiAnthropic | Anthropic | anthropic-ai | |
| Google-ExtendedGoogleCrawler page | Google-Extended | ||
| GoogleOtherGoogleCrawler page | GoogleOther | ||
| Applebot-ExtendedAppleCrawler page | Apple | Applebot-Extended | |
| BytespiderByteDance | ByteDance | Bytespider | |
| CCBotCommon CrawlCrawler page | Common Crawl | CCBot | |
| Meta-ExternalAgentMetaCrawler page | Meta | Meta-ExternalAgent | |
| FacebookBotMeta | Meta | FacebookBot | |
| AmazonbotAmazonCrawler page | Amazon | Amazonbot | |
| cohere-aiCohere | Cohere | cohere-ai | |
| DiffbotDiffbotCrawler page | Diffbot | Diffbot | |
| OmgiliWebz.ioCrawler page | Webz.io | Omgili | |
| TimpibotTimpi | Timpi | Timpibot | |
| PetalBotHuaweiCrawler page | Huawei | PetalBot | |
| AI2BotAllen InstituteCrawler page | Allen Institute | AI2Bot | |
| img2dataset— | — | img2dataset | |
| Kangaroo BotKangaroo LLM | Kangaroo LLM | No robots.txt token | |
| PanguBotHuawei | Huawei | PanguBot | |
| VelenPublicWebCrawlerVelen | Velen | VelenPublicWebCrawler | |
| WebzioWebz.io | Webz.io | Webzio | |
| Sidetrade indexer botSidetrade | Sidetrade | No robots.txt token | |
| ImagesiftBotHiveCrawler page | Hive | ImagesiftBot | |
| DeepSeekBotDeepSeek | DeepSeek | DeepSeekBot | |
| SemrushBot-OCOBSemrush | Semrush | SemrushBot-OCOB | |
| AI search · 14Index pages for AI answers that can cite and link you. Blocking them removes you from those answers. | |||
| OAI-SearchBotOpenAICrawler page | OpenAI | OAI-SearchBot | |
| Claude-WebAnthropic | Anthropic | Claude-Web | |
| Claude-SearchBotAnthropicCrawler page | Anthropic | Claude-SearchBot | |
| PerplexityBotPerplexityCrawler page | Perplexity | PerplexityBot | |
| YouBotYou.comCrawler page | You.com | YouBot | |
| DuckAssistBotDuckDuckGoCrawler page | DuckDuckGo | DuckAssistBot | |
| DoubaoByteDance | ByteDance | Doubao | |
| KimiMoonshot | Moonshot | No robots.txt token | |
| LinerBotLiner | Liner | LinerBot | |
| AndibotAndi | Andi | Andibot | |
| PhindBotPhind | Phind | PhindBot | |
| Brave Leo AIBrave | Brave | No robots.txt token | |
| iAskBotiAsk | iAsk | iAskBot | |
| Bigsur.aiBigsur | Bigsur | No robots.txt token | |
| AI agent · 13Open a page because a person asked an assistant. Some say robots.txt may not apply to them. | |||
| ChatGPT-UserOpenAICrawler page | OpenAI | ChatGPT-User | |
| Claude-UserAnthropicCrawler page | Anthropic | Claude-User | |
| Perplexity-UserPerplexityCrawler page | Perplexity | Perplexity-User | |
| Meta-ExternalFetcherMetaCrawler page | Meta | Meta-ExternalFetcher | |
| MistralAI-UserMistral AICrawler page | Mistral AI | MistralAI-User | |
| Cursor AgentCursor | Cursor | No robots.txt token | |
| DevinCognition | Cognition | No robots.txt token | |
| ManusManus | Manus | No robots.txt token | |
| OpenAI OperatorOpenAI | OpenAI | No robots.txt token | |
| Browser UseBrowser Use | Browser Use | No robots.txt token | |
| FirecrawlAgentFirecrawl | Firecrawl | FirecrawlAgent | |
| NovaActAmazon | Amazon | NovaAct | |
| QuillBotQuillBot | QuillBot | QuillBot | |
| Search engine · 21Classic search engines. Some feed AI answers too — see each crawler page. | |||
| Googlebot-ImageGoogle | Googlebot-Image | ||
| Googlebot-NewsGoogle | Googlebot-News | ||
| Googlebot-VideoGoogle | Googlebot-Video | ||
| Google-InspectionToolGoogle | Google-InspectionTool | ||
| AdsBot-GoogleGoogle | AdsBot-Google | ||
| BingbotMicrosoftCrawler page | Microsoft | Bingbot | |
| AdIdxBotMicrosoft | Microsoft | AdIdxBot | |
| Baiduspider百度 | 百度 | Baiduspider | |
| Baiduspider-render百度 | 百度 | Baiduspider-render | |
| 360Spider360 | 360 | 360Spider | |
| YisouSpider神马(阿里) | 神马(阿里) | YisouSpider | |
| ShenmaSpider神马 | 神马 | ShenmaSpider | |
| ToutiaoSpiderByteDance | ByteDance | ToutiaoSpider | |
| YandexBotYandex | Yandex | YandexBot | |
| DuckDuckBotDuckDuckGo | DuckDuckGo | DuckDuckBot | |
| SeznamBotSeznam | Seznam | SeznamBot | |
| ExabotExalead | Exalead | Exabot | |
| QwantifyQwant | Qwant | Qwantify | |
| MojeekBotMojeek | Mojeek | MojeekBot | |
| CoccocbotCốc Cốc | Cốc Cốc | Coccocbot | |
| PetalSearchHuawei | Huawei | PetalSearch | |
No crawler matches. Try the company name or part of the token.
Groups follow TapCub’s bot rules, which classify traffic by how it behaves. Each crawler page quotes what the operator itself says, and the two can differ — the page is the one to trust for a decision.
Build robots.txt rules
Ticked crawlers in the table become one group each. Paste the result at the end of your robots.txt; nothing else in the file changes.
An Allow group replaces User-agent: * for that crawler, so it can then reach paths your wildcard rules disallow. robots.txt is a request — check the result with the robots.txt tester.
# Block AI crawlers — generated with tapcub.com/tools/ai-bots
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /Version v1.0Updated Update history
Four facts that decide what to do
Every crawler page has the same table. These four rows answer most questions about a crawler.
What it is used for
Training, search or a user’s request. Blocking a training crawler keeps your pages out of future models; blocking a search crawler also removes you from answers that would have linked back.
Follows robots.txt
“Yes” is the operator’s own statement. “Partly” usually means fetches a person starts may skip robots.txt — ChatGPT-User, Perplexity-User and meta-externalfetcher say so.
Official IP list / reverse DNS
The only way to tell a real crawler from one that copies its user agent. If neither is published, the user agent is just a claim.
Not published
We could not find it in the operator’s own documentation. We do not fill gaps from third-party lists — those disagree with each other and go stale.
Facts were checked against the operators’ pages on the date shown on each crawler page. Operators change their docs; the source link is the place to confirm.
From a name in your logs to a rule
Four steps, all in the browser.
-
1
Find the crawler
Type the name from your logs or the company you are thinking of. Filter by purpose to see, say, every AI training crawler.
-
2
Open its page
Read what the operator says it is for and whether it follows robots.txt. Copy its official user agent to compare with your logs.
-
3
Build the rules
Tick crawlers in the table, choose Block or Allow, then copy or download the robots.txt groups.
-
4
Check the effect
Test the file with the robots.txt tester, and use the IP check on a crawler page to confirm a request is genuine.
How crawlers are grouped and how robots.txt applies
The directory groups crawlers the way TapCub’s bot rules do. robots.txt matching follows RFC 9309: a crawler obeys the one group whose User-agent matches its token, and only falls back to * when no group names it.
- A crawler with its own group ignores every
User-agent: *rule - Tokens are matched without regard to case; a typo matches nothing
- Some names are tokens only — Google-Extended and Applebot-Extended never visit
- Agents without a published token cannot be addressed in robots.txt at all
| Group | What it does | Effect of blocking |
|---|---|---|
| AI training | Collects pages for model training or datasets | Your future pages stay out of that operator’s training, if it follows robots.txt |
| AI search | Indexes pages for AI answers with links | You stop appearing as a cited source |
| AI agent | Opens a page for a person’s request | Assistants cannot open your page — when the agent honours robots.txt at all |
| Search engine | Classic search index | You leave that search engine; some, like Bing, also feed AI answers |
A policy many publishers start with
Keep content out of model training, stay visible in AI answers, let assistants open pages people ask about. The tokens below are all documented by their operators.
| Crawler | Rule | Why |
|---|---|---|
| GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent | Disallow: / | Their operators describe them as training crawlers or training controls. |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot | Allow: / or no group | They index for answers that can cite and link you. |
| ChatGPT-User, Claude-User, Perplexity-User | No group | They fetch because a person asked; OpenAI and Perplexity say robots.txt may not apply anyway. |
| Googlebot, Bingbot | Unchanged | Search engines. Bing says its Copilot search relies on the same crawling and index as Bing search. |
Tick these crawlers in the table above to generate the groups. Adjust to your own goals — this is a starting point, not a recommendation for every site.
Where AI crawler policies go wrong
Most problems come from the token or from what the crawler actually is.
Using a company name as the token
“OpenAI”, “Anthropic” or “ChatGPT” are not tokens. Use the exact token from the directory — GPTBot, ClaudeBot, OAI-SearchBot.
Blocking search with training
Blocking OAI-SearchBot or PerplexityBot removes you from AI answers that would link back. Decide per crawler, not per company.
Trusting the user agent
Anyone can send “GPTBot” in a header. Without an IP or reverse-DNS check the request is unverified.
Blocking by IP and losing the opt-out
Anthropic and Cohere warn that blocking their IPs stops them reading robots.txt, so the opt-out may not stick.
See which crawlers actually came
The directory tells you what crawlers say they do. TapCub separates humans from bots with five detection layers and 197+ rules, and reports AI crawlers on their own.
Bot and AI crawler report
User agent, headers, client signals, ASN and rate — five layers that keep bots out of your human numbers.
Learn moreAI content ledger
Pages each AI crawler read next to the visitors it sent back.
Learn moreCrawl health and compliance
Compare robots.txt and llms.txt with what actually crawled you — evidence for a decision.
Learn moreAI bots questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
What are GPTBot and ClaudeBot?
The training crawlers of OpenAI and Anthropic. Both operators say these bots collect public pages that may be used to train their models, and both honour robots.txt: User-agent: GPTBot or User-agent: ClaudeBot with Disallow: / opts you out. Their search crawlers are separate (OAI-SearchBot, Claude-SearchBot), so blocking training does not remove you from ChatGPT or Claude search. Details and sources: GPTBot, ClaudeBot.
Where does the crawler list come from?
From the same rules TapCub uses to detect bots: every AI crawler in those rules, plus search engine bots that have a robots.txt token — 74 in all. Crawler pages add facts from each operator’s own documentation.
Why do some crawlers have no page?
A page needs facts. We build one where the crawler is common enough to matter, and we only write what the operator itself publishes. Others are listed in the table with their token.
Why does a page say “not published”?
Because the operator’s own documentation does not say it. We do not fill gaps from third-party crawler lists, which often disagree and go out of date.
Does blocking AI crawlers hurt my Google ranking?
Google says blocking Google-Extended does not affect Search inclusion or ranking. Blocking other training crawlers does not touch Googlebot. Blocking a search engine bot, of course, removes you from that engine.
Can I verify that a request is really from GPTBot or ClaudeBot?
Yes, for crawlers whose operators publish IP lists or reverse-DNS names. Each crawler page has the official list link and an in-browser IP check; nothing you paste is uploaded.
What about agents with no robots.txt token?
Some browsing agents publish no token, so robots.txt cannot address them. They are listed with “No robots.txt token”; blocking them needs a server-side rule, and TapCub’s bot report still counts them.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Bot and AI crawler rules (same as the TapCub product) — checked 2026-11-10, reviewed every 3 months (next review by 2027-02-10)
- AI crawler operator documentation — checked 2026-11-09, reviewed every 3 months (next review by 2027-02-09)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Rules state intent. Logs show behaviour.
Install the ≈2 KB script and TapCub separates humans, search engines and AI crawlers from the first visit — free plan included.