Free tool AI crawler robots.txt checker
Paste the robots.txt from your site and see, crawler by crawler, what it allows. Then tick the crawlers you want to block and copy the rules. Parsed in your browser — no request is made to your site.
- 43 AI crawler user agents
- Shows the rule that decides
- Runs locally
Paste robots.txt, read the verdict
Open https://your-site.com/robots.txt, copy everything, paste it below. The table updates as you type. The default text is an example file you can replace.
Example file preloaded — replace it with your own. Comments and Sitemap lines are ignored.
Allowed
0
Partly restricted
40
Blocked
3
Not declared
0
Crawler by crawler
A crawler follows the group written for it; if there is none, it follows User-agent: *; if there is neither, it may crawl everything.
| Crawler | Purpose | Deciding rule | Status |
|---|---|---|---|
| GPTBotOpenAI | Model training | Disallow: /own User-agent group | Blocked |
| ChatGPT-UserOpenAI | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| OAI-SearchBotOpenAI | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| ClaudeBotAnthropic | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| Claude-WebAnthropic | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| Claude-UserAnthropic | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| Claude-SearchBotAnthropic | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| anthropic-aiAnthropic | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| PerplexityBotPerplexity | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| Perplexity-UserPerplexity | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| Google-ExtendedGoogle | Model training | Disallow: /own User-agent group | Blocked |
| GoogleOtherGoogle | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| Applebot-ExtendedApple | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| BytespiderByteDance | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| CCBotCommon Crawl | Model training | Disallow: /own User-agent group | Blocked |
| Meta-ExternalAgentMeta | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| Meta-ExternalFetcherMeta | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| FacebookBotMeta | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| AmazonbotAmazon | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| cohere-aiCohere | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| DiffbotDiffbot | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| OmgiliWebz.io | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| YouBotYou.com | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| DuckAssistBotDuckDuckGo | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| MistralAI-UserMistral AI | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| TimpibotTimpi | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| PetalBotHuawei | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| AI2BotAllen Institute | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| img2dataset— | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| PanguBotHuawei | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| VelenPublicWebCrawlerVelen | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| WebzioWebz.io | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| ImagesiftBotHive | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| DeepSeekBotDeepSeek | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| DoubaoByteDance | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| FirecrawlAgentFirecrawl | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| NovaActAmazon | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| LinerBotLiner | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| AndibotAndi | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| PhindBotPhind | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| SemrushBot-OCOBSemrush | Model training | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| QuillBotQuillBot | Fetches on a user’s request | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
| iAskBotiAsk | AI search index | Disallow: /admin/ Disallow: /cartinherits User-agent: * | Partly restricted |
Generate rules
Tick the crawlers you want to keep out. Paste the result at the end of your robots.txt — one group per crawler, nothing else changes.
# AI crawlers — generated with tapcub.com/tools/ai-crawler
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /robots.txt is a request, not a lock. Well-known crawlers honor it; the ones that do not are what the bot report is for.
Version v2.0Updated Update history
NewChecks every AI crawler in the TapCub bot rules (up from 11), and links each one to its own crawler page.
What each status actually means
The four counts at the top summarize the table; the status on each row is a reading of your file, not an enforcement. Two of the four states are easy to misread.
Allowed vs Not declared
"Allowed" means a rule explicitly lets the crawler in; "Not declared" means no rule mentions it at all. The effect is the same — it may crawl — but "Not declared" means a future wildcard rule would start applying to it.
Partly restricted
The crawler is kept out of some paths but not the site. Usually it inherits User-agent: * and only your /admin/ or /cart rules touch it. For most AI crawlers that still means your content is readable.
Blocked
A rule asks the crawler to stay off the whole site. It is a request, not a lock — the deciding-rule column shows which line produced it, so you can confirm it is the one you meant.
The deciding rule
Every verdict names the exact line behind it and whether it came from the crawler’s own group or the wildcard. If a status surprises you, read that line first — it is almost always a token typo or an inherited *.
robots.txt states intent. Whether a crawler obeyed it is a separate question that only a bot report can answer.
The four rules the checker applies
The same logic well-behaved crawlers use, simplified to the cases that matter for AI bots.
-
1
Find the group for the crawler
A group starts with one or more User-agent lines. The crawler matches a group whose token matches its name, case-insensitively. Only that group applies to it.
-
2
Fall back to the wildcard
No group of its own? The crawler follows User-agent: *. The table shows "inherits *" so you know the verdict is indirect.
-
3
Read Disallow and Allow
Disallow: / blocks everything; an empty Disallow allows everything; other paths mean partly restricted. Allow: / overrides a root block.
-
4
No group at all means allowed
A robots.txt that never mentions a crawler and has no * group places no restriction on it. "Not declared" is effectively "allowed".
Not every AI crawler does the same thing
Blocking all of them is a choice, not a default. Training crawlers copy content for model training; search crawlers index it for AI answers that can link back; assistant fetchers load a page because a person asked.
- Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider, CCBot, meta-externalagent; Amazon says Amazonbot may also be used to train its models
- AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot — blocking these removes you from those AI answers
- Assistant fetch: ChatGPT-User, Claude-User, Perplexity-User — triggered by a person, and OpenAI and Perplexity say robots.txt may not apply to theirs
- Policies change; check the vendor documentation for the current token names
What a sensible policy looks like
Block training crawlers if you do not want your content in the next model; keep search and assistant agents if you want to be cited. The snippet generator defaults to that split.
- GPTBotBlocked
- ClaudeBotBlocked
- OAI-SearchBotAllowed
- PerplexityBotAllowed
- ChatGPT-UserAllowed
What the bot report adds
robots.txt tells crawlers what you want. The bot report in TapCub shows what they did: requests per crawler per day, which pages, and whether a blocked one kept coming.
AI crawler requests · 7 days
What the default file says
The example robots.txt blocks three training crawlers by name and gives everyone else the wildcard rules, which only restrict /admin/ and /cart.
User-agent: *
Disallow: /admin/
Disallow: /cart
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /GPTBot, Google-Extended, CCBot → Blocked
Each has its own group with Disallow: /. Nothing in the wildcard group applies to them.
ClaudeBot, PerplexityBot, Bytespider… → Partly restricted
No group of their own, so they inherit User-agent: *, which only keeps them out of /admin/ and /cart. Everything else is open.
If you removed the * group → Not declared
With no wildcard group, crawlers that are not named face no restriction at all. That is the state most sites are in.
Change the text in the checker above and watch the statuses follow.
Why a robots.txt does not do what you meant
The file is simple; the matching rules are not obvious.
Expecting * rules to add to a named group
They do not. A crawler with its own group ignores the wildcard entirely — including your Disallow: /admin/.
Wrong token
"ChatGPT" or "OpenAI" are not tokens. The crawler reads GPTBot, OAI-SearchBot or ChatGPT-User. Typos silently match nothing.
Treating robots.txt as enforcement
It is a request. Compliant crawlers honor it; others ignore it. Watch the bot report to see who actually complies.
Blocking search crawlers by accident
Disallowing OAI-SearchBot or PerplexityBot removes you from AI answers that would have linked to you. Decide per crawler.
See what crawlers actually did
TapCub separates humans from bots with five detection layers and 197+ rules, and reports AI crawlers as their own category — per crawler, per page, per day.
Bot and AI crawler report
Requests by crawler, which pages they fetched, and whether a blocked crawler kept coming.
Learn moreAI search as a traffic source
Visits referred by AI assistants, counted as a channel next to search and social.
Learn moreAn AI crawler ledger
A month of AI crawler traffic on a typical site, and what a sensible policy looks like.
Learn moreAI crawler questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
How do I block AI crawlers in robots.txt?
Add one group per crawler token with Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Tick the crawlers under “Generate rules” above and paste the snippet at the end of your file, then paste the whole file back into the checker: each one you named should show Blocked, with its own group as the deciding rule.
Does the checker fetch my site’s robots.txt?
No. It only parses the text you paste, in your browser. Nothing is requested from your domain and nothing is sent to TapCub.
Will blocking AI crawlers hurt my search ranking?
Blocking training crawlers such as GPTBot or Google-Extended does not affect regular search indexing. Blocking AI search crawlers removes you from those AI answers specifically.
Where does the crawler list come from?
It is exported from the same rules TapCub uses to detect bots: every AI crawler in those rules that has a robots.txt token, 43 in all. Agents that browse on a person’s behalf without a published token cannot be addressed in robots.txt, so they are not listed here; the bot report still counts them.
What if a crawler ignores robots.txt?
Then robots.txt cannot help, and you need to see the traffic. TapCub’s bot report lists requests per crawler so you can decide on a server-side block. See bot detection.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
OpenAI documents three separate agents. GPTBot collects pages that may be used to train models. OAI-SearchBot indexes pages for search in ChatGPT. ChatGPT-User fetches a page when a person asks ChatGPT about it. GPTBot and OAI-SearchBot each follow their own User-agent group, so you can block training and still appear in ChatGPT search. For ChatGPT-User, OpenAI says robots.txt rules may not apply, because a person asked for the page.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v2.0
New featureChecks every AI crawler in the TapCub bot rules (up from 11), and links each one to its own crawler page.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Bot and AI crawler rules (same as the TapCub product) — checked 2026-11-10, reviewed every 3 months (next review by 2027-02-10)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Know who is crawling, not just who you asked not to
Install the ≈2 KB script and the bot report starts separating humans, search engines and AI crawlers from day one — free plan included.