Free tool robots.txt checker and tester for search and AI crawlers
Paste a robots.txt, list the URLs you care about and pick the crawlers. For every URL and crawler you get allowed or blocked, the line that decided it and the group the crawler actually reads, plus a list of lines that crawlers will misread or ignore.
- RFC 9309 matching
- 146 crawler tokens
- Runs locally
Which crawler may fetch which URL
The sample is a small shop. Paste your own file from https://your-site.com/robots.txt and your own paths; the table and the issue list update as you type. Nothing is fetched from your site.
Paste the whole file. Comments and blank lines are fine.
One per line, up to 50. Full URLs work too; only the path and query are compared.
Comma-separated, such as Applebot, YandexBot. Use the token, not the full user-agent string.
User-agent groups
3
Groups found in the file.
Checks
64
URLs × crawlers.
Allowed
28
Blocked
36
Result per URL and crawler
A crawler reads only the group that names it; User-agent: * applies only when no group names it.
| Path | Crawler | Result | Deciding rule | Group used |
|---|---|---|---|---|
/ | Googlebot | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | Bingbot | Allowed | No rule matches: allowed | Own group (line 14) |
/ | Baiduspider | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | GPTBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/ | OAI-SearchBot | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | ChatGPT-User | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | ClaudeBot | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | Claude-SearchBot | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | Claude-User | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | PerplexityBot | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | Perplexity-User | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | Google-Extended | Blocked | Disallow: / (line 12) | Own group (line 9) |
/ | Applebot-Extended | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | CCBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/ | Bytespider | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/ | Meta-ExternalAgent | Allowed | No rule matches: allowed | User-agent: * (line 2) |
/products/red-sneakers?sort=price | Googlebot | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | Bingbot | Allowed | No rule matches: allowed | Own group (line 14) |
/products/red-sneakers?sort=price | Baiduspider | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | GPTBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/products/red-sneakers?sort=price | OAI-SearchBot | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | ChatGPT-User | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | ClaudeBot | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | Claude-SearchBot | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | Claude-User | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | PerplexityBot | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | Perplexity-User | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | Google-Extended | Blocked | Disallow: / (line 12) | Own group (line 9) |
/products/red-sneakers?sort=price | Applebot-Extended | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | CCBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/products/red-sneakers?sort=price | Bytespider | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/products/red-sneakers?sort=price | Meta-ExternalAgent | Blocked | Disallow: /*?sort= (line 6) | User-agent: * (line 2) |
/checkout/help | Googlebot | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | Bingbot | Allowed | No rule matches: allowed | Own group (line 14) |
/checkout/help | Baiduspider | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | GPTBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/checkout/help | OAI-SearchBot | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | ChatGPT-User | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | ClaudeBot | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | Claude-SearchBot | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | Claude-User | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | PerplexityBot | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | Perplexity-User | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | Google-Extended | Blocked | Disallow: / (line 12) | Own group (line 9) |
/checkout/help | Applebot-Extended | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | CCBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/checkout/help | Bytespider | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/checkout/help | Meta-ExternalAgent | Allowed | Allow: /checkout/help (line 3) | User-agent: * (line 2) |
/cart | Googlebot | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | Bingbot | Allowed | No rule matches: allowed | Own group (line 14) |
/cart | Baiduspider | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | GPTBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/cart | OAI-SearchBot | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | ChatGPT-User | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | ClaudeBot | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | Claude-SearchBot | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | Claude-User | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | PerplexityBot | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | Perplexity-User | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | Google-Extended | Blocked | Disallow: / (line 12) | Own group (line 9) |
/cart | Applebot-Extended | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | CCBot | Blocked | Disallow: / (line 12) | Own group (line 9) |
/cart | Bytespider | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
/cart | Meta-ExternalAgent | Blocked | Disallow: /cart (line 5) | User-agent: * (line 2) |
Issues in the file
Problems are checked against RFC 9309 and the major search engines’ documentation.
- NoteLine 15Crawl-delay is not part of RFC 9309. Google ignores it, Yandex has ignored it since 2018 and Baidu does not document it; Bing and Anthropic’s ClaudeBot say they honour it.
- OKNo problems found. Every line is one crawlers understand.
Version v1.0Updated Update history
What each column tells you
Every row is one URL seen by one crawler. Read the result first, then the two columns that explain it.
Allowed or blocked
Blocked means the crawler is asked not to fetch that URL. It is a request, not a lock: compliant crawlers honour it, others do not, and a blocked URL can still appear in Google results without a snippet if other pages link to it.
Deciding rule
The single Allow or Disallow line that won, with its line number. When several rules match, the longest one wins and Allow beats Disallow on a tie. “No rule matches” means the crawler has a group but nothing in it covers the URL, so it is allowed.
Group used
This is where most surprises come from. A crawler with its own group ignores User-agent: * completely. In the sample, Bingbot has a group for /search, so it may fetch /cart and the sorted product URL even though * blocks them for everyone else.
Issues
Errors break rules or the sitemap; warnings are lines some crawlers misread or skip; notes explain non-standard lines such as Crawl-delay. A clean file shows a single OK.
The tester reads the text you paste. Your live file can differ: check that https://your-site.com/robots.txt returns status 200 and plain text, on every host name (www, shop, m) and on both http and https.
Three steps
Nothing is installed and nothing is fetched.
-
1
Paste the file
Open your robots.txt in a browser, select all and paste. To test a change before you publish it, paste the edited version instead.
-
2
List the URLs
Add the pages you must keep crawlable (home, category, product, article) and the ones you meant to block (cart, search, admin). Full URLs are fine.
-
3
Pick crawlers and read
The default set covers the main AI crawlers plus Google, Bing and Baidu. Switch to all AI crawlers or every known token, or add your own. Fix anything in the issue list, then retest.
How a crawler decides, per RFC 9309
RFC 9309 (September 2022) standardised the Robots Exclusion Protocol that Google, Bing and most well-known crawlers follow. The tester applies it exactly:
- Group: the crawler matches its product token against User-agent lines, case-insensitively and as a whole token. Several groups for the same token are merged. Only if none names it does
*apply; with neither, everything is allowed - Rule: among the matching Allow and Disallow lines, the longest path wins; on a tie Allow wins; with no match the URL is allowed
- Path: case-sensitive, compared from the start;
*matches any characters, a final$anchors the end; percent-encoding is normalised first /robots.txtitself is always allowed. Sitemap, Crawl-delay and other lines never split or change a group
| Rules in the group | URL path | Result |
|---|---|---|
Disallow: /fish | /fish.html | Blocked: prefix match |
Disallow: /fish/ | /fish | Allowed: no match |
Disallow: /Fish | /fish | Allowed: paths are case-sensitive |
Disallow: /*.php$ | /index.php?lang=en | Allowed: $ needs the URL to end there |
Disallow: / + Allow: /$ | / | Allowed: /$ is longer |
Disallow: / + Allow: /p | /page | Allowed: /p is longer |
Disallow: /folder + Allow: /folder | /folder/page | Allowed: tie goes to Allow |
Disallow: /%62az | /baz | Blocked: %62 decodes to b |
Four crawlers read the sample file
The sample has three groups. Here is how four crawlers from the default set treat the same URLs, and why.
| Crawler | Group it reads | /products/red-sneakers?sort=price | /checkout/help | /cart |
|---|---|---|---|---|
| Googlebot | * (no own group) | Blocked by /*?sort= | Allowed: Allow is longer than /checkout/ | Blocked |
| Bingbot | Own group: only /search | Allowed | Allowed | Allowed |
| GPTBot | Own group with CCBot and Google-Extended | Blocked by Disallow: / | Blocked | Blocked |
| OAI-SearchBot | * (no own group) | Blocked | Allowed | Blocked |
Bingbot is the trap: giving it a group just to add Crawl-delay dropped every * rule for it. To keep them, repeat the * rules inside the Bingbot group. OAI-SearchBot (ChatGPT search) is not blocked like GPTBot (training), because each OpenAI crawler has its own token.
Why robots.txt does not do what you meant
Most broken files are valid syntax. The logic is what goes wrong.
Expecting groups to add up
A named group replaces * for that crawler; it does not add to it. Repeat shared rules in every named group.
Blocking to deindex
Disallow stops crawling, not indexing. A linked URL can still be listed. To remove a page, allow crawling and add noindex.
Blocking CSS and JavaScript
Rules like Disallow: /assets/ stop search engines from rendering the page, so they may see it as broken or empty.
Wrong host or location
Each host needs its own file at its root: shop.example.com does not read the file on www.example.com, and /blog/robots.txt is never read.
See which crawlers actually obey it
robots.txt states what you want. TapCub’s bot report shows what crawlers did: it separates search engines, AI crawlers and scrapers from human visitors and keeps them out of your human numbers.
Bots and AI crawlers
Five detection layers and 197+ built-in rules split known crawlers from people, with false positives you can mark as human.
Learn moreCrawl compliance
A daily fetch of your robots.txt and llms.txt is compared with what actually crawled you, as evidence for blocking.
Learn moreSearch performance
Google Search Console and Bing Webmaster data next to your pages, to see whether a change hurt clicks.
Learn morerobots.txt tester questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
How do I test whether a URL is blocked by robots.txt?
Paste the file and the URL above. The row for each crawler says allowed or blocked and names the deciding line. For Google’s own view of your live file, Search Console has a robots.txt report (it replaced the old robots.txt Tester at the end of 2023), and URL Inspection shows whether a specific URL is blocked.
Why does Googlebot ignore my User-agent: * rules?
It does not, unless the file also has a group for Googlebot. A crawler that finds a group with its own token reads only that group. Either remove the named group or copy the * rules into it.
Does the order of Allow and Disallow lines matter?
Not under RFC 9309 or for Google and Bing: the longest matching path wins, and Allow wins a tie. Baidu’s documentation, however, says the first matching line applies. Putting Allow lines before Disallow lines gives the same answer under both readings.
Are robots.txt paths case-sensitive?
Yes. Disallow: /Admin does not block /admin. Field names (Disallow, User-agent) and user-agent tokens are case-insensitive; paths are not.
Does blocking a page in robots.txt remove it from Google?
No. It stops crawling. If other pages link to the URL, Google can still list it without a description. To remove a page, let it be crawled and add a noindex robots meta tag or X-Robots-Tag header.
Do AI crawlers follow robots.txt?
The large operators say their automatic crawlers do: OpenAI (GPTBot, OAI-SearchBot), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Google (Google-Extended), Perplexity (PerplexityBot). Fetches a person triggers are different: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it.
What happens if robots.txt returns an error?
Under RFC 9309, a 4xx response (such as 404) means crawlers may fetch everything, while a 5xx response or a network failure means they should treat the whole site as disallowed until the file is reachable again. Crawlers should follow at least five redirects to reach it.
Is my robots.txt uploaded?
No. The file and the URLs are parsed in your browser and nothing is sent to TapCub or to your site. This tool has no share link, so pasted content never ends up in a URL.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Bot and AI crawler rules (same as the TapCub product) — checked 2026-11-10, reviewed every 3 months (next review by 2027-02-10)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Find out which crawlers visit you
Add the TapCub snippet and the bot report shows every search engine and AI crawler that loads your pages, separately from human visitors.