Free tool Fetch as Googlebot and AI crawlers, next to a normal browser
Enter a URL and pick the crawlers. The page is fetched once as Chrome and once as each crawler, and the results sit side by side: status code, final address, size, title, amount of text, indexing rules and signs of a firewall challenge. The robots.txt column says whether each crawler is even allowed to fetch that URL.
- Search and AI crawlers
- robots.txt per crawler
- Sent only when you click
Fetch a URL as a crawler
Enter a page and choose a crawler or a group. Until you click Check, the result is a sample for bbc.com.
check.tapcub.com, which fetches it from Cloudflare’s network with the user-agent shown and a From: tools@tapcub.com header, then returns the status codes, response headers and at most 512 KB of the page. One check uses two to five requests: robots.txt, the page as Chrome and the page as each crawler. These requests come from Cloudflare’s IP addresses, not from Google’s, OpenAI’s or Anthropic’s, so a site that verifies crawlers by IP will see a fake crawler, and may block it. The service records only the host name, status and timing for rate limiting, not the path, query or page content, and stores no results. Limits: 10 requests a minute and 200 a day per IP address.A page on the site. robots.txt is fetched from the same host.
Each crawler is one extra request (limit: 10 a minute).
Differences and checks
Compared with the Chrome row.
- OKEvery crawler got the same status, address, title and roughly the same amount of text as Chrome.
- WarningGPTBot is disallowed for this URL by “Disallow: /” (robots.txt line 154), yet the server served the page anyway. robots.txt is a request, not a lock: a compliant GPTBot would not fetch it.
- WarningClaudeBot is disallowed for this URL by “Disallow: /” (robots.txt line 133), yet the server served the page anyway. robots.txt is a request, not a lock: a compliant ClaudeBot would not fetch it.
- WarningPerplexityBot is disallowed for this URL by “Disallow: /” (robots.txt line 174), yet the server served the page anyway. robots.txt is a request, not a lock: a compliant PerplexityBot would not fetch it.
- NoteThe page is larger than 512 KB; titles and word counts come from the first 512 KB only.
Side by side
Hover the robots.txt result to see the deciding rule.
| Fetched as | robots.txt | Status | Final URL | Size | Title | Words | Index rules | Firewall |
|---|---|---|---|---|---|---|---|---|
| Chrome (desktop) | — | 200 | https://www.bbc.com/ | 695 kB | BBC Home | Breaking News, World News, US News, Sports, Busin… | 2,747 | noodp, noydir | None seen |
| GPTBot | Disallowed | 200 | https://www.bbc.com/ | 695 kB | BBC Home | Breaking News, World News, US News, Sports, Busin… | 2,747 | noodp, noydir | None seen |
| ClaudeBot | Disallowed | 200 | https://www.bbc.com/ | 695 kB | BBC Home | Breaking News, World News, US News, Sports, Busin… | 2,747 | noodp, noydir | None seen |
| PerplexityBot | Disallowed | 200 | https://www.bbc.com/ | 695 kB | BBC Home | Breaking News, World News, US News, Sports, Busin… | 2,750 | noodp, noydir | None seen |
Share these inputs: the values go after the # in the link, so they never reach a server.
https://tapcub.com/tools/fetch-as-bot
Version v1.0Updated Update history
What a difference means
The Chrome row is the baseline. Every difference in a crawler row has one of a few causes.
Different status (403, 429, 503)
Usually a firewall or bot-management rule. Some sites block AI crawlers on purpose; others block them by accident with a “block bots” switch, along with search engines they wanted to keep.
Challenge page
A “Just a moment…” page or another CAPTCHA means the crawler gets no content at all. Search engines’ verified crawlers are normally let through; our requests are not verified, so a challenge here is a hint, not proof.
Less text or a different title
The server sends crawlers a lighter or different page. A lighter page is fine if the main content is still there; a different page for crawlers than for people is cloaking, which Google’s spam policies forbid.
robots.txt disallowed
The site asks this crawler not to fetch the URL. Compliant crawlers honour it, but nothing stops the server from answering, so “disallowed” and “200” together are normal.
Why the IP address matters: Google, Bing, OpenAI, Anthropic and others publish how to verify their crawlers (reverse DNS or IP lists). Our requests carry their user-agent but come from Cloudflare, so a site that verifies crawlers treats them as impostors. If a crawler is blocked here, check your firewall logs for the real crawler before changing anything.
Three steps
Nothing is requested until you click Check.
-
1
Enter a page
Start with a page you want found in search or AI answers: a product, an article, a pricing page.
-
2
Pick crawlers
AI crawlers to see whether ChatGPT, Claude and Perplexity can read you; search engines to check Google, Bing and Baidu.
-
3
Compare and decide
Fix unintended blocks in your firewall or CDN, or keep them and make robots.txt say the same thing so compliant crawlers stop asking.
How the comparison works
All requests go through check.tapcub.com from Cloudflare’s network, with From: tools@tapcub.com:
/robots.txtis fetched once (as Chrome) and evaluated per crawler with RFC 9309 rules: the crawler’s own group, else*; longest match wins; 4xx means no rules, 5xx or no answer means disallow everything- The page is fetched as Chrome on desktop, then once per crawler with that crawler’s published user-agent string; redirects are followed (up to 5)
- From the first 512 KB of HTML: the title, visible text (scripts and styles removed; CJK characters counted one by one), robots meta tags; plus X-Robots-Tag from the headers
- Firewall signs: Cloudflare’s
cf-mitigated: challengeheader and challenge page, Akamai, Imperva, Sucuri, DataDome, HUMAN and AWS WAF markers. A text difference of 30% or more is flagged
| Crawler | Operator | robots.txt token | Used for |
|---|---|---|---|
| Googlebot | Googlebot | Google Search | |
| Bingbot | Microsoft | Bingbot | Bing Search |
| Baiduspider | Baidu | Baiduspider | Baidu Search |
| GPTBot | OpenAI | GPTBot | Model training |
| OAI-SearchBot | OpenAI | OAI-SearchBot | ChatGPT search results |
| ChatGPT-User | OpenAI | ChatGPT-User | Fetches a user asked for |
| ClaudeBot | Anthropic | ClaudeBot | Model training |
| PerplexityBot | Perplexity | PerplexityBot | Perplexity search index |
bbc.com: disallowed in robots.txt, served anyway
The sample on this page was fetched on 2026-11-17.
| Fetched as | robots.txt | Status | Words | Reading |
|---|---|---|---|---|
| Chrome | — | 200 | 2,747 | Baseline |
| GPTBot | Disallowed (Disallow: /) | 200 | 2,747 | Same page; only robots.txt asks GPTBot to stay away |
| ClaudeBot | Disallowed (Disallow: /) | 200 | 2,747 | Same |
| PerplexityBot | Disallowed (Disallow: /) | 200 | 2,750 | Same (3 words of rotating content) |
In this sample the server did not block the crawler user-agents; the restriction lives in robots.txt. That is a common choice: compliant crawlers stop at robots.txt, and blocking by user-agent alone would also catch anyone testing with a tool like this one.
Getting crawler access wrong
Most problems come from treating robots.txt, firewall rules and indexing tags as one thing.
A “block bots” switch
One-click bot blocking in a CDN or security plugin can also stop search engines or AI search crawlers you want. Test after turning it on.
Expecting robots.txt to block
It is a request. To stop a crawler that ignores it, block it at the firewall, ideally by verified IP or DNS, not by user-agent alone.
Disallow plus noindex
A crawler that may not fetch the page never sees its noindex tag. To remove a page from search, allow crawling and use noindex.
Trusting the user-agent
Anyone can claim to be Googlebot, as this tool does. Verify by reverse DNS or published IP ranges before giving a crawler special treatment.
See which crawlers really visit
This tool shows what crawlers would get. TapCub shows which ones actually came, how often and to which pages.
Bots counted separately
Five layers and 197+ rules keep crawlers out of the denominator by default.
Learn moreCrawl compliance
Your robots.txt and llms.txt are fetched daily and compared with what actually crawled you.
Learn moreVerified crawlers
Crawlers sent through the server-side API or log import are checked with reverse and forward DNS and marked verified or suspected.
Learn moreFetch as Googlebot questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
What happened to “Fetch as Google” in Search Console?
Google retired it with the old Search Console in 2019. The URL Inspection tool replaced it: it fetches your page with the real Googlebot from Google’s IP addresses and shows the rendered HTML. Use it for the definitive Google view of sites you own; use this tool to compare several crawlers with a browser on any site.
Is this the same as a real Googlebot visit?
No. The user-agent is Googlebot’s, but the request comes from Cloudflare’s network and does not run JavaScript. Sites that verify Googlebot by IP will treat it as fake, and content added by scripts is not counted.
How do I check if my site blocks AI crawlers?
Pick the AI crawler group and check a few important pages. The robots.txt column shows what you ask; the status and firewall columns show what your server actually does.
Why is GPTBot blocked when my robots.txt allows it?
Something other than robots.txt is blocking it: a firewall rule, a CDN bot setting or a security plugin. It may also be blocking our request only because it does not come from OpenAI’s published IP addresses.
Does it render JavaScript?
No. It reads the HTML the server sends, which is what most AI crawlers see too. Googlebot renders pages later, so content that only appears after scripts run may still be indexed by Google.
Which user-agent strings are used?
The formats each operator publishes, for example Mozilla/5.0 … compatible; GPTBot/1.1; +https://openai.com/gptbot. The request also carries From: tools@tapcub.com, so it is not hidden.
How do I verify a real crawler in my logs?
Do a reverse DNS lookup on the IP and a forward lookup on the result (Google, Bing), or compare the IP with the operator’s published list (OpenAI, Anthropic, Perplexity). The fake bot checker does this for you.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Sample fetches (github.com, apple.com, wikipedia.org, example.com, bbc.com) — checked 2026-11-17, reviewed every 6 months (next review by 2027-05-17)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Check a crawler’s IP address
Seen “Googlebot” or “GPTBot” in your logs? Find out whether it really is.