TapCub is live — analytics, insights and live chat, free on one platform

Free tool Fetch as Googlebot and AI crawlers, next to a normal browser

Enter a URL and pick the crawlers. The page is fetched once as Chrome and once as each crawler, and the results sit side by side: status code, final address, size, title, amount of text, indexing rules and signs of a firewall challenge. The robots.txt column says whether each crawler is even allowed to fetch that URL.

  • Search and AI crawlers
  • robots.txt per crawler
  • Sent only when you click

Fetch a URL as a crawler

Enter a page and choose a crawler or a group. Until you click Check, the result is a sample for bbc.com.

Compare
What is sent, and to whom. Nothing is requested when the page opens: the result below is a dated sample. When you click Check, the address goes to TapCub’s checking service check.tapcub.com, which fetches it from Cloudflare’s network with the user-agent shown and a From: tools@tapcub.com header, then returns the status codes, response headers and at most 512 KB of the page. One check uses two to five requests: robots.txt, the page as Chrome and the page as each crawler. These requests come from Cloudflare’s IP addresses, not from Google’s, OpenAI’s or Anthropic’s, so a site that verifies crawlers by IP will see a fake crawler, and may block it. The service records only the host name, status and timing for rate limiting, not the path, query or page content, and stores no results. Limits: 10 requests a minute and 200 a day per IP address.

A page on the site. robots.txt is fetched from the same host.

Each crawler is one extra request (limit: 10 a minute).

Sample result for https://www.bbc.com/, fetched on 2026-11-17. Click Check to test it live, or enter your own address.

Differences and checks

Compared with the Chrome row.

  • OKEvery crawler got the same status, address, title and roughly the same amount of text as Chrome.
  • WarningGPTBot is disallowed for this URL by “Disallow: /” (robots.txt line 154), yet the server served the page anyway. robots.txt is a request, not a lock: a compliant GPTBot would not fetch it.
  • WarningClaudeBot is disallowed for this URL by “Disallow: /” (robots.txt line 133), yet the server served the page anyway. robots.txt is a request, not a lock: a compliant ClaudeBot would not fetch it.
  • WarningPerplexityBot is disallowed for this URL by “Disallow: /” (robots.txt line 174), yet the server served the page anyway. robots.txt is a request, not a lock: a compliant PerplexityBot would not fetch it.
  • NoteThe page is larger than 512 KB; titles and word counts come from the first 512 KB only.

Side by side

Hover the robots.txt result to see the deciding rule.

Fetched asrobots.txtStatusFinal URLSizeTitleWordsIndex rulesFirewall
Chrome (desktop)—200https://www.bbc.com/695 kBBBC Home | Breaking News, World News, US News, Sports, Busin…2,747noodp, noydirNone seen
GPTBotDisallowed200https://www.bbc.com/695 kBBBC Home | Breaking News, World News, US News, Sports, Busin…2,747noodp, noydirNone seen
ClaudeBotDisallowed200https://www.bbc.com/695 kBBBC Home | Breaking News, World News, US News, Sports, Busin…2,747noodp, noydirNone seen
PerplexityBotDisallowed200https://www.bbc.com/695 kBBBC Home | Breaking News, World News, US News, Sports, Busin…2,750noodp, noydirNone seen
The checks, tables and explanations are worked out in your browser from what the service returns. robots.txt is parsed in your browser with the same RFC 9309 matcher as the robots.txt tester. Nothing you check is saved to an account or shared.

Share these inputs: the values go after the # in the link, so they never reach a server.

Version v1.0Updated Update history

Read the result

What a difference means

The Chrome row is the baseline. Every difference in a crawler row has one of a few causes.

Different status (403, 429, 503)

Usually a firewall or bot-management rule. Some sites block AI crawlers on purpose; others block them by accident with a “block bots” switch, along with search engines they wanted to keep.

Challenge page

A “Just a moment…” page or another CAPTCHA means the crawler gets no content at all. Search engines’ verified crawlers are normally let through; our requests are not verified, so a challenge here is a hint, not proof.

Less text or a different title

The server sends crawlers a lighter or different page. A lighter page is fine if the main content is still there; a different page for crawlers than for people is cloaking, which Google’s spam policies forbid.

robots.txt disallowed

The site asks this crawler not to fetch the URL. Compliant crawlers honour it, but nothing stops the server from answering, so “disallowed” and “200” together are normal.

Why the IP address matters: Google, Bing, OpenAI, Anthropic and others publish how to verify their crawlers (reverse DNS or IP lists). Our requests carry their user-agent but come from Cloudflare, so a site that verifies crawlers treats them as impostors. If a crawler is blocked here, check your firewall logs for the real crawler before changing anything.

How to use it

Three steps

Nothing is requested until you click Check.

  1. 1

    Enter a page

    Start with a page you want found in search or AI answers: a product, an article, a pricing page.

  2. 2

    Pick crawlers

    AI crawlers to see whether ChatGPT, Claude and Perplexity can read you; search engines to check Google, Bing and Baidu.

  3. 3

    Compare and decide

    Fix unintended blocks in your firewall or CDN, or keep them and make robots.txt say the same thing so compliant crawlers stop asking.

Method

How the comparison works

All requests go through check.tapcub.com from Cloudflare’s network, with From: tools@tapcub.com:

  • /robots.txt is fetched once (as Chrome) and evaluated per crawler with RFC 9309 rules: the crawler’s own group, else *; longest match wins; 4xx means no rules, 5xx or no answer means disallow everything
  • The page is fetched as Chrome on desktop, then once per crawler with that crawler’s published user-agent string; redirects are followed (up to 5)
  • From the first 512 KB of HTML: the title, visible text (scripts and styles removed; CJK characters counted one by one), robots meta tags; plus X-Robots-Tag from the headers
  • Firewall signs: Cloudflare’s cf-mitigated: challenge header and challenge page, Akamai, Imperva, Sucuri, DataDome, HUMAN and AWS WAF markers. A text difference of 30% or more is flagged
Crawler tokens used for robots.txt
CrawlerOperatorrobots.txt tokenUsed for
GooglebotGoogleGooglebotGoogle Search
BingbotMicrosoftBingbotBing Search
BaiduspiderBaiduBaiduspiderBaidu Search
GPTBotOpenAIGPTBotModel training
OAI-SearchBotOpenAIOAI-SearchBotChatGPT search results
ChatGPT-UserOpenAIChatGPT-UserFetches a user asked for
ClaudeBotAnthropicClaudeBotModel training
PerplexityBotPerplexityPerplexityBotPerplexity search index
Worked example

bbc.com: disallowed in robots.txt, served anyway

The sample on this page was fetched on 2026-11-17.

https://www.bbc.com/ as Chrome and three AI crawlers
Fetched asrobots.txtStatusWordsReading
Chrome—2002,747Baseline
GPTBotDisallowed (Disallow: /)2002,747Same page; only robots.txt asks GPTBot to stay away
ClaudeBotDisallowed (Disallow: /)2002,747Same
PerplexityBotDisallowed (Disallow: /)2002,750Same (3 words of rotating content)

In this sample the server did not block the crawler user-agents; the restriction lives in robots.txt. That is a common choice: compliant crawlers stop at robots.txt, and blocking by user-agent alone would also catch anyone testing with a tool like this one.

Common mistakes

Getting crawler access wrong

Most problems come from treating robots.txt, firewall rules and indexing tags as one thing.

A “block bots” switch

One-click bot blocking in a CDN or security plugin can also stop search engines or AI search crawlers you want. Test after turning it on.

Expecting robots.txt to block

It is a request. To stop a crawler that ignores it, block it at the firewall, ideally by verified IP or DNS, not by user-agent alone.

Disallow plus noindex

A crawler that may not fetch the page never sees its noindex tag. To remove a page from search, allow crawling and use noindex.

Trusting the user-agent

Anyone can claim to be Googlebot, as this tool does. Verify by reverse DNS or published IP ranges before giving a crawler special treatment.

FAQ

Fetch as Googlebot questions

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What happened to “Fetch as Google” in Search Console?

Google retired it with the old Search Console in 2019. The URL Inspection tool replaced it: it fetches your page with the real Googlebot from Google’s IP addresses and shows the rendered HTML. Use it for the definitive Google view of sites you own; use this tool to compare several crawlers with a browser on any site.

Is this the same as a real Googlebot visit?

No. The user-agent is Googlebot’s, but the request comes from Cloudflare’s network and does not run JavaScript. Sites that verify Googlebot by IP will treat it as fake, and content added by scripts is not counted.

How do I check if my site blocks AI crawlers?

Pick the AI crawler group and check a few important pages. The robots.txt column shows what you ask; the status and firewall columns show what your server actually does.

Why is GPTBot blocked when my robots.txt allows it?

Something other than robots.txt is blocking it: a firewall rule, a CDN bot setting or a security plugin. It may also be blocking our request only because it does not come from OpenAI’s published IP addresses.

Does it render JavaScript?

No. It reads the HTML the server sends, which is what most AI crawlers see too. Googlebot renders pages later, so content that only appears after scripts run may still be indexed by Google.

Which user-agent strings are used?

The formats each operator publishes, for example Mozilla/5.0 … compatible; GPTBot/1.1; +https://openai.com/gptbot. The request also carries From: tools@tapcub.com, so it is not hidden.

How do I verify a real crawler in my logs?

Do a reverse DNS lookup on the IP and a forward lookup on the result (Google, Bing), or compare the IP with the operator’s published list (OpenAI, Anthropic, Perplexity). The fake bot checker does this for you.

Changelog

Update history

New features, fixes and data refreshes for this tool, newest first.

  1. · v1.0

    LaunchedFirst release.

Data this tool relies on

  • Sample fetches (github.com, apple.com, wikipedia.org, example.com, bbc.com) — checked 2026-11-17, reviewed every 6 months (next review by 2027-05-17)

Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics