TapCub is live — analytics, insights and live chat, free on one platform

Free tool AI visibility checker: can AI crawlers reach and read your site?

Before ChatGPT, Claude or Perplexity can cite a page, their crawlers have to be allowed in, get a real response and find readable text. Enter a site and this checker looks at eight things that decide that — robots.txt rules, what the page returns to GPTBot, ClaudeBot and PerplexityBot, server-rendered text, structured data, snippet controls, llms.txt, speed and sitemap — and scores them on a published 100-point scale.

  • Score 0–100, rules published
  • Fetched as 3 AI crawlers
  • 7 requests per check

Check a site’s AI visibility

Enter a domain or a page URL. robots.txt, llms.txt and the sitemap are read from the site root; the page itself is fetched four times. The result below is a dated sample until you click Check.

Checker
What is sent, and to whom. Nothing is requested when the page opens: the result below is a dated sample. When you click the button, the address goes to TapCub’s checking service check.tapcub.com, which requests the pages from Cloudflare’s network with a From: tools@tapcub.com header and returns status codes and the parts of the page this tool needs. The service keeps only host names, status codes and timings for rate limiting, not paths, queries or page content, and stores no results. One check uses 7 requests: robots.txt, llms.txt, the page as Chrome, GPTBot, ClaudeBot and PerplexityBot, and the sitemap. Limit: 10 requests a minute and 200 a day per IP address, so wait a minute between checks.

For example example.com or https://example.com/pricing.

Sample result, checked on 2026-11-17 while building this page. Click the button to check live.

AI visibility score

55

Out of 100. Each part and its rule are listed below.

Breakdown

Fair: some AI crawlers are blocked or get little to readChecked: https://medium.com/
  1. robots.txt lets AI search bots in20 / 20
    • All 6 AI search and user-request bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) may fetch this page.
    • Training crawlers blocked: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider. That keeps content out of model training and does not affect this score; OpenAI and Anthropic document their search bots separately.
    See every AI crawler in robots.txt →
  2. AI crawlers actually get the page0 / 20
    • GPTBot was refused (403).
    • ClaudeBot was refused (403).
    • PerplexityBot was refused (403).
    • These requests came from Cloudflare’s network, not from the crawlers’ published IP ranges. A site that verifies crawlers by IP may let the real ones in; one that blocks by user-agent blocks them too.
    Compare the page as each bot →
  3. Readable text without JavaScript2 / 20
    • Almost no text in the server HTML (42 words): the page is probably rendered by JavaScript. Most AI crawlers do not run JavaScript.
    • No <h1> heading in the HTML.
    Compare the page as each bot →
  4. Structured data (JSON-LD)15 / 15
    • 1 valid JSON-LD block(s): Organization.
    Inspect the structured data →
  5. No snippet or index blocks10 / 10
    • No noindex, nosnippet or max-snippet:0 in meta robots or the X-Robots-Tag header.
    Check index directives →
  6. llms.txt0 / 5
    • No llms.txt (404).
    • llms.txt is a proposal; no major AI company has said its crawlers use it, which is why it carries only 5 points.
    Validate llms.txt →
  7. Response time3 / 5
    • The page answered in 1.8 s. Under a second is better.
    Measure page speed →
  8. Sitemap5 / 5
    • Sitemap declared in robots.txt and reachable: https://medium.com/sitemap/sitemap.xml.
    Check the sitemap’s URLs →
Parsing, scoring and the tables are worked out in your browser from what the service returns. Results stay in this tab and disappear when you leave.

Share these inputs: the values go after the # in the link, so they never reach a server.

Version v1.0Updated Update history

Read the result

What the score does and does not say

The score measures access and readability. It cannot predict whether an AI assistant will cite you.

80–100: open and readable

AI search bots are allowed, get the page, and find text, headings and structured data. What gets cited from here depends on the content and on each assistant’s own ranking.

50–79: partly blocked or thin

Usually one big gap: a firewall refusing bot user-agents, a JavaScript-only page, or search bots blocked in robots.txt. Fix the lowest-scoring part first.

Below 50: largely invisible

AI crawlers are blocked, challenged or get an empty shell. Assistants may still mention you from other sources, but cannot read this page.

Training vs answers

Blocking GPTBot or ClaudeBot (training) is a legitimate choice and costs no points. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot (search and answers) does.

Results are a snapshot from Cloudflare’s network. Crawlers come from their own IP ranges and may be treated differently by your firewall; response time depends on where the request came from.

How to use it

Three steps

No sign-up. One check takes 7 requests and about 10 seconds.

  1. 1

    Enter your site

    A domain checks the home page. Enter a specific URL to check an important page such as pricing, docs or a top article.

  2. 2

    Read the breakdown

    Each of the eight parts shows its points, what was found and a link to the tool that goes deeper.

  3. 3

    Fix the biggest gap

    Start with robots.txt and bot access (40 points together), then server-rendered text. llms.txt and speed are worth little.

How the score works

Eight parts, 100 points, rules in the open

Weights reflect what decides whether a crawler gets usable content. Every rule is applied exactly as written here.

  • robots.txt (20): share of the 6 AI search and user-request bots allowed to fetch the page, under RFC 9309 rules. Missing robots.txt (4xx) = allowed; unreachable (5xx) = blocked
  • Bot access (20): the page fetched as GPTBot, ClaudeBot and PerplexityBot; each bot that gets a 2xx, no challenge page and at least half the browser’s text earns a third
  • Readable text (20): 300+ words in the server HTML = 15, 100–299 = 8; a <title> = 2; an <h1> = 3
  • Structured data (15): valid JSON-LD with an @type = 15, valid without @type = 10, only invalid blocks = 3
  • Snippet controls (10): noindex or none = 0, nosnippet or max-snippet:0 = 3, otherwise 10
  • llms.txt (5): valid = 5, present with errors = 2. Response time (5): under 1 s = 5, under 2.5 s = 3, slower = 1. Sitemap (5): declared in robots.txt and reachable = 5, only at /sitemap.xml = 4, declared but broken = 1
Which crawlers count
CrawlerOperatorUsed forIn score
OAI-SearchBotOpenAIChatGPT search resultsYes
ChatGPT-UserOpenAIFetches when a user asksYes
Claude-SearchBot / Claude-UserAnthropicSearch / user requestsYes
PerplexityBot / Perplexity-UserPerplexitySearch / user requestsYes
GPTBot, ClaudeBotOpenAI, AnthropicModel trainingShown, no points
Google-Extended, Applebot-Extended, CCBot…VariousTrainingShown, no points
Worked example

medium.com on 2026-11-17

The default result above: a site that welcomes AI search bots in robots.txt but refuses AI user-agents at the door.

Where the sample earned and lost points
PartPointsWhy
robots.txt20 / 20All six search and user-request bots allowed; GPTBot, ClaudeBot and other training crawlers blocked
Bot access0 / 20GPTBot, ClaudeBot and PerplexityBot all got 403 from our requests
Readable text2 / 20About 40 words in the server HTML: the home page is built by JavaScript
Structured data15 / 15Valid Organization JSON-LD
Snippets, llms.txt, speed, sitemap18 / 25No snippet limits; no llms.txt; 1.8 s; sitemap declared in robots.txt

Total 55. The 403s came from requests sent from Cloudflare’s network; the real crawlers, coming from their own IP ranges, may be let through. That is exactly the kind of gap worth confirming in your server logs.

Common mistakes

Why sites end up invisible to AI search

Most cases are side effects of settings made for other reasons.

Bot protection on by default

CDN and firewall “block AI bots” switches often catch search bots as well as training crawlers. Allow the search bots you want by name.

One rule for all AI bots

Blocking every AI user-agent to stop training also removes you from ChatGPT search and Perplexity answers. The operators separate these bots for that reason.

Content only after JavaScript

Most AI crawlers read the HTML as sent. A client-rendered page gives them a title and an empty div.

Expecting llms.txt to fix it

llms.txt is a proposal with no confirmed use by the major crawlers. Access and readable HTML matter far more.

FAQ

AI visibility checker questions

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

How do I check if AI crawlers can access my site?

Enter your domain above and click Check. The tool reads your robots.txt rules for the AI search bots, fetches your page as GPTBot, ClaudeBot and PerplexityBot, and compares what they get with what a browser gets.

Does blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search, according to OpenAI: GPTBot is for training, and search results use OAI-SearchBot, which is controlled separately. That is why this score counts the search bots and only shows the training ones.

Why does the tool say GPTBot is blocked when my robots.txt allows it?

robots.txt is only a request; a firewall or CDN rule can still refuse the user-agent. The “Bot access” part shows what the server actually returned. The requests come from Cloudflare’s network, so an IP-based allow rule for the real crawlers may behave differently.

Will a high score get me cited by AI assistants?

No score can promise that. It means the crawlers can get in and read your page; whether an assistant cites it depends on the question, your content and the assistant’s own ranking.

Do I need an llms.txt file?

It is optional. llms.txt is a proposed format, and none of the major AI companies has said its crawlers use it, so it is worth 5 of 100 points here. Robots rules and readable HTML matter much more.

Does nosnippet affect AI answers?

For Google, yes: Google documents that nosnippet and max-snippet also limit what appears in AI Overviews and AI Mode. Other assistants publish no equivalent rule.

How often should I check?

After changing robots.txt, CDN or firewall settings, or moving to a JavaScript framework. In between, crawler visits in your analytics or logs tell you more than repeated checks.

Changelog

Update history

New features, fixes and data refreshes for this tool, newest first.

  1. · v1.0

    LaunchedFirst release.

Data this tool relies on

  • Sample batch, sitemap, link and tracking checks — checked 2026-11-17, reviewed every 6 months (next review by 2027-05-17)

Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates

Next step

Watch AI crawlers and AI referrals over time

TapCub records AI crawler visits separately from people and shows the visitors AI assistants send you.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics