Free tool AI visibility checker: can AI crawlers reach and read your site?
Before ChatGPT, Claude or Perplexity can cite a page, their crawlers have to be allowed in, get a real response and find readable text. Enter a site and this checker looks at eight things that decide that — robots.txt rules, what the page returns to GPTBot, ClaudeBot and PerplexityBot, server-rendered text, structured data, snippet controls, llms.txt, speed and sitemap — and scores them on a published 100-point scale.
- Score 0–100, rules published
- Fetched as 3 AI crawlers
- 7 requests per check
Check a site’s AI visibility
Enter a domain or a page URL. robots.txt, llms.txt and the sitemap are read from the site root; the page itself is fetched four times. The result below is a dated sample until you click Check.
check.tapcub.com, which requests the pages from Cloudflare’s network with a From: tools@tapcub.com header and returns status codes and the parts of the page this tool needs. The service keeps only host names, status codes and timings for rate limiting, not paths, queries or page content, and stores no results. One check uses 7 requests: robots.txt, llms.txt, the page as Chrome, GPTBot, ClaudeBot and PerplexityBot, and the sitemap. Limit: 10 requests a minute and 200 a day per IP address, so wait a minute between checks.For example example.com or https://example.com/pricing.
AI visibility score
55
Out of 100. Each part and its rule are listed below.
Breakdown
-
robots.txt lets AI search bots in20 / 20
- All 6 AI search and user-request bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) may fetch this page.
- Training crawlers blocked: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider. That keeps content out of model training and does not affect this score; OpenAI and Anthropic document their search bots separately.
-
AI crawlers actually get the page0 / 20
- GPTBot was refused (403).
- ClaudeBot was refused (403).
- PerplexityBot was refused (403).
- These requests came from Cloudflare’s network, not from the crawlers’ published IP ranges. A site that verifies crawlers by IP may let the real ones in; one that blocks by user-agent blocks them too.
-
Readable text without JavaScript2 / 20
- Almost no text in the server HTML (42 words): the page is probably rendered by JavaScript. Most AI crawlers do not run JavaScript.
- No <h1> heading in the HTML.
-
Structured data (JSON-LD)15 / 15
- 1 valid JSON-LD block(s): Organization.
-
No snippet or index blocks10 / 10
- No noindex, nosnippet or max-snippet:0 in meta robots or the X-Robots-Tag header.
-
llms.txt0 / 5
- No llms.txt (404).
- llms.txt is a proposal; no major AI company has said its crawlers use it, which is why it carries only 5 points.
-
Response time3 / 5
- The page answered in 1.8 s. Under a second is better.
-
Sitemap5 / 5
- Sitemap declared in robots.txt and reachable: https://medium.com/sitemap/sitemap.xml.
Share these inputs: the values go after the # in the link, so they never reach a server.
https://tapcub.com/tools/ai-readiness-checker
Version v1.0Updated Update history
What the score does and does not say
The score measures access and readability. It cannot predict whether an AI assistant will cite you.
80–100: open and readable
AI search bots are allowed, get the page, and find text, headings and structured data. What gets cited from here depends on the content and on each assistant’s own ranking.
50–79: partly blocked or thin
Usually one big gap: a firewall refusing bot user-agents, a JavaScript-only page, or search bots blocked in robots.txt. Fix the lowest-scoring part first.
Below 50: largely invisible
AI crawlers are blocked, challenged or get an empty shell. Assistants may still mention you from other sources, but cannot read this page.
Training vs answers
Blocking GPTBot or ClaudeBot (training) is a legitimate choice and costs no points. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot (search and answers) does.
Results are a snapshot from Cloudflare’s network. Crawlers come from their own IP ranges and may be treated differently by your firewall; response time depends on where the request came from.
Three steps
No sign-up. One check takes 7 requests and about 10 seconds.
-
1
Enter your site
A domain checks the home page. Enter a specific URL to check an important page such as pricing, docs or a top article.
-
2
Read the breakdown
Each of the eight parts shows its points, what was found and a link to the tool that goes deeper.
-
3
Fix the biggest gap
Start with robots.txt and bot access (40 points together), then server-rendered text. llms.txt and speed are worth little.
Eight parts, 100 points, rules in the open
Weights reflect what decides whether a crawler gets usable content. Every rule is applied exactly as written here.
- robots.txt (20): share of the 6 AI search and user-request bots allowed to fetch the page, under RFC 9309 rules. Missing robots.txt (4xx) = allowed; unreachable (5xx) = blocked
- Bot access (20): the page fetched as GPTBot, ClaudeBot and PerplexityBot; each bot that gets a 2xx, no challenge page and at least half the browser’s text earns a third
- Readable text (20): 300+ words in the server HTML = 15, 100–299 = 8; a
<title>= 2; an<h1>= 3 - Structured data (15): valid JSON-LD with an @type = 15, valid without @type = 10, only invalid blocks = 3
- Snippet controls (10): noindex or none = 0, nosnippet or max-snippet:0 = 3, otherwise 10
- llms.txt (5): valid = 5, present with errors = 2. Response time (5): under 1 s = 5, under 2.5 s = 3, slower = 1. Sitemap (5): declared in robots.txt and reachable = 5, only at /sitemap.xml = 4, declared but broken = 1
| Crawler | Operator | Used for | In score |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results | Yes |
| ChatGPT-User | OpenAI | Fetches when a user asks | Yes |
| Claude-SearchBot / Claude-User | Anthropic | Search / user requests | Yes |
| PerplexityBot / Perplexity-User | Perplexity | Search / user requests | Yes |
| GPTBot, ClaudeBot | OpenAI, Anthropic | Model training | Shown, no points |
| Google-Extended, Applebot-Extended, CCBot… | Various | Training | Shown, no points |
medium.com on 2026-11-17
The default result above: a site that welcomes AI search bots in robots.txt but refuses AI user-agents at the door.
| Part | Points | Why |
|---|---|---|
| robots.txt | 20 / 20 | All six search and user-request bots allowed; GPTBot, ClaudeBot and other training crawlers blocked |
| Bot access | 0 / 20 | GPTBot, ClaudeBot and PerplexityBot all got 403 from our requests |
| Readable text | 2 / 20 | About 40 words in the server HTML: the home page is built by JavaScript |
| Structured data | 15 / 15 | Valid Organization JSON-LD |
| Snippets, llms.txt, speed, sitemap | 18 / 25 | No snippet limits; no llms.txt; 1.8 s; sitemap declared in robots.txt |
Total 55. The 403s came from requests sent from Cloudflare’s network; the real crawlers, coming from their own IP ranges, may be let through. That is exactly the kind of gap worth confirming in your server logs.
Why sites end up invisible to AI search
Most cases are side effects of settings made for other reasons.
Bot protection on by default
CDN and firewall “block AI bots” switches often catch search bots as well as training crawlers. Allow the search bots you want by name.
One rule for all AI bots
Blocking every AI user-agent to stop training also removes you from ChatGPT search and Perplexity answers. The operators separate these bots for that reason.
Content only after JavaScript
Most AI crawlers read the HTML as sent. A client-rendered page gives them a title and an empty div.
Expecting llms.txt to fix it
llms.txt is a proposal with no confirmed use by the major crawlers. Access and readable HTML matter far more.
See which AI crawlers visit, and who they send back
A check shows whether crawlers can get in. TapCub shows whether they actually come, and whether AI assistants send you visitors.
AI crawler report
AI crawlers identified by name and behaviour, apart from human visits.
Learn moreCrawl compliance
Your robots.txt and llms.txt are fetched daily and compared with what actually crawled you.
Learn moreAI referrals
Visitors arriving from ChatGPT, Perplexity and other assistants on their own line.
Learn moreAI visibility checker questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
How do I check if AI crawlers can access my site?
Enter your domain above and click Check. The tool reads your robots.txt rules for the AI search bots, fetches your page as GPTBot, ClaudeBot and PerplexityBot, and compares what they get with what a browser gets.
Does blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search, according to OpenAI: GPTBot is for training, and search results use OAI-SearchBot, which is controlled separately. That is why this score counts the search bots and only shows the training ones.
Why does the tool say GPTBot is blocked when my robots.txt allows it?
robots.txt is only a request; a firewall or CDN rule can still refuse the user-agent. The “Bot access” part shows what the server actually returned. The requests come from Cloudflare’s network, so an IP-based allow rule for the real crawlers may behave differently.
Will a high score get me cited by AI assistants?
No score can promise that. It means the crawlers can get in and read your page; whether an assistant cites it depends on the question, your content and the assistant’s own ranking.
Do I need an llms.txt file?
It is optional. llms.txt is a proposed format, and none of the major AI companies has said its crawlers use it, so it is worth 5 of 100 points here. Robots rules and readable HTML matter much more.
Does nosnippet affect AI answers?
For Google, yes: Google documents that nosnippet and max-snippet also limit what appears in AI Overviews and AI Mode. Other assistants publish no equivalent rule.
How often should I check?
After changing robots.txt, CDN or firewall settings, or moving to a JavaScript framework. In between, crawler visits in your analytics or logs tell you more than repeated checks.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Sample batch, sitemap, link and tracking checks — checked 2026-11-17, reviewed every 6 months (next review by 2027-05-17)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Watch AI crawlers and AI referrals over time
TapCub records AI crawler visits separately from people and shows the visitors AI assistants send you.