TapCub is live — analytics, insights and live chat, free on one platform
Analytics

Agent Analytics: How to Tell AI Agents From Human Visitors

Agent analytics separates AI agent requests from human sessions. Four classes of agent traffic, vendor-declared user agents, how to verify them, and a sample day where agents understated conversion by 29%.

Xu An Published Nov 20, 2026 12 min read Analytics
Sample data: 3,700 of 12,600 sessions (29.4%) were not human. Counting them all as visitors puts conversion at 2.12% instead of 3.00% for the 267 sign-ups.Sample data

Agent analytics means measuring AI agents separately from people. Sort requests into four classes (training crawlers, search crawlers, user-triggered fetchers and browser-driving agents), match published user-agent tokens, verify them against the operator's IP list or DNS, and flag the rest by behaviour. In the sample day below, agents were 29.4% of sessions and made the human conversion rate look 29% lower.

What you will learn

  • Four kinds of agent traffic need four treatments: training and search crawlers honour robots.txt, user-triggered fetchers generally do not, and browser agents have no standard
  • A user-agent name is a claim: verify against published IP lists or reverse DNS and record verified, suspected or unknown
  • Counting agents as visitors turned a 3.00% human conversion rate into 2.12% in our sample day

What counts as AI agent traffic?

"AI agent traffic" is four different things that need four different treatments. Lumping them together is the most common reason agent analytics is confusing.

ClassWhat it doesExample tokensDoes robots.txt apply?Is it a human visit?
Training crawlerCollects pages that may be used to train modelsGPTBot, ClaudeBotYes, vendors say they honour itNo
Search or answer crawlerBuilds the index an AI search feature answers fromOAI-SearchBot, PerplexityBot, Claude-SearchBotYesNo
User-triggered fetcherOpens one page because a person asked an assistant a questionChatGPT-User, Claude-User, Perplexity-UserGenerally no, because a user started itNo, but a person is behind it
Browser-driving agentOperates a full browser to click through a site for a person or a scriptOften a normal Chrome user agentNo standardLooks like one, is not

The first three usually announce themselves in the user agent. The fourth does not, which is why you also need behaviour signals. This guide covers how to separate all four from human sessions, what that does to your conversion numbers, and how to measure what the agents do.

Which user agents declare themselves, and what do the vendors say?

The operators of the large AI assistants publish their crawler names. The facts below were read from the vendors' own documentation on 2026-11-20; version numbers in the strings change, so match on the token, not the full string.

TokenOperator's stated purposerobots.txt
GPTBotCrawls content that may be used to train generative AI foundation modelsApplies
OAI-SearchBotSurfaces sites in ChatGPT's search features; not used for trainingApplies; changes can take about 24 hours
ChatGPT-UserVisits a page when a user asks, or for a Custom GPT action"robots.txt rules may not apply" because a user initiated it
ClaudeBotCollects web content that could contribute to model trainingHonoured; supports a Crawl-delay directive
Claude-UserRetrieves pages when a Claude user's question needs themHonoured per vendor
Claude-SearchBotCrawls to improve the relevance of search responsesHonoured per vendor
PerplexityBotSurfaces and links sites in search results; "not used to crawl content for AI foundation models"Recommended to allow
Perplexity-UserFetches a page when a user's question needs it"generally ignores robots.txt rules"

Sources: OpenAI, Anthropic, Perplexity.

Two consequences follow. First, a robots.txt rule that blocks the training crawler does not stop the user-triggered fetcher from the same operator, because the vendors say those requests are not governed by it. Second, blocking the search crawler can remove you from that assistant's answers, since the vendors describe it as the index builder. Decide per class, not per brand. For the underlying field, see user agent and AI crawler.

TokenStated purposerobots.txt
GPTBotTraining data collectionApplies
OAI-SearchBotSearch features, not trainingApplies
ChatGPT-UserFetch when a user asksMay not apply
ClaudeBotTraining data collectionHonoured
Claude-UserFetch when a user asksHonoured per vendor
PerplexityBotSearch results, not trainingAllow recommended
Perplexity-UserFetch when a user asksGenerally ignored

Read from each operator's documentation on 2026-11-20. Match the token, not the version in the string.

How do you tell a real declared agent from a fake one?

A user-agent string is just text the client chooses to send. Anyone can type "GPTBot" into a script, so a declared name is a claim, not proof. The robots.txt standard says plainly that its rules are "not a form of access authorization" and that crawlers are only "requested to honor" them (RFC 9309).

Verify before you count or block, using what the operator publishes:

  1. Check the published IP list. Perplexity publishes the ranges for both of its agents in JSON files and tells site owners to match each bot's user agent with its own ranges. Anthropic says requests from addresses on its published list at claude.com/crawling/bots.json come from Anthropic. If a request claims the token but comes from outside the list, treat it as unverified.
  2. Use reverse and forward DNS when the operator documents it. Google's procedure for Googlebot is: reverse-DNS the IP, check that the domain is googlebot.com, google.com or googleusercontent.com, forward-DNS that name, and confirm it returns the original IP (Google).
  3. Record the result as a status, not a yes or no. Use three: verified, suspected, unknown. "Unknown" is not the same as "fake": most operators do not publish a reverse-DNS pattern.

Do not block on IP alone without reading the operator's advice. Anthropic, for example, says blocking IPs may not give a persistent opt-out because it stops the bot from reading robots.txt.

How do you catch agents that look like browsers?

Some agents drive a real browser and send an ordinary browser user agent. You cannot find them by name. You can find many of them by what a session does, because human sessions and scripted sessions leave different shapes in a log.

SignalTypical human sessionTypical scripted session
Static assets (CSS, JS, images) per pageManyFew or none when pages are fetched as text
Pages per minute1–3, with pausesDozens, evenly spaced
Fetches robots.txt firstNeverSometimes, for declared crawlers
Runs page JavaScriptYesOften no
ReferrerSearch, social, direct or an assistantNone
Path patternWanders and returns to earlier pagesSweeps sitemap order or a numeric sequence

A rule that is easy to apply and easy to explain: flag a session as suspected agent if it requests at least 30 pages in 2 minutes with zero static-asset requests. Review the matches before acting. Legitimate exceptions exist: link-prefetch, a screen reader that loads text only, an uptime check you set up yourself, or a corporate proxy that hides many people behind one address.

Sample sessions (sample data) show how the rule behaves:

SessionPages in 2 minStatic assetsDeclared tokenClass
S1341noneHuman
S2520GPTBotDeclared training crawler
S310ChatGPT-UserDeclared user-triggered fetch
S4440none (Chrome string)Suspected agent
S550noneHuman (prefetch or screen reader; below the 30-page threshold)

S5 shows why the threshold matters: a few text-only requests are not enough to call something an agent.

What do agents do to your numbers? A worked example

Suppose a site's analytics counted one day of traffic with no bot filtering and got 12,600 sessions. Classifying them by the methods above (sample data, counts recomputed in Python):

ClassSessionsShare of all sessions
Human8,90070.6%
Search engine crawlers1,45011.5%
Declared AI training crawlers1,0208.1%
Declared AI search and answer crawlers6104.8%
Declared user-triggered AI fetchers3402.7%
Suspected browser-style agents2802.2%
Total12,600all sessions

On the same day the site recorded 267 sign-ups. Only people sign up, so:

  • Conversion rate on human sessions: 267 ÷ 8,900 = 3.00%.
  • Conversion rate on all sessions: 267 ÷ 12,600 = 2.12%.

Counting the agents as visitors understated the human conversion rate by 29% (1 − 2.12 ÷ 3.00). Nothing about the site changed; the denominator was polluted. If you then ran a test on the page, the noise would have lowered the baseline and made real lifts harder to detect.

The same table shows that 3,700 of the 12,600 sessions (29.4%) were not human. That share is specific to this sample; yours depends on your site, and you will only know it by measuring it.

app.tapcub.com

Sessions by class, one sample day

12,600 sessions
Human8,900
Search engine crawlers1,450
AI training crawlers1,020
AI search and answer610
User-triggered AI fetch340
Suspected browser agents280
Sample data
Sample data: 3,700 of 12,600 sessions (29.4%) were not human. Counting them all as visitors puts conversion at 2.12% instead of 3.00% for the 267 sign-ups.

What are agents actually doing: reading, or sending people back?

Once humans are separated out, the useful question about agent traffic is an exchange rate. Of the 3,700 non-human sessions, 1,970 were declared AI reads (1,020 + 610 + 340). In the same day, 61 human visitors arrived from an AI assistant's link. That gives:

1,970 reads ÷ 61 visitors sent back = 32.3 reads per human visitor.

Interpret that number with care. It is a description of one day on one site, not a verdict on any operator. A training crawler reading a page is not expected to send anyone back; an answer engine is. So split the ratio by class: 1,020 training reads with no expected return, and 950 answer and fetch reads (610 + 340) against 61 visits is 15.6 reads per visitor. Watch how that moves after you change content or robots.txt rules, rather than comparing the absolute number to other sites.

Three questions give the agent data a purpose:

  1. Which pages do agents read most? If the pages are your pricing and docs, those are the pages assistants will quote, so keep them accurate.
  2. Do they get clean answers? Check the most common status code per crawler. A crawler that mostly receives 404 or 5xx is reading errors.
  3. Do the rules you wrote match the traffic? If the user-triggered fetcher keeps arriving after you blocked training crawlers in robots.txt, that is expected from the vendors' documentation, not a bug. Use a firewall rule if you need to stop it.

What about an agent that reads your analytics?

"Analytics agent" also means the reverse direction: your own AI agent querying your analytics. The safe pattern is read-only access with a scoped key, so the agent can answer questions about last week's traffic but cannot edit settings or delete data. Treat every number it returns the way you would treat a number from a colleague: ask for the query behind it, and check one figure by hand. TapCub's documentation on connecting AI agents with MCP describes this read-only route.

How do you measure agent traffic in TapCub?

Using the example above, here is the same separation done in TapCub.

TapCub AI content ledger listing reads and visitors sent back per operator
Demo data: the AI content ledger on Web analytics › Bots. Figures are from a demo project, not the sample used in the article.
  1. Confirm bots are reported separately. New projects default to detecting bots and reporting them separately, so recognised bots go only to the bot reports and do not inflate human visitors, sessions or conversion rates. Only the Do not detect setting mixes them in, and the documentation does not recommend it.
  2. Open Web analytics › Bots. On the AI & agents tab, the cards show AI requests, AI agents, AI crawlers and visits sent back. The AI content ledger lists, per operator, reads (training crawlers and answer engines in separate columns), human visitors sent back and a return ratio, which is reads divided by visitors. In the example that ratio would be 32.3 : 1 overall.
  3. Check one crawler on All bots. The table lists name, operator, purpose, requests, pages visited, most common status code, last seen, trust level and the detection layers matched. Trust is verified, suspected or unknown; unknown does not by itself mean spoofed. If a row is wrong, report a false positive.
  4. Know the limit. With only the page script, TapCub sees bots that run JavaScript. Crawlers that do not run it, plus status codes and response times, need server-side collection: send those requests from your server through the server events endpoint, which keeps only requests classified as bots. See bots and AI crawlers and read the AI crawler ledger.
  5. Compare with people. On AI visibility and Sources › AI sources, the visits sent back are human sessions, so you can set reads against people in one place. For a fuller walk through the ledger, see the AI crawler ledger.

TapCub does not block or throttle any crawler; it measures them. Blocking happens in robots.txt or at your firewall.

FAQ

What is agent analytics?

Measuring what AI agents do on your site or product: which agents visit, what they read, whether they are verified, and how much human traffic they send back, kept separate from human sessions.

How do you identify AI agent traffic?

Match user-agent tokens published by operators, verify them against published IP ranges or reverse DNS, then flag undeclared sessions by behaviour such as many pages with no static assets.

Do AI agents follow robots.txt?

Training and search crawlers are documented as honouring it. User-triggered fetchers such as ChatGPT-User and Perplexity-User are documented as not being bound by it, because a person started the request.

Should you block AI agents?

Decide by class. Blocking a training crawler opts out of training; blocking a search crawler can remove you from that assistant's answers. Check the ledger's reads against visits sent back before choosing.

Why do some AI crawlers not appear in a script-based report?

A page script only runs when the client executes JavaScript. Crawlers that do not run it appear in server logs, not in a script-based report, which is why TapCub documents server-side collection for them.

Sources

Filed underAnalytics

Was this article helpful?

We use your answers to fix and expand this page.

Xu An

Customer Experience Consultant

Writes about live chat, visitor conversations, and self-service, sharing practical ways to make customer communication clearer and more effective.

See agents apart from people

Bots are reported separately by default, with an AI content ledger that sets reads against human visitors sent back.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics