TapCub is live — analytics, insights and live chat, free on one platform
Analytics

Types of Bots: 10 Kinds and How to Spot Each in Analytics

Ten types of bots sorted by purpose, with user-agent clues, traffic patterns, a way to verify crawlers and a rule for allowing, limiting or blocking each. Includes a worked example of how bots distort conversion rate.

Su He Published Nov 20, 2026 14 min read Analytics
Sample data: 100,000 requests in one week, classified by user-agent, DNS check and behaviour. Automated requests are 46%, the same order as Imperva's 51% for the whole web.Sample data

The main types of bots are search crawlers, AI crawlers, SEO and monitoring tools, link-preview fetchers, scrapers, credential-stuffing bots, spam bots and click-fraud bots. Imperva puts automated traffic at 51% of web traffic and bad bots at 37%, so sorting them is part of reading any traffic number. This guide gives a ten-row table with user-agent and traffic-pattern signs, a three-signal way to check a bot, and a rule for allowing, limiting or blocking each.

What you will learn

  • Sort bots by purpose, then give each a verdict: allow, limit or block; the user-agent alone never proves who is calling
  • Verify claimed crawlers with reverse and forward DNS; in the sample, 1,800 of 9,000 Googlebot hits (20%) were fake
  • Unfiltered bot sessions understated the sample conversion rate by 20% (1.50% instead of 1.875%); robots.txt asks, the server enforces

How are bots grouped, and how much traffic are they?

Bots are usually sorted by what they are for, not by how they are built: crawlers that index or train, tools that monitor or audit, and programs that attack or scrape. The ten-row table above uses exactly that split. The split matters because the same request line in your logs can be a search engine you want, an AI crawler you may want to limit, or a scraper you should block.

Scale first. Imperva's 2025 Bad Bot Report puts automated traffic at 51% of all web traffic and bad bots at 37% of it. Subtract and you get roughly 14 points for everything else that is automated: search crawlers, monitors, feed fetchers and the like. So on an average site, "human traffic" is less than half of the requests, and bad bots outnumber the useful ones by more than two to one. Your own mix will differ, because a small site with no login rarely attracts credential attacks and a large store attracts a lot of scraping.

Use this three-way verdict when you classify a bot:

  • Allow: it helps you get found (search engines) and passes verification.
  • Limit: it is legitimate but costs you bandwidth or gives little back (some SEO crawlers, AI training crawlers).
  • Block: it is trying to take something it has no right to (credential stuffing, scraping behind a login, form spam, click fraud).

The rest of this guide covers how to tell the types apart in your data, what the verified crawlers say about themselves, what bots do to your numbers, and which control to use for which type.

TypeUser-agent cluesTraffic patternVerdict
Search crawlerNames such as Googlebot; verify by DNSSteady, follows links and sitemap, honours robots.txtAllow
AI training crawlerOwn token, e.g. GPTBotBroad fetching of many pages with few visits sent backLimit or allow by policy
AI search crawlerOwn token, e.g. OAI-SearchBotFetches pages that may be cited in answersAllow if you want citations
User-triggered AI fetcherOwn token, e.g. ChatGPT-UserSingle fetch right after a person asksTreat like a visitor; robots.txt may not apply
SEO tool crawlerTool name in the stringLarge site crawls, bursts on one domainAllow or rate-limit
Uptime monitorService name in the stringSame URL at a fixed interval, all dayAllow; exclude from analytics
Link-preview fetcherPlatform name in the stringOne hit on one URL right after a shareAllow
ScraperCopies a browser string or headless ChromeWalks list pages fast; skips images and scripts; rotating IPsBlock
Credential stufferBrowser-like stringMany failed logins across many accountsBlock
Spam or click-fraud botBrowser-like or empty stringForm posts with no page view; paid sessions of one secondBlock

Names in the user-agent column are examples from operator documentation; a name is a claim, not proof.

How can you tell a bot from a visitor?

Use three signals, from cheapest to strongest, and never rely on the first one alone.

1. The user-agent string. It is the client's self-reported identity, as the Apache documentation puts it for the User-agent field of the combined log format. Honest bots put their name in it (Googlebot, GPTBot, OAI-SearchBot, ChatGPT-User). Bad bots copy a normal browser string. A name match tells you who claims to be calling, nothing more.

2. The network. Real crawlers run from addresses their operator can prove. Google's documentation gives a two-step check for Googlebot: run a reverse DNS lookup on the IP from your log, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. For example, host 66.249.66.1 returns a crawl-66-249-66-1.googlebot.com name, and the forward lookup of that name must return the same address. If either step fails, the request is not Googlebot, whatever its user-agent says.

3. The behaviour. Bots leave patterns that people do not:

BehaviourTypical ofWhy it gives them away
Hundreds of pages per minute from one IP or one small rangeScrapers, aggressive crawlersNo person reads a page every 200 ms
Exactly the same interval between hits (for example every 60 seconds)Uptime monitorsHumans are irregular
Many failed logins across many accounts from rotating IPsCredential stuffingOne attempt per account, spread out to avoid lockout
Requests that skip images, CSS and scriptsSimple HTTP scrapersA browser loads all of them
One hit on a single URL right after a link is postedLink-preview fetchersTriggered by a share, not by browsing
Form posts with no preceding page viewSpam botsA real visitor loads the form first

Two cautions. First, a bot that runs a full browser (headless Chrome) looks like a person to a page script, so behaviour and network checks matter more for scrapers than the user-agent. Second, a missing signal is not proof: an unrecognised user-agent from an unverified IP is "unknown", not "malicious".

What do search and AI crawlers say about themselves?

Crawlers from big operators publish their names and rules. Read those pages instead of guessing.

Search crawlers. Googlebot is verified with the DNS check above. Google also lists user-triggered fetchers on separate hostname patterns such as gae.googleusercontent.com or google-proxy-...google.com, so a hit from those is a fetch somebody asked for, not routine crawling.

AI crawlers from one operator. OpenAI's bot documentation separates three user-agent tokens, because they do three different jobs:

TokenWhat it doesrobots.txt effect
GPTBotCollects content that may train generative AI modelsDisallowing it says your content should not be used for training
OAI-SearchBotSurfaces sites in ChatGPT's search featuresDisallowing it keeps your pages out of ChatGPT search answers (links can still appear); changes take about 24 hours
ChatGPT-UserFetches a page when a person asks ChatGPT to, or via GPT ActionsOpenAI says robots.txt rules may not apply, because a user started the action

That table is the practical lesson: "AI bot" is not one type. Blocking the training crawler is a different decision from blocking the one that decides whether you show up in an AI search answer. Other operators publish similar splits, so check each operator's page.

What robots.txt can and cannot do. The robots exclusion standard (RFC 9309, September 2022) states that its rules "are not a form of access authorization". Compliant crawlers follow them; hostile bots ignore them. The RFC also says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, and that parsers must handle at least 500 kibibytes. Two practical rules follow: keep the file small and do not expect a change to apply in minutes.

Which bots are harmful, and what do they look like?

Bad bots are defined by intent, so you spot them by what they do:

  • Scrapers copy prices, listings or articles. They walk lists page by page, often at night, and may use rotating residential IPs and a browser-like user-agent. In analytics they show up as a spike on category or product pages with no clicks on anything else and a near-100% bounce.
  • Credential stuffing tries stolen username and password pairs against your login. The signature is a high rate of failed logins, many distinct accounts, few attempts per account. Your server or auth logs show it; a page script usually does not.
  • Spam bots fill comment, contact and signup forms with links. Look for form submissions with no page view, identical text, or a burst of signups with throwaway addresses.
  • Click-fraud and fake-impression bots hit paid landing pages. A paid campaign whose sessions last one second, arrive from one data-centre network and never scroll is the classic case.
  • Denial-of-service floods are the extreme case: a sudden jump in requests per second, often to one URL.

Cheap defences, in the order to try them: rate limiting per IP, a challenge on login and signup forms, a honeypot field on forms, and blocking by network when one range causes most of the damage. None of them need a vendor, and none of them are analytics, so do them at the server or CDN and use analytics only to see whether they worked.

What do bots do to your analytics numbers?

Three numbers from a made-up site show the damage. All figures here are sample data.

1. Traffic mix. One week, 100,000 requests classified by the methods above: 54,000 human (54%), 13,000 scrapers, 9,000 search crawlers, 6,000 AI crawlers, 6,000 spam, 5,000 credential attempts, 4,000 SEO-tool crawls, 2,000 monitor pings and 1,000 link previews. Automated requests are 46%, the same order of magnitude as Imperva's 51% figure for the web as a whole.

2. Fake Googlebot. The logs contain 9,000 requests claiming to be Googlebot. Running the reverse-then-forward DNS check on every distinct IP verifies 7,200 of them. The other 1,800 (20%) are something else wearing Googlebot's name. If you had allow-listed by user-agent, a fifth of your "trusted" traffic would have been unverified.

3. Conversion rate. The site had 40,000 sessions, 600 conversions and 8,000 sessions from bots that execute JavaScript:

  • Unfiltered conversion rate: 600 ÷ 40,000 = 1.50%.
  • Human-only: 600 ÷ (40,000 − 8,000) = 600 ÷ 32,000 = 1.875%.

The unfiltered number understates the true rate by 20% (1 − 1.50 ÷ 1.875). A team that sees 1.5% and "improves" the checkout is chasing a gap that bots created. The same distortion inflates bounce rate, depresses pages per session and ruins any per-visitor cost figure.

app.tapcub.com

Requests by class, one sample week

% of 100,000
Human visitors54%
Scrapers13%
Search crawlers9%
AI crawlers6%
Spam bots6%
Credential attempts5%
SEO, monitors, previews7%
Sample data
Sample data: 100,000 requests in one week, classified by user-agent, DNS check and behaviour. Automated requests are 46%, the same order as Imperva's 51% for the whole web.

Block, allow or ignore: which control fits which type?

If the bot is…Do thisWhere
A verified search crawlerAllow; watch for 404 and 5xx responses it receivesrobots.txt, server logs
An AI training crawler you do not wantDisallow its token in robots.txt, then check whether requests stoprobots.txt
An AI search or answer crawlerAllow if you want citations; track how many visits it sends backrobots.txt, analytics
A user-triggered AI fetcherTreat as a visitor-like fetch; robots.txt may not apply, so use access control if neededServer or CDN
An SEO tool crawlerAllow, or limit the crawl rate if it loads the serverrobots.txt, rate limit
A monitor you set upExclude from analyticsAnalytics filter
A scraper, credential stuffer or spam botBlock; robots.txt will not stop itServer, CDN, form protection
UnknownDo nothing yet; log it for a week and look at behaviourLogs

The rule behind the table: robots.txt asks, the server enforces. Use robots.txt to state a preference to operators who comply, and use rate limits, authentication and network blocks for everyone else.

How do you see bots separately in TapCub?

Take the 1.50% versus 1.875% problem above (the detection layers are explained in how bot detection works). In TapCub the first fix is a default: a new project is set to Detect and report separately, so recognised bots go only to the bot report and do not inflate human visitors, sessions or pageviews. Here is how to use that:

TapCub bot report showing AI requests, AI reads against visits sent back over a day, and a table of crawler requests and referral visitors by operator
Demo data: the AI & agents view of the bot report, with reads and visits sent back per crawler operator.
  1. Open Web analytics › Bots. The AI & agents tab shows AI requests, the AI content ledger (reads per operator, with training crawlers and answer engines in separate columns, and the human visitors each sent back), and the return ratio.
  2. Open All bots. Each row has the bot name, operator, purpose (search, AI training, AI answer engine, monitoring), requests, pages visited, the most common status code, last seen, a trust level and the detection layers matched. Only Applebot gets a DNS reverse-lookup check; other bots show "unverified", and "unknown" does not by itself mean a bot is fake. If a row is wrong, report a false positive.
  3. Check search crawlers in the health table: fetches, success rate, 404s and 5xx.
  4. Watch the limit stated on the page: with only the page script you see bots that run JavaScript. Crawlers that do not run it, and response codes, need server collection: send access records (user-agent and URL) to the server events interface with an API key from Configuration › Settings › API keys. That interface keeps only requests it classifies as bots and drops human ones. The log file import under All-platform analytics › Data › Data import is for page views and skips crawlers, so it will not fill the bot report.
  5. Compare Web analytics › Realtime: the online count includes only human events, and the live feed is split into people and bots.

TapCub does not block anything. It measures and separates, and you enforce at the server, CDN or robots.txt.

FAQ

What are the main types of bots?

Search engine crawlers, AI crawlers (training, search, user-triggered), SEO and monitoring tools, link-preview fetchers, scrapers, credential-stuffing bots, spam bots and click-fraud bots. The first five are usually legitimate, the last four are not, and AI crawlers depend on your policy.

Are all bots bad?

No. Imperva's 2025 report puts automated traffic at 51% of web traffic and bad bots at 37%, which leaves about 14 points for automation that is not classed as bad. Search crawlers are the clearest example of a bot you want.

Does robots.txt stop bad bots?

No. RFC 9309 says its rules are not a form of access authorization. Compliant crawlers follow it. For bots that ignore it, use rate limits, login requirements, network blocks or a challenge on forms.

How do I check that a visitor claiming to be Googlebot is real?

Run a reverse DNS lookup on the IP, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP.

Not by itself. OpenAI says GPTBot covers training, while OAI-SearchBot decides whether you appear in ChatGPT search answers. They are controlled separately in robots.txt.

See also: Bot Traffic: How to Detect, Filter and Report It.

Sources

Filed underAnalytics

Was this article helpful?

We use your answers to fix and expand this page.

Su He

Data Analyst

Explores traffic sources, conversion funnels, and retention, using clear metric definitions and concrete examples to explain what the numbers mean.

See which bots hit your site

TapCub separates recognised bots and AI crawlers from human traffic, so your visitors and conversion rate stay clean.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics