TapCub is live — analytics, insights and live chat, free on one platform
Glossary

What is bot filtering? How to separate bots from real visitors

Bot filtering keeps automated traffic out of your human reports. How filters tell bots from people, the cost of filtering too little or too much, and a worked example.

Su He Reviewed by Lucas Hayes Also called bot traffic filtering Last updated Nov 17, 2026 Glossary

Quick answer

Bot filtering is the step in web analytics that detects automated traffic, such as crawlers, monitors and scrapers, and keeps it out of human reports so visitors, sessions and conversion rates describe real people. It works best when bots are split into their own report instead of silently deleted.

What is bot filtering?

Bot filtering is the step in web analytics that detects automated traffic (bot traffic: search crawlers, uptime monitors, scrapers, AI crawlers, spam scripts) and keeps it out of your human reports, so visitors, sessions and conversion rates describe people. It usually combines several signals, because no single check catches every bot. The same words are used in email analytics, where filtering removes machine-generated opens and clicks; this page is about websites and apps.

The useful way to think about it is not "deleting some data" but splitting one stream into two: humans and bots, with a number for each. If you can say "8,200 sessions from people and 1,800 from bots", you can explain every change in either. If a tool silently removes traffic and cannot show what it removed, you cannot audit it.

Why it matters: one week, two sets of numbers

Sample data: a site records 10,000 sessions in a week. Unknown to the team, 1,800 are bots (they load pages, almost never convert and stay for a few seconds) and 8,200 are people.

MetricPeople only (8,200)Bots only (1,800)Everything mixed (10,000)
Conversions2460246
Conversion rate3.00%0.00%2.46%
Bounce rate46% (3,772)95% (1,710)54.8%
Average duration120 s4 s99 s

How to recompute: conversion rate = 246 ÷ 8,200 = 3.00%, but 246 ÷ 10,000 = 2.46%. Bounce rate mixed = (3,772 + 1,710) ÷ 10,000 = 54.8%. Average duration mixed = (8,200 × 120 + 1,800 × 4) ÷ 10,000 = 99.1 s.

Nothing changed in the product, yet conversion looks 18% worse and bounce rate nearly nine points higher. Bots do not distort every metric equally, and the direction depends on their behaviour: scrapers that never run scripts may not appear at all, while headless browsers inflate visitors and bounce rate.

How large is the effect in general? Imperva's 2025 Bad Bot Report put automated traffic at 51% of global web traffic in 2024, with bad bots at 37%. That figure comes from Imperva's own customer and monitoring data and includes search engines and other good bots, so it says nothing about your site. Your share can be 3% or 60%, and the only way to know is to measure it.

How a filter tells a bot from a person

Most filters score a request on several layers and call it a bot when enough of them agree.

LayerWhat it checksCatchesMisses
User agentThe browser string against lists of known crawlersDeclared crawlers: search, AI, SEO tools, monitorsBots that pretend to be Chrome
Request headersHeaders real browsers always sendSimple HTTP clientsTools that copy full browser headers
Client signalsWhether JavaScript ran; automation flags such as navigator.webdriver; timing and screen odditiesHeadless browsersBots driven by real browsers
Network (IP / ASN)Address ranges of cloud providers and known operatorsData-centre trafficBots on residential or mobile networks
FrequencyRequest rates no person producesAggressive scrapersSlow, patient ones

Known-bot lists are the cheapest layer and catch the most declared crawlers. For example, Google Analytics 4 excludes traffic from known bots and spiders using Google's research and the IAB International Spiders and Bots List; its help page says the exclusion cannot be disabled and that you cannot see how much was excluded. Whatever tool you use, ask the same question: can I see what was removed?

Too loose or too tight: the two errors

A filter can fail in two directions, and they cost different things. Continue the sample week: 1,800 bots and 8,200 people. Compare a loose filter with a tight one.

Loose filterTight filter
Bots caught1,000 of 1,8001,780 of 1,800
People wrongly flagged0300 of 8,200 (3.7%)
Reported sessions9,000 (800 bots leak in)7,920 (7,900 people + 20 bots)
Visitors vs truth (8,200)+9.8%−3.4%
Conversion rate reported2.73%2.99%
Share of flagged hits that really are bots (precision)100%85.6%
Share of all bots caught (recall)55.6%98.9%

How to recompute: loose conversion = 246 ÷ 9,000 = 2.73%. Tight: assuming flagged people convert at the average 3%, 9 conversions are lost, so 237 ÷ 7,920 = 2.99%. Precision = 1,780 ÷ (1,780 + 300) = 85.6%; recall = 1,780 ÷ 1,800 = 98.9%.

The decision rule: the loose filter inflates visitors and drags rates down but is easy to explain; the tight filter produces cleaner rates but silently deletes real people, and those people are often the unusual ones (privacy browsers, corporate networks, VPNs). If you must pick, keep bots in a separate report so a wrong call is reversible, and tighten the rules only after you have looked at what they flag.

Drop, separate or ignore: three ways to handle a detected bot

OptionWhat happens to a detected botGood forCost
Do not detectBots stay in the human numbersNothing, except comparing with an older toolInflated visitors and distorted rates
Detect and dropThe hit is not stored anywhereKeeping storage and quota free of junkYou cannot audit or count crawlers later
Detect and report separatelyThe hit goes to its own report; human reports exclude itMost sites: clean human numbers and a record of who visitedA second report to read

Notice what none of them does: filtering does not stop the bot. The request still reaches your server and the page is still served. Blocking, rate limiting and robots.txt are separate controls, covered in how to prevent web scraping.

Common mistakes

  • Treating all bots as bad. Search engine crawlers and AI crawlers are how people find you. Separate them, do not delete them, and check their visits in their own report.
  • Judging by user agent alone. A user agent is a string the client chooses; filters that rely on it catch honest crawlers and miss the rest.
  • Confusing bot traffic with referral spam. Spam that appears as fake referrers in your sources report is a related but different problem; see referral spam.
  • Comparing before and after without a note. When filtering changes, history changes. Annotate the date.
  • Forgetting that a script sees only some bots. A tracking script records only clients that run JavaScript. Crawlers that fetch raw HTML never trigger it, which keeps them out of human numbers but also out of any crawl count.
  • Trusting the trust label. "Unverified" means the operator was not confirmed, not that the traffic is fake.

Bot filtering in TapCub

TapCub identifies bots and counts them separately; it does not block them. Using the sample week above:

  1. Check the mode. In Web analytics › Configuration › Web settings, the Bots & server tab has Bot tracking mode with four options: Do not detect, Detect and drop, Detect and report separately (default), and Separate + server-side collection. In the default mode, recognised bots go only to the bot report and are excluded from human visitors, sessions and pageviews. Only Do not detect mixes them in, and it is not recommended.
  2. Read the split. Open Web analytics › Bots. The All bots tab lists each bot with name, operator, purpose (search, AI training, AI answer engines, monitoring), requests, pages visited, most common status code, last seen, trust level and the detection layers that matched. Put these requests next to your human visitor count to get your own "1,800 vs 8,200".
  3. Audit the rules that fired. Detection runs five layers (user agent, request headers, client signals, network and frequency); a hit is marked as a bot when enough of them agree. Trust is verified, suspected or unknown, and unknown does not mean fake. See how bot detection works.
  4. Report a wrong call. If a real visitor is marked as a bot, use Report false positive on the row; a platform admin reviews it and the rule set is maintained centrally. See report a false positive. You cannot edit the rules yourself.
  5. Know what the script cannot see. With only the page script you see bots that execute JavaScript. Status codes, response times and non-JavaScript crawlers need server-side collection through the server events interface.

TapCub does not block bot requests. To stop a crawler, use robots.txt, your CDN or your firewall; TapCub then shows in Monitoring › Crawler compliance whether crawlers obeyed. See are bots counted in my visitors?.

See also: What is bot mitigation? Signals, responses and the false-positive trade-off.

Sources

Was this explanation clear?

We use your answers to fix and expand this page.

Su He

Data Analyst

Explores traffic sources, conversion funnels, and retention, using clear metric definitions and concrete examples to explain what the numbers mean.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What does bot filtering do in analytics?

It detects automated requests from crawlers, monitors, scrapers and scripts and excludes them from your human reports. Without it, visitors are inflated and rates such as conversion and bounce rate are distorted by traffic that never intended to buy or read.

Is bot filtering the same as blocking bots?

No. Filtering decides what counts in your reports; the bot's request still reaches your server and is answered. Blocking, rate limiting and robots.txt are server-side controls that change what the bot can fetch.

Can bot filtering remove real visitors?

Yes. A tight filter can flag privacy browsers, corporate networks or VPN users as bots. That is why a good setup keeps bots in a separate report, shows which signals matched, and lets you report a wrong call.

How much of my traffic is bots?

It depends on the site. Imperva reported automated traffic at 51% of global web traffic in 2024, but that is an average across its customers. Measure your own share by splitting bot and human traffic and comparing the counts.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics