Quick answer
Bot filtering is the step in web analytics that detects automated traffic, such as crawlers, monitors and scrapers, and keeps it out of human reports so visitors, sessions and conversion rates describe real people. It works best when bots are split into their own report instead of silently deleted.
What is bot filtering?
Bot filtering is the step in web analytics that detects automated traffic (bot traffic: search crawlers, uptime monitors, scrapers, AI crawlers, spam scripts) and keeps it out of your human reports, so visitors, sessions and conversion rates describe people. It usually combines several signals, because no single check catches every bot. The same words are used in email analytics, where filtering removes machine-generated opens and clicks; this page is about websites and apps.
The useful way to think about it is not "deleting some data" but splitting one stream into two: humans and bots, with a number for each. If you can say "8,200 sessions from people and 1,800 from bots", you can explain every change in either. If a tool silently removes traffic and cannot show what it removed, you cannot audit it.
Why it matters: one week, two sets of numbers
Sample data: a site records 10,000 sessions in a week. Unknown to the team, 1,800 are bots (they load pages, almost never convert and stay for a few seconds) and 8,200 are people.
| Metric | People only (8,200) | Bots only (1,800) | Everything mixed (10,000) |
|---|---|---|---|
| Conversions | 246 | 0 | 246 |
| Conversion rate | 3.00% | 0.00% | 2.46% |
| Bounce rate | 46% (3,772) | 95% (1,710) | 54.8% |
| Average duration | 120 s | 4 s | 99 s |
How to recompute: conversion rate = 246 ÷ 8,200 = 3.00%, but 246 ÷ 10,000 = 2.46%. Bounce rate mixed = (3,772 + 1,710) ÷ 10,000 = 54.8%. Average duration mixed = (8,200 × 120 + 1,800 × 4) ÷ 10,000 = 99.1 s.
Nothing changed in the product, yet conversion looks 18% worse and bounce rate nearly nine points higher. Bots do not distort every metric equally, and the direction depends on their behaviour: scrapers that never run scripts may not appear at all, while headless browsers inflate visitors and bounce rate.
How large is the effect in general? Imperva's 2025 Bad Bot Report put automated traffic at 51% of global web traffic in 2024, with bad bots at 37%. That figure comes from Imperva's own customer and monitoring data and includes search engines and other good bots, so it says nothing about your site. Your share can be 3% or 60%, and the only way to know is to measure it.
How a filter tells a bot from a person
Most filters score a request on several layers and call it a bot when enough of them agree.
| Layer | What it checks | Catches | Misses |
|---|---|---|---|
| User agent | The browser string against lists of known crawlers | Declared crawlers: search, AI, SEO tools, monitors | Bots that pretend to be Chrome |
| Request headers | Headers real browsers always send | Simple HTTP clients | Tools that copy full browser headers |
| Client signals | Whether JavaScript ran; automation flags such as navigator.webdriver; timing and screen oddities | Headless browsers | Bots driven by real browsers |
| Network (IP / ASN) | Address ranges of cloud providers and known operators | Data-centre traffic | Bots on residential or mobile networks |
| Frequency | Request rates no person produces | Aggressive scrapers | Slow, patient ones |
Known-bot lists are the cheapest layer and catch the most declared crawlers. For example, Google Analytics 4 excludes traffic from known bots and spiders using Google's research and the IAB International Spiders and Bots List; its help page says the exclusion cannot be disabled and that you cannot see how much was excluded. Whatever tool you use, ask the same question: can I see what was removed?
Too loose or too tight: the two errors
A filter can fail in two directions, and they cost different things. Continue the sample week: 1,800 bots and 8,200 people. Compare a loose filter with a tight one.
| Loose filter | Tight filter | |
|---|---|---|
| Bots caught | 1,000 of 1,800 | 1,780 of 1,800 |
| People wrongly flagged | 0 | 300 of 8,200 (3.7%) |
| Reported sessions | 9,000 (800 bots leak in) | 7,920 (7,900 people + 20 bots) |
| Visitors vs truth (8,200) | +9.8% | −3.4% |
| Conversion rate reported | 2.73% | 2.99% |
| Share of flagged hits that really are bots (precision) | 100% | 85.6% |
| Share of all bots caught (recall) | 55.6% | 98.9% |
How to recompute: loose conversion = 246 ÷ 9,000 = 2.73%. Tight: assuming flagged people convert at the average 3%, 9 conversions are lost, so 237 ÷ 7,920 = 2.99%. Precision = 1,780 ÷ (1,780 + 300) = 85.6%; recall = 1,780 ÷ 1,800 = 98.9%.
The decision rule: the loose filter inflates visitors and drags rates down but is easy to explain; the tight filter produces cleaner rates but silently deletes real people, and those people are often the unusual ones (privacy browsers, corporate networks, VPNs). If you must pick, keep bots in a separate report so a wrong call is reversible, and tighten the rules only after you have looked at what they flag.
Drop, separate or ignore: three ways to handle a detected bot
| Option | What happens to a detected bot | Good for | Cost |
|---|---|---|---|
| Do not detect | Bots stay in the human numbers | Nothing, except comparing with an older tool | Inflated visitors and distorted rates |
| Detect and drop | The hit is not stored anywhere | Keeping storage and quota free of junk | You cannot audit or count crawlers later |
| Detect and report separately | The hit goes to its own report; human reports exclude it | Most sites: clean human numbers and a record of who visited | A second report to read |
Notice what none of them does: filtering does not stop the bot. The request still reaches your server and the page is still served. Blocking, rate limiting and robots.txt are separate controls, covered in how to prevent web scraping.
Common mistakes
- Treating all bots as bad. Search engine crawlers and AI crawlers are how people find you. Separate them, do not delete them, and check their visits in their own report.
- Judging by user agent alone. A user agent is a string the client chooses; filters that rely on it catch honest crawlers and miss the rest.
- Confusing bot traffic with referral spam. Spam that appears as fake referrers in your sources report is a related but different problem; see referral spam.
- Comparing before and after without a note. When filtering changes, history changes. Annotate the date.
- Forgetting that a script sees only some bots. A tracking script records only clients that run JavaScript. Crawlers that fetch raw HTML never trigger it, which keeps them out of human numbers but also out of any crawl count.
- Trusting the trust label. "Unverified" means the operator was not confirmed, not that the traffic is fake.
Bot filtering in TapCub
TapCub identifies bots and counts them separately; it does not block them. Using the sample week above:
- Check the mode. In Web analytics › Configuration › Web settings, the Bots & server tab has Bot tracking mode with four options: Do not detect, Detect and drop, Detect and report separately (default), and Separate + server-side collection. In the default mode, recognised bots go only to the bot report and are excluded from human visitors, sessions and pageviews. Only Do not detect mixes them in, and it is not recommended.
- Read the split. Open Web analytics › Bots. The All bots tab lists each bot with name, operator, purpose (search, AI training, AI answer engines, monitoring), requests, pages visited, most common status code, last seen, trust level and the detection layers that matched. Put these requests next to your human visitor count to get your own "1,800 vs 8,200".
- Audit the rules that fired. Detection runs five layers (user agent, request headers, client signals, network and frequency); a hit is marked as a bot when enough of them agree. Trust is verified, suspected or unknown, and unknown does not mean fake. See how bot detection works.
- Report a wrong call. If a real visitor is marked as a bot, use Report false positive on the row; a platform admin reviews it and the rule set is maintained centrally. See report a false positive. You cannot edit the rules yourself.
- Know what the script cannot see. With only the page script you see bots that execute JavaScript. Status codes, response times and non-JavaScript crawlers need server-side collection through the server events interface.
TapCub does not block bot requests. To stop a crawler, use robots.txt, your CDN or your firewall; TapCub then shows in Monitoring › Crawler compliance whether crawlers obeyed. See are bots counted in my visitors?.
See also: What is bot mitigation? Signals, responses and the false-positive trade-off.
Sources
- Google Analytics Help, Known bot-traffic exclusion.
- Imperva, 2025 Bad Bot Report: How AI is Supercharging the Bot Threat.
- Swetrix, What is bot filtering? (definition and the use of several signals).
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Data Analyst
Explores traffic sources, conversion funnels, and retention, using clear metric definitions and concrete examples to explain what the numbers mean.