Quick answer
Bot mitigation is identifying automated traffic and applying a policy to it: allow, count separately, slow down, challenge or block. The aim is to reduce harm (fraud, scraping, load, skewed numbers), not to reach zero bots, because every strict rule also stops some real people.
What is bot mitigation?
Bot mitigation is the work of recognizing automated traffic and applying a policy to it. The policy can be to allow it, count it separately, slow it down, challenge it or block it. The goal is to reduce harm, not to reach zero: scraped content, credential-stuffing attempts, fake signups, wasted server capacity and polluted analytics. Every stricter rule also stops some real people and some useful crawlers, so mitigation is a trade-off you tune, not a switch you turn on.
OWASP's Automated Threats to Web Applications project catalogs the harm in named categories such as credential stuffing, scraping, scalping and denial of inventory. Vendors often use "bot management" and "bot mitigation" interchangeably; this page uses mitigation for the act of limiting harmful automation and leaves product comparison out. For what a bot is, see bot; for how networks of compromised devices run attacks, see botnet.
Which bots are good, which are bad?
Not all automated traffic is unwanted, and the policy depends on the kind.
| Kind | Examples | Usual policy |
|---|---|---|
| Good (wanted) | Search engine crawlers, uptime monitors, feed readers | Allow; verify they are who they say |
| Neutral (depends on you) | SEO tool crawlers, AI crawlers and answer engines | Decide per operator; control with robots.txt |
| Bad (harmful) | Credential stuffing, scrapers that ignore rules, fake-signup scripts, inventory hoarding | Limit, challenge or block |
The neutral row is where most real decisions sit. A crawler that obeys robots.txt (RFC 9309) can be steered with rules; one that ignores them needs enforcement at the server or network edge.
How are bots detected?
No single signal is reliable, so detection combines several. Reliability below is how hard the signal is for a determined bot to fake.
| Signal | What it checks | Reliability | Typical failure |
|---|---|---|---|
| User-Agent string | The name the client declares | Low: any client can claim any name | Scrapers pretend to be a browser |
| Network address | Whether the address belongs to the operator it claims | High for crawlers that publish verification | Residential proxies hide scrapers |
| Reverse and forward DNS | The address resolves to the operator's domain and back | High (Google documents this for Googlebot) | Not available for most bots |
| JavaScript execution | Whether the page's script runs | Medium: simple bots do not run it, headless browsers do | Modern bots render pages |
| Request rate and pattern | Requests per second per address or session | Medium | Slow, distributed bots stay under limits |
| Behavior | Timing, movement, order of pages | Medium to high | Needs history; can misread assistive tools |
| robots.txt compliance | Whether disallowed paths are fetched | Only for crawlers that read it | Bad bots do not read it |
Practical rule: trust a claim only when it can be verified. A request that says "Googlebot" but comes from an address that does not reverse-resolve to Google is not Googlebot, whatever the User-Agent says.
What happens after detection?
| Response | What it does | Cost to real users | Use for |
|---|---|---|---|
| Allow | Lets the request through | None | Verified good bots |
| Count separately | Keeps the request but removes it from human reports | None | Analytics hygiene; neutral bots |
| Rate limit | Caps requests per address or key | Low, but shared addresses can be hit | Scrapers, login endpoints |
| Challenge | Asks for proof (script check, CAPTCHA-style test) | Medium: friction | Suspicious but uncertain traffic |
| Block | Refuses the request | High if wrong | Clear abuse |
| Serve decoys or slow down | Wastes the bot's time | Low | Persistent scrapers |
A sound policy escalates: count first, limit next, challenge uncertain cases, block only what is clearly abusive.
How strict should mitigation be? A worked example
Sample data: a site gets 100,000 requests a day: 88,000 from people, 3,000 from good bots and 9,000 from bad bots. Three rule sets are tried against the 9,000 bad-bot requests. A real visitor converts 3% of the time and a conversion is worth $40.
| Setting | Bad requests stopped | Bad requests left | Real visitors wrongly stopped | False-positive rate | Lost conversions | Lost revenue |
|---|---|---|---|---|---|---|
| Lenient | 5,400 (60%) | 3,600 | 40 | 0.05% | 1.2 | $48 |
| Moderate | 7,200 (80%) | 1,800 | 180 | 0.20% | 5.4 | $216 |
| Strict | 8,100 (90%) | 900 | 450 | 0.51% | 13.5 | $540 |
Recomputation: 450 ÷ 88,000 = 0.51%; 450 × 3% = 13.5 conversions; 13.5 × $40 = $540. Of the 8,550 requests the strict rule stopped, 8,100 were truly bad (94.7%).
The decision rule: going from moderate to strict stops 900 more bad requests (8,100 − 7,200) and wrongly stops 270 more people (450 − 180). The extra lost revenue is 270 × 3% × $40 = $324. Strict is worth it only if each bad request costs you more than $324 ÷ 900 = $0.36. For scraping that costs you almost nothing per request, moderate is the better setting; for fraud that costs $5 an attempt, strict is.
How do bots distort your analytics?
Bots do not only cost money on the server; they change your numbers. Sample data: a site records 20,000 sessions and 400 conversions, so conversion looks like 400 ÷ 20,000 = 2.00%. If 3,000 of those sessions are bots (they never convert), the human rate is 400 ÷ 17,000 = 2.35%, 17.6% higher than reported. Bots also inflate bounce rate, lower average time on page and send false traffic to the pages they scrape. Keeping bots out of the denominator is the first mitigation an analyst controls; see bot filtering.
Common mistakes
- Aiming for zero bots: the last 10% of detection costs the most false positives.
- Trusting the User-Agent string: it is a claim, not proof.
- Blocking search engine crawlers by accident, then wondering why pages drop out of the index.
- Applying one rule to every endpoint: a login form and a blog post deserve different limits.
- Never measuring false positives: without a human-conversion check, strictness only ever goes up.
- Treating analytics filtering as protection: removing bots from a report does not stop them using your server.
Bot mitigation and TapCub
TapCub is not a bot protection product. It does not block, challenge or rate-limit anything; it measures, so your numbers are not polluted. For the example above:
- Count separately by default. New projects use Detect and report separately: recognized bots appear only in Web analytics › Bots and are left out of the human overview, so the 2.35% in the example is what you see, not the 2.00%.
- Read who the bots are. The All bots tab lists name, operator, purpose (search, AI training, answer engines, monitoring), requests, pages visited, trust level and which detection layers matched. AI & agents shows the AI content ledger: reads per operator against visitors sent back.
- Understand trust. Trust is verified, suspected or unknown; unknown does not by itself mean spoofed. If a row is wrong, report a false positive.
- Know the limit. With only the page script, you see only bots that run JavaScript. For status codes and crawlers that do not run it, use server collection or a log import.
- Enforce elsewhere. Rate limits, challenges and blocks belong in your CDN, firewall or web server; use the Bots report to decide which operators to target and to check the effect afterwards.
See also: Brute Force Attack: How It Works and How to Prevent It.
See also: Credential Harvesting: What It Is, Methods and Defences.
Sources
- OWASP, Automated Threats to Web Applications (project page and handbook of 21 threat types).
- IETF, RFC 9309: Robots Exclusion Protocol (2022).
- Google Search Central, Verifying Googlebot and other Google crawlers.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Data Analyst
Explores traffic sources, conversion funnels, and retention, using clear metric definitions and concrete examples to explain what the numbers mean.