Free tool Server log analyzer: who really requests your pages
JavaScript analytics misses most crawlers, because most of them never run your tracking script. Your access log records every request. Drop it here to see which search engines and AI crawlers fetch what, how often, with which status codes, and whether they touch paths your robots.txt disallows.
- The file never leaves your browser
- 197+ bot rules, same as TapCub
- .log, .txt or .gz, up to 1 GB
Analyze an access log
The report below is from a synthetic sample log (2,400 requests over seven days). Choose or drop your own file to replace it; the format is detected from the first lines.
Your log stays on this device
The file is read and parsed by your browser. Nothing is uploaded to TapCub or anywhere else, and IP addresses are shown only as truncated prefixes. You can disconnect from the internet after the page has loaded and it still works.
robots.txt to check against (optional)
Paste your robots.txt to count crawler requests to disallowed paths. The sample's robots.txt is filled in; it is cleared when you load your own log.
Sample dataSynthetic log generated for this page, with documentation-only IP addresses. Not real traffic.
Sample log (synthetic) · Nginx / Apache combined · 2,400 requests · 2026-11-02 to 2026-11-08
Requests
2,400
People
55%
1,314
Search engines
19%
451
AI crawlers
16%
394
Other bots
10%
241
Share of requests
- People (no bot match) 55%
- Search engines 19%
- AI training crawlers 12%
- AI search crawlers 3.5%
- AI agents (user-triggered) 1.1%
- Other bots 10%
Top bots
Matched by user agent with TapCub's built-in rules, first match wins. IP prefixes are /24 for IPv4 and /48 for IPv6.
| Bot | Type | Requests | Share | IP prefixes | Verify |
|---|---|---|---|---|---|
| GooglebotGoogle | Search engine | 278 | 12% | 192.0.2.0/24 ×260203.0.113.0/24 ×18 | Reverse DNS: googlebot.com, google.com |
| GPTBotOpenAI | AI training | 121 | 5.0% | 192.0.2.0/24 ×121 | Reverse DNS: openai.com |
| BingbotMicrosoft | Search engine | 98 | 4.1% | 192.0.2.0/24 ×98 | Reverse DNS: search.msn.com |
| BytespiderByteDance | AI training | 86 | 3.6% | 198.51.100.0/24 ×86 | No published domain |
| AhrefsBotAhrefs | SEO tool | 78 | 3.3% | 203.0.113.0/24 ×78 | No published domain |
| ClaudeBotAnthropic | AI training | 77 | 3.2% | 198.51.100.0/24 ×77 | No published domain |
| Baiduspider百度 | Search engine | 75 | 3.1% | 192.0.2.0/24 ×75 | Reverse DNS: baidu.com, baidu.jp |
| python-requests | Scripted client | 47 | 2.0% | 203.0.113.0/24 ×47 | No published domain |
| UptimeRobotUptimeRobot | Monitoring | 45 | 1.9% | 2001:db8:77::/48 ×45 | No published domain |
| PerplexityBotPerplexity | AI search | 43 | 1.8% | 198.51.100.0/24 ×43 | No published domain |
| OAI-SearchBotOpenAI | AI search | 40 | 1.7% | 192.0.2.0/24 ×40 | No published domain |
| HeadlessChrome | Headless browser | 30 | 1.3% | 203.0.113.0/24 ×30 | No published domain |
| ChatGPT-UserOpenAI | AI agent | 27 | 1.1% | 198.51.100.0/24 ×27 | No published domain |
| facebookexternalhitMeta | Social preview | 25 | 1.0% | 2001:db8:fb::/48 ×25 | No published domain |
| Empty User-Agent | Scripted client | 16 | 0.7% | 203.0.113.0/24 ×16 | No published domain |
Requests to paths robots.txt disallows
Crawlers only (search engines, AI, SEO and social preview bots), using the group for their robots.txt token or User-agent: *. Longest matching rule wins, as in RFC 9309.
| Bot | Group used | Requests | Top disallowed paths |
|---|---|---|---|
| Bytespider | User-agent: Bytespider | 15 | /pricing ×3 · Disallow: //docs/api ×3 · Disallow: //blog/ ×2 · Disallow: //blog/page/3 ×2 · Disallow: //docs/install ×2 · Disallow: / |
| Googlebot | User-agent: * | 13 | /search?q=a ×7 · Disallow: /search/cart ×3 · Disallow: /cart/search?q=b ×3 · Disallow: /search |
| GPTBot | User-agent: GPTBot | 13 | /blog/ ×4 · Disallow: //pricing ×3 · Disallow: //blog/website-analytics-guide ×2 · Disallow: // ×1 · Disallow: //blog/ai-crawlers-explained ×1 · Disallow: / |
Top paths by visitor type
People (no bot match)
| Path | Requests |
|---|---|
/ | 125 |
/assets/logo.svg | 106 |
/favicon.ico | 102 |
/pricing | 100 |
/assets/main.css | 100 |
/blog/ | 95 |
/blog/ai-crawlers-explained | 93 |
/assets/app.js | 92 |
/blog/website-analytics-guide | 84 |
/about | 83 |
Search engines
| Path | Requests |
|---|---|
/blog/ | 56 |
/ | 41 |
/pricing | 41 |
/robots.txt | 36 |
/docs/install | 33 |
/blog/page/2 | 31 |
/docs/api | 31 |
/blog/website-analytics-guide | 31 |
/features | 30 |
/blog/ai-crawlers-explained | 29 |
AI training crawlers
| Path | Requests |
|---|---|
/robots.txt | 184 |
/pricing | 16 |
/blog/ | 15 |
/docs/api | 14 |
/docs/install | 14 |
/blog/ai-crawlers-explained | 12 |
/ | 10 |
/blog/website-analytics-guide | 8 |
/features | 5 |
/about | 4 |
AI search crawlers
| Path | Requests |
|---|---|
/pricing | 27 |
/ | 24 |
/blog/ai-crawlers-explained | 18 |
/blog/website-analytics-guide | 14 |
AI agents (user-triggered)
| Path | Requests |
|---|---|
/docs/install | 14 |
/pricing | 13 |
Other bots
| Path | Requests |
|---|---|
/ | 104 |
/pricing | 22 |
/blog/ai-crawlers-explained | 19 |
/api/v1/users | 13 |
/.env | 12 |
/docs/install | 12 |
/wp-login.php | 11 |
/xmlrpc.php | 9 |
/docs/api | 8 |
/blog/website-analytics-guide | 8 |
Status codes
| Status | People | Search | AI | Other bots |
|---|---|---|---|---|
| 200 | 838 | 312 | 360 | 129 |
| 206 | 0 | 0 | 0 | 8 |
| 301 | 0 | 82 | 0 | 32 |
| 304 | 386 | 0 | 0 | 0 |
| 400 | 0 | 0 | 0 | 8 |
| 403 | 0 | 9 | 5 | 5 |
| 404 | 90 | 48 | 29 | 59 |
Requests by day
| Day | People | Search | AI | Other bots | Total |
|---|---|---|---|---|---|
| 2026-11-02 | 179 | 63 | 62 | 39 | 343 |
| 2026-11-03 | 183 | 65 | 52 | 43 | 343 |
| 2026-11-04 | 191 | 58 | 60 | 34 | 343 |
| 2026-11-05 | 183 | 67 | 63 | 30 | 343 |
| 2026-11-06 | 195 | 70 | 53 | 25 | 343 |
| 2026-11-07 | 201 | 72 | 44 | 26 | 343 |
| 2026-11-08 | 182 | 56 | 60 | 44 | 342 |
Googlebot. The table shows truncated IP prefixes so you can spot a "crawler" coming from an unexpected network. To confirm one, do a reverse DNS lookup and a forward lookup on the full IP (Google, Bing, Baidu, Yandex, Apple and OpenAI publish the domains, listed under "Verify"), or compare against the operator's published IP ranges. See Google's verification guide.Version v1.0Updated Update history
What the report tells you
In the sample, people made 55% of requests, search engines 19%, AI crawlers 16% and other bots 10%. Here is how to read those numbers for your own site.
Share of requests
Requests, not visits: one page view by a person can pull a dozen files (scripts, styles, images), while a crawler usually fetches just the HTML. So people look bigger here than in page counts. Compare the bot shares with each other and over time, not against your analytics sessions.
Top bots and IP prefixes
A real crawler arrives from its operator's network. In the sample, 260 Googlebot requests came from one prefix and 18 from another, which is the pattern a spoofed user agent leaves. Verify suspicious ones with reverse DNS before blocking anything.
Disallowed paths
Requests a crawler made to paths your robots.txt disallows. A few right after you change robots.txt are normal: RFC 9309 lets crawlers cache the file for up to 24 hours. A steady stream means the crawler ignores it, or the user agent is fake.
Status codes and days
Many 404s for search engines point to broken internal links or stale sitemaps; 301 chains waste crawl requests; 403s to a crawler mean your firewall blocks it, which may or may not be what you intended. A spike on one day often matches a deploy or a new sitemap.
"People" means no bot rule matched. Scripts that send a normal browser user agent end up here too, so treat it as an upper bound. TapCub's product adds header, behaviour, network and rate signals on top of the user agent; this tool uses the user agent only.
Get the log, drop it, read the report
You need read access to the server or CDN logs. Larger sites usually rotate logs daily; a week gives a good picture.
-
1
Find the access log
Nginx:
/var/log/nginx/access.log. Apache:/var/log/apache2/access.logor/var/log/httpd/access_log. IIS:C:\inetpub\logs\LogFiles\W3SVC1\. Cloudflare: files from a Logpush job (HTTP requests dataset). Rotated.gzfiles work as they are. -
2
Drop the file
The format is detected from the first 50 lines and the file is read in chunks with a progress bar, so a few hundred megabytes is fine. Paste your robots.txt in the optional box to check disallowed paths.
-
3
Act on it
Block or rate-limit bots you do not want, fix the 404s search engines hit, and export the CSV to compare with next month.
How lines are read and classified
Each request is matched against TapCub's 197+ built-in bot rules in order, case-insensitively, and the first rule that matches decides the bot and its type. An empty user agent counts as a scripted client. These are the same rules TapCub uses for bot traffic.
- Days use the log's own clock: local server time for Nginx and Apache, UTC for Cloudflare and IIS
- Paths are counted without the query string; the robots.txt check uses the full path and query
- Parsing is done line by line; up to 5,000 distinct paths per group are kept, the rest go under "(other paths)"
| Format | Recognised by | Fields used |
|---|---|---|
| Nginx / Apache combined | IP - - [time] "GET /path HTTP/1.1" status bytes "referer" "user agent" | IP, date, request path, status, user agent |
| Apache common | Same without referer and user agent | IP, date, path, status (no bot split) |
| Cloudflare Logpush | JSON per line with ClientRequestUserAgent or ClientRequestURI | ClientIP, EdgeStartTimestamp, ClientRequestURI, EdgeResponseStatus, ClientRequestUserAgent |
| IIS W3C extended | #Fields: header line | date, c-ip, cs-uri-stem, cs-uri-query, sc-status, cs(User-Agent) |
What the sample log shows
The sample is synthetic: a small site over seven days, with a robots.txt that blocks GPTBot and Bytespider from 2 November. The numbers below come straight from the report above.
| Finding | Number | What it means |
|---|---|---|
| Bots in total | 15 different bots | Search engines, AI crawlers, SEO tools, uptime checks, link previews and scripts |
| AI training crawlers | 12% of requests | Mostly GPTBot, ClaudeBot and Bytespider |
| GPTBot on disallowed paths | 13 of 121 requests | All on 2 November; afterwards it only fetches robots.txt, consistent with the 24-hour cache allowed by RFC 9309 |
| "Googlebot" on disallowed paths | 13 requests | All from the second IP prefix (18 requests), hitting /cart and /search: likely a spoofed user agent |
| Bytespider on disallowed paths | 15 requests | Also only on the first day of the block |
With a real log you would confirm the suspicious Googlebot by reverse DNS on its full IP addresses before blocking. The sample uses documentation-only IP ranges, so no lookup is possible there.
Misreading server logs
Logs are precise about requests and silent about intent.
Trusting the user agent
Blocking or allowing by user agent alone is easy to fool. Verify the big crawlers by DNS or IP range and treat unverified ones as unknown.
Comparing requests with visits
A log counts every file. Bot share by requests is not the bot share of page views or sessions; filter to HTML pages or compare like with like.
Reading only one server
Behind a CDN or load balancer, each origin sees part of the traffic, and cached hits never reach it. Use CDN logs or combine all origins.
Expecting robots.txt to block
robots.txt is a request, not a lock. Well-behaved crawlers follow it; for the rest you need firewall rules, rate limits or authentication.
Bot traffic on its own line, every day
TapCub keeps people and bots on separate lines in every report, so crawlers never inflate your numbers, and shows AI crawlers in their own ledger.
Five detection layers
User agent, headers, client signals, ASN and rate, with 197+ built-in rules and DNS verification for major crawlers.
Learn moreAI crawler ledger
Which AI crawlers read your pages and how many visitors their products send back.
Learn moreLog import
Crawlers that do not run JavaScript reach TapCub through a Nginx / Apache log import or the server-side SDK.
Learn moreServer log analyzer questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
How do I see which bots crawl my website?
Look at the server access log, not your analytics: most crawlers do not run JavaScript, so tag-based analytics misses them (the few that render pages, such as Googlebot, are usually filtered out as bots). Drop the log here and the "Top bots" table lists every recognised crawler with its request count, type and IP prefixes.
How can I tell if AI crawlers are visiting my site?
AI crawlers identify themselves in the user agent: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Bytespider (ByteDance), CCBot (Common Crawl) and others. Some opt-outs, such as Google-Extended, are robots.txt tokens only and never appear in logs. The report splits them into training crawlers, AI search crawlers and user-triggered agents, which matters if you want to block training but stay visible in AI answers.
Is my log file uploaded anywhere?
No. The file is read with your browser's File API and processed on your device; there is no upload step and no server-side processing. Only aggregate numbers appear on screen and in the CSV, and IP addresses are cut to /24 or /48 prefixes.
How big a log file can it handle?
Up to 1 GB per file, read in chunks so memory use depends on the number of distinct bots and paths rather than on file size. A compressed .gz is decompressed on the fly. For very large logs, analysing a few representative days is usually enough.
Can a user agent be faked?
Yes, trivially. That is why the report shows IP prefixes and which crawlers publish a verification domain. Google, Bing, Baidu, Yandex, Apple and OpenAI document reverse DNS checks or IP ranges; a "Googlebot" that fails the check is not Google.
Why do crawlers still request pages I disallowed?
Three common reasons: robots.txt changed recently and the crawler still uses its cached copy (RFC 9309 allows up to 24 hours), the user agent is spoofed, or the crawler does not follow robots.txt. Agents acting for a user, such as ChatGPT-User, may not apply robots.txt rules the same way as crawlers.
Why is the people share so high?
Because requests include every image, script and stylesheet a browser loads, and because anything that sends a normal browser user agent, scrapers included, lands in "People". Use it as an upper bound and look at the bot rows for the details.
Which log formats are supported?
Nginx and Apache in combined or common format, Cloudflare Logpush HTTP request logs as JSON lines, and IIS W3C extended logs with a #Fields header. Custom formats work if each line starts with the standard combined fields.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Bot and AI crawler rules (same as the TapCub product) — checked 2026-11-10, reviewed every 3 months (next review by 2027-02-10)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Keep bots out of your numbers automatically
TapCub separates people from bots on every report, keeps a ledger of AI crawlers and can import your server logs, so this check does not have to be a monthly chore.