Free tool Crawler IP range checker: which log addresses are real crawlers?
Search engines and AI companies publish the IP ranges their crawlers use. Paste a list of addresses — or raw lines from an access log — and see which ones sit in an official range, whose it is, and which matching prefix it falls in. 23 lists from 11 operators, 5,966 prefixes, fetched 2026-11-17. Nothing leaves your browser.
- 23 official lists
- Up to 5,000 addresses
- Runs locally, nothing sent
Check addresses against official crawler ranges
One address per line, or paste log lines: the first IP address on each line is used. Duplicates are counted once.
IPv4, IPv6, [2001:db8::1]:443, CSV rows and Nginx / Apache log lines all work. Lines starting with # are skipped.
Addresses
9
Unique, after removing duplicates
On an official list
7
Not on any list
2
Unreadable lines
0
Summary
- OK7 of 9 addresses are on an official crawler list.
- NoteNot on any list: 1. That does not make them fake: most are people, and some operators (Baidu, Yandex) publish no list. Check claimed crawlers one by one with the fake bot checker.
- WarningPrivate or reserved addresses: 1. Real crawlers never come from these; your logs probably record a proxy instead of the client.
By crawler list
One address can be on more than one list (OpenAI uses some ranges for two crawlers).
| Operator | Crawlers | Type | Addresses |
|---|---|---|---|
| Googlebot, Google-InspectionTool, GoogleOther, Storebot-Google | Search engine | 2 | |
| Microsoft | Bingbot | Search engine | 1 |
| Apple | Applebot | Search engine | 1 |
| OpenAI | GPTBot | AI crawler | 1 |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot | AI crawler | 1 |
| Ahrefs | AhrefsBot, AhrefsSiteAudit | SEO tool | 1 |
Addresses
| Address | Result | Operator · crawler | Official prefix | Line |
|---|---|---|---|---|
66.249.66.1 | Official crawler | Google · Googlebot, Google-InspectionTool, GoogleOther, Storebot-Google | 66.249.66.0/27 | 1 |
157.55.39.1 | Official crawler | Microsoft · Bingbot | 157.55.39.0/24 | 2 |
20.171.207.5 | Official crawler | OpenAI · GPTBot | 20.171.207.0/24 | 3 |
216.73.216.12 | Official crawler | Anthropic · ClaudeBot, Claude-User, Claude-SearchBot | 216.73.216.0/22 | 4 |
17.241.208.161 | Official crawler | Apple · Applebot | 17.241.208.160/27 | 5 |
2001:4860:4801:10::1 | Official crawler | Google · Googlebot, Google-InspectionTool, GoogleOther, Storebot-Google | 2001:4860:4801:10::/64 | 6 |
54.36.148.1 | Official crawler | Ahrefs · AhrefsBot, AhrefsSiteAudit | 54.36.148.0/23 | 7 |
203.0.113.77 | Documentation | — | — | 8 |
2a03:2880:f800::1 | Not listed | — | — | 9 |
ip,result,operator,crawler,prefix,source_file
66.249.66.1,official,Google,"Googlebot, Google-InspectionTool, GoogleOther, Storebot-Google",66.249.66.0/27,https://developers.google.com/static/crawling/ipranges/common-crawlers.json
157.55.39.1,official,Microsoft,Bingbot,157.55.39.0/24,https://www.bing.com/toolbox/bingbot.json
20.171.207.5,official,OpenAI,GPTBot,20.171.207.0/24,https://openai.com/gptbot.json
216.73.216.12,official,Anthropic,"ClaudeBot, Claude-User, Claude-SearchBot",216.73.216.0/22,https://claude.com/crawling/bots.json
17.241.208.161,official,Apple,Applebot,17.241.208.160/27,https://search.developer.apple.com/applebot.json
2001:4860:4801:10::1,official,Google,"Googlebot, Google-InspectionTool, GoogleOther, Storebot-Google",2001:4860:4801:10::/64,https://developers.google.com/static/crawling/ipranges/common-crawlers.json
54.36.148.1,official,Ahrefs,"AhrefsBot, AhrefsSiteAudit",54.36.148.0/23,https://api.ahrefs.com/v3/public/crawler-ip-ranges
203.0.113.77,doc,,,,
2a03:2880:f800::1,not_listed,,,,One row per address with the matching list’s file URL, for your firewall or log tooling.
Version v1.0Updated Update history
What a match does and does not mean
An official list proves identity in one direction only.
On a list: it is that crawler
The operator says requests from this range are theirs. Combined with the matching user agent, this is as strong as the DNS check. Allow it if you want to be crawled.
Not on a list: unknown, not fake
Most addresses in a log are people, so “not listed” is normal. It only means fake when the request also claims to be one of the listed crawlers.
Several lists at once
Some ranges serve more than one crawler: OpenAI shares ranges between GPTBot and OAI-SearchBot, and DuckDuckGo publishes one list for both of its bots. All matches are shown.
Private or reserved
An address like 10.0.0.5 or 192.168.1.20 means your server sees a proxy, not the client. Fix the logging (X-Forwarded-For, CF-Connecting-IP) before checking bots.
The lists change every few weeks. This page uses the copy fetched on 2026-11-17; download the operator’s file for the current one.
Three steps
No sign-up. Nothing is uploaded.
-
1
Paste addresses or log lines
Up to 5,000 unique addresses. Raw access log lines work: the first IP on each line is taken.
-
2
Read the summary
How many are official crawlers, from which operators, and which lines could not be read.
-
3
Filter and export
Show only matches or only unlisted addresses, then copy or download the CSV.
Official lists, matched locally
Every operator below publishes a JSON file of CIDR prefixes. TapCub downloads them with a script and ships them as a static file; your browser does the matching.
- Each line is scanned for the first IPv4 or IPv6 address; brackets, ports and zone IDs are removed and IPv4-mapped IPv6 is treated as IPv4
- Each address is tested against every prefix: it matches when its first n bits equal the prefix’s (RFC 4632). Lookups use a sorted table, so 5,000 addresses take a few milliseconds
- When one list has several prefixes containing the address, the most specific one is shown
- Private, loopback, documentation and other special-purpose addresses (RFC 6890) are flagged separately
- The CSV lists the source file of each match so you can audit it
| Operator | Crawlers | Source file |
|---|---|---|
| Googlebot and other common crawlers, special crawlers, user-triggered fetchers and agents | developers.google.com/static/crawling/ipranges/*.json (5 files) | |
| Microsoft | Bingbot | bing.com/toolbox/bingbot.json |
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot | openai.com/gptbot.json and 3 more |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot | claude.com/crawling/bots.json |
| Apple | Applebot | search.developer.apple.com/applebot.json |
| Perplexity | PerplexityBot, Perplexity-User | perplexity.ai/perplexitybot.json, perplexity-user.json |
| DuckDuckGo | DuckDuckBot, DuckAssistBot | duckduckgo.com/duckduckbot.json, duckassistbot.json |
| Amazon | Amazonbot, Amzn-SearchBot, Amzn-User | developer.amazon.com/amazonbot/… (3 pages) |
| Mistral AI | MistralAI-User, MistralAI-Index | mistral.ai/mistralai-*-ips.json |
| Common Crawl | CCBot | index.commoncrawl.org/ccbot.json |
| Ahrefs | AhrefsBot, AhrefsSiteAudit | api.ahrefs.com/v3/public/crawler-ip-ranges |
Six lines from an access log
Click “Try: access log lines” to load them. All six claim to be crawlers in their user agents.
| Address | User agent says | Result |
|---|---|---|
66.249.66.1 | Googlebot | Google common crawlers, 66.249.66.0/27 |
157.55.39.1 | bingbot | Bingbot, 157.55.39.0/24 |
20.171.207.5 | GPTBot | GPTBot, 20.171.207.0/24 |
203.0.113.77 | Googlebot | Documentation address: cannot be a real crawler. In a real log, an address that is not on Google’s list while claiming Googlebot is fake |
216.73.216.12 | ClaudeBot | Anthropic, 216.73.216.0/22 |
2001:4860:4801:10::1 | Googlebot | Google common crawlers, 2001:4860:4801:10::/64 |
The request to /wp-login.php from 203.0.113.77 is the typical pattern: a scanner borrowing Googlebot’s name.
Pitfalls with crawler IP lists
The lists are reliable; how they are used often is not.
Copying the list once
Operators add ranges every few weeks. A firewall allowlist built in spring blocks part of Googlebot by autumn. Automate the download.
Treating “not listed” as a bot
Most unlisted addresses are visitors. Use the list to confirm crawlers, not to accuse everyone else.
Checking the CDN’s addresses
If every address in your log belongs to Cloudflare or your load balancer, nothing will match. Log the original client IP first.
Forgetting user-triggered fetchers
ChatGPT-User, Perplexity-User and Google’s user-triggered fetchers act for a person and may ignore robots.txt. They are on separate lists for a reason.
Bots and AI crawlers, counted for you
Instead of checking logs by hand, see crawler traffic next to your visitors.
Five layers of bot detection
User agent, headers, client signals, network and rate. Bots are kept out of visitor numbers and reported on their own.
Learn moreVerified or suspected
Crawler hits from your server or imported logs are checked with reverse and forward DNS for crawlers that publish verification host names.
Learn moreImport your logs
Server-side events and log imports bring in crawlers that never run JavaScript, so AI crawlers show up too.
Learn moreCrawler IP range questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
Where can I find Googlebot’s IP ranges?
Google publishes them as JSON at developers.google.com/static/crawling/ipranges/: common-crawlers.json (Googlebot and other common crawlers), special-crawlers.json (AdsBot and similar) and three files for user-triggered fetchers and agents. This tool includes all five.
Does OpenAI publish GPTBot’s IP addresses?
Yes: openai.com/gptbot.json for GPTBot, searchbot.json for OAI-SearchBot, chatgpt-user.json for ChatGPT-User and adsbot.json for OAI-AdsBot. OpenAI does not document a reverse DNS check, so the lists are the way to verify its crawlers.
Which crawlers do not publish IP ranges?
Baidu and Yandex document a reverse DNS check instead, and many smaller bots publish nothing at all. Use the fake bot checker for the DNS-verified ones; for the rest, look up the network with the IP lookup tool.
Should I allowlist these ranges in my firewall?
You can, if you refresh them automatically. A stale allowlist blocks new crawler addresses, and an allowlist alone does not stop fakes using other addresses; combine it with user-agent rules.
How often do the lists change?
Each operator updates on its own schedule, and the creation time inside each file shows when it last changed: some files stay the same for years, others change within weeks. This tool shows the date it fetched them (2026-11-17); for a firewall, download the files on a schedule.
Are the addresses I paste uploaded?
No. The lists are downloaded from tapcub.com as a static file and the matching runs in your browser. Nothing you paste is sent or stored.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Official crawler IP ranges (Google, Bing, OpenAI, Anthropic…) — checked 2026-11-17, reviewed every 1 months (next review by 2026-12-17)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
See every crawler without opening a log
TapCub reports search engines and AI crawlers separately from people, with what each one read.