AI bots · Microsoft Bingbot: the crawler behind Bing and Copilot
Bingbot is Bing’s main crawler, and Bing says Copilot search runs on the same crawling and index. Here is what Microsoft publishes, linked to the source, plus rules and checks to copy.
- Search / AI answers
- Follows robots.txt
- IP list + reverse DNS
Bingbot at a glance
What Microsoft publishes about Bingbot, with a link to the page each fact comes from. Copy the robots.txt rule you need, or check whether an IP really belongs to it.
What the operator says about Bingbot
Each row links to the page it comes from. “Not published” means we could not find it in the operator’s own documentation.
| Fact | Value | Source |
|---|---|---|
| Operator | Microsoft | Source |
| Official documentation | bing.com | Source |
| robots.txt token | Bingbot | Source |
| What it is used for | Search / AI answersMicrosoft says Bingbot is Bing’s standard crawler and handles most of its daily crawling for the search index. | Source |
| Follows robots.txt | YesBing says robots.txt tells its crawlers how to interact with your site; it controls crawling, not indexing. Crawl rate is set in Bing Webmaster Tools. | Source |
| Official IP list | www.bing.com/toolbox/bingbot.json | Source |
| How to verify | IP list or reverse DNSBing says to run a reverse DNS lookup (the name must end in search.msn.com) and a forward lookup back to the same IP, or match the IP against bingbot.json, or use its Verify Bingbot tool. | Source |
| Reverse DNS names | *.search.msn.com | Source |
| Checked on | — |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36As published by the operator. W.X.Y.Z stands for the current Edge version. Bing also lists a short form and a mobile string.
robots.txt rules
Add one of these groups to the end of your robots.txt. The token is matched case-insensitively.
User-agent: Bingbot
Disallow: /User-agent: Bingbot
Allow: /A group named for Bingbot replaces your User-agent: * rules for it. To keep it out of one folder only, change / to that folder, e.g. Disallow: /private/.
Check a request really came from Bingbot
# Step 1 — reverse lookup: the name must end in .search.msn.com
host 203.0.113.7
# Step 2 — forward lookup: the name must resolve back to the same IP
host <hostname-from-step-1>Check an IP against the official list
Open www.bing.com/toolbox/bingbot.json, copy everything and paste it below with the IP from your log. The check runs in your browser.
JSON or one range per line. IPv4 and IPv6 ranges are both read.
Paste an IP address to check it.
Also from the operator’s documentation
- Bing says Bing and Copilot search experiences rely on the same crawling, indexing and ranking foundation as traditional search. Source
- Bing says
NOARCHIVEkeeps content out of Copilot answers and grounding results,NOCACHElimits Copilot to the URL, title and snippet, andNOINDEXremoves a page from Bing, Copilot and grounding. Source - Bing asks you to refresh bingbot.json daily because the list can change at any time. Source
Version v1.0Updated Update history
Four facts that decide what to do
Every crawler page has the same table. These four rows answer most questions about a crawler.
What it is used for
Training, search or a user’s request. Blocking a training crawler keeps your pages out of future models; blocking a search crawler also removes you from answers that would have linked back.
Follows robots.txt
“Yes” is the operator’s own statement. “Partly” usually means fetches a person starts may skip robots.txt — ChatGPT-User, Perplexity-User and meta-externalfetcher say so.
Official IP list / reverse DNS
The only way to tell a real crawler from one that copies its user agent. If neither is published, the user agent is just a claim.
Not published
We could not find it in the operator’s own documentation. We do not fill gaps from third-party lists — those disagree with each other and go stale.
Facts were checked against the operators’ pages on the date shown on each crawler page. Operators change their docs; the source link is the place to confirm.
From a name in your logs to a rule
Four steps, all in the browser.
-
1
Find the crawler
Type the name from your logs or the company you are thinking of. Filter by purpose to see, say, every AI training crawler.
-
2
Open its page
Read what the operator says it is for and whether it follows robots.txt. Copy its official user agent to compare with your logs.
-
3
Build the rules
Tick crawlers in the table, choose Block or Allow, then copy or download the robots.txt groups.
-
4
Check the effect
Test the file with the robots.txt tester, and use the IP check on a crawler page to confirm a request is genuine.
How crawlers are grouped and how robots.txt applies
The directory groups crawlers the way TapCub’s bot rules do. robots.txt matching follows RFC 9309: a crawler obeys the one group whose User-agent matches its token, and only falls back to * when no group names it.
- A crawler with its own group ignores every
User-agent: *rule - Tokens are matched without regard to case; a typo matches nothing
- Some names are tokens only — Google-Extended and Applebot-Extended never visit
- Agents without a published token cannot be addressed in robots.txt at all
| Group | What it does | Effect of blocking |
|---|---|---|
| AI training | Collects pages for model training or datasets | Your future pages stay out of that operator’s training, if it follows robots.txt |
| AI search | Indexes pages for AI answers with links | You stop appearing as a cited source |
| AI agent | Opens a page for a person’s request | Assistants cannot open your page — when the agent honours robots.txt at all |
| Search engine | Classic search index | You leave that search engine; some, like Bing, also feed AI answers |
How four robots.txt files treat Bingbot
Bingbot reads the group whose User-agent line matches Bingbot (case does not matter). If there is none it falls back to User-agent: *.
| robots.txt | Result for Bingbot | Why |
|---|---|---|
User-agent: Bingbot Disallow: / | Blocked | Its own group disallows the whole site. |
User-agent: * Disallow: / | Blocked | No group of its own, so it inherits the wildcard. |
User-agent: * Disallow: / User-agent: Bingbot Allow: / | Allowed | A named group replaces the wildcard completely; the * rules no longer apply to it. |
User-agent: Bingbot Disallow: /private/ | Allowed except /private/ | Only paths that start with /private/ are off limits. |
This is how the rules read under RFC 9309. Whether Bingbot follows them is Microsoft’s statement in the facts table above.
Where AI crawler policies go wrong
Most problems come from the token or from what the crawler actually is.
Using a company name as the token
“OpenAI”, “Anthropic” or “ChatGPT” are not tokens. Use the exact token from the directory — GPTBot, ClaudeBot, OAI-SearchBot.
Blocking search with training
Blocking OAI-SearchBot or PerplexityBot removes you from AI answers that would link back. Decide per crawler, not per company.
Trusting the user agent
Anyone can send “GPTBot” in a header. Without an IP or reverse-DNS check the request is unverified.
Blocking by IP and losing the opt-out
Anthropic and Cohere warn that blocking their IPs stops them reading robots.txt, so the opt-out may not stick.
See which crawlers actually came
The directory tells you what crawlers say they do. TapCub separates humans from bots with five detection layers and 197+ rules, and reports AI crawlers on their own.
Bot and AI crawler report
User agent, headers, client signals, ASN and rate — five layers that keep bots out of your human numbers.
Learn moreAI content ledger
Pages each AI crawler read next to the visitors it sent back.
Learn moreCrawl health and compliance
Compare robots.txt and llms.txt with what actually crawled you — evidence for a decision.
Learn moreBingbot questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
What is Bingbot?
Bingbot is run by Microsoft. Microsoft says Bingbot is Bing’s standard crawler and handles most of its daily crawling for the search index.
Does Bingbot respect robots.txt?
Bing says robots.txt tells its crawlers how to interact with your site; it controls crawling, not indexing. Crawl rate is set in Bing Webmaster Tools.
How do I block Bingbot?
Add User-agent: Bingbot and Disallow: / to robots.txt, in its own group. Microsoft says it follows those rules. To keep it out of one folder only, use Disallow: /folder/ instead.
How can I tell a request really came from Bingbot?
Bing says to run a reverse DNS lookup (the name must end in search.msn.com) and a forward lookup back to the same IP, or match the IP against bingbot.json, or use its Verify Bingbot tool.
How do I keep my pages out of Copilot answers but stay in Bing?
Bing says the NOARCHIVE robots meta tag keeps content out of Copilot responses and grounding results, and NOCACHE limits Copilot to the URL, title and snippet. Blocking Bingbot in robots.txt would stop Bing crawling altogether.
How often should I refresh bingbot.json?
Bing asks for a daily refresh, because the list can change at any time.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Bot and AI crawler rules (same as the TapCub product) — checked 2026-11-10, reviewed every 3 months (next review by 2027-02-10)
- AI crawler operator documentation — checked 2026-11-09, reviewed every 3 months (next review by 2027-02-09)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Rules state intent. Logs show behaviour.
Install the ≈2 KB script and TapCub separates humans, search engines and AI crawlers from the first visit — free plan included.