Bingbot: What It Is, How to Verify It and Control Its Crawling

Bingbot crawls the web for Bing, and Bing’s index also feeds other products, so it is often the second most important crawler on a site. This guide covers its user agents, how to verify it, a worked week comparing it with Googlebot, crawl rate control and IndexNow.

What Bingbot is

Bingbot is the web crawler Microsoft uses to discover and refresh pages for the Bing search index. It matters beyond Bing itself: Bing's index also serves other Microsoft products such as Copilot, and other search engines license Bing results.

For many sites Bingbot is the second-busiest search crawler after Googlebot.

Bing's crawlers and user agents

The table below lists the main crawlers Bing documents. The user agent of the standard Bingbot contains bingbot/2.0, for example:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36

The Chrome version in the string changes as Bing updates its rendering engine.

Verify all Microsoft crawlers the same way: reverse DNS to search.msn.com.

How to verify real Bingbot

Anyone can put "bingbot" in a user agent. To verify a request:

1. Take the IP address and do a reverse DNS lookup. Real Bingbot resolves to a host ending in .search.msn.com.
2. Do a forward DNS lookup on that host name and check it returns the same IP.
3. Or paste the IP into Bing's Verify Bingbot tool in Bing Webmaster Tools.

Requests that claim to be Bingbot but fail this check are fake: often scrapers hoping you will not block a search engine.

A worked week: Bingbot next to Googlebot

Server-side crawl data for one site over 7 days (sample numbers):

• Googlebot: 18,400 fetches
• Bingbot (verified): 6,200 fetches, 97% returned 200
• "Bingbot" that failed verification: 900 requests, mostly to search and product listing URLs, from hosting provider IPs
• New sitemap URLs: median wait for a first fetch, 2 days for Google and 5 days for Bing

What to do:

• Block or rate-limit the 900 fake requests at the firewall by IP; do not block the Bingbot user agent.
• To speed up Bing discovery, submit new URLs with IndexNow (below) and keep the sitemap's lastmod accurate.

Controlling Bingbot's crawling

• robots.txt: Bingbot follows Disallow rules for User-agent: bingbot or *. Bing also honours Crawl-delay, unlike Google.
• Crawl control in Bing Webmaster Tools: set crawl rate by hour of day, for example less during your busiest hours.
• Server errors slow it down. Like Google, Bing reduces crawling when it sees many 5xx or timeouts; see crawl errors.
• Do not block it to save load. Blocking Bingbot removes you from Bing and the products that use its index.

IndexNow

IndexNow is an open protocol, supported by Bing and some other search engines, that lets you notify them when a URL is added, updated or deleted. You host a key file on your site and send the changed URLs to an IndexNow endpoint. Many CMS and CDN platforms offer it as a setting. It does not replace a sitemap, but it shortens the wait for new and updated pages.

Seeing Bingbot in TapCub

• Web analytics › Bots separates verified, suspected and unknown bots, and excludes bots from human data by default, so fake Bingbot traffic does not inflate visitors.
• Monitoring › Crawl health shows fetches, success rate, distinct pages, 404 and 5xx by engine, plus how long new sitemap URLs waited for each engine's first fetch.
• Monitoring › Search performance connects Bing Webmaster for clicks, impressions, CTR and position next to Google.

Related: Googlebot, crawl budget and how often Google crawls.