Web crawler: how it works, types of crawlers and how to verify them

Crawlers decide whether your pages get indexed, used by AI models or cited in AI answers, and some of them are not who they say they are. This guide explains how crawlers work, the types you will see in your logs, and how to verify and control them, with a worked week of real-looking log data.

What is a web crawler?

A web crawler (also called a spider or bot) is a program that visits web pages automatically, downloads them, extracts the links they contain and follows those links to find more pages. Search engines use crawlers to build their index; AI companies use them to collect training data and to fetch pages for answers; SEO tools, uptime monitors and price scrapers use them too.

On a typical website, automated requests are a large share of all traffic. Knowing which crawlers visit, what they fetch and whether they are who they claim to be is part of both SEO and security.

How a web crawler works

1. Seed URLs. The crawler starts from known URLs: previously crawled pages, sitemaps, links submitted by site owners.
2. Crawl frontier. URLs waiting to be fetched sit in a queue, prioritized by importance, freshness and how often the page changes.
3. Robots check. Before fetching, a well-behaved crawler reads /robots.txt for the host and skips disallowed paths (the Robots Exclusion Protocol, standardized as RFC 9309).
4. Fetch. It requests the page with its own user-agent string, respecting rate limits so the server is not overloaded.
5. Parse and render. It extracts text, links and metadata. Some crawlers, such as Googlebot, also render JavaScript in a later pass; many AI crawlers do not.
6. Store and schedule. Content goes to an index or dataset; new links join the frontier; the page is scheduled for a revisit.

Types of web crawlers

The difference between AI training and AI search crawlers matters for decisions: blocking a training crawler keeps your content out of a model's training data; blocking a search or user-triggered crawler removes you from that assistant's answers and the visits they send.

Decide per user-agent token in robots.txt.

Worked example: seven days of crawler traffic

A content site analyzed 7 days of server logs, 1,420,000 requests in total (sample figures):

What the logs showed:

• Not every "Googlebot" was Google. 21,400 requests carried a Googlebot user agent; 3,900 of them (18%) came from IPs that failed reverse DNS verification. Those were scrapers using a trusted name to avoid blocks.
• AI crawlers outnumbered search crawlers. AI bots made 1.6 times as many requests as search engines, but most of their fetches went to old archive pages.
• One SEO tool was crawling faceted URLs. 41% of SEO-tool requests hit filter combinations like ?color=red&size=m, which the site does not want indexed anyway.
• Errors hid in bot traffic. Search crawlers got 5xx errors on 1.2% of requests during a nightly backup window; humans rarely visited at that hour, so nobody had noticed.

Actions: block the spoofed Googlebot IPs, add a Disallow for faceted parameters, move the backup out of peak crawl hours, and keep AI search bots allowed while reviewing the training bots.

How to identify and verify crawlers

• User agent: the first clue, but trivially faked.
• Reverse DNS: look up the requesting IP, check the hostname ends in the operator's domain (for Google, googlebot.com, google.com or googleusercontent.com), then do a forward lookup to confirm it resolves back to the same IP.
• Published IP ranges: Google, Bing, OpenAI and others publish JSON lists of crawler IPs. Match against them.
• Behavior: request rate, whether it fetches robots.txt, whether it runs JavaScript, which paths it requests.

How to control crawlers

1. robots.txt to allow or disallow paths per user-agent token. It is a request, not access control: well-behaved crawlers obey, scrapers do not.
2. Meta robots and X-Robots-Tag to keep a crawled page out of the index (noindex) or stop links being followed.
3. Sitemaps to point crawlers at the URLs you want found, with last-modified dates.
4. Rate limiting and WAF rules for crawlers that ignore robots.txt or overload the server; block by verified identity, not user agent alone.
5. Server-side rendering for content you want AI crawlers to read, since many do not execute JavaScript.

Seeing crawlers in TapCub

Web analytics › Bots separates recognized bots from human traffic by default. All bots lists each crawler's name, operator, purpose (search, AI training, AI answer engine, monitoring), requests, pages, most common status code, trust level (verified, suspected or unknown) and the detection layers it matched (How bot detection works). Monitoring › Crawl health checks success rates and whether fetches followed your robots.txt (Check crawl health).

With only the page script, you see only crawlers that execute JavaScript. Most search and AI crawlers fetch HTML without running scripts, so add server collection or import server logs for full coverage. For specific crawlers, see Googlebot and GPTBot; for using logs in SEO, Log file analysis.

Sources

• IETF, RFC 9309: Robots Exclusion Protocol (2022).
• Google Search Central, Verifying Googlebot and other Google crawlers.
• Google Search Central, Overview of Google crawlers and fetchers.