How do I check crawl health and crawler errors?

Short answer

Open Monitoring › Crawl health. It shows how often search engine and AI crawlers fetched your site, the success rate, distinct pages fetched, and 404 and 5xx errors, split by engine. Add your sitemap URL to see which sitemap pages were never fetched and how long new pages waited for each engine's first visit. For status codes, import server logs; the page script alone cannot see them.

What the report shows

Crawler 404s are separate from the 404s real visitors hit on the Pages report.

Set it up

1. Send server records. Crawl health relies on server-side records: import server logs or send access logs as server events. With only the page script you see crawlers that run JavaScript, with no status codes, and you miss crawlers that skip JavaScript entirely, which includes most AI crawlers.
2. Add your sitemap URL in the crawl health settings to enable the sitemap comparison and time to first fetch.
3. Turn on the crawl anomaly alert in notifications. You will be alerted when one engine's fetches in 24 hours fall below 30% of its 7-day daily average.

The page shows how many records came from the script and how many from the server, so you know which view you are reading.

How to read it

1. Start with the engine table. Is Googlebot's fetch volume stable? A sudden drop often follows a robots.txt change, a server outage or a wave of 5xx errors. See how often Google crawls.
2. Fix 404s that crawlers keep requesting. Prioritize URLs that also appear as visitor 404s and URLs still linked internally or listed in the sitemap. Redirect moved pages with a single 301, avoiding redirect chains.
3. Treat 5xx as urgent. Repeated server errors make crawlers slow down and can drop pages from the index.
4. Check sitemap pages never fetched. These are often orphan pages, blocked paths or pages buried too deep. Improve internal links to them.
5. Watch time to first fetch. If new pages wait weeks, your crawl budget is being spent elsewhere, for example on faceted URLs, parameters or soft 404s.

Crawler compliance

Monitoring › Crawler compliance fetches your robots.txt and compares it with real fetches. It flags three kinds of violation: fetching while blocked globally, fetching a disallowed path, and fetching sooner than the declared crawl delay. A generator drafts rules for "block training crawlers", "block answer engines" or "allow major search engines only" for you to publish. The robots.txt file cannot physically stop a fetch; this page is the evidence of who respects it. See the AI crawler ledger.

FAQ

Why is the success rate almost 100%? Records from the page script have no status code and count as success. After a log import, 4xx and 5xx separate out.

Does adding a crawler to exclusion rules block it? No. Exclusion only stops TapCub from counting it. To refuse it on your server, use the block snippet the compliance page provides.

Is this the same as Google Search Console's crawl stats? It overlaps, but covers every engine and AI crawler in one place and joins fetches with your own visitor data. Search clicks and impressions are on Monitoring › Search performance once Search Console is connected.

Related

• How bot detection works
• Log file analysis for SEO
• Crawl errors