Crawl Errors: Types, Causes and How to Fix Them

Crawl errors are search engine requests that fail. A few are normal; a run of server errors or a large group of 404s on important pages is not. This guide explains the types, the usual causes, a priority order, and a worked week of Googlebot data.

What a crawl error is

A crawl error is a request from a search engine bot that does not return the page it asked for. The bot may get no answer (DNS failure, timeout), a server error (5xx), a "not found" (404), or a redirect that goes nowhere useful.

Some errors are normal. Every site has a few 404s from old links. What matters is the type, the volume, and whether important pages are affected.

Site-level vs URL-level errors

• Site-level errors stop the bot reaching the site at all: DNS failures, the server refusing connections, timeouts, or robots.txt returning an error. They are urgent, because search engines slow down crawling the whole site when they see them. Google treats a 5xx on robots.txt as a reason to pause crawling.
• URL-level errors affect single pages: 404, soft 404, 401/403, redirect chains and loops. They matter most when they hit pages you want indexed.

Main types and fixes

The table below lists the common types, what usually causes them and how to fix them.

Blocked by robots.txt is a choice, not an error; check it only if the page should be indexed.

A worked example: one week of Googlebot fetches

Server-side data for one site over 7 days (sample numbers):

• 18,400 fetches by Googlebot
• 16,900 returned 200 (success rate 91.8%)
• 820 returned a redirect (301/302)
• 540 returned 404
• 140 returned 5xx

Reading it:

• The 140 server errors all fall within a 40-minute window during a deploy. That is a deploy problem, not a capacity problem: deploy without downtime, or return 503 with Retry-After during maintenance.
• Of the 540 404s, 410 come from one old category path removed in a redesign. Those deserve 301 redirects to the new category. The other 130 are scattered, mostly old typo links, and can stay 404.
• The 820 redirects include 300 that pass through two hops (http → https → trailing slash). Update internal links and the sitemap to the final URL.

After the fixes, the next week shows a 96% success rate. More of Googlebot's visits now land on real pages; see crawl budget.

What to fix first

1. Site-level errors (DNS, timeouts, robots.txt failures, a run of 5xx).
2. Errors on important pages: anything in your sitemap or linked from navigation.
3. High-volume 404s from one path pattern: one redirect rule fixes many.
4. Redirect chains and loops in internal links and the sitemap.
5. Soft 404s: empty search or category pages that return 200.
6. Leave scattered 404s for removed content alone; 404 is the correct answer.

Where to find crawl errors

• Google Search Console › Pages and Crawl stats shows what Google reports, with a delay of a few days.
• Server logs show every request as it happens; see log file analysis for SEO.
• A site crawler finds broken links and redirect chains, but it shows what a crawler sees, not what search engines actually requested.

Doing it in TapCub

Monitoring › Crawl health counts search engine fetches from the server side, by engine: total fetches, success rate, distinct pages, 404 and 5xx, plus how long new sitemap URLs wait for a first crawl. A "Crawl anomaly" notification fires when fetches in the last 24 hours fall below 30% of the 7-day average, which often follows a run of server errors. Which bots are real is shown in Web analytics › Bots (verified, suspected, unknown). For how Google's crawler behaves, see Googlebot and how often Google crawls.

See also

• How do I import server logs?