What is the X-Robots-Tag? Syntax, examples and when to use it

What is the X-Robots-Tag?

X-Robots-Tag is an HTTP response header that gives crawlers indexing instructions for the URL being served:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

It does the same job as the <meta name="robots"> tag, but because it is a header it works for any file type (PDFs, images, videos, feeds) and can be applied to whole folders or file types in server configuration.

Common directives

Directives can target one crawler: X-Robots-Tag: googlebot: noindex.

X-Robots-Tag vs meta robots vs robots.txt

The classic trap: blocking a URL in robots.txt and adding noindex. The crawler is not allowed to fetch the URL, so it never sees the noindex, and the URL can still show up in results (usually without a snippet). To remove a URL from search, allow crawling and send noindex.

Configuration examples

Nginx, keep all PDFs out of search:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow" always;
}

Apache (.htaccess), same rule:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Cloudflare Worker, noindex a staging host:

const res = await fetch(request);
const out = new Response(res.body, res);
if (new URL(request.url).hostname.startsWith('staging.')) {
  out.headers.set('X-Robots-Tag', 'noindex');
}
return out;

Verifying it

curl -sI https://example.com/files/report.pdf | grep -i x-robots-tag

In Google Search Console, the URL Inspection tool reports "Excluded by 'noindex' tag" once the page has been recrawled.

AI crawlers

Google's crawlers honour X-Robots-Tag; Google-Extended (training for Gemini) is controlled in robots.txt rather than with a header. Other AI crawlers mostly document robots.txt rules only, so use robots.txt or a server-side block for them.

What TapCub can check

TapCub's crawler compliance page (Monitoring › Crawler compliance) fetches your robots rules file and compares it with real crawler fetches, listing crawlers that fetched blocked paths or ignored the declared delay. It does not inspect response headers, so it cannot confirm that an X-Robots-Tag was sent; use curl or Search Console for that. Crawl health shows which pages each search engine fetched and their status codes once server logs are imported. See also robots.txt disallow all.

See also

• Google-Extended: What It Controls and How to Use It in robots.txt