Free tool robots.txt generator with AI crawler controls
Start from a preset, switch individual crawlers on or off, add the paths and sitemaps your site needs, and download the file. Every choice comes with a plain note on what it does, and what robots.txt cannot do.
- 43 AI crawlers
- Copy or download
- Runs locally
Choose a policy, then fine-tune it
The default blocks AI training crawlers and keeps search engines, AI search and assistant fetchers. Change anything; the file below updates as you go.
Start from a preset
Presets change the crawler switches or the paths, not your sitemap. “WordPress defaults” adds the two lines WordPress serves out of the box; WordPress 5.5+ also lists /wp-sitemap.xml.
Block these crawlers from the whole site
One per line, such as /admin/ or /*?sessionid=. Paths are case-sensitive.
One per line: paths inside a disallowed folder that should stay crawlable.
Full URLs, one per line. Sitemap lines apply to every crawler.
Google ignores it; Bing and ClaudeBot honour it. Leave empty unless a crawler overloads your server.
# robots.txt generated with tapcub.com/tools/robots-txt-generator
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /search
# AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: Omgili
User-agent: Timpibot
User-agent: AI2Bot
User-agent: img2dataset
User-agent: PanguBot
User-agent: VelenPublicWebCrawler
User-agent: Webzio
User-agent: ImagesiftBot
User-agent: DeepSeekBot
User-agent: SemrushBot-OCOB
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
Upload it as robots.txt to the root of each host name (https://www.example.com/robots.txt). Then paste it into the robots.txt tester with your key URLs to confirm the result.
What this file will and will not do
- NoteGoogle-Extended is a control token, not a crawler: Googlebot does the fetching. Google says it does not affect inclusion or ranking in Google Search.
- NoteDisallow stops crawling, not indexing. If other sites link to a blocked URL, Google can still list it without a description. For pages that must stay out of results, allow crawling and add noindex.
- Noterobots.txt is public and voluntary. Anyone can read the paths you list, and crawlers that ignore it are not stopped. Protect private pages with a login, not with Disallow.
Share these inputs: the values go after the # in the link, so they never reach a server.
https://tapcub.com/tools/robots-txt-generator
Version v1.0Updated Update history
How the generated file is put together
The output always has the same shape, so it is easy to check and to edit later.
The User-agent: * group
Your Allow and Disallow paths, plus Crawl-delay if you set one. Every crawler without a group of its own follows these lines, search engines included. An empty Disallow: means “nothing is blocked”.
One group per purpose
Blocked crawlers are listed together under a comment for their purpose, each on its own User-agent line, followed by Disallow: /. Several User-agent lines sharing one set of rules is standard (RFC 9309) and keeps the file short.
Allow lines first
Exceptions are written before the paths they carve out of. Google and Bing pick the longest rule so order does not matter to them, but Baidu documents first-match, and this order gives the same answer both ways.
Sitemap at the end
Sitemap lines are independent of groups and apply to all crawlers. List each sitemap or sitemap index once, as a full URL.
A crawler with its own group ignores the * group. That is why blocked crawlers simply get Disallow: /: nothing in * needs to be repeated for them.
Four steps
About two minutes, including the test.
-
1
Pick a preset
Most content sites want “Block AI training, allow AI search”. Shops and SaaS sites that want AI assistants to recommend them should keep AI search and assistant fetchers allowed.
-
2
Adjust crawlers and paths
Switch individual crawlers, add folders you never want crawled (admin, cart, internal search, URL parameters that create duplicates) and the exceptions inside them.
-
3
Add your sitemap
Paste the full sitemap URL. Set Crawl-delay only if a crawler that honours it is putting load on your server.
-
4
Download, upload, test
Save the file as robots.txt in your site root, open it in a browser to confirm, then run your key URLs through the robots.txt tester.
A request to crawlers, not access control
robots.txt asks automated clients to stay away from parts of a site. Well-known crawlers honour it; nothing enforces it.
- It can: tell compliant crawlers which paths to skip, opt out of AI training by a named crawler, and point every crawler to your sitemap
- It cannot: hide a URL (the file is public), keep a page out of search results (use noindex), or stop scrapers that ignore it (block them at the server or CDN)
- User-triggered fetchers can differ: OpenAI, Perplexity and Meta say robots.txt may not apply to fetches a person requests; Anthropic says Claude-User follows it
- Changes are not instant: Google generally caches robots.txt for up to 24 hours; OpenAI says its search systems need about 24 hours to adjust
| Crawler type | Examples | Effect of Disallow: / |
|---|---|---|
| AI training | GPTBot, ClaudeBot, Google-Extended, CCBot | Content not collected for future model training; no effect on search ranking |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Pages not indexed for that AI search, so not cited there |
| AI assistant fetch | ChatGPT-User, Claude-User, Perplexity-User | Depends on the operator: some say robots.txt may not apply |
| Search engine | Googlebot, Bingbot, Baiduspider | Pages not crawled; URLs can still be listed without a snippet |
| SEO tool | AhrefsBot, MJ12bot, DotBot | Site missing from that tool’s backlink and audit data |
Three sites, three policies
The same generator, three different goals. Each row is a preset plus a small edit.
| Site | Choice | What gets written | Why |
|---|---|---|---|
| News publisher | Block AI training, allow AI search | One group of training crawlers with Disallow: /; * blocks /search | Keeps articles out of training sets while staying citable in AI answers |
| Online shop | Allow everything + paths | Disallow: /cart, /checkout/, /*?sort=; sitemap | Wants every assistant to recommend products; hides pages with no search value |
| WordPress blog | WordPress defaults + block all AI | Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, AI groups | Front-end features that call admin-ajax.php keep working for crawlers |
Whatever you choose, keep CSS, JavaScript and image folders crawlable, or search engines cannot render your pages.
What goes wrong with generated files
The file is only half the job; where and how it is served matters as much.
Shipping the staging file
A Disallow: / copied from staging to production quietly removes a site from crawling. Check the live file after every deploy.
Blocking AI search by accident
“Block all AI” also blocks OAI-SearchBot, Claude-SearchBot and PerplexityBot, so those answer engines stop citing you. Decide deliberately.
Listing secret paths
Every line is public. Disallow: /secret-admin-2025/ advertises the URL. Protect it with a login instead.
Uploading the wrong file type
It must be plain text named exactly robots.txt at the host root and return status 200, not an HTML error page.
Check whether the crawlers listen
A robots.txt is a request. TapCub’s bot report shows which crawlers visit, how often and which pages they read, separately from human traffic.
Bots and AI crawlers
Search engines, AI crawlers and scrapers identified by five detection layers and 197+ rules, kept out of your human numbers.
Learn moreCrawl compliance
Your robots.txt and llms.txt are fetched daily and compared with what actually crawled you.
Learn moreRealtime visitors
See who is on the site right now, with bots on their own line.
Learn morerobots.txt generator questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
How do I block AI crawlers with robots.txt?
Add a group for each crawler token with Disallow: /, for example User-agent: GPTBot. The “Block AI training” preset writes this for every training crawler in the list; “Block all AI crawlers” adds AI search and assistant fetchers too. Each operator documents its own tokens, so blocking GPTBot does not block OAI-SearchBot.
Will blocking GPTBot or Google-Extended hurt my SEO?
Not in Google Search. Google says Google-Extended does not affect inclusion or ranking, and GPTBot is OpenAI’s training crawler. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot does remove you from those answer engines.
Do AI assistants obey robots.txt?
It depends on the operator. Anthropic says Claude-User follows robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them, because a person asked for the page. To stop those, block their user agents at your server or CDN.
Where do I put robots.txt?
At the root of each host name: https://www.example.com/robots.txt. A subdomain needs its own file. On WordPress, a real robots.txt file in the web root replaces the virtual one WordPress generates.
Should I use Crawl-delay?
Usually not. Google ignores it, Yandex stopped supporting it in 2018 and Baidu does not document it. Bing and ClaudeBot honour it. For Google, crawl rate adjusts to how your server responds.
Is an empty robots.txt the same as none?
In effect, yes: both allow everything. Serving a small file is still worth it: it stops 404s in your logs and tells crawlers where your sitemap is.
Can robots.txt protect private pages?
No. The file is public and compliance is voluntary. Use authentication for private content and noindex for pages that should not appear in search.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Data this tool relies on
- Bot and AI crawler rules (same as the TapCub product) — checked 2026-11-10, reviewed every 3 months (next review by 2027-02-10)
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Measure the crawlers you just ruled on
TapCub’s bot report lists every search engine and AI crawler that loads your pages, so you can see who honours the file.