Free marketing tools AI crawlers and AI visibility
Tools for deciding which AI crawlers may read your site, and for checking what the rules you wrote actually do. The crawler list is the one TapCub uses to detect bots.
robots.txt, llms.txt and the bots that read your site for AI.
AI crawler robots.txt checker
Paste your robots.txt and see which AI crawlers are allowed, partly restricted or blocked — then generate the rules you want.
- 43 AI crawler user agents, same list as the TapCub bot rules
- Shows the exact rule that decides each one
- Generates a copy-ready robots.txt snippet
llms.txt generator
Write an llms.txt in the llmstxt.org format: site name, summary and sections of links, or start from your sitemap.xml.
- Follows the llmstxt.org proposal exactly
- Paste “Title | URL | note” lines or a sitemap
- Flags missing summary, bad URLs and duplicates
llms.txt validator
Paste an llms.txt and get every problem with its line number, checked against the llmstxt.org format.
- H1, summary, sections and link syntax
- Relative, duplicate and malformed links
- Section-by-section link counts
robots.txt tester
Paste a robots.txt and a list of URLs to see which crawlers may fetch each one, and the exact rule that decides it.
- Follows RFC 9309: groups, longest match, * and $
- Googlebot, Bingbot, Baiduspider and {ai} AI crawlers
- Flags typos, unsupported lines and bad sitemaps
robots.txt generator
Build a robots.txt from presets and per-crawler switches: block AI training, keep AI search, add paths and sitemaps.
- Presets for common policies and WordPress
- 43 AI crawlers grouped by what they do
- Explains what each choice will and will not do
User agent parser
Paste one or many user-agent strings and see browser, engine, OS, device, in-app browser and whether it is a known bot or AI crawler.
- Batch mode: one UA per line, results as a table
- WeChat, DingTalk, Alipay, Douyin, Facebook and other in-app browsers
- Bot check with the same 197 rules TapCub uses
What is my user agent
See your browser's user-agent string, User-Agent Client Hints, language, time zone and screen — and exactly which of these an analytics script like TapCub sends and stores.
- Your UA parsed into browser, engine, OS and device
- Client Hints, including the real Windows 11 / Android version
- What a script can see vs what TapCub keeps
AI bots directory
Every AI crawler TapCub detects — training, AI search and user-triggered agents — with official facts and copy-ready robots.txt rules.
- Searchable by name, operator or token
- One page per major crawler, every fact sourced
- robots.txt block / allow rules to copy
AI referral traffic regex
Build a compact RE2 pattern that matches visits from ChatGPT, Perplexity, Claude, Gemini, Copilot, DeepSeek and other AI assistants, with GA4 setup steps and a test box.
- Pick assistants or add your own domains
- Session source or full page referrer, full or partial match
- Test referrer URLs against the pattern
Server log analyzer
Drop an access log and see how much of your traffic is people, search engines, AI crawlers and other bots. Parsed in your browser.
- Nginx, Apache, Cloudflare Logpush, IIS
- Same 197+ bot rules as TapCub
- robots.txt check and CSV export
Questions about the tools
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
Do I need a TapCub account to use them?
No. The tools are public pages. An account only matters when you want the same numbers computed from your live traffic instead of typed in.
Is anything I type sent to TapCub?
No. Almost every tool runs entirely in your browser. A few lookup tools (DNS, WHOIS, page speed) send only the domain or URL you enter, from your browser straight to a public service such as Cloudflare DNS, the domain registry or Google PageSpeed, and only when you press the button; each of those pages says where the request goes. We record that a tool page was viewed and used, like any other page — never the values.
Are the tools kept up to date?
Yes. Every tool page lists its version, an update history and the date each data source was last checked. Data such as crawler rules, public holidays and platform limits is re-checked on a schedule, and all tool updates are listed in one place.
Can I embed or link to a tool?
Linking is welcome. The pages have stable URLs under /tools. We do not offer an embed, but the formulas are documented on each page so you can reproduce them.
Why do my numbers differ from my analytics tool?
Usually the definition, not the arithmetic: who counts as a visitor, which event converts, and whether bots were removed first. Each tool lists the definition it assumes. See why UV never matches.
Will there be more tools?
When a question comes up often in support we turn it into a page. Suggestions go to contact.