Try it on your own data
Every report in this article is in the free plan. One snippet for web and apps, up to 10 sites.
Start freeApplebot is Apple's search crawler for Spotlight, Siri and Safari; Applebot-Extended is only a robots.txt switch for AI training. Block the wrong token and you switch off Apple search too. This guide covers both agents, three rule choices, verification, and a worked cost example.
What you will learn
- Applebot crawls for Spotlight, Siri and Safari search; Applebot-Extended never crawls — it only decides whether Applebot-collected data may train Apple's AI models
- Disallow Applebot-Extended to opt out of training and keep Apple search; disallowing Applebot itself puts the visits it sends back at risk
- Verify with reverse DNS to *.applebot.apple.com plus a forward check, or Apple's CIDR JSON; watch for the Googlebot-fallback rule that can block Applebot silently
What Applebot is
Applebot is Apple's web crawler. Its data powers search across Spotlight, Siri and Safari, and Apple says the same crawled data may also help train Apple foundation models (Apple Intelligence, Services, Developer Tools) and provide context for AI-generated answers (About Applebot, published 2026-09-04, checked 2026-11-15).
The name covers two things. Applebot is the crawler in your logs. Applebot-Extended is not a crawler: it is a robots.txt token deciding whether Applebot-collected data may train AI. Disallowing Applebot itself turns off the crawl that feeds Apple search; the training-only switch costs none of those visits.
Apple's two agents
From Apple's documentation:
- Applebot (Search) crawls and may render pages for Spotlight, Siri and Safari search. Enabling it in robots.txt lets content appear in those results.
- Applebot-Extended does not crawl webpages. It only determines how Applebot-collected data may be used — specifically, whether it may train Apple's generative foundation models. Pages that disallow Extended can still appear in search results, and Apple states Extended rules are not considered in ranking for Search.
- iTMS (Apple Podcasts) fetches only URLs registered on Apple Podcasts and does not follow robots.txt.
Google uses the same token-only pattern with Google-Extended: a robots.txt name governing data use, not a crawl.
| Agent | What it does | robots.txt effect |
|---|---|---|
| Applebot | Crawls (and may render) for Spotlight, Siri and Safari search; collected data may also train Apple foundation models and power AI answers | Disallow stops the crawl -> content loses Apple search visibility |
| Applebot-Extended | Does not crawl; token deciding whether Applebot-collected data may train Apple foundation models | Disallow = training opt-out; search stays; not a Search ranking factor |
| iTMS (Apple Podcasts) | Fetches only URLs registered on Apple Podcasts | Does not follow robots.txt |
Source: Apple "About Applebot", published 2026-09-04, checked 2026-11-15. The two tokens are independent.
robots.txt rules: three choices
Each token gets its own group. Three goals, three rule sets (copyable version after this section):
- Block training only, keep Apple search:
User-agent: Applebot-ExtendedwithDisallow: /. Applebot keeps crawling; Siri, Spotlight and Safari visibility stays. - Block all Apple crawling:
User-agent: ApplebotwithDisallow: /. This stops the crawl feeding Apple search, so it puts search visibility at risk. - Allow everything: leave both tokens unlisted — the default.
Three documented traps:
- Googlebot fallback. If robots.txt mentions Googlebot but not Applebot, Applebot follows the Googlebot instructions. A disallow written for Googlebot can silently switch off Apple search.
- No crawl-delay. Applebot ignores crawl-delay; it adjusts its crawl rate automatically when a site slows down or returns errors.
- Rendering. Applebot may render pages in a browser. Blocking needed JS, CSS or XHR resources can break rendering — keep them available.
Training and AI answers are separate switches: nosnippet (meta tag or X-Robots-Tag: applebot: nosnippet) opts out of AI-generated answers and snippets; Extended opts out of training. They combine. The same training-versus-search split appears in GPTBot and OAI-SearchBot.
# Option 1 - block training only, keep Siri / Spotlight / Safari (active)
User-agent: Applebot-Extended
Disallow: /
# Option 2 - block all Apple crawling (also turns off Apple search)
# User-agent: Applebot
# Disallow: /
# Option 3 - allow all: leave both tokens unlisted (default)Verify that a request is really Applebot
User-agent strings can be faked. Apple documents two checks:
- Reverse DNS, then forward-confirm. The hostname must end in
.applebot.apple.com, and resolving it must return the same IP. Apple's example:
$ host 17.58.101.179
179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.
$ host 17-58-101-179.applebot.apple.com
17-58-101-179.applebot.apple.com has address 17.58.101.179.
- CIDR match. Apple publishes its IP ranges at search.developer.apple.com/applebot.json: 24 IPv4 prefixes, all 17.x, creationTime 2026-09-15 (checked 2026-11-15). Match each source IP against the list.
The search user agent looks like Safari plus an Applebot suffix, e.g. desktop:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mobile swaps in iPhone; CPU iPhone OS 17_4_1 like Mac OS X and adds Mobile/15E148. Apple occasionally updates the advertised browser version; match the Applebot/ token and verify by DNS or IP.
Check your own data
Count what Applebot fetches and sends back before changing rules. Sample data, 30-day window, server-log import:
- Verified Applebot: 12,000 requests across 3,000 pages, top status 200.
- Human visits from Apple surfaces (Siri / Spotlight / Safari): 900.
- Reads-to-visits ratio: 900 ÷ 12,000 = 7.5% — 75 visits per 1,000 search reads.
- At a 2.0% goal conversion rate: 900 × 2.0% = 18 conversions per month.
- Applebot-Extended: 0 requests — it never crawls.
Compare the two rules:
- Wrong block (
Disallow: /for Applebot). Applebot still reads robots.txt, about once a day: 1 × 30 = 30 requests, 1 page, no new content crawls. The 900 visits and 18 conversions are at risk (an upper bound). Apple publishes no index-decay timing, so none is assumed. - Correct block (
Disallow: /for Applebot-Extended). Applebot stays at 12,000 requests / 3,000 pages; visits stay 900; conversions stay 18. Extended requests are 0 before and after, so the check is the rule's presence, not a traffic drop. - Googlebot-fallback trap. Disallowing Googlebot without mentioning Applebot reproduces the same 12,000 → 30 collapse, silently.
Complete counts need server logs: page-script analytics only sees crawls that execute JavaScript.
Applebot agents · 30-day windows
Wrong rule: Disallow Applebot| Bot | Purpose | Requests | Pages | Top status | Trust |
|---|---|---|---|---|---|
| Applebot (before rule) | Search engine | 12,000 | 3,000 | 200 | Verified |
| Applebot (after wrong rule) | robots.txt only | 30 | 1 | 200 | Verified |
| Applebot-Extended | AI training (token) | 0 | 0 | — | n/a (no crawl) |
Human visits from Apple surfaces: 900
Decide per goal
| Goal | Rule | What you keep or lose |
|---|---|---|
| Out of AI training, stay in Apple search | Applebot-Extended + Disallow: / | Search stays; example keeps 900 visits, 18 conversions |
| Maximum Apple visibility | No rules for either token | Keep key pages returning 200 and renderable |
| Out of Apple search entirely | Applebot + Disallow: / | Example: 900 visits, 18 conversions/month at risk |
| Out of AI answers, still indexed | nosnippet meta or X-Robots-Tag: applebot: nosnippet | Searchable; suggestions show title only |
| Paywalled pages searchable, out of AI context | JSON-LD isAccessibleForFree: false | Search-eligible, not AI answer context; hasPart unsupported |
| Less server load | No crawl-delay; rate auto-adjusts on errors | Keep rules simple; rate-limit at the server |
The switch you want is Extended, not Applebot: the wrong block puts 900 visits and 18 conversions a month at risk. Unsure about a crawler's category? Start with the AI crawler glossary. A rule only states your wish; checking actual fetches shows whether it is honoured.
Applebot in TapCub
Run the example above in TapCub before touching robots.txt: is Applebot healthy, and did your rule hit the right token?
- Collect server records. With only the page script, TapCub sees only bots that run JavaScript. Send server-side access records or import access logs through the API so all 12,000 Applebot requests are counted (setup). Recognised bots stay out of human visitors and conversion denominators by default.
- Find the right row. In Web analytics › Bots, open the All bots tab. Applebot is listed with operator Apple and purpose Search engine. On server records TapCub runs the same check as above, reverse DNS to applebot.apple.com plus a forward lookup, so trust reads Verified when it passes and Suspected when it does not. Applebot-Extended is on TapCub's list as AI training, but since it never crawls, expect no requests under that name.
- Watch the search side. As a search engine, Applebot appears in the search-engine crawl-health table on All bots and on Monitoring › Crawl health: fetches, success rate, 404s and 5xx. In the sample that is 12,000 fetches with status 200.
- Publish the Extended-only rule, then check which token it hit. The Rule file generator on Monitoring › Crawler compliance has a Block all AI training crawlers preset; its list includes Applebot-Extended and leaves Applebot alone. After you publish, Applebot should stay near 12,000 requests on All bots and Crawl health. If it collapses to about 30 robots.txt fetches, you hit Applebot itself or the Googlebot fallback: restore it and disallow only Applebot-Extended. Crawler compliance also lists any fetch made while a path was disallowed.
What TapCub does not do: it measures and verifies, but it does not edit robots.txt or change rankings. Siri and Spotlight visits are not broken out as a source of their own, so the 900-visit side of the example comes from your own referrer and landing-page analysis. Apple offers no webmaster console, and Monitoring › Search performance connects Google Search Console or Bing Webmaster only, so server records plus the Bots report are how you see Applebot.
FAQ
What is Applebot?
Apple's web crawler. It fetches and may render pages to power search across Spotlight, Siri and Safari. Apple says the same data may also train its foundation models and provide context for AI answers; each use has its own control.
What is the difference between Applebot and Applebot-Extended?
Applebot does the crawling for search. Applebot-Extended never crawls webpages — it is a robots.txt token deciding whether data collected by Applebot may be used to train Apple's generative AI foundation models. Disallowing Extended keeps your pages eligible for Apple search results.
Does blocking Applebot remove my site from Siri and Spotlight?
It removes the crawl that feeds them. Apple says enabling Applebot in robots.txt lets content appear in Spotlight, Siri and Safari search, so a blanket disallow puts that visibility at risk. To opt out of training only, disallow Applebot-Extended instead.
Does Applebot follow robots.txt?
Yes for general search crawls, with two documented quirks: if your robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot instructions; and Applebot does not follow crawl-delay. The separate iTMS user agent for Apple Podcasts does not follow robots.txt.
How do I verify a request is really Applebot?
Run a reverse DNS lookup on the source IP: the hostname must end in .applebot.apple.com. Then forward-resolve that hostname and confirm it returns the same IP. Alternatively, match the IP against Apple's published CIDR list at search.developer.apple.com/applebot.json. A matching user agent alone is not proof.
Does blocking Applebot-Extended hurt my Apple search ranking?
No. Apple states that site rules for Applebot-Extended are not considered in ranking for Search, and pages that disallow Extended can still be included in search results. The token only controls whether Applebot-collected content may be used to train Apple's foundation models.
See also
Was this article helpful?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Solutions Engineer
Covers SDKs, API integrations, and data implementation, with an emphasis on validation, system interoperability, and troubleshooting.
