TapCub is live — analytics, insights and live chat, free on one platform
AI Search

Applebot and Applebot-Extended: Two Agents and How to Control Each One

Applebot crawls for Spotlight, Siri and Safari search; Applebot-Extended is only a robots.txt switch for AI training. What each does, three rule choices, DNS and IP verification, and a worked example of what the wrong block costs.

Ethan Brooks Published Nov 16, 2026 10 min read AI Search
Sample data: server records. Applebot verified by reverse DNS to applebot.apple.com; Extended never crawls; the wrong rule leaves only robots.txt fetches.Sample data

Try it on your own data

Every report in this article is in the free plan. One snippet for web and apps, up to 10 sites.

Start free

Applebot is Apple's search crawler for Spotlight, Siri and Safari; Applebot-Extended is only a robots.txt switch for AI training. Block the wrong token and you switch off Apple search too. This guide covers both agents, three rule choices, verification, and a worked cost example.

What you will learn

  • Applebot crawls for Spotlight, Siri and Safari search; Applebot-Extended never crawls — it only decides whether Applebot-collected data may train Apple's AI models
  • Disallow Applebot-Extended to opt out of training and keep Apple search; disallowing Applebot itself puts the visits it sends back at risk
  • Verify with reverse DNS to *.applebot.apple.com plus a forward check, or Apple's CIDR JSON; watch for the Googlebot-fallback rule that can block Applebot silently

What Applebot is

Applebot is Apple's web crawler. Its data powers search across Spotlight, Siri and Safari, and Apple says the same crawled data may also help train Apple foundation models (Apple Intelligence, Services, Developer Tools) and provide context for AI-generated answers (About Applebot, published 2026-09-04, checked 2026-11-15).

The name covers two things. Applebot is the crawler in your logs. Applebot-Extended is not a crawler: it is a robots.txt token deciding whether Applebot-collected data may train AI. Disallowing Applebot itself turns off the crawl that feeds Apple search; the training-only switch costs none of those visits.

Apple's two agents

From Apple's documentation:

  • Applebot (Search) crawls and may render pages for Spotlight, Siri and Safari search. Enabling it in robots.txt lets content appear in those results.
  • Applebot-Extended does not crawl webpages. It only determines how Applebot-collected data may be used — specifically, whether it may train Apple's generative foundation models. Pages that disallow Extended can still appear in search results, and Apple states Extended rules are not considered in ranking for Search.
  • iTMS (Apple Podcasts) fetches only URLs registered on Apple Podcasts and does not follow robots.txt.

Google uses the same token-only pattern with Google-Extended: a robots.txt name governing data use, not a crawl.

AgentWhat it doesrobots.txt effect
ApplebotCrawls (and may render) for Spotlight, Siri and Safari search; collected data may also train Apple foundation models and power AI answersDisallow stops the crawl -> content loses Apple search visibility
Applebot-ExtendedDoes not crawl; token deciding whether Applebot-collected data may train Apple foundation modelsDisallow = training opt-out; search stays; not a Search ranking factor
iTMS (Apple Podcasts)Fetches only URLs registered on Apple PodcastsDoes not follow robots.txt

Source: Apple "About Applebot", published 2026-09-04, checked 2026-11-15. The two tokens are independent.

robots.txt rules: three choices

Each token gets its own group. Three goals, three rule sets (copyable version after this section):

  1. Block training only, keep Apple search: User-agent: Applebot-Extended with Disallow: /. Applebot keeps crawling; Siri, Spotlight and Safari visibility stays.
  2. Block all Apple crawling: User-agent: Applebot with Disallow: /. This stops the crawl feeding Apple search, so it puts search visibility at risk.
  3. Allow everything: leave both tokens unlisted — the default.

Three documented traps:

  • Googlebot fallback. If robots.txt mentions Googlebot but not Applebot, Applebot follows the Googlebot instructions. A disallow written for Googlebot can silently switch off Apple search.
  • No crawl-delay. Applebot ignores crawl-delay; it adjusts its crawl rate automatically when a site slows down or returns errors.
  • Rendering. Applebot may render pages in a browser. Blocking needed JS, CSS or XHR resources can break rendering — keep them available.

Training and AI answers are separate switches: nosnippet (meta tag or X-Robots-Tag: applebot: nosnippet) opts out of AI-generated answers and snippets; Extended opts out of training. They combine. The same training-versus-search split appears in GPTBot and OAI-SearchBot.

robots.txtThree choices - option 1 active: block training only, keep Apple search
# Option 1 - block training only, keep Siri / Spotlight / Safari (active)
User-agent: Applebot-Extended
Disallow: /

# Option 2 - block all Apple crawling (also turns off Apple search)
# User-agent: Applebot
# Disallow: /

# Option 3 - allow all: leave both tokens unlisted (default)

Verify that a request is really Applebot

User-agent strings can be faked. Apple documents two checks:

  1. Reverse DNS, then forward-confirm. The hostname must end in .applebot.apple.com, and resolving it must return the same IP. Apple's example:
$ host 17.58.101.179
179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.
$ host 17-58-101-179.applebot.apple.com
17-58-101-179.applebot.apple.com has address 17.58.101.179.
  1. CIDR match. Apple publishes its IP ranges at search.developer.apple.com/applebot.json: 24 IPv4 prefixes, all 17.x, creationTime 2026-09-15 (checked 2026-11-15). Match each source IP against the list.

The search user agent looks like Safari plus an Applebot suffix, e.g. desktop:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Mobile swaps in iPhone; CPU iPhone OS 17_4_1 like Mac OS X and adds Mobile/15E148. Apple occasionally updates the advertised browser version; match the Applebot/ token and verify by DNS or IP.

Check your own data

Count what Applebot fetches and sends back before changing rules. Sample data, 30-day window, server-log import:

  • Verified Applebot: 12,000 requests across 3,000 pages, top status 200.
  • Human visits from Apple surfaces (Siri / Spotlight / Safari): 900.
  • Reads-to-visits ratio: 900 ÷ 12,000 = 7.5% — 75 visits per 1,000 search reads.
  • At a 2.0% goal conversion rate: 900 × 2.0% = 18 conversions per month.
  • Applebot-Extended: 0 requests — it never crawls.

Compare the two rules:

  • Wrong block (Disallow: / for Applebot). Applebot still reads robots.txt, about once a day: 1 × 30 = 30 requests, 1 page, no new content crawls. The 900 visits and 18 conversions are at risk (an upper bound). Apple publishes no index-decay timing, so none is assumed.
  • Correct block (Disallow: / for Applebot-Extended). Applebot stays at 12,000 requests / 3,000 pages; visits stay 900; conversions stay 18. Extended requests are 0 before and after, so the check is the rule's presence, not a traffic drop.
  • Googlebot-fallback trap. Disallowing Googlebot without mentioning Applebot reproduces the same 12,000 → 30 collapse, silently.

Complete counts need server logs: page-script analytics only sees crawls that execute JavaScript.

app.tapcub.com/web/bots

Applebot agents · 30-day windows

Wrong rule: Disallow Applebot
BotPurposeRequestsPagesTop statusTrust
Applebot (before rule)Search engine12,0003,000200Verified
Applebot (after wrong rule)robots.txt only301200Verified
Applebot-ExtendedAI training (token)00—n/a (no crawl)

Human visits from Apple surfaces: 900

Sample data
Sample data: server records. Applebot verified by reverse DNS to applebot.apple.com; Extended never crawls; the wrong rule leaves only robots.txt fetches.

Decide per goal

GoalRuleWhat you keep or lose
Out of AI training, stay in Apple searchApplebot-Extended + Disallow: /Search stays; example keeps 900 visits, 18 conversions
Maximum Apple visibilityNo rules for either tokenKeep key pages returning 200 and renderable
Out of Apple search entirelyApplebot + Disallow: /Example: 900 visits, 18 conversions/month at risk
Out of AI answers, still indexednosnippet meta or X-Robots-Tag: applebot: nosnippetSearchable; suggestions show title only
Paywalled pages searchable, out of AI contextJSON-LD isAccessibleForFree: falseSearch-eligible, not AI answer context; hasPart unsupported
Less server loadNo crawl-delay; rate auto-adjusts on errorsKeep rules simple; rate-limit at the server

The switch you want is Extended, not Applebot: the wrong block puts 900 visits and 18 conversions a month at risk. Unsure about a crawler's category? Start with the AI crawler glossary. A rule only states your wish; checking actual fetches shows whether it is honoured.

Applebot in TapCub

Run the example above in TapCub before touching robots.txt: is Applebot healthy, and did your rule hit the right token?

  1. Collect server records. With only the page script, TapCub sees only bots that run JavaScript. Send server-side access records or import access logs through the API so all 12,000 Applebot requests are counted (setup). Recognised bots stay out of human visitors and conversion denominators by default.
  2. Find the right row. In Web analytics › Bots, open the All bots tab. Applebot is listed with operator Apple and purpose Search engine. On server records TapCub runs the same check as above, reverse DNS to applebot.apple.com plus a forward lookup, so trust reads Verified when it passes and Suspected when it does not. Applebot-Extended is on TapCub's list as AI training, but since it never crawls, expect no requests under that name.
  3. Watch the search side. As a search engine, Applebot appears in the search-engine crawl-health table on All bots and on Monitoring › Crawl health: fetches, success rate, 404s and 5xx. In the sample that is 12,000 fetches with status 200.
  4. Publish the Extended-only rule, then check which token it hit. The Rule file generator on Monitoring › Crawler compliance has a Block all AI training crawlers preset; its list includes Applebot-Extended and leaves Applebot alone. After you publish, Applebot should stay near 12,000 requests on All bots and Crawl health. If it collapses to about 30 robots.txt fetches, you hit Applebot itself or the Googlebot fallback: restore it and disallow only Applebot-Extended. Crawler compliance also lists any fetch made while a path was disallowed.

What TapCub does not do: it measures and verifies, but it does not edit robots.txt or change rankings. Siri and Spotlight visits are not broken out as a source of their own, so the 900-visit side of the example comes from your own referrer and landing-page analysis. Apple offers no webmaster console, and Monitoring › Search performance connects Google Search Console or Bing Webmaster only, so server records plus the Bots report are how you see Applebot.

FAQ

What is Applebot?

Apple's web crawler. It fetches and may render pages to power search across Spotlight, Siri and Safari. Apple says the same data may also train its foundation models and provide context for AI answers; each use has its own control.

What is the difference between Applebot and Applebot-Extended?

Applebot does the crawling for search. Applebot-Extended never crawls webpages — it is a robots.txt token deciding whether data collected by Applebot may be used to train Apple's generative AI foundation models. Disallowing Extended keeps your pages eligible for Apple search results.

Does blocking Applebot remove my site from Siri and Spotlight?

It removes the crawl that feeds them. Apple says enabling Applebot in robots.txt lets content appear in Spotlight, Siri and Safari search, so a blanket disallow puts that visibility at risk. To opt out of training only, disallow Applebot-Extended instead.

Does Applebot follow robots.txt?

Yes for general search crawls, with two documented quirks: if your robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot instructions; and Applebot does not follow crawl-delay. The separate iTMS user agent for Apple Podcasts does not follow robots.txt.

How do I verify a request is really Applebot?

Run a reverse DNS lookup on the source IP: the hostname must end in .applebot.apple.com. Then forward-resolve that hostname and confirm it returns the same IP. Alternatively, match the IP against Apple's published CIDR list at search.developer.apple.com/applebot.json. A matching user agent alone is not proof.

Does blocking Applebot-Extended hurt my Apple search ranking?

No. Apple states that site rules for Applebot-Extended are not considered in ranking for Search, and pages that disallow Extended can still be included in search results. The token only controls whether Applebot-collected content may be used to train Apple's foundation models.

See also

Was this article helpful?

We use your answers to fix and expand this page.

Ethan Brooks

Solutions Engineer

Covers SDKs, API integrations, and data implementation, with an emphasis on validation, system interoperability, and troubleshooting.

See which Apple agents read your site

Bots by operator and purpose, DNS-verified trust, reads against visits sent back, and robots.txt compliance checks.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics