App Indexing & The New Frontier of SEO: Apple Search + iOS App Indexing

Every year in June, Apple and Google hold conferences to announce their latest technology. With each announcement, both companies seem to be reaffirming their larger company goals and values — Apple asserting its commitment to designing and selling quality products, and Google bolstering its mission to collect and organize the world’s data with the ultimate intent of monetizing the results with advertising.

These goals resurface through many of the company’s business decisions, most recently through the divergent app indexing frameworks promoted by the two companies. For a long time, deep app content has been locked in app code, largely un-indexed and inaccessible to search engine crawlers. The only content that crawlers could get was app titles and descriptions from the various app stores, and from company websites.

Now that both Apple and Google have announced methods for indexing deep content within apps, the situation has dramatically and irreversibly changed. For those interested in the most cutting-edge digital marketing strategies, the concept of SEO will have to expand to include optimization of deep app content for inclusion in the Apple and Google indexes.

Apple’s overall focus on “products” positions the company to gain the most from a primarily app-based world. In Apple’s ideal scenario, the web is only used as an invisible layer that links apps together and allows apps to become their own uniquely-controlled display-layers for private and public content.

Conversely, Google stands to gain the most from a web-based world where data can be most easily collected, organized, and distributed so Google can become the presentation layer of the Internet. While apps are part of their data collection strategy, Google has invested heavily in programs that allow developers to create HTML5 web content to rival apps and thus return users to a web-based ecosystem.

This article is the first in a three-part series about new strategies and opportunities for app indexing in Apple, Google, and other ecosystems.

This first article focuses on the ways in which Apple’s new Search API for iOS 9 encourages and incentivizes an app-centric digital ecosystem and what that means for marketers looking to surface deep app content for Apple users. This is a comprehensive guide of what you as an SEO need to know about Apple’s foray into search engineering, what it means for your company’s app strategy, how it impacts your company’s SEO strategy, and how to take advantage of these new opportunities to surface your company’s app and website in Apple Search.

The subsequent articles in the series will focus on Google’s app indexing opportunities and the associated strategies, and on future app indexing challenges we will face with the growth wearables and other non-standard device apps and device indexes.

What Is Apple Search & Why Does It Matter For SEO?

If you want to surface your app content for iOS users, Apple Search could become a key part of your SEO strategy. Like Google, Apple Search relies on an index (or more accurately, multiple indices) to organize the set of app screens that can be ranked in a search result. Apple refers to their indexes and the software that interfaces with those indexes collectively as “Apple Search.”

Indexed app screens can be surfaced by executing a search in Spotlight or Siri, or through a Spotlight Suggest result that appears when a user types in a query in the address bar of Safari (before hitting “enter”). This means that a user can start a search on Safari and end up being directed to a deep link without ever seeing a single Google result — even where Google is the “default” search engine.

To accomplish this, Apple has introduced the concept of public and private indexing, and three methods for iOS developers to get their app screens indexed for Apple Search: NSUserActivity, CoreSpotlight, and Web Markup.

Despite being limited to iOS app indexing (sorry Android buddies, your apps can’t play here), Apple’s new Search and indexing framework will be an important traffic channel, especially for Apple users who sometimes bypass Google’s algorithms entirely. Unlike Google, Apple has come up with two indexing methods that do not require corresponding web pages to drive app indexing:

1. NSUserActivity Indexing

In this indexing method, Apple indexes “user activities,” which you can think of as a sort of “bookmark + JsessionID + cookie” snapshot of a app screen, complete with the navigational understanding of how a user got there and how they have interacted with that screen in the past. Apple calls these snapshots “NSUserActivities,” and they can be indexed by Apple when a developer adds search eligibility markup to the app’s code itself (no website needed), and a user accesses a screen with the markup.

Each NSUserActivity can also be associated with a contentAttributeSet which includes relevant meta data like titles, keywords, and descriptions. This information is used to create the appearance of the search result and to help determine rankings. Each NSUserActivity should have a uniqueIdentifier and/or include a URL link to its corresponding web content.

2. CoreSpotlight Indexing

The CoreSpotlight API has historically allowed native iOS components like the Calendar and Mail apps to include items in Spotlight search results. This is now being opened up to all apps in the App Store. This indexing method is somewhat analogous to submitting an XML sitemap in SEO, but the index file is submitted with your application manifest, and it is a different, iOS-specific code, created in a file called a “CSSearchableIndex.”

In this indexing method, the app screens to be indexed are called “CSSearchableItem(s),” and each one is associated with a label called a “uniqueIdentifier.” Each CSSearchableItem can be associated with a “CSSerchableItemAttributeSet” that includes relevant metadata like titles, keywords and descriptions. The CSSerchableItemAttributeSet is used to determine algorithmic relevance and populate the search result.

Each CSSearchableItem/uniqueIdentifier combo can also be associated with “domainIdentifier” to help group certain types of app screens in the database, so that they can be added or removed from the CSSearchableIndex as a group. For example, a uniqueIdentifier might be associated with a specific photo screen while a domainIdentifier might indicate all of the photos within an album.

The important thing to note is that NSUserActivity and CoreSpotlight source their search result meta data from code within the apps themselves (not from a website). This means that SEOs need to be more involved in the app development process earlier than ever before in order to ensure that Apple Search optimized title, description, and keyword markup is included in the app at launch (just like you would for a new web page launch back in 1999).

In addition to NSUserActivity and CoreSpotlight, Apple also allows you to get your deep app content indexed using a web crawler and deep link markup, similar to Google. Deep linking is a process in which web markup is used to link web URLs to their corresponding app screens (called URIs), so that a crawler can understand the connection between the URLs and the URIs. This option does require app content to have crawlable corresponding web content:

3. Web Markup Indexing

In this indexing method, Apple’s new web-crawler, “Applebot,” indexes app content from the marketing and/or support URLs that are submitted with the app manifest. Applebot can crawl your website and index corresponding app screens based on the following markup protocols:

• Twitter Card Markup: Twitter Card markup can include a protocol to reference deep links to app screens.
• App Links Protocol: App Links is an external deep linking standard that is also used by Facebook and Bing (and works on iOS and Android).
• Apple’s Smart App Banners: A protocol created by Apple that displays a special Apple banner on websites when they are accessed from iOS devices. The banner either prompts the user to open the app (if installed) or download it from the App Store.

Additionally, though it won’t help you get new app screens indexed, Applebot can crawl two other protocols, looking for metadata to display in search results:

• Open Graph: Facebook’s protocol that makes web content easier to share. Apple supports the full protocol and if OG tagging is already in place, Applebot can crawl it.
• Schema.org: An external markup standard widely embraced across the web. Apple Search is launching with limited support, including the following schemas: AggregateRating, Offers, PriceRange, Interaction Count (likes, views, comments), Organization (phone numbers), Recipes, SearchAction (landing page for search), ImageObject and Actions, limited to: dialing a phone number, getting directions, playing audio or video file.

Web Markup is great for SEOs, because SEOs can enable and control indexing and metadata by adding markup to the website as we are used to doing. iOS app developers need only generate and share the URI deep links with the SEO team. If app URIs are already in place, this can be a very fast and simple implementation that doesn’t require any development resources or an app update.

IMPORTANT REMINDER: When iOS deep links on a web page are clicked (be it from your web page or even Facebook or Twitter), they initiate an “openURL” command that tells the phone Operating System to switch from the browser to an app — openURL must be enabled in your app code for deep links to open in the app.

Make sure your app can actually open deep links before you include them in your web markup. If you are relying exclusively on the Web Markup method for app indexing, you will still need to make this change to your app and submit the update. It’s a small detail, but an important one.

How Does Apple Rank Apps In Apple Search?

In light of an increased global sensitivity to privacy issues (think the “Right to Be Forgotten” in the EU), Apple has begun to focus on privacy concerns as a key differentiator, and selling point for their products and services. Apple’s new app indexing framework showcases a distinct effort to protect users’ personal information by introducing multiple app indexes:

• A private ‘Device Index,’ which is a personal index that is only accessible by the specific user ID, like a small individualized database for each Apple user
• A public ‘Cloud Index,’ which holds content that is accessible from any Apple device through Spotlight, Siri, or Spotlight Suggest in Safari

Because of this architecture, developers can (and should) allow a user’s private app screens to be saved to a private Device Index. Private screens in any app, such as direct messages and user dashboards, are now theoretically indexable.

NOTE: A similar personalized search history is used by Google, but has never been publicly described as a separate “index” or opened up for developer access. Google tested offering ‘desktop search’ years ago, and has recently been experimenting with indexing personal email results from Gmail. We expect this trend to continue and expand for both Google and Apple into the future.

Like Google, Apple has not disclosed the exact details of their search algorithm, but they have described some key ranking factors that should be considered. Many of Apple’s search ranking factors focus on determining how content should rank in a private Device Index vs. the public Cloud Index. SEOs need to be measured and responsible to have a positive impact on rankings because Apple has said that “malicious or poorly considered [app indexing strategy] implementations” will be penalized “or suppressed entirely” (put those black hats away).

A more comprehensive list of Apple Search ranking factors is included below:

Positive Ranking Factors

• App Installation Status. Is the app is installed on the device? (installed apps seem to get preference)
• Personalized App Engagement. Does the individual engage with the screen in the app? This is based on time spent with result that Apple determines from session analytics.
• App Result Click-Through Rate. Do users frequently click through the search result vs. picking another result or searching again?
• Keywords/ Title. Do keywords from the “keywords” and “title” designations in the app markup match up with the user’s query?
• Aggregated Engagement. How many users engage with the app screen?
• Structured Data on Web. Is structured data correctly implemented?
• Canonical App IDs. Is the same screen associated with one unique ID or URL across multiple indexing methods (NSUserActivity, CoreSpotlight, and Web Markup)?
• Strength/Popularity of Web URL. How popular is the website associated with the app deep links? (Presumably, this is based on Applebot’s crawl.)

Negative Ranking Factors

• Low Engagement. Do very few users engage with the app screen? (engagement determined by session analytics)
• Over-Indexing. Does the app have many screens in the index with low or no engagement?
• Returns. Do users return to search results right after looking at the app?
• Keywords Spamming. Are developers stuffing too many irrelevant keywords into the keyword field?
• Interstitials. Is something covering the content in the app or preventing users from accessing it?
• Javascript (web only). Is Javascript preventing Applebot from crawling your site to find new app deep links?
• Low Star Ratings, Low Review Volume, Poor Reviews. Apple has not explicitly called these negative ranking factors for Apple Search, but they are negative ranking factors for the App Store, so we expect Apple to treat them similarly here.

Apple recommends pursuing multiple indexing methods to optimize app visibility, but the overlapping methods will inevitably create duplication across the various indexes. For example, private content could have both a NSUserActivity and CSSearchableItem indexed, and public content could have both a NSUserActivity and a Web Markup deep link indexed.

This is obviously not ideal for controlling Applebot’s efficiency, so Apple strongly recommends associating each NSUserActivity, CSSearchableItem, and Web Markup deep link with the same uniqueIdentifier and/or URL. This is Applebot’s version of a canonical, and it’s so important to Apple that they’ve even made it a ranking factor in the Apple Search algorithm.

How Do I Decide Which Indexing Option Is Most Strategic For My Company?

iOS developers can indicate which indexes they think their app content should be included in. Not all app content must be indexed, so the first question your company needs to answer is: Which content should be included in Apples index and which should not?

Then, since not all indexing methods can assign content to both the public and private indexes, ask: Of the content that should be indexed, which content should be publicly accessible, and which content should be private? The answers to these questions will be the beginning of your app indexing plan.

Generating a list of all app screens and their associated NSUserActivities, then mapping out the public or private nature of the content is an organized way to document this process. Once the private or public nature of app content is determined, the appropriate indexing method can be selected to meet those needs. The NSUserActivities and/or the CoreSpotlight indexing methods can both be used to index content that is intended for the private Device Index while NSUserActivities and or Web Markup can both be used to index content that is intended for the public Cloud Index.

SEOs who are used to Google’s somewhat straightforward and open standards for indexing may have a hard time understanding the 3 different ways that app content can be added to Apple’s Search indexes and how those methods interact.

We created the chart below to help — it outlines the basic functions and limitations for each Apple Search indexing method, described in the previous section, so that the comparison becomes clearer:

NSUserActivity is the only indexing method that has the capability to privately and publicly index content. Engagement statistics from the NSUserActivity session analytics are part of the ranking algorithm, so Apple recommends focusing on the NSUserActivity indexing method first.

Developers can indicate whether a NSUserActivity should be privately or publicly indexed. However, Apple doesn’t automatically publicly index NSUserActivities that are marked for public indexing. Apple still indexes the NSUserActivity to the private Device Index first, and later “promotes” it to the public Cloud Index once a certain number of people have accessed it.

One of the most unique aspects of this new framework is when and how specific NSUserActivities can be promoted to the Cloud Index. Apple’s representative didn’t get into the details, so this aspect of the framework may be a bit unclear until launch. Luckily, he did share a diagram that seemed to indicate that each screen within an app can have a variety of NSUserActivities, some privately indexable and some publically indexable. The developer must associate each indexable NSUserActivity with eligibility code: “var eligibleForSearch” (private indexing) and “var eligibleForSearch” + “var eligibleForPublicIndexing” (public indexing).

In most cases, something like a user dashboard screen will have a publicly indexable default NSUserActivity — an ideal public search result for anyone who has never accessed this screen before. User-specific NSUserActivities that represent the non-default status of the screen should only be privately indexable because they are only appropriate for the specific user that triggered them.

Since both default and customized NSUserActivities are associated with the same app screen, anyone who accesses the screen (public and private) generates engagement data to benefit the publicly indexable, default version of the screen. As a result, the default public version of the screen is a stronger candidate for promotion to the public Cloud Index. Furthermore, because engagement metrics are a positive ranking factor, the default public version of the screen will also likely rank better in all searches, regardless of the user’s historical interactions with the app.

How Can The Different Apple App Indexing Protocols Be Used Together To Benefit SEO?

Developers can use the NSUserActivity indexing method alone, but following the Web Markup indexing method in addition to the NSUserActivity indexing method has some significant advantages (that Apple can’t infer from NSUserActivity alone):

1. When Web Markup is in place, deep linked app screens are automatically added to the public Cloud Index. (No waiting for a certain number of users to discover your content before you can cross the Apple NSUserActivity promotion “threshold.”)
2. When Applebot finds appropriate Web Markup on your site, the web pages can also be added to Apple’s index. A web page may rank in Spotlight Search, instead of the app, if the app is not installed. (Note: This means the Web Markup indexing method gives you two chances to rank: app and web. This is a huge competitive advantage because Apple only indexes web pages that have been submitted to iTunes connect or web pages that contain this app deep link markup. Applebot isn’t a roaming crawler like Googlebot.)
3. “Website Popularity” helps increase your app’s “Relevance Score” in Apple Search, which is a central aspect of the Apple search algorithm.
4. Structured data from the website is an independent ranking factor in the Apple Search algorithm.
5. Action Schema from your website can be pulled in to give your app’s search result additional visual elements (#pizzaz) that make it more appealing to users and drive click-through from the search result, another Apple Search ranking factor.
6. BONUS POINTS: This gets you halfway to getting your deep links indexed in Google Search as well (this will be discussed at length in the next article in this series).

Source: https://developer.apple.com/videos/wwdc/2015/?id=709

Deep links and user activities — whether private or public — only show up as results when the app is installed on the user’s device (or has been in the past). When the app is not installed on a user’s device, Apple will show either an App Store result or indexed web content (discovered by Applebot crawling a site’s Web Markup).

When a user gets the website result, Apple is counting on web developers to have implemented Smart App Banners to refer users from the website to the app download page in the App Store. Apple’s Smart App Banners are great for driving downloads from web traffic that approaches the website from a traditional web referral; but if Apple Search’s ultimate goal is to navigate people away from websites and into an app-only digital space, this strategy misses the boat.

NOTE: This is a notably less direct experience when contrasted with Google. A user without the app installed who taps a Google deep link from a Google search result is immediately delivered to the app download page in Google Play or the iOS App Store.

Apple has previously tested sending these clicks directly to the App Store, but received pushback from users who wanted a website option. While the new solution satisfies those users, it may fail to convert as many searches into app downloads. It’s especially tenuous if users don’t feel compelled to tap on the Smart App Banner or (more likely) if web developers have not enabled the Smart App Banner on their website at all.

Concluding Remarks

SEO practitioners must evolve to keep up with the expanding definition of what constitutes “SEO.” With the new iOS 9 app indexing announcement, SEOs also must evolve to work with Apple’s new understanding of search engineering.

Apple now offers deep app indexing that is significantly different from the app indexing opportunities provided by Google. This is largely due to its public and private indexing frameworks and the methods that marketers can use to get their deep app content indexed.

Apple’s ability to mine iOS user engagement metrics allows them to make engagement a central part of the ranking algorithm, putting less emphasis on traditional ranking factors like titles and descriptions. This is a notable evolution from how Apple currently handles search in the iOS App Store, which relies heavily on titles and keywords to determine app ranking.

The growing opportunity to target more keywords with deep app content should create incentive for app developers to focus slightly more on marketing at earlier stages in app development and force a more strategic and ongoing partnership between SEOs and app developers.

In all the discussion of the new Apple Search and app indexing, the big question that some say has not been adequately addressed is about search volume and tracking. It is unclear how much total search volume and engagement the new Apple Search utilities will get, especially when compared to Google search (and Google’s new ability to surface iOS app deep links in their search results).

Will users be drawn to app deep links over traditional web content, or will they even know the difference? Will users begin to prefer a Spotlight Search to find new apps and app content? Will they still prefer App Store searches, or will the lines between Apple search utilities continue to blur, leading to a single, app-centric Spotlight-Safari-Siri dashboard interface that can fast track app total domination over the web?

For some companies whose apps lack website parity and others whose apps have extensive private or personalized content, Apple Search currently represents the only opportunity to get their app content indexed, but the overall impact of Apple Search remains to seen.

While Apple Search presents many opportunities for new exposure on Apple devices, developers will have to work within Apple’s new app indexing framework, and optimize for a variety of indexes, using new proprietary Apple methods. Success will be limited by the volume of search that Apple Search outlets can carve out from their loyal audience, and the impact of search optimization initiatives will be much more challenging to measure.

Since Apple’s push for privacy is something Google won’t match (at least, not yet, and not without completely changing the company’s data-collection practices), we expect Apple to make more key investments in privacy-focused features. Remember, Apple’s business model is based on turning a profit on their own products, not based on helping marketers turn a profit on theirs. If Apple believes users will pay more for privacy (or the illusion of privacy), they will have no problem restricting data from marketers.

Conversely, if Apple believes they can turn more of a profit by selling access to Apple Search data, we may see new products targeted at marketers in the future — for a price. (Remember how expensive those iAds were?) Apple has historically been late to provide even basic analytics to marketers, so Apple Search could provide little to no access over many of the metrics we already enjoy in Google Analytics.

While we appreciate Apple’s video explanation of what they’re doing with Apple Search and the amount of time they are giving SEOs and developers to prepare for these changes, many questions about the technology remain unanswered. We are left wondering: Does Apple plan to use some version of QDF to determine NSUserActivity promotion into the public Cloud Index?

In other words, will a sudden increase of people engaging with a publicly indexable NSUserActivity (like a Bruins game in the NHL app) cause the activity to get promoted to the public Cloud Index before activities with more total views but less recent traffic (such as a Bruins team roster screen)? Will the fresher activity rank higher because of its timeliness? Conversely, when people stop engaging with fresh user activities, will they get “demoted” back down to the private Device Index (or, say, replaced with a more recent Bruins game)?

What questions do you have? Let us know in the comments.