SEO

Microsoft Clarity AI Scrape-to-Referral insights report

Microsoft Clarity has added another new AI metric, this one is called AI Scrape-to-Referral insights. It puts a single metric on the AI scrape activity against referral traffic. The purpose is to give you insight into whether it is worth having that AI bot crawl your site, and whether there is enough return and value to keep allowing that bot to scrape your website.

What is this metric. Microsoft wrote, “With the new AI Scrape-to-Referral Ratio card and direct links into session recordings, you can evaluate the tradeoff between content extraction and traffic return, then investigate how AI-referred visitors behave once they arrive.”

There is a new AI Scrape-to-Referral Ratio card which compares AI scrape activity against referral traffic in a single metric, helping you understand whether AI platforms are returning value in the form of visits rather than only extracting content.

Here is a graphic Microsoft uploaded of this metric:

Scrape To Referral Cover Image 1 2048x1016 1

This seems to say that this AI tool sent 41 referrals to your website, and if you do the math, it has a 6,000 scrapes-to-1-referral ratio based on its crawl activity related to referral activity.

More metrics. The Bot Analytics report now shows these metrics, as Microsoft said:

  • AI Scrape-to-Referral Ratio card: Compare AI scrape activity against referral traffic in a single metric, helping you understand whether AI platforms are returning value in the form of visits rather than only extracting content. 
  • Operator-level breakdown: Review a ranked list of referring operators to identify which AI sources are sending traffic back and which ones may have high scrape activity with limited referral return. 
  • Mapped-domain coverage safeguards: Ratios are calculated only across correctly mapped domains, helping prevent misleading rollups when CDN and Clarity coverage do not fully align. 
  • Clearer handling of coverage gaps: When your AI bot data and referral traffic data cover different domains, Clarity provides context so you understand how domain coverage may affect the Scrape-to-Referral Ratio.
  • Direct path to session recordings: Click from referral insights into the Recordings tab with filters already applied for the selected AI referral source, making it easier to validate attribution and inspect real user journeys. 
  • Better traffic quality analysis: Move beyond top-line referral counts to see whether AI-referred visitors scroll, engage, convert, or leave quickly, so you can compare which AI sources are driving valuable traffic. 

How do you see it. Microsoft posted these steps to access this data for your site(s):

  1. Open Bot Analytics in the Clarity AI Visibility Dashboard. 
  2. Review the AI Scrape-to-Referral Ratio card to understand overall referral return and operator-level performance. 
  3. Check the coverage context to confirm your mapped domains are aligned across your connected data sources. 
  4. Select a referral source from the card to open session recordings filtered for AI-referred traffic. 
  5. Watch recordings to evaluate visit quality, including engagement patterns and meaningful actions such as purchases, sign-ups, or form fills. 
  6. Use those insights to compare AI sources and refine how you measure the value of AI visibility for your site.

Why we care. Many site operators ask, is it worth allowing AI crawlers to access, scrape, and use my site’s content? Well, here is a new metric that puts a number on that figure, allowing you to be more informed about which AI crawlers you allow and which AI crawlers you disallow.

What is the ratio you are aiming for when making that decision? Well, that depends on the site, the business, and the purpose of the website.

Explore more on

Microsoft Clarity Microsoft Web analytics Web scraping Web traffic Artificial intelligence
Add Search Engine Land as a preferred source on Google

Editorial disclosure

Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.

About the author

Barry Schwartz
Barry Schwartz is a technologist and a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics.

In 2019, Barry was awarded the Outstanding Community Services Award from Search Engine Land, in 2018 he was awarded the US Search Awards the "US Search Personality Of The Year," you can learn more over here and in 2023 he was listed as a top 50 most influential PPCer by Marketing O'Clock.

Barry can be followed on X here and you can learn more about Barry Schwartz over here or on his personal site.