Google explains how it deduplicates Top Stories from main search results

Google will deduplicate a link from its main web results if that same link appears in the first Top Stories slot, so long as the Top Stories section appears before the main web results, according to Danny Sullivan, public search liaison at Google.

However, if this is not the case (e.g., the main web results appear above the Top Stories section), Google will not deduplicate the link (as shown above, to the right). “And again, it’s something we’re reviewing,” Sullivan noted.

What Google said. “Just to cap off with the further clarification I promised, we deduplicate a link from web results if a link appears as the first link in Top Stories and if the Top Stories box appears before web results,” Sullivan said on Twitter, “If it comes after, we don’t.”

This explanation was provided after Dieter Bohn, executive editor at The Verge, called out Google over search listings that were “stealing [The Verge’s] content.”

The query can influence deduplication.  As Sullivan explained, searching by exact headline may not reflect how most people seek out information. Bohn’s example query, “trials and tribulations turning a real camera,” was a partial match for The Verge’s article — Sullivan’s example of what a typical user might search for was “turning camera into webcam.”

For users searching for solutions, “Our systems also are going to generally seek to show the most helpful, reliable info they can,” Sullivan wrote, “That’s why you don’t see a lot of duplicates of your article showing.”

However, headline-oriented searches (like the one Bohn searched for) “typically contain a lot of terms, so our systems shift to return pages that have those terms,” he added. “This means authors are more likely to find duplicates, even though for typical searches that readers would do, these are unlikely to appear.”

Deduplication isn’t new. In January 2020, Google began efforts to deduplicate the right-sidebar featured snippet variant, migrating it into the main results column. Prior to that change, the URL for featured snippets would display twice on the first results page: once as part of the featured snippet and another time as a regular organic listing.

Why we care. Knowing how Google deduplicates results can help you understand how many times your content can appear on the main results page. The more your URLs appear on the main results page, the more likely it is that users will be inclined to click through on one of those listings.

Additionally, this also highlights an opportunity for content that appears in the second, third or fourth slots of the Top Stories section: since they are not the first link, they are not eligible for deduplication and can thus appear in the main results as well, providing the publisher with more search visibility.