Number of websites blocking Google-Extended jump 180%

The New York Times, Yelp and 22 Condé Nast properties are among those who have chosen to block Google-Extended so far.

Chat with SearchBot

More than 250 websites now block Google-Extended, the “standalone product token” Google introduced Sept. 28 to let you block Bard, Vertex AI generative APIs and future generations of models from accessing your content.

That’s according to research shared exclusively with Search Engine Land by the team.

Why we care. There has been much debate and discussion about whether brands and businesses should block any bots (e.g., GPTBot, CCBot) that crawl content that is then used to train LLMs. Only a minority of sites have decided to block so far, but the numbers have continued to climb in recent months – as they don’t want their content to help AI companies profit and compete against them.

A big increase. As of Nov. 19, 252 websites out of a set of 3,000 popular websites had blocked Google-Extended. Just over a month earlier (Oct. 8), only 89 of those sites had blocked Google-Extended.

  • That means the number of sites blocking Google-Extended jumped 180% in the past month.

Websites blocking Google-Extended crawling. They include:

  • Ziff Davis properties (e.g., PC Mag, Mashable).
  • Vox properties (e.g., The Verge and NYMag).
  • The New York Times.
  • Condé Nast (22 sites, including GQ, Vogue, Wired)
  • Yelp (frequent Google critic and legal opponent).

Reminder. While you can block Google-Extended in robots.txt, that does not block your content from appearing in Google’s Search Generative Experience or prevent Google from using your content from training SGE. To opt-out fully, you’d have to block Googlebot (which would also take you out of Search). However, you can opt out of SGE overviews using nosnippet.

About the author

Danny Goodwin
Danny Goodwin has been Managing Editor of Search Engine Land & Search Marketing Expo - SMX since 2022. He joined Search Engine Land in 2022 as Senior Editor. In addition to reporting on the latest search marketing news, he manages Search Engine Land’s SME (Subject Matter Expert) program. He also helps program U.S. SMX events.

Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.

Get the must-read newsletter for search marketers.