Publishers push Common Crawl to stop collecting content for AI training
Could AI lose a key source of training data? Major publishers want Common Crawl to stop collecting and sharing their content.
Topic: Data scraping
Anthropic clarifies how Claude bots crawl sites and how to block them
Anthropic explains how its bots handle AI training, live queries, and search results, and what opting out means for visibility.
Cloudflare: 416 billion AI bot requests blocked since July
Google sees 3.2x more webpages than OpenAI. Cloudflare CEO Matthew Prince wants Google to separate search crawling from AI crawling.
SerpAPI calls Reddit lawsuit a threat to the ‘free and open web’
SerpAPI responds to Reddit’s scraping lawsuit, defending its practices and insisting that public search data should stay open and accessible.
Meet llms.txt, a proposed standard for AI website content crawling
Find out what llms.txt is, how it works, how to think about it, whether LLMs and brands are buying in, and why you should pay attention.
Microsoft AI CEO: Web content is ‘freeware’
Mustafa Suleyman believes almost all web content can be used for AI training unless explicitly restricted by the creator.
How to extract GBP review insights to boost local SEO visibility
Gain a competitive edge by extracting actionable insights from your – and your competitors' – Google Business Profile reviews.
Google introduces Google-Extended to let you block Bard, Vertex AI via robots.txt
You can now prevent Bard and Vertex AI generative APIs, and future generations of models, from from accessing your website, or parts of it.
It’s not a bug; it’s a feature! The Google search term report glitch actually may have been a test; Monday’s daily brief
Plus, Facebook bans researchers looking into political ads
Social Shorts: Snapchat reach campaigns, Facebook attribution change, TikTok downloads can continue
Snapchat pitches Platform Burst reach option for advertisers Snapchat is floating a new advertising offering called Platform Burst. The idea is to guarantee that “campaigns will reach at least 40% of their target audience 15 times” over a three or five day period, according to Digiday. Platform Burst campaigns can encompass different ad formats across […]
Now Official: Google Adds Restaurant Menus To Search Results
That Google menus experiment we told you about a couple weeks ago? It’s now official. But it’s only available in the U.S. at the moment. Google announced that it’s now showing restaurant menus as a OneBox-style answer at the top of its search results. It seems to be primarily triggered by searches that involve both […]
How Facebook Enables The Google Social “Scraping” It’s Upset About
I wrote a very long examination of the issues that Facebook employed a PR firm to publicize, about how Facebook feels Google may be violating privacy with its Google Social Search product. Here’s a shorter look, especially from the angle of how Facebook itself has enabled Google to do what Facebook is now complaining about. […]
Google: Bing Is Cheating, Copying Our Search Results
Google has run a sting operation that it says proves Bing has been watching what people search for on Google, the sites they select from Google’s results, then uses that information to improve Bing’s own search listings. Bing doesn’t deny this. As a result of the apparent monitoring, Bing’s relevancy is potentially improving (or getting […]
Buy Search Indexes Off The Shelf From 80legs
The startup 80legs launched in September of last year to make web crawling affordable and accessible to anyone that wanted to subscribe to the service. The company makes crawling the web or customized slices of it available to even small companies. More flexible than something like the Google Custom Search Engine, pricing ranges from free […]