Publishers want Congress to stop AI crawlers from hiding
More than 300 media executives will press lawmakers to require AI bots to identify themselves and disclose why they’re scraping sites.
Topic: Web scraping
Google’s /goto change: The latest challenge for third-party SEO data
Google’s /goto URLs add to a series of changes making third-party SERP data harder to collect and rank tracking less reliable.
Pagination and SEO: What you need to know in 2027
Pagination can open paths to deeper content — or quietly cut them off. See what matters for crawling, indexing, and canonicalization.
SpyFu guide: How to get the most out of this SEO tool
Master SpyFu with Backlinko's comprehensive guide. Uncover your competitors' strategies, keywords, and ad campaigns with ease.
Google confirms deploying goto URL redirects to search results links
Google said it deploys technical measures against evolving forms of abuse and this is one of those.
How to build an E-E-A-T checker with Claude Code
Build an AI-powered E-E-A-T auditor that crawls websites, applies Google's quality guidance and generates polished reports in minutes.
Reddit’s ChatGPT Search citations fell 86% in four days: Report
Reddit's share of ChatGPT Search citations fell from 3.83% to an average of 0.52% between Aug. 14 and Aug. 17. Promptwatch says.
Inside ChatGPT’s retrieval stack: The index, cache, and pages it actually reads
New research exposes the mechanics behind ChatGPT citations, including what gets retrieved, what gets read, and what ultimately gets cited.
Microsoft Clarity AI Scrape-to-Referral insights report
This report shows you the AI scrape activity against referral traffic as a metric to help you determine of that bot activity returns value.
Google Search testing forcing searchers to sign in to get more search results
This is instead of using a captcha to verify the searcher is human and not a bot.
Google loses key DMCA claims against SerpApi in scraping lawsuit
Google has 21 days to amend part of its lawsuit after the court found its SearchGuard allegations were not sufficiently supported.
Where AI agents get stuck on your site
AI agents retrieve most B2B website content, but pricing often sends them to third-party sources. Learn why and how to fix it.
ChatGPT citations change when hidden search pipelines switch
Two analyses found ChatGPT’s hidden search systems can change its sources, making citation tracking harder than answers suggest.
Publishers push Common Crawl to stop collecting content for AI training
Could AI lose a key source of training data? Major publishers want Common Crawl to stop collecting and sharing their content.
Nearly half of online articles are now AI-generated: Study
Study of 55,000 webpages finds AI-written content surged after ChatGPT and now accounts for about half of sampled articles.
Tools and software to manage AI crawlers accessing your site
Learn how to use AI crawler tools and software to optimize for AI crawlability, allow, block, or limit AI bots, audit crawling behavior, and implement controls.
What are AI crawlers and bots?
What are AI crawlers? Learn about the different types of AI crawlers, their purposes, what they can see, and if you should allow them to crawl your site.
Agentic engine optimization: Google AI director outlines new content playbook
A Google Cloud AI director is urging teams to rethink SEO for the age of AI agents, introducing a framework for how machines consume content.
AI bot traffic surged 300%, hitting publishers hardest: Report
AI fetcher bots siphon traffic and revenue from publishers, delivering real-time answers that bypass your site and cut referral visits.
Google explains how crawling works in 2026
Google went through crawling, fetching, and the bytes it processes.
The push layer returns: Why ‘publish and wait’ is half a strategy
See how AI systems process content through five entry modes — and why relying on crawling alone limits visibility, speed, and revenue
Automated traffic is growing 8x faster than human traffic: Report
Automated traffic now exceeds human activity across the web. Machines aren't just indexing or scraping it. They're actively using it.
Where to focus technical SEO when you can’t do it all
When dev resources are limited, the wrong fixes waste time. Start with architecture, indexing, and performance to drive real gains.
SerpApi asks court to throw out Reddit scraping complaint
In its dismissal bid, SerpApi argues reading public Google results isn't DMCA circumvention and warns Reddit is stretching platform power.
The five infrastructure gates behind crawl, render, and index
Where do AI systems lose confidence in your content? Discovery, selection, crawling, rendering, and indexing hold the answer.
New finding: ChatGPT sources 83% of its carousel products from Google Shopping via shopping query fan-outs
A new study reveals which data sources ChatGPT product carousels prefer. Here’s how we analyzed shopping query fan-outs and what we found.
200+ AI audits reveal why some industries struggle in AI search
Audit data across 10 industries shows access failures, weak evidence, and low utility make some sites easy for AI systems to bypass.
SerpApi moves to dismiss Google scraping lawsuit
SerpApi says Google is stretching the DMCA to protect ad revenue, not copyrights — casting the case as a fight over access to public SERPs.
Why content that ranks can still fail AI retrieval
AI systems don’t evaluate pages the way search engines do. Learn how extraction, embeddings, and structure determine reuse.
Google & Bing don’t recommend separate markdown pages for LLMs
This is a heated debate now, but it seems that the major search engines are not recommending this "strategy."
Google outlines risks of exposing its search index, rankings, and live results
Elizabeth Reid’s affidavit warns that court-ordered data sharing would expose Google's IP, invite spam abuse, and put user privacy at risk.
Why LLM-only pages aren’t the answer to AI search
Creating pages only machines will see won’t improve AI search visibility. Data shows standard SEO fundamentals still drive AI citations.
SEO in 2026: What will stay the same
Hype cycles come and go. But real SEO gains still depend on trust, crawlable data, and meeting user needs — not chasing the latest novelty.
Inside SearchGuard: How Google detects bots and what the SerpAPI lawsuit reveals
We fully decrypted SearchGuard, the anti-bot system protecting Google Search. Here's exactly how Google tells humans and bots apart.
Spam traffic in GA4: How to detect, filter & prevent it
Learn how to identify, filter, and prevent spam traffic in GA4. Fix corrupted analytics, protect data quality, and make better SEO and marketing decisions.
Google sues SerpApi over scraping and reselling Search data
Google accused SerpApi of 'circumventing security measures protecting others’ copyrighted content that appears in Google search results.'
Googlebot fraud (Fake crawlers, bot abuse & how to protect your site)
Learn how Googlebot crawls, renders, and indexes your site. See every crawler type, how crawl budget works, and the technical fixes that improve visibility.
Google vs. publishers: What the EU probe means for SEO, AI answers, and content rights
An antitrust case puts AI Overviews, opt-outs, and zero-click visibility under scrutiny amid growing debate over content use and attribution.
Cloudflare: 416 billion AI bot requests blocked since July
Google sees 3.2x more webpages than OpenAI. Cloudflare CEO Matthew Prince wants Google to separate search crawling from AI crawling.
Canonicalization and SEO: A guide for 2026
Learn how to manage duplicate content, streamline indexing, improve crawl efficiency, and more with these canonicalization best practices.