Tools and software to manage AI crawlers accessing your site
Learn how to use AI crawler tools and software to optimize for AI crawlability, allow, block, or limit AI bots, audit crawling behavior, and implement controls.
Topic: WebKit
What are AI crawlers and bots?
What are AI crawlers? Learn about the different types of AI crawlers, their purposes, what they can see, and if you should allow them to crawl your site.
Google-Agent user agent identifies AI agent traffic in server logs
New user agent reveals when Google-hosted AI completes tasks like browsing or form fills, opening visibility into assisted user journeys.
Googlebot: What it is, how it works & how to optimize
Learn how Googlebot crawls, renders, and indexes your site. See every crawler type, how crawl budget works, and the technical fixes that improve visibility.
Log file analysis for SEO: Find crawl issues & fix them fast
Unlock hidden SEO insights with log file analysis. Learn how to track crawler behavior, spot indexing problems, and optimize your site for search engines.
Google-InspectionTool – the new Google crawler for Google testing tools
Google has added a new Google crawler to its list, this one is uses for testing tools you use in Search Console or other Google properties.
Safari tightens grip on third-party interactions
Advertisers say the change creates a lot of blind spots and a lot of unnecessary headaches.
Report: Google sharing Chrome iOS search revenue with Apple
UK's Competition and Markets Authority is concerned about this previously undisclosed revenue sharing arrangement between Apple and Google.
New Bingbot user-agent now fully live
After a couple years of testing, Microsoft has now stopped using the old BingBot user agent.
Microsoft to go fully live with new Bingbot user-agent change in January 2023
After a couple years of testing, Microsoft said it will stop using the old BingBot user agent next month.
Bingbot user-agent change coming in the Fall of 2022
Bing will stop using the historical user-agent and transition fully to the new user-agents.
WordPress proposes blocking FLoC by default
It’s estimated that 40% of all sites use WordPress. If the proposal is implemented, that could be a devastating blow to Google’s ad-targeting capabilities.
The dangers of misplaced third-party scripts
Most SEO tags like the title, canonical, etc. belong in the HTML HEAD but if placed in the BODY, Google and other search engines will ignore them. Here's how you fix it.
The big PPC shifts of 2019 that will shape how we market in 2020
Automation, full-funnel campaigns, shoppable ads and privacy fueled paid search changes in 2019.
Bing to update BingBot’s user agent
BingBot is getting a new name, just like GoogleBot - are you ready?
Google to update GoogleBot’s user agent
Make sure you are prepared for the new GoogleBot user agent, which launches this December.
Google to change smartphone user-agent of Googlebot from iPhone to Android on April 18th
Google is updating their smartphone user-agent on April 18, 2016. Make sure to test your web site to ensure it will allow Googlebot in.
Bing Adds New Crawlers To Support Non-Responsive Mobile Designs
Bing announced a new set of mobile crawlers with Bingbot Mobile User Agents to better support and recognize mobile websites that are not responsively designed. Lee Xiong from the Bing Crawl team explained Bing has designed these new crawlers to “probe websites with a number of new crawlers with the aim to give us the best […]
Google To Replace Smartphone User-Agent To Improve Smart Phone Indexing
Google announced they are replacing the old smartphone crawler user-agent with a new one in order to make it possible for Google to index more smartphone content. Zhijian He, a smartphone search engineer at Google said that the previous user-agent “made it impossible for Google to index smartphone content of some sites, or for Google […]
Mobile App API As A Future Ranking Factor In Mobile Search Results
Toward the end of 2012, I read an interesting article theorizing that Google will be crawling APIs instead of mobile websites in the future. API means Application Programming Interface which allows software, including websites and mobile apps, to provide direct integration based on functions permitted by the API. Websites and mobile apps use APIs predominantly to share content […]
The Strange Explanation Of Why Windows Phone Users Lost Access To Google Maps
On Friday, Windows Phone users supposedly found themselves unable to access Google Maps. But Google said it made no sudden change on its end to cause this. What happened? It seems likely many people simple noticed for the first time long-standing redirection that Google had in place for those not using Android or iOS devices. Windows Phone […]
SearchCap: The Day In Search, December 15, 2011
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Googlebot Identifies Smartphone Content With New User-Agent Google has announced that its mobile crawler now has a user-agent that’s specific to smartphones. The company has already had a user-agent that targets […]
Googlebot Identifies Smartphone Content With New User-Agent
Google has announced that its mobile crawler now has a user-agent that’s specific to smartphones. The company has already had a user-agent that targets feature phones. The new Smartphone Googlebot-Mobile user agent is: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +https://www.google.com/bot.html) […]
SearchCap: The Day In Search, November 7, 2008
Below is what happened in search today, as reported on Search Engine Land and from other places across the web.
How Bright Is The Outlook For Chrome?
Most of the tech journalists in the San Francisco Bay Area were at Google this morning for a briefing and demo of Chrome. Danny earlier posted lots of detail on the search dimensions of Chrome. And there are quite a few posts and articles about features and first impressions.