What are AI crawlers and bots?
What are AI crawlers? Learn about the different types of AI crawlers, their purposes, what they can see, and if you should allow them to crawl your site.
Topic: User agent
Web Bot Auth, Google’s new experimental method to validate authentic bots
This is just a limited test now for some AI agents hosted on Google's infrastructure but it may roll out more widely sometime in the future.
Google-Agent user agent identifies AI agent traffic in server logs
New user agent reveals when Google-hosted AI completes tasks like browsing or form fills, opening visibility into assisted user journeys.
Inside SearchGuard: How Google detects bots and what the SerpAPI lawsuit reveals
We fully decrypted SearchGuard, the anti-bot system protecting Google Search. Here's exactly how Google tells humans and bots apart.
Googlebot fraud (Fake crawlers, bot abuse & how to protect your site)
Learn how Googlebot crawls, renders, and indexes your site. See every crawler type, how crawl budget works, and the technical fixes that improve visibility.
Google adds Chrome Web Store user agent
Google added a new user agent to the Google user-triggered fetchers named Google-CWS.
Robots.txt: SEO landmine or secret weapon?
Robots.txt tells search engines what to crawl—or skip. Learn how to create, test, and optimize robots.txt for better SEO and site management.
Google adds two new Googlebot crawlers: GoogleOther-Image and GoogleOther-Video
Google said these new crawlers are to "better support crawling of binary data that may be used for research and development."
GPTBot: OpenAI releases new web crawler
You can now prevent OpenAI's ChatGPT from accessing your website, or parts of it, using robots.txt.
Google-InspectionTool – the new Google crawler for Google testing tools
Google has added a new Google crawler to its list, this one is uses for testing tools you use in Search Console or other Google properties.
New Bingbot user-agent now fully live
After a couple years of testing, Microsoft has now stopped using the old BingBot user agent.
Microsoft to go fully live with new Bingbot user-agent change in January 2023
After a couple years of testing, Microsoft said it will stop using the old BingBot user agent next month.
Google search retires Web Light, Google’s method to serve faster, lighter pages to people
Google said this feature is no longer needed these days, as providers and devices are now faster.
How paid marketers are fighting back against the Fake Web
Bots and fake users steal ad clicks, pollute audience targeting, and skew metrics. Paid marketers are learning to protect themselves.
Bingbot user-agent change coming in the Fall of 2022
Bing will stop using the historical user-agent and transition fully to the new user-agents.
Google Search launching Signed Exchanges for desktop users
This change impacts sites that utilities dynamic serving with the vary user-agent header.
Microsoft list of Bingbot IP addresses released
Knowing which bot is really from Microsoft Bing and which is not can help you decide which rogue bots to block from your site.
AMP links in Google Search break with iOS 15
This bug should be resolved soon according to a Google spokesperson.
Why the evergreen Googlebot is such a big deal [Video]
Google’s Martin Splitt discusses the effort behind making Googlebot ‘evergreen’ in this clip from Live with Search Engine Land.
Bing to update BingBot’s user agent
BingBot is getting a new name, just like GoogleBot - are you ready?
Google to update GoogleBot’s user agent
Make sure you are prepared for the new GoogleBot user agent, which launches this December.
Google testing tools now support evergreen Googlebot
After three months, the Google testing tools are now using the latest version of Googlebot.
Google Search Console adds mobile-first indexing features
Google adds the date your site switched over to mobile-first indexing to Search Console reports.
Google adds Googlebot desktop vs. mobile selector for Rich Results test
Here is a new way to test to see if your site has issues with Google's mobile-first indexing.
SearchCap: Super Bowl search ads, testing search ads & more
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Bing doesn’t show ads for queries around Super Bowl commercials, while Google doesFeb 4, 2019 by Barry Schwartz Were you interested in driving online traffic to your Super Bowl ad campaign? […]
FAQ: All about the Google mobile-first index
Here's everything to know about the Google mobile-first index.
Google to change smartphone user-agent of Googlebot from iPhone to Android on April 18th
Google is updating their smartphone user-agent on April 18, 2016. Make sure to test your web site to ensure it will allow Googlebot in.
5 Ways To Crawl A Staging Server Before Important Site Changes Go Live (To Save SEO)
You can uncover many potential SEO problems with a thorough crawl of the staging environment, but what if it's not readily accessible? Columnist Glenn Gabe shares some tips.
Google To Replace Smartphone User-Agent To Improve Smart Phone Indexing
Google announced they are replacing the old smartphone crawler user-agent with a new one in order to make it possible for Google to index more smartphone content. Zhijian He, a smartphone search engineer at Google said that the previous user-agent “made it impossible for Google to index smartphone content of some sites, or for Google […]
The Definitive Guide To Technical Mobile SEO
At SMX Advanced, I moderated a panel about technical SEO. Google’s Maile Ohye spoke about SEO best practices for technical implementation of mobile sites based on how Google crawls, indexed, and ranks mobile content and presents it to searchers on mobile devices. She also talked about Google’s recent announcement that the mobile user experience is […]
Mobile Site Configuration & The Vary HTTP Header
Yesterday, Google announced more changes to the Google mobile search algorithm, which we can expect to roll out shortly. This update is intended to improve the search experience for mobile users by “address[ing] sites that are misconfigured for smartphone users” — presumably by improving mobile rankings for sites that have been optimized according to their […]
Google Offers Tips For A Search Engine Friendly Tablet Experience (Not Recommendations)
Google posted some tips for building a search engine friendly mobile tablet experience for users and search engines. Google is clear to point out these tips are NOT specific “recommendations for building search engine friendly tablet-optimized websites.” Google does not have guidelines or official recommendations for tablet-optimized web sites but they did want to offer […]
When & How Is The New Smartphone Bot Affecting Search Results?
Even though Google has launched a smartphone bot to improve mobile search results and the user experience on smartphones, it doesn’t mean that everything in mobile SEO has changed. Google is still detecting what handset users are searching from, and adapting the results that they send based on the handset; some phones are set up […]
Tips For Multinational Mobile Site Optimisation
In previous posts in this column, I’ve covered the intricacies of multinational search from a technical SEO’s standpoint. Leveraging multinational markup while clearing up in-site duplicate content and avoiding multinational homepage calamities is no easy matter to coordinate for big site SEO. Bringing all of those strategies to bear, and then also attempting to integrate […]
Googlebot Identifies Smartphone Content With New User-Agent
Google has announced that its mobile crawler now has a user-agent that’s specific to smartphones. The company has already had a user-agent that targets feature phones. The new Smartphone Googlebot-Mobile user agent is: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +https://www.google.com/bot.html) […]
OpenOffice.org Is MIA In Bing, But It’s Not Censorship
The home page of OpenOffice.org, the well-known Microsoft Office competitor, is missing from Microsoft’s Bing search engine. While it sounds suspicious, the problem has nothing to do with Bing itself — it’s a technical problem on OpenOffice.org’s end. Ian McAnerin noticed earlier today that OpenOffice.org doesn’t show up in Bing on searches for [open office] […]
Google Writes On Mobile Site SEO Concerns & Techniques
The Google Webmaster Central blog has two excellent write ups on mobile SEO concerns and techniques. One is named Help Google index your mobile site and the other is named Running desktop and mobile versions of your site. The reason for the two different articles is that often webmasters consider smart phone enabled sites as […]
MSNBot 1.1 Gives Way to Bot 2.0
Microsoft is retiring its MSNBot1.1 and introducing its next bot: 2.0B. According to the Bing Community blog: As MSNBot 2.0b enters full-scale production, the time has come to retire our previous generation bot, MSNBot 1.1. By the end of the first week in November, you will no longer see the following user agent in your […]
Key Takeaways From SMX Advanced For In-House SEMs
Perhaps for the first time ever I’m guilty of being slack. Either that or I’ve been seriously buried (a condition I’m sure most in-house SEMs are quite familiar with). Although there was some great live blogging of SMX Advanced, nobody to my knowledge summarized key takeaways for in-house search marketers. So here are what I […]