Tools and software to manage AI crawlers accessing your site
Learn how to use AI crawler tools and software to optimize for AI crawlability, allow, block, or limit AI bots, audit crawling behavior, and implement controls.
Topic: Gecko
Google-Agent user agent identifies AI agent traffic in server logs
New user agent reveals when Google-hosted AI completes tasks like browsing or form fills, opening visibility into assisted user journeys.
GPTBot: OpenAI releases new web crawler
You can now prevent OpenAI's ChatGPT from accessing your website, or parts of it, using robots.txt.
New Bingbot user-agent now fully live
After a couple years of testing, Microsoft has now stopped using the old BingBot user agent.
Microsoft to go fully live with new Bingbot user-agent change in January 2023
After a couple years of testing, Microsoft said it will stop using the old BingBot user agent next month.
Bingbot user-agent change coming in the Fall of 2022
Bing will stop using the historical user-agent and transition fully to the new user-agents.
Bing to update BingBot’s user agent
BingBot is getting a new name, just like GoogleBot - are you ready?
Google to update GoogleBot’s user agent
Make sure you are prepared for the new GoogleBot user agent, which launches this December.
Google to change smartphone user-agent of Googlebot from iPhone to Android on April 18th
Google is updating their smartphone user-agent on April 18, 2016. Make sure to test your web site to ensure it will allow Googlebot in.
Bing Adds New Crawlers To Support Non-Responsive Mobile Designs
Bing announced a new set of mobile crawlers with Bingbot Mobile User Agents to better support and recognize mobile websites that are not responsively designed. Lee Xiong from the Bing Crawl team explained Bing has designed these new crawlers to “probe websites with a number of new crawlers with the aim to give us the best […]
Google To Replace Smartphone User-Agent To Improve Smart Phone Indexing
Google announced they are replacing the old smartphone crawler user-agent with a new one in order to make it possible for Google to index more smartphone content. Zhijian He, a smartphone search engineer at Google said that the previous user-agent “made it impossible for Google to index smartphone content of some sites, or for Google […]