Tools and software to manage AI crawlers accessing your site
Learn how to use AI crawler tools and software to optimize for AI crawlability, allow, block, or limit AI bots, audit crawling behavior, and implement controls.
Topic: KHTML
What are AI crawlers and bots?
What are AI crawlers? Learn about the different types of AI crawlers, their purposes, what they can see, and if you should allow them to crawl your site.
Google-Agent user agent identifies AI agent traffic in server logs
New user agent reveals when Google-hosted AI completes tasks like browsing or form fills, opening visibility into assisted user journeys.
Googlebot: What it is, how it works & how to optimize
Learn how Googlebot crawls, renders, and indexes your site. See every crawler type, how crawl budget works, and the technical fixes that improve visibility.
GPTBot: OpenAI releases new web crawler
You can now prevent OpenAI's ChatGPT from accessing your website, or parts of it, using robots.txt.
Google-InspectionTool – the new Google crawler for Google testing tools
Google has added a new Google crawler to its list, this one is uses for testing tools you use in Search Console or other Google properties.
New Bingbot user-agent now fully live
After a couple years of testing, Microsoft has now stopped using the old BingBot user agent.
Microsoft to go fully live with new Bingbot user-agent change in January 2023
After a couple years of testing, Microsoft said it will stop using the old BingBot user agent next month.
Bingbot user-agent change coming in the Fall of 2022
Bing will stop using the historical user-agent and transition fully to the new user-agents.
Bing to update BingBot’s user agent
BingBot is getting a new name, just like GoogleBot - are you ready?
Google to update GoogleBot’s user agent
Make sure you are prepared for the new GoogleBot user agent, which launches this December.
Google to change smartphone user-agent of Googlebot from iPhone to Android on April 18th
Google is updating their smartphone user-agent on April 18, 2016. Make sure to test your web site to ensure it will allow Googlebot in.
Apple Confirms Their Web Crawler: Applebot
Apple finally confirms they have a web crawler, primarily used for Siri and Spotlight Suggestions.
Bing Adds New Crawlers To Support Non-Responsive Mobile Designs
Bing announced a new set of mobile crawlers with Bingbot Mobile User Agents to better support and recognize mobile websites that are not responsively designed. Lee Xiong from the Bing Crawl team explained Bing has designed these new crawlers to “probe websites with a number of new crawlers with the aim to give us the best […]
Google To Replace Smartphone User-Agent To Improve Smart Phone Indexing
Google announced they are replacing the old smartphone crawler user-agent with a new one in order to make it possible for Google to index more smartphone content. Zhijian He, a smartphone search engineer at Google said that the previous user-agent “made it impossible for Google to index smartphone content of some sites, or for Google […]
SearchCap: The Day In Search, December 15, 2011
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Googlebot Identifies Smartphone Content With New User-Agent Google has announced that its mobile crawler now has a user-agent that’s specific to smartphones. The company has already had a user-agent that targets […]
Googlebot Identifies Smartphone Content With New User-Agent
Google has announced that its mobile crawler now has a user-agent that’s specific to smartphones. The company has already had a user-agent that targets feature phones. The new Smartphone Googlebot-Mobile user agent is: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +https://www.google.com/bot.html) […]