Tools and software to manage AI crawlers accessing your site
Learn how to use AI crawler tools and software to optimize for AI crawlability, allow, block, or limit AI bots, audit crawling behavior, and implement controls.
Topic: Mozilla v. FCC
What are AI crawlers and bots?
What are AI crawlers? Learn about the different types of AI crawlers, their purposes, what they can see, and if you should allow them to crawl your site.
Google-Agent user agent identifies AI agent traffic in server logs
New user agent reveals when Google-hosted AI completes tasks like browsing or form fills, opening visibility into assisted user journeys.
Googlebot: What it is, how it works & how to optimize
Learn how Googlebot crawls, renders, and indexes your site. See every crawler type, how crawl budget works, and the technical fixes that improve visibility.
Google-InspectionTool – the new Google crawler for Google testing tools
Google has added a new Google crawler to its list, this one is uses for testing tools you use in Search Console or other Google properties.
Why I’m glad third-party cookies are dying
Third-party cookies have never been a good way to demonstrate value – not value as most businesses define it, at least.
SEO 2021 in review: Endless updates, title rewrites, GMB becomes GBP and more
Despite facing a still-raging pandemic and a slew of updates, SEOs rose to the challenge. Here’s our recap of the year’s most important SEO stories and news.
Mozilla tests Bing as the default search engine for 1% of users; Tuesday’s daily brief
Plus, ads proliferate in Amazon search results as ad prices surge.
Mozilla tests Bing as the default search engine for 1% of users
Many SEOs often forego Bing optimization in favor of its higher traffic counterpart, but this is another reason to diversify your target search engines.
SEOs rejoice: Core web vitals release moves to June; plus WordPress might block FLoC; Tuesday’s daily brief
And why you should use social media as a key component of your negotiation research
Strange bedfellows Google and Firefox renew their ‘default search’ partnership
Firefox extends 2017 deal with Google as it seeks to diversify revenues.
Ad industry groups launch effort to push for addressable media standards
The Partnership for Responsible Addressable Media includes ad industry trade groups, agencies, advertisers, publishers and ad tech firms.
Bing to update BingBot’s user agent
BingBot is getting a new name, just like GoogleBot - are you ready?
SearchCap: Amazon’s record-breaking holiday season, Google My Business updates, more
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Amazon had a record-breaking holiday season, here’s how advertisers managed it Jan 22, 2019 by Melissa Reilly The holiday season brought improved Sponsored Brand ad copy approval times but policy enforcement […]
SearchCap: Yahoo sues Mozilla, Google Lens review & celebs help Google
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: The lowdown on driving app downloads with Universal App campaignsDec 7, 2017 by Matt Lawson Google’s Universal App campaigns use machine learning to automate app promotion. Columnist and Googler Matt Lawson […]
Yahoo parent sues Mozilla for replacing it with Google as Firefox default search
Mozilla countersues and says that poor Yahoo search quality caused Firefox to lose market share.
Google to change smartphone user-agent of Googlebot from iPhone to Android on April 18th
Google is updating their smartphone user-agent on April 18, 2016. Make sure to test your web site to ensure it will allow Googlebot in.
Meet Marketing Technologist And MarTech Program Chair Scott Brinker
A conversation with MarTech’s conference chair and creator of chiefmartec.com blog
SearchCap: Local Marketing, Content Marketing & A Reader Survey
Below is what happened in search today, as reported on Search Engine Land and from other places across the Web. From Search Engine Land: Global Content Marketing for Enterprise — September 24 Webcast Sep 14, 2015 by Search Engine Land Technologies, tactics and tricks for gaining market share with exceptional content. 14 Billion Reasons You […]
Apple Confirms Their Web Crawler: Applebot
Apple finally confirms they have a web crawler, primarily used for Siri and Spotlight Suggestions.
Report: Yahoo Search Losing Firefox Users Who “Switchback” To Google
Relying on recently released StatCounter data we previously reported Yahoo search gains in the wake of the Firefox “default search” deal announced earlier this year. However a subsequent analysis by RKG’s Mark Ballard argues that many Firefox 34 users are switching back to Google as their default search engine. Ballard looked at paid clicks from […]
Free Ride? In Europe, Google Will Remain Firefox Search Default Despite No Deal
How much does it cost to be Mozilla’s Firefox search provider in Europe? Perhaps nothing — because Google will continue to be the provider there despite not having any formal deal with Mozilla, come this December. Yesterday, it was big news that Mozilla announced new deals making Yahoo the default search provider for Firefox in the US, Yandex […]
Yahoo Replaces Google As Default Search Provider in Firefox
Yahoo and Mozilla just announced a “strategic five-year partnership that makes Yahoo the default search experience for Firefox in the United States on mobile and desktop.” The companies said they will explore other potential “future product integrations and distribution opportunities to other markets.” Yahoo is thus going to displace Google the current Firefox search provider. […]
Bing Adds New Crawlers To Support Non-Responsive Mobile Designs
Bing announced a new set of mobile crawlers with Bingbot Mobile User Agents to better support and recognize mobile websites that are not responsively designed. Lee Xiong from the Bing Crawl team explained Bing has designed these new crawlers to “probe websites with a number of new crawlers with the aim to give us the best […]
SearchCap: Google’s Matt Cutts Regrets, Bing On Firefox OS & Google Doodles
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Google’s Matt Cutts Regrets Not Acting Faster On Paid Links & Content Farms In Google’s Matt Cutts latest video, he answers a question I personally asked about what he regrets, what […]
Google To Replace Smartphone User-Agent To Improve Smart Phone Indexing
Google announced they are replacing the old smartphone crawler user-agent with a new one in order to make it possible for Google to index more smartphone content. Zhijian He, a smartphone search engineer at Google said that the previous user-agent “made it impossible for Google to index smartphone content of some sites, or for Google […]
Google Notifies Sprint Of Spam Penalty; Seeks Advice In Google Help Forums
The latest large brand to be hit with a user-generated content spam penalty notification is Sprint, the large US wireless communications company. Similar to Mozilla’s penalty and BBC’s penalty, Sprint was penalized for user-generated content spam on a portion of their site that was open to anyone to post links and content. Also similar to […]
Q&A With Google’s Matt Cutts On What To Do If You Get A Manual Penalty
What to do if Google sends you a penalty notice, and you can’t figure out exactly what it’s for? Turn to Google’s webmaster help forum, says the head of Google’s Web spam team, Matt Cutts. If you’re still confused after that, you can file a reconsideration request where you might be given more details. The […]
SearchCap: The Day In Search, April 24, 2013
Below is what happened in search today, as reported on Search Engine Land and from other places across the Web. From Search Engine Land: Google Adds Green Arrows To View Cache, Similar Pages & To Share Results On Google+ Google has updated how you access the cached pages, similar results and how you share search […]
Google: That Mozilla Penalty Only Impacted One Page Out Of 22 Million
We reported yesterday that Google penalized Mozilla over user generated content. Today, we learn that it was a really, really small penalty that only impacted a single page out of Mozilla’s ~22 million webpages. Google’s head of search spam, Matt Cutts, added more to the Google thread explaining that this manual penalty was applied in […]
SearchCap: The Day In Search, April 23, 2013
Below is what happened in search today, as reported on Search Engine Land and from other places across the Web. From Search Engine Land: Google Pulls Related Searches Filter Due To Lack Of Usage Google has quietly removed the “related searches” option from the search tools menu within the Google search results page. When Google […]
Google Hits Mozilla With Spam Penalty Over User Generated Content
Have user generated content on your site? Pay attention to what those users are doing. That’s the takeaway from Google hitting Mozilla with a spam penalty this week, along with another takeaway. Despite Google’s saying it’s being more transparent about spam actions, people clearly find it hard to know what they’re in trouble for. Mozilla […]
Firefox 14 Now Encrypts Google Searches, But Search Terms Still Will “Leak” Out
Firefox 14 has officially launched today, which means all Google searches are encrypted by default. However, due to a Google loophole, the encryption will not prevent things you search for from “leaking” out to Google’s advertisers nor potentially showing up as search suggestions or in data reported to web sites through Google Webmaster Central. The Firefox […]
Google, Mozilla Renew Firefox Search Deal For 3 More Years
Google and Mozilla have struck a deal that renews their agreement making Google the default search engine in Firefox browsers. No financial terms were announced, but Mozilla’s blog post says the agreement extends the companies’ agreement “for at least three additional years.” The renewal comes about three weeks after the previous contract expired. The two […]
Googlebot Identifies Smartphone Content With New User-Agent
Google has announced that its mobile crawler now has a user-agent that’s specific to smartphones. The company has already had a user-agent that targets feature phones. The new Smartphone Googlebot-Mobile user agent is: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +https://www.google.com/bot.html) […]
SearchCap: The Day In Search, December 6, 2011
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: SMX West Super Early Bird Rates Expire Soon – Register Now, Save $400 February 28 – March 1 in San Jose, CA Attend SMX West and get expert insights and real-world-proven […]
Twitter, Mozilla Strike Deal That Bakes Twitter Search Into Firefox 8 Search Box
Twitter and Mozilla have struck a deal that makes Twitter search one of the default search options in the new Firefox 8 web browser. It’s included in the Windows, Mac and Linux versions, which can be downloaded now. As Mozilla’s blog post explains, the Twitter search option can be used to search Twitter usernames, hashtags […]
SearchCap: The Day In Search, July 22, 2011
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Bing Tests Ads Within Organic Search Results? The RKGBlog blog posted a screen shot of Bing testing search ads directly in the organic results. Honestly, I almost cannot believe it – […]
SearchCap: The Day In Search, December 1, 2010
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Google: Now Likely Using Online Merchant Reviews As Ranking Signal Earlier this week, the New York Times spotlighted how a merchant with bad reviews nonetheless was ranked well in Google. Today, […]
Microsoft To Replace MSNBot With BingBot October 1, 2010
The Bing Search Blog announced that on October 1st, MSNBot will be renamed to BingBot. Bing has been testing a beta bot since November of last year. On October 1st, the beta label will be removed from the Bot name and it will be named BingBot. You should see the following useragent in your log […]