Apple Confirms Their Web Crawler: Applebot
Apple finally confirms they have a web crawler, primarily used for Siri and Spotlight Suggestions.
Topic: Web crawler
Google To Launch New Doorway Page Penalty Algorithm
Google will take algorithmic action on more doorway pages in the near future. The new algorithm adjustment will impact these pages trying to increase their search footprint.
Google: The Mobile Friendly Update Ranking Factor Will Be Real Time & On A Page-By-Page Basis
Google tells us that the mobile-friendly algorithm launching in about a month is on a page-by-page basis and runs in real-time, which eases the concerns of many webmasters.
Google Now Supports Crawling & Indexing Locale-Adaptive Web Pages
Google adds support for web pages that dynamically change their content based on IP origin or language settings.
Google Mobile Search Descriptions Now Say If You’re Blocking Google
Yet another reason not to block JavaScript, CSS, etc from Google. It may negatively impact your snippet in the mobile search results now.
Google News Now Will Crawl & Index Images Hosted Off Your Domain Name
Google News updated their image crawl process to include images hosted off the publishers domain.
SearchCap: App Extensions Coming To Bing Ads, Apple’s Search Ambitions & How To Win At SEO
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: App Extensions Coming To Bing Ads App extensions will be available to advertisers on Bing Ads starting next spring. Timed to coincide with the removal of explicit mobile OS targeting, app […]
Apple Webcrawler: More Potential Evidence Of Search Ambitions
Apple Insider reports on the discovery of a web-crawling bot originating from Apple’s servers. It was first “outed” by developer Jan Moesen. This is what Moesen saw: Moesen reports that the bot is only crawling HTML, “not the CSS, JavaScript or image files.” Then he asks whether this is an “official Apple project” or just “someone crawling the web […]
SearchCap: BingBot Mobile, Google Penguin Rolling & AdWords Changes
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Link Building with Content: The Experienced Route to SEO – November 11 Webcast Successful SEO is not “content marketing –vs.– link building”. It’s content marketing with link building. Quality content attracts […]
Bing Adds New Crawlers To Support Non-Responsive Mobile Designs
Bing announced a new set of mobile crawlers with Bingbot Mobile User Agents to better support and recognize mobile websites that are not responsively designed. Lee Xiong from the Bing Crawl team explained Bing has designed these new crawlers to “probe websites with a number of new crawlers with the aim to give us the best […]
Google Webmaster Guidelines Updated To Warn About Blocking CSS & JavaScript Files
Google announced they’ve updated their webmaster guidelines to specifically note that blocking your CSS or JavaScript files may have a negative impact on your indexing and search rankings in Google. Pierre Far, Google’s Webmaster Trends Analyst, said the “new guideline specifies that you should allow Googlebot access to the JavaScript, CSS, and image files that […]
Up Close @ SMX: Avoiding SEO Disasters
Contributor Russell Savage recaps Mark Munroe's presentation at SMX East, in which he discussed common SEO disasters, as well as ways to detect and prevent them.
Google May Add Mobile User Experience To Its Ranking Algorithm
At Search Marketing Expo East, Google engineer Gary Illyes talked a lot about user experience and how webmasters really need to focus on that. Over the past week or so, I asked Google about this and tried to understand why Google stressed this point so much at the event. Google told us in a statement: […]
SearchCap: Google Crawling, CTR Study & Focus On The User
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Up Close @ SMX East: How Ads Influence Organic Click-Through Rate On Google If the SERPs are a zero-sum game, where drawing a click in one place takes it away from […]
Google On How They Know When To Slow Or Stop Crawling Your Web Site
Today at SMX East, Google’s Webmaster Trends Analyst, Gary Illyes shared with the audience two technical ways Google determines when GoogleBot, its crawler, should slow down or stop crawling your website. One of the more important factors with SEO is to ensure the search engine crawlers can access your web pages. If they cannot access your […]
Google Says Indexing Its Own Results Was Mistake, Will Be Fixed
Over the week, there were various reports of Google going against its own webmaster guidelines, by indexing its own search results. Last night, Google updated its robot.txt file to ensure it blocks its own search results from being in the Google search results. The guidelines read: Use robots.txt to prevent crawling of search results pages or other auto-generated pages that […]
What’s Tops For “Offers” On Google? Google Offers – Which No Longer Exists
If you search in Google for [offers] Google will return the shuttered Google Offers page as the first search listing. But if you look carefully, you will see that Google has blocked the page from spiders within their robots.txt file and the page itself says “This site is no longer being supported.” Here is a […]
Robots.txt Celebrates 20 Years Of Blocking Search Engines
Today is the 20th anniversary of the robots.txt directive being available for webmasters to block search engines from crawling their pages. The robots.txt was created by Martijn Koster in 1994 while he was working at Nexor after having issues with crawlers hitting his sites too hard. All major search engines back then, including WebCrawler, Lycos […]
Google’s Matt Cutts Explains How Google Search Handles 404 & 410 Status Codes
Google’s head of search spam, Matt Cutts, posted a video today explaining the difference in how Google handles 404 and 410 error status codes. Both 404 and 410s are errors for a web page or document not being available, however a 410 is defined as “gone” forever. So 410s are more of an explicit response […]
SearchCap: Google: News Sites May Have Seen A Decline In GoogleBot Crawling In March
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Google: News Sites May Have Seen A Decline In GoogleBot Crawling In March Google’s Matt Cutts tweeted that news related sites, such as this one, may have noticed a drop in […]
Google Fixing Reverse DNS GoogleBot Verification
Google has confirmed that there are some GoogleBot useragent spiders not properly passing verification protocol. Savvy webmasters noticed that GoogleBot over the .249.70.0 /24 IP range was not returning the proper reverse DNS verification details. The response given was “no such host is known,” but the activity webmasters noticed was that it did appear to […]
SearchCap: The Day In Search, February 17, 2014
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Starbucks & The Economist Admit To Using Google+ For SEO More Than Social Last Friday, a New York Times report on Google+ spotlighted Starbucks and The Economist as two brands currently […]
Google Offers Advice On Faceted Navigation & Infinite Scroll Web Pages
Google has offered up two very technical blog posts on SEO topics covering infinite scroll and faceted navigation search engine friendly techniques. Infinite Scroll Web Pages Infinite scroll is what you see on a lot of newer style web sites such as Pinterest, Google Image search results, many landing pages with a lot of products […]
Google To Replace Smartphone User-Agent To Improve Smart Phone Indexing
Google announced they are replacing the old smartphone crawler user-agent with a new one in order to make it possible for Google to index more smartphone content. Zhijian He, a smartphone search engineer at Google said that the previous user-agent “made it impossible for Google to index smartphone content of some sites, or for Google […]
Google’s Matt Cutts: We Don’t Use Twitter Or Facebook Social Signals To Rank Pages
Google’s head of search spam, Matt Cutts, released a video today answering the question, “are Facebook and Twitter signals part of the ranking algorithm?” The short answer was no. Matt said that Google does not give any special treatment to Facebook or Twitter pages. They are in fact, currently, treated like any other page, according […]
Google Webmaster Tools Crawl Errors Reports Now Showing Errors On Final Redirect URL
Google’s John Mueller announced on the Google Webmaster blog this morning that the Google Webmaster Tools team has updated the crawl errors report to show errors on the final URL of the redirect, as opposed to the origin URL of the redirect. What this means is that Google is now showing the errors on the […]
Google Webmaster Tools Upgrades Structure Data Markup Errors Report
Google’s Webmaster Trends Analyst Mariya Moeva announced, on the Google Webmaster blog, a new update to Google Webmaster Tools where they now show items with errors in the Structured Data dashboard. This can help webmasters better debug issues they are having with their structured data markup. Google is showing the item-level error, but the top-level […]
Google Webmaster Tools Adds Smartphone Specific Crawl Errors
Pierre Far, Google Webmaster Trends Analyst, announced a new feature within Google Webmaster Tools that shows webmasters crawl errors specific to smartphone devices. The issue is: it is often hard for webmasters to detect and thus fix errors that happen on smartphone devices. Google decided to add a section to the crawl errors report that […]
The Curious Case Of Bing Search Results In Google Search Results
Over the weekend, Bill Hartzer noticed that some Google searches returned Bing search results. As of last night, Google search results are once again Bing-free. What happened? Taking a closer look, the Bing search results weren’t www.bing.com/search URLs, which are correctly blocked by Bing’s robots.txt file. They were coming from www.bing.com/entities/search. This pattern is not […]
Google’s Matt Cutts: User Redirection Based On Location Is Not Spam
Google’s head of search spam, Matt Cutts, posted a video yesterday clearly stating the geo-location or redirecting users based on their location is not spam. Google will not consider a site that uses geo-location techniques as spam. What Google does consider spam is when you redirect GoogleBot to a web page of content that users […]
What The Government Shutdown Means For .gov SEO
As part of the US government shut down, many .gov sites have also shut down, replaced by messages like this: “Due to the lapse in federal government funding, this website is not available. We sincerely regret this inconvenience.” But one day, the government will start up again and government agencies will once again focus on ensuring […]
Google: Unlinked URLs Are A Source For Indexing New Content
John Lincoln from Ignite Visibility spotted a Google Hangout where Google’s John Mueller confirmed that Google will use URLs or domain names within content that are unlinked, i.e. without an a href attribute, to discover and index new pages of content. John went on to add that typically no PageRank or trust is flown from […]
Snowden Petition Blocked From Google? Like All Petitions, It Won’t Be When It Gets Enough Signatures
Search for “edward snowden petition” on Google to find the petition filed through the White House petitions site, and you’ll see something odd. The petition has no description, because the White House won’t let Google crawl the page. But it’s not a move against Snowden, as some might think. It’s part of how the petitions site has worked […]
Site Command Limited To 30 Results? Google Says A Temporary Domain Clustering Bug
Over the past day or so, we’ve been seeing reports that some of us can replicate ourselves, where the site command (i.e., site:www.domain.com) in Google was limited to only showing 30 results. The rumors were that with the recent domain clustering change Google pushed out this week has severely impacted the site command. Those rumors […]
Google: Adding Too Many Pages Too Quickly May Flag A Site To Be Reviewed Manually
Google’s Matt Cutts answered a question submitted by another Googler, John Mueller, on YouTube asking, “Should I add an archive of hundreds of thousands of pages all at once or in stages?” The question is, if you build out a new section of your website with tons of content – is it safe to just […]
Bing Offers Recommendations for SEO-Friendly AJAX: Suggests HTML5 pushState
Bing has announced support for HTML5 pushState as a way to implement AJAX on a site in a way that enables Bing to crawl and index the URLs and content. As Google has supported this implementation since early 2012, site owners finally have an AJAX option that can be crawled and indexed by both major […]
Google’s Matt Cutts Awarded Patent On Detecting Hidden Text & Hidden Links
A new Google patent was awarded today named Systems and methods for detecting hidden text and hidden links. The patent was filed on August 25, 2009 by Google’s Matt Cutts and Fritz Schneider. It was awarded today after being in pending status for over 3 years. The abstract reads: A system detects hidden elements in […]
Google Releases Interactive Infographic: “How Search Works”
Ever wondered how Google Search works, finds pages from across the web and decides how to list them in response to a search? If so, Google’s got a new resource designed to answer questions. Called “How Search Works,” the new area announced today is an interactive infographic that explains more about the search process, including […]
New: Bing Sitemap Plugin Helps Webmasters Create XML Sitemap Files
Bing announced, on its Bing Webmaster Blog, a new feature that helps webmasters create two types of XML Sitemaps. Bing created a server-side application that works on both IIS and Apache servers that helps you create a comprehensive Sitemap and, also, a recently updated Sitemap. The binaries can be downloaded at the Microsoft download site. […]
Is Your Local Pricing Strategy Blocking Search Engine Spiders?
eMarketer’s recent report on Global e-commerce growth showed online sales globally exceeded $1 trillion in 2012. They further indicate that global e-commerce will grow by an additional 19% in 2013, with the Asia-Pacific region surpassing North America in online sales. This reemphasizes the importance of articles like Andy Atkins-Krüger’s recent article on international pricing strategies and […]