Topic: HTTP 404

Google On How They Know When To Slow Or Stop Crawling Your Web Site

Today at SMX East, Google’s Webmaster Trends Analyst, Gary Illyes shared with the audience two technical ways Google determines when GoogleBot, its crawler, should slow down or stop crawling your website. One of the more important factors with SEO is to ensure the search engine crawlers can access your web pages. If they cannot access your […]

Google Launches Improved URL Removal Tool For Third-Party Content

Google has released an improved URL removal tool that the company says was specifically designed to improve the ability to remove third-party content from its search engine. John Mueller, a Google Webmaster Trends Analyst, announced the change on the Google Webmaster Central blog. Google’s long had this tool, so what’s new? We checked with Google directly and were […]

Google Webmaster Tools Adds Smartphone Specific Crawl Errors

Pierre Far, Google Webmaster Trends Analyst, announced a new feature within Google Webmaster Tools that shows webmasters crawl errors specific to smartphone devices. The issue is: it is often hard for webmasters to detect and thus fix errors that happen on smartphone devices. Google decided to add a section to the crawl errors report that […]

New Alerts For Crawl Errors Via Google Webmaster Tools

Google announced they are now sending out Google Webmaster Tools alerts for crawl errors. Google grouped the errors in two buckets. (1) Site Error alerts for major site-wide problems (2) URL Error anomaly alerts for potentially less critical issues The Site-Wide Problems Alerts Can Include: Your DNS server is down or misconfigured. Your web server […]

How Microsoft Removes “Junk” From Bing Search Results

Dr. Richard Qian from Bing’s core search team wrote a blog post on the Bing Search blog named Bing Search Quality Insights: Reducing Junk. This is part of Bing’s ongoing effort to provide search quality insights on how Bing works. Bing here explains how they handle removing bad links from the Bing search results, and […]

Google Webmaster Tools Crawl Errors: How To Get Detailed Data From the API

Earlier this week, I wrote about my disappointment that granular data (the number of URLs reported, the specifics of the errors…) was removed from Google webmaster tools. However, as I’ve been talking with Google, I’ve discovered that much of this detail is still available via the GData API. That this detail was available through the […]

Google Webmaster Tools Revamps Crawl Errors, But Is It For The Better?

Google has just revamped the crawl errors data available in webmaster tools. Crawl errors are issues Googlebot encountered while crawling your site, so useful stuff! I originally started this article by writing that in most cases, these changes are for the better and in only a few (really maddening) cases, useful functionality has been removed. […]