Topic: Web crawler

SEO

Meta Robots Tag 101: Blocking Spiders, Cached Pages & More

Last week, I covered a new command for the meta robots tag — one to prevent search engines from using Yahoo titles and descriptions. In doing that, a number of questions came up about the meta robots tag syntax itself. Google Webmaster Central has now posted “Using the robots meta tag,” providing some clarity from […]

YADAC: Yet Another Debate About Cloaking Happens Again

Sigh. Double sigh. Triple sigh. I guess now that the SEO industry has had the required twice-yearly debate about the reputation of SEO, it’s time to do the go round about cloaking once again. A quick word about cloaking has Google’s Matt Cutts trying to clarify concerns that Philipp Lenssen of Google Blogoscoped has been […]

The “About 260” Problem: Google Site: Command Glitch

Dave Naylor noted yesterday that Search Engine Watch appears to have largely disappeared from Google, at least when performing a site: command search. Today, Marketing Pilgrim notes a weirdly similar thing happening to another site and points to an interesting Webmaster World thread on what could be called the "About 260" problem that appears to […]

Finding Search Engine Freshness & Crawl Dates

A reader emailed me today noticing that Google was showing a date next to his listing, which made me think this was a good time to revisit how, when and where search engines show crawl dates for pages. These dates are a useful way for site owners to understand how often they are being revisited […]

Some SEOs Suggest To Verify But Not Submit A Google Sitemap

Google Webmaster Central is an awesome product and I personally love it. It offers ways for SEOs and Webmasters to learn more about how Google is treating their site. It also offers ways for Webmasters to notify Google of content by submitting a Google Sitemap to them. But some SEOs suggest that you should not […]

SEO

Up Close With Yahoo’s New Delete URL Feature

Since Yahoo rolled out a new Delete URL feature this week, a number of questions have come up on how exactly it works. I had time yesterday with some of the Yahoo Site Explorer team to gather answers. Thanks to Priyank Garg and Amit Kumar, who along with Tim Mayer, went through the inner workings. […]

SEO

Yahoo Fixes Lower Case Titles Bug In Search Results

A week ago, we thought that Lower Case Titles In Yahoo Coming From Anchor Text, and asked Yahoo about it. Tim Mayer of Yahoo has commented on original post at the Search Engine Roundtable saying that this has now been fixed. And, based on the examples in our post from last week, a search on […]

Are Lower Case Titles In Yahoo Coming From Anchor Text

At the last WebmasterWorld PubCon, there was a site clinic where one of the issues that came up was Yahoo not using the title tag for a specific site. Instead, Yahoo got titles for these pages in a place no one could figure out, and the titles were all in lower case. Recently, this has […]

Site Diagnostics Bug In Google AdSense Control Panel

If you have looked in your site diagnostics tab in your AdSense control panel lately, you will see a bug that displays a robots.txt error, meaning you are disallowing the Google AdSense mediapartners bot from crawling your site. The only problem is that it is displaying errors based on the Google cache URL, instead of […]

Q&A With Jimmy Wales On Search Wikia

News came out earlier this week that Wikipedia founder Jimmy Wales had a new project in mind, to build a community-driven "Google-killer" search engine. I’ve just finished talking with Jimmy about his plans. Here’s a rundown on his vision and what may come as his Search Wikia project grows over the course of the next […]