Finding Search Engine Freshness & Crawl Dates

A reader emailed me today noticing that Google was showing a date next to his
listing, which made me think this was a good time to revisit how, when
and where search engines show crawl dates for pages. These dates are a useful
way for site owners to understand how often they are being revisited or for
anyone to “squeeze the loaf” of a search engine to see how fresh it is. Here’s a
search engine-by-search engine rundown on date display. I’ll also cover how
we’ve sadly lost crawl dates being embedded next to listings, over the years.
But that’s not all! Read now and you’ll even get a free at-a-glance table
explaining how dates are displayed. Read now — web server operators are
standing by!

Google

When you do a search, some pages may show a date below the description of a
listing.

I thought Google had long done this for certain pages that it revisits on a
super-frequent basis. And when I did a search for
cars today, I saw a date
like this coming up for the cars.com listing as shown above. An hour later, the date was
gone. I then tried that search again using a particular Google data center,
rather than whatever data center was assigned to my browser randomly. Doing the
same
search at that data center gave me dates again.

I’m checking with Google on how long dates have been showing and why they may
come and go as I saw today. I’ll postscript what I’m told at the end of this
story.

The example above shows that only some pages have dates. In contrast, the
Google Cache can
give you dates for nearly any web page.

The Google Cache allows you to view a copy of a page that is stored on
Google’s servers, rather from the website directly. (Don’t like Google caching this
for your site? Learn how to prevent it

here and

here. Don’t see a cached link option? Then the site owner is blocking
caching).

Going back to our search for
cars and the screenshot
above, you’ll see that the disney.go.com listing doesn’t have a date next to it.
To find the date the page was visited, you have to click on the link that says “Cached” under the description of that
listing. That makes the cached page load like

this. At the top of that page, you’ll see this:

This is Google’s

cache of
https://disney.go.com/disneypictures/cars/ as retrieved on
22 Feb
2007 14:34:08 GMT.

See the date and time, which I’ve put in bold? That’s when the page was last visited by Google.

FYI, before
September 2006, that date reflected the last time Google found the page to have
changed, not when it was last visited. In other words, if Google visited the
page in January 2005, then revisited it throughout the year but the page never
changed, the cached date would keep saying January 2005.

Since September 2006,
that’s been different. The date was altered to reflect the last time Google
visited the page — a good change to make. Google explains more about this on
the Google Webmaster Central blog

here, and Google’s Matt Cutts also did a video about it

here.

The options above allow anyone to see the freshness of any pages within
Google, one page at a time (as long as they are cached). What if you want to get industrial strength
and view the freshness of all your own pages at once?
Unfortunately, the Google Webmaster
Central tools don’t let you see the last time all your pages were spidered.
But that’s something they’re considering for the future. The tools will,
however, show you any problems Google had in reaching any of your pages and the
last time a crawl error happened for those pages. Using the “Crawl rate” option
found under the Diagnostics tab, you can also see a general graph of crawling
activity to your site.

There is one other type of date that you might see associated with
listings that has nothing to do when the page was crawled. Look here:

See the “3 visits – Feb 14” part? That’s coming from
Google
Personalized Search and shows that I’ve clicked on that listing 3 times,
with the last visit being on Feb. 14. My
Google Ramps Up
Personalized Search article from earlier this month explains more about how
Google Personalized Search works and can be disabled, if you don’t like it on,
as now happens much more often.

Microsoft Live Search

Microsoft Live Search operates like
Google. Some pages show dates next to them.

As with Google, this seems to happen with pages that are being spidered
frequently, but I’ll check on this. Does a page lack a date? Then click on the
“cached page” link. When the cached page loads, you’ll see something like this
at the top of it:

This is a version of

https://www.pixar.com/theater/trailers/cars/index.html as it looked when
our crawler examined the site on 2/16/2007. The page you see below is the
version in our index that was used to rank this page in the results to your
recent query.

The date (which I’ve but in bold above) tells you when the page was last spidered.

Don’t see a cached page
option? The site owner is probably blocking caching. Are you a site owner that wants to
block caching? Visit the
help area at
Live and
search for “cache” to find more info. I’d point you to the right place, but it
remains impossible to link to particular pages in Microsoft’s absurd help
system.

[Postscript: Microsoft sent this information: “We only show the last-crawl date when it is within a few days. This is a decision to draw attention to the freshest content without highlighting older content. Crawl dates for other documents can be found by looking at the cached page.”]

Ask.com

At Ask.com, you can only get dates by looking at the cached pages,
similar to how that works at Google and Microsoft. Click on the “Cached” link that you’ll see
next to the URL of a listing.

At the top of the page, you’ll see something like this with the date and time
(shown in bold below) that the page was last visited:

Below is a cache or saved snapshot of

https://www.cars.com/  as we found it on February 19, 2007 1:24:56 AM.

Yahoo

At Yahoo, you can only get dates one way, through using
Yahoo Site Explorer. You’ll
have to create an account for your web site, then authenticate your account,
then you’ll be shown last crawl date.

More than any other search engine, Yahoo makes it easy for a site owner to
see the freshness of many pages all at once. However, the huge disadvantage from a
searcher perspective is that you can’t spot check the freshness of any page you
randomly select.

The Date & Freshness Table

I love nothing more than doing tables, so let’s put everything above into a
nice one:

Ideally, I’d like to see that top row — “Dates Next To Listings?” — be
completely “Yes.” Some site owners block caching, which makes it hard to measure
freshness. Putting the dates right next to the listings makes it easy for anyone
who cares to see at a glance if a search engine is stale or fresh.

In fact, I have to laugh. I’ve been asking for this for years. On the old
features chart I used to maintain about dates, I

wrote in 2001:

Along with the page description, some search engines show the date when a web
page was created or modified. As noted above, these dates may not always be
reliable. However, they do provide a useful clue as to how fresh or stale a
search engine’s listings are. Thus, search engines that show a date deserve
praise for doing so.

That was from 2001! Nearly six years later, it’s still the case that dates
aren’t being shown. In fact, it’s a reversal. Back in 2001, the major search
engines of AltaVista, HotBot (Inktomi) and Northern Light all showed dates for
all listings right within search results. Fast forward to today, and none of the
major search engines do.

The reason is simple enough. Over time, the search engines either couldn’t
maintain freshness or didn’t want to show they were sometimes stale. So dates
either went away or never got added. C’mon gang — time to bring them back right
into the search results. If they aren’t there by default, make it an option
people can enable.

Verifying Freshness

In the meantime, there’s a favorite tactic for those search watchers who want
to track freshness. Google’s Matt Cutts
once wrote
about this back in 2005, describing exactly a technique I and others have long
used. You simply find a page that you know carries a date that’s constantly
updated. Look at the cached page and see what the time and date says on it.

But Yahoo doesn’t show a date on cached pages! No, it doesn’t, but you’re not
looking for the date that the search engine inserts. You want the date on the
page itself.

Return Of The Freshness Guarantee?

Finally, I’ll leave you with this trip down memory lane. Back in June 1999,
AltaVista once offered a freshness guarantee that was quickly broken. As I
wrote at
the time:

“AltaVista search is able to
make its Freshness Guarantee: no search site will have fresher results than
AltaVista.”

AltaVista unveiled its first
“Freshness Guarantee” back when it relaunched in June, promising that its
entire index would be refreshed at least once per month. That guarantee was
almost immediately broken, as even AltaVista President Rod Schrock admitted
when we talked recently. “We turned our attention to this new system,” Schrock
said.

OK, fair enough — they wanted
to build something even better. But this new guarantee has already been
broken, as described above. If claims like these are going to be made, then
they should actually be met. And not to meet them in the midst of a huge media
blitz is an incredible blunder.

Freshness is one important component to what makes a good search engine. It’s
not the only thing. Having fresh results means nothing if the results aren’t
relevant. And some pages don’t need to be spidered that often. But putting dates
next to listings is an easy form of search “food” labeling that can give
reassurance about a major search engines. Surely it’s time for dates to make a
comeback.