This gives Google’s spider, GoogleBot, the ability to read comments in AJAX or JavaScript, such as Facebook comments or Disqus comments and others that are dynamically loaded via AJAX or JavaScript. In addition, this means, Google is better at seeing the content behind more of your JavaScript or AJAX.

Postscript: Google now has an official blog post up with more details.

Related Stories:

• Google Offers A Proposal To Make AJAX Crawlable
• Google Offers SEO Advice On AJAX Coding
• Google May Be Crawling AJAX Now – How To Best Take Advantage Of It
• It’s Official: Google’s Proposal For Crawling AJAX URLs is Live
• An Update On Javascript Menus And SEO
• Google Now Crawling And Indexing Flash Content

Typically, Flash has been one of those big No-Nos in the SEO space. I still do not believe any SEO would recommend anyone to build out a site completely built in Flash. However, there are many Flash sites out there that are not concerned with being indexed, either intentionally or not intentionally, but Google wants to index them.

Google has been working with Adobe to continually better index Flash content. In 2008, Google made stride in indexing Flash content and improved on that last year as well. Now, Google is able to index almost all the text and links within a Flash file.

In addition, Google said they can index the latest compatible content for Flash Player 10.1 and they are better at understanding JavaScript to embed SWF content. Finally, Google is also better at detecting and extracting meta data from Flash video files, metadata such as alternate thumbnails.

The example given by Google on how this improves the search results is by showing a search snippet for the query [2002 VW Transporter 888]. In the past, Google would not have been able to show this snippet:

Google began indexing Flash files back in June 2008. Now Google can handle SWFs in many ways, including:

• Index textual content displayed as a user interacts with the file. We click buttons and enter input, just like a user would.
• Discover links within Flash files.
• Load external resources and associate the content with the parent file.
• Support common JavaScript techniques for embedding Flash, such as SWFObject and SWFObject2. Index sites scripted with AS1 and AS2, even if the ActionScript is obfuscated.

Which is great and all, but if “the web has won”, as Vic Gundotra, VP of Engineering at Google proclaimed, then it’s not just application development that’s moved to the web. The potential consumers of these applications have moved to the web too. And Google, more than any other company, knows that search has become the primary navigation point of the web. We’ve become a searching culture and if we don’t see something in the first 10 search results on Google, we may not realize it exists. (A 2008 PEW/Internet survey found that 49% of online Americans use search engines every day, and a 2008 iProspect/JupiterResearch study found that “68% of search engine users click a search result within the first page of results, and a full 92% of search engine users click a result within the first three pages of search results.”)

Web applications need more than technology to thrive. They also need customers. And more often than not these days, those customers are acquired through search. But how well can this new world of the web that Google is ushering in at Google I/O be crawled, indexed, and ranked by search engines such as Google?

If the search engines’ ability to handle the rich internet applications (RIAs) developers have been creating over the last few years are any indication, not very well. Google in particular has recently made strides in this area, as it benefits them in their quest to organize the world’s information and make it universally accessible and useful. But years after Flash and AJAX hit the market, their searchability still isn’t ideal.

When this year’s Google I/O conference was first announced, I talked with Tom Stocky, Director of Product Management for developer products at Google, and asked about the searchability issues with some of the Google Code APIs and products. He noted that Google I/O included a session about searchability: Search Friendly Development, presented by Maile Ohye. Having heard Maile speak about developer issues before, I have no doubt that her content was top notch, but that doesn’t answer my underlying question about the general searchability of the code Google offers to developers. After all, if Google is encouraging developers to “build a business” with their APIs, they should realize that building a business is about more than application construction — it’s about the ability to acquire customers as well.

JavaScript and AJAX

For instance, Google’s AJAX APIs are created with, well, AJAX, which is notoriously difficult to index. A core issue with AJAX is that it dynamically changes the content of the page. The URL doesn’t change as the new content loads. Often, the URL is appended with a hash mark (#). Historically, a # in a URL has denoted a named anchor within a page, and thus search engines generally drop everything in a URL beginning with a # as not to index the same page multiple times.

You can see this implementation on coldfusionbloggers.org, for instance.

Click the Next button, and the URL is appended as follows: http://www.coldfusionbloggers.org/#2

Other AJAX implementations don’t append anything to the URL, but simply dynamically load new content on the page based on clicks. Take a look at an example from the Google Code Playground.

In this example, each of the three tabs (Local, Web, and Blog) contains unique content. The URL of the page doesn’t change as the content reloads. You might expect one of two things to happen:

• search engines associate the content from the tab that appears when the page first loads with the URL
• search engines associate the content from all of the tabs with the URL.

Either of these scenarios could happen, depending on how the code is implemented. What actually happens in this case is less desirable than either of those options. Because the entire tab architecture is loaded in JavaScript, search engines can’t access the content at all.

You can see this in action for an implementation showcased on the Google AJAX APIs blog. The sample site uses a tabbed architecture to organize results, but all that shows up in Google search results is “loading”:

A similar thing happens with this page that uses AJAX to provide navigation through photos:

All Google sees is “loading photos, please wait…”

A dynamic menu such as the one described here has related, but different issues. Each tab loads content from an external HTML file. Google doesn’t see that content as part of the page. Rather, it sees the content as part of those external files and indexes them separately.

There are ways to get the content indexed, of course.

• You can do away with AJAX in this instance and use CSS and divs instead.
• You can use the approach taken on the Yahoo home page. The Featured, Entertainment, Sports, and Life tabs load inline when you click on them, but if JavaScript isn’t enabled, the tabs are shown as links to separate pages. (For instance, the entertainment tab links to http://entertainment.yahoo.com/.)

You can tell that Google sees the version of the Yahoo page in which the tabs link to separate pages because this is the behavior found in the Google cache.

• You can gracefully degrade your AJAX implementation or use progressive enhancement techniques.

• You can implement a technique that Jeremy Keith describes as Hijax: return false from the onClick handler and include a crawlable URL in the href as shown below:

<a href=”ajax.htm?foo=123” onClick=”navigate('ajax.html#foo=123');
return  false”>123</a>

Of course, with an implementation like this, similar to the Yahoo home page experience, search engines may index each page individually or as variations of the same page (depending on what the AJAX code is meant to do), rather than as part of a single, comprehensive page  and when visitors enter your site through search, those individual pages will be how they first experience your site.

Cloaking?

None of these JavaScript/AJAX workarounds are considered cloaking, which can get a site banned from Google, because they don’t present content based on user agent (such as Googlebot). Rather, they present “degraded” content to any user agent without JavaScript support (including screen readers, some mobile devices, and older browsers) or “enhanced” content to any user agent with JavaScript support.

Flash and Flex

Similar issues exist with Adobe technologies such as Flash and Flex. Search engines have historically had trouble crawling Flash. Last year, Adobe made a search crawler version of the Flash player available to Google so it could extract text and links, and more recently launched an SEO knowledge center, but problems remain.

If the Flash application is built with a single URL (rather than changing URLs for each interaction), then the site visitor coming from search will always enter the site at the home page, and have no way of knowing how to get to the content. Not to mention this style of web application is difficult to share. It’s much easier to copy and paste a link to a cute pair of shoes than it is to email a friend instructions like, “go to the home page, then click shoes, then click sandals, then brown, then scroll to the third page and look in the fifth row, second pair over from the left.” Seriously, no matter how good a friend you are to me, I am not following those instructions.

The Flex framework introduces challenges similar to AJAX. It creates new URLs by adding a hash mark (#). As with AJAX, this can make things faster because new pages aren’t loading with each click, but also as with AJAX, search engines drop everything in a URL beginning with the #. Depending on your infrastructure, you might be able to remap or rewrite the URLs, but that sort of defeats the purpose of using a coding framework to begin with.

Should Google Search and Google Code Have Coffee?

I (more than many people, having previously worked in Google search) understand that Google is a big company. And I know that both the search and code teams are working hard at their core goals. And believe me, I really love the people working on both teams. I’ve had the wonderful opportunity to work very closely with many of them, and I can vouch for the fact that any lack of integration between the two is not for lack of caring. Those teams care very much about putting out quality products that their customers are delighted by.

But as can happen in big companies where everyone is working hard, there seems to be a disconnect. Google APIs and other products for developers should be built search-engine friendly right out of the box not only because they’re being built by Googlers who should therefore know better, but because searchability is vital to for a business to be successful on the web today. Code should be secure; it should function properly; it should be crawlable.

At Google I/O’s first keynote session, Vic Gundotra touted the White House’s use of Google Moderator earlier this year as an example of how useful Google tools are to developers. Sure, the AppEngine servers running Google Moderator held up, but none of the discussion could be crawled or indexed by search engines. Maybe this didn’t matter to the White House (although I would think that some American citizens might find it helpful to be able to search through the questions people had). But a small business using Google Moderator as its forum or support framework would almost certainly want that content to be found in Google searches.

Google is working on it

The Google Webmaster Central team has been providing a wealth of education around these issues to help developers build search-friendly web sites. For instance:

• Search-friendly AJAX
• Canonicalization
• Site moves

At Maile’s Search-Friendly Development session at Google I/O, Google announced two advances in their ability to crawl and index RIAs. While both of these advances are great efforts, they were driven entirely by the search team. And unfortunately, they don’t solve the issues with the Google Code APIs. Wouldn’t it be great if the new Web Elements they just announced were search-engine friendly by default?

Flash improvements

Google made improvements in crawling content in Flash files in July of 2008, but wasn’t able to extract content from external files (such as resource files). In addition, many sites loaded Flash via a JavaScript link on the home page, and since Google didn’t follow the JavaScript link, they couldn’t get to the Flash at all. Both of those things have changed as of this week. Google can now access content in external files and can follow JavaScript links to Flash applications.

Both of these are great improvements, but the advice in my original article remains. Don’t use Flash in situations where you really don’t need it. Make sure that all text and links are built into the Flash file as text and links (and not, for instance, in images), and create a new URL for each interaction.

JavaScript improvements

Google has also been crawling some JavaScript for a while. Primarily, they’ve been extracting very simply coded links. As of today, they’re able execute JavaScript onClick events. They still recommend using progressive enhancement techniques, however, rather than to rely on Googlebot’s ability to extract from the JavaScript (not just for search engine purposes, but for accessibility reasons as well).

Googlebot is now able to construct much of the page and can access the onClick event contained in most tags. For now, if the onClick event calls a function that then constructs the URL, Googlebot can only interpret it if the function is part of the page (rather than in an external script).

Some examples of code that Googlebot can now execute include:

• <div onclick="document.location.href='http://foo.com/'">

• <tr onclick="myfunction('index.html')"><a href="#"
  onclick="myfunction()">new page</a>

• <a href="javascript:void(0)" onclick="window.open
  ('welcome.html')">open new window</a>

These links pass both anchor text and PageRank.

The end of progressive enhancement?

Does this mean that web developers no longer need to progressively enhance their JavaScript? I would recommend continuing this practice. Not only does it benefit accessibility and mobile users, but search engines other than Google aren’t yet crawling JavaScript. And in any case, we will have to watch and see what happens in the Google search results before we know exactly how JavaScript content is handled.

For instance, for that Star Trek example above, the entire body code is as follows:

 <body>
    <div id="searchcontrol">Loading</div>
  </body>

The JavaScript function in the head section of the page loads the Google AJAX API code externally, and even with these latest improvements, Googlebot is still only able to interpret code on the page.

What about all those workarounds to accommodate Googlebot’s previous lack of support for JavaScript?

One issue that is worth investigating is how is Google handling <noscript> content and content when the onClick handler returns false? If a developer uses the Hijax method described above as a workaround for URLs with hash marks in them, will Google now see only the non-search-friendly version of the URL? I asked Google about these issues and they told me regarding Hijax that “we try to mimic the browser’s behavior, but it’s still possible for us to discover URLs even though the function returns false.”

As for <noscript>, Google told me that text outside of noscript is best, but they do process information inside of <noscript>. That statement is worth digging into a bit more [and I'll update this post when I have more information]. Does Google prefer text outside of <noscript> because they can now easily crawl all the text inside of JavaScript? Or is it a technique that could be perceived as potentially manipulative? And what does “process  information” mean exactly? [Updated with additional information from Google: as noted below in their help information, <noscript> is a technique that can be useful for graceful degradation, but HTML text is the best option, if possible. And as noted below, the <noscript> text should match what displays in JavaScript-enabled browsers.]

Conventional wisdom has generally been that using <noscript> to provide graceful degradation for those without JavaScript support was fine by the search engines as long as the text in the <noscript> matched the text in the JavaScript exactly. Google itself recommends it in their guidelines. Although, to be fair, there’s always been a bit of concern about this method. In Google’s words (emphasis mine):

If your site contains elements that aren’t crawlable by search engines (such as rich media files other than Flash, JavaScript, or images), you shouldn’t provide cloaked content to search engines. Rather, you should consider visitors to your site who are unable to view these elements as well. For instance:

• Provide alt text that describes images for visitors with screen readers or images turned off in their browsers.
• Provide the textual contents of JavaScript in a noscript tag.

Ensure that you provide the same content in both elements (for instance, provide the same text in the JavaScript as in the noscript tag). Including substantially different content in the alternate element may cause Google to take action on the site.

Sneaky JavaScript redirects

When Googlebot indexes a page containing JavaScript, it will index that page but it cannot follow or index any links hidden in the JavaScript itself. Use of JavaScript is an entirely legitimate web practice. However, use of JavaScript with the intent to deceive search engines is not. For instance, placing different text in JavaScript than in a noscript tag violates our webmaster guidelines because it displays different content for users (who see the JavaScript-based text) than for search engines (which see the noscript-based text). Along those lines, it violates the webmaster guidelines to embed a link in JavaScript that redirects the user to a different page with the intent to show the user a different page than the search engine sees. When a redirect link is embedded in JavaScript, the search engine indexes the original page rather than following the link, whereas users are taken to the redirect target. Like cloaking, this practice is deceptive because it displays different content to users and to Googlebot, and can take a visitor somewhere other than where they intended to go.

Note that placement of links within JavaScript is alone not deceptive. When examining JavaScript on your site to ensure your site adheres to our guidelines, consider the intent.

Keep in mind that since search engines generally can’t access the contents of JavaScript, legitimate links within JavaScript will likely be inaccessible to them (as well as to visitors without Javascript-enabled browsers). You might instead keep links outside of JavaScript or replicate them in a noscript tag.

What about paid links?

On the one hand, this is great news for the web. Google can access more content, which means searchers now have easier access as well. One potential issue is that historically, JavaScript was one of the common methods used for coding a link that was paid (an advertising rather than editorial link). Google has been recommending for some time that site owners either add a nofollow attributes to those links or use a URL redirector and block the redirect via robots.txt, but they haven’t explicitly said that the JavaScript method was no longer allowed.

As late as 2007, Google’s Matt Cutts was espousing JavaScript as a valid option for indicating links that shouldn’t pass PageRank because they were paid for. It’s likely that many sites around the web use this method for “machine-readable” disclosure that the links are advertising. In fact, one of the advertising platforms that Matt showcased as coding links properly is Quigo (now owned by AOL), which appears to still enclose its ad links in JavaScript without adding the extra step of blocking the URL redirect with robots.txt. Look, for instance, at the Sports Illustrated home page:

You can see that the sponsored links are implemented in JavaScript (and when onClick returns false, the link is disabled). The links are redirected through Quigo’s ad server: redir.adsonar.com, but this subdomain isn’t blocked with robots.txt.

What about Microsoft adCenter Content Ads? The example below is from an MSN page.

The ad links are run through a JavaScript redirect (a variation of r.msn.com, in this case, 0.r.msn.com). But this subdomain has no robots.txt. Is MSN in danger of violating the Google webmaster guidelines? Will be Microsoft adCenter be penalized for selling links? Surely not. Surely Google is working out a way to crawl more of the web, while not inadvertently penalizing large portions of it. (Although this may not be an issue in any case. Both of the examples above use a 302 redirect, which likely satisfies Google’s guidelines about coding paid links in such a way that they don’t pass PageRank.)

When I asked Google about this, they told me:

Our onclick processing is becoming more widespread, but keep in mind it’s still an area where we’re constantly improving. We already detect many ads generated by onclick events.

To prevent PR [PageRank] flow, it remains a good practice to do things like have the onclick-generated links in an area that’s blocked from robots, or to use a url redirector that’s robots.txt disallow’d. Penalties for spam techniques have been and will continue to be enforced, but as you know, we work extremely hard to minimize false positives.

Webmaster Tools Message Center already sends emails to developers to inform them when we believe that they are inadvertently violating our guidelines. Whether it’s through our blog or our tools, we’ll continue to find ways to communicate with webmasters, especially as we further innovate in our crawling capability. Processing onclicks is one step of many! :)

I entirely understand their answer, even if I might not entirely like it. They want to crawl and index more of the web and they have to keep evolving to do that. Their aim is to only penalize those sites that intentionally violate their guidelines, but they’re not going to give away the secret sauce of how they detect that intention.

But the truth is that most people who have websites haven’t heard of the Google Webmaster Tools Message Center or Google Webmaster Central blog (or Search Engine Land!). The web contains substantially more site owners than the 52,000 who are subscribed to Google’s webmaster blog. Most site owners don’t know what SEO means. One could argue that anyone who has a web site should know about these things, but most small business owners don’t know really know how to set up accounts payable either, but they’re doing accounting themselves too because they can’t afford to get expert help. And while it’s not Google’s responsibility to ensure business owners know how to run their businesses properly, it is in Google’s best interest to index all of the web. And if they change the rules in inadvertently throw innocent businesses out of their search results, they’re not reaching that goal.

Since Google itself previously recommended JavaScript as a way to block paid links, it seems a bit much for them to now expect the entire web to modify their sites now. Google should take it upon themselves to sort things out. And to be fair, they’re saying that they will. But I imagine some webmasters will be nervous about leaving things up to Google, when the risk of them getting it wrong is being removed from the index.

This isn’t a new issue

As technology on the web continues to advance, web developers will continue to confront these issues.  New infrastructure and platforms will move faster than search engines, which after all, were originally built on the concept of HTML-powered, text-based web pages. So developers will have to create workarounds and then dismantle those workarounds as search engines catch up.

This happened, for instance, with dynamic URLs. Originally, search engines had trouble with URLs that contained characters such as question marks (?) and ampersands (&). In fact, Google advised in its guidelines to avoid using &=sid until mid 2006.

To get around this, some sites encoded their URLs to appear static. This Sitepoint article on dynamic URLs in 2002 explained:

For example, the following URL contains both “?” and “&,” making it non-indexable:

http://www.planet-source-code.com/vb/scripts/ShowCode.asp?
lngWId=3&txtCodeId=769

Below, it has been made search engine-friendly (all “?” and “&” and “=” characters replaced with alternate characters):

http://www.planet-source-code.com/xq/ASP/txtCodeId.769/lngWId.3
/qx/vb/scripts/ShowCode.htm

Dennis Goedegebuure of eBay explains in his blog that eBay employed this technique:

In 2004 search engines were not smart enough to read dynamic URL’s. Especially those URL’s that had a lot of parameters in them to determine sort order or aspects of the product search for shopping sites were a problem to get these indexed. Replacing the dynamic parameters like & or ? with static delimiters was one technique back in the days to make a dynamic URL static for the search engines to crawl.

Now fast forward to 2009, Search Engines have become much smarter and are now able to understand dynamic URL with parameters much better. Last week they even announced their new canonical tag to help website owners to avoid duplicate content issues when it comes to sort order.

In fact, Google is now so good at interpreting dynamic URLs that use traditional patterns that Maile Ohye used eBay as an example of what not to do in a presentation at SMX West:

[At] SMX West a Google rep presented on URL structure. One part of her presentation was about MAVRICK URL’s, and in particular the long and complicated url’s you sometimes see on the Interwebs. i.e. used in her presentation:

http://shop.ebay.com/items/_W0QQ_nkwZipodQQ_armrsZ1QQ_fromZR40QQ_mdoZ

Things on the web have evolved so much that an implementation built entirely to ensure that pages could be crawled by search engines is now being used as an example of what not to do if you want pages to be crawled by search engines!

Continuing the conversation

Clearly, Google and the other major search engines want to crawl and index all of the content on the web. That is, after all, why they continue to evolve their crawlers to adapt to changing technology. And clearly, they want to help site owners and web developers build sites that can be easily found in search. And while it may seem like I’m singling Google out, I’m focusing on them now only because of their messaging of building a business with Google APIs at Google I/O. In truth, Yahoo and Microsoft are also aggressively courting developers and their developer groups seem to be just as disconnected from their search teams. Microsoft’s IIS versions 5 and 6, for instance, are configured to implement redirects as 302s by default, when for search engine value, redirects should be 301s. (Newer versions provide more search-friendly configurations.)

Two upcoming events will provide opportunities for us to continue the discussion about search-engine friendly web development with Google, Microsoft, Yahoo, and Adobe. Next week at SMX Advanced in Seattle, we’ve got a whole set of things for developers including lunch discussion tables and an after-hours Q&A with Adobe, and Matt Cutts will be doing a Q&A session where we can ask him all about the ripple effects of these advances. The following week in San Francisco, Jane and Robot is holding a search developer summit. Again, all the reps will be on hand for some in-depth technical discussion. We’ll also have beer, which we all just might need.

More than three months after Google’s announcement, has anything really changed?

That’s the question Atlanta-based SEO Brian Ussery recently set out to answer. After conducting several case studies, Brian published the results in an article titled 2009 Google Flash SEO.

Here’s a brief summary of Brian’s tests:

• Does Google correctly associate Flash content with the right “parent URL”?
• Can Flash files accrue PageRank?
• How does Googlebot handle #anchors in Flash-based link URLs?
• Does Google translate text found in Flash files?

You’ll have to read Brian’s article for the answers to those questions. He also shares some Flash SEO tips at the end.

This situation hasn’t been ideal for searchers either, as this limitation has kept them from seeing potentially great matches for their queries because they’ve been locked away in Flash files.

According to Adobe and Google, all of that is changing. Google is launching what they tell me is a “deep algorithmic change,” augmented by Flash reader technology supplied by Adobe, that enables them to “read” Flash files and extract text and links from it for better indexing and ranking. This could be great news for both site owners and searchers.

Below, more details about how it all works, as well as some caveats for those who see this development as a Flash panacea and think they no longer have to ensure their Flash applications are search engine friendly.

Google can now crawl and index Flash files

Google has been working on improving how they crawl and index rich content (such as Flash and JavaScript) for some time, and in fact have been able to extract some text and links from Flash files for a while. However, their methods weren’t perfect, and they tell me that this new technology from Adobe makes Google’s algorithms “less error prone” and enables them to access content created in any version of Flash in a variety of languages.

Adobe is providing a Flash player to (some) search engines

Adobe says they have developed an optimized Flash player for search engines and are collaborating with both Google and Yahoo!. Yahoo! has not yet implemented the technology, although they stated that “Yahoo! is committed to supporting webmaster needs with plans to support searchable SWF and is working with Adobe to determine the best possible implementation.” Adobe hasn’t made the technology available to Microsoft’s Live Search, although they say they are “exploring ways to make the technology more broadly available” to “help make all SWF content more easily searchable.” Adobe didn’t comment on whether the fact that Microsoft developing competing Silverlight was a factor in their decision not to collaborate with Microsoft Live Search for this initial announcement.

A big step forward

Previously, Google’s help documentation has warned against the use of Flash-only sites:

“In general, search engines are text based. This means that in order to be crawled and indexed, your content needs to be in text format. This doesn’t mean that you can’t include images, Flash files, videos, and other rich media content on your site; it just means that any content you embed in these files should also be available in text format or it won’t be accessible to search engines.”

They have suggested using Flash sparingly or using a method such as Scalable Inman Flash Replacement (sIFR) to provide an HTML source that can be rendered as either Flash or non-Flash.

That seems to have changed. The help documentation hasn’t been updated, but the post on the Google Webmaster blog says that Googlebot can now extract textual content and links so Google can better crawl, index, and rank the web site.

Both Google and Adobe stressed to me that this is a big win for both site owners and searchers and that it should improve relevancy in search results. They noted that Flash developers don’t have to do anything in their applications to make this new technology work for their sites.

This is certainly great news for the web, as it’s a sign that search engines, which are the primary method of navigating the web, are evolving beyond text to take into account newer web technologies.

Just how much will this change impact search relevance? It’s hard to say until we see the changes, which Google says may take time to percolate through the pipeline. In particular, they note that snippets, the descriptions that display under search results, will be improved. Before, Google often couldn’t extract any content from a Flash file, so the description for a Flash page was often empty or would consist of the only text available from the file, such as the Flash version or the word “loading.”

But although Adobe’s press release talks about “dramatic” improvements in search results and more relevant listings for “millions of RIAs” (rich internet applications), neither Adobe nor Google could give me numbers about how many more pages Google was now about to crawl and index and how much this has impacted search results.

A quick look at how SWF files are currently indexed shows that there’s a lot of room for improvement, so this may indeed be big news for search.

Flash developers should still spend time on Search Engine Optimization

However, this isn’t the perfect solution that it may seem. Adobe assures developers that “RIA developers and rich Web content producers won’t need to amend existing and future content to make it searchable — they can be confident it can now be found by users around the globe.” But that’s not entirely true, particularly for Flash pages that have little textual content.

Only text and links are affected As Danny Sullivan noted last year when word of Google’s work in this area first came up, most Flash content isn’t made up of primarily words. It’s made up of images, video, and animation, and none of that will be surfaced in search results with this advancement. Google’s new Flash algorithms extract text and links only. Everything else is still a black box.

Associate a unique URL with each unique piece of content In addition, the searcher experience is better served by Flash implementations that provide a unique URL for each set of content. Some Flash implementations dynamically load text as the user interacts with the application, but the URL remains the same. In this scenario, Googlebot can now follow those interactions (in a limited way) and if the URL doesn’t change, then all content that is dynamically loaded as the interactions progress is associated with a single URL.

Adobe says the Flash player it is providing to search engines “allows their search spiders to introspect and navigate through a live SWF application as if they were virtual users. The Flash Player technology, optimized for search spiders, enables the ability to traverse and parse all of the different paths in a SWF-based RIA, similar to traversing multiple pages in a standard web application.”

This means that if the content that is dynamically loaded into the Flash application from the fifth interaction matches a searcher query, that Flash application may be served in the search results. But when the searcher clicks over to that result, the content won’t be found on the page. The searcher will have to interact with the application until that content is loaded. Searchers may instead feel frustrated and abandon the page. For the best user experience and higher conversion rates from search, Flash developers should be careful to avoid this situation by creating distinct URLs for each piece of content. This implementation helps the Flash site be more viral as well, as users can email, Digg, and otherwise share the content more easily.

Google acknowledges this scenario may not be an ideal searcher experience, but points out that other non-HTML file formats such as PDFs have the same limitations. When a searcher clicks through the Google search results to a PDF file, the content that matched the query may not be on the first page of that PDF and the searcher has to scroll through the file to find the desired content. Google notes that just as they flag PDF files to alert searchers that the result is non-HTML, they do the same with Flash file, as shown below.

What this means for SEOs

Flash has often been a source of frustration for SEOs who argue that text should be in HTML, with Flash used for non-textual content, such as video illustrations. Can SEOs now remove the “review Flash implementation” line from their checklists? Probably not. However, it should be easier for SEOs to work with Flash-based sites going forward.

SEOs should keep in mind that these new algorithms don’t take into account any meta data or formatting markup in the Flash file and, for now, Google’s cache won’t show a representation of the extracted text so site owners can’t verify what is actually being crawled by viewing the cached copy. In addition, since Googlebot doesn’t execute most JavaScript, Google won’t crawl or index any Flash executed via JavaScript. Any external sources that the Flash file loads will be indexed separately, rather than as part of the Flash file. And as noted earlier, all non-textual content will remain uncrawled. This new Flash support covers all languages other than bidirectional ones (Hebrew and Arabic) and all versions of Flash.

What this means for accessibility and usability

Flash developers should continue to think about not only how well their applications can be found in search, but how usable and accessible they are.

Eric Wittman, director of platform distribution and business development at Adobe, told me that Flash web sites can be built for usability and accessibility. He noted that 98% of desktop computers have Flash support, although he acknowledged that he didn’t know how many have Flash blockers installed, and he didn’t provide numbers on the percentage of mobile devices that don’t support Flash.

He noted that screen reader support has been available since Flash Player 6 and that the newer Flex framework includes support for accessibility.

At the recent Developer Day at SMX Advanced, we talked a lot about making Flash applications accessible and search-engine friendly using graceful degradation, and both sFIR and SWFObject came up several times as good methods for ensuring this. I find that still to be good advice even in light of this announcement.

Overall, this announcement is great news for both content owners and searchers. Web developers are becoming more focused on architecting their sites to ensure they can be found in search engines, as search has become one of the primary acquisition channels online. As web technologies evolve, it’s important for search engines to evolve as well to ensure they provide the most relevant results for searchers. I’m eager to see how substantial a change this proves to be, although web developers should continue keeping search-friendly practices in mind when developing sites.

Scalable Inman Flash Replacement (sIFR) sIFR is a technique that uses Javascript to read in HTML text and render it in Flash instead. The essential fact to focus on here is that the method guarantees that the HTML content and the Flash content are identical. The intended use for this is to render text in an anti-aliased font. This can provide a great improvement in the presentation of your site.

Don’t make too extensive use of the technique. According to sIFR’s creator, Mike Davidson, you shouldn’t try to redo your whole page with sIFR. He says that “sIFR is for heacdlines, pull quotes, and other small swaths of text.” Overuse the technique and suddenly your page will take a long time to load.

How do search engines view this? In July of 2007, Google’s Dan Crow went to a Search Engine Marketing New England (SEMNE) event. Dan, who is head of Google’s crawl team, said that as long as this technique is used as it is intended, that Google was fine with it.

The key thing that drives this statement is that the HTML version of the text is the input used by sIFR to create its output. It is not possible for them to be different. For this reason, this technique is not susceptible to spamming, and there is no reason for a search engine to worry about its implementation.

SWFObject The basic underlying concept with SWFObject is similar, but the implementation is noticeably different. SWFObject starts by detecting if a user agent (a browser or crawler) is capable of executing Flash. If they are not, then the pure HTML version of the given block of content is shown. However, if the user agent is capable of executing Flash, the Flash version of the content is shown instead.

Unlike sIFR, this method does not guarantee that the HTML and the content in the Flash are the same. SWFObject does not reference the text in the HTML at all. It simply runs a pre-compiled Flash movie in place of the HTML.

This technique could conceivably be used by spammers to show one type of content in Flash and another in HTML. In the extreme case, you could have information on travel to a Caribbean island in HTML replaced by a Viagra ad in Flash. This is, of course, not its intended purpose.

At the same SEMNE event, Dan Crow indicated that this technique was “dangerous.” Even though this technique could be used for entirely legitimate reasons (e.g., the same purpose as outlined for sIFR above), there is no way for Google to detect that.

In talking with other search engine representatives, I have also heard it suggested that their concept of a clean implementation of SWFObject would depend on the content in the Flash and the HTML being identical.

In practice, it is really difficult for a search engine to ban or punish a site for using such a technique without human review. I am not saying that sites don’t get hurt by the use of this technique, but there is no way that the search engines could simply punish sites based on the degree of use of SWFObject without creating a great outcry.

Simply put, it’s a legitimate technique for providing accessibility for those with poor eyesight, and providing content to users who have Flash disabled. I think one of the bigger challenges is that you end up with the obligation to craft well thought out implementations in two different mediums at the same time, one in Flash, and one in HTML.

Since the mediums are quite different, you would be tempted to do things quite differently. As noted above, the search engines indicate that they want your HTML content and Flash content to be identical. However, Flash will sometimes lead you down a path to do things quite differently.

This is a good thing, in terms of creating the best user experience with such a site. Search engine comments aside, my own opinion is that a search engine would be hard pressed to justify implementing a penalty on a site if the content of the HTML and the content of the Flash were substantially similar (as opposed to identical).

Note, however, that my opinion notwithstanding, Google has not come out in support of implementing similar but not identical content, and in fact warns that if sites present text in the HTML that is not present in the Flash, the site could risk violating the Google webmaster guidelines.

No Need For A Flash War After All

For a long time SEOs have stood against Flash as a technique. With sIFR and SWFObject, you can implement Flash on your site and still be search engine friendly. Certainly, this brings some additional challenges due to the need to implement the content twice when you use SWFObject, and a certain amount of search engine risk.

Even with SWFObject, I think your risk is low if your implementation is truly clean, but the risk is not zero. The best thing to do to minimize that risk is make your HTML implementation as close to the Flash implementation as possible.

Eric Enge is the president of Stone Temple Consulting, an SEO consultancy outside of Boston. Eric is also co-founder of Moving Traffic Inc., the publisher of Custom Search Guide

Unfortunately, that’s not the case. In fact, these technologies are inherently unfriendly to search engine spiders. So, if you intend to harness Web 2.0 technologies for wider syndication, increased conversion, improved usability and greater customer engagement, you’d better read on or you’ll end up missing the boat when it comes to better search engine rankings.

When it comes to AJAX and Flash, the onus is on you to render them search engine friendly. The major search engines just can’t cope with these Web 2.0 technologies very well at all.

Flash not friendly

Let’s start with Flash, a technology with which many of us already are familiar. Some search engines, including Google, have rudimentary means of extracting content and links from Flash. Nonetheless, any content or navigation embedded within Flash will, at best, rank poorly in comparison to a static, HTML-based counterpart, and at worst, not even make it into the search engine’s index.

Google’s view on Flash is that it doesn’t provide a user-friendly experience. Flash is wholly inaccessible to the vision-impaired, unrenderable on devices such as mobile phones and PDAs, and can’t be accessed without broadband connectivity.

In particular, Google frowns on navigational elements presented exclusively in Flash. Given this stance, Google isn’t likely to make big improvements on how it crawls, indexes and ranks Flash files anytime soon. So, it’s in your hands to either replace those Flash elements with a more accessible alternative like CSS/DHTML or to employ a Web design approach known as “progressive enhancement,” whereby designs are layered in a concatenated manner to provide an alternative experience for non-Flash users. This way, all users, including search engine spiders, will be able to access your content and functionality.

An example of progressive enhancement in action can be found at Amazon.com’s “Create Your Own Ring” on the Web. Simply turn off the JavaScript capabilities in your browser and build your ring—with or without the Flash interaction. All customers are equally served.

Problems with AJAX

AJAX poses similar problems to spiders as Flash does. That’s because AJAX also relies on JavaScript—that’s what the “J” in AJAX stands for, after all (the complete acronym stands for Asynchronous JavaScript and XML). Search engine spiders can’t execute JavaScript commands (or Java either, for that matter). AJAX can be used to pull data seamlessly in the background onto an already loaded Web page, sparing the user from the “click-and-wait” frustrations associated with more conventional Web sites. It’s a great timesaver for users, but the additional content that’s pulled in via AJAX is invisible to the spiders unless it’s preloaded into the page’s HTML and simply hidden from the user via CSS.

Here, progressive enhancement renders a non-JavaScript version of the AJAX application for spiders and JavaScript-incapable browsers. A low-tech alternative to progressive enhancement is to place an HTML version of your AJAX application within noscript tags (see TheCleanerMovie.com for an example).

Other options include rendering static HTML pages from product searches, as the vertical shopping engine Become.com does. Google’s guidelines warn that your search result pages must provide value to users to warrant inclusion in its index. So, extra care must be taken if employing this approach.

Widgets, the mini applications webmasters are encouraged to place on their sites to pull data from an external source, are in most cases inaccessible to search engine spiders. Most widgets are built in search engine unfriendly Flash or AJAX.

A well-loved widget in the blogosphere is Eurekster’s “Swicki,” with its “What’s Popular” buzzcloud, which you may have seen in the sidebars of popular blogs like TechCrunch. The “What’s Popular” buzzcloud. Under our tutelage, Eurekster made its widget more search engine friendly and reaped the benefits with a huge influx of search-referred traffic.

As you head down the road of Web 2.0, just remember that user-friendly doesn’t readily translate into search engine friendly without some assistance. But know that help is available, particularly in the form of progressive enhancement, when you embrace this brave new World Wide Web.

Stephan Spencer is founder and president of Netconcepts, a 12-year-old web agency specializing in search engine optimized ecommerce. He writes for several publications and blogs at the Natural Search Blog. The 100% Organic column appears Thursdays at Search Engine Land.

The meta keywords tag is one of several of meta tags that you can insert into your web pages to provide search engines with information about your pages that isn’t visible on the page itself. For example, my Meta Robots Tag 101: Blocking Spiders, Cached Pages & More article covers how you can use a different meta tag — the meta robots tag — to block pages from being indexed. Users don’t see this information (unless they look at your source code), but search engines do.

Meta Tags & Your Header

Meta tags go within the header area of your web pages. A typical head might look like this:

<head> <title>Welcome To Shoe Central!</title> <meta name=”description” content=”All the best prices on shoes!” /> <meta name=”robots” content=”noodp” /> <meta name=”keywords” content=”shoe, shoes, shoee, shos, footwear” /> </head>

The header is the section that begins <head> and ends </head>. Between those elements, in our example, you have these tags:

• Title: The text here becomes the title that is shown in search engine listings, in most cases.
• Description: The text here is text that search engines sometimes use as a description for your web page when listing it (a meta tag lesson for another time).
• Robots: This particular tag is configured to ensure that the page isn’t described using the a description that the Open Directory might have for it (Meta Robots Tag 101 explains this more).
• Keywords: This tag is the topic of this article, so read on!

History Of Meta Keywords

I’ve long written about search engines and meta tags, but I have never been able to pin down exactly who created the meta keywords tag. There’s a December 1995 internet draft memo that’s the earliest and most authoritative mention of the tag I know of. It says:

<META HTTP-EQUIV= “Keywords” CONTENT= “Italy Product, Italy Tourism”>

The spaces between a comma and a word or vice versa are ignored….

These ‘keywords’ were specifically conceived for exhaustively and completely catalogue the HTML document. This allows the software agents to index at best your own document. To do a preliminary indexing, it’s important to use at least the http-equiv meta-tag “keywords”.

Sounds good, right? Like this is designed for the search engines to use? The issue is that HTML specs like these (especially drafts) are not necessarily used by the search engines. They can use them, ignore them or build upon them as they see fit.

As it turns out, several of the major search engines got together in May 1996 to talk about meta data. That meeting gave birth to a common standard for the meta robots and the meta description tags. As for the meta keywords tag, it was discussed, but no specification emerged.

Despite no specification, both Infoseek (later Go.com, these days no longer crawling the web) and AltaVista (now owned and powered by Yahoo) offered support for the meta keywords tag in 1996. If you looked at their help files at the time, they encouraged site owners to use the tag. Inktomi (now owned by Yahoo) also provided support when it began operations later in 1996, and Lycos (no longer crawling the web) added support in 1997.

That year — 1997 — was the last year that the meta keywords tag enjoyed support among the majority of major crawlers out there (4 out of 7 – Excite, WebCrawler and Northern Light, also crawling the web that year, did not support it).

Support Dies Off

When new search engines emerged in 1998, such as Google and FAST, they didn’t support the tag. The reason was simple. By that time, search engines had learned that some webmasters would “stuff” the same word over and over into the meta keywords tag, as a way of trying to rank better. At the time, search engines didn’t rely so heavily on link analysis, so page stuffing like this was more effective. Alternatively, some site owners would insert words that they weren’t relevant for.

In July 2002, AltaVista dropped its support of the tag. That left Inktomi as the only major crawler still supporting it, causing me to somewhat famously in the SEO world to declare the tag dead, since it was no longer a major ranking factor for even Inktomi:

Traffick.com’s Andrew Goodman wrote recently in an essay about meta tags, “If somebody would just declare the end of the metatag era, full stop, it would make it easier on everyone.”

I’m happy to oblige, at least in the case of the meta keywords tag. Now supported by only one major crawler-based search engine — Inktomi — the value of adding meta keywords tags to pages seems little worth the time. In my opinion, the meta keywords tag is dead, dead, dead. And like Andrew, good riddance, I say!

Since that time, Inktomi was rolled up into Yahoo, which continues to support the meta keywords tag as part of its Yahoo search engine. Or does it?

Search Engine Rep Confusion

Last month, I moderated a panel of search reps when that perennial favorite question came up during the session. Who supports the meta keywords tag?

Sigh. But if this question still coming up wasn’t depressing enough, then the search engine reps starting responding with a load of confusion. To paraphrase:

No, we don’t support it. Well, we read it. We read it, but it doesn’t matter. Actually, maybe we don’t read it.

Even Evan Roseman from Google said at one point that Google reads the meta keywords tag, suggesting no doubt to some that Google uses the tag.

To be clear, Google doesn’t. I’ll prove it further below, but it doesn’t, OK?

I gave Evan (hopefully) some good humored hassle afterward for saying this. He’s at least the second Google rep to declare this on panels I’ve moderated in as many years, and the problem is that the engineers (from any of the search engines) often take the question too literally.

Indexing Versus Retrieval Versus Ranking

To understand, let me talk about three different things a search engine does when it crawls and lists your page:

• Indexing: This is where the search engine effectively makes a copy of your page. The search engine is going to read and store the HTML content it finds — all of it. Evan was right when he said that the meta keyword tag is indexed by Google. Google knows that the tag exists and has recorded what’s in it. But that doesn’t mean it does anything else with it.
• Retrieval: This is where the search engine finds all the matching documents relevant for what you searched for. Most of those documents will actually have the words you searched for on them, in the sections that the search engine searches against (there are some exceptions, such as when anchor text is used to find pages. Google Now Reporting Anchor Text Phrases, Google Kills Bush’s Miserable Failure Search & Other Google Bombs and Google Declares Stephen Colbert As Greatest Living American explain more about this). While the search engine has recorded the entire page, it won’t search against everything indexed for retrieval. In other words, Google will look to see if words you searched for appear in the body area of a document, but it will NOT look in the meta keywords tag for matching words. The keywords tag, while indexed, is not used for retrieval at Google. At Yahoo, it is.
• Ranking: This is where the search engine looks at all those documents retrieved for a search and puts them in order of most importance, according to its algorithm. Retrieval (or what information research professionals call “recall”) is about finding everything). Ranking (or what the IR folks call “precision” — see Tim Bray’s excellent On Search: Precision and Recall document) is about getting the best stuff up to the top. Yahoo, while using the tag for retrieval, really doesn’t assign much weight to it for ranking.

Testing For Retrieval

Back to my panel experience. Since the reps were unclear, I declared to the audience that I’d just go out and test it again myself. It’s literally been about five years since I’ve last tested the tag, because I (and many others) feel it is so useless. There are better things to do with our time. But since that question needs a big old stake to the heart, I rolled up my sleeves and got cracking.

On the Search Engine Land home page, I inserted this meta keywords tag:

<meta name=”keywords” content=”qiskodslajdmnkd, ddakaieciuaj jkdalladpaoaw, wdaopeqndlkakljad” />

I had searched for all of these words on the four major search engines of Google, Yahoo, Microsoft and Ask and found no pages that matched. If these search engines made use of the meta keywords tag, I’d know in short order, if my page started coming up.

The tag went up on August 28. I then needed to wait until I could see each search engine had the most current version of my page (Squeezing The Search Loaf: Finding Search Engine Freshness & Crawl Dates explains more on how to do this).

Google: No

It took two days, until August 30, for Google to show the latest version of my page in its index. I searched for each of the words, and my home page didn’t come up. The meta keyword tag was not used for retrieval and thus not supported.

Microsoft Live: No

It took five days, until September 2, for Microsoft to show a version of my page with the meta keywords tag on it. As an aside, Microsoft is kind of annoying. It will say something like this in the cached copy of the page:

This is a version of http://searchengineland.com/ as it looked when our crawler examined the site on 9/2/2007. The page you see below is the version in our index that was used to rank this page in the results to your recent query. This is not necessarily the most recent version of the page – to see the most recent version of this page, visit the page on the web.

If you glance quickly at the date, you might think the page has been revisited fairly recently. But as the text explains, it might be older. Indeed, when I looked on September 2 (as is the case today), the copy of the page in the index was as of August 30, as I could tell from the stories shown.

As with Google, I searched for each of the words, and my page didn’t come up. The meta keyword tag was NOT used for retrieval and thus not supported.

Yahoo: Yes

It took two days, until August 30, for Yahoo to have my latest page. Searches there did bring up the home page for all words. So the meta keywords tag IS used for retrieval.

Ask: Yes

Ask took the longest to show the most current version of my page, not reflecting the changes until today. Actually, when I look at the cached copy even now, it says that the page is from August 13 and uses a redirection URL rather than my http://searchengineland.com address.

Still, I can tell Ask has a version with the meta keywords tag on it since I’m getting back my home page when searching for words in that tag. As with Yahoo, the meta keywords tag IS used for retrieval.

Should You Use It? Sure, For Misspellings

So there you have it — half of the major crawlers (Yahoo & Ask.com) DO support the tag. Should you begin using it? My advice would be only for misspellings and really unusual words.

As explained, the tag can help with retrieval. A word in the tag is treated as if it were a word visible on the page itself. Now that’s handy for misspellings. For example, say you’re writing about Basset hounds. You suspect some people might misspell the name as Bassett hounds, adding an extra T. You could misspell the word yourself on the visible page, but that makes you look bad. You could insert the word and then try to hide it using CSS styles or putting it in the same color as the page background. But this type of “hidden” text is generally against search engine guidelines.

Enter the meta keywords tag. Just do this:

<meta name=”keywords” content=”bassett” />

Now you’ve got the misspelling on your page in a “legal” means that will be read by Yahoo and Ask. You’re still out of luck for Google and Live.com, but two out four ain’t bad.

But I Want To Rank!

What about ranking better with the tag. I mentioned already that many experienced SEOs don’t find it useful. Believe me, if just putting a single word into that tag was going to rank your page better, everyone would be doing it. Instead, search for anything on Yahoo or Ask. You’ll see plenty of pages ranking well for words without those words appearing in the meta keywords tag. And if you do see the words in the tag, it’s more due to coincidence — the words also appear in the body copy, in the title tag and often in links pointing at the page. The words in the meta keywords tag aren’t the primary reason the page is ranking well. Promise.

Back to our Basset Hound example. Sure, you can add the correct spelling to your meta keywords tag. Go ahead, if you want. Just understand that it is not likely to make you rank any better than if you didn’t include it at all. Moreover, beginners are especially likely to spend far too long worrying about getting the “right” words in the meta keywords tag rather than just writing good body copy.

Comma Conundrum

One of the most common questions I used to get way back in the old days was over using commas in the meta keywords tag. Consider these options:

1. <meta name=”keywords” content=”bassett, hound, hounds, basset” />
2. <meta name=”keywords” content=”bassett,hound,hounds,basset” />
3. <meta name=”keywords” content=”bassett hound, bassett hounds, basset hound, basset hound” />
4. <meta name=”keywords” content=”bassett hound,bassett hounds,basset hound,basset hound” />
5. <meta name=”keywords” content=”bassett hound bassett hounds basset hound basset hound” />
6. <meta name=”keywords” content=”bassett hound basset hounds” />

Sigh. See why I hate this tag so much, when I’ve had to deal with people wondering about commas and spaces and variations like this. Let’s take it from the top, as to the motivations behind these versions:

1. This is someone who thinks that each word should be on its own, separated by a comma and with a space in front of the next word.
2. This is someone who thinks that getting rid of the spaces means they can squeeze in more words.
3. This is someone who thinks that if there are particular phrases they want to be found for, those phrases should be together and set off by commas.
4. As with three, but losing the spaces to squeeze in more words.
5. Similar to three but thinking you don’t need commas at all.
6. This is Mr. or Ms. Paranoid. They’re concerned about saying any word too often. So they lose the commas, restrict repetition and hope that proximity will help (IE, put “basset” behind “hound” rather than in front and maybe you’ll still show up for “basset hound.”

Which way should you go? I’d suggest number three, for these reasons:

• Yahoo has long recommended using commas and in particular supported them as a way to separate out distinct terms for those in their paid inclusion programs. I’ll update this page with the latest advice, but commas still seem to make sense.
• Spaces just make things look nicer, and you shouldn’t be shoving a ton of terms in the tag anyway. How long is too long? No idea! In the past, the search engines just wouldn’t index content beyond around 250 to 1,000 characters. Maybe I’ll test this in the future.
• You do want phrases kept together. “bassett, hound” is probably going to be seen as “bassett hound” anyway, but why risk it?

Other Uses

I mentioned that misspellings were a key use for the tag. You could also use it for synonyms. For example, if you have a page all about shoes and you never say “footwear,” you could put that word in your tag. However, it’s far better if you just find a way to make use of the word in the body copy itself. That text is retrieved by all the major search engines, not just some.

Aside from synonyms, perhaps you have a page that’s all Flash or all images. Use the meta keywords tag to describe the page. Just remember that you’re still not likely to rank better than other pages that have textual information. Search engines are textual creatures. Give them what they want.

Some Official Guidelines

The W3C has guidelines (and here) in HTML 4.0 about meta data and search engines, while the XHTML specs don’t get into it at all. Ignore the specs. YES, IGNORE THE SPECS. Some of them are wrong; some are outdated. The only thing I can see that they explain is the difference between these:

• <meta name=”keywords” content=”bassett”>
• <meta name=”keywords” content=”bassett” />

See how the second tag ends /> rather than > in the first? As best I can tell, this is because a meta tag is an “empty element” in XHTML, where there’s not a “start” and a “finish” (as with a paragraph element: <p> is the beginning, with </p> the end). Empty elements in XHTML need that /> format.

I haven’t tested things without the />, but there are so many (so very, very many) pages out there not following that syntax that it is virtually certain Yahoo and Ask will read the tag either way. Doing it fresh? Do it /> style. But don’t go back and start changing things.

Aside from that, if you want to know how a search engine deals with meta data officially, you go to the search engine itself. Ask’s webmaster guidelines don’t mention the meta keywords tag, so that leaves Yahoo:

• Yahoo Quality Guidelines: “Metadata (including title and description) that accurately describes the contents of a web page.” This is telling you don’t lie with your keywords. Don’t insert words that aren’t somehow related to the topic of your page.
• How do I improve the ranking of my web site in the search results?: “Use a ‘keyword’ meta-tag to list key words for the document. Use a distinct list of keywords that relate to the specific page on your site instead of using one broad set of keywords for every page.” Note that it doesn’t say you’ll automatically rank better by doing this. Also, unique words for each page would be my advice, as well — but do NOT worry if you decide to use the same set of key terms on each of your pages. It isn’t that big of a deal.

Looking for the exact format that you should use for the meta keywords tag from Yahoo? You know, commas, spaces and all that. Sorry — they don’t provide it, which is another sign you’re probably worrying too much about it.

Freaked? Skip It

Overall, here’s the best advice I can offer anyone dealing with this tag. If you begin to feel confused, concern, tired or uncertain when pondering it, SKIP THE TAG ENTIRELY. It’s not going to hurt you to not have it, and it’s not worth the time fretting about it.

Interview: Google’s Dan Crow is the article, and in it, we’re told:

If we hit a web page with a Flash movie on it, we just extract the text out of it and index that text. But a Flash movie is much richer than that. So one of the projects I’m working on is to try and improve our Flash processing - that’s an example of an area where we could be better. But it’s not unique to us.

At the moment our advice is that webmasters need to give us a lot of help – a Flash movie is basically a set of virtual pages. If you used HTML for the links, then we’d be able to see the overall structure. The content will still be in Flash, but we can at least get some of it. Ultimately we’d like to be smart enough to look inside the Flash movie. We’re not quite there today.

Not quite there? Not quite there? Try nowhere, and that’s not really the fault of Google or any search engine.

Look, most Flash content I’ve seen is not made up of words. It’s video. It’s images. It’s non-textual content that cannot be indexed by a search engine designed to process text. There are no words to index.

I have a long-running way I try to explain this to people who build in Flash and who get frustrated that search engines can’t handle their content. I say it’s like taking a TV commercial into a radio station, then getting mad that the radio station won’t play the pictures in the commercial. It can’t. It’s radio. Radio plays only sound. If you want images, that’s TV!

Search engines don’t understand images. They understand text. Show them images, and it’s like trying to play those images on the radio.

Yes, someday in the far, far future, search engines will understand what’s in an image. They’ll see an image and be able to describe it. But that’s not now. That’s not anytime soon. And I say this because after a decade of us being told image recognition is coming, we’re still no significantly further along.

Even when they do, it still won’t be that helpful for general purpose, textually driven web search. I mean, they’ll see an image and be able to tell you "Now the company logo is flying horizontally. Now it is flying vertically. Now it’s getting all big. Now it’s getting all small." How descriptive is that?

Search engines ARE better at finding actual text that can be embedded in Flash movies. So if you have a lot of text in there, then maybe they’ll extract that text, and your page will do better. Search engines, including Google, have been doing that for several years now (and here’s Google’s help page about that). But if you have that much text in Flash — why are you putting it in Flash? Put in regular HTML, and save the Flash for pictures and images.

Overall, don’t expect that things are going to change soon. They haven’t for years, and even suggestions of secret talks with Adobe in this article ("I can’t talk about that") aren’t going to make some dramatic change. It will be terrible if those finally getting the message that Flash means problems with search engines suddenly seize on this interview as hope that a solution is coming. It’s not. You want a solution — try the workarounds you’ll find here, for a start.