Solving the Big URL Issues Presentations from SMX East, October 2010 URLs continue to be a key technical obstacle to search engine optimization. Internal and external linking structure, URL formats, parameters, faceted navigation, breadcrumbs, tracking codes, pagination, canonicalization options, hyphens and underscores. What really matters? What should you rewrite and what should you leave alone? […]
Talk about getting back to your roots. Ask.com, which originally used humans to find answers to questions, is going further back to its origins by abandoning its web crawling technology. The Need, Or Not, To Crawl Search engines like Google and Bing try to have answers to anything someone might search for by “crawling” the […]
As a site owner, you generally want Google to index as many pages of your site as possible. But there are certainly times when you find that you’ve accidentally let Google index confidential content or other information you don’t want published, and you want to get it removed as quickly as possible. Read on for […]
The new search engine Blekko just launched, and it’s making SEO-related data that it has found during its crawling and indexing of the web available for all sites. Just what does it include and how useful is it? Accessing SEO Data You can access SEO data for any domain or URL by clicking the SEO […]
Earlier this week, site owners noticed that substantial Yahoo Site Explorer link data seemed to be missing. A glitch? Gone for good? The first signs of integration into Bing webmaster tools? I talked to Yahoo and got at least some initial answers. Many site owners regularly use Site Explorer for information on external links to […]
At SMX East earlier this week, Maile Ohye of Google showed new parameter handling features in Google Webmaster Tools. Today, those features are available to everyone. Google first introduced parameter handling about a year ago. When I wrote about the launch, I described how optional parameters in URLs could cause canonicalization problems and gave a […]
Dear coders: please write your HTML/XHTML code to be standards-compliant. Yes, I know — we’ve been down this road before. Folks have tested compliant versus non-compliant code, and it doesn’t help with search rankings, so why bother, blah blah blah. Just use the damned standards, OK? Standards compliance does impact rankings, albeit indirectly: Standards compliance […]
The Google Webmaster Help blog announced they are now indexing SVG files and documents. SVG is a royalty-free graphics format typically used on the web for fancy font usage. Google said: We index SVG content whether it is in a standalone file or embedded directly in HTML. The web is big, so it may take […]
The adCenter blog announced they will be temporarily increasing the standard crawl rate they use for detecting ad relevance on adCenter ads. The increase will show a crawl rate of up to 20 to 30qps, which they say is “accepted standards for crawl rates.” Here is the full message from adCenter: Along with our consistent […]
The Google Webmaster Central Blog announced they are stepping up their notifications within Webmaster Tools. They will now send “SiteNotices” for a “significant increase” in unreachable content, that is when GoogleBot cannot reach a number of pages on your web site. Here is a picture of what a SiteNotice may look like: Google will only […]
Today, Microsoft launched a revamp of their Bing Webmaster Tools. I talked to them back in June, when they previewed the tools at SMX Advanced, and they told me that they were starting from scratch and rebuilding the tool from the ground up. So how are things different? They say they are focused on three key […]
Blogger and former journalist Tom Foremski reports he had a conversation with the head of engineering for Google in Brazil and says the guy supported the idea of open-sourcing the search index: I mentioned an idea I had about making the Google index into an open database that anyone could access, I said that this […]
Google released a report on general statistics of web pages on the Internet. The report lists basic information they retrieved over the course of “Google’s crawl and indexing pipeline.” The key highlights include: The average web page takes up 320 KB on the wire. Only two-thirds of the compressible material on a page is actually […]
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Microsoft To Replace MSNBot With BingBot October 1, 2010 The Bing Search Blog announced that on October 1st, MSNBot will be renamed to BingBot. Bing has been testing a beta bot […]
The Bing Search Blog announced that on October 1st, MSNBot will be renamed to BingBot. Bing has been testing a beta bot since November of last year. On October 1st, the beta label will be removed from the Bot name and it will be named BingBot. You should see the following useragent in your log […]
Google announced they added “soft” or “crypto” 404s to their “crawl errors” report in Google Webmaster Tools. Soft 404s or crypto 404s are page not found pages that return the server response code of 200. Technically, 404 pages should return a server status code 404, which means page not found. Sometimes webmasters make a mistake […]
The startup 80legs launched in September of last year to make web crawling affordable and accessible to anyone that wanted to subscribe to the service. The company makes crawling the web or customized slices of it available to even small companies. More flexible than something like the Google Custom Search Engine, pricing ranges from free […]
More egregious mistakes that website owners, web designers/developers, landing page experts, and others make to prevent qualified search engine traffic and conversions.
The home page of OpenOffice.org, the well-known Microsoft Office competitor, is missing from Microsoft’s Bing search engine. While it sounds suspicious, the problem has nothing to do with Bing itself — it’s a technical problem on OpenOffice.org’s end. Ian McAnerin noticed earlier today that OpenOffice.org doesn’t show up in Bing on searches for [open office] […]
Eric Enge has an outstanding interview with Matt Cutts. I have read through the entire interview and pulled out four key takeaways, but there is a lot more to be learned from the interview. I’ll list my takeaways in order of what I find to be most interesting: (1) Matt Cutts confirmed that a 301 […]
Michael VanDeMar has documented possible evidence that Google is using fake referrers, possibly to detect forms of spam. Michael said the GoogleBot is being cloaked and is sending fake referrers for keywords the site is not related to and has no chance in ranking for. Bing has been doing this for years, fixed the issue […]
Back in November 2009, Google News announced they were “in the midst of an exciting transition period” that included a change to the News Sitemap Protocol. News publishers have through April 2010 to modify their News Sitemap to accommodate the new format. What’s so exciting and transitional? I asked Google, thinking that they were changing […]
The Google Reader blog announced you can now track changes to any web page, even the ones without RSS feeds. How does this come in handy? Well, let’s say you want to track when Google changes their webmaster guidelines. All you do is copy the URL into Google Reader and click on “create a feed.” […]
Via Search Engine Roundtable comes news of refreshing candor from a Bing employee about the performance of Bing’s crawler, which is still referred to as MSNbot. The admission happened yesterday in the Bing Webmaster Center forums, when Brett Yount, a program manager for Bing, replied to a question about why Bing has only indexed one […]
Google announced they have added a new supported user agent that can be used to block Google from indexing stories specifically on Google News. You can now specific the user agent Googlebot-News in your robots.txt file to exclude documents from being crawled and displayed in Google News. Google has several user agents for controlling how […]
In the battle between search engines and some mainstream news publishers, ACAP has been lurking for several years. ACAP — the Automated Content Access Protocol — has constantly been positioned by some news executives as a cornerstone to reestablishing the control they feel has been lost over their content. However, the reality is that publishers […]
Microsoft is retiring its MSNBot1.1 and introducing its next bot: 2.0B. According to the Bing Community blog: As MSNBot 2.0b enters full-scale production, the time has come to retire our previous generation bot, MSNBot 1.1. By the end of the first week in November, you will no longer see the following user agent in your […]
For years, Google’s discovery of web pages was solely based on links. If a page had no links to it, Googlebot had no way of knowing about it and therefore, would never index it. Along the way, Google provided an option for submitting individual pages, but that wasn’t really a viable option for site owners […]
Google Webmaster Tools has just launched a “labs” section, where you’ll find new features that may be early in the development cycle and not quite as robust as the rest of the tools. The features available so far are Fetch as Googlebot, which lets you see exactly what Googlebot is served when it requests a […]
It’s one of the most common pieces of SEO advice you’ll find: Don’t build your web site with AJAX if you want it to be crawled and indexed by search engines. AJAX-based web sites are essentially a locked and bolted door when a spider comes crawling. At our SMX East conference today, Google has announced […]
Google introduced its new Maps profile page format, Place Pages, last week on Thursday. It’s a pretty dramatic change that replaces a tabbed version of the “info bubble” on Maps with a full page that features a wide variety of rich content. Each of these pages has its own URL. When I discussed the change […]
A new feature has appeared in the Site Configuration Settings Sections of Google Webmaster Tools. The setting, called Parameter Handling, enables site owners to specify up to 15 parameters that Google should ignore when crawling and indexing the site. Google lists the parameters they’ve found in the URLs on your site, and indicates whether or […]
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Bing Continues With Fake Referrers: What Part Of Stop Don’t They Understand? For over two years now, Microsoft’s search engine has been generating fake information that can make site owners think […]
For over two years now, Microsoft’s search engine has been generating fake information that can make site owners think they’re getting strange search traffic (including porn traffic), when they are not. It’s time for this to come to an end. The culprit? Microsoft’s “crawler,” the software that visits web pages across the web that go […]
Google has just unveiled a “secret project” of “next-generation architecture for Google’s web search“. This new architecture appears to include crawling, indexing, and ranking changes. For the first time, Google isn’t simply incorporating these changes into their existing infrastructure or replacing it. Instead, they’re providing a developer preview and are asking webmasters and power searchers […]
Microsoft announced that MSNBot 2, which is currently in beta, may be ramped up a lot more and webmasters might begin to notice the new bot in their log files. The new bot has been out there since December 2008, and the useragent is just about the same as the original bot. New Bot: msnbot/2.0b […]
The Yahoo Search Blog has announced an index and algorithm update, aka a “weather report.” Yahoo said, “we’re rolling out some updates to our crawling, indexing, and ranking algorithms over the next few days. During this process, you may see some ranking changes and page shuffling in the index.” I spotted this update a couple […]
Cool changes coming to Twitter Search from News.com reports Twitter will be improving Twitter Search to index content beyond Twitter.com by crawling links within Twitter posts and also by adding a ranking algorithm to the search results. In a one-to-one conversation with News.com’s Rafe Needleman and Twitter’s new VP of Operations, Santosh Jayaram – who […]
There is no doubt that you can learn a tremendous amount of information on search engine optimization (SEO) by reading sites like this or ones in our blogroll, but there is always a lot to be learned from getting your hands dirty. Now, you can get your hands dirty by experimenting and trying SEO techniques […]
The Google Webmaster Central blog notified us that Googlers have presented a new study on Sitemaps at the WWW’09 conference in Madrid. The study is absolutely interesting and I recommend printing out the ten page PDF document and reading it. For those of you who don’t have time for that, I hope to highlight the […]