The push layer returns: Why ‘publish and wait’ is half a strategy
See how AI systems process content through five entry modes — and why relying on crawling alone limits visibility, speed, and revenue
Topic: WebCrawler
Indexing and keyword ranking techniques revisited: 20 years later
Learn how keyword-based ranking techniques have evolved – from the 'little old ladies' to the vector space model, to today.
We’ve crawled the web for 32 years: What’s changed?
A look back at the history of search and SEO and a preview of what the next iteration of the internet means for marketers.
10 big changes with search engines over my 20 years of covering them
Search Engine Land founding editor Danny Sullivan looks at how search has changed over the two decades he's been writing about it.
Apple Webcrawler: More Potential Evidence Of Search Ambitions
Apple Insider reports on the discovery of a web-crawling bot originating from Apple’s servers. It was first “outed” by developer Jan Moesen. This is what Moesen saw: Moesen reports that the bot is only crawling HTML, “not the CSS, JavaScript or image files.” Then he asks whether this is an “official Apple project” or just “someone crawling the web […]
SearchCap: Google Redirecting Penalties, Robot.txt 20th & Google Wiretapping Case
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Supreme Court Rules Street View “Wiretapping” Class Action Can Proceed Against Google According to the Wall Street Journal, the U.S. Supreme Court will not review a U.S. Ninth Circuit Court of […]
Robots.txt Celebrates 20 Years Of Blocking Search Engines
Today is the 20th anniversary of the robots.txt directive being available for webmasters to block search engines from crawling their pages. The robots.txt was created by Martijn Koster in 1994 while he was working at Nexor after having issues with crawlers hitting his sites too hard. All major search engines back then, including WebCrawler, Lycos […]
How The “Focus On First” Helps Hide Google’s Relevancy Problems
In January, I was invited to speak to Google’s search quality team about issues I had with Google’s search results. My topic? For queries where I know a subject really well, I often found Google provided some pretty poor results in the top listings along with the good ones. I wanted more perfection! With the […]
Google’s Personalized Results: The “New Normal” That Deserves Extraordinary Attention
On Friday afternoon, Google made the biggest change that has ever happened in search engines, and the world largely yawned. Maybe Google timed its announcement that it was personalizing everyone’s search results just right, so few would notice. Maybe no one really understood how significant the change was. Whatever the reason, it was a huge […]
SearchCap: The Day In Search, September 14, 2009
Below is what happened in search today, as reported on Search Engine Land and from other places across the web. From Search Engine Land: Bing 2.0 “Visual Search” Launches, Allows Search By Pictures Microsoft has unveiled Bing 2.0 at TechCrunch50 today, also known as Bing Visual Search, which allows people to “search by picture.” After […]
A Look Back At The Old Search Engines
The Six Revisions blog complied an excellent list of the classic search engines, some dating back to the 1990s. His list includes screen shots of both the old and current (if available) versions of HotBot, Excite, WebCrawler, Ask Jeeves, Ask.com, Yahoo, Google, Dogpile, AltaVista, Lycos, MSN Search, Bing, AOL Search, Infoseek, Go.com, Netscape, MetaCrawler, and […]
Obit: A West Coast Digerati Deadpools Ask.com
Goodbye, Ask.com. You caught my eye back in 1997 as an unusual meta search engine that asked questions to get answers. By 1998, I counted you alongside Google and Direct Hit as shining examples of what to watch in search. You’d dumped depending on others for search results and started providing answers using your own […]
Google & Human Quality Reviews: Old News Returns
A quote from Google’s director of research Peter Norvig in an article at Technology Review about how human reviewers assess Google’s search quality is starting to pick up play on the web and will no doubt grow. Problem is, it’s not news. Still, I suppose it’s interesting to those who are new to the space […]
Search 3.0: The Blended & Vertical Search Revolution
This has been a remarkable year. After years of no real dramatic evolution in search, the third generation finally arrived. Google calls it Universal Search, and I’ve been tending to say “blended search” as a generic name for the change that’s now hit all the major search engines. But in doing the agenda for our upcoming SMX West conference, a […]
SearchCap: The Day In Search, October 18, 2007
Below is what happened in search today, as reported on Search Engine Land and from other places across the web.
With Dogpile Beating Google In Satisfaction, Owner InfoSpace Needs To Reinvent Itself, Be Risky With Metasearch
InfoSpace recently sold both its local and directory assets and its mobile platform business to Idearc/Superpages and Motricity respectively. So what’s left? Specifically Dogpile and its cousins Metacrawler, Webcrawler and the domain InfoSpace.com. The company also owns the kid-friendly search site Zoo.com. Amid all this, JD Power & Associates has just named Dogpile “highest in […]
Meta Keywords Tag 101: How To “Legally” Hide Words On Your Pages For Search Engines
If there’s anything I particularly hate when it comes to SEO, it’s the meta keywords tag. I so wish it had never been invented. It’s practically useless, yet people still obsess over it. In this article, I’ll explain more about why you shouldn’t worry about it except perhaps for misspellings, as well as which search […]
History Of AOL Search
In preparation for those trying AOL tomorrow as part of our first Google-Free Friday, I thought it would be helpful to give some background on AOL in search. AOL has long offered search — and even owned several web crawling technologies — over the years. Don’t forget to read AOL: Tomorrow’s Google Free Friday Alternative […]