Search engines penalize websites – but why?

There’s hardly another constant hot-button issue as intensely debated within the search industry as search engine penalties. Leading SEO outlets maintain their coverage of that touchy subject from a uniform angle, reinforcing the subjective view that seemingly all but a few websites are penalized. Yet despite this ongoing barrage of information, it seems that more misconceptions are being disseminated than certainties. Consequently the situation only amplifies unwarranted fears among website operators. This opinion piece, strongly influenced by the author’s personal experience while working for Google Search including penalizing websites, is an attempt to clarify why and how leading search engines approach penalization. The focus here is on key search market players. The article does not aspire to provide a comprehensive overview including all search alternatives, such as DuckDuckGo or Ecosia. While they all contribute a valiant effort to enhancing, what otherwise may appear as anemic competition and only sporadic innovation, their individual approaches towards penalizing websites don’t change the gist of the key message: For the most part, search engine penalties are not to be dreaded. 

Why penalize at all?

Search engines are first and foremost commercial enterprises. Their main goal is revenue creation. Towards that objective, cost of operation must be kept low, while user happiness maintained at a constant high. At the same time it is important to remember that search as a product is immensely complex. The data torrents required in the initial steps of crawling and processing, let alone the processing power needed for ranking solutions are tremendous and constantly growing. At any given time approximately 20% of the globally available data is about to vanish, while another 20% is in the process of changing and yet another 20% is brand new and just coming into existence. Paradoxically flux is the only constant factor in search. Therefore, it is little surprise that scalable, algorithmic solutions are much preferred. Here’s where one of the most common misconceptions with regard to search engine penalties takes root. Countless articles and first-hand reports describe sites affected by search engine ”algorithm penalties”. All of them are fundamentally wrong. No website ever was or is affected by an “algorithmic penalty”. How could they, given that algorithms are purely input-driven mathematical formulas designed to calculate information and return results based thereon.

If search engines worked ideally, and their algorithms were close enough to perfection to adequately read all input, then that was the end of the story. There were, in this utopian scenario, no actual penalties. We know it isn’t the case however and that’s because algorithms -even tried and tested ones- are prone to fail occasionally. When that happens, that’s when penalties become a reality. Unlike algorithms they are exceptions to the rule.

Because of their motivations, primary objectives and cost awareness, search engines are rather reluctant to issue penalties. In the first instance, these tend to be costly, individual, limited in scope, barely or not at all replicable and worst of all contributing little to improving the core product. Search engines are averse to penalties because there’s very little gain in them from a purely business standpoint. Contrary to popular notion, legal concerns are not at all a reason why penalties are issued relatively sparsely, all facts considered. After all, no individual or organization has thus far managed to successfully litigate against a search engine and to recover their rankings based on court order. Search engines and their indices are private, proprietary property.

Why different approaches?

In a nutshell, penalizing is the manifestation of a small scale system failure, which requires resource allocation and bares no promise for gains of any kind. This helps to explain why most major search engines are reluctant to commit serious ressources to the endeavor. Among the former giants Yahoo! serves here only as a historical example, referring to another era way before it’s sad slide into search oblivion. At a time when Yahoo! still did enjoy a market share it opted for a rather strict if not completely either-in-or-out strategy. Websites found to be in violation with respective guidelines were either demoted or in case the identified transgression were egregiously entirely removed from the index. These so called Judgements were collected in a database internally referred to as EDB in perpetuity. That database wasn’t identical with the WoW or “worst of web” longtail index, reserved for merely low quality content. Websites included in the EDB did stand a chance to recover. A number of dedicated forms referring to situations such as “Site disappeared from Search Results” or “Improving Site Rank for specific Keywords” promised remedy for repentant website operators. Later the Yahoo! Site Explorer did provide similar two-way communication and resolution opportunities. 

More recently Bing and in-particular Google seem to embrace a more nuanced approach. Bing’s response to their Webmaster Guidelines violations requiring action is initially an attempt to mitigate the undesirable impact black hat SEO techniques have. If that’s not practicable or effective, the offending site is either demoted or ultimately entirely removed from Bing search results. There’s however no preemptive warning to be expected from the Bing Webmaster Tools. While ultimate Bing penalties seem to last for eternity, there is however a ray of hope for site operators willing to repent. Bing does engage in two way communication through a Support Form. While no turn-around time is guaranteed, the fact that website operators can inquire about their individual websites penalties is laudable. In time and with some effort Bing penalties can be lifted.

Google embraced an even more refined approach, issuing granular, selective and specific penalties depending on the type of transgression identified. And proactively highlighting these to respective website operators via their Google Search Console. Despite the confusing naming the messages are not warnings, since there’s no time to prepare for the penalties impact. At the point of receiving the Google message, the penalty highlighted is already in place, even though it may not be felt acutely yet. Google allows manual penalties, which are officially referred to as manual spam actions, to time-out albeit after a prolonged period of time. And, unlike their competitors, Google has a formalized process for site owners who wish to abide by Google Webmaster Guidelines going forward to get back into good graces, called the Reconsideration Request. Here it is important to point out that the set of documents previously referred to has nothing to do with Google Rater Guidelines, which are used exclusively by hired contractors to assist engineers with evaluating algorithmic experiments. While the reconsideration request process is worthy of a deep-dive to foster better understanding, in this instance it is merely important to acknowledge that while no official turn-around time exists. In some cases, several months pass before a response is received. Google nonetheless chooses to commit experienced employees to manually review each and every reconsideration request sent. Even with the scant information provided via Google Search Console and utilizing the Reconsideration Request all Google manual penalties can be expunged, as long as the violation has been eliminated. 

Why do search engines care at all?

Despite this substantially more labor intensive and significantly more accommodating approach towards website owners demonstrated by Bing and Google, the interested public perception frequently highlights a seemingly harsh, even vindictive actions on search engines part. The fact that 99.9% of all escalations of “penalized” websites vented about on forums refer to sites that fail to live up to the site operators expectation due to poor SEO signals is barely known. Most of these discussed cases refer to sites that have not actually been penalized. The constant lamentations suggest however to the online industry that search engines may care for individual websites. Somehow the notion of passionate, loving or hating search engines has taken root. That is fundamentally wrong though because search engines are utterly indifferent towards individual websites or their respective performances in organic search. Their only concern lies with their user satisfaction. Individual websites’ visibility for a given query is of no consequence. This is why search engines allocate only minimum resources towards penalizing websites. Or to lift previously issued penalties. There’s an upside to that dispassionate attitude. It not only allows penalty removal and recovery but it also prevents vindictiveness. Websites are never penalized because of their mere association with known offenders and they can grow in organic search visibility upon their recovery way beyond previously reached highs.

What to make of penalties?

Penalties must be seen for what they truly are: stopgap measures. They are occasional, almost exceptional in the scale of things. For search engines, they are a sore point rather than a hot button issue. Something to deal with swiftly but most definitely not the main theatre of operation. That search engines choose to allocate resources, however limited, to address penalty issues to a small special interest group like online marketers deserves some credit. Yet because they are relatively rare, it appears that penalties have been for far too long receiving unjustified attention within the webmaster community. Rather than dreaded, search engine penalties should be considered as a fact of online life and commerce. Similarly like a driver does not expect to catch a flat tire when getting into the car, his or her driving usually isn’t determined or permanently affected if they actually do catch a flat tire. Some, even skilled webmasters and website owners will have dealt with a search engine penalty at some point in their careers. Few may even experience several penalties over the course of their professional life. Still, these are unlikely to be the defining moments in any way. They are mere bumps on the road of an exciting journey. And the experience gained while dealing with penalties often bears the potential of a virtue out of necessity, ultimately culminating in unprecedented growth and online prosperity.