Microsoft proposes method to automatically submit URLs from WordPress to search engines

Fabrice Canel, the Principal Program Manager at Microsoft Bing, has submitted a proposal, aka feature request, to build into the core of WordPress the ability for WordPress to push new and updated URLs directly to search engines.

The proposal. You can access this feature request at the WordPress ticket site, where Fabrice wrote “today, we propose integrating in WordPress Core the ability to notify not only Bing, but also all participating Search Engines, of any WordPress URL or Content change. Microsoft to develop and maintain the open-source code in close collaboration with WordPress. WordPress to approve, validate and include code.”

“Behind the scenes, WordPress will automatically submit URL or Content ensuring that the WordPress content is always fresh in the Search Engines; in exchange Search Engines will limit crawl or not crawl WordPress sites. Site owners will have the ability to opt-out or select the content they don’t want to submit to search engines,” Fabrice explained.

URL submission API and WordPress Plugin. Doesn’t Bing already have this built into WordPress and its search engine. The answer is not exactly.

Microsoft did create the Bing URL submission API that allows publishers and site owners to automatically feed Microsoft Bing, through Bing Webmaster Tools, new and updated URLs for indexing. This way, Bing doesn’t have to crawl your site to find the new or updated content.

In July, Bing released a WordPress plugin to simplify the process for WordPress sites to use this URL submission API.

What is new. Microsoft wants to avoid all of this and just have WordPress at its core, out of the box, automatically work with the Bing URL submission API, and also support any other search engine that wants to participate. This way you do not need to install the plugin or build your own integration into this API.

More details. Fabrice Canel posted more details in the ticket saying:

• From pulling to pushing: it’s not about pulling (RSS feeds or similar), it’s about pushing, publishing each change, with some throttling logic as already done in Bing Webmaster Tools WordPress plugin, to avoid notification on every keystroke, every 5 seconds save, to the set of search engines having adopted this design open to all search engines and listening to change. Pulling requires crawling, crawling, and crawling again to check if the content has changed (most of the times the content didn’t change), Pulling required also at the first place discovering the site and feeds. Pushing enables search engines to be seconds to minutes behind content change and guaranty that search engines are aware of the change and minimize the need of crawling to discover if something has changed. In case of downtime, search engines will still rely on sitemaps and links to discover news URLs.
• Open to all search engines: Search Engines having an API can be added to the notification.
• Enabled by default: We want to lower the complexity for WordPress users to be found and indexed by search engines. If you are a newbie, your new site should be immediately found and indexed, your latest content and your latest typo fixed should be indexed in minutes… not in weeks.

Why we care. If WordPress does end up supporting this new feature request, then it would simplify the process of WordPress sites getting their content into Microsoft Bing Search. In addition, it may add pressure to Google to begin supporting a content submission API for normal content. Right now, Google only supports with its indexing API being pushed content for job posting URLs and live stream content.

This is not a feature that is live tonight, this is just a feature request that will require months of approvals and testing to eventually go live.

About The Author

Barry Schwartz

Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here.