• Search Engine Land
  • Sections
    • SEO
    • SEM
    • Local
    • Retail
    • Google
    • Bing
    • Social
    • Resources
    • More
    • Home
  • Follow Us
    • Follow
  • Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • Live
  • More
  • Events
    • Follow
  • SUBSCRIBE

Search Engine Land

Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • More
  • Newsletters
  • Home
SEO

Google posts draft to formalize Robots Exclusion Protocol Specification

Google open sources its robots.txt parser as part of this announcement.

Barry Schwartz on July 1, 2019 at 9:22 am
  • More

Google announced this morning it has posted a Request for Comments to the Internet Engineering Task Force to formalize the Robots Exclusion Protocol specification after it being an informal 25-year-old standard for the internet.

The announcement. Google wrote on its blog: “Together with the original author of the protocol, webmasters, and other search engines, we’ve documented how the REP is used on the modern web, and submitted it to the IETF. The proposed REP draft reflects over 20 years of real-world experience of relying on robots.txt rules, used both by Googlebot and other major crawlers, as well as about half a billion websites that rely on REP.”

Nothing is changing. I asked Gary Illyes from Google, who was part of this announcement, if anything is changing and he said: “No, nothing at all.”

So why do this? Since the Robots Exclusion Protocol has never been a formal standard there is no official or definitive guide for keeping it up-to-date or making sure a specific syntax must be followed. Every major search engine has adopted robots.txt as a crawling directive but it isn’t even an official standard. That is going to change.

Google open sources its robots.txt parser. With that, Google announced they are open sourcing the portion of its robots.txt that parses the robots.txt file. “We open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files,” Google said. You can see this library on Github today if you like.

Why we care. Nothing is specifically changing today but with this change to make it a formal standard it does open up the chance for things to change. Keep in mind, the internet has been using this as a standard for 25 years without this being an official standard. So it isn’t clear what will or may change in the future. But for now, if you are building your own crawler, you can use Google’s robots.txt parser to help you.



About The Author

Barry Schwartz
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here.

Related Topics

Channel: SEOGoogle: SEOSEO: Crawling and RobotsSEO: Technical SEO

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter search marketers rely on.
See terms.

ATTEND OUR EVENTS

Lorem ipsum doler this is promo text about SMX events.

February 23, 2021: SMX Report

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

October 2021: SMX Advanced Europe

December 14, 2021: SMX Code

Available On-Demand: SMX

×


Learn More About Our SMX Events

Discover actionable tactics that can help you overcome crucial marketing challenges. Our next conference will be held:

MarTech 2021: March 16-17

MarTech 2021: Sept. 14-15

MarTech 2020: Watch On-Demand

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • Become an Experience-Led Business with an Agile CX Tech Stack
  • Transformational CX: Video-Powered Experiences Meet Consumers’ Demands
  • What Customer Experience Means in 2021
  • 4 Ways to Commodity-Proof Your Company
  • Threat report: How are different industries battling cybercrime?
See More Whitepapers

Webinars

  • The AI Revolution Is Coming to Every Stage of Your Buyer’s Journey
  • The Fundamentals of Link Building for E-Commerce & Affiliate Sites in 2021
  • Your Customer is Calling: Make the Most of Your Marketing Spend with Call Tracking
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

h
Receive daily search news and analysis.

Channels

  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social

Our Events

  • SMX
  • MarTech

Resources

  • White Papers
  • Research
  • Webinars

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok