Fetch, Googlebot! Google’s New Way To Submit URLs & Updated Pages

Today, Google launched a new way for site owners to request that specific web pages be crawled. How is this different from the other ways available to let Google know about your pages and when should you use this feature vs. the others? Read on for more.

This new method for submitting URLs to Google is limited, so you should use it when it’s important that certain pages be crawled right away. Although Google doesn’t guarantee that they’ll index every page that they crawl, this new feature does seem to at least escalate that evaluation process.

To better understand how this feature works, let’s take a look at how Google crawls the web and the various ways URLs are fed into Google’s crawling and indexing system.

How Google Crawls & Indexes the Web

First, it’s important to know a bit about Google’s crawling and indexing pipeline. Google learns about URLs through all of the ways described below and then adds those URLs to its crawl scheduling system. It dedups the list and then rearranges the list of URLs in priority order and crawls in that order.

The priority is based on all kinds of factors, including the overall value of the page, based in part on PageRank, as well as how often the content changes and how important it is for Google to index that new content (a news home page would fall into this category, for instance.

Once a page is crawled, Google then goes through another algorithmic process to determine whether to store the page in their index.

What this means is that Google doesn’t crawl every page they know about and doesn’t index every page they crawl.

How Google Finds Pages To Crawl

Obviously, a page can’t begin the crawling and indexing process until Google knows about it. Below are some of the common ways (including the newest!) Google learns about new or changed pages.

Discovery

Google has lots of ways to discover URLs on the web and they’re pretty good at it. After all, they could never have built up a comprehensive index in the first place without having a good handle on discovering pages on the web.

Originally, Google discovered pages based on links, and that’s the still a core method they use (which is why it’s so vital to ensure that you link to every page on your site internally, as you may not have an external link to every page), although they’ve evolved those methods over time (for instance, they now use RSS feeds for discovery).

Once Google knows about a page, it continues to crawl it periodically. Most sites do fine relying on this method, particularly if the site is well-linked (which of course, is more easily said than done).

XML Sitemaps

The XML Sitemaps protocol enables you to submit a complete list of URLs to Google  and Bing. The search engines don’t guarantee that they’ll crawl every URL submitted, but they do feed this list into their crawl scheduling system.

Public Requests

Google has always had an “Add URL” form available for anyone to request that a URL be added to the index. Historically, this form was intended for searchers who noticed that a particular page seemed to be missing.

As you might imagine, this scenario happened less often over time, and Google likely doesn’t weight these requests as heavily as they do other signals for crawling.

With this latest launch, the Add URL form has been renamed Crawl URL, you have to log in to it with a Google Account, and while you can add any URL (not just those on a site you’ve verified you own), you can submit only up to 50 URLs a week.

Google Crawl URL

Google Webmaster Tools Fetch As Googlebot

This is the new feature launched today. The Fetch as Googlebot feature itself has been around for a while and lets you direct Googlebot to crawl a specific URL on a site you’ve verified you own and see what response Googlebot gets back from the server. This is helpful in debugging issues with the crawl that aren’t obvious when you load the site in a browser as a user.

Since this functionality requires that Googlebot, in fact, crawls the page, there’s no need to start at the beginning of crawling process, so you can now simply request indexing consideration for that page after you fetch it by clicking Submit to index.

Google Submit to Index

When Should You Use The New URL Submission Feature?

XML Sitemaps are still the best way to provide a comprehensive list of URLs to Google and a solid internal link architecture and external links to pages through your site is still the best way to encourage Google to crawl and index those pages.

However, this new feature is ideal for situations when you launch a new set of pages on your site or have major updates. Google also notes you can use this feature to speed up URL removal or cache updates.

You can submit up to 50 URLs a week (although the UI seems to have a bug that starts you with 28). You can also submit the crawled URL and all pages linked from that URL, but those submissions are limited to 10 per month. A common use of that latter option would be when you have launched a new section of your site.

Submit a Single URL to Google’s Index

Google Index Submission

Submit All Pages Linked to a URL to Google’s Index

Google Submit All URLs

You should always use this option over the public Crawl URL submission, as requests by verified site owners are likely given higher prioritization. As Google notes, this request doesn’t ensure the URL will in fact end up in the index, but the page is much farther along in the process.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: SEO | Features: Analysis | Google: SEO | Google: Webmaster Central | Top News

Sponsored


About The Author: is a Contributing Editor at Search Engine Land. She built Google Webmaster Central and went on to found software and consulting company Nine By Blue and create Blueprint Search Analytics< which she later sold. Her book, Marketing in the Age of Google, (updated edition, May 2012) provides a foundation for incorporating search strategy into organizations of all levels. Follow her on Twitter at @vanessafox.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://www.gamerstube.com Joe Youngblood

    “Select if your site has changed significantly. Google will use this URL as a starting point in indexing your site content. Google doesn’t guarantee to index all pages on your site” – how does this work with a noindex, follow page?

    i am guessing google will see the noindex but still update all of the linked pages.

  • http://www.rankexperts.com jasonmz83

    This is great source, thanks for sharing the information, we will definitely provide the information out to our clients and help them getting their links indexed in google faster.

  • http://www.seoblog.co.za DarrenVrede

    I’m glad this topic has come up because I have been wanting to get peoples views on this. What happens if your site or a site that you have a link on gets crawled but does not get indexed. Do you guys think this link is still valuable even though the page it is on does not appear in the index?

  • Hiren vaghela

    Wow it seems good, WMT is always better for webmasters and this feature is added up now.

  • Alex

    I heard about this Fetch as Googlebot feature but it does not work at all so far.
    Submitted couple website to Google year ago.
    Those site are visible on Bing but not on Google.
    Google sucks!

  • http://www.sorezki.com Roi Sorezki

    I’m a bit confused Vanessa. You said “Today”, but how is this a new feature? We’ve been using this for quite some time now. Here’s a screenshot of our last fetch from the 21st of July – http://goo.gl/kKyHH (a couple of weeks prior to this post), and we’ve been using this service for quite some time now.

    Has anything been changed a week ago?

  • http://panicattacksdoctor.com/ Elaine Ryan

    Thanks for posting this.  I am having trouble with one of my new blogs http://panicattacksdoctor.com and have just gone through all the pages and fetched and submitted as googlebot

  • http://www.monicawright.com Monica Wright

    Hi Elaine,

    I suggest asking your question over at our Linked In group. It’s extremely active and many members respond to this type of question. You’ll have a better chance of getting some help there. http://www.linkedin.com/groups?gid=53266

  • http://panicattacksdoctor.com/ Elaine Ryan

     thanks Monica, I will have a look :)

  • joeflyde

    I am just getting into this SEO stuff so this was a very informative post.  ThankYou.

    eatonvilleflorida.us

  • feitian

    Your article really helps me a lot. I had searched the Internet and library for information for a long time.
    I am always interested in this topic. I’m happy to capture your article and I am looking forward to more of your article.
    Thank you very much!!                                                  
    http://www.e-discountchristianlouboutin.com
    http://www.hermesbagsfactory.com

  • Thomas Wang

    aa

  • Thomas Wang

    how it look like? i really don’t know what will it go on for google crawled 
    2012 Christian Louboutin Shoes
    Christian Louboutin Boots
    Christian Louboutin Evening

  • http://twitter.com/longwang5 longwang
  • http://www.buytissotwatches.co.uk/ tissot watches

    Nice Info, Thanks For Sharing at This Blog/Web

  • hong lin

    cheaper high quality watches in http://www.boutique-watch.com,you can visit it,you can got more

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide