This day in search marketing history: January 16

Google explains crawl budget, plus: website outages and blackouts the right way, Google doing 1 trillion searches, and more.

Chat with SearchBot

Google explains crawl budget

In 2017, Google’s Gary Illyes explained what crawl budget is, how crawl rate limits work, what crawl demand is and what factors impact a site’s crawl budget.

As Illyes explained, most sites don’t need to worry about crawl budget.

“Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters,” Illyes said.

More from Illyes:

  • Crawl rate limit is designed to help Google not crawl your pages too much and too fast where it hurts your server.
  • Crawl demand is how much Google wants to crawl your pages. This is based on how popular your pages are and how stale the content is in the Google index.
  • Crawl budget is “taking crawl rate and crawl demand together.” Google defines crawl budget as “the number of URLs Googlebot can and wants to crawl.”

Read all about it in Google explains what “crawl budget” means for webmasters.


Also on this day 


Google Ads bug again for Gmail on desktop Safari browsers

2022: This impacted Google Ads that were served in Gmail for users accessing their email using the Safari desktop browser.


Google January 2020 core update almost done rolling out

2020: Google confirmed: “the update is mostly done…”


Google Search Console unparsable structured data report data issue

2020: Some users may have seen a spike in unparsable structured data errors due to an internal Google misconfiguration.


Google Search Console’s URL inspection tool adds HTTP response, page resources, JS logs and rendered screenshot

2019: The tool added the ability to see a specific URL’s HTTP response code, the page resources, the JavaScript logs and a rendered screenshot.


Bing Ads launches page feeds for easier Dynamic Search Ads management

2019: Page feeds eliminated the need to create targets for specific URLs or groups of URLs in DSA campaigns, making it easier to set up and manage.


Google ordered to submit search index to state sponsorship in Russia

2019: A law passed in 2018 mandated that search engine results be filtered through the federal state information system (FGIS).


Katy Jurado Google doodle marks 94th birthday of film star from the Golden Age of Mexican cinema

2018: Jurado was drawn in a powerful pose against a backdrop inspired by the film “High Noon.”


Google Image Search testing YouTube videos

2017: Google was embedding playable YouTube videos for certain queries.


Martin Luther King Jr. Day Google doodle promotes King’s message of unity

2017: The Doodle, by guest artist Keith Mallett, captured one of the major themes of King’s speeches and writing: “All life is interrelated.”


Google Still Doing At Least 1 Trillion Searches Per Year

2015: Google was sticking with the figure it gave in 2012 but stressed it’s “over” that amount. How much? Google wouldn’t say.


Google’s Knowledge Graph May Show For 25% Of All Search Queries

2015: The figure was important because it meant searchers often didn’t need to leave Google to visit a publisher’s site. 


2015 Oscar Nominations Now Showing In Google Knowledge Graph & Bing Carousel

2015: Searches for [Oscar nominee] and [Oscar nominations] on Google and Bing delivered results for the best picture candidates.


Local SEOs Sound Off On Google+ Local Hijackings

2014: Local SEOs and people familiar with Google+ Local theorized about on how this happened and shared suggestions o how to guard against future hijackings.


Bing: It’s A Myth That Keyword Rich Domain Names Improve Search Rankings

2014: The domain and keywords in that “domain send less and less” ranking signals to the overall Bing ranking algorithm.


RKG Report: PLAs Fuel AdWords Growth, Bing Ads Spend Rises Sharply In Q4

2014: Paid search spend among RKG’s retail-heavy client set rose 23 percent year-over-year. Overall click volume increased 19 percent, and CPCs ticked up just 3 percent.


Covario: Global PPC Click-Through Rates Hit New Highs, Spend Up 17 Percent YoY In Q4

2014: Based on results from their high tech, consumer electronics and retail clients, PPC spending increased 17% year-over-year and 13% over Q3 2013.


Google Now Cards Hit The Desktop For Mac, Windows and Chrome OS Users

2014: The Google Now cards ran through the latest Google Chrome build (Chrome Canary). Users had to be logged into Chrome.


Dian Fossey Commemorated With Google Logo To Mark The Famous Zoologist’s 82nd Birthday

2014: The logo included illustrations of the Silverback gorillas Fossey studied during her time in Rwanda, along with a picture of the Rwanda mountains where Fossey set camp for most of her 18 years with the gorillas.


Report: PLAs Drove 28% Of Google Non-Brand Ad Clicks In Q4

2013: The non-brand ROI for Google was 22% higher than that for Bing.


Play The Frank Zamboni Google Doodle

2013: The interactive Doodle allows searchers to play the Zamboni game by cleaning the ice after it is being used by children, hockey players and ice dancers.


How To Blackout Your Site (For SOPA/PIPA) Without Hurting SEO

2012: Google’s advice was to use a 503 HTTP status code to tell spiders that the website was temporarily unavilable.


Google Honors Martin Luther King Jr. Day With “I Have A Dream” Logo

2012: The Doodle included a couple lines from his famous speech.


Google Plans To Buy Its Way Into Real Estate

2010: Actually … Google’s goal was to acquire one to two companies a month. Not necessarily just small real estate companies.


Why Don’t Google & Yahoo Offer Twitter Search?

2009: if you want to find Twitter’s powerful, compelling material, you went to Twitter. Not to Yahoo. Not to Google, the king of search. 


Microsoft’s Early Mistakes & Missed Opportunities With Paid Search

2009: From bCentral Keywords, to not purchasing Overture, and its failed attempts to buy Yahoo.


Google Names Marchex First ‘Google AdWords Authorized Reseller Technology Platform’

2009: The purpose of this designation was to allow Google to recommend an approved technology partner to AdWords resellers if those partners didn’t have the requisite scalable technology.


Kogan ‘Indefinitely’ Delays Android Phone Launch

2009: Due to technology compatibility issues.


Search In Pictures: Googlers Dancing, Yahoo’s New CEO & Dinner Impossible At Yahoo

2009: The latest images culled from the web, showing what people eat at the search engine companies, how they play, who they meet, where they speak, what toys they have, and more.


December 2008: Search Engine Land’s Most Popular Stories

2009:  Search Engine Land’s 10 most popular stories from December 2008.


Google Writes Google Sitemaps FAQs

2008: Google responded to some of the most asked questions on the Sitemaps protocol and how Google supported it.


Google “Online Marketing Challenge” Has 724 College Teams Signed Up To Be Local SEMs

2008: When Google tried to turn college and business school students into search marketers.


Microsoft And EDGAR Online In Content, Ads Deal

2008: Microsoft became the exclusive third-party provider of display, contextual and video advertising for EDGAR Online and its global audience of more than 2 million unique visitors per month.


Yahoo’s Senior VP Of Entertainment Business Leaves

2008: Media executives at Yahoo were leaving one after another.


Hungry For Yahoo? Feed Yourself From Ya-Hoo! Baking Co.

2008: “not affiliated with Yahoo!, Inc. the search engine company.”


Google No Longer Linking To Yahoo, MapQuest Maps

2007: Previously, a search for an address would have yielded a choice among Google Maps, Yahoo and MapQuest.


comScore: Google Wins Again & IE7 Doesn’t Stop Microsoft’s Slide

2007: When Google had just a 47.3% search market share.


New Google Checkout Promo; New Google Trust Worries

2007: Google added the ability to filter product search results from merchants using Google Checkout – and promoted the new feature within its main search results.


Disclosing When Queries Are Autocorrected

2007: It seemed like Google had moved beyond suggesting spelling corrections to automatically doing them for you. 


MetaGlossary – Two Million Terms, Phrases And Acronyms

2007: It worked in a similar manner to the Google define: function, though it tended to be more comprehensive.


Wikiseek: Leveraging Wikipedia For Web Search, Poorly

2007: A look at the disappointing service along with a revisit to how it was different from Search Wikia.


Vizu Seeking To ‘Democratize’ Market Research

2007: It offered an interesting model, creating financial incentives for publishers to show polls on their sites and a relatively inexpensive way for marketers to get quick user data.


comScore: Email Top Reason For Mobile Internet Access

2007: “33% strongly agree they ‘feel lost’ without their cell phone.”


Past contributions from Search Engine Land’s Subject Matter Experts (SMEs)

These columns are a snapshot in time and have not been updated since publishing, unless noted. Opinions expressed in these articles are those of the author and not necessarily Search Engine Land.


< January 15 | Search Marketing History | January 17 >


About the author

Danny Goodwin
Staff
Danny Goodwin has been Managing Editor of Search Engine Land & Search Marketing Expo - SMX since 2022. He joined Search Engine Land in 2022 as Senior Editor. In addition to reporting on the latest search marketing news, he manages Search Engine Land’s SME (Subject Matter Expert) program. He also helps program U.S. SMX events.

Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.

Get the must-read newsletter for search marketers.