• Search Engine Land
  • Sections
    • SEO
    • SEM
    • Mobile
    • Local
    • Retail
    • Google
    • Bing
    • Social
    • Home
  • Follow Us
    • Follow
  • Search Engine Land
  • SEO
  • SEM
  • Mobile
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • More
  • Events
    • Follow
  • SUBSCRIBE

Search Engine Land

Search Engine Land
  • SEO
  • SEM
  • Mobile
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Newsletters
  • Home
SEO

Sign up for our daily recaps of the ever-changing search marketing landscape.

Note: By submitting this form, you agree to Third Door Media's terms. We respect your privacy.


Bing says it is improving web crawler efficiency

Bing is working on making sure its crawler doesn't miss new content and at the same time overload your web servers.

Barry Schwartz on October 17, 2018 at 9:19 am
  • More

Fabrice Canel, principal program manager for Bing Webmaster Tools, provided an update on his team’s efforts to improve the efficiency of their web crawler, BingBot.

Responding to user feedback. The update is a follow up to his talk at SMX Advanced in June, during which he announced an 18-month effort to improve BingBot. Canel asked the audience to submit suggestions and feedback.

In a blog post Tuesday, Canel said the team has made numerous improvements based on this feedback and thanked the SMX audience for its contributions. He said they will “continuing to improve” the crawler and share what they’ve done in a new “BingBot series” on the Bing webmaster blog.

BingBot’s goal. In this first post, Canel outlined the goal for BingBot, which is to use an algorithm to determine “which sites to crawl, how often, and how many pages to fetch from each site.” To ensure site’s servers aren’t overloaded by the crawler, the goal of BingBot is to limit its “crawl footprint” on a site while ensuring content in its index is as fresh as possible.

This “crawl efficiency” is the balance Bing is working to strike at scale. Canel said, “We’ve heard concerns that bingbot doesn’t crawl frequently enough and their content isn’t fresh within the index; while at the same time we’ve heard that bingbot crawls too often causing constraints on the websites resources.” It’s a work in progress.

Why should you care? Bing is clearly listening to the webmaster and SEO community. The Webmaster Tools team is making changes to ensure its crawler does not overload your servers while at the same time are faster and more efficient when it comes to finding new content on your web site. Bing is actively working on this and says it will continue to work on this.

How does this impact you? If you add new content to your web site and Bing doesn’t see it, it won’t rank it. That means searchers using Bing will not find your new content.

Recently Bing shut down the anonymous submit URL tool, and we have seen reports that Bing is not listening to submit URL requests even in Bing Webmaster Tools. It is possible the tweaks and changes Bing is making is causing some of this slowness with crawling and indexing now. But ultimately, Bing is clearly working on the issue.

Canel will be speaking at SMX East in New York City next week. See the full agenda here.



About The Author

Barry Schwartz
Barry Schwartz is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on SEM topics.

Related Topics

Bing SEOChannel: SEOSEO: Crawling and Robots

Sign up for our daily recaps of the ever-changing search marketing landscape.

Note: By submitting this form, you agree to Third Door Media's terms. We respect your privacy.


We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

ATTEND OUR CONFERENCES

Lorem ipsum doler this is promo text about SMX events.

May 21-22, 2019: SMX London

Sept 30 - Oct 1, 2019: SMX Advanced Europe

November 18-19, 2019: SMX Paris

February 19-20, 2020: SMX West

March 18-19, 2020: SMX Munich

June 8-10, 2020: SMX Advanced

November 11-12, 2020: SMX East

×


Learn More About Our SMX Events

Gain new strategies and insights at the intersection of marketing, technology, and management. Our next conference will be held:

April 15-17, 2020: San Jose

October 6-8, 2020: Boston

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • How Good Partnerships Should Reduce Your Workflow
  • Digital Insights 2019: How marketers confront the obstacles of digital customer engagement
  • 14 Industry Experts on the Future of Attribution
  • The Essential Guide to Protecting Your Ad Spend from Invalid Traffic
  • How to Turn Leads into Repeat Customers
See More Whitepapers

Webinars

  • The Fantastic 5: Emerging strategies to drive conversions & growth
  • CCPA Goes Live in Weeks: Is your marketing measurement ready?
  • 4 Ways to Get Started With Agile Marketing
See More Webinars

Research Reports

  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Enterprise SEO Platforms
  • Call Analytics Platforms
See More Research

Search Engine Land’s Guide To SEO

Master the Science of SEO
Sign up for our NEW daily brief.
Search Engine Land
Download the Search Engine Land App on iTunes Download the Search Engine Land App on Google Play

Channels

  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social

Our Events

  • SMX West
  • SMX London
  • SMX Advanced
  • SMX East
  • MarTech West
  • MarTech East

Resources

  • White Papers
  • Research
  • Webinars
  • Search Marketing Expo
  • MarTech Conference

About

  • About Us
  • Contact
  • Privacy
  • Advertise
  • Marketing Services
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • Youtube
  • iOS App
  • Google Play

© 2019 Third Door Media, Inc. All rights reserved.