Search Engine Land
  • SEO
    • > All SEO
    • > What Is SEO?
    • > SEO Periodic Table
    • > Google: SEO
    • > Bing SEO
    • > Google Algorithm Updates
  • PPC
    • > All PPC
    • > What is PPC?
    • > Google Ads
    • > Microsoft Ads
    • > The Periodic Tables of PPC
  • Focuses
    • > Local
    • > Commerce
    • > Shopify SEO Guide
    • > Content
    • > Email Marketing Periodic Table
    • > Social Media Marketing
    • > Analytics
    • > Search Engine Land Awards
    • > All Focuses
  • SMX
  • Webinars
  • Intelligence Reports
  • White Papers
  • About
    • > About Search Engine Land
    • > Newsletter
    • > Third Door Media
    • > Advertise

Processing...Please wait.

Search Engine Land » Bing » Bing SEO » Bing says it is improving web crawler efficiency

Bing says it is improving web crawler efficiency

Bing is working on making sure its crawler doesn't miss new content and at the same time overload your web servers.

Barry Schwartz on October 17, 2018 at 9:19 am

Fabrice Canel, principal program manager for Bing Webmaster Tools, provided an update on his team’s efforts to improve the efficiency of their web crawler, BingBot.

Responding to user feedback. The update is a follow up to his talk at SMX Advanced in June, during which he announced an 18-month effort to improve BingBot. Canel asked the audience to submit suggestions and feedback.

In a blog post Tuesday, Canel said the team has made numerous improvements based on this feedback and thanked the SMX audience for its contributions. He said they will “continuing to improve” the crawler and share what they’ve done in a new “BingBot series” on the Bing webmaster blog.

BingBot’s goal. In this first post, Canel outlined the goal for BingBot, which is to use an algorithm to determine “which sites to crawl, how often, and how many pages to fetch from each site.” To ensure site’s servers aren’t overloaded by the crawler, the goal of BingBot is to limit its “crawl footprint” on a site while ensuring content in its index is as fresh as possible.

This “crawl efficiency” is the balance Bing is working to strike at scale. Canel said, “We’ve heard concerns that bingbot doesn’t crawl frequently enough and their content isn’t fresh within the index; while at the same time we’ve heard that bingbot crawls too often causing constraints on the websites resources.” It’s a work in progress.

Why should you care? Bing is clearly listening to the webmaster and SEO community. The Webmaster Tools team is making changes to ensure its crawler does not overload your servers while at the same time are faster and more efficient when it comes to finding new content on your web site. Bing is actively working on this and says it will continue to work on this.

How does this impact you? If you add new content to your web site and Bing doesn’t see it, it won’t rank it. That means searchers using Bing will not find your new content.

Recently Bing shut down the anonymous submit URL tool, and we have seen reports that Bing is not listening to submit URL requests even in Bing Webmaster Tools. It is possible the tweaks and changes Bing is making is causing some of this slowness with crawling and indexing now. But ultimately, Bing is clearly working on the issue.

Canel will be speaking at SMX East in New York City next week. See the full agenda here.


New on Search Engine Land

    What are your secrets to overcoming marketing challenges? Take our survey

    10 reasons to join us at SMX Advanced online this June

    How to ensure influencers help your SEO campaigns

    Webinar: Dominate your competition with Google auction insights and search intelligence

    Vimeo adds structured data to all public videos to improve Google Search visibility

About The Author

Barry Schwartz
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Related Topics

Bing SEOSEO

Get the daily newsletter search marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS

Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.

March 8-9, 2022: Master Classes (virtual)

June 14-15, 2022: SMX Advanced (virtual)

November 15-16, 2022: SMX Next (virtual)

Learn More About Our SMX Events

Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.

Start Discovering Now: Spring (virtual)

September 28-29, 2022: Fall (virtual)

Learn More About Our MarTech Events

Webinars

Data-Driven Answers to Achieve Omnichannel Success

Is Your Marketing Stack Ready for Omnichannel CX?

Outrank in Organic Search with These 5 Core Tactics

See More Webinars

Intelligence Reports

Enterprise SEO Platforms: A Marketer’s Guide

Enterprise Identity Resolution Platforms

Email Marketing Platforms: A Marketer’s Guide

Enterprise Sales Enablement Platforms: A Marketer’s Guide

Enterprise Digital Experience Platforms: A Marketer’s Guide

Enterprise Call Analytics Platforms: A Marketer’s Guide

See More Intelligence Reports

White Papers

Realising the power of virtual events for demand generation

The Progressive Marketer’s Ultimate Events Strategy 2022 Worksheet

CMO Guide: How to Plan Smart and Pivot Fast

The Retail Renaissance Report, USA Edition: 4 Keys to Predicting Online & In-Store Demand Across Global Markets

Thinking Bigger About Marketing Budgets

See More Whitepapers

Receive daily search news and analysis.

Processing...Please wait.

Topics

  • SEO
  • PPC

Our Events

  • Search Marketing Expo - SMX
  • MarTech

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2022 Third Door Media, Inc. All rights reserved.