Google explains the use cases for its different crawler types
There is Googlebot, special-case crawlers and user-triggered fetchers.
Topic: Robots.txt
Google launches a new crawler named GoogleOther
This crawler is similar to Googlebot, but it will be used internally by Google teams to crawl the public web.
Should you block ChatGPT’s web browser plugin from accessing your website?
It's a hot debate in SEO right now: You can block OpenAI's ChatGPT-User bot with robots.txt – but should you? Short answer: No. Here's why.
Crawl efficacy: How to level up crawl optimization
Crawl budget is a vanity metric. Your goal should be to guide Googlebot in crawling important URLs fast once they are published or updated.
Technical SEO testing: How Googlebot handles iframes
Here are some test scenarios that may lead Google to index the content inside an iFrame, while assigning that content to its parent URL.
Putting lipstick on a pig: Fix the website or fail at SEO
If a website is not performing well, optimizing it is just putting lipstick on a pig. Here’s what to do to ensure your site succeeds in SEO.
We’ve crawled the web for 32 years: What’s changed?
A look back at the history of search and SEO and a preview of what the next iteration of the internet means for marketers.
Microsoft Bing drops anonymous sitemap submission due to spam issues
You can now only submit sitemaps to Bing using robots.txt or Bing Webmaster Tools.
Google Search Console’s URL parameter tool is officially not working
Google will not use any of the legacy rules in the tool, so make sure to watch your site's SEO health over the next days and weeks.
5 ways to use the Wayback Machine for SEO
Use the tool to recover lost information and glean insights into the direction of competitor strategies.
4 technical SEO tasks that are critical to organic success
Technical SEO is a necessity. Here are four key technical tasks SEOs should take care of to help increase organic visibility.
Google’s URL Parameters tool is going away
Google said this is because the tool has "low value."
SEO dos and don’ts of website migration
Site migrations are necessary at times, but rankings and traffic can drop if you're not careful. At SMX Next, Bruce Clay showed how marketers can ensure SEO success during site migrations.
Deepcrawl launches technical SEO app for Wix
Wix’s SEO efforts continue with an app designed for small and mid-sized enterprises.
Google adds new robots tag indexifembedded
This new robots tag lets you define if you want your content indexed when it's embedded through iframes or other means.
Microsoft Advertising will deprecate ETAs too; Wednesday’s daily brief
Plus, October 8 is the Diversity Award nomination deadline
We asked, you answered: Should Google add noindex support to robots.txt? Poll results
In total, 84% of the 87 respondents said “yes,” they would like this feature.
Fancy like????… why brands should pay attention to TikTok
Plus, Microsoft’s new credit card ads
Google categorizes what organic search traffic drops look like
This is the first time Google illustrated what different types of traffic drops look like.
Google rankings, quality and relevancy; Wednesday’s daily brief
Also, non-AMP content now in top stories carousel.
Google is pushing hard to make it easier to create ads; Tuesday’s daily brief
Plus, Shopify site owners can finally edit their robots.txt files.
Finally, Shopify site owners can edit their robots.txt files
The rollout began on Friday and should now be available for all sites.
Skyrocket your rankings with a head-to-toe website audit
On average, each crawled web page has at least one critical error—make sure to fix yours
Google local car dealership inventory search results
Google seems to be getting into the local car inventory space with a new search service.
Google updates the Search Console coverage report
The updated report now gives you very specific categories and labels for errors and responses.
Bing revamps Site Explorer in new Bing Webmaster Tools
This tool gives SEOs information on how Microsoft Bing does just that and then gives you more data on clicks, impressions, links, errors, warnings, indexing issues and much more.
Bing’s new robots.txt tester can help SEOs identify crawling issues
The new Bing Webmaster Tool can also be used to edit robots.txt files and check URLs for errors instantly.
Microsoft launched revamped Bing Webmaster Tools
With this launch, Bing added a URL inspection tool with some exciting features.
Bing Webmaster Tools gets a new robots.txt tester
Here is another tool you can use from Bing to see how BingBot can or cannot access your web pages.
Common Shopify SEO pitfalls and how to avoid them
What Shopify store owners and SEOs can do to overcome platform challenges and rank higher.
Does Google respect the URL parameters tool?
E-commerce sites should not make assumptions about Google’s crawling parameter - check the log files to confirm activity.
WSJ: GoogleBot can add products to shopping carts
Make sure to double check those abandoned cart metrics; they may be inflated.
Common JavaScript errors that can harm your SEO [Video]
And, why JavaScript may not be the real cause of your crawling issues.
Common oversights that can impede Google from crawling your content [Video]
SEOs and developers shouldn’t over-engineer workarounds when basic, reliable solutions already exist, says Martin Splitt in this clip from Live with Search Engine Land.
Don’t try to reinvent the SEO wheel, says Google’s Martin Splitt
Splitt addresses the most common indexing and JavaScript mistakes, why he doesn’t take ranking questions and shares why Google’s communications aren’t always as proactive as we’d like.
Oops, LinkedIn temporarily drops out of Google Search
Even the biggest sites on the internet can have problems with their SEO and Google.
Mixed Directives: A reminder that robots.txt files are handled by subdomain and protocol, including www/non-www and http/https [Case Study]
Since robots.txt files impact crawling, it’s important for site owners to understand how they are treated by search engines. Columnist Glenn Gabe covers two cases involving multiple robots.txt files (by subdomain and protocol).
Google’s new treatment of nofollow links has arrived
Here’s what you need to be aware of as Google begins viewing nofollow links as a hint for crawling and indexing.
Google Search Console launches new removals tool
The new tool temporarily block URLs from showing in Google, it shows outdated content and content filtered by SafeSearch.
Online tax-filing services can no longer hide free services from Google Search results
TurboTax and others are prohibited from noindexing their free e-file landing pages, per new IRS rules.