AI optimization: How to optimize your content for AI search and agents
To be visible, optimize your site with clean HTML, metadata, fast responses, and bot-friendly configurations.
Topic: Robots.txt
Top 5 takeaways from the 2024 Web Almanac SEO Chapter
Discover key learnings from a deep dive into web – including robots.txt files, robots directives, canonical use, dynamic rendering, and more.
Your guide to sitemaps: best practices for crawling and indexing
Sitemaps make it easier for search engines to discover and index your pages. Here's all you need to know and how to create one.
11 essential SEO elements you should be tracking
Stay on top of your SEO performance by tracking both technical and on-page changes, and using the right tools to simplify your efforts.
How to deal with SEO emergencies
From rankings drops to site issues, learn which SEO problems require urgent action and how to prioritize your efforts effectively.
What is edge SEO?
Streamline SEO updates by implementing changes at the network's edge, improving website speed and flexibility.
Google updates crawlers and user-triggered fetchers documentation
With that, Google expanded how Google products are affected by each crawler and gave robots.txt examples for each crawler.
Reddit CEO to Microsoft and AI search engines: Pay for our content
Reddit CEO Steve Huffman does not believe all web content – especially theirs – is free for AI models and search engines to crawl and use.
Website migration checklist: 11 steps for success
Website migration is complicated. Eliminate errors and help streamline the site migration process with these steps.
Microsoft confirms Reddit blocked Bing Search
Reddit updated its robots.txt file prohibiting Bing and some other search engines from crawling its site.
Google streamlines product listings via website crawl
Google is making it easier for merchants to surface product listings across its surfaces through automated 'website crawl' feeds.
Driving traffic to gated content and paywalled sites: SEO tips + examples
Explore SEO tactics to effectively market premium content and drive site traffic while preserving the value of your gated assets.
Google adds two new Googlebot crawlers: GoogleOther-Image and GoogleOther-Video
Google said these new crawlers are to "better support crawling of binary data that may be used for research and development."
13 questions to diagnose and resolve declining organic traffic
Is your site's organic traffic taking a nosedive? Use these questions to investigate and fix issues impacting your SEO performance.
Best of SearchBot: Guide me through a technical SEO audit
We ask Search Engine Land's SearchBot how to carry out an effective technical SEO audit.
6 things that happen when Googlebot can’t crawl your website
An experiment by technical SEO expert Kristina Azarenko reveals what happens you disable Googlebot from crawling a website using robots.txt.
Number of websites blocking Google-Extended jump 180%
The New York Times, Yelp and 22 Condé Nast properties are among those who have chosen to block Google-Extended so far.
Google Search Console adds robots.txt report
Google has sunset the robots.txt tester.
12 SEO pitfalls to avoid during a website platform migration
Learn how to safeguard your website's SEO performance during a platform migration with these tips for a smooth transition.
Google updates Favicon search documentation
Google removed the Favicon user agent and said you must allow Googlebot and Googlebot-Image for Google to index your site's favicon.
Google-Extended does not stop Google Search Generative Experience from using your site’s content
Instead, you need to block SGE by blocking Googlebot completely.
Overstock’s Bed Bath & Beyond rebrand: The SEO impact
Merging two websites is difficult and complex. Find out how Bed, Bath & Beyond did – plus SEO considerations for a migration.
How SEOs can detect and address user data leaks
Discover the hidden dangers of user data exposure and why SEOs play a crucial role in data safeguarding and privacy protection.
Google introduces Google-Extended to let you block Bard, Vertex AI via robots.txt
You can now prevent Bard and Vertex AI generative APIs, and future generations of models, from from accessing your website, or parts of it.
26% of the top 100 websites are now blocking GPTBot
The number of popular websites blocking ChatGPT has increased 250% in the past month. Pinterest, Indeed and more big brands now block GPTBot.
Google to block Bard’s shared chats from showing in Google Search
After Google released shared conversations in Bard, Google Search began crawling and indexing those public chats.
Microsoft adds controls to disallow content in Bing Chat
This utilizes NOCACHE and NOARCHIVE tag and more features may come in the future.
What your enterprise SEO audit may be missing
Build a strong SEO foundation for your enterprise website. Learn how to bridge the gaps in your technical SEO with this guide.
Bing explains the new Sitemap Index Coverage report
After quietly rolling out the new sitemap in Webmaster Tools, Bing explains how it works.
Dozens of big brands have blocked GPTBot, OpenAI’s new web crawler
At least 15% of the top 100 websites and 7% of the top 1,000 websites are blocking GPTBot, a new analysis finds.
XML sitemaps and SEO: What you need to know
You can’t trust search engine bots to find all your URLs. Give them a guide with your XML sitemap. Learn the SEO benefits here.
Google: All online content should be available for AI training unless publishers opt out
The search engine has sent its proposal to the Australian government, urging policymakers to change existing copyright laws.
GPTBot: OpenAI releases new web crawler
You can now prevent OpenAI's ChatGPT from accessing your website, or parts of it, using robots.txt.
How long SEO takes to work
Learn the average timeframe to see results from SEO and key factors that influence your SEO program's performance.
Robots.txt is not the answer: Proposing a new meta tag for LLM/AI
Using robots.txt for managing data usage in LLMs is the wrong approach in this new age of generative AI products. Here's why.
Crawlers, search engines and the sleaze of generative AI companies
AI companies see the openness of the web as permitting large-scale crawling to obtain training data. So what should publishers do?
How to improve enterprise-level crawling and indexing efficiency
Here are three real examples of what happens when enterprise SEO goes wrong – and how you can safeguard large websites from costly mistakes.
A guide to diagnosing common JavaScript SEO issues
Learn how to spot SEO issues in JavaScript-powered sites and what you need to know when it comes to rendering.
Google to explore alternatives to robots.txt in wake of generative AI and other emerging technologies
Google said it will engage with the web and AI communities with public discussions during this process.
Google to deprecate Sitemaps ping endpoint later this year
Instead, Google is recommending an accurate lastmod date in your Sitemap file.