Why traditional keyword research is failing and how to fix it with search intent

After 25 years of working in SEO, I’ve seen firsthand how traditional keyword research methods fail to keep up with Google’s advancements. 

In my SMX Next presentation, I challenged SEOs to go beyond outdated keyword methodologies and embrace an intent-driven approach. 

Here are six key insights from that session.

1. Traditional keyword research is failing us

Traditional keyword research is no longer enough. 

We’ve relied on tools that provide data on competition, search volume, and relevance, but they don’t uncover the hidden context behind searches.

For years, SEOs have prioritized high-volume, low-competition keywords, assuming this would drive results. 

While this may have worked for the simpler, lexical-based Google algorithm of the early 2000s, this approach falls short because it ignores search intent.

For example, a keyword like “solar panels” may have high search volume. 

But without context, it’s impossible to determine whether users are looking for products, financing options, or general information. 

Without understanding intent, marketers risk attracting traffic that never converts. 

Today, success depends on moving beyond search volume and focusing on search intent.

Dig deeper: How to optimize for search intent: 19 practical tips

2. Google is an AI search engine

Google isn’t one monolithic AI algorithm – it’s a collection of AI systems working together to:

• Understand queries.
• Classify content.
• Deliver the best results.

Here’s what’s changed:

• Google has improved its understanding of keywords and content.
• There is a strong emphasis on user experience, with Google prioritizing content that is easy for users to consume.
• Google ranks pages based on relevance to intent, even if the exact keywords are missing.

For SEOs, this means that content must align with search intent – not just keywords. 

Well-structured, high-value content that directly addresses users’ questions will outperform pages optimized solely for keyword density.

Dig deeper: Content mapping: Who, what, where, when, why and how

3. The best way to uncover intent? Read the SERPs

The number one way to understand search intent is to study the search engine results pages (SERPs).

Rather than guessing what a keyword means, analyzing what Google is already ranking provides a clear picture of the dominant intent behind a query.

For example, I once worked with an ecommerce company selling biscotti cookies. 

Initially, they targeted high-volume keywords like “chocolate biscotti,” expecting strong results. 

However, a quick SERP analysis revealed that most top-ranking results were recipes, not product listings.

This indicated that searchers weren’t looking to buy biscotti – they wanted to bake it. 

Instead of chasing high-volume terms with mismatched intent, the company shifted its focus to lower-volume keywords with strong purchase intent, ultimately improving conversions.

Blindly following keyword tools without SERP analysis can lead to content that attracts traffic but fails to convert.

4. Prioritize search intent over keywords

The real question isn’t just what keywords people are searching for – it’s why they’re searching.

As Google increasingly prioritizes intent over keywords, SEO strategies must evolve accordingly. A three-step process can help align keyword research with search intent:

Identify target intents

Before diving into keyword research, define 5-6 core search intents that align with business goals. Examples include:

• “Compare mortgage rates” (for financial services)
• “Best protein powders for weight loss” (for fitness brands)

Filter keywords by intent

Rather than focusing solely on search volume and competition, filter keywords based on clear purchase or action intent. 

This approach refines traditional keyword research to focus on what actually drives conversions.

Choose content formats that match intent

Content should match the searcher’s intent, which often requires moving beyond standard blog posts. Some high-performing content formats include:

• Comparison articles (“Best budget vs. premium running shoes”)
• Niche buying guides (“How to choose an ergonomic office chair”)
• Interactive tools (e.g., mortgage calculators, pricing estimators)

By aligning keywords with intent and content formats, SEOs can dramatically improve engagement and conversion rates.

Dig deeper: Rethinking your keyword strategy: Why optimizing for search intent matters

5. Invest in content formats that convert better

Middle-of-the-funnel content – like comparison pages, niche buying guides, and Q&A pages – tends to rank better and convert more effectively than generic blog content.

With AI-driven search results delivering direct answers, traditional educational blog posts are losing traction. 

To stay competitive, marketers must create high-value content that serves the searcher’s next step.

Some of the best-performing content types include:

• Comparison content (“Best DSLR cameras under $1,000”).
• Niche buying guides (“Ultimate guide to ergonomic keyboards”).
• Interactive tools (e.g., ROI calculators, pricing estimators).
• Video-first content, which improves engagement and differentiation.

Shifting to intent-driven content formats can significantly boost both rankings and conversions.

Dig deeper: Writing people-first content: A process and template

6. Use AI wisely, but prioritize customer insights

AI tools are valuable for analyzing SERPs and understanding search intent, but they are not a substitute for real customer insights.

The best way to understand what searchers want is to talk to actual customers. Conversations, chat logs, and feedback from sales teams offer deeper intent insights than AI alone.

For those who don’t have direct access to customers, speaking with sales representatives can be just as effective. 

Sales teams repeatedly hear the same customer questions, making them an excellent source of content ideas and keyword strategy insights.

Dig deeper: How to optimize your 2025 content strategy for AI-powered SERPs and LLMs

[Watch] Next-generation SEO keyword research: Shift from traffic to search intent

Want to take your SEO strategy to the next level? Watch my full SMX Next 2024 session here.