ChatGPT shortlists: How to benchmark visibility and find room to compete

AI search is quietly becoming our personal procurement department. The question is: are you on the vendor shortlist?

A recent Semrush survey of B2B professionals shows how AI search is becoming a gatekeeper for buying decisions:

• 97% say AI has helped them discover new vendors.
• 92% say AI has shaped their vendor shortlist.
• 83% say AI influenced their final vendor decision.

But what if your brand isn’t showing up?

I analyzed ChatGPT’s shortlists for workforce management software across five prompt variations, running each prompt 20 times and examining the responses, sources, and fan-out queries.

Here’s how to use that research to choose which shortlists to compete for. The research is available to download at the end of the article.

What is a ChatGPT shortlist?

A ChatGPT shortlist is a curated list of brands chosen from a larger pool of possibilities.

But how short are these lists? In our research on workforce management (WFM) software, ChatGPT shortlists featured an average of 5.6 brands across 100 responses.

That’s a narrow selection in a crowded category. The SHRM vendor directory has 215 listings in WFM’s time, attendance, and expense reporting category.

Here’s how one of the shortlists looks:

There are three key elements to notice in this table:

• Mentions: Named brands, in this case, are listed in the “Software” column.
• Best For: ChatGPT describes which audience or need each recommendation suits.
• Inline citations: Links within the table that help users research the recommendations.

This represents one instance of one prompt. To select the shortlists a brand decides to compete for, we need to account for two challenges:

• AI prompts are harder to predict than traditional searches.
• AI search can deliver different responses to the same prompt.

Find the questions that create shortlists

ChatGPT search prompts are long and conversational. Traditional searches were short and precise.

Comparing January to February 2025 with the same period in 2026, Semrush found that “the average prompt length for search-enabled queries nearly doubled: from 4.7 words to 8.7 words.”

Longer, more conversational prompts give buyers more ways to describe the same need. We don’t have to predict their exact words to study which brands ChatGPT recommends.

Start with a buying question in the category, then look for these qualities in the answer:

• Does the response create a shortlist of recommended brands?
• Is this list relevant to your buyers?
• Does your brand offer a solution to this need?

Don’t look at a prompt and think: This is or isn’t something my prospect would type. Look at the response and see if it includes a shortlist your buyer would use.

Dig deeper: What blog posts should you write to be mentioned in ChatGPT?

Track shortlist visibility across a topic

We started with a specific question: “What is the best WFM software for time and attendance?”

But buyers can approach that same topic through comparisons, alternatives, or specific tasks.

I use a framework of five related questions to sample shortlists across a topic.

These five questions cover different ways into the topic. But ChatGPT recommends different brands even when the question stays the same. That’s why we need to run each prompt multiple times.

Establish a shortlist visibility baseline

Run each prompt multiple times to establish a baseline showing which brands make the shortlist and how often. A single response can’t tell us whether a brand appears regularly or only occasionally.

In AI search, randomness is a feature, not a bug.

As Search Engine Land’s Danny Goodwin wrote:

• “SEO has been deterministic until now. Certain inputs resulted in fairly predictable outputs.”

AEO is probabilistic: Type the same prompt twice in a row, and the results are different by design.

Graphite’s research on randomness in AI answers recommends that you “run each prompt at least 10 times for a quick estimate.”

For our WFM research, we ran each of the five prompts 20 times, capturing the response, sources, and fan-out queries together from Chrome DevTools for each run. That gave us 100 captures. We started a new Temporary Chat for each run and kept the prompt wording exactly the same.

Next, we’ll combine the results into a scorecard showing which brands earned shortlist seats across the topic.

Dig deeper: 80% of ChatGPT product recommendations change when search is enabled: Study

See who’s winning the shortlist seats

The brands consistently winning shortlist seats don’t necessarily have the highest authority and traffic. Our scorecard combines 100 responses across five prompts to show who gets recommended, how their domains compare, and which pages ChatGPT cites.

Here’s how to read the scorecard:

• Shortlist seats: How many of the 100 runs put the brand in a shortlist.
• Domain authority score and domain organic traffic: We’re using Semrush data. Use your preferred tool.
• Most-cited page: The page linked to most often within that brand’s shortlist-table rows.

Look for these patterns in the scorecard:

• Who dominates: Do a few brands appear across most shortlists, or are seats spread across many competitors?
• Who outperforms: Which brands earn frequent recommendations despite lower domain authority and traffic?
• Which content gets cited: Are recommendations supported by product pages, research, reviews, or other page types?

A seat on the shortlist puts a brand into the buyer’s consideration set. The “Best For” column helps the buyer decide if it’s a fit. But does that description match how the brand wants to be positioned?

What does ChatGPT say your brand is best for?

Shortlist visibility matters, but so does the positioning that comes with it. Being described inaccurately is bad for lead generation.

When ChatGPT assigns “Best For,” it’s looking across many content sources (brand, competitor, third party, etc.). Monitor these descriptions to see whether ChatGPT presents your brand accurately.

In our research, we grouped “Best For” for the five brands with the most shortlist seats:

Look for two dimensions in “Best For”:

• Audience: Are these the buyers you want to reach?
• Use Case: Is this the need you’re targeting?

Now that we’ve looked at how the shortlist gets presented, let’s see how it gets created.

Dig deeper: How to become the answer in AI search: The ASC framework

Not every shortlist seat is created equal

When we see five brands named on a shortlist, we think we’ve got five chances to break in.

But ChatGPT isn’t starting with a blank slate when it builds a shortlist. In our WFM research, it named specific brands in its search queries before any search results came back.

Behind the scenes, ChatGPT turns a prompt into multiple web searches, a process called query fan-out.

Across our 100 captures, we recorded 570 fan-out queries issued before any search results came back. The examples below show how brands were already named as ChatGPT searched the category, vendor websites, features, reviews, and product releases.

Some brands are already part of the search before the shortlist is created. For a brand looking to break in, the question is how many seats go to brands outside that starting group.

Find the shortlists your brand can break into

When choosing a shortlist topic to compete for, first look for openings.

Which brands does ChatGPT name in its fan-out queries? And which other brands make the resulting shortlist?

We compared each shortlist with the queries from the same run. Brands not named before search results returned held 89 of 563 seats (16%) and appeared in 45 of the 100 tables.

Use this table to identify openings. Then, examine the pages cited for brands earning those seats: What type of content is ChatGPT citing, and whose sites publish it?

AI search is the new gatekeeper

AI search is taking on market exploration and discovery for buying teams. It narrows a crowded market to a small set of vendors.

These shortlists have become both gatekeeper and influencer, deciding who gets presented and how buyers see them.

G2 found that 69% of surveyed software buyers said an AI chatbot led them to select a different software vendor than initially planned. Thirty-three percent purchased from a vendor they’d never previously heard of.

Brands need to know:

• Which prompts produce shortlists.
• Which brands earn seats and how often.
• What ChatGPT says each brand is “Best for.”
• Which pages those shortlists cite.
• Which brands are fast-tracked into the list.

If your brand isn’t on the shortlist, that’s the starting point. Find out who’s on it, how ChatGPT presents them, how they present themselves, and where there’s room to compete.

The research, prompts, and methodology are available to download here.