ChatGPT fails: 13 common errors and mistakes you need to know

I’ve written about ways to use ChatGPT for SEO and content marketing-related tasks, like keyword research, crafting title tags and local SEO.

I use it at least once daily for a variety of tasks. The more you use the platform, the more valuable it will be to you to understand its limitations.

The founder of OpenAI (the company that built ChatGPT) has called the tool “incredibly limited” and a “horrible product” and has cautioned users about how much they rely on it:

ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness.

it’s a mistake to be relying on it for anything important right now. it’s a preview of progress; we have lots of work to do on robustness and truthfulness.

— Sam Altman (@sama) December 11, 2022

Multiple publications have published content with errors on meaningful YMYL (“Your Money or Your Life”) topics like personal finance and health.

This article discusses the most common technical errors and the frequent mistakes when using ChatGPT and how to address them.

Common ChatGPT errors

If you’ve been using ChatGPT for a “long” time (i.e., since late 2022), you’ve probably seen the prompt below at some point.

Obviously, a new software product in an emerging technology field that’s massively popular will run into problems.

If you’re making ChatGPT an integral part of your daily workflow, knowing the common issues you may encounter is helpful.

ChatGPT also has issues familiar to any website, such as 404 not found errors, 5xx internal server errors, 403 access denied errors, and Cloudflare errors like 1020 and 524 errors.

Often these can be fixed by clearing your cache, using (or turning off) a VPN, or ensuring you’re typing the web address correctly.

And there are also several errors more specific to the platform:

1. API issues

Often, users may hit specific character limits, token limits, or limits in functionality with the ChatGPT API.

2. Character limits

There are limits to how many characters you can enter into ChatGPT and how many characters ChatGPT responses can be.

They vary depending on your version (e.g., GPT-3.5 has an input limit of 2048, and GPT-4 is 4,096) and in the web interface versus the API.

3. Connection time out on long responses

If ChatGPT is processing a long and complex response, it will sometimes just time out.

You might see this when asking the platform to perform functions like writing code or analyzing and responding to long inputs.

4. Unfinished responses

Not an error per se, but frequently, if you ask ChatGPT to write code or a long text, it will just stop mid-way.

You can type continue, and it will finish within the next few prompts in these instances.

5. Limited messages

ChatGPT, at the time of writing, limits the number of messages you can use per time period, even for paid accounts. 

Similarly, there is specific rate-limiting for different GPT versions, OpenAI tools, and account types.

6. Login issues

Depending on the level of access required or session duration. ChatGPT may log you out, or you may not have access to certain features with certain account statuses.

Most of these issues can be resolved by:

• Reworking your prompts to fit into character limits.
• Making sure you’re logged in and can access the functionality you’re attempting to use with your account level.
• Asking ChatGPT to regenerate responses.
• Starting a new chat.

---

Common ChatGPT mistakes

ChatGPT makes many mistakes, including:

• Getting verifiable facts wrong.
• Detailing a process it “supposedly” used to generate an answer that is impossible given the platform’s limitations.
• Generating broken code.
• Broken formatting or ignoring instructions in the output.
• And more.

Other users have detailed instances of the platform lying to get a CAPTCHA solved and issues with simple math problems related to age and distance.

Some of this has improved with GPT-4, but not all. Here is a prompt and output using the latest version of ChatGPT:

While it’s fun to do “gotcha” prompts and catch ChatGPT saying incorrect or goofy things, it’s more beneficial to be aware of areas where ChatGPT can be wrong – the “categories of wrongness,” if you will.

Understanding these will make you both a better prompt engineer and output editor. Let’s walk through the most common categories of mistakes I know.

7. Confident but wrong

The “hallucination” issue is a significant problem for ChatGPT and other generative AI solutions.

ChatGPT will answer all kinds of questions that it doesn’t know the answer to in a compelling way. That doesn’t make the answer right.

The example screenshot above shows that it doesn’t use any qualifying language or hedge at all. You get a decisive answer without equivocating.

8. Not understanding word and character counts

ChatGPT is terrible at estimating and understanding character counts in any precise way. 

If you’re doing work related to SEO and content, this is a major limiting factor. You might be looking to write title tags, meta descriptions, or content where meeting a word count is necessary:

This is actually pretty close. I’ve had instances where the tool was much farther off.

9. Mistakes in reasoning

I searched for “logic puzzles” and grabbed the first list of “easy logic puzzles” I found.

ChatGPT got the first three puzzles correct, then the fourth one wrong:

10. Coding errors

There are several real-life examples now of people using ChatGPT to build websites, apps and browser extensions or troubleshoot coding issues.

That said, ChatGPT can also generate code that doesn’t work. I’ve encountered the following issues when using ChatGPT to help code:

• The code just doesn’t function properly. (Actions not being executed when users use a tool.).
• Calculations built by ChatGPT’s code being incorrect.
• Using the wrong API. (Often, it’s because it isn’t aware of the API you want to use because the API came out or was updated after the training data period for GPT, which ended in late 2021. Despite this, ChatGPT doesn’t let you know.)
• Ignoring or “forgetting” instructions and project requirements or constraints.

Again, it’s imperative you always check the code you’re deploying carefully and test the output of anything ChatGPT helps with. 

Some users have also reported that GPT- 4 or even legacy mode (rather than the new default, Turbo) leads to better results for coding tasks.

11. Outdated information

The training data for ChatGPT may be updated at some point and various plugins and tools have started to integrate web crawling and real-time data.

Still, the web version of ChatGPT currently can’t crawl the web and isn’t aware of information from before late 2021.

This is important, particularly when coupled with the “confident but wrong” issue.

12. Short memory

There is a limit to how many characters within a chat ChatGPT will “remember” so it will frequently make mistakes due to losing the context of your early prompts or its early answers.

13. Formatting mistakes

As with coding, if you ask ChatGPT to perform functions like creating a table or formatting text in a certain way, it will sometimes make mistakes or ignore specific instructions.

Avoiding and addressing ChatGPT’s mistakes

So how can you get the most out of ChatGPT while dealing with the platform’s issues and limitations?

Focusing on these key areas can help you avoid much of the potential downside of using ChatGPT in your day-to-day tasks:

Be aware of limitations

Staying up to date with specific product limitations like (i.e., character limits for prompts and responses, the date of training data for GPT, etc.) will help you craft prompts that generate good responses and QA outputs better. 

OpenAI does a good job of updating this information with new releases. Some items here are things they’ve explicitly said they are addressing.

Review and edit outputs carefully

This should be fairly obvious, but you must carefully edit any text and QA code before publishing or deploying it. 

Due to how convincingly confident ChatGPT can be, I’d strongly recommend having a domain expert review content and code before it’s deployed. 

It’s relatively easy to be fooled by some of the content and assertions put out by ChatGPT, especially if a topic isn’t your area of expertise.

But if you have deep experience on a topic, it’s easier to catch misleading or inaccurate statements.

Layer tools and processes on top of the API 

OpenAI has rolled out plugins and made a ChatGPT-specific API available. 

You can address some of the platform’s limitations by using different tools that others create or by creating some yourself.

Now that you know the potential pitfalls of ChatGPT, you can go on and establish guardrails for using the tool – leveraging its benefits while avoiding many of the negatives.