How Accurate Is ChatGPT?

When you use ChatGPT, have you ever stopped in your tracks, looking at this marvellous creation of human beings and wondered, “Just how accurate is ChatGPT?“

We certainly did! We did some research and asked ChatGPT about this (getting the information straight from the figurative horse’s mouth). We have explored how accurate ChatGPT is by examining three factors that influence the accuracy and what their implications are.

This blog post presents what we found out. As far as possibble, we will break down the information into simple terms and avoid technical jargon, making it easy for everyone to understand.

Training Data Quality

The quality of the data ChatGPT was trained on is crucial – that’s where ChatGPT’s “intelligence” came from after all.

ChatGPT learns from a diverse range of information from online and offline sources. This includes books, articles, websites, published papers, and journals. Ideally, the training data has to be accurate, authoritative, diverse, balanced, and comprehensive. The higher the quality of the data, the better ChatGPT’s responses will be.

Thanks to the vast and diverse training data it has been exposed to, ChatGPT’s accuracy is typically high. However, inherent limitations, including potentially inaccurate and outdated information, can influence its accuracy.

Model Architecture and Size

Behind ChatGPT is a sophisticated system that is trained on vast amounts of data and generates responses to user requests. Technical folks call this ‘model’.

The processing capabilities of the model, as well as its size, affect ChatGPT’s performance. Larger and more advanced models can capture more complex patterns and generate more accurate responses. For example, GPT-4, with 1.76 trillion parameters, is more capable than GPT-3.5 with “only” 175 billion parameters.

It is generally agreed that the advanced architecture and large size of ChatGPT enable it to produce high-quality, impressive responses — although there is always room for improvement.

Model Updates

ChatGPT is constantly evolving. For example, from 2022 to 2024, it has evolved from ChatGPT-3.5 to ChatGPT-4, and subsequently ChatGPT-4o and 4o mini.

With each update, more recent and diverse training data are fed into ChatGPT, helping the model stay up-to-date and handle a wider range of topics accurately. The underlying algorithms that power ChatGPT are also enhanced to improve its capabilities to understand context, handle ambiguity, and generate more coherent responses. Fixes and improvements based on user feedback are implemented too.

These regular updates are key factors that help improve ChatGPT’s accuracy.

Common Limitations and Errors

Anyone using ChatGPT for long enough will inevitably run into some common limitations and errors. Let’s see if the following examples are familiar to you.

Factual Inaccuracies

ChatGPT is known for occasionally making up information that isn’t accurate. This behaviour is called ‘hallucination’ in the AI world. To be fair, this is a common issue not just with ChatGPT but also other AI models like Google’s Bard. The many research papers generated with AI tools that contain fake academic references are good examples of this.

Biases

ChatGPT can reflect biases found in the data it’s trained on. In other words, if the data includes biased views on certain topics, the responses from ChatGPT might be biased too.

Lack of Common Sense

ChatGPT lacks the common sense that humans possess, because its responses are generated based on patterns rather than a true understanding of scenarios.

For example, if you ask it, “I have a 12 litre jug and a 6 litre jug and I want to measure 6 litres. How do I do it?” (This was a question that Prof Yejin Choi has employed to test ChatGPT’s common sense.)

The most straightforward answer is to fill the 6-liter jug, but ChatGPT might suggest more complex methods due to its lack of common sense reasoning.

(Captured on 25 July 2024 from Prof Yejin Choi’s TED Talk on YouTube, “Why AI Is Incredibly Smart and Shockingly Stupid”, https://www.youtube.com/watch?v=SvBR0OGT5VI)

Vagueness and Ambiguity Handling

While ChatGPT is good at understanding context, it can struggle with vague and ambiguous prompts. For example, if you ask ChatGPT to “Tell me about cat”. The query is so broad that even a human will not be certain about what you’re expecting — let alone the AI.

Are you interested in the common characteristics of cats? Scientific information? Fun facts?

Without a clear direction, ChatGPT could either only offer some general responses or make guesses. And from what we’ve experienced, it’s not very good at guessing, and therefore might not fully satisfy the user’s specific needs.

Conclusion

We have looked at key factors that influence ChatGPT’s accuracy, including the quality of its training data, the architecture and size of its model, and the impact of regular updates. These elements work well together for ChatGPT, enabling it to provide impressive and high-quality responses.

However, it’s important to be aware of its common limitations and errors. Remember that its responses could be factually inaccurate sometimes. Your takeaway from this video? Do maintain a healthy dose of scepticism and perform due diligence while you enjoy the conveniences offered by ChatGPT – and any AI tools!