Artificial Insecurity: threats to information integrity

Every week, hundreds of millions of people ask ChatGPT deeply personal health and wellbeing questions, often uploading their blood test results, scan images, or medical records in the process. Putting aside obvious concerns about such data remaining private and secure — as we explored in our previous blog on confidentiality, that’s far from a given — the question arises: can you trust the answers churned out by chatbots and other AI tools? In this second part of our blog series on digital security and AI, let’s dive into this topic, and investigate large language models (LLMs) from the perspective of information integrity.

// Does AI really just make stuff up? 

The short answer is yes. According to the confidentiality-integrity-availability (CIA) triad, safeguarding integrity means ensuring that data is “trustworthy, complete, and [hasn’t] been accidentally altered or modified by an unauthorized user.” With traditional software systems, threats to integrity might involve unauthorized third parties tampering with data, but the most relevant problem for LLMs is their tendency to repeatedly and confidently produce incorrect and flawed outputs.

The reason for this is simple: LLMs are akin to fancy auto-complete programs, predicting a likely response to your prompt in the same way a messaging app suggests the next words as you type out a sentence. A chatbot’s answers sound plausible, and are indeed sometimes accurate, because LLMs are trained on huge amounts of text, images, and other media. But as a rule, we should be surprised when they don’t make mistakes, given their propensity for error. 

Whether you call these mistakes ‘hallucinations,’ ‘confabulations,’ or ‘failed approximations’ (our preferred term, as it frames all LLM outputs as approximations, whether successful or not), a growing body of research suggests they are technically inevitable. This doesn’t mean, however, that these errors cannot be mitigated to some degree. One widespread approach, known as Retrieval Augmented Generation (RAG), allows LLMs to retrieve information from specific external sources, rather than inventing a seemingly plausible answer. For instance, were you to ask ChatGPT for the address and phone number of a particular business in your city, it might return a completely or partially false result. RAG forces the chatbot to consult a reliable database, such as the Golden Pages, and retrieve any relevant information before answering your query.

This sounds great in theory, but while RAG techniques are successful in reducing errors, they do not reliably or completely eradicate them. Even when using the most advanced error reduction methods, LLMs, and systems built on them, are consistently prone to generating incorrect information. This issue persists even with the most advanced LLMs. Meanwhile there is evidence that smaller, more specialized models may be more accurate within specific domains, suggesting that bigger isn’t better when it comes to LLMs’ accuracy.

// How dangerous are AI’s failed approximations? 

The consequences of AI churning out inaccurate information can be life-changing and, in some cases, life-threatening. One place where you absolutely don’t want to see AI-generated failed approximations cropping up is a courtroom; yet a recent study of just two months’ worth of U.S. court cases identified more than 22 separate cases in which “courts or opposing parties found non-existent cases within filings.” Similar instances have been noted in the UK and Australia, involving lawyers, but also people representing themselves. 

Of course, dishonest lawyers and unscrupulous lawyering predate the use of advanced AI tools, and professional bodies are developing guidance to prevent unethical usage; even if, in the face of a push for the legal profession to adopt AI tools, lawyers may neglect to learn about appropriate mitigation measures and inappropriate uses. But it is deeply problematic that AI tools are being marketed in such a way that suggests they are suitable for, and capable of doing, independent specialized legal work, especially when the consequences of AI getting vital details wrong can be severe, for lawyers and litigants alike. 

When we look at people’s use of medical tools for medical purposes, the risks are even higher. As mentioned earlier, people are increasingly turning to ChatGPT and similar chatbots to answer health-related questions. But recently-published research from the University of Oxford demonstrated that, when asked about specific, real-world symptoms, chatbots were mostly unable to correctly identify a person’s medical condition, even if they could identify the same condition in a medical licensing exam scenario. Meanwhile, researchers at Brown University have shown how, when used for mental health advice, chatbots “systematically violate ethical standards of practice” adhered to by mental health practitioners. In a high-stakes, real-world situation where people’s physical or mental safety may be on the line, this is downright dangerous, and may have tragic consequences. 

// Why we can’t avoid threats to information integrity

In the face of such threats, one might think there’s a simple solution to protecting yourself from AI-generated misinformation: just don’t ask ChatGPT, or any other chatbot for that matter, to answer questions about serious or sensitive matters. But the integration of AI overviews and summaries into ‘traditional’ search engines makes avoiding AI-generated answers impossible. In 2024, Google launched AI overviews across its Search offering (which represents a global market share of almost 90%), immediately increasing the amount of LLM-generated outputs that the public is exposed to at the top of search results, whether they like it or not. Unsurprisingly, the tool has been mired in scandal since it launched, most memorably advising the use of non-toxic glue for sticking cheese to pizza and recommending that humans eat one rock per day. 

While such errors, which Google called “isolated incidents,” may seem absurd, others are less of a laughing matter. Similar to the health misinformation generated by AI chatbots, a recent investigation by The Guardian identified a range of medical inaccuracies in Google’s AI-generated summaries, including false information related to cancer or liver disease. While Google has since removed these specific results, and continues to claim that such incidents are unexpected anomalies produced by an otherwise reliable tool, the opposite is true: Google AI Overviews are fundamentally unsafe, insecure, and not fit for purpose. Yet they, and other AI-generated content summaries, are being pushed on people despite the unproven benefits and proven harms.

// From AI assistant to AI salesperson

Beyond the risks posed by specific features, there’s another worrying trend emerging where the very business models underpinning the roll-out of LLMs are compounding threats to information integrity, most noticeably via the introduction of advertising into AI chatbots.

Last month OpenAI announced that it would begin running targeted advertisements in both the free version of ChatGPT and the USD $8/month subscription version. OpenAI claimed that these ads would be clearly labelled as ‘sponsored’ and would not influence chatbot responses. Beyond the privacy risks of introducing targeted ads into potentially sensitive chatbot conversations, such a business model risks jeopardizing the integrity of the information people receive, since chatbot companies will face financial pressure from advertisers to tailor outputs to drive traffic toward their products or services.

So what can we do about these risks? Firstly, people using these services should at the very least have an opt-out, and preferably an opt-in, to LLM-driven features, while the design and marketing of such tools should clearly highlight their limitations. Secondly, people should be able to review clear breakdowns of how models arrive at a particular answer (the ‘chain of reasoning’), as well as which sources were consulted to compile the answer. As we acknowledge the inevitable nature of mistakes in LLMs, it’s critical that we can determine exactly where a system has gone awry — and fix it. 

We must also resist the narrative that we need to integrate ever bigger LLMs across every aspect of our lives. LLMs have their uses, but also clear limitations and drawbacks. As we’ll discuss in the next blog, the frenzied push for “LLMs everywhere, all the time” is also jeopardizing access to and the availability of data, services, and systems we all depend on.

---