AI Doesn’t Hallucinate. It Bullshits! (In the Psychological Sense)

In the context of AI, hallucination is a misleading metaphor that fools you into making incorrect assumptions about what AI is and how it works. By using humanlike language like “hallucination”, AI companies deflect accountability and build unwarranted hype around their product, making it seem much more capable than it is.

Let’s unpack why the term hallucination isn’t a good metaphor, why it’s more accurate to call all AI output bullshit, and why what you call it matters.

Disclaimer: In psychology, bullshit means communication that’s indifferent to both truth and falsehood. In this article, every time the word is used, it’s referring to this kind of communication.

• Calling undesired AI output hallucinations is misleading, as it implies hallucinations are bugs. Hallucinations in LLMs aren’t bugs — the process that produces desired and undesired outputs is one and the same.
• LLMs hallucinate all the time; you only notice it when the output is particularly bad.
• Using humanlike language for AI terms creates a serious business risk by masking the technology’s unpredictability.

Metaphorical extensions — what they are and how they’re supposed to work

Metaphorical extension is what you get when a word’s meaning expands to include new concepts based on structural or functional similarities.

Take viruses, for example. A virus is a biological entity that invades a host organism and disrupts biological functions. A computer virus is malicious code that invades a host file or program and corrupts system operations.

One may be biological and the other digital, but you can see how calling this malicious code a virus is helpful. It lets even people who don’t know much about computers quickly and correctly intuit what a computer virus is.

This is a metaphorical extension at its best. And, if you think human hallucinations and AI hallucinations share these kinds of similarities, think again.

Hallucination isn’t a metaphorical extension — it’s glosslighting

Using the term hallucination for inaccurate AI output doesn’t help you to intuit what AI hallucinations are. That’s because it’s not a metaphorical extension — it’s glosslighting.

Glosslighting is a naming convention that deceptively uses evocative, humanlike words to build hype, obscure understanding, and shape public perception, all while hiding behind a narrow technical description for plausible deniability.

Let’s see how this works by using AI agents as an example. The word “agent” suggests an autonomous entity with free will and intent. The narrow technical definition of an AI agent is “a program that maps inputs to outputs, possibly with a reward signal or iterative loop.”

The humanlike term inspires more confidence and suggests grander capabilities than the technical definition. It also deflects blame from the company to the agent.

The same applies to every humanlike term used in AI, from intelligence and understanding to thinking and reasoning — they’re all glosslighting; not one is a metaphorical extension that lets you intuit what’s actually going on.

Glosslighting lets companies get all the financial and social benefits of tricking the public into thinking AI is practically human, while dodging legal, scientific, and moral accountability. Investors and politicians hear these impressive words and assume the technology is incredibly advanced, which helps the AI companies drive excitement, boost stock prices, and secure massive funding.

And hallucination is the most harmful example of glosslighting for you, because it misleads you about the sheer scope of inaccuracy you should expect from AI. This can seriously harm your business and its reputation, so it’s good to understand exactly what you’re dealing with.

So what are hallucinations, really?

In humans, a hallucination is an error in perception and neurochemistry. Something in your brain goes haywire, and you see or hear things that aren’t real as a result. You can think of it like a bug. It’s also binary — you’re either hallucinating, or you’re not.

None of this is true with AI hallucinations — they’re not bugs, and they’re not binary.

The technical definition of AI hallucinations is: “Output from a language model that is fluent and plausible but factually wrong, unsupported, or fabricated.” This technical definition most closely resembles the psychological concept of bullshit — grammatically accurate text or speech that’s indifferent to truth and lies.

A liar is someone who knows the truth and intentionally deceives. A bullshitter doesn’t care either way if what they say is true or false. Large language models (LLMs) are bullshitters.

But let’s focus on the 2 key differences between human hallucinations and LLM bullshit, because they reveal a lot about the scope of the glosslighting at play here.

AI hallucinations aren’t bugs — they’re baked in

To explain why hallucinations aren’t bugs, let’s quickly go over how LLMs work. An LLM is just a next-token predictor.

Think of LLM tokens as words — it’s not 100% correct, but it’s close enough for you to completely understand the point.

The LLM looks at all the tokens in the conversation so far and outputs the next token based on complex statistical probability calculations. Because an LLM isn’t a database, it has nowhere to retrieve answers from, so the token-prediction process is basically guessing. The LLM just guesses one token at a time.

Here’s an example: When you ask an LLM what the capital of Japan is, it doesn’t know the answer. Instead, it calculates that the token for Tokyo correlates the most highly with tokens for Japan and capital city. The math is more complex than I’m making it out to be, but it always ends with the LLM assigning 1 token the dominant probability.

If it outputs “Tokyo”, we say that it’s telling the truth, and if it outputs “Kyoto”, we say that it’s hallucinating. But there is no difference between desirable and undesirable outputs to the LLM. The exact same process, working as designed, generates both outputs, which means the undesirable output isn’t the result of a bug. It’s the result of a gamble that didn’t pay off.

To use a simple analogy, if you threw a die hoping for a 6, and it landed on a 1, you wouldn’t say that the die hallucinated. It did exactly what it was supposed to; it generated a statistically probable output. (You just didn’t like the output.)

AI hallucinations aren’t binary — LLMs always hallucinate

It’s better for AI companies if you think hallucinations are binary, because it frames hallucinations as rare errors produced by an otherwise truth-telling machine. But this just isn’t the case. To quote former OpenAI founding member Andrej Karpathy: “Hallucination is all LLMs do.” We only notice it when the output is especially obvious.

To illustrate how this works, let’s once again compare LLM outputs to dice rolls, only this time, we’ll throw 10 dice. The sum total of this roll is anywhere between 10 and 60, with 35 being the average.

The LLM’s output is only completely accurate when you roll 35. If you get a number that’s way off from the average, like 13 or 54, it’ll be obvious to you that the AI hallucinated. But what if the output is 27 or 39? The hallucination could be small enough that you don’t notice it, but perhaps not so small that it can’t cause harm.

So, LLMs hallucinate all the time; it’s just a matter of how much. Even if the output is completely factually accurate, it’s still a result of the same gambling process — it’s a dice roll that you got lucky on; bullshit that happened to be true.

Why calling LLM output bullshit matters

Using glosslighting instead of proper metaphorical extensions for inaccurate outputs reframes a predictable failure mode of LLMs as a humanlike quirk. To paraphrase a United Nations University article, this doesn’t just mislead people and hinder public understanding of AI; it also “fuels both breathless hype and dystopian fear.”

This can have actively harmful effects at both the business and personal levels (and at the societal level as well, though the effects there are more speculative).

On a business level, undesirable LLM output can lead to monetary and reputational harm:

• Lawyers who used ChatGPT to write a legal brief went to court with invented case law citations and fictitious quotes, leading to embarrassment and sanctions.
• Alphabet Inc. lost an estimated $100 billion in market value after their Bard chatbot claimed the James Webb Space Telescope took the first-ever picture of a planet outside the solar system — something that had been done 20 years prior.
• Deloitte had to pay a partial refund to the Australian government after the $290,000 report they were hired to make was flagged as full of AI hallucinations. 

On a personal level, undesirable LLM output can be deadly.

So let’s call a spade a spade, which means calling both desirable and undesirable LLM output what it is — bullshit that sometimes just so happens to be true.