I just corrected an AI after it kept insisting a given date was a Wednesday. The correct answer, trivially verified, was Thursday.
After Claude confidently and incorrectly declared no less than three times that April 16 was indeed, a Wednesday, and "perfect for your webinar" - I finally corrected it. "You are absolutely right" - it said - and we continued brainstorming.
This is not a rare occurrence. Anyone who has spent serious time using large language models has had the same experience: a response that is impressively articulated and broadly correct, coupled with an error that a reasonably attentive human would catch immediately.
The reaction this tends to provoke is confusion. How can something that writes so fluidly and synthesizes information so well stumble on something so basic?
The answer, unsatisfying though it may be, is that these mistakes are not failures of the system. They are simply a direct consequence of how these models work.
Hallucinations: more feature than bug
For years now, the industry has referred to these errors as "hallucinations." The term implies a malfunction, something that could in principle be fixed with the right patch or training method.
But hallucinations are not a failure mode. They are a structural property of the technology.
At its core, a large language model is performing next-token prediction. Given a sequence of text, it calculates the probability distribution of the next word (or token) and selects one according to that distribution. Everything else emerges from that process.
The model has no internal mechanism for verifying a claim against an external ground truth. [1] It does not "check" facts in the way a person might look up a date or mentally reconstruct a calendar. Instead, it generates the sequence of words that most plausibly follows from the input and its training data.
When the statistical signal is strong, the output can be remarkably accurate. When it is weak or ambiguous, the model still produces an answer. The structure of the system rewards fluency, not epistemic caution.
This is why LLM responses often carry an unusual combination of strengths and weaknesses. They are excellent at producing coherent language and synthesizing patterns from large corpora. They are far less reliable when precise factual verification is required.
Calling these mistakes hallucinations makes it sound as if the system briefly loses its grip on reality. In truth, it never had one.
Humans are wired to recognize patterns, real or imagined, especially in communication. We attribute personality even to inanimate things (do you give your car a name? a raging storm...), we attribute a lot of feelings to non-verbal interactions, such as with animals.
When something like an LLM talks to us, coherently and politely and seemingly authoritatively - all our instincts are to believe it.
The Temptation to Anthropomorphize
A great deal of confusion around AI systems comes from the persistent temptation to treat them as approximations of human cognition.
The interface encourages this. We converse with these systems in natural language. They respond with sentences that appear thoughtful and structured. The interaction resembles a dialogue with an informed assistant.
It is therefore natural to assume that something analogous to human reasoning must be happening inside the model.
But the resemblance is largely superficial. The underlying mechanism is fundamentally different from how human cognition operates.
Human reasoning draws on structured mental models of the world. We form causal relationships, test hypotheses, and evaluate claims against experience. Our errors usually stem from incomplete knowledge, faulty assumptions, or cognitive biases.
Large language models operate on statistical associations in text. They learn patterns in how words tend to follow one another across billions of documents. That training produces systems capable of astonishing linguistic competence, but it does not produce an internal model of the world in the way humans possess one.
This distinction matters because it explains why the systems behave the way they do.
And whether LLMs have a ceiling to their abilities.
A person who knows how calendars work would rarely confuse Wednesday and Thursday in a specific date context. A language model, by contrast, is generating a plausible answer based on patterns in its training data and the prompt it received. If those patterns are weak or inconsistent, the output may simply be wrong.
The mistake is not surprising once we remember what the system actually is.
Why Transformers Alone May Not Be Enough
When transformer models first appeared in the late 2010s, their capabilities expanded so rapidly that many observers began to assume we had discovered the core architecture that would eventually produce general artificial intelligence.
I have been skeptical of that idea from the beginning.
Transformers are extraordinarily effective at modeling language. The architecture excels at identifying relationships between tokens across long contexts, which makes it well suited to tasks like translation, summarization, and code generation.
But language modeling is not the same thing as general reasoning.
Reasoning typically requires a different set of capabilities: persistent memory, structured representations of knowledge, the ability to perform iterative verification, and mechanisms for interacting with external information sources.
The transformer architecture provides some approximations of these capabilities through attention mechanisms and large context windows. But those approximations remain limited.
It is entirely possible that future systems will integrate transformers with other architectures that provide more explicit reasoning structures. Some of that experimentation is already underway. But the notion that scaling next-token prediction indefinitely will automatically produce human-level general reasoning remains unproven.
I myself am experimenting along this vector. Some luminaries of the field (e.g., Yann LeCun) believe that the transformer architecture itself is unsound - that we need some sort of world model that curdles up from a different machine-learning architecture. I'm not so sure that it is needed or at least the only way to address present limitations.
I am in awe of the enormous leaps in AI that have emerged from the transformer architecture. I don't think we need to scrap it and start afresh. I think we can bolt a missing piece onto it that will help tame the accuracy as well as personalize it for our applications and expertise. I think that missing piece may well be bringing back the deterministic "expert systems" type of AI, and marrying it with the creative and extremely capable next-token-prediction machines. I will be sharing more on my experimentation soon. Some very promising results!
In the meantime, we continue to encounter the peculiar combination of brilliance and fragility that characterizes current models.
The Expectation Problem
Much of the frustration people feel with AI systems is really an expectation mismatch.
If we believe these models should exhibit human-level reasoning, then every basic mistake becomes baffling. We find ourselves asking the same question over and over: how can something so intelligent fail so obviously?
But the premise is wrong.
Large language models are not general-purpose thinkers. They are statistical language engines with remarkable generative abilities. Expecting them to behave like human reasoners sets us up for constant disappointment.
This misunderstanding has produced two equally misguided camps.
On one side are the skeptics who dismiss the technology as nothing more than "stochastic parrots." The phrase, popularized in a 2021 research paper critiquing large language models, captures an important truth: these systems reproduce patterns from their training data rather than generating knowledge in the human sense.
But the phrase is often used in a way that understates the real capabilities of these models. Pattern recognition at the scale of modern training corpora is not trivial. It enables forms of synthesis and abstraction that are genuinely useful.
On the other side are the enthusiasts who believe organizations can replace large portions of their workforce with autonomous AI agents.
This view makes the opposite mistake. It assumes that systems capable of producing fluent language can also handle the messy, context-dependent reasoning that much real work requires.
Neither position is substantiated by the current state of the technology or the arc of its progress.
Where the Real Value Appears
The practical value of LLMs emerges once we stop asking them to be something they are not.
They are extraordinarily effective tools when used within structured workflows that compensate for their weaknesses.
In practice, this means building scaffolding around the models.
Retrieval-augmented generation (RAG) systems allow models to pull information from verified databases rather than relying solely on internal training data. Model Context Protocol (MCP) frameworks provide structured ways for systems to interact with external tools and services. Review loops introduce human or automated validation steps before outputs are accepted as final.
Agent frameworks extend this idea further by breaking complex tasks into smaller steps, each of which can be evaluated or corrected along the way.
None of this eliminates hallucinations entirely. But it dramatically reduces their impact.
More importantly, it reframes the role of the model. Instead of acting as an autonomous decision-maker, the LLM becomes one component in a larger system designed to produce reliable results.
This is not unlike how we structure human workflows. Complex projects rarely rely on a single individual working in isolation. They involve layered processes, reviews, and tools that support the final outcome.
AI systems benefit from similar structures.
Productivity Gains Without the Hype
When used this way, the productivity gains are real.
Developers use LLMs to accelerate coding tasks, generate test cases, and document complex systems. Analysts employ them to summarize research and explore preliminary interpretations of data. Writers use them to draft and reorganize ideas.
In each of these cases, the model functions less like an independent expert and more like a highly capable assistant that produces drafts and suggestions.
The human remains responsible for verification, judgment, and final decisions.
This hybrid approach may not be as dramatic as the narratives that dominate technology headlines. It does not promise to eliminate entire categories of jobs or usher in fully autonomous knowledge work.
But it aligns much more closely with how the technology actually behaves.
The gains accumulate quietly: faster iteration cycles, reduced friction in routine tasks, and the ability to explore ideas more quickly than before.
Those improvements may prove significant over time, even if they fall short of the grander predictions.
The Middle Ground
Every transformative technology tends to produce the same pattern of reaction. Early enthusiasm overshoots reality. Skepticism then swings too far in the opposite direction. Eventually, the technology settles into a more pragmatic role.
Large language models appear to be moving through that cycle now.
They are neither magical reasoning engines nor trivial parrots of internet text. They are powerful statistical systems that happen to operate in the medium of language.
That capability alone opens up a surprising range of applications. But it also imposes limits that are unlikely to disappear simply through scale.
The sooner we recognize both sides of that equation, the sooner we can use these tools effectively.
In the meantime, it remains entirely possible that an AI will confidently tell you that today is Tuesday, when in fact it is a Monday.
Footnotes
- People build all kinds of scaffolding around the foundation model (i.e., a "harness"), to mitigate this to some degree.