AI: The Illusion of Understanding

Abstract

In 1980 John Searle imagined a man locked in a room with a rulebook, shuffling Chinese characters he could not read, and asked whether anything there understood anything. The thought experiment has framed the debate ever since. This paper argues that it no longer fits the machines we now argue about, and that the human understanding it quietly holds up as the standard is in no shape to serve as one. Searle's insistence on the difference between real understanding and a convincing imitation belongs to its moment: the same years gave us Baudrillard on simulation, a counterfeiting industry large enough to need its own federal statute, and a Coca-Cola that had to relabel itself Classic because the product alone could no longer prove it was the original. Insisting on genuineness is what people do when the copies are getting good. The thought experiment also relies on our being able to look over the man's shoulder and see that he is only matching symbols — and in a modern network there is no rulebook to read, not even for the people who built it. The usual escape, that Searle was only ever talking about the clunky AI of his day, is one he closed himself when he widened his position in later work. The paper then turns a term borrowed from AI engineering, context rot, back on us. Our own understanding decays in much the same way: memory rewrites what it stores, emotion tilts the scales before we notice, and every speaker runs a private version of the language that never quite matches anyone else's. Watch the word justice in the mouths of Putin, Trump and Khomeini — each is certain he is simply using it correctly. Recent findings from AI interpretability, on traits that pass between models through data that never mentions them, on memory that is periodically rebuilt, and on emotion-like states that measurably change what a system does, are brought to bear on an argument written in 1980. The conclusion turns the title around. The illusion of understanding may not be a machine problem at all. It may be what understanding has always been — partial, shaped by where you stand, and good enough to act on. The real question is how useful that illusion is to us.