The Best Guess You Ever Made

Read this sentence slowly and notice something strange: by the time your eyes reach the middle of any line, you already know roughly where it is going. You anticipate the shape of the next word before it arrives. This is not a party trick of literacy. It is the ordinary machinery of a brain doing what it does every waking moment, which is to run slightly ahead of the world and bet on what comes next.

We like to imagine that seeing is a passive act. Light enters the eye, the world imprints itself on the retina, and the brain, like a diligent clerk, files an accurate copy. It is a comforting picture. It is also almost entirely wrong. The brain is not a recording device. It is a prediction engine, and the reality you inhabit is less a photograph than a hypothesis, continually tested against the thin stream of data your senses can supply.

Most of the time the hypothesis is good enough that you never notice it is a hypothesis at all. You call it reality and get on with your day. But the guesses do fail. And when they fail, they leave fingerprints. Illusions, misheard words, the peculiar case of a blue dress that half the internet swore was white. Each of these is a moment when the machine slips and shows its hand.

The clerk that was never really a clerk

The idea that perception is a matter of inference rather than transcription is not new. It goes back at least to the German physicist and physiologist Hermann von Helmholtz, who in the 1860s proposed that seeing involves what he called unconscious inference ¹. The retina, Helmholtz pointed out, receives an impoverished and ambiguous signal. A given pattern of light could have been produced by countless different arrangements of objects in the world. To settle on one interpretation, the brain must fill in what the data leave out, drawing on assumptions built from a lifetime of experience.

Helmholtz understood that this filling-in happens beneath awareness and at extraordinary speed. You do not consciously reason your way from a smear of color to the recognition of a friend’s face. The inference is already made by the time the face arrives in consciousness. What you experience as direct contact with the world is in fact contact with a model of the world, assembled on the fly and delivered to you as finished fact.

There is a physical reason the brain cannot afford to simply wait and see. Neural signals are slow. Light strikes the retina, but the information has to be converted into electrical activity and ferried through several stages of processing before it reaches the regions that produce conscious vision. By the time you become aware of a moving object, it has already moved. A brain that only reported where things had been would be permanently behind, and a creature so lagged would be poor at catching prey or dodging a thrown stone. The solution is to extrapolate. The brain projects the near future and presents that forecast as the present, quietly reconciling the accounts when the real signal catches up.

Predictions down, errors up

Modern neuroscience has taken Helmholtz’s hunch and given it a formal architecture. The framework is called predictive coding, and its central claim is deceptively simple. The brain is organized as a hierarchy. Higher regions send predictions downward about what lower regions should be sensing. Lower regions compare those predictions against the incoming data and send back only the discrepancy: the prediction error, the part the higher level failed to anticipate ².

This is a strikingly economical way to run a nervous system. If a prediction is accurate, almost nothing needs to travel upward. The brain has already accounted for what is there. Only surprise, the mismatch between expectation and reality, is worth the metabolic cost of transmission. In this picture perception is a negotiation. The top of the hierarchy proposes a story about the world, the bottom pushes back wherever the story diverges from the evidence, and the story is revised until the argument settles.

The most ambitious version of this idea belongs to Karl Friston, a neuroscientist at University College London, who folded predictive coding into something he named the free energy principle ³. Friston’s proposal is sweeping to the point of being vertiginous. Any system that persists over time, he argues, must resist the tendency of things to fall into disorder, and it can do so only by minimizing surprise about the states it encounters. A brain, on this account, exists to keep its predictions and the world in agreement, either by updating the prediction to fit the world or by acting on the world to make it fit the prediction. Every glance, every reach, every belief is in the service of reducing uncertainty. The paper in which Friston laid out the groundwork has since been cited tens of thousands of times, a measure of how thoroughly the field has taken the prediction hypothesis to heart.

Caught in the act

The theory would be little more than elegant speculation if it could not be seen at work. It can, and the clearest demonstrations come from the moments when prediction goes wrong.

Watch a magician perform the vanishing ball illusion. She tosses a ball into the air twice, and on the third throw the ball simply disappears, seemingly at the top of its arc. In fact there was no third throw. The magician palmed the ball and only mimed the toss, following the imaginary trajectory with her eyes. Yet spectators vividly report seeing the ball rise and vanish. Their brains, having learned the pattern from the first two throws, predicted the flight of a ball that was never launched. The prediction was so confident that it generated a visual experience with no corresponding light. The trick works entirely inside the forecast.

The late British psychologist Richard Gregory spent much of his career arguing that illusions are precisely this, the brain’s guesses caught in the act ⁴. His favorite example was the hollow mask. Take a mask of a human face and turn it around so you are looking at the concave inside. Under most conditions you will not see a hollow at all. You will see a normal, convex face bulging toward you, and if it rotates, it will appear to move in an impossible direction. The reason is that your brain holds an overwhelming prior belief: faces are convex. It has seen billions of them and never once encountered a face that caved inward. Confronted with sensory evidence of a concave face, the brain simply overrules the evidence. The prior wins the argument, and you see the face your model insists must be there.

Something similar happens in hearing. In a noisy environment, on a bad phone line, in a crowded room, the acoustic signal reaching your ears is riddled with gaps and interference. You do not perceive those gaps. Your brain patches them with its best linguistic guess, supplying phonemes and whole words that never physically arrived. Researchers have shown that when a sound is spliced out of a spoken sentence and replaced with a cough or a burst of noise, listeners hear the missing sound as clearly as if it had been present, a phenomenon known as phonemic restoration ⁵. You are, in a real sense, hearing your expectations. A meaningful fraction of everyday conversation is reconstructed rather than received.

The dress, and other controlled hallucinations

The neuroscientist Anil Seth, at the University of Sussex, has a phrase for all this that manages to be both provocative and precise. Ordinary waking perception, he says, is a controlled hallucination ⁶. A hallucination is what the brain produces when its predictions run free of correction. Perception is the same generative process, only reined in and kept honest by sensory input. When the guesses happen to match the world, we honor them with the name reality. The difference between seeing and hallucinating is not the mechanism. It is only whether the world gets a vote.

No single episode illustrated this more publicly than the photograph of a striped dress that circulated in 2015. Some people saw it as blue and black. Others, looking at the identical image on the identical screen, saw white and gold, and neither group could understand how the other had gone so wrong. The image was genuinely ambiguous about the light falling on the fabric, and each brain made an unconscious assumption to resolve it. Brains that assumed cool, bluish daylight discounted the blue and saw white and gold. Brains that assumed warm, artificial light discounted the yellow and saw blue and black. The pixels were the same for everyone. What differed was the prediction each brain made about the illumination, and that hidden bet decided the color that reached awareness.

The reach of prediction does not stop at the senses. Lisa Feldman Barrett, a psychologist at Northeastern University, has argued that emotions themselves are constructions the brain builds rather than reactions it merely undergoes ⁷. A pounding heart and a jolt of arousal do not arrive with a label attached. That same bodily state could be fear before a job interview, excitement on a first date, or the flush of anger in an argument. The brain assigns the meaning, drawing on context and a lifetime of prior episodes, predicting which emotion best fits the situation and then serving that interpretation up as raw feeling. On this view you do not simply have emotions. Your brain forecasts them, and the forecast becomes the experience.

Prediction is not part of perception. It is perception.

It is tempting to treat all of this as a set of curiosities, a collection of quirks and bugs bolted onto an otherwise faithful sensory system. That would be a mistake, and it would miss the real force of the idea. Prediction is not a feature the brain adds on top of perceiving. According to predictive coding, prediction is the act of perceiving. The generative model comes first. It proposes the world, and the senses are relegated to a checking role, correcting the proposal only where it strays too far from the incoming evidence.

This inverts the intuitive order of things. We assume that data flows in and belief flows from it. The predictive brain works largely the other way around. Belief flows down, in the form of expectations about what should be sensed, and data flows up only to the extent that it violates those expectations. What you consciously experience is not the sensory signal. It is the winning hypothesis. The signal’s job is merely to arbitrate between competing guesses, and when the guess is good, the signal barely needs to speak at all.

Living inside the forecast

Once you take the prediction hypothesis seriously, a surprising amount of ordinary mental life comes into sharper focus. Consider how stubborn a first impression can be. Meet someone in an unflattering moment, and your brain forms a prediction about who they are. Every subsequent encounter is then filtered through that expectation, and the evidence that would revise it tends to be discounted as noise, exactly as the hollow mask discounts the concavity it cannot believe in. The prior clings on, not out of malice or laziness, but because the machine is built to trust its model and demand a great deal of contrary evidence before rewriting it.

The framework has reached further, into medicine and mental health, where its implications are still being worked out. Some researchers now think of certain forms of chronic pain as predictions that have outlived their cause. Pain is not a simple readout of tissue damage. It is the brain’s best guess about the threat to the body, and like any prediction it can become self-sustaining, persisting after an injury has healed because the model has learned to expect it and keeps generating the experience ⁸. Anxiety has been described in similar terms, as a predictive system stuck in a mode that forecasts threat where little exists, so that the world is perpetually experienced as more dangerous than it is.

These descriptions are not merely poetic. If experience is generated by prediction, then changing the prediction ought to change the experience, and there is growing interest in therapies that work on exactly this principle, teaching the brain to expect safety, or to reinterpret a bodily sensation, and letting the perception shift with the expectation. The placebo effect, long dismissed as a nuisance to be controlled for, starts to look like a straightforward demonstration of the point. Believe a treatment will ease your pain, and for many people it genuinely does, because the belief is itself an input to the model that constructs the feeling.

There is something almost consoling in all of this. To learn that you have never had unmediated access to reality, that you have only ever known your brain’s rolling forecast of it, might sound like a loss. But it is also an account of a remarkable achievement. From a trickle of noisy, delayed, ambiguous signals, your brain builds a stable and useful world, and does it so seamlessly that you spend your whole life mistaking the construction for the thing itself.

So the next time an illusion catches you out, when a hollow face bulges toward you or a ball vanishes into thin air or a dress refuses to settle on a color, resist the urge to feel foolish or fooled. You have not been deceived by a defect. You have been granted a rare look behind the curtain. For an instant, the guess and the world came apart, and you got to watch your brain doing what it does every waking second, out loud and in the open: making its best guess about what is out there, and calling it the truth.

Sources

1. Helmholtz, H. von, Handbuch der physiologischen Optik, 1867 (English: Treatise on Physiological Optics) — https://en.wikipedia.org/wiki/Unconscious_inference
2. Rao, R. P. N. & Ballard, D. H., Predictive coding in the visual cortex, Nature Neuroscience, 1999 — https://www.nature.com/articles/nn0199_79
3. Friston, K., The free-energy principle: a unified brain theory?, Nature Reviews Neuroscience, 2010 — https://www.nature.com/articles/nrn2787
4. Gregory, R. L., Knowledge in perception and illusion, Philosophical Transactions of the Royal Society B, 1997 — https://royalsocietypublishing.org/doi/10.1098/rstb.1997.0121
5. Warren, R. M., Perceptual restoration of missing speech sounds, Science, 1970 — https://www.science.org/doi/10.1126/science.167.3917.392
6. Seth, A. K., Being You: A New Science of Consciousness, Faber & Faber, 2021 — https://www.penguinrandomhouse.com/books/566315/being-you-by-anil-seth/
7. Barrett, L. F., How Emotions Are Made: The Secret Life of the Brain, Houghton Mifflin Harcourt, 2017 — https://lisafeldmanbarrett.com/books/how-emotions-are-made/
8. Moseley, G. L. & Butler, D. S., Fifteen Years of Explaining Pain, The Journal of Pain, 2015 — https://www.jpain.org/article/S1526-5900(15)00680-2/fulltext

Related reading

• The Theater That Runs Behind Your Eyes
• The Rep You Cannot See

More from the Mind edition →