Why surprise counts as information
Information theory measures how much a message changes what you can reasonably expect. If a symbol is certain in advance, seeing it confirms your prediction but teaches you nothing new. If several symbols are possible, learning which one appeared reduces uncertainty. The less likely an outcome was, the more information its arrival carries.
This is a measure of surprise, not importance. A rare character can be informative even if it has no practical meaning; a familiar warning can matter enormously despite being easy to predict.
A message needs structure, not just surprises
Consider a stream of random static. Its next sound is difficult to predict, so it can carry a great deal of surprise from moment to moment. But the sounds do not form words or tell a coherent story. Randomness alone is not the same as a useful message.
Structure makes signals interpretable. In ordinary language, some letters and words are more likely than others, and grammar narrows what might come next. That predictability can make a passage easier to follow while also reducing how much new information each character contributes. A genuinely unexpected word can stand out precisely because the surrounding sentence sets up expectations.
Why predictability can be useful
Predictable messages are not worthless. They can be easier to understand, remember, or transmit reliably. In data compression, for example, common patterns can be represented compactly because a decoder can anticipate them; unusual details need more bits to specify. The useful balance is neither maximum predictability nor maximum randomness, but enough structure to make a signal meaningful and enough novelty to tell the reader something they did not already know.
