Why a rare word can be a better clue
A search for “jaguar” might retrieve pages about the animal, the car, and a sports team. Add a distinctive term such as “habitat,” and the results become easier to sort. The reason is that a word’s usefulness depends not only on whether it appears in a page, but also on how common it is across the collection being searched.
Inverse document frequency captures that second part. A document is one item in a search index, such as a web page. If a term appears in nearly every document, finding it tells the search engine little about which page is relevant. If it appears in only a small fraction, its presence is a stronger clue.
Turning rarity into a ranking signal
The calculation works roughly like this: count how many indexed documents contain a word, then give that word less weight as the count rises. In practice, the score is usually scaled with a logarithm, so an extremely rare term does not overwhelm every other clue.
Search systems can combine this rarity score with term frequency—how often the word appears in a particular page. A word repeated in a page may matter to that page, but it should not dominate just because it is common everywhere. Together, the two signals help distinguish a page that is specifically about a topic from one that merely contains a few matching words.
Why the idea still matters
The method does not understand meaning the way a person does. It cannot, by itself, tell that “jaguar” has several senses or that two different words can describe the same thing. Its strength is more basic: it turns the distribution of words across many documents into a practical ranking clue. That principle remains useful even as search engines add newer methods, because a rare, relevant term can still help point toward the page a reader actually wants.
