How to Lie with Statistics

“If your sample is large enough and selected properly, it will represent the whole well enough for most purposes. If it is not, it may be far less accurate than an intelligent guess and have nothing to recommend it but a spurious air of scientific precision.”

Huff uses this quote to discuss statistical samples; it is part of his more extensive discussion on correct sampling. He criticizes using bad sampling or sampling that is too small for statistical analysis. Bias in the sample often skews the results far enough from reality that it is useless. The only thing it has for support is the veneer of respectability gained from being “scientific.” As he says, the reader is better off guessing than trusting the results of a bad sample.

“It is equally true that the result of a sampling study is no better than the sample it is based on. By the time the data have been filtered through layers of statistical manipulation and reduced to a decimal-pointed average, the result begins to take on an aura of conviction that a closer look at the sampling would deny.”

Huff again talks in this quote about issues with faulty sampling practices. The statistician can attempt to make the results appear more legitimate, but this legitimacy is surface-level. If the sample has too many problems, the statistic is flawed from the start, and no amount of dressing it up can change that.

“The operation of a poll comes down in the end to a running battle against sources of bias, and this battle is conducted all the time by all the reputable polling organizations. What the reader of the reports must remember is that the battle is never won.”

Huff warns that an ideal sample free of all sources of bias is an impossible goal. This doesn’t mean that the attempt is unimportant. The work of good statisticians and polling entities is to remove as many sources of bias from their samples as possible. It is also the reader’s job to be aware that unknown biases might still appear even in the case of the best possible sample.

“This bias is toward the person with more money, more education, more information and alertness, better appearance, more conventional behavior, and more settled habits than the average of the population he is chosen to represent.”

Huff calls out one of the significant unconscious biases that appears in choosing what part of the population to select for polling. Polls tend to bias toward reinforcing societal norms or standards of “success” because those who create the polls have their own biases that appear in the construction.

“That is the essential beauty of doing your lying with statistics. Both those figures are legitimate averages, legally arrived at. Both represent the same data, the same people, the same incomes.”

This quote establishes Huff’s discussion on the topic of averages. In the preceding example, Huff gave two “averages” for neighborhood income that varied wildly enough that one could be perceived as an outright lie. Huff uses it to make his point here that statistics do not technically have to lie to be deceptive. Even one change in the analysis method can create a skewed picture of reality.

“You have in reality the case that sounds like a joke or a figure of speech: Nearly everybody is below average.”

This quote is part of Huff’s continued discussion on using the wrong average when discussing incomes. He used the mean to obtain the high number, which skews the average in favor of the high-earning outliers. The average does not reflect reality. Instead, it creates an impossible standard that most people can never meet.

“The importance of using a small group is this: With a large group any difference produced by chance is likely to be a small one and unworthy of big type.”

This quote establishes why a large sample size is crucial and why many misleading statistics rely on small samples. In a large group, the variations disappear unless they have real significance. With a small sample, the differences look more dramatic and can be better used to prove a point. In cases of advertising or opinion-making, the smaller size can lead to a better campaign for whatever the entity wants to support.

“Hardly anybody is exactly normal in any way, just as one hundred tossed pennies will rarely come up exactly fifty heads and fifty tails. Confusing ‘normal’ with ‘desirable’ makes it all the worse.”

Huff warns against taking “normal” too seriously. He tried to make clear through his discussions on averages that “average” does not reflect any individual person. It is calculated from a population, which includes individual variances. As a result, some are bound to be “above” average and some “below.” This is only a part of statistics, not a value judgment.

“But comparisons between figures with small differences are meaningless. You must always keep that plus-or-minus in mind, even (or especially) when it is not stated.”

The “plus-or-minus” Huff references in this quote is the degree of error in a statistic. If the error is left off the published statistic, it throws any comparisons into question, especially if the difference isn’t significant. Without it, there is no way to know if the difference is substantial or if the numbers overlap. Even a tiny difference can appear more significant than it is in reality.

“There is terror in numbers. Humpty Dumpty’s confidence in telling Alice that he was master of the words he used would not be extended by many people to numbers. Perhaps we suffer from a trauma induced by grade-school arithmetic.”

This quote serves as the introduction to the chapter on graphs. It uses Huff’s signature quips to reinforce one of his main points. Some manipulative statistics are chosen to be represented in an image because of accessibility to the general public. The less one must think about the actual numbers, which he describes as a “terror,” the better.

“It is a subtler equivalent of editing ‘National income rose ten percent’ into ‘…climbed a whopping ten percent.’ It is vastly more effective, however, because it contains no adjectives or adverbs to spoil the illusion of objectivity. There’s nothing anyone can pin on you.”

This quote highlights the usefulness of using visuals to support manipulative statistics over text. While the general public views data as subjective, people tend to have some understanding of how words can be used to lie. Because a graph is seen as objective, the tricks it uses to hide its manipulations go unnoticed more easily.

“The eye saw two furnaces, one of them close to three times as big as the other. To say ‘almost one and one-half’ and to be heard as ‘three’—that’s what the one-dimensional picture can accomplish.”

This quote highlights Huff’s points regarding the issues of pictorial graphs. While the second is “almost one and one-half” times larger than the first furnace, because of changes in width to accommodate proportionality it appears to the viewer that the difference is even larger. This is the issue with manipulative imagery. The image tends to be what the audience sees, while the actual numbers are ignored, perpetuating an incorrect assumption.

“Showing this increase by drawing two cows, one three times the height of the other, will exaggerate the impression in the manner we have been discussing. But the effect on the hasty scanner of the page may be even stranger: He may easily come away with the idea that cows are bigger now than they used to be.”

This is another example of the issue with pictorial graphs not set up in a way to be easily intuitive and of The Importance of Critical Thinking when encountering statistical information. It also points out an issue that must be considered when presenting data: Be aware that the viewer may come away with an entirely different idea than the one intended. The manipulation in the example comes from the proportion issue but has the unintended consequence of also suggesting incorrect information.

“You have achieved something remarkable by careful use of a semiattached figure. The worse things get, the better your poll makes them look.”

This line is the closing of Huff’s example of how data collected from racist individuals can be twisted to imply a lack of racism. The higher their number in the poll, the better things looked for the Black population’s job prospects in the statistic. Huff notes that the result did not reflect the situation’s reality, nor did it reflect conditions getting better. Rather, it was an effect of sample bias.

“There are often many ways of expressing any figure […] The method is to choose the one that sounds best for the purpose at hand and trust that few who read it will recognize how imperfectly it reflects the situation.”

Huff states the basic way to achieve any deceptive presentation of statistics. It relies on the perceived objectivity of the data, which hides the subjectivity of the manipulation happening to the data. Underscoring this quote is the implication that if more people recognized the statistic’s faulty nature, the potency of its manipulations would diminish.

“Not all semiattached figures are products of intentional deception. Many statistics, including medical ones that are pretty important to everybody, are distorted by inconsistent reporting at the source. There are startlingly contradictory figures on such delicate matters as abortions, illegitimate births, and syphilis.”

Huff steps away from his critiques of the manipulative nature of many semiattached figures to make the point that sometimes it results from data issues rather than conscious choice. Data can go unreported when compared to reality due to a variety of factors. If reporting on something might lead to a person being ostracized or facing legal consequences, that thing tends to be woefully underreported.

“The point is that when there are many reasonable explanations you are hardly entitled to pick one that suits your taste and insist on it. But many people do.”

Huff points out the issues with choosing an explanation when the question of causation appears among correlated examples. Each possible explanation may be just as likely as the next to be the cause, which is a factor of reality. However, individuals with an agenda tend to latch onto an explanation supporting their position, even if they have no evidence for it.

“When you find somebody—usually an interested party—making a fuss about a correlation, look first of all to see if it is not one of this type, produced by the stream of events, the trend of the times.”

The context here comes from an example when women were perceived as walking a certain way because of their age, but this was actually due to having grown up in a time when walking that way was fashionable. Huff uses this as part of his broader discussion of sources of causation being ignored. People can never be divorced from the context in which they exist.

“Permitting statistical treatment and the hypnotic presence of numbers and decimal points to befog causal relationships is little better than superstition.”

This quote is another of Huff’s warnings to remain critical of statistical data. If the readers do not, the people creating manipulative statistics go unchallenged. In this case, they are muddying cause-and-effect relationships. Unlike what is discussed in some of the other chapters in the book, manipulation that confuses correlation with causation does not have the same basis in legitimate data.

“As long as the errors remain one-sided, it is not easy to attribute them to bungling or accident.”

Huff implies in this quote that while some deceptive statistics result from genuine errors in the process, many result from intentional manipulations. If the faulty statistic was always the result of error, one would expect it to have variations in who or what it supports. Instead, the results tend to favor the interests of the people creating them.

“The fact is that, despite its mathematical base, statistics is as much an art as it is a science. A great many manipulations and even distortions are possible within the bounds of propriety. Often the statistician must choose among methods, a subjective process, and find the one that he will use to represent the facts.”

Huff presents support for his argument that statistical analysis is subjective and not infallible. He spent the prior chapters explaining the ways bias and choice encroach on statistical results. Here, he synthesizes the argument that statistics, good or bad, rely on the subjective decision-making of the statistician creating them.

“But arbitrarily rejecting statistical methods makes no sense either. That is like refusing to read because writers sometimes use words to hide facts and relationships rather than to reveal them.”

Huff attempts to prevent a potential problem from arising from his book. The previous chapters provided ways of recognizing deceptive statistics, but they could also build a distrust in the reader for all statistics, even those based on good data and reporting. Huff underlines the importance of not rejecting statistics altogether by pointing out that statistical analysis is a tool. The usage of statistics—not the field of statistics itself—should be critically viewed.

“Not all the statistical information that you may come upon can be tested with the sureness of chemical analysis or of what goes on in an assayer’s laboratory. But you can prod the stuff with five simple questions, and by finding the answers avoid learning a remarkable lot that isn’t so.”

Huff makes one of his critical points regarding the purpose of the book in this quote. While his readers cannot test the data for themselves due to a lack of access, training, or other factors, that does not mean they need to take statistics at face value. They can use the questions he provides in Chapter 10 to protect themselves. The point is not to understand the minutiae of the statistic in question but to understand when they are being manipulated.