Bad statistics: causes, implications, and detection methods

¶ … video, three different forms of bad statistics were mentioned. The first was bad sampling, the second was bad use of measures of central tendency and the third was leading questions. Leading questions are a form of sampling error, in which survey respondents are led to specific conclusions. Their responses are calculated in the results, the context in which those responses arose was not noted. The context, however, played a significant role in framing the responses. There are many ways to lead people with questions. I think a lot of times the leading questions are emotional in nature, to put somebody in a specific emotional state that frames their response. The question about Santa Claus in the video was a good example of this -- "Do you really still believe in Santa Claus?" But this happens with all types of surveys. "A lady found a finger in her taco last week at Taco Bell. Did you hear about this?" "No, that's shocking, sir." "If you had a choice for dinner tonight, would you prefer pizza or Mexican?" The entire interview could be about food safety until the last question, planting the idea that Mexican food is unsafe, and then the question highlighted in the public communications is that pizza is far more popular than Mexican food.

The second error was the bad sampling error. In order for a sample to be extrapolated to the population at large, it has to be representative of the population. The example used in the video of the housewives was a good example. The sample provided accurate information about housewives within the sample community, but was erroneously extrapolated to the broader population, which should not have happened. You can see how this works. Let's say you wanted to find out what the best burrito in Los Angeles is. You conduct your survey in Malibu. Well, good luck getting a useful answer because people up there probably don't get down to the parts of town where one might find the actual best burrito in LA. The sample is not representative of what you are trying to find out.

The third error was the wrong use of central tendency. This is where the Michael Jordan fallacy was described. The statistic used was correct, but the context in which that statistic was misleading, because there was an outlier.

The Michael Jordan fallacy was simply taking a measure of central tendency out of context. In this case, it was the median salary of UNC geography grads, which was skewed by Michael Jordan's NBA salary. The fallacy could also apply to other misuses of central tendency. In some surveys, for example, the mode could be 0, but this might be quite out of line with the mean and median; it just so happens that 10% of people did not eat tacos last year, so the most common answer to "how many tacos did you eat last year?" was zero. The mean and median were both much higher, indicating that the other 90% of people, those who do eat tacos, consume a lot more than zero tacos. But because the range is dispersed from 10-200, the 10% of non-taco eaters provided the mode in this hypothetical example.

These errors, unfortunately, are common. It is important to understand that such errors are impossible to detect when they are presented in news media or advertisements. In order to evaluate the quality of the research reported, you need to look at the source material. That can either be completely unavailable or only available at a high cost. Also, most people do not have the knowledge of statistics that is needed in order to truly understand the information that is being presented. Hence the quote at the beginning of the video explaining that it is important to understand statistics, and why it is deficient that most people know nothing of the subject. It really affects their abilities to think critically.

I would say that the most misrepresented media these days comes from politics. This is something I see a lot, but certainly during election campaigns or on any number of the propaganda shows. Soundbites are routinely taken entirely out of context to portray a political opponent in a bad light. Attack ads will take a five word clip from six years ago and use that to frame an argument today. This is completely ridiculous and no intelligent person would fall for this, but there are a lot of emotional, unintelligence and uninformed voters out there. Instead of researching a candidate's actual track record, they are more likely to be swayed by a soundbite framed not by the actual situation in which it occurred but in the situation in which the person running the ad or talk show wants you to hear it.