Critical Thinking About Statistics: Evaluating Data Claims

Introduction: How Statistics Can Mislead or Inform

Statistics is an approach to research in which data collected from a sample is used to draw conclusions about a population. To this end, statistics provides a crucial basis for decision-making, as well as a means of understanding population complexities. However, despite their usefulness, statistics have one major flaw that cannot be overlooked: since they are created by human beings, and the counting and analyses therein are based solely on human definitions, statistics can neither be regarded as magical nor taken as always true. Statistics can mislead a reader as much as they can educate, and can deceive just as much as they can depict the truth. The key to discerning good statistics from bad lies in the evaluation approach chosen. Naivety and cynicism cause people to lose a great deal, but a critical mindset leads a person to effectively evaluate any statistics presented to them and draw comprehensive conclusions.

Three Mindsets: Naive, Cynical, and Critical

One may ask: what, then, constitutes a critical mindset? Before addressing that question directly, it is helpful to briefly discuss the two other approaches to statistical thinking — naivety and cynicism (Best 15).

The Naive Mindset makes the presumption "that statistics are generally accurate, that they mean what they seem to mean" (Best 15). This mindset fully trusts any statistics presented to it, and neither questions why the figures are the way they are, nor wonders how the producers' interests could have influenced them (Best 16).

The Cynical Mindset views statistics as nothing but manipulative efforts. It does not trust numbers, nor the people who produce them (Best 16). Cynical mindsets are always "suspicious of statistics; they are convinced that numbers are probably flawed, and that those flaws are probably intentional" — and hence statistics simply cannot be used to prove anything (Best 16).

The Critical Mindset is neither hostile nor negative, because it understands that even the best statistics cannot be perfect (Best 17). All a critical mindset does is "evaluate numbers to distinguish between good statistics and bad statistics" (Best 17). Unlike the naive and cynical mindsets, this approach is statistically literate (Schield 1).

A critical mindset evaluates presented statistics on the basis of: the study type used; the sample used for the study; the measurements derived; the graphs generated from the gathered data; the claims and probability statements made from the data; the study's limitations; and the quantity of information provided to the end consumer (Ganio 6).

Having recognized that no single statistic is perfect, a critical thinker is often more concerned with "whether a particular statistic's flaws are severe enough to damage its usefulness" (Best 17). Relevant questions include: How broad is the definition — does it encompass an excess of false positives, or leave out significant false negatives? How significant would the statistical change be if the definition were altered? How does the sample or measurement choice influence the statistic? How would the statistic change if a different sample were used? Has the statistic been correctly interpreted, or has it become a mutant statistic? How appropriate are the comparisons made?

Fundamental Distinctions in Critical Statistical Thinking

Critical thinking begins with making three fundamental distinctions about presented statistics: causation vs. association, population vs. sample, and a test's quality vs. its predictive power (Schield 2).

Distinguishing causative statements from associative statements is a crucial step in critical thinking because it conveys how strongly the evidence given supports the statistic or claim (Schield 2). A magazine article might claim, for instance, that violent TV programs have contributed to rising rates of antisocial behavior and violent crime. Such an article might assert that: (1) the more violent TV entertainment a teenager is exposed to, the higher their chances of developing antisocial behavior; (2) there is a positive association between TV violence and antisocial behavior; and (3) if teenagers watched less violent entertainment, "they would exhibit less antisocial behavior" (Schield 2).

At a glance, one would conclude that claim 3 is true as long as claim 2 is true. Critical thinking would, however, lead one to realize that the difference between the two is also the difference between causation and association. Therefore, even though claim 2 provides evidence for claim 3, its truth "is not sufficient to prove the truth of #3" (Schield 3).

Distinguishing between a population parameter and a sample parameter is an important component of critical thinking. Failure to make this distinction gives rise to the presumption that "a statistic obtained from a sample is actually a property of the entire population" (Schield 2). Consider the claim that 70% of New York adults support a reduction of violent TV programs. Unless this result was obtained from a census, the statistic comes from a sample survey, and hence the phrase "New York adults" does not represent the adult population of New York in its entirety (Schield 2).

The question that arises concerns the sample's degree of representativeness: Was the sample collected from a neighborhood mostly inhabited by young adults? Was the sample reflective of New York's population in terms of size and composition?

Another fundamental component of critical thinking involves distinguishing between a test's quality and its predictive power (Schield 3). Whereas quality "is measured on subjects whose disease status is known prior to the test, the predictive power of a test is measured on subjects whose disease status is unknown prior to the test" (Schield 3). A lack of statistical literacy leads readers to presume that a test's quality automatically reflects its predictive power, which is not always the case.

Conclusion

Statistics can never be perfectly accurate because they are created by humans and are therefore prone to error. The error's degree of significance is the real issue. This degree of significance can be established through critical thinking, in which the reader establishes the truth and representativeness of a statistic by examining causative and associative relationships, population and sample parameters, and the quality and predictive power of the test used. This process of critical thinking helps a reader discern good statistics from bad ones and, consequently, to rightly decide whether a given statistic qualifies as a reliable guide to decision-making.

Best, Joel. "Critical Thinking about Statistics." Comcast. Web. 7 May 2014.

Ganio, Lisa. "Teaching Critical Thinking (in Statistics) for Natural Resource Education." 8th Biennial Conference on University Education in Natural Resources. 2010. Web. 7 May 2014.

Schield, Milo. "Statistical Literacy: Thinking Critically About Statistics." The Inaugural Issue of the Journal of Significance. 1999. Web. 7 May 2014.