The Scientific Method Is Not What It is Cracked Up To Be

Testing the Limits of the Scientific Method

In his article, “The Truth Wears Off” (December 5, 2010), Jonah Lehrer makes the troubling point that the effectiveness and even the validity of the scientific method has been increasingly called into question in recent years as researchers encounter unexpected and frequently inexplicable results from their studies. Not only have researchers been experiencing problems in replicating their findings, the strength of their initial findings has been declining as the research process progresses. In addition, Lehrer also claims that some previously proven theories that have widespread influence are being disproven despite the use of scientifically accepted research methods.

Although some of these outcomes can be attributed to the operation of the regression to the mean statistical analyses that produce a more robust picture, these trends have been taking place in a wide range of fields of study and researchers are concerned that their reliance on the tried-and-true scientific method is no longer as justifiable as they once believed. Likewise, as noted above, Lehrer points out that researchers are struggling to why they produce significant results at the outset of an experiment but the strength of these results continues to decline over time. In other words, some researchers have found strong support for their guiding hypotheses during the initial stages of their study, but these findings tend to diminish in strength when the experiment is repeated even though all protocols remained the same.

While Lehrer presents plenty of evidence in support of his claims, his analysis suffers from some of the very same constraints that he cites with respect to other research. For example, Lehrer points out that selective reporting and publication bias may account for the high percentage of studies that actually find their way into juried publications since everyone prefers positive over negative results, especially researchers with a financial interest in the outcome of their experiments.

Although Lehrer makes a good case in support of his thesis that the scientific method may be more limited than many researchers would like to believe, he does concede that researchers work in the real world where randomness prevails. Furthermore, ongoing efforts to address the fundamental constraints that have been experienced with the scientific metho can only go so far in addressing randomness in research. In this regard, Lehrer admits that, “Although such reforms would mitigate the dangers of publication bias and selective reporting, they still wouldn’t erase the decline effect. This is largely because scientific research will always be shadowed by a force that can’t be curbed, only contained: sheer randomness.”

Even though Lehrer does not attempt to explain the decline effect with randomness, he does emphasize that researcher bias and subjectivity can creep into the most well-designed and rigorous scientific experiment and there is always a caveat emptor quality to accepting the results of any study. For instance, Lehrer points out that, “We like to pretend that our experiments define the truth for us. But that’s often not the case. … When the experiments are done, we still have to choose what to believe.” In other words, it is not necessarily the scientific method that is limited but rather the manner in which the results of scientific research are achieved and published that is at fault. These findings mean that while international relations research that desires to be empirical must rely on empirical evidence, it must also take into account the unstated potential limitations and constraints that were involved in order to make an informed decision concerning its relevance and legitimacy.