This paper examines the foundations and limitations of quantitative research methodology, contrasting it with qualitative approaches and exploring the complex realities of statistical analysis in the social sciences. It discusses null hypothesis testing, the importance of statistical power, and the challenges of isolating variables in studies involving human subjects. The paper also addresses how improper use of statistical conventions—including handling of outliers and missing data—can distort results, and how deliberate data manipulation (falsification and fabrication) undermines research integrity. The paper concludes that numerical data is not inherently objective and that rigorous methodology remains essential to valid scientific inquiry.
Quantitative research produces numerical data that can be analyzed in a statistical fashion. Quantitative research frequently — although not exclusively — deploys the scientific method, whereby a hypothesis is tested in a controlled fashion. One group, the experimental group, is subjected to an intervention known as the independent variable, while another, otherwise similar group is designated the control group and not subjected to that variable. The dependent variable is the change, or lack of change, that results from the intervention, and the results prove or disprove the initial hypothesis. Quantitative research can also take the form of a survey or other instrument designed to collect raw data about a particular population.
In contrast, qualitative research is designed to explore the evolution of a particular phenomenon in narrative form. Responses from test subjects may be coded and subjected to data analysis, but ultimately the goal of this type of research is to record the particular experience of a population in a holistic fashion, not to test a theory within limited parameters. This contrast means that qualitative research is often seen as subjective, versus the supposedly superior objective claims of quantitative methodologies. However, there are many persistent problems with quantitative research that complicate this schematic notion. It has been observed that "poor statistics" make for "poor science," but this is true of all disciplines — indeed, in the social sciences, where variables are more difficult to isolate within populations, rigorous statistical methodology to eliminate error is even more significant (Gardenier & Resnik, 2002, p. 70). Also, in quantitative research, using effective statistical testing is vital regardless of the experiment, given the ethical implications of having human subjects take the risk of participating in a study with questionable utility and value (Gardenier & Resnik, 2002, p. 66).
In an experiment involving statistical analysis of a population, the formal null hypothesis is tested — that is, the theory that nothing will happen. The null hypothesis is actually a statement contrary to what researchers want to prove. In general, it is assumed that false rejection of the null hypothesis is less damaging than false acceptance; in other words, overestimating the potential impact of a variable is considered less troubling than failing to recognize its impact (Baroudi & Orlikowski, 1989, p. 88). As Cortina (2002) explains, "the embedded null approach involves embedding a hypothesis of no effect within an interaction framework. The framework is then used to show that, under certain conditions, the manipulation/predictor variable in question does produce an effect or relationship, while under the conditions of primary interest, the effect or relationship does not appear" (p. 342).
The cautious approach to tracking change makes sense given that the selection of the test population may be imperfect and contain too many outliers. That is why a statistically significant alteration must be in evidence, not simply any change at all. "The reason that we avoid concluding a lack of effect from studies that show minimal or non-significant results is that there are many alternative explanations for this finding" (Cortina, 2002, p. 343). Particularly in the social sciences, it can be difficult to isolate all variables that could potentially affect the test population, given that human beings live in the real world and are the sum of lived experiences that cannot necessarily be controlled by the test's creator. Only "if one can rule out some of these alternative explanations" as to why a phenomenon occurred, while "offering sound theoretical justification for a hypothesis of no effect, then there is no reason to avoid no effect conclusions" (Cortina, 2002, p. 343).
Beyond the statistical analysis itself, the ways in which groups are allocated and designed can also yield a powerful impact upon results, underscoring the importance of test construction. This is yet another reason we cannot assume that data is "objective" simply because it is quantitative in nature. For example, when constructing an experiment, "an extreme groups design (e.g., assigning participants to high or low conditions) maximizes the variances of the components of the product term; it also results in much more power with respect to the interaction effect than would the corresponding observational design" (Cortina, 2002, p. 343). Conversely, conducting an experiment "in the field" is likely to yield a less statistically significant impact because of the inability to control the extremity of the variables.
A study of the statistical power of research in the social sciences revealed that only 40% of all MIS studies had adequate statistical power to ensure that the null hypothesis would be rejected correctly at all times (Baroudi & Orlikowski, 1989, p. 87). Significance criteria, sample size estimates, and effect size can all influence statistical power, and when dealing with human subjects, many additional variables can further affect it (Baroudi & Orlikowski, 1989, p. 87).
"Outliers and missing data distort findings"
"Deliberate manipulation and fabrication of data"
All of this does not mean that we should abandon statistical research. However, it is important to recognize that the appearance of numbers is no guarantee that the research in question is of better quality, more accurate, or more generalizable than a small qualitative study. The cliché that "numbers don't lie" is untrue, given that numbers are always accumulated by human intelligence and via human-created research designs.
You’re 69% through this paper. Sign up to read the remaining 2 sections.
Sign Up Now — Instant Access Already a member? Log inAlways verify citation format against your institution’s current style guide requirements.