Experimental Research Methods in Business Experimental Research Methods

The author provides a survey of the literature illustrating applied experimental research methods in cross-sections of business and organization types. The advantages and disadvantages of the experimental research methods are discussed for each of the examples provided which run the gamut from depression-era agricultural economics to research conducted for the National Science Institute. While the article focuses on business research methods, the range of examples from multiple disciplines serves to demonstrate the adaptability of various methods to distinct contexts, the importance of thoughtfully developed research questions, and perceptions in the field regarding scientific rigor. The article is intended to guide students in their exploration of the breadth and depth of experimental research methods and to convey a sense of the challenges of applied scientific inquiry.

Introduction

The study of business topics has not always been inherently scientific. Certainly the work of Max Weber and Frederick Winslow Taylor brought rigor to the science of management. However, as with all emerging disciplines, business has occasionally gone down the quasi-scientific path in response to astonishing social acclaim. The use of Meyers-Briggs personality tests by organizational development and human resource specialists is one example. Adoption of the open plan office space is another. Researchers in the areas of attention and cognitive processing have demonstrated -- through empirical research (Treasure, 2011) -- that the highly distracting environment of the open plan shared office space, even when modified by the ubiquitous cubicle -- does not result in high rates of innovative thinking or productivity, and it is bad for our health (Oommen, et al., 2008). In fact, the strain of having to think and work within a cacophony that resembles the Tower of Babble is altogether exhausting, eroding inordinate amounts of human energy and reducing productivity by a whopping 33% (Demarco, et al.). Yet, the cubicle reigns.

Over the past half century, however, business-related research has become as robust and evidence-based as many other disciplines. This paper provides a survey -- as in survey course not survey questionnaire -- of the literature illustrating applied experimental research methods in cross-sections of business and organization types. It is neither an exhaustive nor representative survey. Rather, the articles reviewed provide insight into the problem sets and challenges of conducting experimental or quasi-experimental research in vivo or in simulated environments.

Discussion

Experimental Research Methods

Cooper and Schindler (2011) begin the PowerPoint presentation in the supplementary materials with this quote from Richard Buckminster Fuller, the renowned engineer and architect of geodesic fame: "There is no such thing as a failed experiment, only experiments with unexpected outcomes." In this survey of applied experimental research methods, the one researcher who is most likely to enjoy aligning himself with Fuller is Alistair Campbell, a family therapist and a student of research methods. Campbell's research work, which is scientifically robust, is conducted in the field of social services, specifically family therapy. I have included him first because the attitude he chooses to convey regarding experimental research is at once jaded, irreverent, and cavalier. I felt it best to address the skepticism that many people today harbor about scientific research. We are not so far from the time to not heed the remarks published in the North American Review of "Chapters from My Autobiography" by Mark Twain, in which he wrote, "Figures often beguile me -- particularly when I have the arranging of them myself, in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics" (Twain, 1906).

Family therapy research. The approach that Campbell (2004) takes to tackling the "research-reporting monoculture" in which we live is fresh, cogent, and utilitarian. In this article, Campbell covers the research forms of randomized control trials, cohort studies, cross-sectional research, case studies, case control studies, systematic reviews, and meta-analyses. The origins and inclinations of randomized control trials (RCT) are explained by Campbell in a manner that helps to demystify the deification of experimental control. Though Campbell may be accused of taking a simplistic...

Campbell says precisely that, "the biggest weakness for RCTs is that what they make up for by controlling as many factors as possible (internal validity) they lose in being actually applicable in the real world (external reliability)" (2004, p. 165). Case studies are defended by Campbell (2004) as the place where all scientific inquiry begins, and Campbell reminds the researcher that case studies do "represent the perfect vehicle for the articulation of tacit knowledge" which is the learning that one acquires through their praxis and their experience (2004, p. 166). Tacit knowledge is valuable and is often the catalyst for our "aha" reactions, which -- not coincidently -- often evolve into research questions or methodology that results in answers to research questions. The difference between tacit knowledge and explicit knowledge is that one "can be codified and transmitted in formal language" and shared with a community of researchers, in the time-tested way that a literature is built (2004, p. 167).
A refreshingly candid review of meta-analyses is provided in this article. Campbell tackles the limitations of the technique -- which are many and massive -- head on. His point is well-taken that meta-analyses are quite useful as places to begin research on a line of inquiry since researchers who use meta-analysis are quite driven to identify studies that meet their methodological requirements in order to be included in their meta-analysis (Campbell, 2004, p. 167). As a result, their research is practically exhaustive.

Research paradigms. In 1962, Thomas Kuhn came up with construct of a paradigm and it was broadly and readily accepted by people across many different disciplines. Perhaps it was the general applicability of the term that caused people to so readily adopt it. It was 15 years later that Kuhn was pinned down sufficiently to clarify just what it was -- exactly -- that he meant by the term paradigm. A paradigm, Kuhn explained, was just a general concept that stood for the phenomenon that occurs when a "group of researchers having a common education and an agreement on 'exemplars' of high quality research or thinking" (Kuhn, 1977). Johnson, et al., () in the subsection below entitled Toward a definition of mixed methods research, describe their version of a paradigm -- a research paradigm -- as "a set of beliefs, values, and assumptions that a community of researchers has in common regarding the nature and conduct of research" (Johnson, 2007).

Johnson, et al. (2007) states that the beliefs that act as scaffolding to a research paradigm include "ontological beliefs, epistemological beliefs, axiological beliefs, aesthetic beliefs, and methodological beliefs" (p. 24). This terminology we recognize as stemming from philosophy, but this fact does not prevent us from thinking that the word beliefs does not seem like a scientific term at all. There is, however, an entire branch of simulation or machine learning research that is based on algorithms and probabilistic structural equation modeling. The simulation research is also called Bayesian Beliefs Modeling, so named because an integral aspect of structural equation modeling is the expert knowledge that is brought to bear on the machine learning. The scientist's beliefs, we are then led to understand, truly matter in experimental research that employs algorithms. A researcher's thinking skills are necessary for automatic simulation research, just as they are for the process of identifying the best set of research questions to address a research problem, and for isolating the best research approaches to apply toward answering those research questions.

From Johnson, et al. (2007), we understand that research paradigm and research culture are roughly equivalent, the synonym of which, the authors argue, is methodological paradigm. Just as Kuhn meant for the term paradigm to be a general, adaptable term -- before he was pinned down by other scientists -- Johnson, et al. (2007) meant for the term to mature based on what it means to a group of researchers to conduct research and how they go about undertaking that research. In their discussion of mixed methods research, Johnson, et al., (2007) apply the terms research paradigms or methodological paradigms as organizers to separate what they perceive to be distinct types of research: Qualitative research, quantitative research, and mixed research.

Toward a definition of mixed methods research. Johnson, et al. (2007) explored definitions of mixed methods, provided a definition of mixed methods, and placed mixed methods approach in the research. Further, the authors argue that there are three research paradigms: Quantitative research, qualitative research, and mixed methods research. Expanding on the single general definition of mixed methods research, Johnson, et al. (2007) provide 19 distinct definitions of mixed methods research which they summarize in discussion and through a content analysis. Finally, the authors define qualitative dominant mixed methods research and quantitative dominant mixed methods research (Johnson, et al., 2007). The general definition of mixed methods research for which the authors found agreement, along with…