Experimental Research Methods in Business and Organizations

Introduction

The study of business topics has not always been inherently scientific. Certainly the work of Max Weber and Frederick Winslow Taylor brought rigor to the science of management. However, as with all emerging disciplines, business has occasionally gone down the quasi-scientific path in response to astonishing social acclaim. The use of Myers-Briggs personality tests by organizational development and human resource specialists is one example. Adoption of the open-plan office space is another. Researchers in the areas of attention and cognitive processing have demonstrated — through empirical research (Treasure, 2011) — that the highly distracting environment of the open-plan shared office space, even when modified by the ubiquitous cubicle, does not result in high rates of innovative thinking or productivity, and it is bad for our health (Oommen et al., 2008). In fact, the strain of having to think and work within a cacophony resembling the Tower of Babel is altogether exhausting, eroding inordinate amounts of human energy and reducing productivity by as much as 33% (Demarco et al., 2008). Yet, the cubicle reigns.

Over the past half century, however, business-related research has become as robust and evidence-based as many other disciplines. This paper provides a survey — as in a survey course, not a survey questionnaire — of the literature illustrating applied experimental research methods across a cross-section of business and organization types. It is neither an exhaustive nor a representative survey. Rather, the articles reviewed provide insight into the problem sets and challenges of conducting experimental or quasi-experimental research in vivo or in simulated environments.

Experimental Research Methods Overview

Cooper and Schindler (2011) open with this quote from Richard Buckminster Fuller, the renowned engineer and architect of geodesic fame: "There is no such thing as a failed experiment, only experiments with unexpected outcomes." In this survey of applied experimental research methods, the one researcher most likely to align himself with Fuller is Alistair Campbell, a family therapist and student of research methods. Campbell's research work, which is scientifically robust, is conducted in the field of social services — specifically family therapy. He is introduced first because the attitude he conveys regarding experimental research is at once jaded, irreverent, and candid. It is worth addressing the skepticism that many people today harbor about scientific research, not far removed from Mark Twain's remark published in the North American Review: "Figures often beguile me — particularly when I have the arranging of them myself, in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics'" (Twain, 1906).

The approach that Campbell (2004) takes to tackling the "research-reporting monoculture" in which we live is fresh, cogent, and utilitarian. In this article, Campbell covers the research forms of randomized control trials, cohort studies, cross-sectional research, case studies, case control studies, systematic reviews, and meta-analyses. The origins and inclinations of randomized control trials (RCT) are explained by Campbell in a manner that helps to demystify the deification of experimental control. Though Campbell may be accused of taking a simplistic approach, his criticism of randomized control trials is pointed: "the biggest weakness for RCTs is that what they make up for by controlling as many factors as possible (internal validity) they lose in being actually applicable in the real world (external reliability)" (2004, p. 165).

Case studies are defended by Campbell (2004) as the place where all scientific inquiry begins. Campbell reminds the researcher that case studies "represent the perfect vehicle for the articulation of tacit knowledge" — the learning one acquires through praxis and experience (2004, p. 166). Tacit knowledge is valuable and is often the catalyst for "aha" reactions which, not coincidentally, often evolve into research questions or methodology that results in answers to research questions. The difference between tacit knowledge and explicit knowledge is that the latter "can be codified and transmitted in formal language" and shared with a community of researchers, in the time-tested way that a literature is built (2004, p. 167).

A refreshingly candid review of meta-analyses is also provided. Campbell tackles the limitations of the technique — which are many and substantial — head on. His point is well-taken that meta-analyses are quite useful as starting points for a line of inquiry, since researchers who use meta-analysis are driven to identify studies that meet their methodological requirements for inclusion. As a result, their research tends to be practically exhaustive (Campbell, 2004, p. 167).

In 1962, Thomas Kuhn introduced the construct of a paradigm, which was broadly and readily accepted across many disciplines. Perhaps it was the general applicability of the term that made it so easy to adopt. It was 15 years later that Kuhn was pressed sufficiently to clarify what he meant by the term. A paradigm, Kuhn explained, was a general concept representing the phenomenon that occurs when a "group of researchers having a common education and an agreement on 'exemplars' of high quality research or thinking" share a common orientation (Kuhn, 1977). Johnson et al. (2007) describe a research paradigm as "a set of beliefs, values, and assumptions that a community of researchers has in common regarding the nature and conduct of research" (p. 24).

Johnson et al. (2007) state that the beliefs acting as scaffolding to a research paradigm include "ontological beliefs, epistemological beliefs, axiological beliefs, aesthetic beliefs, and methodological beliefs" (p. 24). This terminology stems from philosophy, yet the word beliefs does not seem like a strictly scientific term. There is, however, an entire branch of simulation and machine learning research based on algorithms and probabilistic structural equation modeling — also called Bayesian Beliefs Modeling — in which expert knowledge is integral to the machine learning process. The scientist's beliefs, therefore, genuinely matter in experimental research employing algorithms. A researcher's thinking skills are necessary for automated simulation research, just as they are for identifying the best research questions and isolating the best research approaches to answer those questions.

From Johnson et al. (2007), we understand that research paradigm and research culture are roughly equivalent — the synonym of which, the authors argue, is methodological paradigm. Just as Kuhn intended the term to be general and adaptable, Johnson et al. (2007) meant for the term to mature based on what it means to a group of researchers to conduct research and how they undertake that research. In their discussion of mixed methods research, Johnson et al. (2007) apply the terms research paradigm or methodological paradigm as organizers to distinguish three types of research: qualitative research, quantitative research, and mixed research.

Research Paradigms and Mixed Methods

Johnson et al. (2007) explored and provided a definition of mixed methods, placing the mixed methods approach within the broader research landscape. The authors argue that there are three research paradigms: quantitative research, qualitative research, and mixed methods research. Expanding on this, Johnson et al. (2007) provide 19 distinct definitions of mixed methods research, which they summarize through discussion and content analysis. They further define qualitative-dominant and quantitative-dominant mixed methods research. The general definition on which the authors and a deep bench of research methods experts found agreement is that mixed methods research occurs when "a researcher or team of researchers combines elements of qualitative and quantitative research approaches for the broad purposes of breadth and depth of understanding and corroboration" (2007, p. 123). The research approaches referred to in this definition include data collection and analysis, inference techniques, qualitative perspectives, and quantitative perspectives. When placing mixed methods as a type of research, the authors suggest that research employing mixed methods "would involve mixing within a single study; a mixed method program would involve mixing within a program of research and the mixing might occur across a closely related set of studies" (Johnson et al., 2007, p. 123).

Greene (2006) further parsed research methodologies into four domains: (1) "philosophical assumptions and stances;" (2) "inquiry logics;" (3) "guidelines for practice;" and (4) "sociopolitical commitments," the latter being most appropriate for social inquiry methods, though still applicable with regard to the larger context in which research is conducted. Greene (2006) argues that the term methodology is essentially the inquiry logics that guide topic selection, the development of research questions, the standards for quality and rigor, the study purposes, and the writing forms that guide the researcher's perspective. The domain of guidelines for practice includes any tools, procedures, and techniques used to conduct the research — the operational stage that tells a researcher how to structure and carry out the work.

Scientists and researchers would like to believe they are above the common pressures of the larger social context in which they work — but that is decidedly not the case. There is, within and surrounding every research endeavor of any substantive scope, a socio-political and economic context characterized by power relations, institutional commitments, and individual and institutional interests. These pressures can determine what types of inquiry will be employed, in effect vetoing the preferences of research departments and methodological experts. In such cases, it is often the anticipated reception of the intended audience that becomes the deciding factor in how research design is structured. This is precisely one of the tenets that Suter (2005) argues in his work on multiple methods.

Research in science and mathematics education often employs multiple methods of experimental study (Suter, 2005). When single methods are used, they are most often a version of experimental design or case study. That said, multiple methods prevail in educational practice research, and these designs frequently attempt too great a reach. Suter (2005) argues that too little attention is paid to the development of focused research questions that pursue a problem over time through sustained effort. At issue is that funding sources frequently require studies of a certain magnitude addressing specific governmental priorities. Funders can reasonably be faulted for encouraging research that overreaches and results in inconclusive outcomes. This concern is less of a problem in business research, though research conducted under the auspices of a university-affiliated business school can face similar limitations. A primary goal of research design is therefore to focus on "research questions that match the question with a method" (Suter, 2005, p. 180), after which the selection of appropriate research methods will reasonably follow.

Just as the selection of generative research questions must fit with the chosen research methods, the special area of decision aid research must calculate how best to use research to solve problems and support decisions.

Experimental decision aid research has applicability to a wide range of disciplines. A primary advantage of decision aid research is its usefulness as a support to understanding the outcomes of experimental research. A decision aid is precisely what it sounds like — a tool for facilitating decision-making or problem-solving that provides embedded or inherent information relevant to the decision or problem. Decision aids can consist of simple paper-based supports or computerized algorithms that transform input into useful output in the form of data (Wheeler & Murthy, 2011). A substantive issue with decision aids is the effect that use of the experimental decision aid has on the user — agreement about how to solve this problem has not yet solidified (Wheeler & Murthy, 2011).

Conclusion

Sorensen, F., Mattsson, J., & Sundbo, J. (2010). Experimental methods in innovation research. Research Policy, 39, 313–322.

Suter, L. E. (2005, October). Multiple methods: Research methods in education projects at NSF. International Journal of Research & Method in Education, 28(2), 171–181.

Treasure, J. (2009, October). TED: The 4 ways sound affects us. TED Talks.

Wheeler, P., & Murthy, U. (2011). Experimental methods in decision aid research. International Journal of Accounting Information Systems, 12, 161–167.