Experimental Research Methods in Business Experimental Research

Experimental Research Methods in Business

Experimental Research Methods

The author provides a survey of the literature illustrating applied experimental research methods in cross-sections of business and organization types. The advantages and disadvantages of the experimental research methods are discussed for each of the examples provided which run the gamut from depression-era agricultural economics to research conducted for the National Science Institute. While the article focuses on business research methods, the range of examples from multiple disciplines serves to demonstrate the adaptability of various methods to distinct contexts, the importance of thoughtfully developed research questions, and perceptions in the field regarding scientific rigor. The article is intended to guide students in their exploration of the breadth and depth of experimental research methods and to convey a sense of the challenges of applied scientific inquiry.

Introduction

The study of business topics has not always been inherently scientific. Certainly the work of Max Weber and Frederick Winslow Taylor brought rigor to the science of management. However, as with all emerging disciplines, business has occasionally gone down the quasi-scientific path in response to astonishing social acclaim. The use of Meyers-Briggs personality tests by organizational development and human resource specialists is one example. Adoption of the open plan office space is another. Researchers in the areas of attention and cognitive processing have demonstrated -- through empirical research (Treasure, 2011) -- that the highly distracting environment of the open plan shared office space, even when modified by the ubiquitous cubicle -- does not result in high rates of innovative thinking or productivity, and it is bad for our health (Oommen, et al., 2008). In fact, the strain of having to think and work within a cacophony that resembles the Tower of Babble is altogether exhausting, eroding inordinate amounts of human energy and reducing productivity by a whopping 33% (Demarco, et al.). Yet, the cubicle reigns.

Over the past half century, however, business-related research has become as robust and evidence-based as many other disciplines. This paper provides a survey -- as in survey course not survey questionnaire -- of the literature illustrating applied experimental research methods in cross-sections of business and organization types. It is neither an exhaustive nor representative survey. Rather, the articles reviewed provide insight into the problem sets and challenges of conducting experimental or quasi-experimental research in vivo or in simulated environments.

Discussion

Experimental Research Methods

Cooper and Schindler (2011) begin the PowerPoint presentation in the supplementary materials with this quote from Richard Buckminster Fuller, the renowned engineer and architect of geodesic fame: "There is no such thing as a failed experiment, only experiments with unexpected outcomes." In this survey of applied experimental research methods, the one researcher who is most likely to enjoy aligning himself with Fuller is Alistair Campbell, a family therapist and a student of research methods. Campbell's research work, which is scientifically robust, is conducted in the field of social services, specifically family therapy. I have included him first because the attitude he chooses to convey regarding experimental research is at once jaded, irreverent, and cavalier. I felt it best to address the skepticism that many people today harbor about scientific research. We are not so far from the time to not heed the remarks published in the North American Review of "Chapters from My Autobiography" by Mark Twain, in which he wrote, "Figures often beguile me -- particularly when I have the arranging of them myself, in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics" (Twain, 1906).

Family therapy research. The approach that Campbell (2004) takes to tackling the "research-reporting monoculture" in which we live is fresh, cogent, and utilitarian. In this article, Campbell covers the research forms of randomized control trials, cohort studies, cross-sectional research, case studies, case control studies, systematic reviews, and meta-analyses. The origins and inclinations of randomized control trials (RCT) are explained by Campbell in a manner that helps to demystify the deification of experimental control. Though Campbell may be accused of taking a simplistic tack, his criticism of randomized control trials is tantamount to the little boy pointing out that the Emperor's new clothes -- well -- aren't quite there. Campbell says precisely that, "the biggest weakness for RCTs is that what they make up for by controlling as many factors as possible (internal validity) they lose in being actually applicable in the real world (external reliability)" (2004, p. 165). Case studies are defended by Campbell (2004) as the place where all scientific inquiry begins, and Campbell reminds the researcher that case studies do "represent the perfect vehicle for the articulation of tacit knowledge" which is the learning that one acquires through their praxis and their experience (2004, p. 166). Tacit knowledge is valuable and is often the catalyst for our "aha" reactions, which -- not coincidently -- often evolve into research questions or methodology that results in answers to research questions. The difference between tacit knowledge and explicit knowledge is that one "can be codified and transmitted in formal language" and shared with a community of researchers, in the time-tested way that a literature is built (2004, p. 167).

A refreshingly candid review of meta-analyses is provided in this article. Campbell tackles the limitations of the technique -- which are many and massive -- head on. His point is well-taken that meta-analyses are quite useful as places to begin research on a line of inquiry since researchers who use meta-analysis are quite driven to identify studies that meet their methodological requirements in order to be included in their meta-analysis (Campbell, 2004, p. 167). As a result, their research is practically exhaustive.

Research paradigms. In 1962, Thomas Kuhn came up with construct of a paradigm and it was broadly and readily accepted by people across many different disciplines. Perhaps it was the general applicability of the term that caused people to so readily adopt it. It was 15 years later that Kuhn was pinned down sufficiently to clarify just what it was -- exactly -- that he meant by the term paradigm. A paradigm, Kuhn explained, was just a general concept that stood for the phenomenon that occurs when a "group of researchers having a common education and an agreement on 'exemplars' of high quality research or thinking" (Kuhn, 1977). Johnson, et al., () in the subsection below entitled Toward a definition of mixed methods research, describe their version of a paradigm -- a research paradigm -- as "a set of beliefs, values, and assumptions that a community of researchers has in common regarding the nature and conduct of research" (Johnson, 2007).

Johnson, et al. (2007) states that the beliefs that act as scaffolding to a research paradigm include "ontological beliefs, epistemological beliefs, axiological beliefs, aesthetic beliefs, and methodological beliefs" (p. 24). This terminology we recognize as stemming from philosophy, but this fact does not prevent us from thinking that the word beliefs does not seem like a scientific term at all. There is, however, an entire branch of simulation or machine learning research that is based on algorithms and probabilistic structural equation modeling. The simulation research is also called Bayesian Beliefs Modeling, so named because an integral aspect of structural equation modeling is the expert knowledge that is brought to bear on the machine learning. The scientist's beliefs, we are then led to understand, truly matter in experimental research that employs algorithms. A researcher's thinking skills are necessary for automatic simulation research, just as they are for the process of identifying the best set of research questions to address a research problem, and for isolating the best research approaches to apply toward answering those research questions.

From Johnson, et al. (2007), we understand that research paradigm and research culture are roughly equivalent, the synonym of which, the authors argue, is methodological paradigm. Just as Kuhn meant for the term paradigm to be a general, adaptable term -- before he was pinned down by other scientists -- Johnson, et al. (2007) meant for the term to mature based on what it means to a group of researchers to conduct research and how they go about undertaking that research. In their discussion of mixed methods research, Johnson, et al., (2007) apply the terms research paradigms or methodological paradigms as organizers to separate what they perceive to be distinct types of research: Qualitative research, quantitative research, and mixed research.

Toward a definition of mixed methods research. Johnson, et al. (2007) explored definitions of mixed methods, provided a definition of mixed methods, and placed mixed methods approach in the research. Further, the authors argue that there are three research paradigms: Quantitative research, qualitative research, and mixed methods research. Expanding on the single general definition of mixed methods research, Johnson, et al. (2007) provide 19 distinct definitions of mixed methods research which they summarize in discussion and through a content analysis. Finally, the authors define qualitative dominant mixed methods research and quantitative dominant mixed methods research (Johnson, et al., 2007). The general definition of mixed methods research for which the authors found agreement, along with a deep bench of experts on research methods, is that mixed methods research can be said to occur when "a researcher or team of researchers combines elements of qualitative and quantitative research approaches for the broad purposes of breadth and depth of understanding and corroboration" (2007, p. 123). The research approaches to which the authors refer in their definition include, for example, data collection and analysis, inference techniques, qualitative perspectives, and quantitative perspectives. When placing a mixed methods approach in the research -- as a type of research -- the authors suggest research employing mixed methods "would involve mixing within a single study; a mixed method program would involve mixing within a program of research and the mixing might occur across a closely related set of studies" (Johnson, 2007, p. 123).

As we have said, research methods can be described as fitting within any one of three research paradigms: The qualitative research paradigm, the quantitative research paradigm, or the mixed methods research paradigm. Greene (2006) further parsed research methodologies into four domains: (1) "philosophical assumptions and stances;" (2) "inquiry logics;" (3) "guidelines for practice;" and (4) "sociopolitical commitments," the latter being most appropriate for social inquiry methods, but still having applicability with regard to the larger context in which the research is conducted. When discussing methodology, researchers benefit from articulating the fundamental epistemological or philosophical assumptions that undergird a select methodology. Greene (2006) argues that the term methodology is essentially the inquiry logics that guide topic selection, the development of research questions, the standards for quality and rigor, the study purposes, and the "writing forms that guide the researcher's gaze or point-of-view." The domain of guidelines for practice include any tools, procedures, techniques that are used to conduct the research -- this is the rubber-hits-the-road stage that tells a researcher how to structure and carry out the research. Scientists and researchers would like to believe that they are above the common pressures of the larger social context in which they do their work -- but that is decidedly not the case.

There is within -- and without -- every research endeavor of any substantive scope or size a socio-political and economic context, which is characterized by the power relations, institutional commitments, and individual and institutional interests of the society in which the research is conducted. In fact, these very same pressures can determine what types of inquiry will be employed, in effect vetoing the preferences of research departments and those with expertise in research. In such cases, it is often the anticipated reception of the intended audience for the research that is the deciding factor in the way the research design is structured. This is precisely one of the tenets that Suter (2005) purports in his work on multiple methods. It is worth noting that Suter is affiliated with the National Science Foundation.

Multiple methods. Research in the areas of science and mathematics education often employs multiple methods of experimental study (Suter, 2005). When single methods are employed in science and mathematics education research, the methods most often used, according to Suter (2005) are a version of experimental design or case study. That said, multiple methods prevail in research on educational practice, and the tendency is for the research designs in this area to attempt too great a reach. Suter (2005) argues that too little attention is paid to the development of focused research questions that pursue a problem over time through sustained research effort. At issue in educational practices research is that funding sources frequently require studies of a certain magnitude and that research that addresses the funding priorities of a governmental agency. Funders can reasonably be faulted for encouraging research that overreaches and results in inconclusive outcomes. This concern is less of a problem in business research, although research conducted under the auspices of a university-affiliated business school can suffer the same limitations. An important point derived from consideration of this issue is that a primary goal of research design is to focus on "research questions that match the question with a method" (Suter, 2005, p. 180). The selection of appropriate research methods will reasonably follow.

Just as the selection of generative research questions must fit with the research methods chosen for a study, the special area of decision aid research must calculate how to best use research to solve problems and make decisions.

Decision aid research. Experimental decision aid research has applicability to a wide range of disciplines. A primary advantage of decision aid research is its usefulness as a support to understanding the outcomes of experimental research. A decision aid is quite what it sounds like -- a tool for facilitating decision-making or problem-solving that provides embedded or inherent information relevant to the decision or problem. Decision aids can consist of simple paper-based supports or computerized algorithms that transform input into useful output in the form of data (Wheeler & Murthy, 2011). Interestingly, a substantive issue with decision aids is the effect that the use of the experimental decision aid has on the user -- agreement about how to solve this problem has not solidified (Wheeler & Murthy, 2011).

Making decisions based on experiments conducted the during manufacturing process is not is an easy as employing computerized modeling strategies. To be fair, 12 years separate the two research experiments. It is entirely probable that some of the problems experienced by Schultz (1999) during his experiments in operations management could have been eliminated through the use of more powerful computer systems and more flexible computer modeling programs such as those in use today.

Research in management accounting. In the discipline of accounting, experimental research has been employed in the sub-discipline of auditing, but has not similarly utilized in management accounting. Addressing what he perceived as a scarcity of appropriate empirical research in management accounting, Schultz (1999).applied experimental methods to a study of organizational structure, feedback, and production-system performance in the field of management accounting. Schutz's research explored ways to foster coordination of interdependent tasks in a machining operation and, as such, was fundamentally a study in operations management. His study illustrated the difficulty of applying traditional empirical research methods to an in situ manufacturing operation.

From the manufacturing plant floor to the agricultural field, applied experimental research makes known its idiosyncratic challenges. For Elliott (1929), the challenges are embedded not only in the experimental research processes, but also in the comparison between experimental research paradigms and economics research paradigms.

Research in applied agricultural economics. Elliott (1929) addresses the controversial tension between economic and experimental research, the former known for the propensity to suppose that all other things are equal, the later to require rigorous demonstration that all other things are equal enough to make and qualify conclusions. Elliott argues that statistical methods have matured sufficiently to enable agricultural economists to achieve the same level of confidence regarding their research conclusions as is evident in other fields. There is a cogent elegance in Elliott's statement that the inability to control all variables in a study results in situations where "problems will arise incident thereto which will have to be solved apart from those pertaining directly to the question under study in the experiment" (1929). Thusly have scientists steadily lamented. The agricultural economics research that is the focus of Elliott's work emphasizes the returns of different approaches to agriculture, including change outs of machinery, techniques, practices, and operating units. The research outlined in Elliott's (1929) article is a harbinger of the types of inquiries carried out by agricultural extensions of universities today.

The difficulties of reconciling experimental research procedures and techniques with those of behavioral economics plague Page (2010) just as they did Elliott (1929). The blithe assumptions that are made in economics -- presumably due to the recognition that there is absolutely no way that global economic variables can be "controlled" so they must be assumed to be stabile -- that all things remaining equal do not have any impact on economic research. The economic forecast is always qualified -- in experimental research, this would be considered bad form. In experimental research, one must control for, or correct for, or weight the variables over which one has no influence. Page (2010) copes by making the wild cards* all part of the game and an expected component of the psychology of the situation. This is the beauty of behavioral economics -- it establishes a paradigm based on the belief that humans will not act rationally, and that the tendency to behave in a non-rational manner can be mediated -- and this phenomenon can be demonstrated through gaming or research psychology.

Experimental economics. Education economists may employ experimental research methods to the study of student behavior, the beliefs and preferences that form the basis for student behavior, and the success or failure of a school (Page, 2010). The purpose of the study is to learn what variables influence the persistence of educational inequalities, with particular focus on financial variable and the individual motivation of students (Page, 2010). To the traditional economic methods, the author suggests the addition of experimental economics (Page 2010). Variables considered include risk preferences, time preferences, beliefs & expectations, concern with relative position, and identity (Page, 2010). Page suggests that experimental economics are "closely related to game theory, behavioural economics, and the experimental tradition in social psychology" (2005, p. 784). The author does an excellent job of explaining the limitations of behavioral economic research while suggesting ways to tie experimental economics to the conventions of research in the field of economics (Page, 2005).

Just as economics research tends to study the way things are -- in order to predict the way that things will be, as forecast, experimental research in healthcare administration is interested in how extant in-care services have been provided. Katz, et al. (2006), demonstrates the power of data mining and that is reminiscent of research conducted in the field of economics -- an enterprise that is built on its ability to collect large amounts of data.

Research in healthcare administration. Katz, et al. (2006) studied the feasibility of using administrative data to determine the quality of care in family practice. The authors argue that patient outcomes are not good measures of quality of care because they so often depend on socio-economic status (Katz, et al., 2006). The authors obtained data from three databases: The Population Health Research Data Repository, the Drug Program Inform Immunization Monitoring Program (MIMS), and the Physician Resource Database (Katz, et al., 2006). Using qualitative research methods that enabled data coding and database integration, administrative data was linked to responsible physicians according to relevant indicators, such as drugs prescribed (Katz, et al., 2006). Eight in-care services and chronic disease management indicators were developed and measured. Katz et al. (2006) found that the quality of care by physicians could be measured using administrative data. They concluded that the methodology they established provides suitable and reliable instrumentation for assessing quality of care and for developing in-care service and chronic disease management improvement initiatives (Katz, et al., 2006).

The capacity that Katz et al. (2006) demonstrated through data mining was an innovative demonstration in the use of what is known as "large" data. Essentially, the research conducted by Katz et al. (2006) was quasi-experimental in that there was not a way to randomly assign patients to groups. The research looked for relationships between the data, employing mathematical models to detect those relationships, and the measured the correlation among relevant variables. Sorensen, et al. (2011) also adapted experimental research procedures to fit their objectives and to fit the social context, referring to soft research procedures that sound remarkably like those used in action research, action learning, and reflexive practice. Sorensen, et al. (2011) described four experimental research studies for which adapted methods or approaches were used to research innovative programs.

Innovation research. Sorensen et al. (2011) argues that experimental research methods have not been widely used in business innovation and organizational innovation. This is a missed opportunity, according to Sorensen, for furthering the innovative development of methods and tools in businesses and organizations. Organizational development has a long history of experimental research, but Sorensen et al. (2011) suggest that the methods used have been limited, with case studies and surveys dominating the field. Open innovation is a contemporary of the 360 degree performance review. In an open innovation interactive system, ideas about innovation may be derived from any available source capable of contributing some applicable knowledge. With this framework in mind, it does not make sense to limit innovative input to conventional research configurations. Using a case study format, Sorensen, et al. (2011) compared four experiments in open innovation within service organizations and service businesses. Sorensen, et al. (2005) found that there was particular value in innovation experiments that are based in practice, and attributed this to researcher-practitioner interaction -- something that would be controlled for in conventional research. In this way, open innovation research appears to be a cross between action research (Levin, 2002) and usability research.