Experimental Research Design: Methods and Processes

Introduction to Experimental Research

The research process is described as one that utilizes scientific techniques to investigate phenomena and is focused on the acquisition of new knowledge. The experimental method requires the basis of "observable, empirical and measurable evidence" to be considered scientific, and it must also "follow some principles of reasoning" (Experiment Resources, 2009).

Experimental research is generally used across the sciences, including sociology, psychology, biology, chemistry, and other disciplines. It is defined as "a systematic and scientific approach to research in which the researcher manipulates one or more variables, and controls and measures any change in other variables" (Experiment Resources, 2009).

Experimental research is appropriate when a time priority exists in a causal relationship — that is, when the cause precedes the effect — when there is consistency in a causal relationship such that a cause always produces the same effect, and when the magnitude of the correlation is large (Experiment Resources, 2009).

Although experimental research carries a broad range of definitions, in its strictest sense it refers to what is termed a "true experiment," in which "the researcher manipulates one variable, and control/randomizes the rest of the variables" (Experiment Resources, 2009). A true experiment has a control group, subjects are selected and assigned through a random process, and the researcher tests only one effect at a time.

More broadly, experimental research is also defined as a "quasi-experiment," in which "the scientist actively influences something to observe the consequences. Most experiments tend to fall in between the strict and the wide definition. A rule of thumb is that physical sciences, such as physics, chemistry and geology tend to define experiments more narrowly than social sciences, such as sociology and psychology, which conduct experiments closer to the wider definition" (Experiment Resources, 2009).

The Experimental Design Process

The experimental design process includes the following steps:

(1) Research question (Hypothesis)
(2) Design experiment
(3) Data collection
(4) Data analysis
(5) Conclusions

Once a researcher decides to test something, they identify the research problem and operationalize it in order to define how it will be measured. The test results will depend on the exact measurements chosen. By defining the research problem, the researcher is able to formulate a research hypothesis, which can then be tested against the null hypothesis (Experiment Resources, 2009). An "ad hoc analysis" is an additional type of hypothesis that is sometimes "added to the results of an experiment to try to explain away contrary evidence."

Constructing an experiment involves several aspects of planning. Planning ahead ensures that the experiment is carried out properly, that the results reflect the real world as accurately as possible, and that this is accomplished through the best method available.

The work of Congdon and Dunham (1999), "Defining the Beginning: The Importance of Research Design," states that a research plan consists of two general areas: (1) research concepts and context, and (2) research logistics. Success or failure is often determined by how well the research is planned and "how well the steps in the plan are integrated" (Congdon and Dunham, 1999). It is important that data sheets are designed specifically for collecting data in order to minimize mistakes and omissions. Computer programs should ideally be used in managing data.

Sampling is an important aspect of experimental research design and must be correctly executed. When there is more than one condition in the experiment, one group serves as the "control group whereas the others are tested under the experimental conditions" (Experiment Resources, 2009). Group samples can be determined in several ways, including:

Sampling and Experiment Construction

(1) Randomization
(2) Quasi-randomization
(3) Pairing (Experiment Resources, 2009)

These are among the most commonly used sampling methods. When constructing the experiment, researchers often adjust the sample size to reduce the chances of random sampling errors. Factors that affect the research design chosen include time, money, ethics, and measurement problems. "The design of the experiment is critical for the validity of the results" (Experiment Resources, 2009).

Typical designs used in experiments include the following:

(1) Pre-test: Used to check whether the groups differ prior to the experiment; known to sometimes influence the effect.
(2) Post-test: Measures the effect or effects.
(3) Control Group: Designed to measure research bias and measurement effects, such as the Hawthorne Effect. A control group is a group that does not receive the same manipulation. Experiments frequently have two conditions, but rarely more than three at the same time.
(4) Solomon Four-Group Design: Uses two control groups and two experimental groups to test both the effect and the effect of a pre-test.
(5) Double-Blind Experiment: Neither the researcher nor the participants know which is the control group. Results can be affected if either party knows this information.
(6) Bayesian Probability: This method uses Bayesian probability to "interact" with participants. It can be used in settings where there are many variables that are difficult to isolate. The researcher begins with a set of initial beliefs and tries to adjust them based on how participants have responded (Experiment Resources, 2009).

A pilot study may be conducted prior to the actual experiment to ensure that the experiment measures what it is intended to measure. The pilot study helps identify minor errors that have the potential to compromise the experiment. When the experiment involves human participants, "a common strategy is to first have a pilot study with someone involved in the research, but not too closely, and then arrange a pilot with a person who resembles the subject(s). Those two different pilots are likely to give the researcher good information about any problems in the experiment" (Experiment Resources, 2009).

Conducting an experiment is accomplished through "manipulating a variable, called the independent variable, affecting the experimental group. The effect that the researcher is interested in, the dependent variable(s), is measured" (Experiment Resources, 2009). It is necessary to identify and control non-experimental factors that the researcher does not want influencing the results. "This is often done by controlling variables, if possible, or randomizing variables to minimize effects that can be traced back to third variables. Researchers only want to measure the effect of the independent variable(s) when conducting an experiment, allowing them to conclude that this was the reason for the effect" (Experiment Resources, 2009).

Conclusion: Research Design and Rigor

Publication of recent research indicates that "case-based research can be as useful and rigorous as other research methods" (Naslund, 2005). Naslund further relates that "the use of triangulation suggests that logistics researchers see value in combining methods to increase theory development and improve data collection" (2005).

Logistics problems are described by Naslund (2005) as "often ill-structured, even messy, real-world problems." Logistics uses multi-disciplinary and cross-functional approaches. In order to provide value to industry and education, logistics researchers must gain "extreme relevance" by understanding what is occurring within and between organizations (Naslund, 2005). Logistics research will realize benefits when researchers invest time observing and communicating with professionals performing logistics in practice. Ultimately, the strength of any research lies in how carefully the design is planned, how well each step is integrated, and how rigorously the methodology is applied to the research problem at hand.