Evidence-Based Instruction: Methods, Standards, and Limits

Defining Evidence-Based Instruction

Proponents of evidence-based instruction represent one end of the teaching methods continuum, where practices that have been tested empirically using rigorous research designs are considered the only valid form of instruction (Odom et al., 2005). At the other end of the spectrum are methods that may have some basis for use—such as intuition or theory—but that have not been subjected to empirical scrutiny and are nonetheless considered valid to use. Evidence-based instruction, or scientific research-based instruction, consists of instructional practices or programs for which empirical data have been collected to determine effectiveness (Odom et al., 2005).

In these types of practices and programs, rigorous research designs have been conducted to evaluate effectiveness. Such designs can include randomized controlled trials, quasi-experiments, single-subject designs, correlational methods, and qualitative research. The most empirically sound designs—randomized controlled experiments—are used to demonstrate that students exposed to a practice show significant improvements compared to a matched group of students who are not exposed to it. In these tightly controlled studies, researchers have greater confidence that observed changes are due to the specific practice or intervention being tested, rather than to other variables such as teacher characteristics, student characteristics, or environmental conditions.

The types of educational practices with the strongest research evidence have been examined across large numbers of students, in a wide range of settings, and with diverse subgroups. Often, relatively few studies meet such rigorous criteria, and reviewing available research to determine whether it fits this category can be a lengthy, complex, and expensive process (Odom et al., 2005).

Federal Policy and Scientific Research in Education

When considering the federal government's involvement in educational research and its engagement with the National Academies, it becomes clear that several fundamental assumptions about education must underlie that involvement. The first assumption is a widespread belief that educational research should—or at least can—be scientific, at least in part. The second is that, with respect to education policy, the federal government distinctively seeks out scientific research when making decisions about policy and practice. A third assumption is that the quality of educational research was considered to fall short of necessary standards in at least some areas, thus making further research necessary. Finally, there is an assumption that establishing a scientific basis for educational research is itself worthy of serious attention and should not be subject to political influence.

With respect to educational research, the No Child Left Behind Act is probably the most well-known advocate for requiring teachers to use empirically validated techniques in the classroom. Many major agencies focused on empirically validated instruction have maintained that the "gold standard" for validating an educational practice or program is the randomized controlled design (e.g., What Works Clearinghouse [WWC], 2012). However, in recent years there has been concern in other fields about the types and quality of scientific evidence acceptable for validating a premise, program, or theoretical assumption (White & Smith, 2002).

The National Research Council has stated that questions regarding the effectiveness or utility of practices in education encompass different types of questions requiring different methodological approaches (Shavelson & Towne, 2002). The emphasis on randomized controlled trials most likely originated in the medical field, where such trials are necessary to test the effectiveness of medications and medical treatments against no treatment or against existing treatments for specific diseases or ailments. For the question "Which is generally more effective, treatment A or treatment B?" this method is the gold standard when "effective" is easily operationalized. This is often the case with medications or medical treatments where both the pathology and "wellness" are well-defined. In education, however, the "treatment effect" can be considerably more variable, and subject variables can be more diverse and involve significantly more complex interactions than those seen in medical research.

Research Question Types and Appropriate Methodologies

The primary aim of education research policy should be to improve the quality of education for everyone. The National Research Council proposed that research questions in education fall into three broad types:

1. Descriptive — "What is happening?"
2. Causal — "Is there a systematic effect?"
3. Process-focused — "How or why is it happening?" (Shavelson & Towne, 2002)

Each of these questions requires a different method to answer. Most medical research has traditionally concentrated on the second question, which is important but not the only question worth considering—hence the emphasis on randomized controlled trials. For the third type of question, correlational or qualitative methods (such as surveys or observational studies) would be more appropriate. For the first, correlational methods would generally suffice. There should be an appropriate match between the question being asked and the method used to answer it.

As a result, the definition of what constitutes evidence-based research has broadened somewhat with the recognition that educational research questions are often wider in scope than those of the medical field. Studies may employ a range of methodologies, including systematic empirical methods that rely on observational or experimental designs suited to the questions asked. When reviewing such research, intervention studies using experimental or quasi-experimental designs are often still given the highest priority (August & Shanahan, 2006; Condelli & Wrigley, 2004). However, descriptive, correlational, or ethnographic studies may be used to support experimental findings or to assist in the development of theory (August & Shanahan, 2006). Studies that rely on measurement or observational methods providing valid data across multiple observers, evaluators, and measurement occasions are also considered to offer empirical evidence for appropriate research questions (August & Shanahan, 2006; Condelli & Wrigley, 2004). Regardless of method, these studies should always include rigorous data analyses sufficient to test researchers' hypotheses and justify their conclusions.

References

August, D., & Shanahan, T. (Eds.). (2006). Executive summary. Developing literacy in second-language learners: Report of the National Literacy Panel on Language-Minority Children and Youth. Mahwah, NJ: Erlbaum.

Condelli, L., & Wrigley, H. S. (2004). Identifying promising interventions for adult ESL literacy students: A review of the literature. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

Foorman, B. R., & Torgesen, J. (2001). Critical elements of classroom and small-group instruction that promote reading success in all children. Learning Disabilities Research & Practice, 16, 203–213.

Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71, 137–149.

Roberts, G., Torgesen, J. K., Boardman, A., & Scammacca, N. (2008). Evidence-based strategies for reading instruction of older students with learning disabilities. Learning Disabilities Research & Practice, 23, 63–69.

Shavelson, R. J., & Towne, L. (2002). Scientific research in education. Washington, DC: National Academy Press.

What Works Clearinghouse. (2012). WWC evidence review protocol for science interventions. Washington, DC: Author.

White, E., & Smith, C. S. (2002). Theme issues on scientific research in education. Educational Researcher, 31(8), 3–29.