Evaluation Research and Social Science Methods Compared

Overview of Evaluation Research and Research Methods

Over the course of the semester, we reviewed three major types of research designs in the social sciences: quantitative, qualitative, and mixed methods. However, there is another classification — evaluation research — that is frequently deployed in organizations. Evaluation research may make use of all of these different methodologies, even though it has a different goal than academic research. As Trochim (2006) explains, "The generic goal of most evaluations is to provide 'useful feedback' to a variety of audiences including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies."

Colleague One also noted that evaluation research and research methods are not simply applications for research but are also purposes for research (Laureate Education, Inc., 2009). The correlation when using these approaches is that they all work together to distinguish a common reason, intent, goal, or idea, which is used in acquiring knowledge (Laureate Education, Inc., 2009). In social science, all of the research methods are applied to evaluation research, and in finding social change, all of the purposes of research are utilized in the search for new knowledge. Social science is an area of study with an ever-changing knowledge base, and the use and development of research purposes are tools used to seek that change (Laureate Education, Inc., 2009).

Evaluation Research Strategies and Methodological Overlap

In contrast to quantitative research's reliance on the scientific method, evaluation research is more intent upon offering practical data that can be used to maximize organizational resources, and it is specifically devoted to program evaluation. Some evaluation research does use standard qualitative methods in the tradition of the scientific method. However, an equally common approach is that of management-oriented systems models. Two of the most common of these are PERT, the Program Evaluation and Review Technique, and CPM, the Critical Path Method (Trochim, 2006). Other forms of evaluation research use anthropological models in the qualitative tradition, such as studying workers through participant interviews or observing interactions with clients. Mixed methods designs embrace both methodologies and combine them for a more holistic perspective — for example, assessing quantitatively gathered sales data alongside customer satisfaction feedback obtained from focus groups.

Colleague Two also observed that scientific-experimental models are the most common evaluation strategies and include experimental and quasi-experimental designs as well as theory-driven evaluation (Trochim, 2006). Qualitative and anthropological models, in which observation is emphasized and human interpretation is valued, represent another important class of evaluation strategies (Trochim, 2006). In both research methods and evaluation research, the role of the researcher or evaluator is critical: both must remain as objective as possible when interviewing, questioning, and reporting findings, and both must ensure the reliability and validity of the data collected (Silver, 2004). In short, both research methods and evaluation research rely on rigorous and systematic procedures for collecting data.

Not-for-profit organizations such as schools, and programs within the justice department and other government agencies, may also use evaluation research. Formative assessments — such as quizzes administered by teachers or short-term evaluations of data such as arrest records for police officers — provide immediate feedback about whether current programs are working. As Trochim (2006) explains, "Formative evaluations strengthen or improve the object being evaluated — they help form it by examining the delivery of the program or technology, the quality of its implementation, and the assessment of the organizational context, personnel, procedures, inputs, and so on." Summative evaluations, such as standardized tests or final reports, are designed to pass judgment on whether a particular program, method, or approach is working. A common summative evaluation technique is a cost-benefit analysis, in which it is assessed whether the costs or drawbacks of a particular initiative outweigh the benefits of implementing it.

Formative and Summative Evaluation in Practice

By virtue of necessity and scarce financial resources, quantitative methods often appear to be preferred within many disciplines — it is essential to demonstrate that a program works for a wide range of individuals, not simply a small and potentially non-representative sample. However, qualitative research and mixed methods designs have their own value in allowing for more individualized feedback, and statistics can be misleading if not grounded in participants' lived experiences. As Trochim (2006) concludes, "There is no inherent incompatibility between these broad strategies — each of them brings something valuable to the evaluation table."

Colleague Two noted that over the past decades quantitative methods have dominated education research; however, more recently there has been a shift toward qualitative methods (Szyjka, 2012). This shift has been driven by increased interest in topics such as student learning concepts, conceptual change, and social, cultural, and gender issues — all of which are best evaluated using qualitative approaches (Szyjka, 2012). There has also been a rise in program evaluation designed to measure outcomes relative to program objectives. In higher education, a common evaluation method is student feedback regarding courses and experiences gathered through questionnaires, which may be followed up with structured interviews or small-group discussions leading to a plenary session (Silver, 2004). In education more broadly, evaluation research is commonly used to assess program objectives, curriculum effectiveness, teaching quality, institutional quality, and the effectiveness of standardized tests (Silver, 2004). Ultimately, both research methods and evaluation research "add to new knowledge, stimulate and benefit from the development of theory and contribute to a science of education" (Silver, 2004).