Evaluation Methodologies: Qualitative vs. Quantitative

Introduction to Evaluation Designs

Evaluation methodology has constantly developed over the past decades as the complexity of the evaluated group, task, or activity has continued to grow. This complexity has given rise to a multitude of methodologies and designs, of which this paper will highlight some of the most important. The paper also aims to discuss key differences among these evaluation designs and methodologies, and to draw conclusions from the comparisons made.

Single-Group and Time-Series Designs

Within a single group, the evaluator deals with the simplest types of design and evaluation. In general, there is a sole program targeting a single group with a particular purpose. This means that the evaluation can be as straightforward as determining whether participants met an objective or improved after the program was completed. This can be evaluated either qualitatively or through simple mathematical formulas.

For example, if one wished to determine whether a smoking-cessation program targeting a group was successful, the simplest evaluation would compare the number of smokers in the group before and after the program. If the number after the program is smaller than the number before, the program can be evaluated as having been somewhat successful. The single-group evaluation is probably the simplest, least time-consuming, and least expensive of all evaluation methodologies.

With more complex situations, the methodologies become more complex as well. One design that has been used extensively over the last couple of decades across various fields—including economics—is the time-series design or analysis. The main characteristic of this type of design is that information is gathered from different time periods. The methodology offers several important advantages: collecting information over a longer period increases the likelihood that the evaluation will reflect actual trends in the analyzed element, rather than patterns true only for a limited span of time. In the economic field, for example, this approach allows the analyst to adopt a broader perspective and examine economic cycles or other features of economic trends.

One potential problem with time-series analysis is that it can be difficult to determine whether a change in trend or pattern noticed around the time of a program's implementation is actually attributable to that program. However, the broader perspective afforded by time-series analysis can be an asset here as well, allowing the analyst to draw more pertinent conclusions from the data.

Experimental Methodology

Some modern evaluation methodologies propose using experiments as one of the primary tools for analysis. There are several important advantages to this approach, including the fact that it is perhaps the most objective of all the types of analysis discussed here (Posavac & Carey, 2006). The experimental approach also tends to provide results most closely aligned with the object of analysis, since the experimental design is typically built around the research purpose and does not deviate far from it. If a researcher wants to determine the effect of a certain cream on the skin, for instance, the experiment will involve the cream and human subjects and little else.

However, several important drawbacks must be considered when comparing experiments with other methodologies. It can be difficult to identify subjects willing to participate in an experiment; despite safety precautions, some individuals may be reluctant due to the risk of adverse effects from the products being tested. There are also high costs associated with certain experiments, as well as a subjective element that does not carry some of the positive traits found in the qualitative methodologies discussed below.

Qualitative Evaluation and the Role of Interviews

No matter how thorough they are, quantitative and statistical methodologies are not always sufficient. There are cases in which a qualitative design or methodology either complements the quantitative approach or replaces it altogether for better results. The qualitative design involves the direct participation of the evaluator and direct contact with the evaluated group, rather than interaction through a statistical intermediary. It also entails direct observation of the group, interviews with group members, and similar techniques.

The question of whether to select a qualitative or a quantitative approach to the evaluation process is an ongoing discussion in the field. Unless the best methodology is one that combines the two approaches, it is often the specific subject matter or situation at hand that will determine the most appropriate method to use (Dobrovolny & Fuentes, 2008).

The differences between qualitative evaluations and the statistical and quantitative evaluations previously discussed also reside in the final purpose of the evaluation. While quantitative evaluation is primarily aimed at assessing certain trends and drawing conclusions about the volume or quantity of a particular output, qualitative evaluation is more likely to probe the underlying causes of a given quantitative result.

If the observer plays the central role in a qualitative evaluation, then the interview is probably the most important instrument available in the qualitative design. As noted above, qualitative evaluation relies on direct contact between the observer and the individuals being studied. A well-conducted interview is the best means by which the observer can gather the information needed for analysis, as well as obtain details on the underlying reasons for certain behaviors or patterns within the group. As Kvale defined it, interviews are "attempts to understand the world from the subjects' point-of-view, to unfold the meaning of peoples' experiences, to uncover their lived world prior to scientific explanations" (Kvale, 1996).

This discussion naturally leads to a comparison of the financial and temporal costs of quantitative versus qualitative evaluation. In both cases, the depth to which the evaluator wishes to go is a determining factor. On one hand, quantitative analysis tends to be less costly than a qualitative one, primarily because statistical data covering a given time period is generally the type of public information available to all. On the other hand, an interview instrument must be designed to precisely target the individuals being evaluated and to cover all the relevant elements the observer wishes to analyze.

The interview also brings an additional level of interaction between the observer and the subject, which may require a period of sustained direct contact. Additional costs arise from this interaction, such as travel or local expenses and compensation paid to interview subjects.

However, a quantitative evaluation can also require significant investment before the right data are gathered. For example, if an evaluation concerns the impact of a new product launch on a company's consumer base, specific consumer data will be needed rather than general market statistics that are publicly available. The organization may need to invest in a different kind of interview—closer to a survey than a qualitative interview—that can provide relevant data on consumer preferences.

References

Combining Qualitative and Quantitative Methods for Evaluation. (2006, March). Retrieved July 20, 2009, from http://intelligentmeasurement.wordpress.com/2006/03/13/combining-qualitative-and-quantitative-methods-for-evaluation/

Dobrovolny, J. L., & Fuentes, C. (2008). Quantitative vs. qualitative evaluation: A tool to decide which to use. Performance Improvement, 47(4), 7–14.

Kvale, S. (1996). Inter-views: An introduction to qualitative research interviewing. Sage.

Posavac, E., & Carey, R. (2003). Program evaluation: Methods and case studies (6th ed.). Prentice Hall.