The impact of politics on program evaluation and vice versa

Politics and Program Evaluation

Getting from Good to Best:

The Role of the Politics in the Evaluation of Public Programs

Any program designed for the public good is by its nature inextricably linked to politics, not only through motive but in many instances through more practical elements as well. The political system provides the directives and often the financial support for these programs, and thus the programs themselves are prone to many of the same strengths and weaknesses as the political system. Program evaluation is also part of this link. Eleanor Chelimsky (1987) points out that "evaluation, with its purpose of providing high-quality information to decision makers, automatically claims a role for itself in the political process" (p. 24). What this role should be, and what it ultimately is in reality, are questions of fundamental importance to those involved in public policy administration and in the political world at large.

No one doubts the overall usefulness of program evaluation. Evaluation, according to Berk and Rossi (1999), is rooted in "the commonsense idea that social programs should have demonstrable benefits" (p. 1). As "commonsense" as this may seem, what qualifies as "demonstrable" and what qualifies as a "benefit" are not always simple matters. Even when the evidence is clear and the benefits are undisputed, how this information should be used in the political sphere is often a matter of contention.

There has been a long-standing concern in the evaluation research community that program evaluations are not used enough or in the proper manner by political decision makers. Carol Weiss elucidated this concern in her 1973 landmark paper "Where Politics and Evaluation Research Meet." In this paper, Weiss states that "relatively few evaluation studies have had a noticeable effect on the making and remaking of public policy" (qtd. In Cheminsky 1987, p. 24). This situation can be frustrating to the sociologist and especially disheartening to the program evaluator, since "being taken seriously in policy matters would appear to be part of [the evaluator's] professional identity" (Rothbart 1975, p.23).

A salient example of this dismissal of evaluation results by policy makers can be seen in the case of the Head Start evaluation performed by the Westinghouse Corporation in 1969. Despite the fact that this evaluation showed the benefits of Head Start to be short-lived if present at all, federal policy makers decided to fund the program at an even higher rate. The methodology of the Westinghouse Study was found to be questionable, but subsequent studies have confirmed the disappointing results. However, these results seem to have no effect on the public perception of the Head Start program, or on its political viability. Labeled "a program that works" by Jimmy Carter and expanded significantly under Ronald Reagan and George W. Bush, Head Start has at times commanded an annual budget of more than $1 billion (Besharov and Hartle 1985, para.1).

This case points to a fundamental difference between the priorities of program evaluators and those of policy makers. Referring to the Head Start case study, George Rothbart (1975) noted that, while Head Start fails as a program if approached in a methodological sense, "political elites may view Head Start, not in terms of its capacity to maximize educational change, but in terms of the alternatives that might be necessary to satisfy community demands" (p. 25). While the aim of the evaluator is to reach a fair and data-driven conclusion about the effectiveness and focus of the program, the goal of the policy maker in interpreting and using the evaluation is determined in no small way by the public popularity of the program and the political price to be paid by changing or terminating it.

However, there need not always be a disjunction between the aim of the evaluator and the ultimate priorities of political decision makers. According to Huey-tsyh Chen (2005), "evaluation is most useful when it contributes [my italics] to the political process" (p. 269). This is most likely to happen when policy makers and evaluators "have a mutual understanding on the evaluation needs and the evaluation approaches to be used" (Ibid). How this mutual understanding comes to be is not always clear, and often depends on the philosophical assumptions of the program administrators, the program evaluators, and the politicians creating the program policy.

Eleanor Chelimsky broke down the philosophical differences between these parties in her 1987 paper "The Politics of Program Evaluation." The program evaluators often feel "that evaluations cannot be used because of bureaucratic relationships and conflicts, & #8230;and that evaluations will only be used if management really wants them" (p.25). On the other hand, there is often an assumption on the part of the users that evaluations are "an ivory tower process…too late to be useful, too full of jargon to be understood, too lengthy [to read]…, and too likely to be answering a question quite different from the policy question originally posed" (Ibid).

The last user complaint set forth by Chelimsky -- that the question answered is often not the question posed -- points to the problem of what role, if any, the policy makers themselves should have in forming the evaluation criteria. This problem was a source of pointed debate after the publication of the Equality of Educational Opportunity Study (also known as "the Coleman Report") in 1966. This study was commissioned by the United States Department of Health, Education, and Welfare to determine the effectiveness of the Civil Rights Act in ensuring equal educational opportunities for people of all race, color, religion, and national origin. It found that disparities in educational opportunities remained high, not as a factor of race or religion, but as a factor of socio-economic conditions. While it highlighted the need for a war on poverty, it also gave segregationalists statistical fuel for the argument that school integration would have no effect on equalizing educational opportunities.

This led Glen Cain and Harold Watts to question Coleman's methodology in their 1970 paper "Problems in Making Policy Inference from the Coleman Report." In this critique, they argued that Coleman chose his variables based on "broad and disinterested scientific concerns" (Rothbart 1975, p.23). While this seems an entirely appropriate approach to determining variables from a social scientist's standpoint, Cain and Watts argued that the social scientist must also play the role of social engineer, claiming that variables must be chosen "for their potential role in policy manipulation" (Ibid). In his reply, Coleman retorted that his job was one of scientific discovery, not outcome manipulation.