Program Evaluation Benefits for Health Care Managers

Introduction to Program Evaluation

Program evaluation is an important part of the health program planning, implementation, review, and change process. Patton (1997) defines program evaluation as "the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming." As this definition suggests, program evaluations can be carried out to serve different purposes, for example:

• To identify and describe potential program clients and their service needs.
• To improve program operations.
• To determine whether the program has achieved its objectives.
• To assess the program's impact.

There are a variety of different approaches and benefits to evaluation (Porteous et al., 1997). Therefore, in order to maximize the potential usefulness of an evaluation, managers must choose the approach or approaches that provide the highest quality, most useful information to answer evaluation questions, given the program context and the resources available to conduct the investigation.

This paper addresses the key issues regarding program evaluation in an effort to determine its value to health care management. It also demonstrates how and why different approaches to program evaluation should be considered, and how they can be most useful to managers.

When examining program evaluation, it is important to consider the term "program" in general. In most cases, organizations base their work on their mission to identify several overall goals that must be reached to accomplish that mission (McNamara, 1998). In the health care industry, each of these goals is often a program. Health care programs are organized methods to provide certain related services to patients, and therefore programs must be evaluated to decide if they are useful to clients. For a pharmaceutical company, for example, a program is often a one-time effort to produce a new product.

In this light, program evaluation means thoroughly and carefully collecting data about a program in an effort to make necessary decisions about it. According to McNamara (1998), "Program evaluation can include any or a variety of at least 35 different types of evaluation, such as for needs assessments, accreditation, cost/benefit analysis, effectiveness, efficiency, formative, summative, goal-based, process, outcomes, etc." The type of evaluation chosen to improve programs depends on what managers want to learn. Therefore, managers must concentrate on what information they need to make the necessary program decisions, as well as how they can accurately collect and understand that information.

Basic Myths About Program Evaluation

Some managers believe that program evaluation holds little value, arguing that it often produces large amounts of banal information with useless conclusions (McNamara, 1998). In the past, this was a significant problem, as program evaluation methods were selected mainly on the basis of complete scientific accuracy, reliability, and validity. This approach resulted in extensive data from which only scientifically accurate conclusions were drawn. For the most part, generalizations and recommendations were avoided in favor of raw facts.

As a result, evaluation reports essentially restated the obvious and left program administrators skeptical of evaluation's value. Today, however, evaluation focuses much more on utility, relevance, and practicality, rather than concentrating solely on scientific validity.

Another myth is that program evaluation is specifically designed to determine the success or failure of a program — assuming that success means implementing the perfect program and never having to revisit it. This is not a realistic goal. Instead, success means remaining open to continuous feedback and adjusting the program to meet changing needs. Evaluation provides managers with this continuing feedback.

In addition, many people believe that evaluation is a unique and complex process that occurs at a specific time in a specific way, and almost always requires the participation of outside experts. Many managers feel they must completely understand terms like "validity" and "reliability." However, this is not true. Managers must consider what information they need in order to make current decisions about program issues or needs (McNamara, 1998), and they must be willing to commit to understanding what is really going on.

Many managers regularly undertake some form of program evaluation and fail — not because evaluation is inherently flawed, but because they do not approach it in a formal, methodical manner. As a result, they fail to draw accurate conclusions and miss significant opportunities to make a greater difference for their customers and clients, or to earn greater returns.

The Importance of Program Evaluation in Health Care

Most health care managers today recognize the importance of measuring outcomes and costs (Grannemann, 2002). While most organizations have some basic measures of outcomes, cost, and service quality, many managers have questions about the cost-effectiveness of their programs and their real value to clients. For many health care organizations, managers need to determine exactly what they must invest in order to obtain better performance measures for their managed care programs. Several factors contribute to designing this type of evaluation: the programs themselves, the customers served, and the organization's willingness to use research results.

Understanding the answers to the following questions can be extremely useful for a manager in determining whether an investment in program evaluation is worthwhile (Grannemann, 2002).

1. What is program evaluation? Program evaluation is a scientific approach used to determine the value of managed care programs. The methods of formal program evaluation have traditionally been used to assess health care programs such as Medicare and Medicaid. Program evaluation helps managers assess costs and outcomes. It usually includes a scientific research design and statistical analysis designed to measure program effects, and the analysis may control for factors such as case mix, characteristics of the local health care system, and price differences. This process may also include cost-benefit or cost-effectiveness analysis.

Because program evaluation focuses on measuring program effects, it goes beyond simple outcomes measurement or benchmarking. It provides valuable feedback about the effects a program achieves, how it achieves them, and what its value is to the managed care organization and its clients.

2. Will it help an organization gain a competitive advantage? Program evaluation can help a health care organization gain a competitive edge by enabling managers to make better strategic decisions and to drive ongoing improvements in quality and efficiency. Managers need accurate information when faced with strategic decisions, such as what services to offer, how to introduce new products, what type of new products to develop, or whether to adopt a new health care management technique. Program evaluation provides this information and enables managers to identify what contributions a program will make to the organization, what program features will produce the desired results, and which elements should be replicated in future programs.

Many managers believe that the greatest benefit of program evaluation is the way it helps an organization move more quickly toward developing efficient, cost-effective programs — showing managers what works before their competitors discover it. Better programs enable organizations to offer clients lower costs or superior services.

3. Will it help with marketing? Program evaluations can help managers keep their clients more informed about their programs. The stronger clients' interest in quality and value, the greater the importance of documenting the effectiveness of programs. For example, in many markets, employers or health care purchasing coalitions examine evidence of quality and cost-effectiveness in their local market. If employers use this information when selecting managed care providers, organizations can gain a competitive edge by presenting credible, research-based evidence of their programs' cost-effectiveness.

Furthermore, by demonstrating the benefits of particular programs, evaluation enables managers to position various programs in the market to take advantage of real strengths. Evaluation results also help managers communicate that position clearly to their clients.

4. Is evaluation worth the cost? Each evaluation project should be weighed by its net benefits to the organization. Costs — research design, data collection, and analysis — are more easily estimated once a project is designed. Benefits, in terms of lower costs and/or improved outcomes, accumulate to the managed care organization, its business customers, and its patients. In many cases it is difficult to determine how much of this benefit is directly attributable to the program evaluation project. Evaluation is more of an enabling tool that helps managers develop cost-effective solutions and make the best strategic decisions. It also helps employers see the value of a managed care program and allows patients to receive better care.

As an enabling tool for decision-makers, program evaluation derives its value from the importance of the program being evaluated. Thus, evaluation is most important when applied to large programs and when it will help managers make significant decisions about those programs.

Is program evaluation a choice or a necessity? For many organizations, some type of evaluation is necessary if long-term success depends on maintaining a competitive advantage through cost-effective care. The real question is how much managers should invest in program evaluation to avoid mistakes and make the best decisions about future programs. Each evaluation project must be carefully designed to provide the best information for the money spent. Broadly defined evaluation projects can serve as a comprehensive assessment of a program, while limited, focused evaluations address specific questions. Managers must determine research needs and identify the value of addressing each issue, then design an evaluation project that addresses the key questions within an affordable budget.

A well-designed evaluation project should make a positive net contribution to an organization's long-term profitability — perhaps through reduced service utilization costs, reduced operating costs for the managed care program, or improved marketing (Grannemann, 2002).

How much should be spent on program evaluation? When deciding on an investment in program evaluation, managers should estimate its rate of return. The key variables in this estimation are the expected ultimate size of the program, the cost of the evaluation, and the expected contribution of the evaluation to improved program efficiency. Even if an organization does not currently have a budget for program evaluation, estimating the costs and potential benefits of a specific project may persuade organizational leaders that formal evaluation is a sound use of resources.

Conclusion

Overall, there are many benefits of program evaluation, especially in the health care industry. First and foremost, program evaluation measures performance. Health care managers in all sectors must measure program performance in order to sustain a competitive advantage (U.S. Department of Housing and Urban Development, 1997).

Program evaluation produces evidence that managers are directing resources into programs that benefit clients. It is also helpful in directing resources to support programs that work, and it can be used to identify what does not work. It further provides information to improve an organization's existing efforts.

Program evaluation demonstrates program benefits to investors, clients, and the community. If a hospital has a disease prevention program that works, for example, it can share this success with investors, clients, and the community. Investors will see that they are supporting a successful project, and this information can be used to attract additional investors or clients. Many investors require programs to measure performance or provide information on program results, service quality, and client satisfaction.

Evaluation can also provide valuable information about the impact of a particular program to audiences in a position to support such efforts, such as government officials, neighborhood associations, and community leaders (U.S. Department of Housing and Urban Development, 1997). It also helps organizations attract clients and provides the evidence managers need to obtain support for a program — whether to solicit new funding, request additional funds to expand a program, or to launch a similar initiative.

Another significant benefit of conducting a program evaluation is that the findings often directly improve programs. Program evaluations create an opportunity to share information about what works with similar departments or affiliates. If a manager has implemented a program that evaluation has shown to be effective, that information can be shared with other departments or used to create a new program. Program evaluations provide documented findings that confirm a program design's success, allowing it to be replicated with confidence.

In summary, program evaluation provides great value to health care managers by improving program effectiveness, demonstrating accountability, and supporting sound strategic decision-making. It is not simply an optional exercise but a strategic imperative for health care organizations committed to quality, efficiency, and continuous improvement.

Bibliography

Association of periOperative Registered Nurses (AORN). (January 2000). Evaluating patient care programs. AORN Journal, 71(1).

Beyea, et al. (November 1999). Using data to support decision making. AORN Journal, 70, 912–913.

Boruch, R. F. (1997). Experiments for planning and evaluation: A practical guide. Berkeley: University of California Press.

Grannemann, Thomas. (August 2002). When to invest in health care program evaluation. Andover Economic Evaluation. Retrieved from http://www.andoverecon.com/index.html.

House, E. R. (1993). Professional evaluation. London: Sage.

McNamara, Carter. (February 16, 1998). Nuts-and-bolts guide to nonprofit program design, marketing and evaluation. New York, NY: Authenticity Consulting, LLC.

Nachmias, D. (1995). The role of evaluation in public policy. In Public policy: The essential readings (pp. 173–180). Englewood Cliffs: Prentice Hall.

Nelson, et al. (1998). Improving clinical care: A clinical improvement sheet. In E. C. Nelson, P. B. Batalden, & J. C. Ryer (Eds.), Clinical improvement action guide. Oakbrook Terrace, IL: Joint Commission on Accreditation of Healthcare Organizations.

Nelson, et al. (1998). Measuring outcomes and costs: The clinical value compass. In Clinical improvement action guide (pp. 51–76). Oakbrook Terrace, IL: Joint Commission on Accreditation of Healthcare Organizations.

Patton, M. Q. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage.

Porteous, N. L., Sheldrick, B. J., & Stewart, P. J. (1997). Program evaluation tool kit: A blueprint for public health management. Ottawa: Ottawa-Carleton Health Department.

Rossi, P., & Freeman, H. E. (1993). Evaluation: A systematic approach. Newbury Park, CA: Sage.

U.S. Department of Housing and Urban Development, Office of Policy Development and Research. (1997). A guide to evaluating crime control of programs in public housing (pp. 1.2–1.4). Washington, DC: KRA Corporation.

Weiss, C. (1977). Introduction. In Using social research and public policy making. Lexington, MA: D. C. Health.

Wholey, J. S., Hatry, H., & Newcomer, K. (1994). Handbook of practical program evaluation. San Francisco: Jossey-Bass.