Usefulness of Program Evaluation to Management

¶ … Program Evaluation to Health Care Managers

Program evaluation is an important part of the health program planning, implementation, review, and change process. Patton (1997) defines program evaluation as "the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming." As this definition suggests, program evaluations can be carried out to serve different purposes, for example:

To identify and describe potential program clients and their service needs.

To improve program operations.

To determine whether the program has achieved its objectives.

To assess the program's impact.

There are a variety of different approaches and benefits to evaluation (Porteous, et al., 1997). Therefore, in order to maximize the potential usefulness of an evaluation, managers must choose the approach or approaches that provide the highest quality, most useful information to answer evaluation questions given the program context and the resources available to conduct the investigation.

This paper addresses the key issues regarding program evaluation, in an effort to determine its value to healthcare management. It will also demonstrate how and why different approaches to program evaluation should be considered and how they can be most useful to managers.

The Basics of Program Evaluation

When examining program evaluation, it is important to look at the term "program" in general. In most cases, organizations base their work on their mission to identify several overall goals that must be reached to accomplish their mission (McNamara, 1998).

In the health care industry, each of these goals is often a program. Health care programs are organized methods to provide certain related services to patients. Therefore, programs must be evaluated to decide if the programs are useful to clients. For a pharmaceutical company, for example, a program is often a one-time effort to produce a new product.

In this light, program evaluation means thoroughly and carefully collecting data about a program in an effort to make necessary decisions about the program. According to McNamara (1998), "Program evaluation can include any or a variety of at least 35 different types of evaluation, such as for needs assessments, accreditation, cost/benefit analysis, effectiveness, efficiency, formative, summative, goal-based, process, outcomes, etc."

The type of evaluation taken improve programs depends on what managers want to learn about the program. Therefore, managers must concentrate on what information they need to make the necessary program decisions, as well as how they can accurately collect and understand that information.

Basic Myths About Program Evaluation

Some managers believe that program evaluation holds little value, arguing that it often produces hoards of banal information with useless conclusions (McNamara, 1998). In the past, this was a major problem, as program evaluation methods were selected mainly on the basis of complete scientific accuracy, reliability and validity. This approach resulted in extensive data from which only scientifically accurate conclusions were drawn. For the most part, generalizations and recommendations were avoided in favor of facts.

As a result, evaluation reports basically just stated the obvious and left program administrators critical of the value of evaluation. However, today, evaluation focuses much more on utility, relevance and practicality, rather than just concentrating on scientific validity.

Another myth regarding program evaluation is that it is specifically designed to determine the success or failure of a program. This myth assumes that success is implementing the perfect program and never having to deal with employees, customers or clients again. However, this is not a realistic goal. Instead, success is remaining open to continuous feedback and adjusting the program to meet changing needs. Evaluation provides manager with this continuing feedback.

In addition, many people believe that evaluation is a unique and complex process that occurs at a specific time in a specific way, and almost always includes the participation of outside experts. Many managers feel that they must completely understand terms like validity and reliability. However, this is not true.

Managers must consider what information they need in order to make current decisions about program issues or needs (McNamara, 1998). In addition, they have to be willing to commit to understanding what is really going on.

Many managers regularly undertake some nature of program evaluation and fail. This is because they do not take the time to understand and do not perform it in a formal manner. Thus, they fail to draw accurate conclusions and basically waste their time. Many researchers argue that program evaluation cannot be accurate if not done methodically. As a result, these managers miss significant opportunities to make more of difference for their customer and clients, or earn greater profits.

The Importance of Program Evaluation in the Health Care Industry

Most health care managers today recognize the importance of measuring outcomes and costs (Grannemann, 2002). While most have some basic measures of the outcomes, cost, and quality of an organization's services, many have questions about aspects of the cost-effectiveness of the programs and their real value to clients.

For many health care organizations, managers need to determine exactly what they need to invest in order to obtain better performance measures for their managed care programs. There are several factors that contribute to designing this type of evaluation: programs, customers, and the organization's willingness to use research results. Understanding the answers to the following questions can be extremely useful for a manager in determining whether an investment in program evaluation is worthwhile for the organization (Grannemann, 2002).

1. What is program evaluation? Program evaluation is a scientific approach used to determine the value of managed care programs. The methods of formal program evaluation have been traditionally used to assess health care programs like Medicare and Medicaid. Program evaluation helps manager to assess costs and outcomes. It usually includes a scientific research design and statistical analysis designed to measure program effects. Analysis may control for the following factors: case mix, characteristics of the local health care system, and price differences. This process may include cost-benefit or cost-effectiveness analysis.

Due to the fact that program evaluation focuses on measuring program effects, it goes beyond simple outcomes measurement or benchmarking. It also provides valuable feedback about the effects a program achieves, how it achieves them, and what its value is to the managed care organization and its clients.

2. Will it help an organization gain a competitive advantage? Program evaluation can help a health care organization gain a competitive edge by helping managers make better strategic decisions and helping managers make ongoing improvements in quality and efficiency.

Basically, managers need accurate information when faced with strategic decisions, such as what services to offer, how to introduce new products, what type of new products to make, or whether to adopt a new health care management technique. Program evaluation provides this information and enables managers to identify what contributions a program will make to the organization. It helps managers see what program features will produce the desired results and should be replicated in future programs.

Many managers believe that the best benefit of program evaluation is the way it helps move an organization more quickly in terms of developing more efficient and cost-effective programs. Program evaluation shows managers what works before their competitors do. Evaluation can help managers update an organization's managed health care programs to beat the competition. Better programs enable organizations to offer their clients lower costs or superior services.

3. Will it help with marketing? Basically, program evaluations can help managers keep their clients more informed about their programs. The stronger the clients' interest in quality and value, the greater is the importance of documenting the effectiveness of programs. For example, in many markets, employers or health care purchasing coalitions examine evidence of quality and cost-effectiveness in their local market. If the employers in a particular market use this type of information in selecting their managed care providers, organizations can gain a competitive edge if they provide credible research-based evidence of the cost-effectiveness of health care programs.

In addition, by showing the benefits of particular programs, evaluation enables managers to see how to position various programs in the market to take advantage of real strengths and ability to produce results. Evaluation results also help managers make that position clear to their clients.

4. Is evaluation worth the cost? Each evaluation project should be weighed by its net benefits to the organization. The costs can be more easily weighed than the benefits. Costs of research design, data collection, and analysis are easily estimated once an evaluation project is designed. Benefits, in terms of lower costs and/or improved outcomes, accumulate to the managed care organization, its business customers, and patients.

In many cases, it is difficult to determine how much of this benefit is due to the program evaluation project. Evaluation is more of an enabling tool that helps managers develop cost-effective solutions and make the best strategic decisions. It also helps employers to see the value of a managed care program and allows patients to receive better care that will restore good health.

As an enabling tool for decision makers, program evaluation gets its value from the importance of the program that is being evaluated. Thus, evaluation is more important when applied to large programs and when it will help managers make important decisions about those programs.

Is program evaluation a choice or a necessity? For many organizations, evaluation is a must. Some type of evaluation is necessary if an organization's long-term success depends on maintaining competitive advantage by offering cost-effective care. The real question is how much managers should invest in program evaluation to avoid mistakes and make the best decisions about future programs.

For this reason, the design of an evaluation is important. Each evaluation project must be carefully designed to provide the best information for the money spent. Broadly defined evaluation projects can serve as a detailed and comprehensive assessment of a program, while limited, focused evaluations address specific questions. Managers must determine research needs and identify the value of addressing each issue. Subsequently, an evaluation project can be designed, addressing the key research questions with an affordable budget.

A well-designed evaluation project should make a positive net contribution to an organization's long-term profitability, perhaps through reduced costs of service utilization, reduced cost of operating the managed care program, or improved marketing (Grannemann, 2002).

How much should be spent on program evaluation? When making a decision about an investment in program evaluation, manager should estimate its rate of return. An estimated calculation of costs and expected benefits can help guide decisions. The key variables in this estimation are: expected ultimate size of the program, the cost of the evaluation, and the expected contribution of the evaluation to improved program efficiency.

In many cases, health care managers discover that the expected rate of return from an investment in program evaluation is very expensive. Even if an organization does not have a budget for program evaluation, determining the costs and potential benefits of a specific evaluation project may be convincing in showing organization leaders that formal evaluation is a good use of an organization's resources.

Designing a Useful Program Evaluation

When designing an evaluation approach, manager should review the following three types of evaluations, which are common in most organizations, in order to maximize the usefulness of their evaluations. Often the manager will act as the evaluator in the evaluation.

Weiss describes academia's view of evaluation's impact, "It would give cause-and-effect theories for policymaking, so that statesmen would know which variables to alter in order to effect desired outcomes" (1977). Ideally, the evaluator would design a hypothesis as close to the policy goals as possible. The error of probability would be no low and the research would be conducted. At the conclusion of the study, if the program's performance reflected the desired policy goals, the program would have a greater chance of continued funding and continuance.

However, if the results where negative or at best questionable, managers could at this point reduce funding or disband the program with scientific assurances of their decision (Nachmias, 174-175). Much is expected of managers and their use of the information presented by researchers. With project evaluation, therefore, it is important to conduct the right evaluation for the organization.

The first type of evaluation is goals-based evaluations, which are helpful in determining whether or not programs are achieving their overall, predetermined objective (McNamara, 1998)s. In many cases, programs are designed to meet one or more specific goals. These goals are often described in the manager's blueprint for program plans and goals.

Goal-based evaluations aim to determine which programs are successfully meeting predetermined goals or objectives. There are a series of questions that manager must address when designing an evaluation, including (McNamara, 1998):

1. How were the program goals established? Was the process effective?

2. What is the status of the program's progress toward achieving the goals?

3. Will the goals be achieved according to the timelines specified in the program implementation or operations plan? If not, then why?

4. Do personnel have adequate resources (money, equipment, facilities, training, etc.) to achieve the goals?

5. How should priorities be changed to put more focus on achieving the goals?

6. How should timelines be changed?

7. How should goals be changed (be careful about making these changes - know why efforts are not achieving the goals before changing the goals)? Should any goals be added or removed? Why?

8. How should goals be established in the future?

Process-based evaluations aim to determine how a program really works, as well as its strengths and weaknesses (McNamara, 1998). Process-based evaluations, when properly designed, help managers to understand how a program works and how it produces the results that it does.

These evaluations are useful if programs were established a long time ago and have changed over the years, if employees, clients or patients report a large number of complaints about the program, if there appear to be significant problems in delivering program services and if they are also useful for portraying to outside parties (such as shareholders) how a program truly operates.

Managers must address several questions in a process evaluation, including (McNamara, 1998):

1. On what basis do employees and/or the customers decide that products or services are needed?

2. What is required of employees in order to deliver the product or services?

3. How are employees trained about how to deliver the product or services?

4. How do customers or clients come into the program?

5. What is required of customers or client?

6. How do employees select which products or services will be provided to the customer or client?

7. What is the general process that customers or clients go through with the product or program?

8. What do patients or clients consider being strengths of the program?

9. What do employees consider to be strengths of the product or program?

10. What typical complaints are heard from employees and/or customers?

11. What do employees and/or customers recommend to improve the product or program?

12. On what basis do employees and/or the customer decide that the product or services are no longer needed?

Outcomes-based evaluations are important in the health care industry, as they are helpful in identifying benefits to clients. An outcomes-based program allows a manager to determine if the company is really using the right program activities to bring about the outcomes needed by clients, rather than just engaging in busy activities that appear to be correct. Outcomes are benefits to clients from participation in the program, and are usually in terms of enhanced learning or conditions, including increased literacy, self-reliance, and more.

The following table demonstrates the major methods used for collecting data during program evaluations, as well as how they can be useful and challenging to managers (McNamara, 1998):

Method

Overall Purpose

Advantages

Challenges questionnaires, surveys, checklists when need to quickly and/or easily get lots of information from people in a non-threatening way can complete anonymously

-inexpensive to administer

-easy to compare and analyze

-administer to many people

-can get lots of data

-many sample questionnaires already exist might not get careful feedback

-wording can bias client's responses

-are impersonal

-in surveys, may need sampling expert

- doesn't get full story interviews when want to fully understand someone's impressions or experiences, or learn more about their answers to questionnaires get full range and depth of information

-develops relationship with client

-can be flexible with client can take much time

-can be hard to analyze and compare

-can be costly

-interviewer can bias client's responses documentation review when want impression of how program operates without interrupting the program; is from review of applications, finances, memos, minutes, etc.

A get comprehensive and historical information

-doesn't interrupt program or client's routine in program

-information already exists

-few biases about information often takes much time

-info may be incomplete

-need to be quite clear about what looking for -not flexible means to get data; data restricted to what already exists observation to gather accurate information about how a program actually operates, particularly about processes view operations of a program as they are actually occurring

-can adapt to events as they occur can be difficult to interpret seen behaviors

-can be complex to categorize observations

-can influence behaviors of program participants

-can be expensive focus groups explore a topic in depth through group discussion, e.g., about reactions to an experience or suggestion, understanding common complaints, etc.; useful in evaluation and marketing quickly and reliably get common impressions

-can be efficient way to get much range and depth of information in short time

- can convey key information about programs can be hard to analyze responses

-need good facilitator for safety and closure

-difficult to schedule 6-8 people together

Benefits to Health Care

According to the Association of periOperative Registered Nurses (AORN) (2000), program evaluation of patient care can effectively assist health care managers when making decisions about clinical innovations. In health care, many managers implement innovations but never fully evaluate their effect in a consistent manner.

Until recently, no unified program evaluation framework or structure existed in clinical practice. Program managers have simply guessed that an innovation will work and proceeded to develop that program accordingly. Thus, while managers become very involved in developing and implementing an innovation, they often fail to collect the necessary information to understand the full effect of their ideas and programs.

For example, in nursing, failure to perform a patient care program evaluation occurred with the inception of primary nursing during the 1970s (AORN, 2000). In this case, many nurses accepted the idea that care delivered via a primary nursing model was better for nurses, patients, family members, and providers, in comparison to team nursing. This concept was widely adopted and popular, yet very few nursing departments actually conducted a program evaluation of the outcomes related to primary nursing care delivery.

When health care reform began and cost-cutting initiatives were implemented, the benefits of primary nursing had not been formally substantiated. Nursing managers and hospital administrators attempted to justify continuing with what was viewed as a costly innovation.

However, with little evidence to support the belief that primary nursing care had improved patient outcomes and decreased lengths of stay, it was removed. Many hospitals, in a redesigning attempt, have systematically eliminated RN positions and the primary nursing care model.

This scenario demonstrates the importance of program evaluation. Evaluation can help health care managers understand and evaluate their programs, to prove what works and what does not. Ideally, it demonstrates how a specific program contributes to the health of the patients and family members it serves and how the program's costs relate to those outcomes.

Important program evaluation data includes patient outcomes, costs, and quality. Additional factors may be measured concurrently and examined, depending on the type of program and its goals. A program's type and purpose determines the specific measurement efforts.

According to, to successfully evaluate a program, manager should use a framework to conceptualize the process. The clinical value compass (CVC) provides a framework to measure (AORN, 2000):

biological outcomes, functional health status, satisfaction, and costs.

Biological outcomes refer to morbidity, mortality, complication rates, diagnostic tests, and laboratory findings. Functional outcomes include measures of physical and mental function, social and role function, and health status. Satisfaction addresses outcomes include the patient's and family members' satisfaction with program processes and clinical outcomes. Direct costs include the cost of care, follow-up visits, medications, and other treatments, and indirect costs include the patient's time away from work or usual activities (AORN, 2000, Nelson, et al., 1998).