Summative Training Evaluation: Measuring Program Effectiveness

Introduction

Developing a training program is important both to the company and its personnel. It is through this training that employees become more confident in their abilities, and the company can experience higher productivity, better employee morale, and lower turnover rates. However, it is not enough to analyze training needs, then design and implement a training program. The company should also evaluate the training to ensure that the goals set based on the needs assessment are being met.

This phase of the training system is much overlooked in the process, yet it is important in determining the effectiveness of the design. There are various reasons for this oversight: organizations may lack confidence that programs will impact the company, or the HRD professional may not have the ability to conduct an evaluation. Sometimes the barrier is attributable to a "lack of resources and expertise" (Wang & Wilcox, 2006), and sometimes the organizational culture simply does not support the efforts behind conducting a training evaluation.

This article examines the process of summative evaluation, which is conducted at the end of the training program. This type of evaluation draws its conclusion from "measurement and assessment of program outcomes" (Wang & Wilcox, 2006). This review critiques the article and delivers an argument regarding its validity and practical applicability.

What is Summative Evaluation?

According to the Evaluation Toolbox, "Summative evaluation looks at the impact of an intervention on the target group" ("Summative Evaluation," 2012). Another explanation is that "summative evaluation is undertaken to test the validity of a theory or determine the impact of an educational practice so that future efforts may be improved or modified" ("Types of Evaluations," 2010). It identifies the benefits of the training in what the participants have learned and how their job performances were enhanced. Summative evaluations provide an opportunity to justify the investment in human resource development. This approach gives value to the implemented interventions and justifies the money allotted to the training budget.

A summative evaluation can also identify those interventions that did not result in meeting the goals and expectations. It is important to note that, while said interventions may not have been successful, they do provide opportunities for redesign and improvements in the process. The authors of the referenced article state that "training evaluation may be classified into three types" (Wang & Wilcox, 2006): cognitive, skill-based, and affective.

Cognitive evaluations reveal the participants' knowledge and the techniques they apply based on that knowledge. Skill-based evaluations assess whether participants demonstrate the learned information automatically and spontaneously. Affective evaluations reveal the attitudes of the participants and their motivational levels following the training.

According to the Social Research Methods Knowledge Base, summative evaluation can also be subdivided into five groups:

It is the HRD professionals' task to decide which data collection techniques they will use and which tools they will use to evaluate the results.

Short and Long Term Outcomes

According to the authors, one would think that "engaging in formative evaluation within the training phases of ADDIE and taking action on the findings will improve the quality of the training intervention" (Wang & Wilcox, 2006). However, even the best training plan and formative evaluation cannot guarantee successful summative evaluation, especially once the performers return to their work settings.

Four outcome evaluations can be realized from summative evaluations. Short-term evaluations measure "participants' perceptions and reactions on training that occur during and toward the end of the implementation phase" (Wang & Wilcox, 2006). These are often unreliable because they rely on participants' reactions and learning, which are affected by the participants' understanding of their particular learning needs. Carter McNamara states that feelings are a poor indicator of whether or not the training had an impact on the learners. He defines their reactions on learning as "attitudes, perceptions, or knowledge" (McNamara, n.d.)

Long-term outcomes involve on-the-job behavior and the impact on the organization, including return on investment. According to McNamara, this involves implementing the changes on the job and the improved performance due to enhanced behaviors. It is the long-term evaluation that has "received increasing attention in organizations" (Wang & Wilcox, 2006). Transferring the training to the job involves maintaining the skills, knowledge, and attitudes during a particular period of time. This means that the climate of the workplace must also be conducive to the maintenance of learned knowledge. There must be support from peers and supervisors, as well as the opportunity to use the information.

Organizational Impact and Nonfinancial Alternatives

One way of determining the organizational impact of training is return on investment (ROI). As Fred Nikols of the Training Journal states, "training does not have any direct financial impact—other than as the expense it represents" (Nikols, 2011). He further states that determining the ROI of the investment in training tells management what they got for their money. Organizations want to know that their investment has not only increased the skill levels of employees but also brought in more money than was spent on the training. However, according to Nikols, there is no true ROI associated with training.

Nikols argues that "training, whatever form it takes, is an expense, not an investment, and it has no direct financial impact except as an expense. Whatever positive financial impact training might have is always indirect, realized only through the effects of the actions of the people who have been trained and never as a direct result of training itself" (Nikols, 2011).

Studies have shown that many of the measurement requests are "based on a lack of understanding and misconceptions about evaluation" (Wang & Wilcox, 2006). Other authors argue that the training itself should not be the object of the evaluation, but rather one should evaluate how well the company uses the training. In this article, one researcher, Brinkerhoff, "demonstrated that effective training impact should involve all relevant stakeholders of a training program" (Wang & Wilcox, 2006). This approach would require that the stakeholder requirements be woven into all stages of the training, which would increase their interest in the outcomes and evaluate those outcomes in a way that would be meaningful to the stakeholders.

Conclusion

This review concludes that the article was an excellent compilation of information from researchers and training specialists regarding the impetus behind a training program as well as the expectations from management and stakeholders. HRD professionals must hone their skills on program design and implementation, but they must also develop measurement skills in order to create the instruments of evaluation that are valid and reliable.

There are differing opinions regarding the evaluation process and how it should be measured. The central point is that it is not only important to train employees, but to evaluate the training for job skills improvement and how the training met the goals of the organization. Management desires to know that the investment in training is paying for itself by improving job performance and, ultimately, the bottom line.