Staffing Organizations: Reliability and Validity in Testing

Interpreting Reliability Results for Tests and Work Samples

The reliability of test results and work samples can be interpreted using the coefficient alpha measure. A high coefficient alpha implies that numerous items on the sample or results measure the same construct — they appear correlated and consistent (Bechet, 2008). An employer would expect a low coefficient alpha where a written job knowledge test is used to measure heterogeneous job skills. This suggests that the focal job demands a diverse range of skills and knowledge, which may include managerial, mechanical, clerical, and mathematical abilities. In such cases, the test results are intentionally measuring various forms of knowledge, and the samples are not expected to correlate strongly. This is the primary intention of the staffing professional, who will therefore anticipate a low coefficient alpha.

Retesting the reliability of low-scoring tests may be desirable when the work samples under investigation are unstable. In such cases, the work sample being measured will exhibit varied levels of the attribute at different points in time. Psychological factors such as attitudes and moods are expected to vary from one work sample to another during the intervals between tests and re-tests. In cases involving long intervals, attributes connected to achievement and ability would be expected to appear as results in the reliability of low test-retest scores (Heneman, Judge, & Kammeyer-Mueller, 2012).

The validity of results is interpreted through predictive validation. In this approach, the validity study begins with a job analysis aimed at identifying and defining the important tasks of the focal job. The organization would then assess the motivation required to perform those tasks. Underlying tasks are arrayed systematically within a job requirement matrix. Performance measures of tasks are achieved through existing measurement instruments (Bechet, 2008). It is necessary to develop predictor measures in a systematic manner based on the identified job analysis. Tests are administered to employees with the purpose of developing a criterion for job performance scores. The ability test generates predictor scores that are then linked to newly created criterion scores. This relationship is used to determine whether prevailing employee abilities are related to their current job performance as measured by the criterion scores. A high correlation indicates high concurrent criterion validity, suggesting that the identified abilities accurately reflect performance on the relevant tasks.

Interpreting the Validity of Results for Tests

For content validity to be interpreted, a job analysis must first be completed and a requirement matrix developed. Interpreting content validity is then a judgmental procedure carried out by internal or external experts. These experts must be thoroughly informed and knowledgeable about the nature of the focal job and its tasks. They deliver a judgment on whether the abilities reflected in the test results are a true representation of the organization's job analysis findings. Assessment of content validity tends to be judgmental and does not require the creation of a criterion measure, unlike criterion validity testing (Heneman, Judge, & Kammeyer-Mueller, 2012).

There are three major limitations that must be kept in mind when interpreting results and making decisions about whether to use clerical tests or work samples:

I. The level of similarity between the job applicants and the samples used in the study. A lack of similarity may indicate that the study's results are not generalizable to the broader population.

II. Whether the criterion measures used — such as complaint rates and error rates — are valid indicators of customer service representative (CSR) performance. If they are not, the test results will fail to predict important dimensions of CSR performance.