Reliability and validity assessment of Phonemin's hiring tests

Customer Service Evaluation

How do you interpret the reliability results for the clerical test and work sample? Are they favorable enough for Phonemin to consider using them "for keeps" in selecting new job applicants?

The clerical sample is the most reliable of the three tests given to applicants as its test-retest r score is .92 at the .05 level of significance. The paradox of the test-retest r, r with speed and r with error rate is the lack of validity over time of these results however. They are reliable in measuring clerical proficiency yet the relative low levels of r with error rate and very high standard deviations (4.70 and 5.11) relative to the work samples make them not as reliable as the clerical test.

The dilemma the hiring company has is that one of their tests, the clerical on, is highly reliable, yet not valid for all hires. The work sample tests lack validity of measuring skills to the extent the clerical test does. Phonemin needs to use the insights gained from the clerical test to create a more valid test across all skill sets, and get beyond concentrating on clerical speed and accuracy in the context of their hiring tests.

The results of using these tests as they are today will be hiring clerical workers who excel at taking tests yet have work samples that vary significantly in terms of quality and validity from an ongoing performance standpoint. The effective design of hiring tests for employment needs to include actual skill set mastery and an orientation to reliability of application of knowledge, not just measuring cognitive test-taking ability (Heneman III, Judge & Kammeyer-Muller, 2012). Due to these factors the reliability of the two work samples needs further work and redefinition.

2. How do you interpret the validity results for the clerical test and work sample? Are they favorable enough for Phonemin to consider using them "for keeps" in selecting new job applicants?

The validity results for the clerical test are troubling in that they have such a very wide standard deviation and show lack of consistency as a result. The clerical test has a very low level of validity as a result, and needs to be redefined by Phonemin before it can be used reliably.

The validity of the work samples however is very high. First, the standard deviations for Work Sample T. with Time 1 and Time 2 are .93 and 1.01 respectively and with Work Sample ( C ), .99 and 1.10 standard deviations. This low level of variance in scores shows that the work samples are measuring the propensity of potential employees to get work done; yet they vary with r with work sample levels of .81 and .77, singif8icant at the .05 level, shows that the tests while valid, are not reliable in measuring ongoing performance gains. The r with complaints for Work Sample (T) of .37 and .35 also indicate that the test is highly valid yet lacks reliability in ensuring long-term customer service levels. Work Sample (T) shows that it is only sporadically valid as well with the very low r with speed and negative values for r with error rate. The Work Sample tests are ideal for capturing applicants' ability to manage complaints yet cannot be used with any validity over other areas of their responsibilities. In conclusion these tests also need modification before they can be used "for keeps" as part of the overall methodology for hiring at the company.

3. What limitations in the above study should be kept in mind when interpreting the results and deciding whether to use the clerical test and work sample?

There are many limitations that need to be kept in mind in interpreting the study. The first and most significant is that the Clerical Test is specifically designed for reliability of test taking skills and not necessarily the effectiveness of the applicant across the entire breadth of responsibilities. This limitation is common in tests to evaluate employability and skills sets, as activity-based test show proficiency yet lack measures of validity when they become too myopic in nature (Heneman III, Judge & Kammeyer-Muller, 2012). This is the problem with the Clerical test; it is effective in measuring a very narrow set of skills yet must be redesigned to ensure a broader definition of reliability in determining employability.