Reliability and Validity in Psychological Testing

Introduction

In any academic and professional testing context, it is important to obtain at least some degree of reliability and validity. Without these qualities, tests cannot produce results that are consistent or usable in an academic setting, since they cannot be verified in terms of repeatability or compared meaningfully against other results. In psychological testing, which is more often studied by qualitative rather than quantitative means, it is often difficult to establish reliability and validity because the specific numerical benchmarks required to do so are frequently lacking. Nevertheless, there are methods available to ensure an optimal level of both properties in this kind of testing.

Reliability in Psychological Testing

According to Kline (2013, p. 7), reliability comes in two distinct forms: reliability in terms of consistency over time, and reliability in terms of internal consistency. Consistency over time is determined by administering tests to the same individuals on more than one occasion. When the results are the same or similar across administrations, reliability is considered high. This is also known as an instrument's test-retest reliability. Internal consistency reliability, on the other hand, concerns the relationship of test items to each other. The danger with this type of reliability is that test items may simply be paraphrases of one another, which negatively affects the validity of the instrument (Kline, 2013, p. 13).

The importance of the reliability factor in test instruments is illustrated by a study conducted by Narrow et al. (2013). In that study, the level of reliability for several populations was measured when administering test items for cross-cutting symptoms using the DSM-5 measurement instrument. The authors found the test-retest reliability of the instrument items to be good to excellent. In terms of the populations interviewed, parents proved reliable in reporting symptoms in their children. Child respondents, however, showed less uniformly reliable responses, while clinicians were able to rate psychosis reliably but not clinical domains related to psychosis in children or suicide across age groups.

Validity in Psychological Testing

The validity of a psychometric test is considered high when the test demonstrably measures what it was intended to measure. According to Kline (2013, p. 17), many psychometric tests have surprisingly low validity, which reflects the complicated nature of both the construct and its measurement. In psychological testing, validity is often difficult to assess because of the fluid nature of the concept itself; there is no singular measure or statistic against which a test can be evaluated for validity. For those attempting to determine the validity level of a psychometric test, several distinct types of validity can be applied.

Face validity, for example, refers to the appearance of a test in terms of whether it seems to measure what it intends to measure (Kline, 2013, p. 18). Concurrent validity refers to the correlation of the test with other instruments measuring the same variable at the same time. Construct validity is closely related, referring to the correlation of the test results with other established tests that are administered on a regular basis.

Conclusion

In conclusion, the importance of validity and reliability in psychometric testing depends largely on the type of test being administered. In most cases, reliability is of primary importance, while somewhat low validity is often an acceptable outcome — particularly for newly developed instruments. One might therefore conclude that reliability is the more critical property, though only by a relatively narrow margin.

References

Kline, P. (2013). Handbook of Psychological Testing. New York: Routledge.

Narrow, W. E., Clarke, D. E., Kuramoto, S. J., Kraemer, H. C., Kupfer, D. J., Greiner, L., & Regier, D. A. (2013, January). DSM-5 field trials in the United States and Canada, Part III: Development and reliability testing of a cross-cutting symptom assessment for DSM-5. The American Journal of Psychiatry, 170(1).

Raz, S., Bar-Haim, Y., Sadeh, A., & Dan, O. (2014). Reliability and validity of online continuous performance test among young adults. Assessment, 21(1).

Salzer, M. S., & Brusilovsky, E. (2014, April). Advancing recovery science: Reliability and validity properties of the Recovery Assessment Scale. Psychiatric Services, 65(4).