Best Practices in I-O Psychology Research Design

Introduction

Research practices depend on clearly defined guidelines. Those guidelines include general suggestions for how to conduct research effectively, how to apply research to clinical practice, and how to maximize research reliability and validity. The scientist-practitioner model has become the "framework for many training programs in clinical psychology" (Belar & Perry, 1992, p. 71). However, it is also important to pay attention to specific statistical analyses due to the potential for misinterpreting data. Cortina (1993) points out the significance of coefficient alpha, noting that proper interpretations of alpha enhance research validity and reliability. Alpha can often be misunderstood, particularly in the realm of scientist-practitioner and other types of applied research. It is not just misinterpretation of the alpha coefficient that can stymie research validity in the social sciences. Measurement errors, attenuation, and related biases can also impede research validity (Schmidt & Hunter, 1996).

Another core area of concern in applied psychology research is whether to use broad versus narrow constructs in research design, and the efficacy of core self-evaluations (Judge & Kammeyer-Muller, 2012). Depending on the area of applied research — such as intelligence testing or job skills testing — researchers can determine what type of construct to use. Yet research does not only guide practice; psychological research also informs theory. As Bacharach (1989) points out, researchers also need ground rules for how to interpret data results in ways that can reinforce existing theories, challenge them, or propose new theories for newly observed phenomena. A review of the literature on best practices in psychological research reveals ways to improve research design and its application, ensuring validity and reliability.

The Scientist-Practitioner Model in Applied Psychology

Belar and Perry (1992) present the findings of the National Conference on Scientist-Practitioner Education and Training for the Professional Practice of Psychology, addressing mainly the scientist-practitioner model in general, its relevance for applied psychology, and especially its merits for informing best practices in fields like human resources and other industrial-organizational (I-O) psychology fields. Exposing the core professional practice points, Belar and Perry (1992) do not necessarily focus on specific issues impacting research reliability and validity so much as they outline the merits of the scientist-practitioner framework and how it can be used and improved. Because many industrial-organizational psychologists rely on the scientist-practitioner model, it is important to keep the core nine points the authors outline in mind, while also paying close attention to research on specific issues — such as those related to statistical analysis.

Statistical Validity: Coefficient Alpha and Measurement Error

Cortina (1993) and Schmidt and Hunter (1996) reflect on specific statistical and analytical biases that can impede psychology research, adversely affecting the reliability of results. Researchers need to pay close attention to every element of research design, including how research is framed and how results are interpreted depending on the type of data analysis being used. A large number of psychology studies rely on the use of the alpha coefficient for factors like standard deviations in order to determine reliability and validity. Unfortunately, there are many ways to use alpha, and the researcher also determines the alpha coefficient value — a researcher-determined value that could give rise to researcher bias.

Cortina (1993) also offers insight into different alpha constructs, such as Cronbach's coefficient alpha, which is commonly used in psychological metrics. By presenting the results of different hypothetical tests that manipulate alpha, Cortina (1993) illustrates how the way alpha is used has a direct bearing on how results will be interpreted and applied. For scientist-practitioners, this information is especially important because the results of a study could have a direct impact on the lives of subjects, raising pertinent ethical questions. Schmidt and Hunter (1996) focus on another type of statistical issue: measurement validity. Researchers need to know how to correct for measurement biases, and the authors therefore offer methods by which to correct for problems in the survey instrument itself. Taken together, the Schmidt and Hunter (1996) and Cortina (1993) articles are extremely useful for research psychologists with an interest in applied psychology such as I-O.

Conclusion

All together, this review of the literature shows how scientist-practitioners in the field of I-O psychology can improve their research competencies. By attending to the scientist-practitioner framework, carefully interpreting statistical measures such as coefficient alpha, correcting for measurement error, and thoughtfully selecting constructs grounded in sound theory, I-O psychologists can enhance the validity, reliability, and ethical integrity of their research and practice.

References

Bacharach, S. B. (1989). Organizational theories: Some criteria for evaluation. Academy of Management Review, 14(4), 496–515.

Belar, C. D., & Perry, N. W. (1992). National conference on scientist-practitioner education and training for the professional practice of psychology. American Psychologist, 47(1), 71–75.

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104.

Judge, T. A., & Kammeyer-Muller, J. D. (2012). General and specific measures in organizational behavior research: Considerations, examples, and recommendations for researchers. Journal of Organizational Behavior, 33(2), 161–174.

Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1(2), 199–223.