Publication bias in industrial and organizational psychology journals

Kepes, S., McDaniel, M. (2013). How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology, 6: 252-268.

The purpose of the experiment conducted in the study by Kepes and McDaniel (2013) is to discern the extent of the credibility of the scientific literature published in academic journals that relates to industrial and organizational psychology. It is the hypothesis of the researchers that either the sector of psychology as a whole is now "omniscient" based on the perception that "most hypotheses in I-O psychology journals are confirmed" or these same academic/scholarly journals "are publishing an unrepresentative sample of completed research" (p. 252).

The method by which the researchers attempt to assess their experimental hypothesis is to use "evidence from publication bias analyses that estimates the extent to which the purported values are likely to be distorted" as they fail to represent the whole of research on a particular subject (p. 253). Thus, it is a literature review methodology. The researchers assess the structural problems within the scientific process by noting that studies in which "large magnitude effect sizes" are depicted tend to be those which published most frequently (p. 254). The research participants in the study are thus other studies that have been published over the years by I-O scholarly journals.

The measurements are based on qualitative analysis rather than on quantitative analysis. Thus the quality of the journal publications is the focus of this research rather than a numerical or statistical assessment. That qualitative data is gleaned from an intuitive, almost phenomenological approach, although it is never discussed as such. The immersive process by which the researchers engage with the data, however, is evident from their extensive use of citations and the breadth of their research in terms of supporting nearly every one of their claims with a resource and reference. The procedure is thus situated in an intuitive method that notes a certain phenomenon in the publishing of I-O literature and then seeks to understand the reasons for it. In this sense, the research is causative in that it attempts to examine the deeper explanations for why particular studies are published more than others and if the results are indicative of a wider or universal phenomenon or if they are particular to the field of I-O literature. Statistical evidence is offered, but it is not the finding of the researchers themselves but rather quantitative data found in the literature review that is used to support their overall thesis. For example, they cite a study in which more than half of a faculty department had known of a colleague who had altered findings in a study in order to have a "better" result for publication. This sort of evidence is used to make their own case more convincting -- which is that the world of scholarly journals and publications is not infallible and is not representative of the research field as a whole.

Based on their findings, the researchers offer recommendations as such as the creation of research registries, which act as databases (required) that all researchers must register worth prior to conducting their studies. The registry format exists in the medical field and could be used in the I-O field in order to better serve the process of credibility.

A second recommendation is for changes to be implemented in the editorial review process. A two-step process is suggested by the researchers: the first-step involves the submission of only half the manuscript (specifically, the introduction, methodology and analysis approach) while the rest is reserved so that reviewers can judge the study on the merits of its approach and intent rather than based on the "findings," which the researchers have shown can be altered in order to better facilitate publication.

A third recommendation is that null effect publications and replications be published. In doing so, the scientific process would be greatly stabilized and supported regardless of the lack of sensational findings or conclusions drawn.

My personal opinion is that while these recommendations would surely serve the research field in general and may be applicable in a universal sense, their utilization would also cause the publishing field to lament in terms of falling readership, as the poit of publishing is to cull readership, not lose it. What editors no doubt have realized is that in order to have readers, articles and studies must cover new ground rather than old ground. Thus, the aims of journals and of researchers are fundamentally at odds. The former is motivated not by scientific truth or necessity but by sensation and attention (that is what brings about higher circulation). The latter is meant to be driven by truth, but after the tradition of the journals may also be driven by the need for attention.