Validity and Reliability in Feedback Interaction Research
This paper critically evaluates the validity and reliability of Harrison and Rouse's (2015) inductive study of feedback interactions over the course of creative projects, published in the Academy of Management Journal. The review focuses on two primary threats: the subjective nature of the study's sampling logic, which relied on researcher judgment to identify "relevant" creative contexts, and the acknowledged gaps in the methods section that introduce variability into the findings. The paper argues that greater precision in defining comparable incidents, documenting hierarchical relationships, and minimizing subjective sampling decisions would strengthen the study's replicability and overall scientific rigor.
- Introduction: Purpose and scope of the validity review
- Sampling and Its Threats to Validity and Reliability: Subjective sampling logic undermines replicability and validity
- Methodological Gaps and Reliability Concerns: Acknowledged gaps and undefined terms weaken reliability
- Conclusion: Authors partially controlled threats but could improve
✍️ How to write this paper — guide, tools & examples ▾
What makes this paper effective
- The paper identifies concrete, specific threats to validity and reliability rather than making vague general criticisms, anchoring each concern to a direct quotation from the reviewed article.
- It connects methodological critique to practical remedies — for each identified gap, the author suggests what researchers could do differently in future studies.
- The discussion of subjective sampling is well-developed, using the modern dance example to illustrate how opinion-based framing can undermine replicability in a tangible way.
Key academic technique demonstrated
The paper demonstrates critical source evaluation: using direct quotations from a peer-reviewed article as evidence for methodological critique. Rather than summarizing what the article claims, the author interrogates the article's own language — phrases like "overriding sampling logic" and "comparable incidents" — and explains why that language signals reliability or validity concerns. This technique is central to research methods coursework and literature review writing.
Structure breakdown
The paper opens with a brief statement of purpose, moves into an extended critique of the article's sampling strategy (the longest and most developed section), then addresses gaps in the methods section more briefly, and closes with a short conclusion that references the article's own "Limitations" section. The structure mirrors a standard article review: purpose, analysis, synthesis, and evaluation.
Introduction
This report reviews a peer-reviewed journal article with the specific purpose of evaluating it for implications regarding validity and reliability. Where gaps exist in either dimension, this review identifies them and suggests what could be done to avoid such issues in future or different studies. A recent study about feedback interactions by Harrison and Rouse (2015) revealed several gaps that could constitute threats to both reliability and validity.
Sampling and Its Threats to Validity and Reliability
To evaluate the validity and reliability of the selected peer-reviewed article, it is useful to begin with the study's sampling approach. As noted in the applicable section, the authors state that "our overriding sampling logic was to find contexts that had shown a history of successfully using feedback in creative work, enabling us to see feedback in action frequently and transparently" (Harrison & Rouse, 2015). That sounds reasonable on its surface, but it is methodologically perilous when it comes to validity and reliability.
First, what different researchers deem "relevant" and "proper" for such a study tends to be subjective and, therefore, potentially quite different from one researcher to the next. This makes the possibility of replication and reliability unlikely, because the raw data used would tend to differ as well. Similarly, validity is brought into question because the conclusions drawn — and the reasons those conclusions are drawn — all flow from the data. There must be a meaningful absence of subjectivity in the sampling process. Obviously, looking at data that is inapplicable will not help. However, selectively choosing what is and is not assessed is a methodological risk. A study that cherry-picks its context based on subjective impressions of "creativity" cannot easily be replicated.
A subsection of the sampling section discusses modern dance, and the very first sentence in that section characterizes modern dance as having set a "high bar for creativity." That is an opinion. It may be a prevailing opinion and may hold up across many studies, but it is not an empirically established fact common to all research contexts. Rather than relying on such subjective framing, the authors should have focused on the specific interactions they were studying. They also should have been precise about what types of interactions were occurring and who was involved — including the rank and relative power of the individuals, the performers, and others in the creative hierarchy. Such precision allows a study to be replicated, which is how reliability and validity can be meaningfully assessed across different research teams and settings.
Conclusion
The "Limitations and Opportunities for Future Research" section of Harrison and Rouse (2015) admits that there is a considerable amount of complexity in the interactions and feedback instances being analyzed and recorded. As a result, any results and conclusions must be understood in light of the scope, depth, and breadth of what was analyzed and what was specifically being examined. The authors did a reasonable job of acknowledging and partially controlling for these sources of variability, but greater methodological rigor in sampling definition and incident classification would have — and in future studies should — strengthen both the reliability and validity of this line of organizational behavior research.
References
Harrison, S. H., & Rouse, E. D. (2015). An inductive study of feedback interactions over the course of creative projects. Academy of Management Journal, 58(2), 375–404. https://doi.org/10.5465/amj.2012.0737
Create your account
Always verify citation format against your institution’s current style guide requirements.