Limitations in research methods and their effects on study results

Introduction
In any academic study, there are inevitably going to be limitations to the study that have to be taken into account when evaluating the results of the study. The authors are supposed to build these limitations into their conclusions, but if they do not then the reader must. Therefore, it is important to understand what the limitations of any given study are, and how they might have influenced the results.
Britz & Dunn
Britz & Dunn (2009) studied the relationship between self-care and quality of life. The authors begin with the hypothesis that decreased quality of life among patients with heart failure arise from self-care deficits. In other words, if people don't look after themselves the don't do as well. So, rocket science, but okay.
The limitation of this study are listed by the authors as having a small sample size and the potential homogeneity of the sample. These two limitations can influence the ability of a researcher to extrapolate these results across a wider population. A small samples size can reduce the reliability of the results. Homogeneity of the subjects makes the findings less reliable across a wider population, because in that case the sample is not representative of the entire population. The sample can still be representative of a population, just not the broadest segment; it would still be representative of members of that particular in-group.
The authors posit that there is another limitation. However, they are poor writers and cannot actually explain in clear language what this limitation is. Terms like "low reliabilities" and "subscale alphas" are used to describe the limitation, but those words have no meaning unless given context, which they are not. Extra credit for the five dollar word "coefficient" and for referencing a Greek letter. Writing rambling gobbledegook is less a limitation of the study than of the authors themselves, who with such purple prose cast doubt as to their knowledge of the subject matter. If they knew what they were talking about, they would be able to express their ideas clearly and simply, rather than hiding behind a blizzard of academic buzzwords.
The study was conducted with a convenience sample from a single hospital. They made a point to exclude patients from facilities where they receive care, to isolate patients who were responsible for self-care. The patients were interviewed in order to fulfill the study. The interviews were then subject to evaluation by the researchers. The qualitative inputs were pumped into SPSS, yet interestingly the authors don't bother to explain how they coded the qualitative answers to fit into statistical software. Each answer would have to have been converted into a number to be subject to statistical analysis, yet there is nothing in the paper that describes this process.
The data analysis procedure is thus missing a fairly important gap. If the researchers have interviews, and then they skip directly to an explanation of the statistics that they got from SPSS, that leaves out a rather important step. While we're on the subject of limitations, this isn't necessarily a limitation in the study but it is one in the paper because it puts the reader in a position of not being able to verify the results. A sound paper has to be vetted by people other than the authors – the reader has to be able to read the paper and know exactly how the authors reached their conclusions. The omission of the how the authors translated interviews into statistics did not go unnoticed.
Tang et al
Tang et al (2014) examine the role of depression in medication adherence. The basic principle is that medication adherence is correlated with health outcomes for heart failure patients. Thus, it is worth investigating potential risk factors for lack of medication adherence. Interestingly, the hypothesis of the study was that depression is associated with lower self-reported medication adherence than objectively measured medication adherence. Whatever the merits of that study might be – what people report doesn't matter in terms of health outcomes, what matters is what they do. Barking up the wrong tree aside, let's take a look at the limitations of the study.
The authors argue that one limitation of the study was the exclusion of severely depressed individuals. They offered up a pretty flimsy excuse for this, which is weird, because there's a good one to be had – that there is a significant difference between the severely depressed and more mild forms of depression. Some examination of the research would probably show that there are differences between different levels of depression that can affect things like taking medication – it's hard to imagine why the authors passed up the opportunity to look up a few papers an instead offered up a weak excuse for their decision to exclude that particular group from the study.
They also noted a further limitation, in the lack of qualitative data to inform the condition of medication adherence; they note "why" someone chose to adhere and "how" they did so. Ok, two things. One, they could have asked these questions if they wanted to know the answers. Two, if these questions weren't asked then they are not part of the study. The study is about something else, and these questions represent opportunities for future research. Arguably, however, the why is actually pretty important. Without the why, the authors are only testing for correlation, not causation. Given that they are trying to investigate this link, one has no choice but to ask why the authors stopped short and only tried to examine correlation. It would be better to list as a limitation that they tried as find out "why" and simply were unable to get conclusive answers. To not ask is a fault in the research design that will have to be remedied by other scholars in the future. The authors actually recommend this, de facto admitting this as an error.
The authors used a prior study – information gathered from that study, but excluding patients based on different criteria. The authors do a good job of explaining why certain patients were excluded from the study. They note that the used the Basel Assessment of Adherence Scale as their means of translating qualitative answers into quantitative in order to input the responses into statistical software. By outlining the means by which they evaluated the answers, they help the reader to be able to evaluate the quality of the study; this is the desired level of transparency.
The authors use SPSS to run their stats, and they explain the methods by which they calculated the results. This again provides a level of transparency.
Other Aspects
In both studies there was adherence to norms with respect to human rights. This is one of the benefits of excluding patients with severe depression, for example, but ultimately there were no issues with either study with respect to the treatment of the participants, all of whom agreed to partake of the study.
Assumptions in the two studies were fairly benign. The Tang study had its flaws, but the baseline assumption is that it is important to test for depression in patients with heart failure, because depression will impact on the health outcomes of those patients. Being aware that a patient suffers from depression will help medical practitioners to give that person better care – for example more stringent oversight to ensure that they are take the medications. So the underlying assumption is valid, and important to health care practice.
The Britz study is useful as well, though it is rather stating the obvious. Still, it is worth testing to see if the obvious actually holds true in the real world, so in that respect the underlying assumption was tested, and confirmed what a reasonable person would have suspected – that a person's ability to self-care will affect their health outcomes.
On the issue of threats to validity, the Tang study excluded the severely depressed as a means of isolating a particular patient group, or at least that was the outcome, and this made the study more valid when taken in its particular context. The Britz study shouldn't have any real threats to its validity as its intent was to confirm what any reasonable person would already know – that people who are healthier and pretty sure they can take care of themselves in fact do have better outcomes than unhealthy people who aren't sure if they can take care of themselves. Validity isn't the issue that study, relevance is.
Conclusions
The studies both explore variables that relate to health outcomes for people with heart failure. The Britz study is pointless – small sample size, obvious premise and sloppy writing undermine the study's value significantly, despite its apparent validity. Apparent being the correct term when one cannot actually verify how the qualitative responses were codified for use in SPSS.
The Tang study has its own weaknesses, but again validity is less of one than certain issues with the construction of the study. The methodology is explained clearly, and if there are thoughts that they could have tested other things, well, they didn't and the only thing left to do is just to accept what they did test and hope that the more valuable aspects of the study are taken up by somebody else in the future.





References

Britz, J. & Dunn, K. (2009) Self-care and quality of life among patients with heart failure. Journal of the American Academy of Nursing Practitioners. Vol. 22 (2010) 480-487.

Tang, H., Sayers, S., Weissinger, G. & Riegel, B. (2014) The role of depression in medication adherence among heart failure patients. Clinical Nursing Research. Vol. 23 (3) 231-244.