Evaluating the Reliability and Validity of the Results

¶ … Reliability and Validity of the Results

Accuracy

Precision

Validity

Reliability

Analyzing Internal Validity of the Research Design

Maturation

Testing

Statistical Regression

Selection

Experimental Mortality

Analyzing External Validity of the Research Design

Interaction

Pretesting

Multiple Treatments or Interventions

Recommendation of a Better Design

Evaluate the Reliability and Validity of the Results

In the Washington Post (2015), an article titled "How was Sexual Assault Measured," by Scott Clement, there are many factors needed to be evaluated in order to better test the true accuracy, precision, validity and reliability of the survey presented. This particular post outlines the research survey undertaken by Kaiser Family Foundation, which assessed the extent and prevalence of sexual assault. The results of the study brought about the conclusion that 20% of the prevailing and recent female students in college, that subside within the campus or near it have reported of being sexually assaulted during this time as they attend school (Clement). The following section will make an attempt to evaluate and assess the reliability and validity of these results. Subsequently, it will point out a number of flaws and faults perceived in the research itself.

Accuracy

Firstly, let us examine the accuracy of this article by seeking the degree to which the measurement used in the survey represents the true value of what it is the surveyor is looking for. In the case of sexual assault, the topic is ambiguous, and according to the article, "measuring the prevalence of sexual assault is a tricky task in surveys for two reasons." The first being that asking about sexual assault plainly could result in producing unreliable results because the definition of what sexual assault entails may be relative, and not a common belief among all individuals. The second is that sexual assault is a highly sensitive topic and respondents may not be willing to report their true responses. It can be argued that the telephone interviews and survey were undertaken because access to the participants was minimal. However, this in itself becomes a flaw in the article. It is imperative to note that accuracy is sensitive to change, particularly with regard to detail, such as the dates, people present and the like. The fact that the interviews were undertaken through telephone makes it difficult to ascertain the information being given by the participants. The survey undertaken makes the mistake of making the assumption of accepting that the information being given is reliable and accurate. It is important to always question implicitly the responses to the questions being asked and look for indications of deception or self-deception by the participants. In this case, this was not possible. This is because telephone interviews make it difficult to look at the participants and make a determination as to whether they are telling the truth or they have an indication of deception. Solely listening to the voice of the participants to make this determination indicates that the information obtained is not accurate.

Precision

Taking into consideration the aforementioned aspects, we must also examine the precision aspect of what it is we are seeking, which is basically the degree to which the results of this study will resemble those of other studies under similar circumstances. In the article, it states that "the post-Kaiser survey found that 20% current and recent female students report being assaulted by force or while incapacities, compared with the 13.7% in the 2007 survey among current college students only." In this case of comparing these two surveys, we would not be able to assume that there is a random error because these surveys may have been perceived completely differently, even though their goal may have been very common. There is no certainty that the questions have been asked in the same exact manner and fashion. A lack of precision would imply a random error, but in this case, unless the questions were identically stated, we cannot determine if one exists, or does it state in the article if there were any precision issues between the individual surveys.

Validity

In definition, validity seeks to make a determination as to whether the research truly gives a measure of what it was intended and purposed to measure or also how truthful the results of the research are. In other words, this is to question whether the research instrument used enables the researcher to hit the mark of the research object. When it comes to validity, we must make sure that the study itself measures exactly what it is intended to measure. In the article, the terminology of sexual assault is broken down specifically into 5 different types of assault, (1) forced sexual touching, (2) oral sex, (3) sexual intercourse, (4) anal penetration and (5) sexual penetration with a finger or object. This way, the term itself is not as subjective, nor ambiguous. Respondents have a better understanding of what it is they are asked, and surveyors have a better way of analyzing their data (Creswell and Miller, 125).

Reliability

Lastly, reliability also plays a key role. Reliability can be defined as the magnitude to which the results obtained are consistent over time and have an accurate or precise representation of the total population under the research study. More so, if the results and outcomes attained in the study can be reproduced yet again under a similar methodology, then the instrument of the research is considered to be reliable. It seeks to discover how one will get the same values if the measurement is repeated. Taking all factors constant such as trauma, fear, and other aspects, the participants can be expected to provide dissimilar results from the response of the same questions if an identical methodology is undertaken. There are some flaws that can be perceived, in particular, with regard to this article (Golafshani, 602).

Three elements are considered when assessing this aspect. The first one is the magnitude to which the measurement, given incessantly, remains the same. The second one is the stability of the measurement over time, and third of all, the similarity of measurements within a given time period. It is imperative to note that consistency is the main measure of reliability. One of the main flaws with regard to the article, concerning the article by Clement is the aspect of proximity to the events. The survey that was undertaken by Post-Kaiser did not take this aspect into consideration. This is imperative for the reason that events that took place a long time ago could be changed and not given an accurate and precise description of what happened during the occasion. Therefore, this questions the reliability of the study (Golafshani, 600).

In overall, there are a number of elements that can be perceived to have flaws within the research article. Some of these flaws include the use of telephone interviews rather than face-to-face interviews. This brings a limitation in determining whether the responses given by the participants have any form of deception. Analyzing responses through the telephone is quite difficult and the participants can easily give false or altered answers. Another important element is assessing the time period from the event. This is because events that took place far long from the time of the interview do not give a precise indication and are bound to be changed by the participants. In addition, sensitivity is another element as sexual assault is a highly sensitive topic and respondents may not be willing to report their true responses (Kirk and Miller, 52).

Part 2

The following section will analyze the internal validity and external validity of the case study "Marketing Analysts and promotional Specialists, Inc." Subsequently, a recommendation for a better research design will be given to take into account any flaws that might arise.

Analyzing Internal Validity of the Research Design

In accordance to McDaniel et al. (Chapter 10), internal validity is defined as the magnitude to which the contending explanations and justifications for the experimental results witnessed can be dismissed and not taken into consideration. In other words, internal validity seeks to ascertain how well one can make the ascertainment that the Y variable, which is the dependent variable, is caused by the X variable, which is the independent variable, and not other factors. By critically evaluating and assessing the research design, it is imperative to establish the dependent variable and the independent variables. The dependent variable, Y, is the sales of the different beers included in the research experiment. On the other hand, the independent variable is the placement of the products on the shelf, either in the fridge or along the aisles. The degree of control exercised over prospective extraneous variables decides the level of internal validity. When undertaking an evaluation of internal validity of a study, the main consideration is whether the conclusions or outcomes follow the data as well as the procedures used.

For starters, with regard to the research in the case study being analyzed, there is treatment effect and therefore there is a cause-and-effect relationship to assess. As is indicated, the sales increased in treatment 2 by 5% as a result of the change in the allocation of where the Dixie beer was placed. It was well documented that the cold beer be placed at the far left side of the fridge, and the warm beer be placed in the middle of the isle since consumers may want to "shop" or walk around before making a selection. Therefore, the internal validity can be assessed since the statistical conclusion is valid (Winter).

The internal validity of the research design will be established through the analysis of the following factors:

History

This factor takes into account whether some unexpected events took place while the experiment was under way and whether such events had an influence on the dependent variable. It is imperative to note that history is a threat for a one group design. With regard to the one group pre-posttest design, the impact of the treatment is the variance in the pre-test and post-test results or outcomes. This dissimilarity might be as a result of the treatment or history. Therefore, in this particular case, history as a factor does not impact the internal validity. This is because this particular research study is a treatment design, which is a two group design that makes a comparison between the treatment group and the control group. The history threat does not take place and therefore, does not have any adverse impact on the internal validity of the research design (Isaac and Michael, 84).

Maturation

These are natural, instead of experiment enforced changes that take place as a result of the regular passage of time. This takes into consideration whether the changes in the dependent variable are as a result of ordinary developmental procedures operating within the participants as a function of time. For instance, with more time passing by in a research study, there is a greater likelihood that the participants will come to be fatigued, uninterested and less motivated. However, this particular factor is only a threat to the one group and not the two group design. Bearing in mind that the research experiment in this case study is a two group design, implies that this does not adversely influence the internal validity of the research (Isaac and Michael, 84).

Testing

In several experiments, the participants are pretested to establish whether all of them are beginning the research study at roughly the same level. The downside to this is that this can alter the performance of the participants during subsequent tests, for example, those employed as dependent variables, which measure the similar field beyond any impacts instigated by the treatment itself. In this particular case, there is no indication that M.A.P.S pretested any of the participants in the study. Therefore, this does not adversely influence the internal validity of the research study.

Instrumentation

This encompasses the changing or alteration of the methods of measurements or how they are administered during the course of a research and how it influences what is measured. In addition, if the research study employs human observers, then it might be the judgment and conclusion of the observer that alter over time, instead of the performance of the participants. In this particular case, the researchers did not alter the methods of measurement. Throughout both treatment 1 and treatment 2, M.A.P.S employed the percentage of beer purchases for every brand as the method in the study. Therefore, this factor does not serve as a threat or adversely impact the internal validity of the research study.

Statistical Regression

This particular threat comes about when the participants in a research study are chosen and selected to take part for the reason that they had an extremely low or extremely high score with regard to some measure of performance. As a result, retesting of the participants will more or less always yield a dissimilar distribution of scores, and the mean for this new distribution will be almost similar to that of the population. For instance, if the selected subjects all had high scores originally, the group's mean on the retest will be lower compared to how it was initially. On the other hand, if the group's average was initially low, their retest average would be greater. In this case, the initial scores during treatment 1 were not extreme on either ends. Since there was no retest of the participants, treatment 2 generated a similar distribution of percentage scores with minimal variance (Slack and Draugalis).

Selection

The participants in comparison, for example, between the control group and treatment group, ought to be functionally similar at the start of a study. This is largely for the reason that if they actually are, then observed dissimilarities between the groups, as measured by the performance dependent variable, at the culmination of the research study, are more probable to be instigated only by the independent variable rather than other organismic variables. It is imperative to note that if the control group and treatment group are dissimilar from one another at the start of the study, then the results that are obtained are biased. In this particular case study, there is quite some ambiguity with regard to this aspect. In particular, there is no outlining whether there was a similarity or difference between the participants in the control group and in the treatment group. Therefore, this particular factor could be perceived as a threat to the internal validity of the research study (Isaac and Michael, 76).

Experimental Mortality

Experimental mortality comes about if and when the participants withdraw from the research study in question. It is imperative to note that in case one comparison group, that is either the control group or treatment group, experiences a greater level of withdrawal or mortality of the participants compared to the other, then the observed difference between the groups end up being questionable. In this particular case study, M.A.P.S does not give any indication whether there was any issue in participants dropping out. Therefore, it has to be considered that this particular aspect would not act as a threat to the internal validity of the case study (McDermott, 30).

Analyzing External Validity of the Research Design

On the other hand, in accordance to McDaniel et al. (Chapter 10), external validity is defined as the magnitude to which the underlying relationships that are measured in an experiment can be given generalizations to external individuals, settings and timings. For instance, it is deemed that field experiments have greater external validity. Since internal validity has been established, it implies that there is treatment effect to generalize and therefore external validity can be established. According to Blanche et al. (165), a study is considered to have external validity when its conclusions or the findings attained may be universal beyond the limits of the design and the study setting. When undertaking an evaluation of external validity of a study, the main consideration is the degree or magnitude to which the conclusions can be extrapolated beyond the context of the study (Blanche et al., 166).

The external validity of the research design will be established through the analysis of the following factors:

Interaction

One of the factors that can negatively hinder the external validity of a study is whether an interaction amid the participants had been selected as well as the treatment. In this particular case, the participants in the both studies had been randomly chosen from the population. As a result, taking this into consideration, it can be deemed that the demographic individualities and features could not have had any bias on their performance. For this reason, the results of the research study are appropriate to this population as well as other groups, which more precisely signifies population characteristics. This factor does not adversely affect the external validity of the research (Bracht and Gene, 440).

Pretesting

The aspect of pre-testing the participants in a research study might instigate them to respond more or less intensely to the treatment in comparison to how they might have reacted if they had not experienced or gone through any pre-test. In this particular research study, there was no pretesting of participants for treatment 1 and treatment 2. Therefore, it can easily be concluded that the other random population members might perform within an identical and similar manner as the participants did within this study. Therefore, with no pretesting done, this factor could does not adversely impact or influence the external validity of the research study (Bracht and Gene, 442).

Setting

The participants' performance in some research studies has been more of an outcome or response to the setting of the experiment, for instance, where the research study is being carried out, than it is in reacting with the independent determinant. Case in point, the participants who know and are aware that they are taking part in a research study or are cognizant that they are being observed, might respond in a different way to the treatment being carried out than a participant who underwent the treatment; however, had been uninformed that he was being monitored. In this case, there is no indication that the participants were reacting due to the setting of the study or the store where the research was being conducted. Therefore, this factor did not impact the external validity of the research (Slack and Jolaine, 2176).