Evaluating the Kaiser Family Foundation sexual assault survey's reliability and validity

¶ … Reliability and Validity of the Results

Accuracy

Precision

Validity

Reliability

Analyzing Internal Validity of the Research Design

Maturation

Testing

Statistical Regression

Selection

Experimental Mortality

Analyzing External Validity of the Research Design

Interaction

Pretesting

Multiple Treatments or Interventions

Recommendation of a Better Design

Evaluate the Reliability and Validity of the Results

In the Washington Post (2015), an article titled "How was Sexual Assault Measured," by Scott Clement, there are many factors needed to be evaluated in order to better test the true accuracy, precision, validity and reliability of the survey presented. This particular post outlines the research survey undertaken by Kaiser Family Foundation, which assessed the extent and prevalence of sexual assault. The results of the study brought about the conclusion that 20% of the prevailing and recent female students in college, that subside within the campus or near it have reported of being sexually assaulted during this time as they attend school (Clement). The following section will make an attempt to evaluate and assess the reliability and validity of these results. Subsequently, it will point out a number of flaws and faults perceived in the research itself.

Accuracy

Firstly, let us examine the accuracy of this article by seeking the degree to which the measurement used in the survey represents the true value of what it is the surveyor is looking for. In the case of sexual assault, the topic is ambiguous, and according to the article, "measuring the prevalence of sexual assault is a tricky task in surveys for two reasons." The first being that asking about sexual assault plainly could result in producing unreliable results because the definition of what sexual assault entails may be relative, and not a common belief among all individuals. The second is that sexual assault is a highly sensitive topic and respondents may not be willing to report their true responses. It can be argued that the telephone interviews and survey were undertaken because access to the participants was minimal. However, this in itself becomes a flaw in the article. It is imperative to note that accuracy is sensitive to change, particularly with regard to detail, such as the dates, people present and the like. The fact that the interviews were undertaken through telephone makes it difficult to ascertain the information being given by the participants. The survey undertaken makes the mistake of making the assumption of accepting that the information being given is reliable and accurate. It is important to always question implicitly the responses to the questions being asked and look for indications of deception or self-deception by the participants. In this case, this was not possible. This is because telephone interviews make it difficult to look at the participants and make a determination as to whether they are telling the truth or they have an indication of deception. Solely listening to the voice of the participants to make this determination indicates that the information obtained is not accurate.

Precision

Taking into consideration the aforementioned aspects, we must also examine the precision aspect of what it is we are seeking, which is basically the degree to which the results of this study will resemble those of other studies under similar circumstances. In the article, it states that "the post-Kaiser survey found that 20% current and recent female students report being assaulted by force or while incapacities, compared with the 13.7% in the 2007 survey among current college students only." In this case of comparing these two surveys, we would not be able to assume that there is a random error because these surveys may have been perceived completely differently, even though their goal may have been very common. There is no certainty that the questions have been asked in the same exact manner and fashion. A lack of precision would imply a random error, but in this case, unless the questions were identically stated, we cannot determine if one exists, or does it state in the article if there were any precision issues between the individual surveys.

Validity

In definition, validity seeks to make a determination as to whether the research truly gives a measure of what it was intended and purposed to measure or also how truthful the results of the research are. In other words, this is to question whether the research instrument used enables the researcher to hit the mark of the research object. When it comes to validity, we must make sure that the study itself measures exactly what it is intended to measure. In the article, the terminology of sexual assault is broken down specifically into 5 different types of assault, (1) forced sexual touching, (2) oral sex, (3) sexual intercourse, (4) anal penetration and (5) sexual penetration with a finger or object. This way, the term itself is not as subjective, nor ambiguous. Respondents have a better understanding of what it is they are asked, and surveyors have a better way of analyzing their data (Creswell and Miller, 125).

Part 2

The following section will analyze the internal validity and external validity of the case study "Marketing Analysts and promotional Specialists, Inc." Subsequently, a recommendation for a better research design will be given to take into account any flaws that might arise.

Analyzing Internal Validity of the Research Design

In accordance to McDaniel et al. (Chapter 10), internal validity is defined as the magnitude to which the contending explanations and justifications for the experimental results witnessed can be dismissed and not taken into consideration. In other words, internal validity seeks to ascertain how well one can make the ascertainment that the Y variable, which is the dependent variable, is caused by the X variable, which is the independent variable, and not other factors. By critically evaluating and assessing the research design, it is imperative to establish the dependent variable and the independent variables. The dependent variable, Y, is the sales of the different beers included in the research experiment. On the other hand, the independent variable is the placement of the products on the shelf, either in the fridge or along the aisles. The degree of control exercised over prospective extraneous variables decides the level of internal validity. When undertaking an evaluation of internal validity of a study, the main consideration is whether the conclusions or outcomes follow the data as well as the procedures used.

For starters, with regard to the research in the case study being analyzed, there is treatment effect and therefore there is a cause-and-effect relationship to assess. As is indicated, the sales increased in treatment 2 by 5% as a result of the change in the allocation of where the Dixie beer was placed. It was well documented that the cold beer be placed at the far left side of the fridge, and the warm beer be placed in the middle of the isle since consumers may want to "shop" or walk around before making a selection. Therefore, the internal validity can be assessed since the statistical conclusion is valid (Winter).

The internal validity of the research design will be established through the analysis of the following factors:

History

This factor takes into account whether some unexpected events took place while the experiment was under way and whether such events had an influence on the dependent variable. It is imperative to note that history is a threat for a one group design. With regard to the one group pre-posttest design, the impact of the treatment is the variance in the pre-test and post-test results or outcomes. This dissimilarity might be as a result of the treatment or history. Therefore, in this particular case, history as a factor does not impact the internal validity. This is because this particular research study is a treatment design, which is a two group design that makes a comparison between the treatment group and the control group. The history threat does not take place and therefore, does not have any adverse impact on the internal validity of the research design (Isaac and Michael, 84).

Maturation

These are natural, instead of experiment enforced changes that take place as a result of the regular passage of time. This takes into consideration whether the changes in the dependent variable are as a result of ordinary developmental procedures operating within the participants as a function of time. For instance, with more time passing by in a research study, there is a greater likelihood that the participants will come to be fatigued, uninterested and less motivated. However, this particular factor is only a threat to the one group and not the two group design. Bearing in mind that the research experiment in this case study is a two group design, implies that this does not adversely influence the internal validity of the research (Isaac and Michael, 84).

Testing

In several experiments, the participants are pretested to establish whether all of them are beginning the research study at roughly the same level. The downside to this is that this can alter the performance of the participants during subsequent tests, for example, those employed as dependent variables, which measure the similar field beyond any impacts instigated by the treatment itself. In this particular case, there is no indication that M.A.P.S pretested any of the participants in the study. Therefore, this does not adversely influence the internal validity of the research study.

Instrumentation

This encompasses the changing or alteration of the methods of measurements or how they are administered during the course of a research and how it influences what is measured. In addition, if the research study employs human observers, then it might be the judgment and conclusion of the observer that alter over time, instead of the performance of the participants. In this particular case, the researchers did not alter the methods of measurement. Throughout both treatment 1 and treatment 2, M.A.P.S employed the percentage of beer purchases for every brand as the method in the study. Therefore, this factor does not serve as a threat or adversely impact the internal validity of the research study.

Statistical Regression

This particular threat comes about when the participants in a research study are chosen and selected to take part for the reason that they had an extremely low or extremely high score with regard to some measure of performance. As a result, retesting of the participants will more or less always yield a dissimilar distribution of scores, and the mean for this new distribution will be almost similar to that of the population. For instance, if the selected subjects all had high scores originally, the group's mean on the retest will be lower compared to how it was initially. On the other hand, if the group's average was initially low, their retest average would be greater. In this case, the initial scores during treatment 1 were not extreme on either ends. Since there was no retest of the participants, treatment 2 generated a similar distribution of percentage scores with minimal variance (Slack and Draugalis).

Selection

The participants in comparison, for example, between the control group and treatment group, ought to be functionally similar at the start of a study. This is largely for the reason that if they actually are, then observed dissimilarities between the groups, as measured by the performance dependent variable, at the culmination of the research study, are more probable to be instigated only by the independent variable rather than other organismic variables. It is imperative to note that if the control group and treatment group are dissimilar from one another at the start of the study, then the results that are obtained are biased. In this particular case study, there is quite some ambiguity with regard to this aspect. In particular, there is no outlining whether there was a similarity or difference between the participants in the control group and in the treatment group. Therefore, this particular factor could be perceived as a threat to the internal validity of the research study (Isaac and Michael, 76).

Experimental Mortality

Experimental mortality comes about if and when the participants withdraw from the research study in question. It is imperative to note that in case one comparison group, that is either the control group or treatment group, experiences a greater level of withdrawal or mortality of the participants compared to the other, then the observed difference between the groups end up being questionable. In this particular case study, M.A.P.S does not give any indication whether there was any issue in participants dropping out. Therefore, it has to be considered that this particular aspect would not act as a threat to the internal validity of the case study (McDermott, 30).

Analyzing External Validity of the Research Design

On the other hand, in accordance to McDaniel et al. (Chapter 10), external validity is defined as the magnitude to which the underlying relationships that are measured in an experiment can be given generalizations to external individuals, settings and timings. For instance, it is deemed that field experiments have greater external validity. Since internal validity has been established, it implies that there is treatment effect to generalize and therefore external validity can be established. According to Blanche et al. (165), a study is considered to have external validity when its conclusions or the findings attained may be universal beyond the limits of the design and the study setting. When undertaking an evaluation of external validity of a study, the main consideration is the degree or magnitude to which the conclusions can be extrapolated beyond the context of the study (Blanche et al., 166).

The external validity of the research design will be established through the analysis of the following factors:

Interaction

One of the factors that can negatively hinder the external validity of a study is whether an interaction amid the participants had been selected as well as the treatment. In this particular case, the participants in the both studies had been randomly chosen from the population. As a result, taking this into consideration, it can be deemed that the demographic individualities and features could not have had any bias on their performance. For this reason, the results of the research study are appropriate to this population as well as other groups, which more precisely signifies population characteristics. This factor does not adversely affect the external validity of the research (Bracht and Gene, 440).

Pretesting

The aspect of pre-testing the participants in a research study might instigate them to respond more or less intensely to the treatment in comparison to how they might have reacted if they had not experienced or gone through any pre-test. In this particular research study, there was no pretesting of participants for treatment 1 and treatment 2. Therefore, it can easily be concluded that the other random population members might perform within an identical and similar manner as the participants did within this study. Therefore, with no pretesting done, this factor could does not adversely impact or influence the external validity of the research study (Bracht and Gene, 442).

Setting

The participants' performance in some research studies has been more of an outcome or response to the setting of the experiment, for instance, where the research study is being carried out, than it is in reacting with the independent determinant. Case in point, the participants who know and are aware that they are taking part in a research study or are cognizant that they are being observed, might respond in a different way to the treatment being carried out than a participant who underwent the treatment; however, had been uninformed that he was being monitored. In this case, there is no indication that the participants were reacting due to the setting of the study or the store where the research was being conducted. Therefore, this factor did not impact the external validity of the research (Slack and Jolaine, 2176).

Multiple Treatments or Interventions

Any research study that makes use of multiple treatments or interventions might have restricted generalizability for the reason that the early and initial treatments might have an amassing impact on the performance of the subject. In simple terms, for example, in case a group of participants experienced treatment Y1, and this initial treatment had been followed-up with a succeeding treatment Y2, their performance that is measured subsequent to Y2 will be influenced with both these treatments and not just that of Y2. This is for the reason that the impacts of Y1 cannot be erased. In this particular case, the participants in the research study experienced treatment one and thereafter treatment 2. This implies that the significant outcomes perceived in treatment 2 were also influenced by treatment 1. This implies that there is treatment effect to generalize and therefore external validity can be established.

In general, it can be perceived that within any group research, the key methodologies and approaches used to attain validity (whether internal or external) have been randomization, utilizing a research design as well as employing statistical analyses that are fitting to the kinds of data gathered as well as the main question that the researcher is trying to answer. All of these elements are taken into consideration in this particular research study. The participants within the research are randomized as they are represented by the different consumers who walk into the store to purchase a beer. Secondly, there is a research design used as there is the comparison between treatment 1 and treatment 2 to analyze the difference and give a solution to the researcher's question. Third of all, the statistical analysis employed is quite basic. The threats to internal validity and external validity have been analyzed and show no indication of having an adverse impact on the validity of the research. Therefore, in overall, the research can be deemed valid, and in being valid can be considered to be reliable in future settings (Ferguson, 20).

Recommendation of a Better Design

An alternative to using the research design demonstrated in the article could be a correlational study to determine whether the placement of beer and sales is directly correlated. One may also use time of year (month) and sales of beer to determine a relationship as well. In addition, an observational study could also be used to view how and where consumers look at when shopping for beer. Although the 'what 'and 'how' would perhaps be discovered, the more important variable would not, the "why." For this, the researcher would perhaps need to use survey and discover why certain people buy certain brand of beers over others. Is it cost related and have taste preferences; does the time of year have anything to do with it? Many variables could be examined.

Another element that could be taken into consideration in the new design is having treatment 1 and treatment 2 undertaken at the same time in two different stores. This is for the reason that it can be argued that the timing of the research also has an influence on the results. For instance, at the end or start of the month is usually the time when individuals have just received their monthly income and therefore have greater spending power. On the other hand, in the middle of the month, the consumers usually have lower spending rates. Therefore, perhaps this influences the results of the study. In addition, it is also imperative to understand whether Dixie beer has a lower price compared to Miller beer. This is because the consumers might be purchasing the former simply because it is a cheaper option and not solely due to the placement and alignment of the beers along the aisle and fridge sections. Another aspect that could be included in the research design is having the first round and second round of treatment analysis in different stores. This will give certainty that the change in the position of the beer has an impact on the purchases.