Variables Impacting Internal Validity in Program Evaluation

Internal Validity

Is maturation a possible threat to the internal validity of the study? Why or why not?

For measures of academic achievement, six months (which is the intended length of the study) is not a long enough period of time to be very concerned about maturation as a threat to internal validity. Most academic achievement tests have longer durations. Maturation is something to be considered because of the varying ages of the study participants, and the fact that we know children's brains mature at different rates and some mathematical concepts are not understood until the human brain does mature to certain levels. Moreover, since the boys will move in and out of residency, their exposure to other reading instructions methods and tutoring cannot be controlled.

Is history a possible threat to the internal validity of the study? Why or why not?

Yes. History is the occurrence of some unanticipated event, such as the revelation about the director lying on his resume about attending school and being found out. History is generally not a threat to internal validity in a two-group experimental design since it is unlikely to be of sufficient magnitude to impact the outcome for the entire single group. However, the nature of the historical event could have direct bearing on the attitudes of the boys towards school and learning.

3. Is testing a possible threat to the internal validity of the study? Why or why not? No. Both groups will experience the same testing schedule and type of tests, so the pre-test variable is not a threat to the internal validity.

4. Is instrumentation a possible threat to the internal validity of the study? Why or why not?

Yes. Any change to the way that the dependent variable is measured can be a threat to internal validity. Since the investigator is planning to measure GPA while the boys are in school differently from when the boys leave school (by asking them to report it following receipt of a letter), a threat to internal validity does exist.

5. Is selection bias a possible threat to the internal validity of the study? Why or why not?

Yes. Unless the selection of participants achieves a randomly selected sample, selection bias is a very real threat to the internal validity of the study. The participants in the study should be assigned to their groups through random selection, yet the investigator is planning to enlist the boys who have performed the most poorly in their last reading test.

6. Is statistical regression a possible threat to the internal validity of the study? Why or why not?

Yes. The instrument that the investigator is planning to use to measure the boys attitudes toward school has never been piloted. More importantly, the participants will be selected on the basis of worst reading score, so any improvements measured can simply be a regression to the mean and not due to the effectiveness of the intervention. Also, without repeated testing, outliers are more likely to emerge; whereas, with repeated testing, these outliers are likely to be absorbed into a more normal distribution.

7. Is mortality a possible threat to the internal validity of the study? Why or why not?

Yes. Since a number of the boys will leave the facility after six months and others will be there for three to five years, and because past history says that it will be difficult to contact the boys once they leave the facility, with actual the actual percentage of boys still accessible running about 10%, mortality is a threat to internal validity.

8. Of the above questions, how many did you provide a "yes" response?

Four.

9. Given your responses to the previous questions, how confident are you in the study's ability to attribute changes in the dependent variable to the independent variable? In other words, what is your level of confidence that the independent variable caused the changes in the dependent variables? Why?