Participating in Post-Tenure Review, One

¶ … participating in post-tenure review, one primary concern consists of the need to ensure that envisioned research is conducted within the limits and expectations of acceptable statistical methodology. Because of the potential conflicts of cultural differences in faculty oversight (Wood and Johnsrud, 2001), the need to understand and promote objective research methodology is crucial to faculty adhesion. Through controlling such factors as research validity through appropriate methodological constraints, the researcher can avoid some of the potential pitfalls of cultural disagreements which may arise in the absence of such controls. Therefore, this brief paper will consider the role of design validity, specifically focusing on external, construct, and internal validity, as a means of facilitating discussions involved in post-tenure review. Through definition of the concepts and discussion of possible problems that may arise in research projects if appropriate controls are not in place, the paper will present an outline for ensuring that research design validity is enhanced.

External validity, according to Trochim (2008), consists of generalizable "truths" about a given population. The goal of controlling for external validity is to choose a sample from the total population that has the same characteristics as the population has. When this is done properly, the findings related to the sample should be generalizable to the total population with a higher degree of confidence. When done improperly or not at all, the study cannot be said to hold appropriate inplications about the population. The concept of gradient of degreed confidence is important here, as the sample will likely never be perfectly representative. The researcher wants to choose a sample set in a particular time and place with a particular group that gets as close as possible to approximating such perfect representation, however. Threats to external validity include the possibilities that the wrong people were chosen, or that the wrong time or location was selected from which to choose subjects. Additionally, it is possible that the researcher is not entirely certain what the total population for a study looks like, and therefore sample selection is complicated. In order to control for external validity, the researcher should identify the population in terms of personal and group characteristics, time and location. Then, randomized selection techniques should be utilized in order to make sure that selector bias does not occur. Finally, the researcher should take care to make sure that properly randomly-selected subject do not drop out, but actually participate in the study, thus ensuring high response rates and enhance validity.

Construct validity consists of the way the measurement instrument is constructed, including the labeling and conceptualizing of variables to be measured and the scales for use in measuring them. Trochim points out that there are a number of factors which impact upon construct validity. Face validity consists of a common sense assessment of whether the items and instrument seem like good measurements of the relevant concepts. Content validity consists of ensuring that the various criteria involved in the concept are all accounted for in the measurement instrument. Predictive validity asks whether the variables in the instrument are associative of and predictive for the concepts with which they ought to be related. Concurrent validity measures whether the instrument allows for distinctions to be made between groups that should be separately identifiable. Convergent validity asks whether responses to items in the instrument which are conceptually related move in similar ways relative to each other. And divergent validity measures whether operationalized variables in the instrument are separately identifiable from other, unrelated concepts. The threats to construct validity include not clearly defining variables operationally before instrument construction, using a too-narrow concept for measurement or a too-narrow treatment for measurement that does not reflect the full nature of the concepts measured, improperly measuring the effects of multiple treatments related to the variable in ways that do not account for interaction of other variables, failure to account for the effects of such considerations as subject apprehension or testing effects on subject responses, and experimenter bias. The researcher can control for these threats by linking the theoretical rationale for measurement with the actual measurements, conducting multivariate methodological treatments to ensure that variables moved in the expected directions and with the appropriate similar variables, and through pattern matching to show how variables relate to one another, relevant to expectations.