problems with random assignment experiments on principals

In “An Examination of the Benefits, Limitations, and Challenges of Conducting Randomized Experiments with Principals,” the researchers use a case study method to show why a professional development program for principals failed to achieve its desired outcomes. Focusing on one school district with 48 principals, the authors state, “We developed and understanding of why the program failed to achieve its intended goals,” (p. 188). The District Professional Development (DPD) program was designed to improve leadership capacities among principals with the ultimate goal of improving school outcomes.

Based on interviews with district officials, a systematic review of the DPD materials, observations of the actual DPD training sessions, attendance records, and the principals’ personal logs and journals, the researchers collected substantive qualitative evidence that can be used to supplement their quantitative experimental research on the DPD. In other words, this study is a classic mixed-methods research design that uses both qualitative and quantitative methods judiciously.

The qualitative methods are needed to illustrate why the program did not work, which is not something the data from the quantitative study could ever reveal. As it turned out the reason why the program failed to achieve its intended goals were related as much to methodological constraints as to subjective issues like lack of motivation. The researchers outline several main challenges to implementing a professional development program as ambitious as the DPD.

One challenge was the effects of changes to educational policy on the professional identity and roles of principals during the delivery of the DPD. Another was the fact that true random assignment was not even possible, due to what they call “subversion,” or not showing up when they said they would or showing up to the DPD training even when not registered (p. 201). The third challenge was reduced program delivery, and the decline in attendance over the duration of the course.

Although there were some notable outcomes in the DPD experimental group, such as increased time spent on planning and goal setting, the authors nevertheless brand the intervention a failure due to low attendance and poor compliance. The qualitative aspects of the case study help the researchers to unearth the core assumptions that guided the experimental intervention in the first place. Theoretical orientation, worldview, values, and other guiding principles may need to be changed in order to design more effective professional development programs.

Due to the complex array of variables involved in leadership, including situational constraints, individual differences, organizational culture, and the multiple components of the program itself, the overly reductionist qualities of an experimental design using binary responses (yes/no or true/false) “will be of limited utility to policymakers, leaders, and practitioners,” (p. 211). The review of principal journals and other qualitative data sources augment the quantitative data to show how principals were interacting with the DPD.

More importantly, the researchers used their qualitative data to demonstrate weaknesses in the experimental design. The program ended up not being delivered fully to enough principals due to noncompliance and other issues. The DPD was also not administered uniformly to all principals. There are simply too many variables in program delivery to allow for a meaningful.