Charter Schools: Research Methodology Review

¶ … charter schools: Research methodology

Review of Methodology of a Study

Research methodology review

Greene, Jay P., Greg Foster, & Marcus Winters. (2003, July). Apples to apples:

An evaluation of charter schools serving general student populations. Center for Civic Innovation at the Manhattan Institute. Education Working Paper. Retrieved May 22, 2010 at http://www.manhattan-institute.org/pdf/ewp_01.pdf

Research methodology review

Greene, Jay P., Greg Foster, & Marcus Winters. (2003, July). Apples to apples:

An evaluation of charter schools serving general student populations. Center for Civic Innovation at the Manhattan Institute. Education Working Paper. Retrieved May 22, 2010 at http://www.manhattan-institute.org/pdf/ewp_01.pdf

One of the most striking and controversial developments in recent years has been the explosion of charter schools in America. Charter schools are publically-funded schools that are released from some of the restrictions that affect the policies of public schools (such as requiring teacher certification or specific aspects of curriculum management). In exchange for these special provisions, charter schools agree to meet specific goals, often for students with needs that are not met by the standard public educational system. These goals might pertain to the education of the gifted or the disabled; the need to offer higher-level math and science or intensive immersion in the arts; or helping high-risk youth graduate from high school.

But while the personalized nature of charter schools has caused many politicians and educators alike to cite them as 'the solution' to the crisis of poorly performing school districts, according to the 2003 Manhattan Institute educational research working paper entitled "Apples to apples: An evaluation of charter schools serving general student populations," little research has been conducted on charter schools. This is partially due to the idiosyncratic nature of many of these educational facilities. Many are designed to suit the needs of challenging populations, such as special needs students or delinquent youth or conversely the schools 'cherry pick' the best and brightest for their student bodies. A comparison with public schools within their district is neither fair to the charter school nor to the mainstream public institutions nearby. As nearly a quarter of all charter schools tailor their curriculums to suit the needs of specific demographics, useful cross-comparative studies have proved elusive. The authors hope that by deploying a large-scale quantitative approach that compares charter schools with similar demographic populations to their corresponding public school counterparts will offer a sound quantitative perspective upon the potential gains charter schools can yield for students. If the charter schools have similar aims to that of the nearby, more traditional school district, the comparison is fair. "Since the goal of our study is to overcome the problem of differences in student populations, it is particularly important that we remove the influence of student background characteristics to the extent that we can. Excluding targeted charter schools that are unlikely to have a counterpart in the regular public school system [such as science magnet schools or schools for students with special needs] is by far the most important way in which we do this; comparing each charter school to its closest regular public school is another" (Green, Foster, and Winters, 2003, p.9).

The number of charter schools is growing, and this growth has been encouraged by the nation's political leaders. Although the comparison between charter and standard public schools may be imperfect, the need to establish some basis of comparison cannot be ignored, state the study authors. At the time of the study, there were 2,700 charter schools within America. 684,000 children were enrolled in such schools nationwide (Green, Foster, and Winters, 2003, p.1).The study was conducted for the duration of a single year, and involved eleven states: Arizona, California, Florida, Texas, Michigan, Wisconsin, Ohio, Colorado, North Carolina, Minnesota, and Pennsylvania, all of which "had large enough numbers of charter schools to give us a good chance of yielding sufficiently large samples after targeted schools were eliminated. We also included four other states -- Illinois, New York, New Jersey, and Indiana -- that had fewer charter schools but were of particular interest" because of their diverse demographic composition (Green, Foster, & Winters, 2003, p.6). However, every state has its own regulations as to how charter schools are approved and what educational regulations can be waived for charters (Green, Foster, & Winters, 2003, p.2-3).

The assessment of the quality of the schools under review was evaluated solely on the basis of standardized test scores. Test score improvements were measured, comparing the test case year to the previous year and the following year. It was found that on average "charter schools serving the general student population outperformed nearby regular public schools on math tests by 0.08 standard deviations, equivalent to a benefit of 3 percentile points for a student starting at the 50th percentile. These charter schools also outperformed nearby regular public schools on reading tests by 0.04 standard deviations, equal to a benefit of 2 percentile points for a student starting at the 50th percentile" (Green, Foster, and Winters, 2003, p.1).

However, a single year, even when compared to the previous and following school year is still an extremely short period of time in a child's education. This was, in short, not the type of study where a longitudinal analysis was conducted to determine if attending a charter school improved a child's likelihood of succeeding in college. "Another approach, useful when multiple years of individual student test score data are available but individual demographic data are not, is to use what statisticians call a 'fixed-effects' model. This method keeps track of each student's performance separately, comparing a student's performance in one year to his performance in the next year, the year after that, and so on. By comparing each student only with himself, this method removes much of the influence of differences in student populations" (Green, Foster, & Winters, 2003, p.5). Yet the relatively short existence of most charter schools means that such long-term studies comparing individual students are difficult to formulate. The researchers believe that a longitudinal study with a meaningfully-sized population of charter school graduates would have been impossible, hence their single-year approach. A longitudinal study would also not have given the authors the type of immediate results called for in the current political situation: politicians and parents alike want to understand how to allocate scarce financial resources in the most effective manner possible. In the future, if the number of charter schools in existence remains constant, a more extensive longitudinal study might be more helpful in determining how ostensible gains in test scores translate into individual student's long-term academic performance, and thus the overall efficacy of charter vs. public schools.

A second, perhaps more serious question raised by the researcher's methodology is if the charter schools selected are really representative of charter schools, given that so many schools are begun with a population-specific approach and thus were excluded from the study. Public schools targeting specific population groups were also excluded from the study. The relatively small test gains boasted by students, which only held true in a few states, suggest that even the statistically significant results, might be somewhat idiosyncratic and not really proof that charter schools provide higher-quality instruction for most student populations.

One advantage of the study was the fact that the researchers did segment the results by state and provided a basis of cross-comparison not simply between charter and public schools, but also between states. For example, in some states, such as Arizona, California, and North Carolina, the results were inconclusive. "The study's strongest results came in Florida and Texas. In Texas, charter schools achieved year-to-year math score improvements 0.18 standard deviations higher than those of comparable regular public schools and reading score improvements 0.19 standard deviations higher. These benefits are equivalent to 7 and 8 percentile points, respectively, from the 50th percentile. Florida charter schools achieved year-to-year math and reading score improvements that were each 0.15 standard deviations greater than those of nearby regular public schools, equivalent to a gain of 6 percentile points for a student starting at the 50th percentile" (Green, Foster, and Winters, 2003, p.9).

This seemingly leads to the conclusion that charter schools in Florida and Texas are 'better' than those in other states. However, the study results were not based on a general test for all subjects, but student performance on different state tests. The results for the state might meant that students in charter schools were better-prepared for standardized exams than those in non-charter schools, but not necessarily better-educated. The emphasis on 'accountability' for charter schools, in terms of proving their right to exist based upon statistical gains on exams could suggest that there is an added incentive for such schools to specifically 'teach to the test,' more so than public schools whose entire existence is less likely to be threatened by minute losses in percentage points. (the authors did not entertain this possibility).

In the final paragraphs summarizing their findings, the authors admit that no definite conclusion can be drawn regarding the efficacy of charter schools, given the modest statistical differences between populations. They attribute the small benefits conveyed by charter schools to the schools' newness, and to a lesser extent, a lack of funding and over-regulation of supposedly individualized curricula. This means the schools may not be so different from their public counterparts. The author suggest, for further research, a randomized comparison of charter and public school students in terms of standardized state test performance to provide better insight into the question of what is 'better,' overall, for students.