Improving Literacy in the Seattle

Improving Literacy in the Seattle School District

Team Read Program

To outline a method of evaluating the Team Read Project

Team Read has completed its first year, and has grown significantly. Growth can be reasonably expected to continue over the coming school years, resulting in a significant increase in funding requirements. In order to acquire these funds, Team Read needs to demonstrate its effectiveness to the financiers. We feel that we should measure the impacts directly at the individual student level as this will provide the most specific feedback. We also wish to track the variables within the program more closely, in order to improve the effectiveness of the program to the benefit of both the readers and the financiers.

Problem/Background: Team Read is wrapping up its first full year of operation. During this year we grew rapidly, and now have 10 schools, 300 coaches and 355 students. We overcame several challenges over the course of the year, including logistical and staffing issues. We went from having to convince schools to take on the program to a point where we are now receiving unsolicited requests from elementary schools for Team Read. The future of the program is very bright, and we are most appreciative of your support. We understand that future support depends on the ability of Team Read to deliver strong results in creating readers.

Seattle is home a diverse population. Many of our schools are majority non-white. There is also a significant gap in income levels between student bodies. The performance of our elementary schools in areas the core areas of reading, writing and mathematics shows a very wide range results. When those results are weighed against demographics statistics, we can see certain trends emerge. A correlation appears between poverty, family structure and poor academic performance. There is also a correlation between students who are non-native English speakers and academic performance at the elementary school level. We feel that Team Read can help to bridge these gaps.

However, at present we do not have a specific system in place to measure the success of Team Read. We feel that before the next school year commences, we should have such a system in place. Team Read cost $248, 000 this past year to operate, not including the tuition that has accrued for our coaches who opted for that compensation option. The total cost of the year therefore could be closer to $300,000 once all the accrued costs have been tallied. There is demand amongst elementary school principals to expand the program, and we have identified that as many as 20 school have as little as 1/3 of students achieving WASL reading standards. Over 30 schools have a high proportion of at-risk students as determined by poverty rates, single-parent homes or non-English speaking homes. Therefore, the cost of the program has the potential to expand significantly over the next few years. This will require additional sources of funding, and for that we will need to present specific figures with regards to the program's success, both in terms of end results and in an analysis of the success factors that contribute to those end results.

Stakeholders:

Team Read has many key stakeholders - the readers, the coaches, the school staff and the financiers foremost with parents, and the community at large as secondary stakeholders.

The readers' stake in Team Read is their literacy. Students who fall behind in their reading skills at the elementary school level are at a disadvantage throughout their educational careers. This is especially true for students who come from homes where English is a first language. ESL students often catch up to other students by the time they reach high school, but ESL students with learning difficulties are a high-risk group because those difficulties may be masked by the language barrier.

Students who struggle with reading often become discouraged with education in general, since reading is such an integral part of the learning experience in our schools. These students face the risk of a lifetime of hardships as a result of their reading difficulties. We feel that Team Read offers them not only the opportunity to learn to read but the opportunity to give them hope. They have perhaps the biggest stake because of the long-term impacts of reading difficulties.

The coaches' stake is largely in the experience of working as a coach. The financial benefits are limited, given the relatively few hours available working as a coach. However, coaching is one of the most fulfilling jobs available to a high school student. For them the opportunity to do a job that offers genuine benefit to another human being is invaluable.

The stake that school staff have in Team Read is the opportunity to improve the effectiveness of their schools and enhance the lives of the readers. The pay gained from an eight-hour a week part time job is not one of the major benefits of Team Read to school staff. The principals benefit in that they are given a way to reach out to the students most in need, and over the long run improve their schools' test scores. Most principals and staff genuinely care about improving the lives of their students. The main point of concern is the longevity of the program - many such programs have lacked long-term effectiveness because they could not sustain themselves. Team Read at this point has won the support of many school administrators and staff but remains at risk of becoming yet another failed program.

The financiers of Team Read have many philanthropic opportunities. The cost of Team Read will ultimately be weighed against its effectiveness and the opportunity cost of financing other beneficial programs. The cost of last year's run, once the 50% tuition bonus and four-year scholarships have been paid, will likely exceed $300, 000. Based on our needs assessment, Team Read could easily double or even triple in size in the coming years, bringing costs up toward the $1 million dollar mark. For that kind of money, this group of stakeholders will need to know that the project is having a definable and meaningful impact.

The secondary stakeholders are those impacted by the results of Team Read, both the parents and the public at large. By improving literacy rates, and salvaging the educational careers of at-risk students, both of these groups stand to receive significant benefit from the program. The parents of these students often cannot readily afford private tutoring so would either be forced to make significant sacrifices for their children or, as is most often the case, be forced to sacrifice their children's education. Team Read helps to remove some of that burden. Society at large is forced to bear the costs of students who do not succeed in the education system. Poverty rates for such students are much higher, leading to increased welfare usage, criminality and other adverse consequences. By getting to these students at an early age, we can help to prevent them from falling into this cycle, by building in them the skills and confidence needed to overcome obstacles and find success.

Policy Alternatives:

The main issue at present is to identify the measures we will use to evaluate the program. We have identified several approaches to the issue. The first is to evaluate outputs only. Within this alternative there are several methods and measures at our disposal. We already track test scores and compile school performance. One way to analyze the effectiveness of the program is to compare schools that are involved in Team Read vs. those that are not. This alternative is easy to implement, in that the statistics are readily available from government testing. One of the main drawbacks is that the number of students involved in Team Read may not be statistically significant. In any given school, the number of students in the program is around 25, which depending on the school is between 5 and 10% of the student population. At these levels, the impacts of Team Read could be obscured by year-over-year variability in test results. With a constantly changing student body, shifts in demographics and staff turnover, the comparables may not be sufficiently corollary to provide valuable information. Additionally, this alternative would require several years of data in order to provide truly useful information.

Another option is to base our evaluations on specific student performance. This information is available through test scores.

It gives specific figures on a student's performance at the end of the year, which can be compared with that student's performance the previous year. This can be compared against students who did not participate in the program. The drawbacks to this information is that it can be difficult to separate improvement from participation in Read First and improvement that would have otherwise occurred. Each year, the student will have a new teacher, the student could a be late bloomer or the results could be impacted by the impact of outside variables that affect the underlying causes of the reading difficulty. Additionally, many of the most at-risk students are relatively transient, which can reduce the pool of information available, or reduce the relevance, for example if the student has switched schools it can be difficult to ascertain the cause for a change in their reading ability.

Another alternative is to not only measure one or both of the above, but to also analyze the impact of other factors, so that we can not only identify the outputs of the program, but break down those outputs in terms of changes in inputs. For example, next year we will be tracking the race of the coaches. This can help us to determine if matching coaches and students by race has any impact on the students' success rates. More in-depth analysis will require more money to measure. Also, it may not provide useful data unless the pool of information is larger, which is ultimately a function of available funding. The benefit, however, is that this information will allow us to increase the effectiveness of the program, so as to get better use out of the money we spend.

Evaluation Criteria

The most important evaluation criteria is the specificity of the information. There are so many variables in education that it can be difficult to accurately gauge the effectiveness of a program like Team Read. We need to seek out the information that not only provides specific and direct feedback but isolates as many key variables as possible.

The second most important criteria is that whatever method we choose should address the needs of the most important stakeholder - the readers. Their educational needs should ultimately guide the structure of Team Read, and we should be able to gather information that helps us to understand those needs and the best ways to address them.

The third criteria should be timeliness. There is little doubt that Team Read will proceed next year, but there is doubt as to the specific level of funding that will be made available. We need to be able to provide feedback to the financiers as soon as possible so that we can adequately prepare for the coming school year, in terms of having a budget and being able to identify the number of schools we are able to help. The ad hoc manner in which we grew this year was a necessary requirement of a first-year program, but as the size of the program increases both those of us administering Team Read and those funding it will need to have greater certainty. Every summer we will have two months to wrap up one school year and prepare for the next. The approximate timeline is to gather the data, present it along with our proposals for the following year, receive feedback on the funding levels that we will receive, and then design the program. This means that we have a short window in which to gather information on the effectiveness of the program.