Evaluation of Massachusetts District and School Turnaround Assistance: Impact of School Redesign Grants (SRG)

Christina LiCalsi, Martyna Citkowicz, Lawrence B. Friedman, Megan Brown

Executive Summary

The Massachusetts Office of District and School Turnaround (ODST) assists the Commissioner’s Districts (the 10 largest districts in the state) and schools within those districts. The assistance focuses on turning around the lowest performing schools in the district while building district capacity to support improvement in other district schools. The three major strategies examined in this study are as follows:

District Liaisons. Liaisons include ODST staff members who serve as project managers and coordinate support to the districts, overseeing implementation of the state’s strategy for school turnaround.
Priority Partners. Partners include external organizations that support turnaround efforts in four areas of support: maximizing learning time; the effective use of data; social, emotional, and health needs; and district systems of support.
School Redesign Grants (SRGs). SRGs are competitive funds that support turnaround efforts in persistently underperforming schools. The first year grants were awarded was 2010-11.

American Institutes for Research (AIR) contracted with the Massachusetts Department of Elementary and Secondary Education (Massachusetts ESE) to conduct an evaluation of ODST assistance to Commissioner’s Districts and schools. This report summarizes the results from an impact analysis focused on the impact of the SRG receipt. SRGs are provided through federal School Improvement Grant (SIG) funding. Federal guidelines allow states flexibility in how they administer and monitor their distribution of SIG funds. The Massachusetts ESE allocates SIG funds as SRGs through a competitive application process. They commissioned this research to better understand how the process it designed has impacted or has failed to impact schools and how the process might be improved. This report answers the following research question: What is the impact of SRG receipt on student outcomes (e.g., academic achievement and attendance)?”

Using a comparative interrupted time series (CITS) design, AIR researchers examined whether, when compared with non-SRG schools and controlling for selected background characteristics, students in SRG schools experienced better academic outcomes and attendance.

Methods

AIR used a CITS design to measure the impact of SRG receipt on student outcomes, including student achievement and attendance. The basic principle of CITS is that the SRG effect can be detected by comparing changes in the outcomes of SRG schools to changes in the outcomes in a comparison group during the same time period. This approach draws on information from both the treatment and comparison schools to estimate what performance in SRG schools would have been if the program had not been implemented. The deviation from this prediction is the estimated treatment effect of SRG receipt.

The sample for this study included all students in cohorts 1 (beginning grant year of 2010–11), 2 (beginning grant year of 2011–12), and 3 (beginning grant year of 2012–13) of the SRG schools, plus students within the same districts in comparison schools. Comparison schools were those in the same districts as the SRG schools but did not win an SRG. We used multilevel regression models to control for confounding factors (e.g., student characteristics), nesting of students within schools and years, and any changes in the given indicator across time that was not caused by the intervention itself. In addition, we controlled for student-level covariates (race, special education status, free- or reduced-price lunch status, and English language learner [ELL] status) and school-level factors (grant year, district, and whether the school was a high school) and allowed for baseline differences between schools.

Findings

The impact study found the following:

When considering prior achievement trends, students in the SRG schools performed better on the English language arts (ELA) and mathematics sections of the Massachusetts Comprehensive Assessment System (MCAS) compared with students in comparison schools. The effects were statistically significant after the first, second, and third years of SRG implementation on both the ELA and mathematics sections.
Positive SRG impacts on achievement were found for schools serving Grades 3–8 across all three years on both the ELA and mathematics sections. For schools serving Grade 10, positive impacts were found in Years 1 and 2 on the ELA section and in Year 1 on the mathematics section.
Positive impacts on both the ELA and mathematics sections were found in all three years of program implementation in Boston and for the first two years in Springfield and the other districts.
SRG receipt was associated with a decrease in the achievement gap on both the ELA and mathematics sections between ELL and non-ELL students in SRG schools compared with the change in the achievement gap between students in the comparison schools. These effects were found in all three years of program implementation on the ELA section and in the first two years on the mathematics section. The achievement gap also was decreased for students who had free- or reduced-price lunch status versus those who did not have such status in the SRG schools compared with similar students in comparison schools in the first year on the mathematics section but then increased in the second year. No changes were detected for students with special education status.
When considering prior achievement trends, students in the SRG schools appear to have slightly lower attendance rates compared with students in the comparison schools in the third year of SRG receipt. However, when examining these rates separately by school level, the effect appears to be negative for schools serving Grades 1–8 and positive for schools serving Grades 9–12 in the first year of program implementation. The effects appear negative for Years 2 and 3 in Boston, positive for Year 3 in Springfield, and positive for Year 1 in other districts.
SRG grant receipt was associated with a decrease in the attendance gap between students who were in special education versus who were not in special education in the SRG schools compared with the change in the attendance gap between similar students in the comparison schools in all three years of program implementation. No changes in the attendance gap were detected for ELL versus non-ELL students and students with or without free- or reduced-price lunch status in the SRG schools compared with the comparison schools.

Conclusion

The results from this evaluation suggest that the disbursement of SIG grants in the process designed by ESE as SRGs have consistently positive effects on student academic achievement, particularly on standardized state assessments. Moreover, these results are generally robust across districts and school levels, and they are particularly strong for students who are ELLs. The results are neither strong nor consistent for attendance, suggesting that, overall, SRG receipt does not affect attendance.