The Zambia eSchool 360 Evaluation Project

Low- and middle-income countries have made significant progress toward placing children into schools, but student learning and achievement are often dreadfully low. Globally, approximately 387 million children of primary school age are not acquiring basic reading and mathematics skills, even though about two-thirds are in school and will complete primary school. Zambia faces many common educational challenges.

Impact Network’s eSchool 360 program represents a promising approach to improving educational outcomes in rural Zambia by incorporating three potentially high-impact components that could create important synergies. The multifaceted program comprises an e-learning technology component, ongoing teacher training and professional development, and community ownership among students in community schools in rural Zambia. The e-learning component includes electricity via solar power, projectors, and tablets loaded with materials in the local language, structured around a curriculum approved by the Zambian government.

Each component could, on its own, have positive impacts on student outcomes by engaging the three main actors in the education system: students, teachers, and parents, respectively. However, combining these elements into a single program may have particular impact by working to align all three actors toward improving the educational outcomes of the students. Earlier research has suggested that these complementarities may be substantial with higher impacts from educational technology programs that include a strong focus on pedagogical practices.
 

The Evaluation

AIR designed and implemented, a two-stage mixed-methods evaluation to measure the effect of Impact Network’s eSchool 360 program on learning outcomes among students in community schools in rural Zambia as well as the cost-effectiveness of any measured impacts. The use of cost-effectiveness analyses helped estimate the cost of achieving certain benefits, such as improvements in learning outcomes. These estimates, in turn, guided policymakers in assessing the value for money of investing in the eSchool 360 program.

The evaluation used a cluster-randomized controlled trial in which 63 eligible schools were randomly assigned to either receive Impact Network’s eSchool 360 program (“treatment schools”) or not receive the program (“control schools”). The evaluation design began with quantitative baseline data collection, followed by quantitative and qualitative data collection to inform the scale-up of the program one year and three years after the start of the baseline, by measuring aggregate scores on the early grade reading assessment (EGRA), early grade math assessment (EGMA), and Zambian achievement tests; the secondary outcomes are the EGRA, EGMA, and Zambian achievement test subtasks as well as measures of oral reading fluency.

In addition to the EGRA, EGMA, Zambian achievement test, and oral reading fluency measures, we estimated the program impacts on school attendance and enrollment, parents’ perceptions of school and education quality, student-level aspirations, and parent-level aspirations. Using these outcome measures enabled us to determine impacts along the causal chain of the theory of change and allowed us to examine the mechanisms underlying the program impacts.
 

Key Findings

• The cluster-randomized controlled trial of the program showed that the program increased early grade reading scores by 0.40 standard deviations and early grade mathematics scores by 0.21 standard deviations for all children eligible to enroll in first grade 14 months after program start. 
• Follow-up impact evaluation results more than four and a half years after the start of the program continue to show substantive and statistically significant impacts on early grade reading scores of 0.26 standard deviations. While we did not find positive average effects on mathematics scores, the study did show positive and statistically significant impacts on mathematics scores for boys. The evaluation found these results slightly more than one year after the restart of primary schooling following the COVID-19 pandemic in Zambia. 
• Findings from a comparison with government schools revealed that Impact Network students continued to score higher on all assessments than students in government schools in Grade 5 more than four and a half years after the start of the program.
• Process evaluation findings indicate that the eSchool 360 program was implemented with a high fidelity of implementation.