An Analysis of the Validity of Student Eligibility Status for NSLP as a Proxy Variable for Socioeconomic Status in Reporting NAEP Results After the Implementation of the Community Eligibility Provision

Qingshu Xie
,

This analysis uses the 2009–2019 NAEP mathematics data to determine whether the implementation of the Community Eligibility Provision (CEP) in the National School Lunch Program (NSLP) has reduced the validity of student NSLP eligibility status as a proxy for socioeconomic status (SES) in the reporting of NAEP results. Concerns about the validity of student NSLP eligibility status as a proxy for SES in NAEP are based on the supposition that CEP implementation might substantially increase the overall percentage of NSLP-eligible students and the proportion of NSLP-eligible students across schools at lower poverty levels. That is, a substantial number of students who were previously ineligible for NSLP and attended lower poverty schools might now be counted as NSLP-eligible by NAEP.

This paper is part of a series of AIR-NAEP working papers that showcase AIR’s expertise and experience not only with NAEP but with other large-scale assessments and survey-based longitudinal studies. Explore all the AIR-NAEP working papers.

In general, NSLP-noneligible students score higher on NAEP than NSLP-eligible students. If many higher-achieving students who would have been classified as NSLP-noneligible based on pre-CEP criteria were now grouped with NSLP-eligible students due to the implementation of CEP, the average NAEP score of the NSLP-eligible group would increase. This would cause the achievement gap by NSLP eligibility status to decrease and the variance in NAEP achievement explained by NSLP eligibility status to diminish. However, if these trends are not observed, concerns about the validity of student NSLP eligibility status, as defined and reported by NAEP, as a proxy variable for SES may not be warranted.

Based on what we would expect to observe if NSLP eligibility status is becoming less valid due to the implementation of CEP, the study uses four criteria to determine whether there is evidence against the validity of NSLP eligibility as reported by NAEP. Specifically, if the validity of the NSLP measure has been compromised by the adoption of CEP, the following trends should be observed:

  1. There should be a substantial increase in the overall percentage of NSLP-eligible students.
  2. There should be a substantial increase in the percentage of NSLP-eligible students from schools at lower poverty levels that participated in school-wide free lunch programs.
  3. The achievement gap between NSLP-eligible and NSLP-noneligible students, as defined and reported by NAEP, should decrease over time.
  4. The variance explained by the association between NSLP eligibility status and NAEP performance should also decrease over time.

The analysis was conducted at the national, state, and Trial Urban District Assessment (TUDA) levels to examine the impact of the adoption of CEP on reported NSLP eligibility in NAEP.

The major findings using these criteria are:

  • At the national level, our analyses do not support the concerns that student NSLP eligibility status is no longer a valid SES proxy variable due to the implementation of CEP.
     
    1. There has been no substantial increase in the overall percentage of NSLP-eligible students since the national implementation of CEP. The percentage of NSLP-eligible students from CEP-all schools remained small and unchanged between 2015 and 2019 after the national implementation of CEP.
    2. There has been no fundamental change in the percentage of reported NSLP-eligible students from schools that participated in school-wide free lunch programs by the level of poverty concentration. 
    3. NAEP achievement gaps by student NSLP eligibility status at the national level have also remained very stable over time. There has been no decrease in the NAEP mathematics achievement gap by student NSLP eligibility status. 
    4. The variance in student NAEP mathematics achievement explained by student NSLP eligibility status, as measured by R-squared, has remained largely unchanged at all three grades from 2009 to 2019. 
       
  • At the state level, there is no evidence of a continuing, across-the-board increase in the percentage of students recorded as eligible for NSLP or a widespread decrease in NAEP achievement gaps by student NSLP eligibility status.
     
    1. Between 2011 and 2019, no state experienced a decrease in the achievement gap at grades 4 and 8. 
    2. During the same period, only nine states at grade 4 and four states at grade 8 saw a decrease in the variance explained by student NSLP eligibility status. This suggests that, for most states, student NSLP eligibility status remains a valid proxy variable for student SES at the state level.
       
  • At the TUDA district level, there has not been a widespread, dramatic increase in the percentage of students eligible for NSLP in most participating districts following the nationwide implementation of CEP. Additionally, the reporting of NAEP achievement gaps at the TUDA district level has not been affected by the implementation of CEP.
     

Taken together, these results suggest that, to date, the implementation of CEP has not affected the validity of student NSLP eligibility status as a proxy variable for SES at the national, state, or TUDA levels.

The likely reason for the disconnect between the often-voiced concerns about the decreased validity of NSLP eligibility status due to CEP and these empirical results is the misconception that NAEP cannot differentiate between NSLP-eligible and NSLP-noneligible students in schools implementing CEP. In fact, in most schools where all students can receive free lunch, NAEP can still differentiate between students who are economically disadvantaged (coded as NSLP-eligible) and those who are not (coded as NSLP-noneligible), as defined by respective state standards.

This technical memorandum provides important empirical evidence for the continuing validity of NSLP eligibility as an SES proxy variable for NAEP reporting and for its inclusion in a NAEP SES index.