Evaluation of Bias Correction Methods for “Worst-case” Selective Non-participation in NAEP

Random non-participation introduces random error into NAEP estimates. More worrisome is the possibility of selective non-participation at the top or bottom of the ability distribution, which would introduce a bias into statewide mean scores. Selective non-participation could be an unintended consequence of NAEP’s new role under No Child Left Behind. In anticipation of such an outcome, this study estimated the potential bias from "worst-case" scenarios of selective non-participation and examined the extent to which statistical methods can correct for that bias. Correction for non-participation bias using information about students’ state test scores was found to be partially effective, eliminating roughly half of the bias when a linear equating method was used. A useful finding was that exact state test scores could be replaced with state test score quartiles in these non-participation bias corrections without serious degradation of results.