Standards and Assessments Ecosystem

AIR supports state education agencies (SEAs), regional education agencies (REAs), and local education agencies (LEAs) as they collaborate with diverse constituents to plan and facilitate standards and assessment development. AIR utilizes evidence-informed protocols and tools to support engagement with diverse voices and groups in the development or revision of academic standards and assessments. 

• Image

  

  Conducting landscape scans: Landscape scans help inform constituents of current context and set the stage for the standards development or revision process. Subject matter experts at AIR conduct national and international landscape scans of standards and assessments to provide current context that informs the updating or revising of standards. The scans may include overviews of key national and international frameworks, current frameworks for national and international assessments, and analyses of other states’ standards and assessment policies. Landscape scans are also used to engage constituent voices in each state and can help set the stage for the development or revision of state standards and assessment frameworks.  

• Conceptualizing evidence-based engagement strategies: AIR’s evidence-based engagement strategies are informed by resources developed by AIR experts using research and best practices from the field and deep experience in technical assistance. For instance, AIR-created evidence-informed guides that offer techniques to encourage brainstorming and implement effective constituent engagement strategies such as developing key messages and creating two-way feedback loops. AIR also developed a collaborative, research-based process called co-interpretation to help constituents analyze and share data. This process helps constituents engage with the data in meaningful ways and make data-based decisions.  

• Analyzing constituent feedback to make data-informed decisions: AIR has experience successfully convening groups both in person and virtually to engage constituents in education policy discussions and analyze feedback captured through constituent engagement. AIR also has experience collecting constituent feedback through user-friendly surveys on various platforms. Surveys allow for a more focused approach to capturing public feedback about proposed standards. AIR experts analyze survey data to identify major themes to assist in the standards development process. 

AIR has extensive expertise and experience supporting states, particularly SEAs, in the development and revision of standards. AIR guides SEA through this complex process from start to finish. We apply evidence-informed processes and support the creation of guidance and communications that SEA leaders can use to ensure that new standards are high quality and that the process involves and effectively communicates to key constituents.  

• Image

  

  Applying research-informed practices to facilitate standards development: AIR has led the field in developing tools and resources for the standards revision process. This includes creation of the Standards Review and Revisions Toolkit and partnering with the Council of Chief State School Officers to publish the Guide to Developing and Implementing Social Studies Standards. We help clients utilize these practices throughout every step of the process—from analyzing stakeholder feedback before the revision process to using the Criteria for High-Quality Standards throughout the process to gauge the quality of draft standards while they are being written.  

• Convening constituents and experts to revise standards: AIR works with states to convene teams of teachers, school leaders, constituents, experts, businesses, and community members across the state to develop and evaluate standards. AIR designed processes and rubrics that states can utilize to form teams that are representational in terms of subject areas covered, grade levels, geographic location, and other factors.  

• Using tools and resources during the standards development process: AIR prepares and supports educators and constituents in developing and revising standards. We help build a common language for standards, using research and evidence to clarify the definition of high-quality academic standards, and collaboratively conduct interrater reliability. AIR supports educators and constituents in gathering international, national, and state academic standards and frameworks, and reviewing them with a critical lens, in preparation for conducting a review of their own existing standards. 

• Using evidence to ensure standards are high quality: AIR uses the Criteria for High-Quality Standards throughout the revision or review process to continuously check the standards against established criteria in order to ensure the standards are rigorous, coherent, and in line with best practices. AIR has also engaged subject matter experts to provide feedback to teams that are writing or reviewing standards to ensure multiple pieces of evidence are gathered so that teams can make data-informed decisions.  

Implementing new state standards requires the support of multiple education leaders—the SEA, REAs, school and district administrators, teachers, and others, all of whom hold important and varying roles in the education system. AIR uses evidence-informed processes and resources to create guidance to support standards implementation. 

• Image

  

  Working with constituents to design and monitor implementation plans: Implementation plans provide a roadmap for successful standards implementation. AIR helps education leaders and educators evaluate current needs and identify key constituents to engage in order to best support implementation. This information is then used to draft an implementation plan, which identifies the steps necessary for successful implementation and highlights key resources to support implementation.  

• Facilitating the development of resources that support implementation: Once new standards are developed, user-friendly resources need to be developed and/or rolled out to best support those standards. AIR has a history of creating and facilitating development of the various resources needed to support implementation.  

• Identifying high-quality instructional materials to support implementation: AIR has worked with several states and constituents to create rubrics and other tools to identify high-quality instructional materials aligned to new standards. This is a critical step in helping educators and district personnel understand the kinds of instructional shifts that are necessary to best implement the standards.  

• Aligning standards-based systems to assessment and accountability systems and policies: AIR conducts implementation research on standards-based reforms and accountability systems. This research examines how standards are implemented, their impact on student learning, and the tools and measures used to support implementation. The insights gained from implementation research support states and districts as they seek to create high-quality standards, develop aligned assessments, identify instructional tools and resources, and build accountability systems for districts and schools that are based on educational results.   

AIR works collaboratively with states and districts to conduct highly audits of curricula. AIR’s individual school curriculum audits are designed to provide information to school and district leadership teams about the curriculum and the instruction taking place in their buildings so that the teams can identify specific areas on which to focus improvement efforts.  

• Image

  

  Conducting curriculum audits to confirm evidence base: AIR has an extensive background in examining the effectiveness and evidence base of curricula. We incorporate an evidence review methodology that leverages existing processes and infrastructure from AIR’s Evidence Support Center (ESC), a practitioner-focused, evidence-based clearinghouse populated with educational programs, practices, products, and policies aligned to all four of the Every Student Succeeds Act (ESSA) tiers of evidence. We leverage our own resources as well as publicly available data from the What Works Clearinghouse and other sources to conduct comprehensive curriculum audits.  

• Evaluating curriculum for standards alignment: AIR’s curriculum audits consist of a mixed method evaluation, including analysis of student performance data, state and district policy documents, classroom observations, and interviews and focus groups with teachers, principals, students, parents, and district officials. Through this approach, AIR provides information about the gaps between the intended, enacted, and learned curriculum. The evaluation teams meet with state and district leadership teams to interpret the results in the audit documents, working together to identify strong themes and findings. The process culminates in a state- or district-owned action plan for addressing gaps in curriculum, instruction, and assessment and identifying approaches to closing those gaps.  

• Performing equity audits of curriculum: AIR brings a wealth of in-depth knowledge and understanding to conducting audits and has worked extensively with districts and schools of all sizes and demographics in urban, suburban, and rural settings. In addition to conducting equity audits, our highly collaborative audit work has included reviewing district special education programs, conducting comprehensive school needs assessments, and adapting established data collection protocols and grounding the work in a set of quality indicators. 

• Evaluating high-quality instructional materials including curriculum and digital learning platforms to align with standards: AIR has worked with clients ranging from SEAs to private entities to help them determine whether the curriculum and instructional materials they use are evidence-based, high-quality, and effective. This includes evaluating high-quality instructional materials (HQIM), developing rubrics and evidence guides to assess curriculum and instructional material alignment with applicable standards, evaluating the effectiveness of curricula and materials, and producing analysis reports to provide needed feedback to improve online and blended learning curricula.  

• Aligning and monitoring of curriculum implementation: AIR works with clients to understand and develop an action plan to monitor the facets of high-quality curriculum implementation. This ranges from conducting gap analysis of standards and enacted curricula to developing tools and other resources to monitor the implementation of effective and evidence-based curricula over time.  

AIR has a long history of developing and applying methodologies to analyze the content of standards and assessments. Our 30+ studies in this area over the past two decades have been undertaken for both comparative and evaluative purposes and have examined how various standards or assessments may be similar or different from one another, how standards or assessments may or may not change over time, and if standards and assessments that are intended to be aligned with each other are, indeed, aligned. Our capabilities also include analysis of assessment items, using cognitive labs with students, to validate their measurement of standards and support ESSA reporting requirements.  

• Image

  

  Comparing the content of standards and assessments across programs: For over 20 years, AIR has designed and led studies that compare the content of curricular standards and assessments. These studies have helped our clients understand how their standards or assessments relate to other standards or assessments—identifying the content and skills that may overlap and those that are unique. This in turn supports our clients’ ability to report on and further develop their own programs.  
   
  Our comparison studies have examined standards and assessments at the international, national, and state levels; we are leaders particularly in international-to-national comparisons. Programs we have compared have included each of the major international student assessments; National Assessment of Educational Progress (NAEP); national standards such as the Next Generation Science Standards (NGSS) and the Common Core State Standards (CCSS) in mathematics and English language arts and the standards and assessments of various U.S. states, territories, and the Department of Defense Education Activity (DoDEA). We review curricular standards, assessment frameworks, and assessment items with both qualitative and quantitative approaches, and we engage collaboratively with subject-matter experts inside and outside of AIR.  

• Examining grade progression within and across standards and assessments: AIR has experience conducting analyses related to grade-level progression in both cross-program and within-program studies and projects. For example, in several cross-program studies, AIR has examined the extent to which different assessments or standards cover content at different (higher or lower) grade levels. This was a key aspect of a study that compared the NGSS with the NAEP science and other frameworks and another AIR study that compared the NAEP mathematics framework with all 50 states’ mathematics standards.
  
  Cross-grade analyses have also been a central aspect of AIR's work supporting the completion of the curriculum questionnaires for the Trends in International Mathematics and Science Study (TIMSS), in which the TIMSS topics are rated in terms of their coverage in U.S. national and state standards. AIR researchers methodically analyzed TIMSS data across multiple assessment cycles. By examining student performance on a set of assessment items, they identified student understandings—and misunderstandings—related to two core concepts, gravity in physics and linear equations in mathematics. AIR researchers developed an Educator Tool for teachers to better understand students' misconceptions about these topics. AIR researchers plan to analyze other concepts and add them to the Educator Tool.

• Assessing alignment of standards and assessments over time and within programs: AIR also has experience analyzing standards and assessments for the specific purpose of evaluating alignment—or the degree to which a framework maintains consistency over time or an assessment measures the intended content and skills specified in related standards. For example, for the National Center for Education Statistics, AIR compared updated NAEP frameworks in two subjects (reading and mathematics) with the prior versions of the frameworks to analyze their similarities and differences and to ensure that changes did not compromise NAEP’s trend-reporting capability.  

• Validating assessments with a focus on ESSA reporting requirements: AIR’s work in analyzing the content of assessments extends to item analysis, specifically with a focus on determining the extent to which items measure what is intended based on either framework or related standards. For example, as part of AIR’s NAEP support work, we evaluated the 2005 science assessment items for potential use in the 2009 assessment as part of the item development process for the new 2009 framework. In upcoming work, we will again conduct cognitive labs to pilot test items developed according to an international assessment framework in a novel area: learning in a digital world.  

AIR works with federal, state, and local clients to support their efforts to use standards and assessment policies and practices to improve teaching and learning. 

• Image

  

  Using co-interpretation of assessment data or curriculum audits to analyze and develop action plans: AIR’s co-interpretation services include a facilitated and collaborative process that engages a variety of constituents to examine data, interpret the data, and have an active voice in determining the areas in which improvement is required. This collaborative approach is facilitated by AIR experts, but the power lies in how the process is facilitated, which encourages ownership of the results from those closest to the work.  

• Analyzing and reporting assessment results: AIR experts in large-scale surveys can analyze item-level data to determine common misconceptions across the learning progression. We have established a methodology to identify misconceptions or errors using distractors or incorrect/partial score categories and to tease out what teachers can learn from them and how to help students remove them. AIR researchers developed an Educator Tool for teachers to better understand students' misconceptions. Our work includes how misconceptions in the earlier grades can lead to lower performance or more misconceptions in higher grades. Our method is transparent and can be reproduced, earning trust in its results.  

• Developing tools for teachers to better access and use assessment results: Through sponsorship from the National Center for Education Statistics, AIR organizes the Research on International Studies in Education (RISE) Webinar Series, which showcases research using data from international studies and promotes sharing and discussion about how data-based evidence can be used to improve educational outcomes.

• Conducting research and policy analyses: In early 2023, AIR researchers and other research partners authored, "Looking Back to Look Forward: Quantitative and Qualitative Reviews of the Past 20 Years of K-12 Education Assessment and Accountability Policy," which provides a comprehensive analysis of policies related to high-quality academic standards, assessments, and accountability as tools for educational equity. On the state level, AIR technical assistance experts studied the Nebraska’s academic standards review policies and   drafted a report including, a summary of the review of other states’ processes, identification of key elements in a standards revision process, identification of Nebraska Department of Education strengths and gaps and recommendations to fill the gaps.