What Works Clearinghouse: Statistics, Website, and Training (SWAT)

The What Works Clearinghouse (WWC) is a leading federal source of evidence-based information about education programs, policies, and interventions that show promise for improving student outcomes.

Since becoming the the Statistics, Website, and Training (SWAT) contractor in 2018, AIR has assisted the Institute of Education Sciences (IES), in the U.S. Department of Education, in methodological discussions, training, and dissemination, and helps facilitate deliberations about the WWC Standards and Procedures across all WWC contracts.

These projects included:

• Updating the WWC Standards and Procedures handbooks;
• Including ESSA Evidence Tier tags and data;
• Revising online training materials;
• Conducting a WWC rapid review of evidence; and
• Maintaining and updating the WWC’s infrastructure.
   

Updating the WWC Standards and Procedures Handbooks

The WWC has published multiple updates to the Standards and Procedures Handbook since AIR became the SWAT contract.

• In January 2020, the WWC published Version 4.1 of the Standards and Procedures Handbooks. The updated Handbooks included new procedures for synthesizing findings in WWC products using meta-analytic techniques, estimating intervention effects in single-case design studies, creating effectiveness ratings for Intervention Reports, and other statistical and methodological innovations.
• In late 2020, the WWC published a supplement to the Handbooks that derived several new effect size standard error formulas and updated previously outdated calculations.
• In 2022, the WWC published the draft Version 5.0 of the Handbook with plans to finalize the release in the summer of 2022.
   

Including ESSA Evidence Tier Tags and Data

Prior to the 2018, the WWC’s website and products did not include ESSA Evidence Tier information. Since becoming the SWAT contract, AIR has helped the WWC launch several ESSA Evidence Tier initiatives, including:

• Revisions to the Data from WWC Study Reviews web portal to include additional filters and ESSA Evidence Tier information;
• Icons and graphics, as well as functional algorithms, to identify studies that meet ESSA Evidence Tiers on the Reviews of Individual Studies webpage; and
• Numerous infographics, webinars, and videos to inform audiences on how to interpret ESSA Evidence Tiers and use them effectively.
   

Revising Online Training Materials

The WWC’s rigor depends on the quality of the information selected from studies included within WWC products, a process the WWC refers to as “study reviews”. The WWC relies on individuals to conduct study reviews, so that the information can be translated and disseminated accurately. The individuals assigned to conduct the study reviews must therefore receive training to ensure high quality. AIR has helped advance this mission by:

• Updating the process for receiving certification as a WWC study reviewer;
• Creating online training materials and certification for Regression Discontinuity Designs as well as Single-Case Designs;
• Designing Version 4.1 re-certification tests and materials; and
• Producing webinars and videos across a range of methodological issues facing WWC study reviewers, such as: searching the literature, reviewing studies using the Online Study Review Guide, defining and extracting study context data, and understanding study confounds.
   

Conducting a WWC Rapid Review of Evidence

The onset of the COVID-19 pandemic resulted in an immediate transition of in-person to online, distance learning. The Institute of Education Sciences called on researchers and educators to bring forth evidence-based programs and practices that could support the use of technology in online classrooms. AIR assisted the Department in conducting what IES referred to as the “first-ever cooperative rapid evidence synthesis.” Through this process, AIR assisted in the conduct of:

• The search and screening of relevant distance learning studies;
• The peer-review of all eligible WWC study reviews;
• The creation of an easy to understand flow diagram infographic depicting the state of the field; and
• The production of an official WWC product: A What Works Clearinghouse rapid evidence review of distance learning programs (WWC 2021-005REV) and technical appendix.
   

Maintaining and Updating the WWC’s Infrastructure

Ensuring the WWC is a primary resource for all education stakeholders requires a team of professionals who have relevant experience and expertise across numerous impacted sectors. In addition to the projects listed above, AIR also assisted in various other activities, such as:

• Maintaining and updating the Online Study Review Guide and supporting documentation, which allows WWC reviewers to upload study review information in a reliable and efficient manner;
• Responding to WWC Help Desk questions about various issues related to conducting study reviews, the systematic review process, and how to identify appropriate resources. Through March 2022, the Help Desk team has responded to more than 3,200 individual requests for information, assistance, or feedback;
• Conducting peer-reviews of WWC products and study reviews, which ensures that the information listed on the WWC’s website and supported databases remains high-quality; and
• After a series of discussions with education stakeholders, AIR assisted in the re-launching of the WWC’s website to bring enhanced navigation and user experiences online. More updates and enhancements are planned for 2022 and 2023.