Incubating the Use of Artificial Intelligence for Conducting High-Quality Research Syntheses

Image
person working on laptop with overlay of data visualization

Findings from research syntheses inform policymakers, practitioners, and researchers in a variety of contexts and settings. Producing a high-quality research synthesis, however, requires vast resources because the process is largely without automation, forcing personnel to spends hours conducting manual, tedious, routine tasks and processes. AIR and its Methods of Synthesis and Integration Center (MOSAIC), along with institutional partners and subject matter expert consultants, seek to accelerate and transform the practice of research synthesis using breakthrough artificial intelligence (AI) tools and models, especially in the fields of natural language processing and large language models.
 

Our Vision

The long-term vision of this Mid-Scale Research Infrastructure (MSRI) Incubator project is to inject greater automation throughout the synthesis process; we aim to do so using AI applications and tools that can be built to reduce costs to users and use fair and unbiased algorithms. The short-term goal of this project is to lay the groundwork for this long-term vision, establishing partnerships and processes that will inform future potential MSRI projects.

Our integrative, community-based, and ethical vision contrasts with the disjointed existing efforts to augment research syntheses with AI. We propose to create freely available infrastructure via MetaReviewer that will enable broad accessibility and robust transparency. Unlike proprietary tools that are often driven by for-profit motives—with hefty subscription fees, limited transparency, closed-source code, and/or undisclosed poor performance at certain synthesis tasks—our comprehensive efforts will also support an integrated workflow for the entire synthesis process. Our vision is therefore novel and unique among other AI-based synthesis applications.
 

Project Objectives

The project team is guided by four objectives:

  1. To foster relationships and increase diverse participation across a broad range of established and newly burgeoning experts in the fields of research synthesis, AI, computer science, and STEM education.
  2. To identify the research infrastructure needs of synthesists and the research synthesis processes that will enable greatest gains in efficiency, trustworthiness, and usefulness.
  3. To create rapid testing and validating procedures necessary to understand the promise and pitfalls of available AI tools and models. 
  4. To develop an MSRI Implementation proposal that capitalizes on the newly formed relationships and identified needs, by the end of the second year. 

We envision a platform that has a core ethos of community engagement and open-source production. Our core insight is that standardized application programming interfaces and a strong open-source commitment will result in using the same back-end technical infrastructure across multiple platforms. The MetaReviewer platform will be our guiding focus for integrating existing efforts but we also seek to coordinate with other platforms for broader impact. Our proposed incubator will continue to seek these partnerships for mutual gain, ultimately helping to align common efforts and provide the infrastructure for bringing promising open-source prototypes to scale.
 

Incubator Team

Our team includes members from diverse backgrounds and organizations.

Project leaders:

Steering committee:

  • Mario Kubek, Assistant Professor, Georgia State University
  • Katie Shilton, NSF AI Institute: TRAILS Institute, University of Maryland
  • Doug Downey, Allen Institute for AI, Northwestern University
  • David Yeager, NSF Incubating Infrastructure for Experimentation on Inclusive STEM Teaching Practices, University of Texas at Austin

Network participants:


This project is supported by the National Science Foundation’s Division of Research on Learning (DRL-2425651).