AU Professor and Students Join Program to Discover if Machines Can Determine Credibility of Research Claims

AU Professor and Students Join Program to Discover if Machines Can Determine Credibility of Research Claims
AU Associate Professor of Psychology Dr. Chris Chartier describes the first Psychology Accelerator Project study on face perception to his team of AU students who are serving as research assistants for the new DARPA program.

2/5/19 ASHLAND, Ohio – A collaborative team of scientists, organized by The Center for Open Science (COS) and Ashland University Associate Professor of Psychology Dr. Christopher R. Chartier and his students, has been selected to participate in the Defense Advanced Research Project Agency’s (DARPA) new program to evaluate if machines can determine the credibility of massive amounts of published scientific claims.

This program, titled Systematizing Confidence in Open Research and Evidence (SCORE), represents a major investment by DARPA to assess and improve the credibility of social and behavioral science research and further establishes AU faculty and students as key contributors to revolutionizing research practices in these fields. The three-year cooperative agreement will cost more than $7.6 million.

Chartier said he believes that the team’s work is likely to have downstream consequences for evaluating published findings across all scientific disciplines. “This is a great opportunity for me and my students to contribute to such an endeavor with global implications,” he said.

With the SCORE project, DARPA hopes to develop and deploy automated tools to assign ‘confidence scores’ to research results and claims. These confidence scores will be quantitative measures that should enable Department of Defense members or any consumer of social-behavioral science research to understand the degree to which a particular claim or result is likely to be reproducible or replicable. If successful, researchers, funders, policymakers and the public would have readily available information about the uncertainty associated with research evidence.

To achieve this lofty goal, the SCORE project team will create four primary products:  

  1. A massive database of approximately 30,000 claims from published papers in the social-behavioral sciences. The database will be enhanced with evidence about those claims that is automatically and manually extracted from the paper itself and merged into the database from other sources -- such as how often the work has been cited, and whether the data are openly accessible or the research was preregistered.  
  2. Experts will review and score about 3,000 of those claims in surveys, panels or prediction markets for their likelihood of being reproducible findings
  3. Teams of data scientists will use the database of information about the claims to generate algorithms (artificial intelligence) to score the same claims as the experts
  4. Hundreds of researchers will conduct replications on a sample of the claims to test the experts’ and algorithms’ ability to predict reproducibility.  

Attempting to conduct hundreds of replications is particularly daunting, but the SCORE project team has gained substantial experience coordinating large-scale replication projects with contributions from hundreds of members of the research community.

“The success of this project will rely on the enthusiastic and capable contributions of many researchers working toward an ambitious common goal,” said Chartier, who will lead the sourcing team for conducting the replications. “There is such tremendous community spirit for improving research practices. In terms of number of contributors, this project could become one of the largest studies ever conducted in the social-behavioral sciences.”

Chartier’s lead role in this project builds on previous investments by the University to create the Ashland University International Collaboration Research Center (AUICRC), based in the College of Arts and Sciences. Since 2017, Chartier and his AUICRC students have initiated and directed a research consortium with 497 collaborating researchers and 387 laboratories representing 60 countries.

Dr. Dawn Weber, dean of the College of Arts and Sciences, said, “This is very exciting news, especially because it provides an opportunity for Ashland undergraduate students to be involved in such a significant international research project.”

Ashland University students -- Dana Awlia, a junior majoring in Psychology; Nicholas Bloxsom, a senior majoring in Psychology and Criminal Justice; Emma Graves, a freshman majoring in Psychology and Music Performance; Mackenzi Gray, an incoming freshman majoring in Psychology;  Brynna Leach, a sophomore majoring in Psychology; Brianna Jurosic, a junior majoring in Psychology and Criminal Justice; Savannah Lewis, a junior majoring in Psychology; Bryce McClish, a senior majoring in Psychology; Tiffany Pryce, a senior majoring in Psychology, Public Relations and Strategic Communications; and Elizabeth Takacs, a senior majoring in Psychology -- will support Dr. Chartier’s efforts to assemble the necessary team, likely to exceed 600 individual researchers, to conduct the replication studies.

These student also will conduct a handful of the replications focused on psychological science findings in their AUICRC laboratory. Their involvement is likely to result in co-authorship on high impact scientific journal articles and presentations at regional, national and international conferences.

“DARPA’s investment signals the onset of the next phase of the reformation that is underway in the social and behavioral sciences” said Brian Nosek, executive director of the Center for Open Science. “For the last eight years, the research community has been scrutinizing the reproducibility of its findings and the quality of its research practices. Now, that learning is being translated into opportunities to improve research practices to accelerate the pace of discovery.”

Ashland University, which has been ranked in the top tier of colleges and universities in U.S. News and World Report’s National Universities category, is a mid-sized, comprehensive private university conveniently located a short distance from Akron, Cleveland and Columbus, Ohio. Religiously affiliated with the Brethren Church, Ashland University ( deeply values the individual student and offers a unique educational experience that combines the challenge of strong, applied academic programs with a faculty and staff who build nurturing relationships with their students. ###