Deficiencies in the scientific community’s centralized peer review process can impact more than a researcher’s career. Faults in the process can ultimately affect the kinds of scientific discoveries that are made, the distribution of information about findings, the technology innovations that do – or don’t – result, and even the economic impact of scientific research work.
With that premise and a goal and plan to improve the process, School of Information Studies (iSchool) Assistant Professor Daniel Acuna has attracted a $531,339 grant from the National Science Foundation.
For the next three years, he and two co-researchers will determine ways to optimize scientific peer review by creating a better understanding of the factors that affect the process and developing insights and plans to improve it. Acuna is the principal investigator for the project. He is working with co-principal investigators Konrad P. Kording, Penn Integrated Knowledge Professor at the University of Pennsylvania, and James A. Evans, Director of Knowledge Lab, Professor of Sociology, and Director of the Computational Social Science program at the University of Chicago.
Acuna explains that some researchers have found that the scientific community’s centralized peer review process can be slow and low-quality, while other studies have quantified biases in the process, such as prejudices against certain ideas and inconsistencies in opinions from different groups of peers. Those deficiencies can delay or shorten the dissemination of important research, he says. Such issues also can have larger effects, and not simply on the careers of scientific researchers. More broadly, those issues can affect the types of research that gets done, thus the direction science takes, plus the emergence of technology innovations and even the impact of scientific research on the economy.
“This grant can help illuminate one of the oldest and perhaps least understood aspects of how modern scientific knowledge is disseminated,” Acuna says. “Several research groups have shown that scientists can be against novel ideas or ideas produced by distant scientific relatives. Moreover, scientists can be inconsistent, judging essentially similar pieces of work wildly differently. These problems percolate into the peer review system and can be exacerbated by its anonymity. We hope to address some of these issues.”
In its scope, the research analyzes factors that affect peer review outcomes and uses those to improve reviewer selection. Ultimately, plans call for the development of software that will help optimize reviewer assignments and evaluate the resulting models. The analysis is being done in the real-world context of a scientific journal, major scientific conferences, and massive open, online courses (MOOCs).
In its first phase, the project will quantify problems in bias, variance, timing, and quality of reviews, then identify bias as attributed to author and reviewer (such as age, gender, minority status, and visibility and centrality within the field). The second phase of research consists of assessing techniques to estimate the characteristics of potential reviewers. Then, the study will use that information to propose a review panel that is balanced in terms of reviewer choice, timing, and quality of the process. In the last segment, the study will test the techniques that automatically assign reviewers and analyze the output of the process.