The $1,000,000 Preregistration Challenge, launched one month ago by the Center for Open Science (COS), is testing how addressing scientists' incentives can elicit new behaviors and improve the reproducibility of published research. 1,000 researchers will earn $1,000 each for publishing the results of preregistered research.
"We needed a way to catch the attention of busy, practicing scientists," says David Mellor, Project Manager at COS, "to nudge them toward a small change in their workflow that could have a big impact."
Preregistration increases the credibility of statistical tests by having scientists specify details of their research plans in advance of conducting a study. That planning produces clear confirmatory tests of research hypotheses. This is distinct from exploratory analysis in which the scientist uses data to discover new hypotheses that could then be tested later. Both methods of research are important, and preregistration clarifies which is which. "In the complexity of doing research, it can be easy to blur the line between generating and testing hypotheses with data. The difference matters because if we are not careful, using statistical inference with exploratory analysis risks making 'discoveries' that are really just random noise in the data," said Courtney Soderberg a statistical consultant at COS.
"No one has ever conducted a nearly worldwide competition to encourage more robust research practices." said Jolene Esposito, Project Manager at COS. "For this to succeed, we needed to build a process that is easy to use and supports a wide variety of research approaches across scientific disciplines."
The Preregistration Challenge was designed with input from scientists in many domains, including biomedical, social, behavioral, and environmental sciences. This way, the Preregistration Challenge provides tools for supporting research planning addressing the key issues that are common across many scientific disciplines.
"So far, a majority of the scientists entering the competition are psychologists," says David Mellor, "which is not too surprising, given the attention these issues have received in that field. Preregistration is already seen in psychology as a way to address some of their reproducibility challenges. However, reproducibility issues are not isolated to only one field, so we are engaging researchers from more domains to preregister their research. We've had several entrants from political and biomedical science, and we want to encourage more."
When scientists enter a preregistration into the competition, they choose whether to make it public immediately, or embargo it for up to four years. Preregistrations must be made public by the time the study is published to be eligible for the $1000 award.
"So far, over 60% of researchers are choosing to submit public preregistrations," says Sara Bowman, Project Manager at COS. "A frequently heard concern in the field is the fear of being scooped by a competitor. Researchers may be recognizing that by preregistering publicly they are establishing a time-stamped certification of when the idea occurred, establishing their priority."
To be eligible for the award, each preregistration is reviewed for completeness and rule compliance. Though not as thorough as a complete peer review process, there were early concerns that this would slow down the natural pace of scientific investigations. "Reviewing each preregistration is definitely one of the riskier aspects of this competition," says David Mellor. "We could not allow this to slow down the pace of work in a researcher's lab, but we had to have a way to make sure prizes will only be awarded to research with sufficiently detailed plans. We've committed to providing reviews in 10 business days or less, and so far have been able to keep that to about 1-3 days."
This is the first time that a large number of preregistrations will be done outside of clinical trials, where it is required by law. This will give the scientific community an unprecedented opportunity to evaluate its effectiveness. "There's limited empirical data out there about how preregistration really affects the basic sciences," says Courtney Soderberg. "This will be the first time that we can really measure its effect on researchers and on the reproducibility of science."
The Preregistration Challenge is built on top of the Open Science Framework, a free scholarly commons maintained by COS. Researchers use the OSF for preregistration, project and collaboration management, and archiving of data, materials, and code. COS also supports a variety of services that nudge incentives toward more open and reproducible research such as badges to acknowledge open research practices, the TOP Guidelines for journals and funders to promote transparency, and Registered Reports- a publishing model in which peer review occurs prior to conducting the study.
"I love working on the cutting edge of improving science," says Mallory Kidwell, Project Coordinator at COS. "Our services aim to meet scientists where they are, providing tools to facilitate greater efficiency in their work while also making their science better." Results of the first preregistered studies in the Preregistration Challenge are expected in late 2016, and all 1000 awards are expected to be administered by the end of 2018.
###