January 19, 2017 | Posted by Team in Reproducibility |
Reproducibility has re-emerged at the forefront of public awareness this week, as the first five replication studies executed by the Reproducibility Project: Cancer Biology (RP:CB) have just been published in the open-access journal eLife.
The project is a collaboration between Science Exchange and the Center for Open Science (COS) to independently replicate key experiments from high-impact, published cancer biology studies. Unlike other assessments of reproducibility, the RP:CB studies and their results are completely open to the public.
The RP:CB studies highlight some of the practical considerations associated with replicating an existing study. For example, the RP:CB studies tackle the questions:
- How do we define “replicate”?
- What are the minimum requirements for reporting to enable a replication study?
- How much time do replication studies take?
- How much do replication studies cost?
The preliminary results of the RP:CB project, as eloquently summarized in The Atlantic, indicate that replication studies are lengthy and difficult.
Are the resources required for replication studies worth the benefits? Undoubtedly.
High-profile reports, from researchers at Amgen, Bayer, and elsewhere, illustrate the industry’s concerns that this lack of reproducibility might be driving the low success rate of drug candidates. Despite the costs of irreproducibility, researchers have few incentives to replicate studies. Results from replication studies have reduced chances of being published in traditional journals and are rarely prioritized for grant funding. The Reproducibility Project: Cancer Biology is helping initiate a cultural shift in the research community to motivate scientists to perform independent replication.
Our mission at Science Exchange is to facilitate collaboration between the world’s best scientific labs.We hope to play a big part in that cultural shift.
Still have questions? Download our FAQ that answers the most-asked questions on this project.
October 14, 2014 | Posted by Reproducibility Project Core Team in Reproducibility |
There has been growing concern in the scientific community over the last several years about a lack of reproducible results in the biomedical research community. Recently, two large pharmaceutical companies (Amgen and Bayer) announced that they could only reproduce a small fraction of published preclinical cancer biology studies. These results have shocked the scientific community, and have lead to calls mandating an overhaul of both funding and publishing practices to address the crisis. The NIH, as well as the journals Nature and Science, are all proposing strategies to help improve the research process.
However, a major question remains: Why weren’t these experiments reproducible? Valid arguments exist suggesting scientists are falling prey to poor experimental design, flawed statistical analysis, and/or biased data interpretation, all of which can prevent their results from being replicable. However, there are many innocuous reasons why a particular experiment might fail to replicate the original results, from errors or changes in the protocol, to a lack of expertise in performing a particular technique, to unknown factors that produce variability in results. Unfortunately, it’s hard to draw conclusions from the Amgen and Bayer studies because these companies made none of their data or methods public.
The birth of the Reproducibility Project: Cancer Biology
We believe that in order to really understand the crisis in reproducibility, including its prevalence, scope and underlying causes, we need a large dataset of actual replication experiments. These replications must be conducted in a rigorously empirical fashion, using detailed protocols as close to the original study as possible, and conducted by expert scientists trained in the original techniques. Most importantly, the details of these replication datasets must be freely available to everyone.
These criteria led us to create the Reproducibility Project: Cancer Biology (RP:CB), a large-scale initiative to systematically replicate key findings from 50 highly impactful recent papers in the field of cancer biology. The project is a partnership between Science Exchange (and our network of expert service providers) and the Center for Open Science, and is funded through a grant from the Arnold Foundation, as well as through donations from many generous vendors. The goal of the project is to clarify the variety of challenges that exist for reproducibility, and encourage discussion of data-driven solutions from researchers themselves, as well as for policy makers at funding, publishing, and government institutions. To that end, all our findings will be published by the open-access journal eLife. Additionally, all of the methods, data, and results of the replication studies to be available for anyone to review on the Open Science Framework. Read the rest of this entry »
Follow Us