There has been growing concern in the scientific community over the last several years about a lack of reproducible results in the biomedical research community. Recently, two large pharmaceutical companies (Amgen and Bayer) announced that they could only reproduce a small fraction of published preclinical cancer biology studies. These results have shocked the scientific community, and have lead to calls mandating an overhaul of both funding and publishing practices to address the crisis. The NIH, as well as the journals Nature and Science, are all proposing strategies to help improve the research process.
However, a major question remains: Why weren’t these experiments reproducible? Valid arguments exist suggesting scientists are falling prey to poor experimental design, flawed statistical analysis, and/or biased data interpretation, all of which can prevent their results from being replicable. However, there are many innocuous reasons why a particular experiment might fail to replicate the original results, from errors or changes in the protocol, to a lack of expertise in performing a particular technique, to unknown factors that produce variability in results. Unfortunately, it’s hard to draw conclusions from the Amgen and Bayer studies because these companies made none of their data or methods public.
The birth of the Reproducibility Project: Cancer Biology
We believe that in order to really understand the crisis in reproducibility, including its prevalence, scope and underlying causes, we need a large dataset of actual replication experiments. These replications must be conducted in a rigorously empirical fashion, using detailed protocols as close to the original study as possible, and conducted by expert scientists trained in the original techniques. Most importantly, the details of these replication datasets must be freely available to everyone.
These criteria led us to create the Reproducibility Project: Cancer Biology (RP:CB), a large-scale initiative to systematically replicate key findings from 50 highly impactful recent papers in the field of cancer biology. The project is a partnership between Science Exchange (and our network of expert service providers) and the Center for Open Science, and is funded through a grant from the Arnold Foundation, as well as through donations from many generous vendors. The goal of the project is to clarify the variety of challenges that exist for reproducibility, and encourage discussion of data-driven solutions from researchers themselves, as well as for policy makers at funding, publishing, and government institutions. To that end, all our findings will be published by the open-access journal eLife. Additionally, all of the methods, data, and results of the replication studies to be available for anyone to review on the Open Science Framework.
How does our process work?
So, how do you conduct rigorous, empirical replication experiments? We believe the answer lies in using Registered Reports, a new format for articulating experimental design, materials and methods, and planned statistical analyses prior to data collection. For the RP:CB, we are creating Registered Reports for each of the 50 studies being replicated. These Reports are shared with both the original authors and the replicating labs, both of which have the opportunity to provide input. The finalized Reports are sent to eLife for peer review, where a panel of expert scientists and statisticians can review the planned replication experiments. Once the Registered Reports are published, we begin experimental data collection.
By seeking to control for as many variables as possible up front (getting detailed protocol information from the original authors, finding experts to conduct each experiment, having experts review our intended protocols and analyses), we can hopefully collect meaningful data addressing reproducibility in the field of cancer biology, which will begin clarifying and addressing the issues surrounding reproducibility.
The 10 phases of a successful replication
In order to achieve rigorous reproducibility, each study must pass through the following phases:
Phase 1: A selection of key experiments are identified for each original paper
Phase 2: Protocols are drafted for each necessary experiment
Phase 3: Protocols are transferred to the Registered Report format
Phase 4: The protocols are shared with the original authors for review and feedback and requests for necessary reagents are made.
Phase 5: Each experiment is matched with an expert provider from the Science Exchange network of labs.
Phase 6: The Registered Report is peer reviewed through eLife
Phase 7: Experimental work is begun by the replicating lab
Phase 8: Experiment work is finished
Phase 9: The reproducibility of the replication experiments is analyzed and evaluated
Phase 10: The final Replication Study is published in eLife for each original paper
Tracking our progress
Each month, we will provide updates regarding our process, and we will detail how many studies are in each phase of the project. You can track our progress here. Our current status, as of October 2014, is detailed below:
While we have not begun experimentation for any of the projects yet, we are starting to successfully move projects through peer review! Stay tuned for next month’s progress from us (Joelle Lomax, Fraser Tan, and Tim Errington).