My TEDMED talk about scientific reproducibility was released today, so I wanted to take the opportunity to provide some additional thoughts about the importance of replication studies.
Every year, billions of dollars are spent funding biomedical research, resulting in more than one million new publications presenting promising new results. This research is the foundation upon which new therapies will be developed to enhance health, lengthen life, and reduce the burdens of illness and disability.
In order to build upon this foundational research, these results must be reproducible. Simply put, this means that when an experiment is repeated, similar results are observed. Over the last five years, multiple groups have raised concerns over the reproducibility of biomedical studies, with some estimates indicating only ~20% of published results may be reproducible (Scott et al. 2008, Gordon et al. 2007, Prinz et al. 2011, Steward et al. 2012, Begley and Ellis 2012). The National Institutes of Health (NIH), the largest public funder of biomedical research, has stated, “There remains a troubling frequency of published reports that claim a significant result, but fail to be reproducible. As a funding agency, the NIH is deeply concerned about this problem”.
Despite the growing concern over lack of reproducibility, funding for replication studies, the only way to determine reproducibility, is still absent. With no funding systematically allocated to such studies, scientists almost never conduct replication studies. It would be interesting to obtain the exact numbers, but it appears that last year the NIH allocated $0 to funding replication studies, out of a $30B+ budget. In the absence of replication studies, scientists end up wasting precious time and resources trying to build on a vast, unreliable body of knowledge.
It is easy to see why funders might shy away from funding replication studies. Funders want to demonstrate their “impact,” and it is tempting for them to solely focus on funding novel exploratory findings that can more easily be published in high profile journals. This is a mistake. Funders should instead focus on how to truly achieve their stated goals of enhancing health, lengthening life, and reducing the burdens of illness and disability. Although allocating a portion of funding towards replication studies would divert funds from new discoveries, it would enable scientists to efficiently determine which discoveries were robust and reproducible and which were not. This would allow more rapid advancements by allowing scientists to build upon the most promising findings and avoid wasting their time and funding pursuing non-robust results.
Some researchers find the idea of replicating previous studies unnecessary or even offensive. However, it is the responsibility of the scientific community, including funders, to work as quickly and cost effectively as possible to make progress. Introducing replication studies as part of the process provides an effective way to enable this.
If you would like to see funding specifically allocated for replication studies, please register your support. We will share this information with funders in the hope that it will encourage them to establish funding programs specifically for replication studies to improve the speed and efficiency of progress in biomedical research.
by Elizabeth Iorns, Ph.D.
CEO and Co-Founder
About Science Exchange
Science Exchange is the world’s leading marketplace for outsourced research. The Science Exchange network of 3000+ scientific service providers has run the experiments for the major replication studies that have been conducted to date including the largest biomedical replication study undertaken (Reproducibility Project: Cancer Biology). Additional details are available here: https://www.scienceexchange.com/applications/reproducibility
- Scott et al. Amyotroph Lateral Scler. 9, 4-15 (2008)
- Gordon et al. Lancet Neurol. 6, 1045–1053 (2007)
- Prinz et al. Nat Rev Drug Discov. 10, 712 (2011)
- Stuart et al. Experimental Neurology 233, 597–605 (2012)
- Begley and Ellis. Nature. 483, 531-3 (2012)