First open-access reproducibility project reveals roadblocks to performing replication studies

January 19, 2017 | Posted by Team in Reproducibility |

Reproducibility has re-emerged at the forefront of public awareness this week, as the first five replication studies executed by the Reproducibility Project: Cancer Biology (RP:CB) have just been published in the open-access journal eLife.

The project is a collaboration between Science Exchange and the Center for Open Science (COS) to independently replicate key experiments from high-impact, published cancer biology studies. Unlike other assessments of reproducibility, the RP:CB studies and their results are completely open to the public.

The RP:CB studies highlight some of the practical considerations associated with replicating an existing study. For example, the RP:CB studies tackle the questions:

  • How do we define “replicate”?
  • What are the minimum requirements for reporting to enable a replication study?
  • How much time do replication studies take?
  • How much do replication studies cost?

The preliminary results of the RP:CB project, as eloquently summarized in The Atlantic, indicate that replication studies are lengthy and difficult.

Are the resources required for replication studies worth the benefits? Undoubtedly.

High-profile reports, from researchers at Amgen, Bayer, and elsewhere, illustrate the industry’s concerns that this lack of reproducibility might be driving the low success rate of drug candidates. Despite the costs of irreproducibility, researchers have few incentives to replicate studies. Results from replication studies have reduced chances of being published in traditional journals and are rarely prioritized for grant funding. The Reproducibility Project: Cancer Biology is helping initiate a cultural shift in the research community to motivate scientists to perform independent replication.

Our mission at Science Exchange is to facilitate collaboration between the world’s best scientific labs.We hope to play a big part in that cultural shift.

Still have questions? Download our FAQ that answers the most-asked questions on this project.

The Importance of Replication Studies

July 28, 2016 | Posted by Team in Reproducibility |

My TEDMED talk about scientific reproducibility was released today, so I wanted to take the opportunity to provide some additional thoughts about the importance of replication studies.

Every year, billions of dollars are spent funding biomedical research, resulting in more than one million new publications presenting promising new results. This research is the foundation upon which new therapies will be developed to enhance health, lengthen life, and reduce the burdens of illness and disability.

In order to build upon this foundational research, these results must be reproducible. Simply put, this means that when an experiment is repeated, similar results are observed. Over the last five years, multiple groups have raised concerns over the reproducibility of biomedical studies, with some estimates indicating only ~20% of published results may be reproducible (Scott et al. 2008, Gordon et al. 2007, Prinz et al. 2011, Steward et al. 2012, Begley and Ellis 2012). The National Institutes of Health (NIH), the largest public funder of biomedical research, has stated, “There remains a troubling frequency of published reports that claim a significant result, but fail to be reproducible. As a funding agency, the NIH is deeply concerned about this problem”.

Despite the growing concern over lack of reproducibility, funding for replication studies, the only way to determine reproducibility, is still absent. With no funding systematically allocated to such studies, scientists almost never conduct replication studies. It would be interesting to obtain the exact numbers, but it appears that last year the NIH allocated $0 to funding replication studies, out of a $30B+ budget. In the absence of replication studies, scientists end up wasting precious time and resources trying to build on a vast, unreliable body of knowledge.

It is easy to see why funders might shy away from funding replication studies. Funders want to demonstrate their “impact,” and it is tempting for them to solely focus on funding novel exploratory findings that can more easily be published in high profile journals. This is a mistake. Funders should instead focus on how to truly achieve their stated goals of enhancing health, lengthening life, and reducing the burdens of illness and disability. Although allocating a portion of funding towards replication studies would divert funds from new discoveries, it would enable scientists to efficiently determine which discoveries were robust and reproducible and which were not. This would allow more rapid advancements by allowing scientists to build upon the most promising findings and avoid wasting their time and funding pursuing non-robust results.

Some researchers find the idea of replicating previous studies unnecessary or even offensive. However, it is the responsibility of the scientific community, including funders, to work as quickly and cost effectively as possible to make progress. Introducing replication studies as part of the process provides an effective way to enable this.

If you would like to see funding specifically allocated for replication studies, please register your support. We will share this information with funders in the hope that it will encourage them to establish funding programs specifically for replication studies to improve the speed and efficiency of progress in biomedical research.

by Elizabeth Iorns, Ph.D.

CEO and Co-Founder

Science Exchange

About Science Exchange

 

Science Exchange is the world’s leading marketplace for outsourced research. The Science Exchange network of 3000+ scientific service providers has run the experiments for the major replication studies that have been conducted to date including the largest biomedical replication study undertaken (Reproducibility Project: Cancer Biology). Additional details are available here: https://www.scienceexchange.com/applications/reproducibility

 

References

  1. https://www.nih.gov/about-nih/what-we-do/budget#note
  2. http://www.ncbi.nlm.nih.gov/pubmed
  3. https://www.nih.gov/about-nih/what-we-do/mission-goals
  4. Scott et al. Amyotroph Lateral Scler. 9, 4-15 (2008)
  5. Gordon et al. Lancet Neurol. 6, 1045–1053 (2007)
  6. Prinz et al. Nat Rev Drug Discov. 10, 712 (2011)
  7. Stuart et al. Experimental Neurology 233, 597–605 (2012)
  8. Begley and Ellis. Nature. 483, 531-3 (2012)
  9. http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586
  10. http://www.nature.com/news/reproducibility-the-risks-of-the-replication-drive-1.14184

 

 

Science Exchange Won Four Grants from the International Initiative for Impact Evaluation (3ie), which is funded by the Gates Foundation, to reanalyze published articles dealing with health in developing countries.

October 28, 2015 | Posted by Keith Osiewicz in Reproducibility |

We won four grants to reanalyze four published journal articles in the field of public health. These grants covere four subjects: Cash transfers and sexually transmitted diseases, the necessary training of healthcare providers, circumcision and HIV and the affect of the US government’s spending in Africa on mortality. The grants come from 3ie, which is funded by the Gates Foundation. The work will be performed by a lab listed on Science Exchange: the University of Nebraska’s Center for Collaboration on Research Design and Analysis.

The grants cover various aspects of healthcare in developing countries. The first grant will analyze a paper published in Lancet in 2012 by Sarah Baird, Richard Garfein, Craig McIntosh and Berk Ozler. This paper, Effect of a Cash Transfer Programme for Schooling on Prevalence of HIV and Herpes Simplex Type 2 in Malawi: A Cluster Randomized Trial, showed that direct cash transfers decreased the prevalence of HIV and herpes simplex virus 2 (HSV-2) as well as the sexual behavior of the young women receiving transfers for 18 months. The second grant will reanalyze the 2012 paper published in Lancet, Task shifting of antiretroviral treatment from doctors to primary-care nurses in South Africa (STRETCH): a pragmatic, parallel, cluster-randomised trial. This paper examined the affect of using nurses, instead of scarce doctors to administer anti-retroviral treatment to patients with HIV. They found that STRETCH was not inferior to standard care and supports expanding the pool of ART prescribers beyond doctors to nurses. The third grant seeks to replicate the paper from 2011, Effect of circumcision of HIV-negative men on transmission of human papillomavirus to HIV-negative. This paper addresses an important question about HPV prevention and it evaluates male circumcision as a HPV prevention strategy among rural African HIV-negative women who lack both resources and vaccines that cover the existing high-risk HPV genotypes. The results from Wawer et al. (2011) provide strong support for use of male circumcision for HPV prevention and cervical neoplasia in HIV negative female partners. The fourth grant will examine a study published in JAMA in 2012 titled, HIV Development Assistance and Adult Mortality in Africa. This study investigates the relationship between increased funding to countries receiving aid through the President’s Emergency Plan for AIDS relief (PEPFAR) and adult mortality more generally. PEPFAR is the initiative developed by President George W Bush which increased funding to select countries from 2004 to 2010. The main finding of the paper is that PEPFAR countries had dramatically lower mortality than non-PEPFAR countries.

Dr. Nicole Perfito, Science Exchange lead for these projects says, “Using Science Exchange to gather the resources to reanalyze these experiments means it can be done faster and cheaper than would normally occur.

The analysis of these papers will bring extra scientific rigor to the study of health in developing countries. The University of Nebraska’s Center for Collaboration on Research Design and Analysis will do the analysis of these journal articles under the direction of Dr. Nicole Perfito at Science Exchange. Once the results are complete, these replicated studies will be published in peer-reviewed journals.

Two Papers Published in the Online Journal PeerJ; First Step to Reproducing Critical Prostate Cancer Findings.

September 22, 2015 | Posted by Keith Osiewicz in Reproducibility |

Science Exchange published two papers in PeerJ, the online journal, that are being funded by the Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative. This initiative seeks to address growing concerns about reproducibility in scientific research by conducting replications of recent papers in the field of prostate cancer.  It is a collaboration between the Prostate Cancer Foundation, the Movember Initiative, and Science Exchange.  These two papers represent the first step to reproducing the original experiments. Today’s papers are meant to report what the collaborators will do so the scientific community has a full understanding of the process. PeerJ will publish the final results of the replications.

The first paper, The Androgen Receptor Induces a Distinct Transcriptional Program in Castration-Resistant Prostate Cancer in Man by Sharma and colleagues, was originally published in Cancer Cell in 2013. Of thousands of targets for the androgen receptor (AR), the authors elucidated a subset of 16 core genes that were consistently down-regulated with castration and re-emerged with castration resistance. These 16 AR binding sites were distinct from those observed in cells in culture. The authors suggested that cellular context can have dramatic effects on downstream transcriptional regulation of AR binding sites. The present study will attempt to replicate Fig. 7C by comparing gene expression of the 16 core genes identified by Sharma and colleagues in xenograft tumor tissue compared to androgen treated LNCaP cells in vitro.

The second paper Androgen Receptor Splice Variants Determine Taxane Sensitivity in Prostate Cancer by Thadani-Mulero and colleagues was published in Cancer Research in 2014. The experiment that will be replicated is reported in Fig. 6A. Thadani-Mulero and colleagues generated xenografts from two prostate cancer cell lines; LuCaP 86.2, which expresses predominantly the ARv567 splice variant of the androgen receptor (AR), and LuCaP 23.1, which expresses the full length AR as well as the ARv7 variant. Treatment of the tumors with the taxane docetaxel showed that the drug inhibited tumor growth of the LuCaP 86.2 cells but not of the LuCaP 23.1 cells, indicating that expression of splice variants of the AR can affect sensitivity to docetaxel.

Labs listed on Science Exchange will perform the lab work. These labs include Nobel Life Sciences, ProNovus Bioscience LLC, and the Stem Cell and Xenograft Core at the University of Pennsylvania.

The Reproducibility Project: Cancer Biology – Experiments have begun

July 10, 2015 | Posted by Keith Osiewicz in Reproducibility |

The RPCB is a first of its kind attempt to directly replicate a subset of high-impact, pre-clinical cancer biology papers. Importantly, the methodology, quality control steps and replication data will be open and accessible on the Open Science Framework.

We are very excited to report that 13 Registered Reports have been accepted in eLife, and experiments from 12 of those studies are underway. These include:

  • Registered Report: BET bromodomain inhibition as a therapeutic strategy to target c-Myc
  • Registered Report: Interactions between cancer stem cells and their niche govern metastatic colonization
  • Registered Report: Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs
  • Registered Report: Discovery of preclinical validation of drug indications using compendia of public gene expression data
  • Registered Report: Intestinal inflammation targets cancer-inducing activity of the microbiota
  • Registered Report: Tumour vascularization via endothelial differentiation of glioblastoma stem-like cells
  • Registered Report: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors
  • Registered Report: Transcriptional amplification in tumor cells with elevated c-Myc
  • Registered Report: Senescence surveillance of pre-malignant hepatocytes limits liver cancer development
  • Registered Report: Widespread potential for growth-factor-driven resistance to anticancer kinase inhibitors
  • Registered Report: Tumour micro-environment elicits innate resistance to RAF inhibitors through HGF secretion
  • Registered Report: Melanoma genome sequencing reveals frequent PREX2 mutations

Before each replicating lab begins experimental work, critical reagents (often kindly shared by authors of the original studies) are quality checked. For example, all of the cell lines are authenticated and mycoplasma tested, plasmid sequences are sequenced, and rodents are pathogen tested. These quality check steps will be included on the Open Science Framework along with the data for the replication experiments themselves.

 

Tracking Our Progress

Keep track here as we continue to move projects forward. Our current status as of July 2015 is described below:

Reproducibility project progress

Phase:

  1. Replication experiments identified for each original paper
  2. Protocols drafted
  3. Protocols transferred to Registered Report format
  4. Review and feedback from original authors (requests for necessary reagents)
  5. Expert provider identified
  6. Registered Report peer reviewed at eLife
  7. Experimental work
  8. Experiment work is finished
  9. Replication experiments analyzed and evaluated
  10. Replication Study published in eLife

The Reproducibility Project: Cancer Biology Moves Forward

February 9, 2015 | Posted by Reproducibility Project Core Team in Reproducibility |

“Reproducibility is actually the heart of science. The fact that not everything is reproducible is not a surprise.” – Eric Lander, head of the Broad Institute at MIT in a recent Washington Post article. 

“We’re always in a gray area between perfect truth and complete falsehood,” The best researchers can do is edge closer to truth. – Giovanni Parmigiani, Dana-Farber Cancer Institute in a recent ScienceNews article

The Reproducibility Project, a collaboration between Science Exchange and the Center for Open Science, is independently replicating some of the most impactful studies in cancer biology. Along the way, not only will the collaboration shepherd 50 studies through the process of replication and meta-analysis, but it will also help to mature the discussion around reproducibility more generally. Where do the biggest challenges lie? What are some of the key predictors of whether experiments are reproducible? The answer to these questions will be critical as the reproducibility initiative gains traction.

Since December, experimental work has begun on four more replication studies, and three more Registered Reports have been published in eLife (with a fourth* accepted and on the way):

In total, eleven replications have begun or are poised to begin in the coming weeks. Read the rest of this entry »

First Registered Reports for the Reproducibility Project: Cancer Biology are published

December 17, 2014 | Posted by Reproducibility Project Core Team in Reproducibility |

We’re excited to announce that our introductory article and the first three of our Registered Reports have been published by our partner eLife.

In “An open investigation of the reproducibility of cancer biology research”, the Reproducibility Project: Cancer Biology core team details the impetus for and the specific goals of the project:

“The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility” (1).

The first three Registered Reports are:

In addition, Sean Morrison, director of the Children’s Medical Institute at the University of Texas–Southwestern and a senior editor at eLife, has written an editorial introducing the Reproducibility Project: Cancer Biology, highlighting the role this project could play in beginning to reform scientific discovery methods to maximize reproducibility. He notes that:

“to be responsible stewards of the public’s investment in this work we have to maximize the pace of discovery and the efficiency with which discoveries get translated to the benefit of patients. By gauging the fraction of high-impact results that are not reproducible, we can consider what further steps should be taken to promote good science….[M]easuring the magnitude of the problem with efforts like the Reproducibility Project: Cancer Biology is an important step in the right direction” (2).

You can find all five articles on eLife’s website: http://elifesciences.org/collections/reproducibility-project-cancer-biology Read the rest of this entry »

Science Exchange will be reproducing studies for the Movember Foundation-PCF Scientific Reproducibility Initiative

October 28, 2014 | Posted by Elizabeth in Reproducibility |

pcf1

We are proud to announce today that we have partnered with the Prostate Cancer Foundation (PCF), with funding from the Movember Foundation, to reproduce findings that have implications for prostate cancer patients. We will be collaborating with PCF to identify faster, high-impact biomedical findings that that can improve early detection and new cures.

PCF’s Chief Science Officer, Dr. Soule stated “This first-in-field foundation initiative is all about getting the smart stuff to patients quicker. We will see an acceleration of progress due to the mobilization of resources against the robust findings.”

Our Software Engineer Michael Kompanets with his Movember mustache.

Our Software Engineer Michael Kompanets with last year’s Movember mustache.

Science Exchange has a been long-time fan and supporter of PCF and the Movember Foundation (see picture to the right), so we are thrilled to be working with them to incorporate replication studies into their funding strategy. We will be utilizing the best practices that we’ve established for our Reproducibility Project: Cancer Biology to enable confirmation of high potential exploratory research results. Our hope is that by identifying robust reproducible results, we can accelerate prostate cancer research.

“The Movember Foundation is committed to accelerating the translation of promising discoveries into new tests and treatments,” said Paul Villanti, Executive Director of Programs, Movember Foundation. “Through quicker validation of the science, and if the science is true, we can help find new cures and prevent prostate cancer in more men at a faster rate. The Movember Foundation is confident that this initiative will play an important role in supporting this goal.” Read the rest of this entry »

Can Registered Reports help diagnose a reproducibility crisis?

October 14, 2014 | Posted by Reproducibility Project Core Team in Reproducibility |

There has been growing concern in the scientific community over the last several years about a lack of reproducible results in the biomedical research community. Recently, two large pharmaceutical companies (Amgen and Bayer) announced that they could only reproduce a small fraction of published preclinical cancer biology studies. These results have shocked the scientific community, and have lead to calls mandating an overhaul of both funding and publishing practices to address the crisis. The NIH, as well as the journals Nature and Science, are all proposing strategies to help improve the research process.

However, a major question remains: Why weren’t these experiments reproducible? Valid arguments exist suggesting scientists are falling prey to poor experimental design, flawed statistical analysis, and/or biased data interpretation, all of which can prevent their results from being replicable. However, there are many innocuous reasons why a particular experiment might fail to replicate the original results, from errors or changes in the protocol, to a lack of expertise in performing a particular technique, to unknown factors that produce variability in results. Unfortunately, it’s hard to draw conclusions from the Amgen and Bayer studies because these companies made none of their data or methods public.

The birth of the Reproducibility Project: Cancer Biology

We believe that in order to really understand the crisis in reproducibility, including its prevalence, scope and underlying causes, we need a large dataset of actual replication experiments. These replications must be conducted in a rigorously empirical fashion, using detailed protocols as close to the original study as possible, and conducted by expert scientists trained in the original techniques. Most importantly, the details of these replication datasets must be freely available to everyone.

These criteria led us to create the Reproducibility Project: Cancer Biology (RP:CB), a large-scale initiative to systematically replicate key findings from 50 highly impactful recent papers in the field of cancer biology. The project is a partnership between Science Exchange (and our network of expert service providers) and the Center for Open Science, and is funded through a grant from the Arnold Foundation, as well as through donations from many generous vendors. The goal of the project is to clarify the variety of challenges that exist for reproducibility, and encourage discussion of data-driven solutions from researchers themselves, as well as for policy makers at funding, publishing, and government institutions. To that end, all our findings will be published by the open-access journal eLife. Additionally, all of the methods, data, and results of the replication studies to be available for anyone to review on the Open Science Framework. Read the rest of this entry »

eLife will publish Reproducibility Project: Cancer Biology results

August 28, 2014 | Posted by Elizabeth in Reproducibility |

We are excited to announce that eLife has joined our partnership with the Center for Open Science to work on the Reproducibility Project: Cancer Biology!

eLife is an open access journal co-founded by the Howard Hughes Medical Institute, the Wellcome Trust and the Max Plank Institute. We are proud to have the work of the RP:CB published through them.

Each study in the RP:CB will undergo two rounds of review and publication. The first round will present the proposed replication plan to the public in the form of a Registered Report. This Registered Report will ensure that the proposed protocols have been reviewed by scientific and statistical experts prior to experimental work commencing. The completed work and all data will then be published as a Replication study. All data generated will be freely available to the public through eLife’s open access platform. Registered Reports are now under review by the eLife Board of reviewing editors and will be published in the eLife journal as available.

The Reproducibility Project: Cancer Biology aims to replicate key findings from 50 high profile papers from the field of cancer biology.

“We need an objective way to evaluate reproducibility,” said Randy Scheckman, who is the Editor-in-Chief of eLife and a Nobel prize winning cell biologist at the University of California- Berkeley. “This project is a valuable opportunity to generate a high-quality dataset to address questions about reproducibility constructively and rigorously.”

For more information, please see eLife (http://elifesciences.org/eLife-the-Center-for-Open-Science-and-Science-Exchange-partner-to-assess-the-reproducibility-of-cancer-biology-research) and the Reproducibility Project: Cancer Biology (http://validation.scienceexchange.com/#/cancer-biology). Read the rest of this entry »

About Science Exchange

We are transforming scientific collaboration by creating a marketplace where scientists can order experiments from the world's top labs.

Check the Science Exchange blog for updates, opinions, guest posts and the latest happenings at Science Exchange HQ!

Visit Science Exchange →

Subscribe to the blog
Never miss a post! Science Exchange blog posts delivered right to your inbox.
Thank you for joining the SciEx revolution!
Powered By WPFruits.com