First open-access reproducibility project reveals roadblocks to performing replication studies

January 19, 2017 | Posted by Team in Reproducibility, Research |

Reproducibility has re-emerged at the forefront of public awareness this week, as the first five replication studies executed by the Reproducibility Project: Cancer Biology (RP:CB) have just been published in the open-access journal eLife.

The project is a collaboration between Science Exchange and the Center for Open Science (COS) to independently replicate key experiments from high-impact, published cancer biology studies. Unlike other assessments of reproducibility, the RP:CB studies and their results are completely open to the public.

The RP:CB studies highlight some of the practical considerations associated with replicating an existing study. For example, the RP:CB studies tackle the questions:

  • How do we define “replicate”?
  • What are the minimum requirements for reporting to enable a replication study?
  • How much time do replication studies take?
  • How much do replication studies cost?

The preliminary results of the RP:CB project, as eloquently summarized in The Atlantic, indicate that replication studies are lengthy and difficult.

Are the resources required for replication studies worth the benefits? Undoubtedly.

High-profile reports, from researchers at Amgen, Bayer, and elsewhere, illustrate the industry’s concerns that this lack of reproducibility might be driving the low success rate of drug candidates. Despite the costs of irreproducibility, researchers have few incentives to replicate studies. Results from replication studies have reduced chances of being published in traditional journals and are rarely prioritized for grant funding. The Reproducibility Project: Cancer Biology is helping initiate a cultural shift in the research community to motivate scientists to perform independent replication.

Our mission at Science Exchange is to facilitate collaboration between the world’s best scientific labs.We hope to play a big part in that cultural shift.

Still have questions? Download our FAQ that answers the most-asked questions on this project.

Two Papers Published in the Online Journal PeerJ; First Step to Reproducing Critical Prostate Cancer Findings.

September 22, 2015 | Posted by Keith Osiewicz in Research, Science Exchange News |

Science Exchange published two papers in PeerJ, the online journal, that are being funded by the Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative. This initiative seeks to address growing concerns about reproducibility in scientific research by conducting replications of recent papers in the field of prostate cancer.  It is a collaboration between the Prostate Cancer Foundation, the Movember Initiative, and Science Exchange.  These two papers represent the first step to reproducing the original experiments. Today’s papers are meant to report what the collaborators will do so the scientific community has a full understanding of the process. PeerJ will publish the final results of the replications.

The first paper, The Androgen Receptor Induces a Distinct Transcriptional Program in Castration-Resistant Prostate Cancer in Man by Sharma and colleagues, was originally published in Cancer Cell in 2013. Of thousands of targets for the androgen receptor (AR), the authors elucidated a subset of 16 core genes that were consistently down-regulated with castration and re-emerged with castration resistance. These 16 AR binding sites were distinct from those observed in cells in culture. The authors suggested that cellular context can have dramatic effects on downstream transcriptional regulation of AR binding sites. The present study will attempt to replicate Fig. 7C by comparing gene expression of the 16 core genes identified by Sharma and colleagues in xenograft tumor tissue compared to androgen treated LNCaP cells in vitro.

The second paper Androgen Receptor Splice Variants Determine Taxane Sensitivity in Prostate Cancer by Thadani-Mulero and colleagues was published in Cancer Research in 2014. The experiment that will be replicated is reported in Fig. 6A. Thadani-Mulero and colleagues generated xenografts from two prostate cancer cell lines; LuCaP 86.2, which expresses predominantly the ARv567 splice variant of the androgen receptor (AR), and LuCaP 23.1, which expresses the full length AR as well as the ARv7 variant. Treatment of the tumors with the taxane docetaxel showed that the drug inhibited tumor growth of the LuCaP 86.2 cells but not of the LuCaP 23.1 cells, indicating that expression of splice variants of the AR can affect sensitivity to docetaxel.

Labs listed on Science Exchange will perform the lab work. These labs include Nobel Life Sciences, ProNovus Bioscience LLC, and the Stem Cell and Xenograft Core at the University of Pennsylvania.

The Reproducibility Project: Cancer Biology Moves Forward

February 9, 2015 | Posted by Reproducibility Project Core Team in Reproducibility |

“Reproducibility is actually the heart of science. The fact that not everything is reproducible is not a surprise.” – Eric Lander, head of the Broad Institute at MIT in a recent Washington Post article. 

“We’re always in a gray area between perfect truth and complete falsehood,” The best researchers can do is edge closer to truth. – Giovanni Parmigiani, Dana-Farber Cancer Institute in a recent ScienceNews article

The Reproducibility Project, a collaboration between Science Exchange and the Center for Open Science, is independently replicating some of the most impactful studies in cancer biology. Along the way, not only will the collaboration shepherd 50 studies through the process of replication and meta-analysis, but it will also help to mature the discussion around reproducibility more generally. Where do the biggest challenges lie? What are some of the key predictors of whether experiments are reproducible? The answer to these questions will be critical as the reproducibility initiative gains traction.

Since December, experimental work has begun on four more replication studies, and three more Registered Reports have been published in eLife (with a fourth* accepted and on the way):

In total, eleven replications have begun or are poised to begin in the coming weeks. Read the rest of this entry »

First Registered Reports for the Reproducibility Project: Cancer Biology are published

December 17, 2014 | Posted by Reproducibility Project Core Team in Reproducibility |

We’re excited to announce that our introductory article and the first three of our Registered Reports have been published by our partner eLife.

In “An open investigation of the reproducibility of cancer biology research”, the Reproducibility Project: Cancer Biology core team details the impetus for and the specific goals of the project:

“The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility” (1).

The first three Registered Reports are:

In addition, Sean Morrison, director of the Children’s Medical Institute at the University of Texas–Southwestern and a senior editor at eLife, has written an editorial introducing the Reproducibility Project: Cancer Biology, highlighting the role this project could play in beginning to reform scientific discovery methods to maximize reproducibility. He notes that:

“to be responsible stewards of the public’s investment in this work we have to maximize the pace of discovery and the efficiency with which discoveries get translated to the benefit of patients. By gauging the fraction of high-impact results that are not reproducible, we can consider what further steps should be taken to promote good science….[M]easuring the magnitude of the problem with efforts like the Reproducibility Project: Cancer Biology is an important step in the right direction” (2).

You can find all five articles on eLife’s website: http://elifesciences.org/collections/reproducibility-project-cancer-biology Read the rest of this entry »

Science Exchange will be reproducing studies for the Movember Foundation-PCF Scientific Reproducibility Initiative

October 28, 2014 | Posted by Elizabeth in Reproducibility |

pcf1

We are proud to announce today that we have partnered with the Prostate Cancer Foundation (PCF), with funding from the Movember Foundation, to reproduce findings that have implications for prostate cancer patients. We will be collaborating with PCF to identify faster, high-impact biomedical findings that that can improve early detection and new cures.

PCF’s Chief Science Officer, Dr. Soule stated “This first-in-field foundation initiative is all about getting the smart stuff to patients quicker. We will see an acceleration of progress due to the mobilization of resources against the robust findings.”

Our Software Engineer Michael Kompanets with his Movember mustache.

Our Software Engineer Michael Kompanets with last year’s Movember mustache.

Science Exchange has a been long-time fan and supporter of PCF and the Movember Foundation (see picture to the right), so we are thrilled to be working with them to incorporate replication studies into their funding strategy. We will be utilizing the best practices that we’ve established for our Reproducibility Project: Cancer Biology to enable confirmation of high potential exploratory research results. Our hope is that by identifying robust reproducible results, we can accelerate prostate cancer research.

“The Movember Foundation is committed to accelerating the translation of promising discoveries into new tests and treatments,” said Paul Villanti, Executive Director of Programs, Movember Foundation. “Through quicker validation of the science, and if the science is true, we can help find new cures and prevent prostate cancer in more men at a faster rate. The Movember Foundation is confident that this initiative will play an important role in supporting this goal.” Read the rest of this entry »

Can Registered Reports help diagnose a reproducibility crisis?

October 14, 2014 | Posted by Reproducibility Project Core Team in Reproducibility |

There has been growing concern in the scientific community over the last several years about a lack of reproducible results in the biomedical research community. Recently, two large pharmaceutical companies (Amgen and Bayer) announced that they could only reproduce a small fraction of published preclinical cancer biology studies. These results have shocked the scientific community, and have lead to calls mandating an overhaul of both funding and publishing practices to address the crisis. The NIH, as well as the journals Nature and Science, are all proposing strategies to help improve the research process.

However, a major question remains: Why weren’t these experiments reproducible? Valid arguments exist suggesting scientists are falling prey to poor experimental design, flawed statistical analysis, and/or biased data interpretation, all of which can prevent their results from being replicable. However, there are many innocuous reasons why a particular experiment might fail to replicate the original results, from errors or changes in the protocol, to a lack of expertise in performing a particular technique, to unknown factors that produce variability in results. Unfortunately, it’s hard to draw conclusions from the Amgen and Bayer studies because these companies made none of their data or methods public.

The birth of the Reproducibility Project: Cancer Biology

We believe that in order to really understand the crisis in reproducibility, including its prevalence, scope and underlying causes, we need a large dataset of actual replication experiments. These replications must be conducted in a rigorously empirical fashion, using detailed protocols as close to the original study as possible, and conducted by expert scientists trained in the original techniques. Most importantly, the details of these replication datasets must be freely available to everyone.

These criteria led us to create the Reproducibility Project: Cancer Biology (RP:CB), a large-scale initiative to systematically replicate key findings from 50 highly impactful recent papers in the field of cancer biology. The project is a partnership between Science Exchange (and our network of expert service providers) and the Center for Open Science, and is funded through a grant from the Arnold Foundation, as well as through donations from many generous vendors. The goal of the project is to clarify the variety of challenges that exist for reproducibility, and encourage discussion of data-driven solutions from researchers themselves, as well as for policy makers at funding, publishing, and government institutions. To that end, all our findings will be published by the open-access journal eLife. Additionally, all of the methods, data, and results of the replication studies to be available for anyone to review on the Open Science Framework. Read the rest of this entry »

eLife will publish Reproducibility Project: Cancer Biology results

August 28, 2014 | Posted by Elizabeth in Reproducibility |

We are excited to announce that eLife has joined our partnership with the Center for Open Science to work on the Reproducibility Project: Cancer Biology!

eLife is an open access journal co-founded by the Howard Hughes Medical Institute, the Wellcome Trust and the Max Plank Institute. We are proud to have the work of the RP:CB published through them.

Each study in the RP:CB will undergo two rounds of review and publication. The first round will present the proposed replication plan to the public in the form of a Registered Report. This Registered Report will ensure that the proposed protocols have been reviewed by scientific and statistical experts prior to experimental work commencing. The completed work and all data will then be published as a Replication study. All data generated will be freely available to the public through eLife’s open access platform. Registered Reports are now under review by the eLife Board of reviewing editors and will be published in the eLife journal as available.

The Reproducibility Project: Cancer Biology aims to replicate key findings from 50 high profile papers from the field of cancer biology.

“We need an objective way to evaluate reproducibility,” said Randy Scheckman, who is the Editor-in-Chief of eLife and a Nobel prize winning cell biologist at the University of California- Berkeley. “This project is a valuable opportunity to generate a high-quality dataset to address questions about reproducibility constructively and rigorously.”

For more information, please see eLife (http://elifesciences.org/eLife-the-Center-for-Open-Science-and-Science-Exchange-partner-to-assess-the-reproducibility-of-cancer-biology-research) and the Reproducibility Project: Cancer Biology (http://validation.scienceexchange.com/#/cancer-biology). Read the rest of this entry »

Reproducibility Project: Cancer Biology will receive more than $500,000 worth of reagents and models

May 1, 2014 | Posted by Elizabeth in Science Exchange News |

I’m excited to announce that top scientific suppliers BioLegend, Charles River Laboratories, Corning Incorporated, DDC Medical, EMD Millipore, Harlan Laboratories, LI-COR Biosciences, Mirus Bio, Novus Biologicals, and Sigma-Aldrich will provide more than $500,000 worth of research reagents and models to support one of our validation projects, the Reproducibility Project: Cancer Biology. The donation of reagents and models will increase the number of replication experiments that can be conducted for the Reproducibility Project: Cancer Biology, a collaboration between Science Exchange and the Center for Open Science, supported by a $1.3 million grant from the Laura and John Arnold Foundation.

These companies chose to donate to the project, because they are committed to improving the quality of research and we are thrilled to have their support!

The Reproducibility Project: Cancer Biology is independently replicating 50 recent, high-impact cancer biology studies using the Science Exchange network of expert labs. The aim of the project is to use independent replication studies to identify best practices that maximize reproducibility and facilitate an accurate accumulation of knowledge, enabling potentially impactful novel findings to be built upon by the scientific community.

Studies from Amgen and Bayer report that the majority of published results cannot be independently reproduced, but there has been no open systematic review of replication in cancer biology. The Reproducibility Project: Cancer Biology will generate an open replication dataset made available on the Open Science Framework that can be used to examine the rate of reproducibility in this field and to study factors associated with the reproducibility of experimental results.

We continue to be amazed by the wide-ranging support for this project from the scientific community – thank you so much!

Of course, the more scientific supplies that are donated the more we can get done, so if you are involved with a company that is interested in donating please email me here.

About the author

Elizabeth Iorns is the CEO of Science Exchange and Director of the Reproducibility Initiative. Elizabeth conceived the idea for Science Exchange while an Assistant Professor at the University of Miami and as CEO she drives the company’s vision, strategy and growth. She is passionate about creating a new way to foster scientific collaboration that will break down existing silos, democratize access to scientific expertise and accelerate the speed of scientific discovery. Elizabeth has a Ph.D. in Cancer Biology from the Institute of Cancer Research in London, and conducted postdoctoral research in Cancer Biology from the University of Miami’s Miller School of Medicine where her research focused on identifying mechanisms of breast cancer development and progression.

Reproducibility through peer review

April 16, 2014 | Posted by Guest in Reproducibility |

This week we are featuring a guest post on how peer review can improve reproducibility. Check out Aimee Whitcroft from Publons’ thoughts below.

There has been much talk over the last few years about the fact that most research, particularly in the medical fields, may not be reproducible – a stunning waste of time and resources.

At Publons, we’ve been following the crisis closely, and we at think improved peer review is a vital first step towards reproducibility in academic and scientific literature. Read the rest of this entry »

Reproducing the STAP Stem Cell Method

March 13, 2014 | Posted by Elizabeth in Reproducibility |

When the stimulus-triggered acquisition of pluripotency (STAP) stem cell papers were published there was tremendous excitement in the scientific community. The papers described a seemingly simple method to reprogram differentiated somatic cells into pluripotency  – a process that usually involves the addition of multiple transcription factors.

The controversy around the papers comes from two separate issues. The initial controversy concerns the images submitted by the authors. First, an image used in Dr. Obokata’s doctoral thesis may have also been used in the Nature papers. However, the image from her thesis was from different experiments and time periods than those reported in the Nature paper. Secondly, a lane in their genomic analysis gel seems to be spliced. Lastly, images from two different placentas look nearly identical. Questionable images are a red flag, and this may be what causes the papers to be retracted.

But the larger issue brought up by these papers is reproducibility, which is much more complex. While it is terrific to see the crowdsourced replication attempts reported on the Knoepfler blog, the attempts did not use the same cells as those reported in the original studies, thus limiting interpretation of the attempts as replications. Read the rest of this entry »

About Science Exchange

We are transforming scientific collaboration by creating a marketplace where scientists can order experiments from the world's top labs.

Check the Science Exchange blog for updates, opinions, guest posts and the latest happenings at Science Exchange HQ!

Visit Science Exchange →

Subscribe to the blog
Never miss a post! Science Exchange blog posts delivered right to your inbox.
Thank you for joining the SciEx revolution!
Powered By WPFruits.com