First open-access reproducibility project reveals roadblocks to performing replication studies

January 19, 2017 | Posted by Team in Reproducibility |

Reproducibility has re-emerged at the forefront of public awareness this week, as the first five replication studies executed by the Reproducibility Project: Cancer Biology (RP:CB) have just been published in the open-access journal eLife.

The project is a collaboration between Science Exchange and the Center for Open Science (COS) to independently replicate key experiments from high-impact, published cancer biology studies. Unlike other assessments of reproducibility, the RP:CB studies and their results are completely open to the public.

The RP:CB studies highlight some of the practical considerations associated with replicating an existing study. For example, the RP:CB studies tackle the questions:

  • How do we define “replicate”?
  • What are the minimum requirements for reporting to enable a replication study?
  • How much time do replication studies take?
  • How much do replication studies cost?

The preliminary results of the RP:CB project, as eloquently summarized in The Atlantic, indicate that replication studies are lengthy and difficult.

Are the resources required for replication studies worth the benefits? Undoubtedly.

High-profile reports, from researchers at Amgen, Bayer, and elsewhere, illustrate the industry’s concerns that this lack of reproducibility might be driving the low success rate of drug candidates. Despite the costs of irreproducibility, researchers have few incentives to replicate studies. Results from replication studies have reduced chances of being published in traditional journals and are rarely prioritized for grant funding. The Reproducibility Project: Cancer Biology is helping initiate a cultural shift in the research community to motivate scientists to perform independent replication.

Our mission at Science Exchange is to facilitate collaboration between the world’s best scientific labs.We hope to play a big part in that cultural shift.

Still have questions? Download our FAQ that answers the most-asked questions on this project.

Science Exchange will be reproducing studies for the Movember Foundation-PCF Scientific Reproducibility Initiative

October 28, 2014 | Posted by Elizabeth in Reproducibility |

pcf1

We are proud to announce today that we have partnered with the Prostate Cancer Foundation (PCF), with funding from the Movember Foundation, to reproduce findings that have implications for prostate cancer patients. We will be collaborating with PCF to identify faster, high-impact biomedical findings that that can improve early detection and new cures.

PCF’s Chief Science Officer, Dr. Soule stated “This first-in-field foundation initiative is all about getting the smart stuff to patients quicker. We will see an acceleration of progress due to the mobilization of resources against the robust findings.”

Our Software Engineer Michael Kompanets with his Movember mustache.

Our Software Engineer Michael Kompanets with last year’s Movember mustache.

Science Exchange has a been long-time fan and supporter of PCF and the Movember Foundation (see picture to the right), so we are thrilled to be working with them to incorporate replication studies into their funding strategy. We will be utilizing the best practices that we’ve established for our Reproducibility Project: Cancer Biology to enable confirmation of high potential exploratory research results. Our hope is that by identifying robust reproducible results, we can accelerate prostate cancer research.

“The Movember Foundation is committed to accelerating the translation of promising discoveries into new tests and treatments,” said Paul Villanti, Executive Director of Programs, Movember Foundation. “Through quicker validation of the science, and if the science is true, we can help find new cures and prevent prostate cancer in more men at a faster rate. The Movember Foundation is confident that this initiative will play an important role in supporting this goal.” Read the rest of this entry »

Can Registered Reports help diagnose a reproducibility crisis?

October 14, 2014 | Posted by Reproducibility Project Core Team in Reproducibility |

There has been growing concern in the scientific community over the last several years about a lack of reproducible results in the biomedical research community. Recently, two large pharmaceutical companies (Amgen and Bayer) announced that they could only reproduce a small fraction of published preclinical cancer biology studies. These results have shocked the scientific community, and have lead to calls mandating an overhaul of both funding and publishing practices to address the crisis. The NIH, as well as the journals Nature and Science, are all proposing strategies to help improve the research process.

However, a major question remains: Why weren’t these experiments reproducible? Valid arguments exist suggesting scientists are falling prey to poor experimental design, flawed statistical analysis, and/or biased data interpretation, all of which can prevent their results from being replicable. However, there are many innocuous reasons why a particular experiment might fail to replicate the original results, from errors or changes in the protocol, to a lack of expertise in performing a particular technique, to unknown factors that produce variability in results. Unfortunately, it’s hard to draw conclusions from the Amgen and Bayer studies because these companies made none of their data or methods public.

The birth of the Reproducibility Project: Cancer Biology

We believe that in order to really understand the crisis in reproducibility, including its prevalence, scope and underlying causes, we need a large dataset of actual replication experiments. These replications must be conducted in a rigorously empirical fashion, using detailed protocols as close to the original study as possible, and conducted by expert scientists trained in the original techniques. Most importantly, the details of these replication datasets must be freely available to everyone.

These criteria led us to create the Reproducibility Project: Cancer Biology (RP:CB), a large-scale initiative to systematically replicate key findings from 50 highly impactful recent papers in the field of cancer biology. The project is a partnership between Science Exchange (and our network of expert service providers) and the Center for Open Science, and is funded through a grant from the Arnold Foundation, as well as through donations from many generous vendors. The goal of the project is to clarify the variety of challenges that exist for reproducibility, and encourage discussion of data-driven solutions from researchers themselves, as well as for policy makers at funding, publishing, and government institutions. To that end, all our findings will be published by the open-access journal eLife. Additionally, all of the methods, data, and results of the replication studies to be available for anyone to review on the Open Science Framework. Read the rest of this entry »

Reproducing the STAP Stem Cell Method

March 13, 2014 | Posted by Elizabeth in Reproducibility |

When the stimulus-triggered acquisition of pluripotency (STAP) stem cell papers were published there was tremendous excitement in the scientific community. The papers described a seemingly simple method to reprogram differentiated somatic cells into pluripotency  – a process that usually involves the addition of multiple transcription factors.

The controversy around the papers comes from two separate issues. The initial controversy concerns the images submitted by the authors. First, an image used in Dr. Obokata’s doctoral thesis may have also been used in the Nature papers. However, the image from her thesis was from different experiments and time periods than those reported in the Nature paper. Secondly, a lane in their genomic analysis gel seems to be spliced. Lastly, images from two different placentas look nearly identical. Questionable images are a red flag, and this may be what causes the papers to be retracted.

But the larger issue brought up by these papers is reproducibility, which is much more complex. While it is terrific to see the crowdsourced replication attempts reported on the Knoepfler blog, the attempts did not use the same cells as those reported in the original studies, thus limiting interpretation of the attempts as replications. Read the rest of this entry »

Independent Antibody Validation to Improve Research Quality

July 30, 2013 | Posted by Team in Reproducibility |

validation_slide

Science Exchange partners with antibodies-online.com to independently validate commercial antibodies enabling researchers to choose high quality research reagents

PALO ALTO, Calif. — Press Release — Science Exchange, in partnership with the world’s largest marketplace for antibodies, antibodies-online (www.antibodies-online.com), announced today the launch of a program to independently validate thousands of commercial antibodies via the Science Exchange Independent Validation Service (www.scienceexchange.com/validation). This program will help scientists identify high quality antibodies, improving the quality of research results and preventing the waste of resources spent on ineffective antibodies.

“More than 70% of published research cannot be independently reproduced,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “This has significant consequences for our ability to make scientific advances. One cause of this serious problem is the quality of reagents used in research studies. Our antibody validation program will directly tackle this problem, enabling scientists to identify independently validated antibodies that they can trust for their research.”

The Reproducibility Initiative

August 14, 2012 | Posted by Team in Reproducibility |

Palo Alto, California – August 14, 2012 – Science Exchange, in partnership with the open-access publisher PLOS and open data repository figshare, announced today the launch of the Reproducibility Initiative (www.reproducibilityinitiative.org) – a new program to help scientists, institutions and funding agencies validate their critical research findings.

“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation—the Reproducibility Initiative directly tackles these missing pieces.”

The Reproducibility Initiative provides both a mechanism for scientists to independently replicate their findings and a reward for doing so.  Scientists who apply to have their studies replicated are matched with experimental service providers based on the expertise required.  The Initiative leverages Science Exchange’s existing marketplace for scientific services, which contains a network of over 1000 expert providers at core facilities and contract research organizations (CROs). “Core facilities and commercial scientific service providers are the solution to this problem,” said Dr. Iorns. “They are experts at specific experimental techniques, and operate outside the current academic incentive structure.”

Read the rest of this entry »

About Science Exchange

We are transforming scientific collaboration by creating a marketplace where scientists can order experiments from the world's top labs.

Check the Science Exchange blog for updates, opinions, guest posts and the latest happenings at Science Exchange HQ!

Visit Science Exchange →

Subscribe to the blog
Never miss a post! Science Exchange blog posts delivered right to your inbox.
Thank you for joining the SciEx revolution!
Powered By WPFruits.com