Lessons in reproducibility from the Organic Syntheses journal.
As we begin our work to make all areas of scientific research more reproducible with the Reproducibility Initiative and the Science Exchange validation service, I was really interested to discover the journal Organic Syntheses. Organic Syntheses is the first journal (that I am aware of) that requires reproducibility of a procedure to be confirmed prior to publication. Chemists that I’ve spoken to trust that the procedures and results described in Organic Syntheses are likely to be able to be reproduced in their own laboratories.
OrgSyn’s website states that “In order for a procedure to be accepted for publication, each reaction must be successfully repeated in the laboratory of a member of the Editorial Board at least twice, with similar yields (generally ±5%) and selectivity similar to that reported by the submitters.”
To learn more about their process, I interviewed Chuck Zercher, Associate Editor of Organic Syntheses and Professor in the Department of Chemistry at the University of New Hampshire http://unh.edu/chemistry/charles-k-zercher.
Q: How are procedures chosen for checking by OrgSyn?
A: Researchers submit their procedures for checking and publication in OrgSyn and the board of editors reviews whether the synthesis is of widespread importance and the likelihood that it will work prior to accepting it for checking. The board of editors also reviews the literature for important procedures and invites the researchers to submit their procedure for checking and publication in OrgSyn.
Q: What percentage of the procedures accepted for checking make it through the checking process?
A: 90-95%. ~30 checked procedures are published each year in OrgSyn.
Q: Why is the percentage of procedures that are successfully checked so high?
A: Because the submitter has provided full details of the procedure and has agreed to have it checked. Also, the board of editors is selective in terms of the procedures that they choose to check.
Q: What percentage of researchers invited to submit their procedure for checking don’t respond or decline to have their procedure checked by OrgSyn?
A: ~60 invitations are sent out each year, with ~30% of invitations not responded to or declined.
Q: What are the reasons stated for declining to have the procedure checked?
A: The main reasons are the time and resources required to scale up the procedure to work for synthesizing gram quantities (since many syntheses reported in the literature are only in the milligram quantity). Another reason frequently stated is that the authors believe that they have provided enough detail in the literature already, so people should be able to reproduce it if needed.
Q: Who does the checking?
A: The 12 labs of the members of the Organic Syntheses Board of Editors (available here: http://orgsyn.org/boe.asp). The board includes members from academia, and industry from North America, Europe and Asia.
Q: Who pays for the checking?
A: The materials for each check are paid for by OrgSyn or are donated by the checker’s organization. The cost of materials varies considerably, but is in the region of hundreds to thousands of dollars per check. Junior checkers (students associated with members of the Board of Editors) who conduct the replication experiments also receive an honorarium from OrgSyn for their efforts, but otherwise all labor is donated.
Q: Is the checking process anonymous?
A: Yes, the checker usually prefers to remain anonymous until the checking is completed. We often interact with the original submitter to get the synthesis details finalized to enable a successful check to be conducted and in that case, I act as the intermediate. Once the check is completed, the submitter gets to review the procedure and results and the checkers are made public.
Q: Who is the author on the publication of the procedure in OrgSyn?
A: The submitter is the author. The checkers are acknowledged by a statement “Checked by XXX”.
Many of the principles employed by OrgSyn are very similar to those envisaged by the Reproducibility Initiative (www.reproducibilityinitiative.org) to enable reproducible methods, reagents and results to be identified for the wider research community. In time, we hope to generate the same level of trust in results validated by the Reproducibility Initiative as OrgSyn generates for its validated procedures.
[about_box image=”http://thebenchapp.s3.amazonaws.com/wp-content/uploads/2012/03/Elizabeth-80.png”]Elizabeth Iorns is Co-Founder & CEO of Science Exchange. Elizabeth conceived the idea for Science Exchange while an Assistant Professor at the University of Miami and as CEO she drives the company’s vision, strategy and growth. She is passionate about creating a new way to foster scientific collaboration that will break down existing silos, democratize access to scientific expertise and accelerate the speed of scientific discovery. Elizabeth has a B.S. in Biomedical Science from the University of Auckland, a Ph.D. in Cancer Biology from the Institute of Cancer Research in London, and conducted postdoctoral research in Cancer Biology from the University of Miami’s Miller School of Medicine where her research focused on identifying mechanisms of breast cancer development and progression.[/about_box]