What should the Cochrane Collaboration do about research that is, or might be, fraudulent?

  • By: John Carlisle, Nathan Pace, Jane Cracknell, Ann Møller, Tom Pedersen & Mathew Zacharias
  • On: May 31, 2013, 09:00
thumbnail image: What should the Cochrane Collaboration do about research that is, or might be, fraudulent?

The question in the title was prompted by an editorial in the European Journal of Anaesthesiology[1] concerning a Cochrane Review[2] that is stated to have "uncritically included" data under suspicion. We write together as an author of that review (J Carlisle) and the editors (J Cracknell, AM, NP, TP, MZ) of the Cochrane Anaesthesia Review Group (CARG). Our purpose is to correct some misunderstandings about the writing and editing of this review. We also challenge The Cochrane Collaboration to authorise a working group to develop and implement a methodology that estimates data integrity and standards for the inclusion or non-inclusion of studies under suspicion but not yet retracted. Systematic reviewers can potentially play a crucial role in the scientific community's struggle against the pollution of fraud. But potential is all that we will have until we help ourselves.

In June 2005 a draft version of the review, on drugs for preventing postoperative nausea and vomiting, included allegations of fraud and fabricated data against an author of some of the included studies. Now, in 2013, 183 (of 193) studies by that author, Dr Yoshitaka Fujii, have been retracted, making him the most prolific scientific fraudster in any discipline, ever. But in 2005, the review presented the CARG and others with two problems. Firstly, it was large. Secondly, there was the problem of the allegation of fraud against Fujii. In 2005 no journal had questioned or retracted a Fujii-authored publication, and no employer had investigated his integrity. After much discussion between the authors, the editors, and the Collaboration's then leadership, it was decided to change the text to convey the authors' meaning in a way that reduced the risk of successful litigation. However, the review included analyses of any interaction between intervention effects and authorship (Fujii versus others).

The retraction of 183 Fujii studies resulted from a combination of statistically unlikely data distributions (age, height, sex, etc), the lack of regulatory approval, and the absence of any experimental study records or data. In a previous scandal, again involving an anaesthetic researcher (Professor Joachim Boldt), 88 retractions were based largely on failures to gain appropriate ethical approval. The retention of details of the investigation's results by Boldt's employers meant that the consortium of journal editors that retracted these papers had to state "the research results are [not] per se fraudulent".[3] Boldt's studies, both retracted and active, continue to be incorporated into systematic reviews.[3] One of us (J Carlisle) has applied the techniques used on Fujii's studies[4] to 79 publications naming Boldt as an author, 44 of which have been retracted. Of the 35 studies not retracted, 10 displayed aberrant statistical characteristics that undermine the assumption that they are genuine (in comparison with 13/44 retracted studies). An additional study from the same researchers, but without Boldt as a co-author, was also retracted on the basis of procedural irregularity; this study also displays statistical anomalies.[5]

The Cochrane Collaboration should therefore consider the following questions:

  • How will it alert authors whose systematic reviews include retracted studies?
  • How will it flag retracted studies in the registry?
  • Should any retracted study be excluded from any systematic review?
  • Should any study be excluded if any of its authors has been shown to fabricate data, whether or not that particular study has been retracted?
  • Should any author who has co-authored with a fabricator be considered under suspicion, with their studies being subject to exclusion in sensitivity analyses?
  • What statistical 'barometer of doubt' might we use to gauge how much we should trust a study report, in addition to the methodology we currently use?
  • Should a particular group be tasked with developing statistical techniques, generally investigating the statistical trustiness of published studies, and specifically investigating cases referred by concerned Cochrane Review Groups?

We also support closer collaboration with journals against this common enemy, the Collaboration helping to remove published fraud, the journals helping to prevent its publication in the first place.[6]

John Carlisle1, Jane Cracknell2, Ann Møller3, Nathan Pace4, Tom Pedersen5, Mathew Zacharias6

1John Carlisle (john.carlisle@nhs.net), Consultant Anaesthetist and Intensivist, Torbay Hospital, Torquay, UK; 2Jane Cracknell (jane_cracknell@yahoo.com), Managing Editor, Cochrane Anaesthesia Review Group, Rigshospitalet, Copenhagen, Denmark; 3Ann Møller, Co-ordinating Editor, Cochrane Anaesthesia Review Group, and Professor, Department of Anaesthesiology, Herlev Hospital, Herlev, Denmark; 4Nathan Pace (n.l.pace@utah.edu), Co-ordinating Editor, Cochrane Anaesthesia Review Group, and Professor of Anesthesiology, University of Utah, Salt Lake City, USA; 5Tom Pedersen (doctp@yahoo.com), Director, Head and Orthopaedic Center, Rigshospitalet, Copenhagen, Denmark; 6Mathew Zacharias (mathew.zacharias@otago.ac.nz), Specialist Anaesthetist, Dunedin Hospital, Dunedin, New Zealand.

How to cite: Carlisle J, Cracknell J, Møller A, Pace N, Pedersen T, Zacharias M. What should the Cochrane Collaboration do about research that is, or might be, fraudulent? [editorial]. Cochrane Database of Systematic Reviews 2013;5:ED000060. dx.doi.org/10.1002/14651858.ED000060


1. Tramèr MR. The Fujii story: a chronicle of naive disbelief. European Journal of Anaesthesiology 2013; 30: 195-8. dx.doi.org/10.1097/EJA.0b013e328360a0db

2. Carlisle J, Stevenson CA. Drugs for preventing postoperative nausea and vomiting. Cochrane Database of Systematic Reviews 2006;3:CD004125. dx.doi.org/10.1002/14651858.CD004125.pub2

3. Wilkes MM, Navickis RJ. The Boldt affair: a quandary for meta-analysts. Anesthesiology News 2013;39:4.

4. Carlisle JB. The analysis of 168 randomised controlled trials to test data integrity. Anaesthesia 2012;67:521-37. dx.doi.org/10.1111/j.1365-2044.2012.07128.x

5. Ochmann C, Tuschy B, Beschmann R, Hamm F, Röhm KD, Piper SN. Supplemental oxygen reduces serotonin levels in plasma and platelets during colorectal surgery and reduces postoperative nausea and vomiting. European Journal of Anaesthesiology 2010;27:1036-43. dx.doi.org/10.1097/EJA.0b013e32833b04e4

6. Vlassov V, Groves T. The role of Cochrane Review authors in exposing research misconduct [editorial]. Cochrane Database of Systematic Reviews 2010;12:ED000015. dx.doi.org/10.1002/14651858.ED000015

Competing interests: The authors have declared (1) no receipt of payment or support in kind for any aspect of the article; (2) no financial relationships with any entities that have an interest related to the submitted work; (3) no other relationships or activities that could be perceived as having influenced, or giving the appearance of potentially influencing, what was written in the submitted work.

Image credit: Volker Steger/Science Photo Library

Contact the Editor in Chief, Dr David Tovey (dtovey@cochrane.org): Feedback on this editorial and proposals for future editorials are welcome.


The Cochrane Library - Independent high-quality evidence for health care decision making

Copyright © 2014 John Wiley & Sons, Ltd. All rights reserved