Why the Cochrane risk of bias tool should not include funding source as a standard item

  • By: Jonathan Sterne
  • On: December 20, 2013, 12:00
thumbnail image: Why the Cochrane risk of bias tool should not include funding source as a standard item

This editorial is based on a presentation given as part of a debate held at the Cochrane Methods Symposium in Québec City, Canada, October 2013. You can also read Lisa Bero's editorial putting the case for including funding source in the risk of bias tool.

Read comments on this editorial.

Since its introduction in 2008, The Cochrane Collaboration's risk of bias (RoB) tool has been used in hundreds of Cochrane and non-Cochrane systematic reviews. Its development was based on a combination of empirical and theoretical considerations, leading to a focus on risk of bias rather than study 'quality' and a division of assessments into six bias domains.[1] Despite the substantially increased workload for review authors compared with previous approaches, this major change to Cochrane Review methodology has been widely accepted.

The Collaboration is a diverse organisation, and a number of review authors and groups have freely adapted the RoB tool. Some adaptations, such as including an assessment of whether sample size calculations were reported, were explicitly rejected in the Cochrane Handbook chapter describing the RoB tool.[2] Others, such as including assessment of risk of bias due to early stopping, are the subject of ongoing debate and disagreement. Including the funding source as a standard item in the tool is an adaptation that has caused particular controversy.

There is widespread agreement about the importance of knowing who funded a study. The way that the medical research community addresses and documents conflicts of interest has been transformed over the last decade, and most journals now require detailed declarations, including a stipulation to err on the side of complete disclosure, to be completed by each author at the time a paper is submitted. These innovations are based on well-documented controversies such as the series of events leading to the withdrawal of the COX-2 inhibitor rofecoxib (Vioxx), and more recently the resistance from Roche to full release of data from trials of oseltamavir (Tamiflu), despite heroic efforts by Chris Del Mar, his colleagues in the Cochrane Acute Respiratory Infections Group, and others.[3] More than 61,000 individuals have signed the AllTrials petition (www.alltrials.net), and more than 400 organisations (including GlaxoSmithKline and about 140 patient groups) have signed up to support the campaign for all trials to be registered and all results to be reported.[4]

However, the problem with including funding source in the risk of bias tool and (I assume) classifying all pharmaceutical industry-funded studies as at high risk of bias, is that there is little evidence (so far that I am aware) that trial methods are more likely to be flawed if a trial is industry-funded. Nor am I aware of evidence that fraud is more likely if a trial is industry-funded. Indeed, industry-funded trials tend to be well-resourced, done by highly skilled and experienced professionals, and based on detailed and extensively documented standardised operating procedures.

In my opinion the problems associated with industry-funded trials lie mainly in two areas: selective reporting of outcomes and whole studies, and problematic choice of comparator. Unfortunately, assessment of selective reporting in the current RoB tool does not work well. Many review authors classify this item as 'unclear', reliability is low, and most importantly it makes little sense to classify all findings from a study as biased on the basis that it failed to report one or more particular results (eg, relating to an adverse effect). In such circumstances the risk of bias is in the meta-analysis examining the adverse effect rather than the study's contribution to the outcomes on which it does report. The issue is more akin to traditional publication bias, when eligible studies remain unpublished for reasons related to their results. The Collaboration needs a better-structured approach to assessment of the risk that a meta-analysis is affected by reporting biases, which should be completely or mainly separated from the RoB tool. Associated guidance should focus on searching for, identifying and accessing unpublished information; and include information on how to access and use trial protocols, trial registries and information available to regulators. Resulting risk of bias assessments would be at the level of the meta-analysis (for an outcome), not at the level of an individual trial. The focus of the RoB tool should remain on ways in which a trial's methods render its results at risk of bias.

There is no doubt that choice of comparator in industry-funded trials can pose a major problem to medical decision-makers. There is an obvious incentive for those with a commercial interest in the success of a new drug to plan a comparison that is likely to show it in a favourable light and facilitate regulatory approval and marketing campaigns. Too often, a new drug is compared with placebo or an inadequate alternative when the question of interest is whether it is better than its major competitor. However, the results of a well-conducted and fully reported trial with inappropriate comparator are not at risk of bias – they are simply of limited utility to those wishing to make informed medical decisions. Unhelpful or inappropriate choices of comparator should be addressed through sensible interpretations of pairwise meta-analyses, routine use of mixed treatment comparisons (network meta-analyses) that focus on the question "What is the best intervention from among all the candidates?", and increased use of methods that integrate effects on benefits and harms in order to facilitate informed treatment choices.

Transparent and structured assessments of conflict of interest in included trials should be a routine component of Cochrane systematic reviews. Such assessments relate mainly to the context in which the review results should be interpreted. It is vital to know whether most of the information comes from a company with a commercial interest in the intervention of interest, but this is not in itself a reason to dismiss the accumulated evidence. It may nonetheless be a reason to search particularly hard for unreported or selectively reported evidence and to carefully scrutinise the chosen comparators.

In summary, conflict of interest in reporting of medical research is a substantial and unresolved problem. Welcome progress has been made through better regulation of pharmaceutical industry interactions with the medical profession, increased use of online trial registries, better journal publication policies, and campaigns to make all trial results accessible. However, the suppression of trial results remains a major problem that is far from being solved. There are particular problems associated with pharmaceutical industry-funded research, but these should be dealt with by: (1) reporting and commenting on conflicts of interest as a standard component of Cochrane systematic reviews; (2) better procedures and an improved tool to assess reporting biases; and (3) more extensive use of mixed treatment comparisons. Adding source of funding as a bias domain in the risk of bias tool would send an extremely negative message to pharmaceutical industry colleagues with whom we should be happy to work, and it might have the unintended consequence of labelling high-quality trials as biased while diverting attention from appropriate solutions to problems associated with pharmaceutical industry trials.

Jonathan AC Sterne

Professor of Medical Statistics and Epidemiology, School of Social and Community Medicine, University of Bristol, UK jonathan.sterne@bristol.ac.uk

How to cite: Sterne JAC. Why the Cochrane risk of bias tool should not include funding source as a standard item [editorial]. Cochrane Database of Systematic Reviews 2013;(12):ED000076.

References

1. Cochrane Bias Methods Group. Assessing risk of bias in included studies. bmg.cochrane.org/assessing-risk-bias-included-studies (accessed on 18 December 2013)

2. Higgins JPT, Altman DG, Sterne JAC on behalf of the COchrane Statistical Methods Group and the Cochrane Bias Methods Group. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available at handbook.cochrane.org.

3. Doshi P. Neuraminidase inhibtors: the story behind the Cochrane Review. BMJ 2009;339:B5164. dx.doi.org/10.1136/bmj.b5164

4. All Trials Campaign. All trials registered and results reported. Available at www.alltrials.net/2013/all-trials-registered-and-results-reported (accessed on 18 December 2013).

Competing interests: The author has completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request) and declares no conflicts of interest.

Image credit: GP Kidd/Science Photo Library

Feedback: Please contact David Tovey, Cochrane Library Editor in Chief, with comments on this editorial and proposals for future editorials.

Comments

Comment by Peter Gøtzsche, 9 January 2014

Lisa Bero argued the case for the inclusion of funding source as a standard item in the Cochrane risk of bias tool.[1] It is easy to see that Bero wins this debate, as she provides solid empirical evidence for her standpoint, which Sterne does not. I outline here what the major problems are with Sterne's arguments. Sentences in quotation marks are taken from Sterne's editorial.

1. "There is little evidence (so far that I am aware) that trial methods are more likely to be flawed if a trial is industry-funded." Bero argues convincingly why the current five bias domains we use cannot explain why industry-funded trials are more positive than other trials, and she cites my book for flaws in coding, which can distort the results completely without leaving a trace of the misconduct, as we cannot get access to the raw data at drug companies.[2]

2. "Nor am I aware of evidence that fraud is more likely if a trial is industry-funded. Indeed, industry-funded trials tend to be well-resourced, done by highly skilled and experienced professionals, and based on detailed and extensively documented standardised operating procedures." This comment is misleading. It looks fine on the surface, but deliberate manipulations and even fraud during coding, data analysis, and writing of manuscripts are well documented in industry trials.[2] And it is very easy to understand why we cannot trust industry trials. The difference between an honest and a not-so-honest data analysis can be worth billions of euros on the world market, and the fraudulent trials of some of the COX-2 inhibitors for arthritis and the SSRIs for depression in children and adolescents are good examples of this.[2]

3. "In my opinion the problems associated with industry-funded trials lie mainly in two areas: selective reporting of outcomes and whole studies, and problematic choice of comparator." I agree that these are important problems but there are many others, some of which are more serious than cherry-picking, e.g. omitting lethal harms.[2]

4. "It is vital to know whether most of the information comes from a company with a commercial interest in the intervention of interest, but this is not in itself a reason to dismiss the accumulated evidence." We have not argued that all industry trials should be dismissed, only that industry funding is a separate source of bias.

5. "Welcome progress has been made through better regulation of pharmaceutical industry interactions with the medical profession, increased use of online trial registries, better journal publication policies, and campaigns to make all trial results accessible." In his book, Ben Goldacre calls these solutions "fake fixes".[3] Despite these developments, companies have continued to find ways to circumvent them. This has included resorting to criminal activities that have led to large fines that are still almost certainly insufficient deterrence against future misconduct. And most drug companies are continuing to fight tooth and nail against the new openness policy the European Medicines Agency (EMA) has launched. The two court cases against the EMA that two US drug companies, AbbVie and InterMune, have started are backed by the drug industry associations in both Europe and the United States.[4]

6. Sterne suggests "better procedures and an improved tool to assess reporting biases; and more extensive use of mixed treatment comparisons." Fake fixes again. Many of the tricks cannot be revealed by better tools, as Bero documents, and network meta-analyses are not the solution either, as they cannot resolve the issues with selective reporting, biased coding and other manipulations with the available data or the way the trials were designed.[2]

7. "Adding source of funding as a bias domain in the risk of bias tool would send an extremely negative message to pharmaceutical industry colleagues with whom we should be happy to work, and it might have the unintended consequence of labelling high-quality trials as biased while diverting attention from appropriate solutions to problems associated with pharmaceutical industry trials." Those who work in industry know very well that they have a tremendous conflict of interest when they are their own judges by conducting the trials themselves and not letting anyone outside the company see the data. As epidemiologist Jan Vandenbroucke has explained, this is not science.[2] It is marketing. Without the possibility of open debate about what the data shows, science simply ceases to exist. And why would we be happy to work with people in an industry that doesn't allow access to its data? As whistleblower Aubrey Blumsohn has stated, no self-respecting scientist could ever be expected to publish findings based on data to which they do not have free and full access.[2] And again, we only talk about a risk of bias, not bias per se in every industry funded trial, so Sterne's comment is also misleading in this respect.

Peter C Gøtzsche, Director, Nordic Cochrane Centre

References

1. Bero L. Why the Cochrane risk of bias tool should include funding source as a standard item [Editorial]. Cochrane Database of Systematic Reviews 2013;(12):ED000075.

2. Gøtzsche PC. Deadly Medicines and Organised Crime: How Big Pharma has Corrupted Health Care. London: Radcliffe Publishing, 2013.

3. Goldacre B. Bad Pharma. London: Fourth Estate; 2012.

4. Gøtzsche PC. AbbVie considers harms to be commercially confidential information: sign a petition. BMJ 2013;347:f7569. dx.doi.org/10.1136/bmj.f7569

Keywords:

The Cochrane Library - Independent high-quality evidence for health care decision making

Copyright © 2014 John Wiley & Sons, Ltd. All rights reserved