Why the Cochrane risk of bias tool should include funding source as a standard item

  • By: Lisa Bero
  • On: December 20, 2013, 12:00
thumbnail image: Why the Cochrane risk of bias tool should include funding source as a standard item

This editorial is based on a presentation given as part of a debate held at the Cochrane Methods Symposium in Québec City, Canada, October 2013. You can also read Jonathan Sterne's editorial presenting the case for not including funding source in the risk of bias tool.

Rigorous systematic reviews aim to identify and reduce any bias that can adversely influence interpretation of study outcomes. The Cochrane Handbook defines bias as "a systematic error, or deviation from the truth, in results or inferences" and goes on to describe a number of biases, including allocation concealment and blinding, that have been shown empirically to systematically influence results.[1] As Cochrane strives to improve its methods of bias assessment, we must consider the empirical data showing that funding source is a risk of bias.

A recent Cochrane Review provides evidence that there is bias associated with study funding sources.[2] The review examined the association of study funding source with clinical drug study outcomes, and the included studies defined 'favorable' results as those showing greater efficacy or less harm for the sponsor's product than for the comparator. Pharmaceutical industry-sponsored studies were more likely to have favourable efficacy results (risk ratio 1.32, 95% confidence interval 1.21 to 1.44) and harm results (risk ratio 1.87, 95% confidence interval 1.54 to 2.27) than studies not sponsored by industry. The review found mixed results for effect size. This is completely expected since the effect sizes of different drugs can vary widely and depend on dose, the outcome being measured, and other factors. For example, the influence of reporting biases on effect sizes has varies considerably among drugs.[3] Industry-sponsored studies have favourable conclusions more often than non-industry-sponsored studies, even when controlling for effect sizes.[4] Importantly, the consequence of industry funding is the same, independent of the effect size: industry-funded trials show treatments to be more efficacious and less harmful than non-industry funded trials.

The current Cochrane risk of bias tool is insufficient to assess bias related to study funding sources. The Cochrane tool includes sequence generation, allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting. The Cochrane Review that examined funding bias found no difference between industry-funded and non-industry-funded drug studies in sequence generation, concealment of allocation, or loss to follow-up.[2] Industry-funded studies did have lower risk of bias related to blinding, but this is not surprising as many drug studies are conducted for regulatory approval and must adhere to certain study design standards. Furthermore, the lower risk of performance and detection bias in industry trials would be expected to lead to less, not more, favourable results.

Since the observed funding bias cannot be fully explained by the study characteristics assessed with the Cochrane risk of bias tool, there must be other possible mechanisms by which this works. Bias related to funding source can results from systematic influences on how the study is actually conducted, the methodology of the study, whether the full results and analyses of the study are published, or a combination of these mechanisms.[5] Drug study results can be biased to maximise efficacy and minimise harm through such mechanisms as choice of inferior comparators (either by dose, drug or administration route),[6,7] biased coding of outcomes,[6] bias in how data are analysed,[8] and selective outcome reporting and publication bias.[9] Risks of bias are not mutually exclusive; a study may have multiple risks of bias and we may not be able to identify all of them. Funding source bias is a known bias that should be assessed.

Furthermore, bias may be related to funding source even when all studies are industry-funded. For head-to-head comparisons of statins with other drugs, funding bias is associated with which statin manufacturer funds the study, and not just industry funding per se. The head-to-head comparison was more likely to favour the sponsor's product than the competitor drug, even when other risks of bias in the studies were taken into account.[10] So, when there is competition between drugs in a class, funding bias can influence the outcome to favour the commercial interests of the sponsor.

If readers of Cochrane Reviews wanted to do their own assessment of bias related to funding source, they would find this difficult. Cochrane Reviews are not doing an adequate job of disclosing funding sources of included trials. Roseman et al have shown that the funding sources of the included trials were fully or partially disclosed in only 46 of 151 Cochrane Reviews and that these disclosures were scattered around seven different places in the reviews.[11] The MECIR (Methodological Expectations of Cochrane Intervention Reviews) reporting standards require details of funding sources for each included study and declarations of interest of the primary researchers of the included studies to be placed in the 'Characteristics of included studies' table.[12] It would be more appropriate to find this information along with other risks of bias in the risk of bias assessment table.

Those opposed to adding funding source to the Cochrane risk of bias tool argue that all mechanisms of bias can be identified, perfectly measured, and incorporated quantitatively in the results of meta-analysis. It is impossible or highly impractical to accurately measure all mechanisms of bias. For example, internal documents from the industry sponsor may be needed to detect how a study was deliberately biased in design or to determine whether outcomes were selectively reported or analysed. It is more realistic to assess bias in studies included in systematic reviews by using empirical methods to identify factors that are associated with research results, as has been done for funding source bias. The impact of the bias can be assessed descriptively or by using subgroup analysis, comparing industry-funded to non-industry-funded studies, as is commonly done in Cochrane Reviews. A bias should not be ignored even if we do not fully understand its mechanism, just as we should not ignore harms of interventions if we don't understand how they arose, or ignore the harm of smoking because we don't know how smoking causes cancer. Therefore, a study's funding source should be evaluated as an independent risk of bias.

In summary, the Cochrane risk of bias tool should include funding source as a standard item because:

1. Funding source fits the definition of bias

2. There is empirically-based evidence of bias related to funding source

3. The observed bias related to funding source cannot be captured by the risk of bias criteria currently assessed with the risk of bias tool

4. Risks of bias are not mutually exclusive

5. Bias may be related to funding source even when all studies are industry-funded.

Lisa A Bero

Co-Chair, Cochrane Collaboration Steering Group; Director, San Francisco Branch of the US Cochrane Center; and Professor, Department of Clinical Pharmacy and Institute for Health Policy Studies, University of California San Francisco, USA berol@pharmacy.ucsf.edu

How to cite: Bero LA. Why the Cochrane risk of bias tool should include funding source as a standard item [editorial]. Cochrane Database of Systematic Reviews 2013;(12):ED000075.

References

1. Higgins JPT, Altman DG, Sterne JAC (editors). Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from handbook.cochrane.org

2. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews 2012;12:MR000033. dx.doi.org/10.1002/14651858.MR000033.pub2

3. Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 2012;343:d7202. dx.doi.org/10.1136/bmj.d7202

4. Als-Nielsen B, ChenW, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? JAMA 2003;290(7):921–8. dx.doi.org/ 10.1001/jama.290.7.921

5. Odierna DH, Forsyth SR, White J, Bero LA. The cycle of bias in health research: a framework and toolbox for critical appraisal training. Accountability in Research 2013;20(2):127-141. dx.doi.org/10.1080/08989621.2013.768931

6. Gøtzsche PC. Deadly Medicines and Organised Crime: How Big Pharma Has Corrupted Healthcare. London: Radcliffe Publishing, 2013.

7. Safer DJ. Design and reporting modifications in industry-sponsored comparative psychopharmacology trials. Journal of Nervous and Mental Disease 2002;190:583–92.

8. Juni P, Dieppe PA. Are selective COX 2 inhibitors superior to traditional non steroidal anti-inflammatory drugs? BMJ 2002:324:1287. dx.doi.org/10.1136/bmj.324.7349.1287

9. Vedula SS, Bero L, Scherer RW, Dickersin K. Outcome reporting in industry-sponsored trials of Gabapentin for off-label use. New England Journal of Medicine 2009;361(20):1963–71. dx.doi.org/10.1056/NEJMsa0906126

10. Bero L, Oostvogel F, Bacchetti P, Lee K. Factors associated with findings of published trials of drug-drug comparisons: why some statins appear more efficacious than others. PLOS Medicine 2007;4(6):e184. dx.doi.org/10.1371/journal.pmed.0040184

11. Roseman M, Turner EH, Lexchin J, Coyne JC, Bero LA, Thombs BD. Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. BMJ 2012;345:e5155. dx.doi.org/10.1136/bmj.e5155

12. The Cochrane Collaboration. Standards for the reporting of new Cochrane Intervention Reviews Version 1.1 [updated 17 December 2012]. Available from www.editorial-unit.cochrane.org/mecir

Competing interests: The author has completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request) and declares that she is a co-author of some of the studies cited in this editorial.

Image credit: GP Kidd/Science Photo Library

Feedback: Please contact David Tovey, Cochrane Library Editor in Chief, with feedback on this editorial and proposals for future editorials.

Keywords:

The Cochrane Library - Independent high-quality evidence for health care decision making

Copyright © 2013 John Wiley & Sons, Ltd. All rights reserved