Assessing risk of bias in randomised clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge

  • By: Asbjørn Hróbjartsson, Isabelle Boutron, Lucy Turner, Douglas G Altman & David Moher
  • On: April 30, 2013, 17:00
thumbnail image: Assessing risk of bias in randomised clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge

Randomised clinical trials are often inadequately reported and may be inadequately conducted. Any associated biases could impact seriously on the findings and conclusion of a systematic review. Authors of systematic reviews thus need to assess the risk of bias in included randomised clinical trials. In this 20th Anniversary editorial, we look at the evolution of guidance on appraising studies included in Cochrane Reviews.

Assessing the methodological 'quality' of included trials was addressed from the earliest days of The Cochrane Collaboration, although the phrase 'risk of bias' came into use later. In 1994 one of the first editions of the Cochrane Collaboration Handbook recommended that reviewers should routinely assess the adequacy of allocation concealment, and that they could consider assessing blinding and attrition, based on a seminal empirical study by Schulz and colleagues.[1] Over the next decade several Cochrane Review Groups developed different recommendations for assessing risk of bias. Of 50 Cochrane Review Groups surveyed in 2007, 41 recommended using specific trial characteristics to assess risk of bias and nine either recommended using a quality scale or made this optional. Most groups suggested assessing the randomisation procedure (including concealment of allocation), blinding, and attrition.[2]

In 2008 the Cochrane risk of bias tool was released with Review Manager 5.0, following three years of development. It included six characteristics: 'generation of the allocation sequence', 'concealment of the allocation sequence', 'blinding', 'incomplete outcome data', 'selective outcome reporting', and 'other bias'. Selective outcome reporting was included based on a landmark paper documenting a tendency for statistically significant trial outcomes to be selected for reporting.[3] Subsequent research has replicated this finding. In 2011 a revised version of risk of bias tool split blinding into 'blinding of participants and personnel' and 'blinding of outcome assessment'.[4]

Now is a good time to reflect on two decades of assessing risk of bias. The risk of bias tool provides a standardised approach, based on items selected on both theoretical and empirical grounds, and following broad consultations with clinical research methodologists. Furthermore, a more appropriate terminology has been developed emphasising 'risk of bias' instead of 'methodological quality', and the initial approach mainly based on a single trial characteristic to bias has matured into a structured multidimensional approach.[4]

The risk of bias tool is a comparatively recent development that still likely needs refinement. The modest inter-rater agreement rates[5] will hopefully be improved by modifications to the questions and enhanced training courses. However, authors of Cochrane Reviews tend to be reluctant in designating an overall risk of bias for each trial or outcome and also reluctant to incorporate the risk of bias assessment in analyses and conclusions. The next version of Review Manager, scheduled for release by the end of 2014, will enable authors to see the risk of bias table jointly with the forest plot, thus facilitating a cohesive interpretation of effects and risk of bias for each outcome.

However, risk of bias assessment has more fundamental challenges. Empirical analyses of bias in randomised trials typically rely on meta-epidemiological studies.[1,6] Such studies involve comparisons within several meta-analyses of the estimated treatment effects in trials with the characteristic present (such as adequate allocation generation) and trials without the characteristic (such as inadequate or unclear allocation generation). The risk of confounding in such comparisons is pronounced, as compared trials may differ for other reasons, such as allocation concealment, type of outcome, blinding, and the trial's sample size. Furthermore, reporting inadequacies in trial publications is an additional concern. It thus remains an open question whether inadequate allocation generation is truly causally linked to bias or whether it is an indirect marker of other factors associated with bias.

To establish reliable causal relationships of bias in observational studies may be even more difficult than to establish reliable causal relations in epidemiology in general. The assessment of risk of bias is to a large extent based on common sense and theoretical considerations, with an empirical basis of observational studies with a considerable risk of confounding.

This highlights a peculiar circularity. Meta-analysis of randomised clinical trials is the core methodology for reliable estimates of treatment effects, and is thus the core methodology for evidence-based medicine. This is partly based on the reasonable view that randomised trials are more reliable than observational studies in assessing effects of health care interventions. Still, the empirical evidence underlying the assessment of risk of bias in trials – an assessment necessary for ensuring that biased trials do not lead to biased systematic reviews – is based on observational studies.

An increasing number of meta-epidemiological studies report associations between a trial characteristic and exaggerated treatment effects: funding status,[7] number of centres participating in a trial,[8] early stopping of a trial,[9] and developing country status.[10] For many of these characteristics it is unclear whether they represent a unique bias, confounding, publication bias, spurious findings, or a combination of these and/or other unknown factors. It is, nonetheless, helpful to be aware of such associations, sometimes called meta-bias.[11]

Funding status is a major concern in randomised trials. The exaggerated effects reported for industry trials[7] may to some extent be explained as a result of publication bias or other characteristics included in the risk of bias tool, for example selective outcome reporting. However, companies that stand to gain financially by a positive result have substantial conflicts of interest when they control the planning, funding, conduct, and reporting of a trial. It is not clear that the risk of bias tool in its present version addresses this problem adequately.

It is important to assess risk of bias in randomised clinical trials included in systematic reviews. In the last 20 years risk of bias assessment in Cochrane Reviews has been refined several times. For the next two decades and beyond the process is likely to continue. The why is easy, the how is a challenge.

Asbjørn Hróbjartsson1, Isabelle Boutron2, Lucy Turner3, Douglas G Altman4, David Moher5, on behalf of the Cochrane Bias Methods Group (bmg.cochrane.org)

1Asbjørn Hróbjartsson (ah@cochrane.dk), Senior researcher, Nordic Cochrane Centre, Copenhagen, Denmark; 2Isabelle Boutron (isabelle.boutron@bch.aphp.fr), Assistant Professor, Centre d'Epidémiologie Clinique, Hôpital Hôtel-Dieu and Université Paris Descartes, Paris, France; 3Lucy Turner (lturner@ohri.ca), Statistician, Ottawa Hospital Research Institute, Ottawa, Canada; 4Douglas G Altman (doug.altman@csm.ox.ac.uk), Director, Centre for Statistics in Medicine, Oxford, UK; 5David Moher (dmoher@ohri.ca), Senior Scientist, Ottawa Hospital Research Institute, Ottawa, Canada.

How to cite: Hróbjartsson A, Boutron I, Turner L, Altman DG, Moher D. Assessing risk of bias in randomised clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge [editorial]. Cochrane Database of Systematic Reviews 2013;4:ED000058. dx.doi.org/10.1002/14651858.ED000058

References

1. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12. dx.doi.org/10.1001/jama.1995.03520290060030

2. Lundh A, Gøtzsche PC. Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Medical Research Methodology 2008;8:22. dx.doi.org/10.1186/1471-2288-8-22

3. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291(20):2457-65. dx.doi.org/10.1001/jama.291.20.2457

4. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. dx.doi.org/10.1136/bmj.d5928

5. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009;339:b4012. dx.doi.org/10.1136/bmj.b4012

6. Savović J, Jones HE, Altman DG, Harris RJ, Jüni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine 2012;157(6):429-38. annals.org/article.aspx?articleid=1359238

7. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews 2012;12:MR000033. dx.doi.org/10.1002/14651858.MR000033.pub2

8. Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P. Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study. Annals of Internal Medicine 2011;155(1):39-51. annals.org/article.aspx?articleid=747012

9. Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JP. Comparative effect sizes in randomised trials from less developed and more developed countries: meta-epidemiological assessment. BMJ 2013;346:f707. dx.doi.org/10.1136/bmj.f707

10. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA 2010;303(12):1180-7. dx.doi.org/10.1001/jama.2010.310

11. Goodman S, Dickersin K. Metabias: a challenge for comparative effectiveness research. Annals of Internal Medicine 2011;155(1):61-2. annals.org/article.aspx?articleid=747016

Competing interests The authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request) and declares (1) no receipt of payment or support in kind for any aspect of the article; (2) no financial relationships with any entities that have an interest related to the submitted work; and (3) that DM is a member of The Cochrane Library Oversight Committee, but no other relationships or activities that could be perceived as having influenced, or giving the appearance of potentially influencing, what was written in the submitted work.

Image credit: TH Foto-Werbung/Science Photo Library

Contact the Editor in Chief, Dr David Tovey (dtovey@cochrane.org): Feedback on this editorial and proposals for future editorials are welcome.

Keywords:

The Cochrane Library - Independent high-quality evidence for health care decision making

Copyright © 2013 John Wiley & Sons, Ltd. All rights reserved