Archie Cochrane's seminal work was titled "Effectiveness and efficiency: random reflections on health services", reflecting his belief that healthcare interventions should not only do more good than harm, but should also represent a good use of resources. That is, efficiency in the use of resources is important because wasted resources represent sacrifices of effectiveness elsewhere in the healthcare system. Since its inception 20 years ago, The Cochrane Collaboration has produced over 5000 systematic reviews of the effectiveness of interventions, but these have largely ignored the challenge to provide information on efficiency. Is this a problem and, if so, what can be done about it?
Since Cochrane was writing in 1972, growing pressures on healthcare budgets have meant that issues of efficiency have grown in prominence. This is demonstrated by the growth in the number of bodies, such as the UK National Institute for Health and Care Excellence (NICE), that consider evidence on both clinical and cost-effectiveness. The needs for evidence on effectiveness and efficiency are closely aligned.
The Collaboration has offered considerable support to the development of economics methods. The Cochrane Economics Methods Group, now the Campbell and Cochrane Economics Methods Group (CCEMG), was formally established in 1998 but as a group was developing methods as early as 1993. In addition, economics workshops have been held at many Cochrane meetings, there is an economics methods chapter in the Cochrane Handbook, and some health economists have contributed as advisors or editors of Cochrane Review Groups. So why do so few published Cochrane intervention reviews incorporate formal reviews of economic evidence? It could be that, despite these efforts, the importance of reviewing economic evidence is still not apparent to many authors and editors of Cochrane Reviews. It is perhaps more likely that the increasing methodological sophistication of Cochrane Reviews has limited the capacity of author teams and editorial bases to grapple with economic evidence as part of Cochrane Reviews. It may also be difficult to recruit health economists to contribute to Cochrane Review Groups and individual reviews. This is an issue the CCEMG has been trying to address, but many health economists work across several clinical fields and may not be prepared to invest a significant amount of time on a single clinical topic.
There are also issues with the economic data themselves. It is usually assumed that, unlike estimates of relative clinical effect, economic data are not generalisable and that therefore the production of a pooled estimate would be meaningless. This may be true of cost data, which clearly vary between settings and over time due to differences in prices, and there are methods by which these prices might be converted. However, estimates of resource use are less likely to vary, particularly within the context of a clinical trial, where practice is largely controlled by the trial protocol. There may be some variation in resource use due to differences in local practices that reflect local circumstances (e.g. the case mix of patients seen), but such variations would also impact on clinical outcomes, which are often thought to be sufficiently similar and generalisable to be pooled. As with data for health outcomes, pooling 'economic' data may also help us investigate the extent and causes of variation between estimates of resource use or costs from studies conducted in different places and settings.
Another problem is that it may not be clear how best to extract and analyse resource-use data within the context of a Cochrane Review. This issue has recently been addressed by Brunetti et al, who describe how to use the GRADE approach to include considerations of resource use. They argue that the key steps in considering resource use and rating the quality of economic evidence within GRADE are:
1. Identifying items of resource use that may differ between management strategies and that are potentially important to decision-makers
2. Finding evidence for the differences in resource use, making judgments regarding confidence in effect estimates using the same criteria used for health outcomes
3. Valuing the resource use in terms of costs for the specific setting for which recommendations are being made.
It might be argued that the desire to include an economic perspective is not sensible because the amount and quality of existing economic evidence is so limited. That may have been true historically, but economic evaluation methods have advanced substantially in the last 20 years, including the increasing sophistication of both model based and trial based economic evaluations that have had additional impacts elsewhere but most notably the spur given to the development of Bayesian methods of meta-analysis. Related to this is the development of methods to handle imprecision surrounding results that are considerably more sophisticated than those found in the majority of current trials. Coupled with the development of economic evaluation methods there are greater expectations from funders for new research to include an economic evaluation, and this has led to an increase in the number of trials and other studies that include high-quality economic evaluations. This trend is continuing, and the application of methods economic evidence in systematic reviews needs to meet this expectation. CCEMG has developed materials and methods that help systematic review authors incorporate economic perspectives in Cochrane Reviews.
With efforts such as these, coupled with moderate investment in further methods development and capacity building to support the application of economics methods in reviews, we believe that over the next 20 years the Collaboration will be well-placed to address the full challenge raised by Archie Cochrane to develop and routinely incorporate economic perspectives and evidence into its reviews. In turn, this will increase the relevance and usefulness of Cochrane Reviews to the growing number of decision-makers who need information on both the clinical and cost-effectiveness of healthcare interventions.
Mike Drummond1, Ian Shemilt2, Luke Vale3, on behalf of the Campbell and Cochrane Economic Methods Group
1Mike Drummond (email@example.com), Centre for Health Economics, University of York, York, UK; 2Ian Shemilt (firstname.lastname@example.org), Behaviour and Health Research Unit, University of Cambridge, Cambridge, UK; Luke Vale (email@example.com), Institute of Health & Society, Newcastle University, Newcastle upon Tyne, UK; on behalf of the Campbell and Cochrane Economic Methods Group (www.c-cemg.org).
How to cite: Drummond M, Shemilt I, Vale L, on behalf of the Campbell and Cochrane Economic Methods Group. Should The Cochrane Collaboration be producing reviews of efficiency? [editorial]. Cochrane Database of Systematic Reviews 2013;(10):ED000071 dx.doi.org/10.1002/14651858.ED000071
1. Cochrane AL. Effectiveness and Efficiency: Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust, 1972.
3. Shemilt I, Mugford M, Byford S, Drummond M, Eisenstein E, Knapp M, et al. Chapter 15: Incorporating economics evidence. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from handbook.cochrane.org.
4. Anderson R, Shemilt I. Chapter 3: The role of economic perspectives and evidence in systematic review. In: Shemilt I, Mugford M, Vale L, Marsh K, Donaldson C (editors), Evidence-Based Decisions and Economics: Health Care, Social Welfare, Education and Criminal Justice. Oxford: Wiley-Blackwell, 2010.
5. Campbell and Cochrane Economic Methods Group, EPPI Centre. Cost Converter: a free online tool for adjusting cost estimates expressed in one currency and price year to another target currency and price year. eppi.ioe.ac.uk/costconversion (accessed August 2013).
6. Brunetti M, Shemilt I, Pregno S, Vale L, Oxman AD, Lord J. GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence. Journal of Clinical Epidemiology 2013;66(2):140-50. dx.doi.org/10.1016/j.jclinepi.2012.04.012
7. Campbell and Cochrane Economic Methods Group. Resources. Available at www.york.ac.uk/inst/crd/CCEMG/resources_ccemg.html (accessed 14 October 2013).
Competing interests: The authors have completed the ICMJE form for disclosure of potential conflicts of interest (forms available on request) and declare no conflicts of interest.
Image credit: The Cochrane Collaboration
Feedback: Please contact the Editor in Chief, David Tovey with feedback on this editorial and proposals for future editorials.