Measuring the performance of The Cochrane Library

  • By: The Cochrane Library Oversight Committee
  • On: November 14, 2012, 13:53
thumbnail image: Measuring the performance of The Cochrane Library

How well does The Cochrane Library achieve its objective—to provide accessible and credible evidence to guide decision making in medicine and public health? And how should we measure success or failure? Regular users of The Cochrane Library will have a view of its quality, and we hope that you think that it’s good and improving. One of our tasks as the Cochrane Library Oversight Committee (CLOC) is to report to the Steering Group of The Cochrane Collaboration on the performance of the Library and its Editor in Chief.[1] In general we are satisfied with both. But we thought it important to try to introduce some objectivity into the evaluation of The Cochrane Library by devising a set of metrics. Together with the Cochrane Editorial Unit we have done so, and they are shown in Table 1 along with their values for the past three years. We welcome your feedback on the metrics we have devised. To be useful, metrics must provide information on progress in relation to the aims of the Library. In addition, it must be possible to measure them precisely and relatively easily. They must also change over time at a speed that is useful—not as fast as hourly but not as slowly as five yearly.

The number of new reviews is clearly an important measure and has ranged between 402 and 449 per year for the previous four years with no clear time trend. The extent to which the Collaboration achieves its aim to update all reviews regularly also seems an important measure as reviews must be updated to be useful. The number of reviews updated has been fairly constant, ranging between 479 and 524 per year for the previous four years with no clear time trend. As the Library is cumulative the total number of reviews is increasing, but the proportion that are up to date (updated within the past two years) is steadily declining—from 39.8% in 2009 to 35.8% in June 2012. It's worrying both that the figure is low and declining, given that Cochrane Reviews are living documents, readers expect Cochrane Reviews to be up to date, and review authors commit to maintaining their reviews every two years in order that the findings remain accurate. As we can see, it does not seem possible to achieve the original aspiration of the Cochrane pioneers. In April 2012 the Editor in Chief proposed that there should no longer be an expectation that every review would be updated every two years. Some reviews cover questions that are of historic interest only—with interventions that are no longer used, or covering comparisons that are considered to have been adequately evaluated. This implies that all reviews will need to be labelled appropriately, but they may not require updating. This will leave a smaller but better prioritised pool of reviews that need regular maintenance, perhaps more frequently than two years in some cases. We will modify the metric for next year to be the proportion of reviews that need to be updated that have been updated.

Table 1. Metrics for assessing the performance of the Cochrane Database of Systematic Reviews and The Cochrane Library

Metric

2012
(to end of June)

2012 (extrapolated where possible)

Projected increase 2012-2011

2011

Increase 2011-2010

2010

Increase 2010-2009

2009

Number of new reviews published

210

420

1.0%

416

-7.3%

449

11.7%

402

Number of updated reviews published

241

482

3.0%

468

-10.7%

524

9.4%

479

Total number of active reviews

4,905

5,097

8.1%

4,713

8.9%

4,329

9.4%

3,958

Percentage of active reviews that are
up to date (within two years)

35.8

 

 

36.5

0.6%

36.3

-3.5%

39.8

Number of registered authors

22557 (calculated in Jan)

22,557

11.4% 

 20,246

-4.2% 

21,123

17.0%

18,057

Number of registered authors from
low- and middle-income countries

4,943
(calculated in Jan) 

 4,943

 13.7%

4,346

 11.8%

3886 

 1.3%

3838 

Percentage of authors from low- and
middle-income countries

21.9%

21.9% 

 0.4%

 21.5%

 3.1% 

18.4%

-2.9%

21.3

Mean time (months) from protocol
publication to review publication

 

 

 

 

30.1

3.4%

29.1

Median time (mnonths) from
protocol publication to review publication

 

 

 

 

 

23

0.0%

23

Full text access

1,980,715 (to end of April)

5,942,145

37.0%

4,337,045

9.6%

3,957,567

13.9%

3,473,141

Amount of substantive feedback

34

68

-11.7%

77

-8.3%

84

-22.2%

108

Impact factor

 

 

 

5.912

-4.4%

6.186

9.4%

5.653

Number of citations

 

 

 

 29,593

8.1% 

27,366

18.5%

23,102

Media mentions

1,715

3,430

-35.0%

5,277

33.3%

3,958

15.3%

3,434


One of the most important measures of Cochrane content is its methodological quality and adherence to best current standards as described in the Cochrane Handbook for Systematic Reviews of Interventions.[2] At the moment we do not have a measure for this, but the Collaboration has recently developed minimum standards for the conduct and reporting of reviews,[3] and these will be incorporated into the metrics in the next year.

Producing systematic reviews is demanding work, and the more authors that there are to contribute to the Cochrane Database of Systematic Reviews the better. So our metrics include the total number of registered authors, and because the Collaboration wants to increase the geographical and linguistic diversity of its contributors we have included the proportion of authors from low- and middle-income countries (LMICs). The latter is an example of a metric that has to be calculated manually, in this case by Claire Allen and colleagues in the Cochrane Operations Unit. This metric is a case where we might set a target. Should we perhaps aim to increase the proportion of authors from LMICs from 22% to 30% by 2015? We would like your views.

It seemed to be important to include a measure of timeliness and efficiency of review production in the metrics, and we selected the time from publication of a protocol to publication of the review. The average is a slow 30 months, but this figure includes a number of older reviews with very delayed publication, so the median is a better measure—and stands at 23 months. Is two years a reasonable time to take to publish a review? This metric is, of course, a measure of both the time taken by the review authors to undertake the research and also the work done by editors and peer reviewers. We should perhaps include a metric of the time from submission to publication of the completed review, a stage that is more under the control of Cochrane Review Group editorial teams.

The main point of The Cochrane Library is that its reviews should be accessed and used to improve the quality of health care and policy. Usage of the full text of Cochrane Reviews is thus a very important metric, and it is pleasing that it is high and increasing rapidly. We opted for usage of full text rather than hits on the website because it better reflects serious usage. The amount of feedback received might have been included under the heading of content, but we chose to include it here because people must access and use the website in order to send feedback. As feedback represents a form of post-publication review and as the Library is interested to increase debate, the amount of feedback is disappointingly low, and we are implementing ways to try and increase it, partly by simplifying an over-elaborate and slow process. In future we should also measure responsiveness of review authors to published feedback.

The Collaboration wants the Cochrane Library to have as much impact as possible, and so we have included measures of impact—with the impact factor in pride of place. Most readers are probably familiar with the impact factor, which has become very important to academics in many countries in the past two decades, but for those who do not know it is the number of citations of articles in a publication divided by the number of citable articles. To be more precise and to use the example of the impact factor for 2011 it is the number of citations in 2011 of articles published in 2009 and 2010 divided by the number of citable articles published in 2009 and 2010. Despite its importance to some academics, the impact factor has many problems and increasingly is gamed by editors.[4] The impact factor matters to a publication as one measure of its use but also because it influences whether academics will submit work. The Cochrane Database of Systematic Reviews' impact factor of about 6 puts it among the highest ranked publications and is adequate for most academics to be happy to publish there. Because it is possible to have a high impact factor and publish very little (indeed, reducing the amount they publish is one of the strategies used by editors to increase their journal's impact factor) we thought it important as well to include a metric of the total number of citations. This is impressively high. As another measure of impact the Collaboration is seeking ways to measure the use of Cochrane Reviews in clinical practice guidelines.

Our final metric is the number of times that Cochrane Reviews are mentioned in the media (measured using a consistent methodology over the previous five years, and including Google News). The Collaboration is interested in influencing the broader world, including patients and politicians, so this is a useful measure although it has fluctuated considerably over recent years and can be highly sensitive to small numbers of extreme outliers. This was exemplified by the 2011 review of zinc and its role in preventing and treating the common cold,[5] which attracted a level of media interest that was unprecedented and out of proportion to its clinical importance.

We hope that you will agree that the metrics we have selected are meaningful and useful, but we will be very grateful for feedback on these metrics and suggestions for others that we might consider and the reasons why.

The Cochrane Library Oversight Committee: Lisa Bero1, Godwin Busuttil2, Cindy Farquhar3, Tracey Pérez Koehlmoos4, David Moher5, Magne Nylenna6 (deputy chair); Richard Smith7 (chair), David Tovey8

1Lisa Bero, Professor of Clinical Pharmacy and Health Policy, University of California, San Francisco, USA; 2Godwin Busuttil, Barrister, UK; 3Cindy Farquhar, Postgraduate Professor of Obstetrics and Gynaecology, University of Auckland, New Zealand; 4Tracey Pérez Koehlmoos, Head of the Health and Family Planning Systems Programme, ICDDR B, Bangladesh; 5David Moher, Clinical Epidemiologist, Ottawa Hospital Research Institute and University of Ottawa, Canada; 6Magne Nylenna, Head, Norwegian Electronic Health Library, Norway; 7Richard Smith, Director, UnitedHealth Chronic Disease Initiative, London, UK; 8David Tovey, Editor in Chief, The Cochrane Library, London, UK

How to cite: The Cochrane Library Oversight Committee. Measuring the performance of The Cochrane Library [editorial]. Cochrane Database of Systematic Reviews 2012 Nov 14;(11):ED000048 http://www.thecochranelibrary.com/details/editorial/3620281/Measuring-the-performance-of-The-Cochrane-Library.html (accessed Day Month Year).

References:

1. The Cochrane Library Oversight Committee http://www.thecochranelibrary.com/view/0/CochraneLibraryOversightCommittee.html (accessed 10 November 2012)

2. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.

3. Cochrane Editorial Unit. Methodological Expectations of Cochrane Intervention Reviews (MECIR). http://www.editorial-unit.cochrane.org/mecir (accessed 10 November 2012)

4. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:497.1 DOI:10.1136/bmj.314.7079.497

5. Singh M, Das RR. Zinc for the common cold. Cochrane Database of Systematic Reviews 2011;(2):CD001364. DOI:10.1002/14651858.CD001364.pub3

Image credit: Gustoimages/Science Photo Library

Contact: Feedback on this editorial is welcome and should be sent to the Chair of The Cochrane Library Oversight Committee, Richard Smith (richardswsmith@yahoo.co.uk), or to the Editor in Chief, Dr David Tovey (dtovey@cochrane.org). 

Keywords:

The Cochrane Library - Independent high-quality evidence for health care decision making

Copyright © 2014 John Wiley & Sons, Ltd. All rights reserved