Production and Citation of Cochrane Systematic Reviews: a Bibliometrics Analysis Jiantong Shen 1, Youping Li *1, Mike Clarke2, Liang Du 1, Li Wang 1,Dake Zhong1 1. Chinese Cochrane Center, West China Hospital, Sichuan University, China 2 .Queen's University Belfast, Northern Ireland *Corresponding author
Email: [email protected]
Abstract Objective: To evaluate the production and utilization of Cochrane systematic reviews(CSRs) and to analyze its influential factors, so as to improve the capacity of translating CSRs into practice. Methods: All CSRs and protocols were retrieved from the Cochrane Library ISSUE 2, 2011 and citation data were retrieved from SCI database. Citation analysis was used to analyze the situation of CSRs production and utilization. Results: CSR publication had grown from an annual average of 32 to 718 documents. Only one developing country was among the ten countries with the largest amount of publications. High income countries accounted for 83% of CSR publications and 90.8% of cited counts. 34.7% of CSRs had a cited count of 0, while only 0.9% had been cited more than 50 times. Highly cited CSRs were published in England, Australia, Canada, USA and other high income countries. The countries with a Cochrane center or a Cochrane methodology group had a greater capability of CSRs production and citing than others. The CSRs addressing the topics of diseases were more than those targeted at public health issues. There was a big gap in citations of different interventions even for the same topic. Conclusion: The capability of CSR production and translation grew rapidly, but varied among countries and institutions, which was affected by several factors such as the capability of research, the resourcesand the applicability of the evidence. It is important to improve
This article has been accepted for publication and undergone full peer review but has not been through the copyediting, typesetting, pagination and proofreading process, which may lead to differences between this version and the Version of Record. Please cite this article as doi: 10.1111/jebm.12101. This article is protected by copyright. All rights reserved.
1
evidence translation through educating, training and prioritizing the problems based on real demands of end user.
Keywords: Cochrane systematic review, citation analysis, production, utilization
Systematic reviews are essential foundations for EBM practice, and the importance of which is being widely recognized by policymakers, researchers, clinicians, patients and the public[1]. The Cochrane Collaboration, an international non-profit organization, specializes in systematic reviews production, collection, and dissemination. Its product, Cochrane systematic reviews(CSRs), were further being recognized by the medical community for their specialized organization, standardized methods, and strict quality management. As the usage data of Cochrane Library showed, “Every day someone, somewhere searches the Cochrane Library every second, reads an abstract every two seconds and downloads a full-text article every three seconds”[2].The Cochrane Collaboration had expended to 52 Cochrane Review Groups(CRGs) and had produced more than 4500 full Cochrane reviews and 2000 protocols listed in Cochrane Library for Cochrane reviews [3]. However, the utilization statuses for these were uncertain. The impact of CSRs on clinical practice and decision making is difficult to measure comprehensively. However, one important measure of impact is the use of citations in the published literatures. Citations could reflect the usage of a document in a certain level and used widely as an indicator of document importance, influence and quality. Citations have limitations[4],but they provide an objective measurement of how often scientists use a specific published work. This report analyzed the citations of CSRs in hopes of providing evidence for production, dissemination, and utilization of systematic reviews.
This article is protected by copyright. All rights reserved.
2
1. Materials and Methods We searched the Cochrane Library ISSUE 2, 2011 on April 19th, 2011 and retrieved 4,594 CSRs and 2,008 protocols. The SCI database was searched with publication name “Cochrane Database of Systematic Reviews” on April 14th; 4,270 CSRs were retrieved with a total of 24,591 citations and an average of 5.7 citations per publication. We downloaded 4270 records of CSRs from SCI and extracted total citations, authors’ countries and institutions from records. Whole counting was used to count publication, which mean all unique countries will receive a credit of 1 publication once a country appears in the address [5]. If a country appears more than once in the address list it will still get only 1 credit. The whole counting was used widely and adopted by ISI. We used whole counting to compare publication and citation of different group countries and institutions. Countries were grouped using the criteria of World Bank [6]. The indicator of citation was total citation without considering authors self citing. The Cochrane review question was specified by the types of population (participants), types of interventions (and comparisons), and the types of outcomes that are of interest (PICO)[7]. The themes of CSRs were grouped by population (participants) which were mapped to ICD10. 2. Results 2.1 CSR Production Status As seem in Figure 1, from the establishment of the Cochrane Library in 1996 to 2010, the annual publication has exhibited with steady growth (especially dramatic increase in publication in 2006 and 2007), increased from annual publication of 32 papers to 718 papers. There were 2008 protocols listed in the Cochrane Library, which have yet to become full CSRs. Protocols had increased greatly since 2006, when CL was indexed by SCI. Protocols published before 2009 accounted for 61% and 21% of which had published before 2006.
This article is protected by copyright. All rights reserved.
3
2.2 CSR Citation Status As seen in Table 1, the 4,270 CSRs searched in SCI were published by 79 countries (include co-authorship publications); of these, 39 high-income countries have published 3,553 CSRs (accounted for 83.3%), 23 medium-high-income countries have published 319 CSRs (accounted for 7.5%), 13 medium-low-income countries have published 383 CSRs (accounted for 9%), and 4 low-income countries have published 12 CSRs (accounted for 0.3%).
The cited frequencies of developed countries accounted for 90.8% of the total, while medium-high-income countries, medium-low-income countries and low-income countries respectively accounted for 4.8%, 4.3% and 0.1%. Average cited count per CSR was 6.01 as for developed countries, while average cited count for medium-low and low-income countries did not even reach 3.0 per CSR. As seen in Table 2, citation of CSRs differed greatly depending on economic status of the country; the most frequently citing countries were high-income countries while the least countries were low-income countries. When observing relative numbers, CSRs published by a country of a certain status were more likely to be cited by a country of the same status, emphasizing the importance of local evidence and external applicability of evidence for translating it into practice.
This article is protected by copyright. All rights reserved.
4
2.3 Production and Citation of Top Ten Countries and Institutions China was the only developing country among the ten countries with the largest amount of CSRs; all the other nine were developed countries as England, Australia, Canada, USA, Netherlands, Scotland, Italy, New Zealand, and Germany. The publications of these ten countries accounted for 88% of all CSR publications, while England, Australia, Canada, and USA totally accounted for 65%. The ten institutions with the largest amount of CSRs were University of Oxford, University of Liverpool, Sichuan University, University of Auckland, McMaster University, University of Manchester, University of Sydney, University of Toronto, University of Leeds and University of Adelaide. Of these, only Sichuan University is located in a developing country – China (accounting for 61% of CSRs published in China); the rest institutions are mainly located in England (four institutions), Canada, Australia, and New Zealand. These countries & institutions and their publication & utilization statuses could be seen in Figures 2 and 3.
As seen in Figure 2, of the top 10 countries with most publication, England had the most publications and citations; China had the lowest cited counts with an average of 1.7 counts per paper. The average cited counts of the other nine countries were around 5.4 cited times per paper.
As seen in Figure 3, of the top 10 institutions with most publication, University of Oxford had the highest publication and citation; Sichuan University in China and University of
This article is protected by copyright. All rights reserved.
5
Liverpool were respectively ranked the third and second by publications but had the lowest citations.
2.4 Factors Affecting CSR Publication and Citation As shown in Table 3, of the 14 Cochrane Centers, four Centers were located in developing countries while the other ten were located in developed countries. Of the top ten CSRspublishing countries, nine contained a Cochrane Center, while one, New Zealand, had a branch-center. The fourteen countries with a Cochrane Center listed the top 22 CSRs publishing countries in the world. In general, CSRs publication in developed countries was greater than that in developing countries; however, China had surpassed many developed countries in CSRs production, ranked 7th in the world. Some developed countries also ranked lower than developing countries. 34.7% of CSRs had never been cited before. The ratios of 0-cited CSRs of USA, Australia and Brazil were lower than the global average while the ratios of other countries were mostly greater than the global average. China, France, and Germany had the largest ratios of 0-cited CSRs. 48.9% of CSRs had cited frequencies between 1 and 10; 6.3% of CSRs had cited frequencies over 20, 69% of which were from England, Australia, Canada and USA. China and Spain had no CSRs with cited frequencies over 20. Only 0.9% of CSRs had cited frequencies over 50, which were all published by England, Australia, Canada, USA, Netherlands and Denmark, and Denmark had the highest ratio. Cited frequency of CSRs published by developed countries was generally higher than that of CSRs published by developing countries; however, there were some developed countries with cited counts lower than those of developing countries. Of the four developing countries as China, Brazil, South Africa and India, China’s CSR publication number was greater than those of the other three countries; however, China had
This article is protected by copyright. All rights reserved.
6
no CSRs with cited frequencies more than 20, while Brazil, South Africa, and India do. The proportion of 0-cited CSRs from China was higher over that of the other three countries, while the CSRs proportions in other categories of cited frequency were lower than those of the others. Brazil had the lowest proportion of 0-cited CSRs and the highest of 1 to 10-cited CSRs. South Africa had a relatively large proportion of CSRs with over 11 cited counts. All four countries had virtually no CSRs with over 50 cited counts. South Africa, Brazil, and India all had two CSRs with cited more than 20 times. The themes of these CSRs were post traumatic stress disorder, hypertension of pregnancy, obesity, acute bronchitis, schizophrenia, and tubercular meningitis. The themes of main countries of 0-cited CSRs were listed in Table 4. 92% of 0-cited CSRs were about diseases and only a few CSRs were about public health. Four themes were shared among 6 countries, as psychiatric & nervous system disease, neoplasm, circulation system disease and digestive system disease. The most published themes were psychiatric & nervous system disease, neoplasm, and respiratory system disease. Australia and United States had most 0-cited CSRs, however, the ratio of 0-cited CSR were lower than global average; while Germany and France had less 0-cited CSRs but the ratios of 0-cited CSR were higher than global level.
As seen in Table 5, 66% of high-cited CSRs were about high burden diseases, with chronic diseases as the most prevalent topic. High-cited CSRs about public health were relatively uncommon and the themes were about smoking cessation, obesity, nutritional deficiencies, and antibiotic abuse.
This article is protected by copyright. All rights reserved.
7
The theme of 41% of CSRs published by China were about Traditional Chinese Medicine and 95% of them focused on diseases. Compared with high-cited CSRs, the special themes of Chinese CSRs were oculopathy, dermal diseases and side effect. 3. Discussion 3.1 CSR Production In 1993, the Cochrane Collaboration, a non-profit academic organization was established to produce and disseminate high-quality medical evidences. Since then, publication of CSR protocols and full texts has undergone with yearly growth (this growth became even more pronounced after indexed in ISI); currently, systematic reviews are being published at a rate of 11 papers per day[8], while protocols are published at an even greater rate. The Cochrane Collaboration requires the full CSRs to be published within two years after the protocol submission[7]. As in 2010, there were 2008 protocols that had yet to be published as full texts; 70.3% of them (1,412 of 2,008) were published two years ago. Only a small proportion of protocols were published in other periodicals[9], hindering the creation and production of knowledge, and preventing public access to high-quality evidences, resulting in a major misuse of resources. This suggests that the 52 CRGs in the Cochrane Collaboration ought to strengthen protocols supervision, in order to complete protocols into full text of CSRs in timely. The median time for protocols publication was 1.5 years; the median time for full CSRs publication varied from 1.25 to 2.4 years[10,11]. Median update time was 5.5 years[12]. If CSRs publication takes too long, newly-published CSRs will already have lost much of their relevance as soon as they are published, lowing quality of evidence. The production time of CSRs was shorter than that of systematic reviews of other magazines (31 weeks vs. 65 weeks); however, the MeSH word indexing process for CSRs took longer (21 weeks vs. 4
This article is protected by copyright. All rights reserved.
8
weeks)[13]. CL should improve workflow processes to decrease indexing time and increase the timeliness of CSR s updating. Language, economics, ability, methods, and organizations are major factors that affect CSRs production, of which method and ability are the most impactful. Almost half CSR-producing countries were developed countries, accounting for 83% of CSR publications; countries with Cochrane Centers generally had more CSR publication than others, while countries with Cochrane methodology groups (England, Australia, Canada, and USA) had the highest CSR publication. Although China, Brazil, South Africa and India are developing countries as nonEnglish speaking ones, under the guidance of Cochrane Centers, they have overcame language and economic barriers, exhibiting publication counts greater than many of those developed countries. Universities could be labeled as the most efficient environment for CSRs research, production, and utilization. They provide conditions for superior human resource, information, space, teamwork, and long-term mechanism. As a result, the top ten institutions for publishing CSRs are all universities. Most CSRs were about diseases, most of which were chronic diseases. The hottest themes were psychiatric & nervous system disease, respiratory system disease, digestive system disease, neoplasm, and circulation system disease. One reason for the less number of public health CSRs is that there are less original studies about public health for production of evidences from secondary studies. The other reason is that external adaptability of evidences about public health is weak, which is more easily influenced by factors as local economy, social and environment. So it is urgent, difficult and important to produce local evidences about public health. 3.2 CSR Citation In 2011, the impact factor of CL reached 6.18. Citation frequency for CSRs experienced a decrease suggesting that the speed of dissemination and impact of CSRs are increasing daily;
This article is protected by copyright. All rights reserved.
9
it can be concluded that CSRs are fast becoming an important source of evidence in all fields of decision-making[14,15]. Average citation per document peaked in 2007 as 10.5 cites, but has fallen to 4.9 cites ever since. Documents with over 20 cites only account for 17% of all CSRs; 35% of CSRs still not have been cited. Even in England, the country with the highest citations, 36% of CSRs have never been cited. This suggests that research topics must be more prioritized in order to increase the precision and applicability of CSRs. Due to language, economy and resource, the capacities of CSR use in high-income countries greatly surpass those of other countries, while CSRs published in these countries also boast higher cited frequencies than those of developing countries. Our results suggested the importance of creating local evidences to promote evidence use. The diversities between medical needs and resource of developed and developing countries is relatively large; even a systematic review applied in developed country successfully, the same SR may not been necessarily apply to a developing country. Even as for countries with similar economic status, the health problems they faced were different. These may be reasons why so many CSRs have never been cited. The goal of the Cochrane Collaboration is to provide high-quality systematic reviews for global users and as a result, should focus on global health problems. As estimated, 10 000 Cochrane reviews are needed to cover a substantial proportion of the studies relevant to health care[16]. There are many factors that influence research priority[17]; however, alternative interventions were addressed to optimize the best solutions[18], which may be the reason as why the same topic CSRs had significant different citations. Some interventions such as Tradition Chinese Medicine were not been international confirmed and applied but were widely applied in certain local areas for their unique effectiveness. These local evidences should be produced and communicated widely. In order to effectively deal with the needs of patients, clinicians, and medical policy-makers, primary researches and funding determining should begin with SR, which may reduce the misuse of limited
This article is protected by copyright. All rights reserved.
10
resources[19-21]. Countries without a methodology group have published relatively few highly-cited CSRs. Apart from CSR theme, applicability and relevance, major causes of this phenomenon may also include capability, methods and CSR quality. Primary studies are raw materials of systematic reviews, and their quality and quantity directly affect the quality of SRs. Unfortunately, many countries, in particular developing countries, lack scientific design and plan, as well as methodology training and guidance, and lack sufficient funds, limiting production of high-quality local researches. A large number of primary clinical studies are poorly designed and unable to correctly conduct random and double blind trials, resulting in a low quality of evidences[22-24]. Primary studies included in published CSRs are mostly from USA, England, Canada, Australia, and other developed countries[25]. As a result, CSRs produced from these studies provide the highest-quality evidence, while under specific conditions in developing countries, due to the large difference between the applications conditions and primary studies conditions, quality of evidence will decrease. Many CSRs have not been updated in timely [11,26], and have methodology defects, resulting in various biases[27-29]; the guidance of a methodologist is needed to correct these defects. Research shows that the number of studies and participants included in CSRs was less than other systematic reviews, resulting in a relatively large inclusion bias[26]. 3.3 Suggestions for Improvement (1) In order to achieve the global goal of “2020 health for all”, the most important task is to identify and solve current issues for each country. This involves learning from WHOCochrane Collaboration’s strategic cooperation model. Because of the pre-existing work and advantages of the cooperation between the Cochrane Collaboration and the Campbell Collaboration, the world produce high-quality evidence in urgent need for WHO and it’s
This article is protected by copyright. All rights reserved.
11
member states through employing global view design, standard training and collaboration. All participants are putting their best efforts to realize the global goal. (2) It is advised to increase the speed of dissemination and training; improve recognition of evidence; strengthen end users’ abilities of evidence search, use and production; improve methodology training for researchers and improve the quality of primary and secondary studies to produce high-quality evidence. (3) Taken on the major burdens and health problems, Cochrane Collaboration produce high quality evidences to solve complex problems and improve the applicability and relevance of evidence. Different countries or regions perform research based on their own specific needs and produce local evidence to increase evidence translation. In order to improve the quality of research, primary and secondary research format should be standardized and research registration should be strictly implemented[30], research process must be fully transparent research quality must be controlled and reports should be standardized. Systematic reviews must also reduce production time and indexing time, complete protocols into full-texts of CSRs and update timely to improve the CSR timesensitive[31,32].
4. Limitations This report considered CSRs usage from citation analysis, however, this is a common way of CSRs utilization evaluation. Utilization and translation of CSRs in practice may not be completely reflected. The citation of recent publications have not been adjusted for time, which may exaggerate the number of low cited CSR.
This article is protected by copyright. All rights reserved.
12
5. Conclusion The capability of CSR production and utilization grows rapidly, but the situation were different among evidences and countries. Language, economics, methods, capability, resource, relevance and applicability of evidence affect the production and utilization of evidence. It is crucial to support the efficiency of evidence translation both locally and globally through publicity, education and training, capacity building and prioritized problem based on real demands of end users to create local high quality evidence.
Acknowledgements We sincerely appreciate the help of Wentao Xie (Jason Xie) in English translation and review. This study was funded by the China Medical Board (CMB).
This article is protected by copyright. All rights reserved.
13
References 1. Lavis JN, Davies HT, Gruen RL, Walshe K, Farquhar CM (2006) Working Within and Beyond the Cochrane Collaboration to Make Systematic Reviews More Useful to Healthcare Managers and Policy Makers. Healthcare Policy 1: 21-33. 2. Starr M, Chalmers I, Clarke M, Oxman AD (2009) The origins, evolution, and future of The Cochrane Database of Systematic Reviews. International Journal of Technology Assessment in Health Care 25: 182–195. 3. Allen C, Richmond K. (2011) The Cochrane Collaboration: International activity within Cochrane Review Groups in the first decade of the twenty-first century. Journal of EvidenceBased Medicine 4: 2–7. 4. Garfield E (1996) How can impact factors be improved ? BMJ 313:411-413. 5. Gauffriau M, Larsen PO, Maye I, Roulin-Perriard A, von Ins M (2008) Comparison of results of publication counting using different methods. Scientometrics 77:147–176. 6.Country and Lending Groups. http://data.worldbank.org/about/countryclassifications/country-and-lending-groups 7. Higgins J, Green S (2008) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration Available from www.cochrane-handbook.org. 8. Bastian H, Glasziou P, Chalmers I (2010) Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up. PLoS Medicine 7: E1000326. 9. Tricco AC, Pham B, Brehaut J, Tetroe J, Cappelli M, et al. (2009) An international survey indicated that unpublished systematic reviews exist. Journal of Clinical Epidemiology 62: 617-623. 10. Middleton P (2004) Life Cycle of Cochrane Reviews. Chinese Journal of Evidence Based Medicine 4: 488-507. 11. Tricco AC, Brehaut J, Chen MH, Moher D (2008) Following 411 Cochrane Protocols to Completion: A Retrospective Cohort Study. PLoS One 3: e3684. 12. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S et al. (2007) How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. Ann Intern Med 147: 224-233. 13. Sampson M, Shojania KG, Garritty C, Horsley T, Ocampoa M, et al. (2008) Systematic reviews can be produced and published faster. Journal of Clinical Epidemiology 61: 531-536. 14. Miranda C, Kathie C, Daren S, Arne O (2004) The good news: Evidence of the dissemination and influence of Cochrane systematic reviews. 12th Cochrane Colloquium, Ottawa, Canada, 2–6 October 2004. 15. Patsopoulos NA, Analatos AA, Ioannidis JP (2005) Relative citation impact of various study designs in the health sciences. JAMA 293: 2362-2366. 16. Mallett S, Clarke M (2003) How many Cochrane reviews are needed to cover existing evidence on the effects of health care interventions? ACP J Club 139: A11. 17. John PA, Ioannidis C, Isaakidis P, Volmink J (2002) Re: Priority setting in research. BMJ 324: 702. 18. Fleurence RL, Torgerson DJ (2002) Priority setting in research. BMJ 324: 702. 19. Chalmers I, Glasziou P (2009) Avoidable waste in the production and reporting of research evidence. Lancet 374: 86-89. 20. Ncayiyana DJ (2002) Africa can solve its own health problems. BMJ 324: 688-689. 21. Clarke M, Hopewell S, Chalmers I (2010) Clinical trials should begin and end with systematic reviews of relevant evidence:12 years and waiting. Lancet 376: 20-21. 22. Zhang D, Yin P, Freemantle N, Jordan R, Zhong N, et al. (2008) An assessment of the quality of randomised controlled trials conducted in China. Trials 9: 22.
This article is protected by copyright. All rights reserved.
14
23. Hewitt C, Hahn S, Torgerson DJ, Watson J, Bland JM (2005) Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ 330: 1057-1058. 24. Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, et al. (2006) Evidence of bias and variation in diagnostic accuracy studies. CMAJ 174: 469-476. 25. Wolff RF, Reinders S, Barth M, Antes G (2011) Distribution of Country of Origin in Studies Used in Cochrane Reviews. PLoS One 6: e18798. 26. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG (2007) Epidemiology and Reporting Characteristics of Systematic Reviews. PLoS Medicine 4: e78. 27. PLoS Medicine Editors (2007) Many Reviews Are Systematic but Some Are More Transparent and Completely Reported than Others. PLoS Medicine 4: e147. 28. Schwarzer G, Antes G, Tallon D, Egger M (2001) Review publication bias? Matched comparative study of Cochrane and journal meta-analyses. 9th International Cochrane Colloquium, Lyon, France, 9–13 October 2001. 29. Kirkham JJ, Altman DG, Williamson PR (2010) Bias Due to Changes in Specified Outcomes during the Systematic Review Process. PLoS One 5: e9810. 30. Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, et al. (2011) An international registry of systematic-review protocols. Lancet 377: 108-109. 31. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, et al. (2008) When and how to update systematic reviews. Cochrane Database Syst Rev 23: MR000023. 32. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, et al. (2007) A systematic review identified few methods and strategies describing when and how to update systematic reviews. J Clin Epidemiol 60: 1095-1104.
This article is protected by copyright. All rights reserved.
15
Figures
Figure 1. Annual publication of CSRs and Protocols
This article is protected by copyright. All rights reserved.
16
Figure 2. Publication and citation of the top 10 countries *The size of the circle represents average cited count per year; the slope of the circle represents cited count per article.
This article is protected by copyright. All rights reserved.
17
Figure 3. Publication and citation of the top ten institutions
This article is protected by copyright. All rights reserved.
18
Tables Table 1. CSRs publication and citation classified by country Publication and citation
High-Income
Medium-High
country（%）
Income （%）
Medium-Low
Low Income （%）
Income（%）
Publication
3,553 (83.3)
319 (7.5)
383 (9.0)
12(0.3)
Cited Count
21,368 (90.8)
1,120(4.8)
1,004 (4.3)
34(0.1)
Average Citation per Article
6.01
3.51
2.62
2.83
Table 2. CSR citation categorized by economy status of a country High-Income Citing Country High Medium-High Medium-Low Low Total
Cited Frequency 19,844 842 685 127 21,498
% of Group 92.31 3.92 3.19 0.59 100
Medium-HighIncome Cited % of Frequency Group 1,234 81.40 165 10.88 71 4.68 46 3.03 1,516 100
This article is protected by copyright. All rights reserved.
Medium-LowIncome Cited % of Frequency Group 1,051 81.92 65 5.07 127 9.90 40 3.12 1,283 100
Low-Income Cited Frequency 46 9 4 3 62
19
% of Group 74.19 14.52 6.45 4.84 100
Table 3. Citation of CSR produced by countries with Cochrane Centers Countries
0 cited（%）
1-10 cited
11-20 cited
21-50 cited
>50 cited
（%）
（%）
（%）
（%）
Total
UK
612（36.0）
846（49.8）
151（8.9）
75（4.4）
16（0.9）
1700
Australia
176（31.2）
294（52.0）
61（10.8）
28（4.9）
6（1.1）
565
Canada
177（39.3）
189（42.0）
53（11.8）
26（5.8）
5（1.1）
450
USA
126（30.9）
209（51.2）
47（11.5）
21（5.2）
5（1.2）
408
Netherlands
90（36.4）
125（50.6）
21（8.5）
8（3.3）
3（1.2）
247
China
85（51.8）
73（44.5）
6（3.7）
0（0）
0（0）
164
Italy
54（35.3）
71（46.4）
23（15.0）
5（3.3）
0（0）
153
Germany
58（43.6）
53（39.9）
13（9.7）
9（6.8）
0（0）
133
Brazil
34（33.3）
58（56.9）
8（7.8）
2（2.0）
0（0）
102
South Africa
36（40.5）
40（44.9）
11（12.4）
2（2.3）
0（0）
89
Denmark
29（42.7）
28（41.2）
6（8.8）
3（4.4）
2（2.9）
68
Spain
21（40.4）
26（50.0）
5（9.6）
0（0）
0（0）
52
India
18（36.0）
28（56.0）
2（4.0）
2（4.0）
0（0）
50
France
13（38.2）
15（44.1）
4（11.8）
2（5.9）
0（0）
34
1410（34.7）
1988（48.9）
428（10.4）
206（5.4）
36（0.9）
4270
ALL
This article is protected by copyright. All rights reserved.
20
Table 4. The themes of 0-cited CSR published by 6 countries Australia
USA
China
Germany
Brazil
France
Total
16
4
18
22
5
2
67
13
12
18
5
3
2
53
Respiratory system diseases
29
11
8
2
Musculoskeletal system diseases
16
9
1
4
Circulatory system disease
10
9
9
3
Pregnancy and childbirth
12
15
1
3
Urologic and Genital Diseases
14
3
5
5
27
5
13
4
4
26
1
5
5
6
6
6
4
6
2
2
Perinatal diseases
14
4
Blood and immune
4
5
Bye diseases
1
11
Perioperative period
4
3
Dermal diseases
5
2
1
1
Side effects
1
2
1
1
Injure
4
2
Pain
2
1
Ear diseases
2
Disease Neurological & psychiatric disorders Neoplasm
Infectious diseases Digestive system diseases Endocrine and metabolic diseases
50 4
34 2
33 31
3
26 20 18
4
1
14 12
1
1
9 9
1
6 6 3
Congenital malformation
1
3
3
3
Public health Exercise
4
Occupational diseases
2
Smoking cessation
3
1 2 1 1
Drink Cessation
4 4
2
Nutritional deficiencies
Breast feeding
5
2
1
2
1
1 1
Old care
1
Health education
1
1
Immunization
1
1
Adolescent behavior
1
1
1
1
Education Others Total
7
6
6
176
126
85
58
34
13
（31.2%）
（30.9%）
（51.8%）
（43.6%）
（33.3%）
（38.2%）
This article is protected by copyright. All rights reserved.
19
21
491
Table 5. The themes of high cited CSRs and Chinese CSRs Disease Themes Neurological& psychiatric disorders Respiratory system disease Digestive system disease Neoplasm Circulatory system disease Musculoskeletal system Urologic and Genital Disease Endocrine and metabolic disease
Public health High cited
Others
High Cited
Chinese
High cited
Chinese
10
35
Smoking cessation
8
2
Medical practice & doctorpatient relationship
5
-
9
17
Drink cessation
1
1
Methodology
3
-
8
10
8
30
6
26
6
5
Obesity Nutritional deficiencies Antibiotic abuse Medical insurance
5
-
Education
1
-
2
-
Others
5
4
2
-
-
1
5
8
Primary care
1
-
3
6
Breast feeding
1
-
Themes
Chinese
Themes
-
1
5
1
2
3
Infectious disease
2
9
Injure
1
-
Side effect
-
4
Skin disease
-
1
Perinatal disease
1
-
66
156
20
4
14
4
（66%）
（20%）
（14%）
Eye disease Pain Pregnancy and childbirth
Total
This article is protected by copyright. All rights reserved.
22
Email: [email protected]
Abstract Objective: To evaluate the production and utilization of Cochrane systematic reviews(CSRs) and to analyze its influential factors, so as to improve the capacity of translating CSRs into practice. Methods: All CSRs and protocols were retrieved from the Cochrane Library ISSUE 2, 2011 and citation data were retrieved from SCI database. Citation analysis was used to analyze the situation of CSRs production and utilization. Results: CSR publication had grown from an annual average of 32 to 718 documents. Only one developing country was among the ten countries with the largest amount of publications. High income countries accounted for 83% of CSR publications and 90.8% of cited counts. 34.7% of CSRs had a cited count of 0, while only 0.9% had been cited more than 50 times. Highly cited CSRs were published in England, Australia, Canada, USA and other high income countries. The countries with a Cochrane center or a Cochrane methodology group had a greater capability of CSRs production and citing than others. The CSRs addressing the topics of diseases were more than those targeted at public health issues. There was a big gap in citations of different interventions even for the same topic. Conclusion: The capability of CSR production and translation grew rapidly, but varied among countries and institutions, which was affected by several factors such as the capability of research, the resourcesand the applicability of the evidence. It is important to improve
This article has been accepted for publication and undergone full peer review but has not been through the copyediting, typesetting, pagination and proofreading process, which may lead to differences between this version and the Version of Record. Please cite this article as doi: 10.1111/jebm.12101. This article is protected by copyright. All rights reserved.
1
evidence translation through educating, training and prioritizing the problems based on real demands of end user.
Keywords: Cochrane systematic review, citation analysis, production, utilization
Systematic reviews are essential foundations for EBM practice, and the importance of which is being widely recognized by policymakers, researchers, clinicians, patients and the public[1]. The Cochrane Collaboration, an international non-profit organization, specializes in systematic reviews production, collection, and dissemination. Its product, Cochrane systematic reviews(CSRs), were further being recognized by the medical community for their specialized organization, standardized methods, and strict quality management. As the usage data of Cochrane Library showed, “Every day someone, somewhere searches the Cochrane Library every second, reads an abstract every two seconds and downloads a full-text article every three seconds”[2].The Cochrane Collaboration had expended to 52 Cochrane Review Groups(CRGs) and had produced more than 4500 full Cochrane reviews and 2000 protocols listed in Cochrane Library for Cochrane reviews [3]. However, the utilization statuses for these were uncertain. The impact of CSRs on clinical practice and decision making is difficult to measure comprehensively. However, one important measure of impact is the use of citations in the published literatures. Citations could reflect the usage of a document in a certain level and used widely as an indicator of document importance, influence and quality. Citations have limitations[4],but they provide an objective measurement of how often scientists use a specific published work. This report analyzed the citations of CSRs in hopes of providing evidence for production, dissemination, and utilization of systematic reviews.
This article is protected by copyright. All rights reserved.
2
1. Materials and Methods We searched the Cochrane Library ISSUE 2, 2011 on April 19th, 2011 and retrieved 4,594 CSRs and 2,008 protocols. The SCI database was searched with publication name “Cochrane Database of Systematic Reviews” on April 14th; 4,270 CSRs were retrieved with a total of 24,591 citations and an average of 5.7 citations per publication. We downloaded 4270 records of CSRs from SCI and extracted total citations, authors’ countries and institutions from records. Whole counting was used to count publication, which mean all unique countries will receive a credit of 1 publication once a country appears in the address [5]. If a country appears more than once in the address list it will still get only 1 credit. The whole counting was used widely and adopted by ISI. We used whole counting to compare publication and citation of different group countries and institutions. Countries were grouped using the criteria of World Bank [6]. The indicator of citation was total citation without considering authors self citing. The Cochrane review question was specified by the types of population (participants), types of interventions (and comparisons), and the types of outcomes that are of interest (PICO)[7]. The themes of CSRs were grouped by population (participants) which were mapped to ICD10. 2. Results 2.1 CSR Production Status As seem in Figure 1, from the establishment of the Cochrane Library in 1996 to 2010, the annual publication has exhibited with steady growth (especially dramatic increase in publication in 2006 and 2007), increased from annual publication of 32 papers to 718 papers. There were 2008 protocols listed in the Cochrane Library, which have yet to become full CSRs. Protocols had increased greatly since 2006, when CL was indexed by SCI. Protocols published before 2009 accounted for 61% and 21% of which had published before 2006.
This article is protected by copyright. All rights reserved.
3
2.2 CSR Citation Status As seen in Table 1, the 4,270 CSRs searched in SCI were published by 79 countries (include co-authorship publications); of these, 39 high-income countries have published 3,553 CSRs (accounted for 83.3%), 23 medium-high-income countries have published 319 CSRs (accounted for 7.5%), 13 medium-low-income countries have published 383 CSRs (accounted for 9%), and 4 low-income countries have published 12 CSRs (accounted for 0.3%).
The cited frequencies of developed countries accounted for 90.8% of the total, while medium-high-income countries, medium-low-income countries and low-income countries respectively accounted for 4.8%, 4.3% and 0.1%. Average cited count per CSR was 6.01 as for developed countries, while average cited count for medium-low and low-income countries did not even reach 3.0 per CSR. As seen in Table 2, citation of CSRs differed greatly depending on economic status of the country; the most frequently citing countries were high-income countries while the least countries were low-income countries. When observing relative numbers, CSRs published by a country of a certain status were more likely to be cited by a country of the same status, emphasizing the importance of local evidence and external applicability of evidence for translating it into practice.
This article is protected by copyright. All rights reserved.
4
2.3 Production and Citation of Top Ten Countries and Institutions China was the only developing country among the ten countries with the largest amount of CSRs; all the other nine were developed countries as England, Australia, Canada, USA, Netherlands, Scotland, Italy, New Zealand, and Germany. The publications of these ten countries accounted for 88% of all CSR publications, while England, Australia, Canada, and USA totally accounted for 65%. The ten institutions with the largest amount of CSRs were University of Oxford, University of Liverpool, Sichuan University, University of Auckland, McMaster University, University of Manchester, University of Sydney, University of Toronto, University of Leeds and University of Adelaide. Of these, only Sichuan University is located in a developing country – China (accounting for 61% of CSRs published in China); the rest institutions are mainly located in England (four institutions), Canada, Australia, and New Zealand. These countries & institutions and their publication & utilization statuses could be seen in Figures 2 and 3.
As seen in Figure 2, of the top 10 countries with most publication, England had the most publications and citations; China had the lowest cited counts with an average of 1.7 counts per paper. The average cited counts of the other nine countries were around 5.4 cited times per paper.
As seen in Figure 3, of the top 10 institutions with most publication, University of Oxford had the highest publication and citation; Sichuan University in China and University of
This article is protected by copyright. All rights reserved.
5
Liverpool were respectively ranked the third and second by publications but had the lowest citations.
2.4 Factors Affecting CSR Publication and Citation As shown in Table 3, of the 14 Cochrane Centers, four Centers were located in developing countries while the other ten were located in developed countries. Of the top ten CSRspublishing countries, nine contained a Cochrane Center, while one, New Zealand, had a branch-center. The fourteen countries with a Cochrane Center listed the top 22 CSRs publishing countries in the world. In general, CSRs publication in developed countries was greater than that in developing countries; however, China had surpassed many developed countries in CSRs production, ranked 7th in the world. Some developed countries also ranked lower than developing countries. 34.7% of CSRs had never been cited before. The ratios of 0-cited CSRs of USA, Australia and Brazil were lower than the global average while the ratios of other countries were mostly greater than the global average. China, France, and Germany had the largest ratios of 0-cited CSRs. 48.9% of CSRs had cited frequencies between 1 and 10; 6.3% of CSRs had cited frequencies over 20, 69% of which were from England, Australia, Canada and USA. China and Spain had no CSRs with cited frequencies over 20. Only 0.9% of CSRs had cited frequencies over 50, which were all published by England, Australia, Canada, USA, Netherlands and Denmark, and Denmark had the highest ratio. Cited frequency of CSRs published by developed countries was generally higher than that of CSRs published by developing countries; however, there were some developed countries with cited counts lower than those of developing countries. Of the four developing countries as China, Brazil, South Africa and India, China’s CSR publication number was greater than those of the other three countries; however, China had
This article is protected by copyright. All rights reserved.
6
no CSRs with cited frequencies more than 20, while Brazil, South Africa, and India do. The proportion of 0-cited CSRs from China was higher over that of the other three countries, while the CSRs proportions in other categories of cited frequency were lower than those of the others. Brazil had the lowest proportion of 0-cited CSRs and the highest of 1 to 10-cited CSRs. South Africa had a relatively large proportion of CSRs with over 11 cited counts. All four countries had virtually no CSRs with over 50 cited counts. South Africa, Brazil, and India all had two CSRs with cited more than 20 times. The themes of these CSRs were post traumatic stress disorder, hypertension of pregnancy, obesity, acute bronchitis, schizophrenia, and tubercular meningitis. The themes of main countries of 0-cited CSRs were listed in Table 4. 92% of 0-cited CSRs were about diseases and only a few CSRs were about public health. Four themes were shared among 6 countries, as psychiatric & nervous system disease, neoplasm, circulation system disease and digestive system disease. The most published themes were psychiatric & nervous system disease, neoplasm, and respiratory system disease. Australia and United States had most 0-cited CSRs, however, the ratio of 0-cited CSR were lower than global average; while Germany and France had less 0-cited CSRs but the ratios of 0-cited CSR were higher than global level.
As seen in Table 5, 66% of high-cited CSRs were about high burden diseases, with chronic diseases as the most prevalent topic. High-cited CSRs about public health were relatively uncommon and the themes were about smoking cessation, obesity, nutritional deficiencies, and antibiotic abuse.
This article is protected by copyright. All rights reserved.
7
The theme of 41% of CSRs published by China were about Traditional Chinese Medicine and 95% of them focused on diseases. Compared with high-cited CSRs, the special themes of Chinese CSRs were oculopathy, dermal diseases and side effect. 3. Discussion 3.1 CSR Production In 1993, the Cochrane Collaboration, a non-profit academic organization was established to produce and disseminate high-quality medical evidences. Since then, publication of CSR protocols and full texts has undergone with yearly growth (this growth became even more pronounced after indexed in ISI); currently, systematic reviews are being published at a rate of 11 papers per day[8], while protocols are published at an even greater rate. The Cochrane Collaboration requires the full CSRs to be published within two years after the protocol submission[7]. As in 2010, there were 2008 protocols that had yet to be published as full texts; 70.3% of them (1,412 of 2,008) were published two years ago. Only a small proportion of protocols were published in other periodicals[9], hindering the creation and production of knowledge, and preventing public access to high-quality evidences, resulting in a major misuse of resources. This suggests that the 52 CRGs in the Cochrane Collaboration ought to strengthen protocols supervision, in order to complete protocols into full text of CSRs in timely. The median time for protocols publication was 1.5 years; the median time for full CSRs publication varied from 1.25 to 2.4 years[10,11]. Median update time was 5.5 years[12]. If CSRs publication takes too long, newly-published CSRs will already have lost much of their relevance as soon as they are published, lowing quality of evidence. The production time of CSRs was shorter than that of systematic reviews of other magazines (31 weeks vs. 65 weeks); however, the MeSH word indexing process for CSRs took longer (21 weeks vs. 4
This article is protected by copyright. All rights reserved.
8
weeks)[13]. CL should improve workflow processes to decrease indexing time and increase the timeliness of CSR s updating. Language, economics, ability, methods, and organizations are major factors that affect CSRs production, of which method and ability are the most impactful. Almost half CSR-producing countries were developed countries, accounting for 83% of CSR publications; countries with Cochrane Centers generally had more CSR publication than others, while countries with Cochrane methodology groups (England, Australia, Canada, and USA) had the highest CSR publication. Although China, Brazil, South Africa and India are developing countries as nonEnglish speaking ones, under the guidance of Cochrane Centers, they have overcame language and economic barriers, exhibiting publication counts greater than many of those developed countries. Universities could be labeled as the most efficient environment for CSRs research, production, and utilization. They provide conditions for superior human resource, information, space, teamwork, and long-term mechanism. As a result, the top ten institutions for publishing CSRs are all universities. Most CSRs were about diseases, most of which were chronic diseases. The hottest themes were psychiatric & nervous system disease, respiratory system disease, digestive system disease, neoplasm, and circulation system disease. One reason for the less number of public health CSRs is that there are less original studies about public health for production of evidences from secondary studies. The other reason is that external adaptability of evidences about public health is weak, which is more easily influenced by factors as local economy, social and environment. So it is urgent, difficult and important to produce local evidences about public health. 3.2 CSR Citation In 2011, the impact factor of CL reached 6.18. Citation frequency for CSRs experienced a decrease suggesting that the speed of dissemination and impact of CSRs are increasing daily;
This article is protected by copyright. All rights reserved.
9
it can be concluded that CSRs are fast becoming an important source of evidence in all fields of decision-making[14,15]. Average citation per document peaked in 2007 as 10.5 cites, but has fallen to 4.9 cites ever since. Documents with over 20 cites only account for 17% of all CSRs; 35% of CSRs still not have been cited. Even in England, the country with the highest citations, 36% of CSRs have never been cited. This suggests that research topics must be more prioritized in order to increase the precision and applicability of CSRs. Due to language, economy and resource, the capacities of CSR use in high-income countries greatly surpass those of other countries, while CSRs published in these countries also boast higher cited frequencies than those of developing countries. Our results suggested the importance of creating local evidences to promote evidence use. The diversities between medical needs and resource of developed and developing countries is relatively large; even a systematic review applied in developed country successfully, the same SR may not been necessarily apply to a developing country. Even as for countries with similar economic status, the health problems they faced were different. These may be reasons why so many CSRs have never been cited. The goal of the Cochrane Collaboration is to provide high-quality systematic reviews for global users and as a result, should focus on global health problems. As estimated, 10 000 Cochrane reviews are needed to cover a substantial proportion of the studies relevant to health care[16]. There are many factors that influence research priority[17]; however, alternative interventions were addressed to optimize the best solutions[18], which may be the reason as why the same topic CSRs had significant different citations. Some interventions such as Tradition Chinese Medicine were not been international confirmed and applied but were widely applied in certain local areas for their unique effectiveness. These local evidences should be produced and communicated widely. In order to effectively deal with the needs of patients, clinicians, and medical policy-makers, primary researches and funding determining should begin with SR, which may reduce the misuse of limited
This article is protected by copyright. All rights reserved.
10
resources[19-21]. Countries without a methodology group have published relatively few highly-cited CSRs. Apart from CSR theme, applicability and relevance, major causes of this phenomenon may also include capability, methods and CSR quality. Primary studies are raw materials of systematic reviews, and their quality and quantity directly affect the quality of SRs. Unfortunately, many countries, in particular developing countries, lack scientific design and plan, as well as methodology training and guidance, and lack sufficient funds, limiting production of high-quality local researches. A large number of primary clinical studies are poorly designed and unable to correctly conduct random and double blind trials, resulting in a low quality of evidences[22-24]. Primary studies included in published CSRs are mostly from USA, England, Canada, Australia, and other developed countries[25]. As a result, CSRs produced from these studies provide the highest-quality evidence, while under specific conditions in developing countries, due to the large difference between the applications conditions and primary studies conditions, quality of evidence will decrease. Many CSRs have not been updated in timely [11,26], and have methodology defects, resulting in various biases[27-29]; the guidance of a methodologist is needed to correct these defects. Research shows that the number of studies and participants included in CSRs was less than other systematic reviews, resulting in a relatively large inclusion bias[26]. 3.3 Suggestions for Improvement (1) In order to achieve the global goal of “2020 health for all”, the most important task is to identify and solve current issues for each country. This involves learning from WHOCochrane Collaboration’s strategic cooperation model. Because of the pre-existing work and advantages of the cooperation between the Cochrane Collaboration and the Campbell Collaboration, the world produce high-quality evidence in urgent need for WHO and it’s
This article is protected by copyright. All rights reserved.
11
member states through employing global view design, standard training and collaboration. All participants are putting their best efforts to realize the global goal. (2) It is advised to increase the speed of dissemination and training; improve recognition of evidence; strengthen end users’ abilities of evidence search, use and production; improve methodology training for researchers and improve the quality of primary and secondary studies to produce high-quality evidence. (3) Taken on the major burdens and health problems, Cochrane Collaboration produce high quality evidences to solve complex problems and improve the applicability and relevance of evidence. Different countries or regions perform research based on their own specific needs and produce local evidence to increase evidence translation. In order to improve the quality of research, primary and secondary research format should be standardized and research registration should be strictly implemented[30], research process must be fully transparent research quality must be controlled and reports should be standardized. Systematic reviews must also reduce production time and indexing time, complete protocols into full-texts of CSRs and update timely to improve the CSR timesensitive[31,32].
4. Limitations This report considered CSRs usage from citation analysis, however, this is a common way of CSRs utilization evaluation. Utilization and translation of CSRs in practice may not be completely reflected. The citation of recent publications have not been adjusted for time, which may exaggerate the number of low cited CSR.
This article is protected by copyright. All rights reserved.
12
5. Conclusion The capability of CSR production and utilization grows rapidly, but the situation were different among evidences and countries. Language, economics, methods, capability, resource, relevance and applicability of evidence affect the production and utilization of evidence. It is crucial to support the efficiency of evidence translation both locally and globally through publicity, education and training, capacity building and prioritized problem based on real demands of end users to create local high quality evidence.
Acknowledgements We sincerely appreciate the help of Wentao Xie (Jason Xie) in English translation and review. This study was funded by the China Medical Board (CMB).
This article is protected by copyright. All rights reserved.
13
References 1. Lavis JN, Davies HT, Gruen RL, Walshe K, Farquhar CM (2006) Working Within and Beyond the Cochrane Collaboration to Make Systematic Reviews More Useful to Healthcare Managers and Policy Makers. Healthcare Policy 1: 21-33. 2. Starr M, Chalmers I, Clarke M, Oxman AD (2009) The origins, evolution, and future of The Cochrane Database of Systematic Reviews. International Journal of Technology Assessment in Health Care 25: 182–195. 3. Allen C, Richmond K. (2011) The Cochrane Collaboration: International activity within Cochrane Review Groups in the first decade of the twenty-first century. Journal of EvidenceBased Medicine 4: 2–7. 4. Garfield E (1996) How can impact factors be improved ? BMJ 313:411-413. 5. Gauffriau M, Larsen PO, Maye I, Roulin-Perriard A, von Ins M (2008) Comparison of results of publication counting using different methods. Scientometrics 77:147–176. 6.Country and Lending Groups. http://data.worldbank.org/about/countryclassifications/country-and-lending-groups 7. Higgins J, Green S (2008) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration Available from www.cochrane-handbook.org. 8. Bastian H, Glasziou P, Chalmers I (2010) Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up. PLoS Medicine 7: E1000326. 9. Tricco AC, Pham B, Brehaut J, Tetroe J, Cappelli M, et al. (2009) An international survey indicated that unpublished systematic reviews exist. Journal of Clinical Epidemiology 62: 617-623. 10. Middleton P (2004) Life Cycle of Cochrane Reviews. Chinese Journal of Evidence Based Medicine 4: 488-507. 11. Tricco AC, Brehaut J, Chen MH, Moher D (2008) Following 411 Cochrane Protocols to Completion: A Retrospective Cohort Study. PLoS One 3: e3684. 12. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S et al. (2007) How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. Ann Intern Med 147: 224-233. 13. Sampson M, Shojania KG, Garritty C, Horsley T, Ocampoa M, et al. (2008) Systematic reviews can be produced and published faster. Journal of Clinical Epidemiology 61: 531-536. 14. Miranda C, Kathie C, Daren S, Arne O (2004) The good news: Evidence of the dissemination and influence of Cochrane systematic reviews. 12th Cochrane Colloquium, Ottawa, Canada, 2–6 October 2004. 15. Patsopoulos NA, Analatos AA, Ioannidis JP (2005) Relative citation impact of various study designs in the health sciences. JAMA 293: 2362-2366. 16. Mallett S, Clarke M (2003) How many Cochrane reviews are needed to cover existing evidence on the effects of health care interventions? ACP J Club 139: A11. 17. John PA, Ioannidis C, Isaakidis P, Volmink J (2002) Re: Priority setting in research. BMJ 324: 702. 18. Fleurence RL, Torgerson DJ (2002) Priority setting in research. BMJ 324: 702. 19. Chalmers I, Glasziou P (2009) Avoidable waste in the production and reporting of research evidence. Lancet 374: 86-89. 20. Ncayiyana DJ (2002) Africa can solve its own health problems. BMJ 324: 688-689. 21. Clarke M, Hopewell S, Chalmers I (2010) Clinical trials should begin and end with systematic reviews of relevant evidence:12 years and waiting. Lancet 376: 20-21. 22. Zhang D, Yin P, Freemantle N, Jordan R, Zhong N, et al. (2008) An assessment of the quality of randomised controlled trials conducted in China. Trials 9: 22.
This article is protected by copyright. All rights reserved.
14
23. Hewitt C, Hahn S, Torgerson DJ, Watson J, Bland JM (2005) Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ 330: 1057-1058. 24. Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, et al. (2006) Evidence of bias and variation in diagnostic accuracy studies. CMAJ 174: 469-476. 25. Wolff RF, Reinders S, Barth M, Antes G (2011) Distribution of Country of Origin in Studies Used in Cochrane Reviews. PLoS One 6: e18798. 26. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG (2007) Epidemiology and Reporting Characteristics of Systematic Reviews. PLoS Medicine 4: e78. 27. PLoS Medicine Editors (2007) Many Reviews Are Systematic but Some Are More Transparent and Completely Reported than Others. PLoS Medicine 4: e147. 28. Schwarzer G, Antes G, Tallon D, Egger M (2001) Review publication bias? Matched comparative study of Cochrane and journal meta-analyses. 9th International Cochrane Colloquium, Lyon, France, 9–13 October 2001. 29. Kirkham JJ, Altman DG, Williamson PR (2010) Bias Due to Changes in Specified Outcomes during the Systematic Review Process. PLoS One 5: e9810. 30. Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, et al. (2011) An international registry of systematic-review protocols. Lancet 377: 108-109. 31. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, et al. (2008) When and how to update systematic reviews. Cochrane Database Syst Rev 23: MR000023. 32. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, et al. (2007) A systematic review identified few methods and strategies describing when and how to update systematic reviews. J Clin Epidemiol 60: 1095-1104.
This article is protected by copyright. All rights reserved.
15
Figures
Figure 1. Annual publication of CSRs and Protocols
This article is protected by copyright. All rights reserved.
16
Figure 2. Publication and citation of the top 10 countries *The size of the circle represents average cited count per year; the slope of the circle represents cited count per article.
This article is protected by copyright. All rights reserved.
17
Figure 3. Publication and citation of the top ten institutions
This article is protected by copyright. All rights reserved.
18
Tables Table 1. CSRs publication and citation classified by country Publication and citation
High-Income
Medium-High
country（%）
Income （%）
Medium-Low
Low Income （%）
Income（%）
Publication
3,553 (83.3)
319 (7.5)
383 (9.0)
12(0.3)
Cited Count
21,368 (90.8)
1,120(4.8)
1,004 (4.3)
34(0.1)
Average Citation per Article
6.01
3.51
2.62
2.83
Table 2. CSR citation categorized by economy status of a country High-Income Citing Country High Medium-High Medium-Low Low Total
Cited Frequency 19,844 842 685 127 21,498
% of Group 92.31 3.92 3.19 0.59 100
Medium-HighIncome Cited % of Frequency Group 1,234 81.40 165 10.88 71 4.68 46 3.03 1,516 100
This article is protected by copyright. All rights reserved.
Medium-LowIncome Cited % of Frequency Group 1,051 81.92 65 5.07 127 9.90 40 3.12 1,283 100
Low-Income Cited Frequency 46 9 4 3 62
19
% of Group 74.19 14.52 6.45 4.84 100
Table 3. Citation of CSR produced by countries with Cochrane Centers Countries
0 cited（%）
1-10 cited
11-20 cited
21-50 cited
>50 cited
（%）
（%）
（%）
（%）
Total
UK
612（36.0）
846（49.8）
151（8.9）
75（4.4）
16（0.9）
1700
Australia
176（31.2）
294（52.0）
61（10.8）
28（4.9）
6（1.1）
565
Canada
177（39.3）
189（42.0）
53（11.8）
26（5.8）
5（1.1）
450
USA
126（30.9）
209（51.2）
47（11.5）
21（5.2）
5（1.2）
408
Netherlands
90（36.4）
125（50.6）
21（8.5）
8（3.3）
3（1.2）
247
China
85（51.8）
73（44.5）
6（3.7）
0（0）
0（0）
164
Italy
54（35.3）
71（46.4）
23（15.0）
5（3.3）
0（0）
153
Germany
58（43.6）
53（39.9）
13（9.7）
9（6.8）
0（0）
133
Brazil
34（33.3）
58（56.9）
8（7.8）
2（2.0）
0（0）
102
South Africa
36（40.5）
40（44.9）
11（12.4）
2（2.3）
0（0）
89
Denmark
29（42.7）
28（41.2）
6（8.8）
3（4.4）
2（2.9）
68
Spain
21（40.4）
26（50.0）
5（9.6）
0（0）
0（0）
52
India
18（36.0）
28（56.0）
2（4.0）
2（4.0）
0（0）
50
France
13（38.2）
15（44.1）
4（11.8）
2（5.9）
0（0）
34
1410（34.7）
1988（48.9）
428（10.4）
206（5.4）
36（0.9）
4270
ALL
This article is protected by copyright. All rights reserved.
20
Table 4. The themes of 0-cited CSR published by 6 countries Australia
USA
China
Germany
Brazil
France
Total
16
4
18
22
5
2
67
13
12
18
5
3
2
53
Respiratory system diseases
29
11
8
2
Musculoskeletal system diseases
16
9
1
4
Circulatory system disease
10
9
9
3
Pregnancy and childbirth
12
15
1
3
Urologic and Genital Diseases
14
3
5
5
27
5
13
4
4
26
1
5
5
6
6
6
4
6
2
2
Perinatal diseases
14
4
Blood and immune
4
5
Bye diseases
1
11
Perioperative period
4
3
Dermal diseases
5
2
1
1
Side effects
1
2
1
1
Injure
4
2
Pain
2
1
Ear diseases
2
Disease Neurological & psychiatric disorders Neoplasm
Infectious diseases Digestive system diseases Endocrine and metabolic diseases
50 4
34 2
33 31
3
26 20 18
4
1
14 12
1
1
9 9
1
6 6 3
Congenital malformation
1
3
3
3
Public health Exercise
4
Occupational diseases
2
Smoking cessation
3
1 2 1 1
Drink Cessation
4 4
2
Nutritional deficiencies
Breast feeding
5
2
1
2
1
1 1
Old care
1
Health education
1
1
Immunization
1
1
Adolescent behavior
1
1
1
1
Education Others Total
7
6
6
176
126
85
58
34
13
（31.2%）
（30.9%）
（51.8%）
（43.6%）
（33.3%）
（38.2%）
This article is protected by copyright. All rights reserved.
19
21
491
Table 5. The themes of high cited CSRs and Chinese CSRs Disease Themes Neurological& psychiatric disorders Respiratory system disease Digestive system disease Neoplasm Circulatory system disease Musculoskeletal system Urologic and Genital Disease Endocrine and metabolic disease
Public health High cited
Others
High Cited
Chinese
High cited
Chinese
10
35
Smoking cessation
8
2
Medical practice & doctorpatient relationship
5
-
9
17
Drink cessation
1
1
Methodology
3
-
8
10
8
30
6
26
6
5
Obesity Nutritional deficiencies Antibiotic abuse Medical insurance
5
-
Education
1
-
2
-
Others
5
4
2
-
-
1
5
8
Primary care
1
-
3
6
Breast feeding
1
-
Themes
Chinese
Themes
-
1
5
1
2
3
Infectious disease
2
9
Injure
1
-
Side effect
-
4
Skin disease
-
1
Perinatal disease
1
-
66
156
20
4
14
4
（66%）
（20%）
（14%）
Eye disease Pain Pregnancy and childbirth
Total
This article is protected by copyright. All rights reserved.
22
