European Journal of Clinical Nutrition (2014) 68, 143–145 & 2014 Macmillan Publishers Limited All rights reserved 0954-3007/14 www.nature.com/ejcn
REVIEW
Conclusiveness of the Cochrane Reviews in Nutrition: a systematic analysis S Cohen1,2, D Mandel2,3, FB Mimouni2,4, R Marom2,3 and R Lubetzky2,4 BACKGROUND/OBJECTIVE: To assess the conclusiveness of the Cochrane Reviews (CRs) in the field of Nutrition, we tested the hypotheses that: (1) the majority of CRs is inconclusive; (2) the majority of CRs recognizes the need for further and better studies and (3) the ability to reach a conclusion is dependent on the number of studies performed and number of patients enrolled. SUBJECTS/METHODS: We selected all 87 CRs in the field of Nutrition available in Cochrane library. Each CR was analyzed for the number of randomized clinical trials (RCTs) found, number of RCTs included for analysis, number of patients enrolled, the stated need for further studies and the reason(s) for it and the conclusiveness of the CR. RESULTS: Fifty-six out of eighty-seven CRs (64.4%) were conclusive. The average number of available articles, the percentage of articles included, the average number of RCT’s retained in the analyses and the total cumulative number of patients enrolled in the studies retained for analysis were significantly higher in conclusive CRs than in non-conclusive ones. The majority of inconclusive CRs (70.9%) recognized the need for further studies, a percentage not significantly different from that found in conclusive ones (58.9%, P ¼ 0.26). The percentage of conclusive CRs was not affected by year of publication. CONCLUSIONS: The majority of CRs in Nutrition is conclusive, but most of them emphasize the need for further studies. The ability for a CR to reach a conclusion is affected by the cumulative patient sample size and number of RCT’s included in the analysis. European Journal of Clinical Nutrition (2014) 68, 143–145; doi:10.1038/ejcn.2013.252; published online 11 December 2013 Keywords: nutrition; meta-analysis; randomized clinical trial
INTRODUCTION The Cochrane Collaboration is an international, independent, not-for-profit organization of over 28 000 contributors from more than 100 countries (http://www.cochrane.org). Its stated aims are to ‘making up-to-date, accurate information about the effects of health care readily available worldwide’ (http://www.cochrane.org). The contributors of the Cochrane Reviews (CR) work together to produce systematic reviews of health care interventions, which are then published online in the Cochrane Library. An intended aim of the CRs is to ‘help providers, practitioners and patients make informed decisions about health care’ based on ‘comprehensive, reliable and relevant source of evidence on which to base these decisions’ (http://www.cochrane.org). The CRs constitute a large database of systematic reviews and meta-analyses, and have become a key resource in evidencebased medicine (http://en.wikipedia.org/wiki/Cochrane_Library). From a few reviews published in 1995, at the inception of the project, there are currently more than 5000 reviews available (http://www.cochrane.org). The CRs can be withdrawn from the active database when they become out-of-date or are replaced by new CRs in a similar subject area (http://www.cochrane.org). On 7 August 2012, the database comprised 5131 published reviews, 2235 protocols, of which 29 were new reviews, 29 updated reviews, 68 new protocols and 10 updated protocols, at the same time there was 1 withdrawn review and 5 withdrawn protocols. Typically, a given CR provides a systematic analysis of available clinical trials, and ends up in a specific recommendation on the
efficacy of a particular medical intervention (for example, surgical or pharmaceutical) (http://www.cochrane.org). Often, because the CR did not find any strong evidence for the efficacy of a particular treatment or strategy, it ends up recommending that ‘additional medical studies should be performed before a final conclusion can be reached’ (http://www.cochrane.org). Therefore, we conducted the present study to systematically analyze the conclusions and recommendations of all CRs related to the field of Nutrition available until 1 July 2012. We aimed to verify whether the reviews are conclusive or not from a clinical practice standpoint. For this purpose, we defined conclusiveness as the ability or not to conclude that a specific treatment was beneficial.1 We tested the hypotheses that: (1) the majority of CRs is inconclusive; (2) the majority of CRs recognizes the need for further and better studies and (3) the ability to reach a conclusion is dependent on both the number of studies performed and number of patients enrolled. A ‘time effect’ was also considered, as we hypothesized that newer CRs are more likely to be conclusive than older ones. MATERIALS AND METHODS We selected all 87 CRs in Nutrition available in the Cochrane library (http://www.cochrane.org) on 15 June 2012, starting from 1999. In order to avoid interobserver variability, only one author (SC) took charge of the CRs analysis. The following were recorded: number of randomized clinical trials (RCTs) retrieved by the CR authors on the topic, number of RCTs retained in the final analysis (that is, that fulfilled the criteria for inclusion in the CR),
1 Department of Pediatric Gastroenterology and Nutrition, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel; 2Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel; 3Department of Neonatology, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel and 4Department of Pediatrics, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel. Correspondence: Dr S Cohen, Department of Pediatric Gastroenterology and Nutrition, Tel Aviv Sourasky Medical Center, 6 Weizman Street, Tel Aviv 64239, Israel. E-mail: [email protected] Received 4 July 2013; revised 25 October 2013; accepted 1 November 2013; published online 11 December 2013
Conclusiveness of the Cochrane Reviews in Nutrition S Cohen et al
144 total cumulative number of patients enrolled in the included studies and type of conclusion reached by the reviewers. The above-mentioned information was retrieved from the complete review itself. We classified each CR in the following manner: (1) one strategy or drug is ‘better’ than the alternative; (2) there are no significant differences in strategies or drugs; (3) the quality of the studies is inadequate, thus no decision can be reached; (4) there are not enough data, thus no decision can be reached. The first two categories were defined by us as ‘conclusive’, and the last two as ‘inconclusive’.1 Importantly, conclusiveness was determined on the basis of the main outcome (as opposed to secondary outcomes). The need for additional studies and its reason(s) were recorded, whenever stated. In particular, we noted whether the stated reason was that the sample size or the number of studies was too small, there were not enough data in specific subgroups of patients or there were not enough data for analysis of rare side effects. The Minitab version 16 (State College, PA, USA) was used for statistical analyses. We used Kruskal–Wallis tests to test differences between ‘conclusive’ and ‘inconclusive’ studies, and linear regression analysis to study the correlation between number of studies and sample size, and between percentage of conclusive reviews and year of publication. Results are expressed as mean±s.d., or median and range. A P-value of o0.05 was considered significant.
RESULTS Eighty-seven CRs related to Nutrition were retrieved. Thirty-nine of them (45%) were related to neonatal nutrition topics, and 16 (18%) were related to the rest of the pediatric literature (chronic illness 15 CRs, and cancer 1 CR). There were 32 adult CRs (37%), 5 relating to pregnancy, 4 to trauma or surgery, 2 to cancer, 2 to IBD and 19 to chronic diseases or infections. Table 1 depicts the mean±s.d., median and range of number of RCTs examined, number of RCTs included for analysis, total cumulative number of patients enrolled, stated need or not for further and better studies and type of conclusion reached by the reviewers. We found that 56 CRs (64.4%) were conclusive and 31 (35.6%) inconclusive. In 23 (41.1%) of the 56 ‘conclusive’ studies, one strategy or drug was ‘better than the alternative’ and in the 33 (58.9%) remaining ones there were no significant differences between the two strategies or drugs. In 9 (29.1%) of the 31 studies classified as inconclusive, the reason for inconclusiveness was because the studies were of insufficient quality, and in 22 (70.9%) others because there were insufficient data. Table 1 depicts the comparison between conclusive and inconclusive reviews. The average number of available articles was significantly higher in conclusive reviews (P ¼ 0.002). The percentage of articles included in conclusive studies was also significantly higher than in inconclusive ones (P ¼ 0.014). Thus, the average number of RCT’s retained in the analyses was significantly higher in conclusive CRs (2.6 times higher) (P ¼ 0.001). The total cumulative number of patients enrolled in the studies retained for analysis was also significantly higher (median 11 times higher) than in the conclusive CRs (Po0.0001). The majority of inconclusive CRs (70.9%) recognized the need for further studies, a percentage not significantly different from that found in conclusive ones (58.9%, P ¼ 0.26). In 41% of the conclusive studies there was no reason given for the need for further studies, compared with 29% of the inconclusive studies. Whenever a reason was given for the need for additional studies, there were no differences between conclusive studies and inconclusive studies. In majority of the cases, the reason given was not enough studies or too small a sample size (77.7% of conclusive studies and 86.3% of inconclusive studies, P ¼ 0.40 by w2 analysis). In linear regression analysis, there was no correlation between the percentage of conclusive reviews and the year of publication. DISCUSSION We found that a relatively small majority (64.4%) of CRs in Nutrition were conclusive, contrary to what we hypothesized. In a previous study in a very different field of Medicine (that is, European Journal of Clinical Nutrition (2014) 143 – 145
Table 1.
Study characteristics of the CRs in Nutrition P
‘Conclusive’ CRs (n ¼ 56)
‘Inconclusive’ CRs (n ¼ 31)
40.7±56.0 (19, 1–260)
16.2±20.2 (12, 0–110)
0.002
Percent of included articles mean±s.d. (median, range)
46.6±25.7 (45.9, 1.81–100)
30.8±32.0 (19.6, 0–100)
0.014
Number of retained articles mean±s.d. (median, range)
12.1±12.8 (8, 1–62)
4.7±7.4 (1, 0–27)
0.001
1930±6290 (49, 0–31 833) 70.9%
o0.0001
Available articles mean±s.d. (median, range)
Total number of patients enrolled mean±s.d. 5577±21 626 (median, range) (538, 49–157 628) Percent of reviews where 58.9% further studies are needed
0.26
Abbreviation: CRs, Cochrane Reviews. Data are expressed as mean±s.d., median, range.
Neonatology), we found very similar results, in that 67.7% of Cochrane Neonatal Reviews fulfilled our criteria for being conclusive, whereas only 32.3% were inconclusive.1 Thus, from the standpoint of the practicing nutritionist/dietitian expert, a CR on a specific topic relevant to his/her field of expertise has a good likelihood of being conclusive. This fact cannot necessarily be translated into clinical decision making. Indeed, many factors may potentially limit the clinical applicability of meta-analyses, and many of them have already been discussed extensively in the Medical literature (http://en.wikipedia.org/wiki/ Cochrane_Library).2 For instance, the publication bias2,3 relates to the fact that studies with positive results are more often published than those with negative results. They even have a better chance of being published in a journal of higher impact factor than those with negative results.4 Another example of bias is the location bias, which relates to the fact that not all types of populations are similar, and therefore studies may have a certain effect in a specific population and a very different one in another population. The language bias provide a specific advantage to articles published in English,2,5 which are more widely read and understood than articles published in a less common language. Another type of bias is the citation bias, which relates to the fact that articles with negative results, less frequently quoted, might be missed during the search phase of the meta-analysis. Finally, a multiple publication bias is created when authors report more than once a positive study, increasing its chance of being found in the search.2 It may appear paradoxical that the need for further studies was very similar in conclusive (58.9%) and inconclusive (70.9%) ones. It appears that often, the reason for this need is not stated in the study, and whenever stated is very similar in conclusive and inconclusive studies (most of the time ‘not enough studies’ or ‘too small a sample size’). Rarely, the need for additional studies is ‘justified’ by the authors of the CR by the need to study some specific subgroups of patients, or by the fact that in specific subgroups there are not enough patients to reach valid conclusions. We believe that any conclusion or advice regarding the need of further studies given at the end of a CR needs more detailed information, for example, on the kind of studies, which should be done or any strategies to overcome the weakness and shortcomings of the available studies. Consistent with our hypothesis, we found that conclusiveness was dependent on both the number of studies retained in the analysis and the cumulative number of patients enrolled. This is logical, in that meta-analyses are precisely performed in order to & 2014 Macmillan Publishers Limited
Conclusiveness of the Cochrane Reviews in Nutrition S Cohen et al
145 ‘increase’ the sample size and reduce the type II error, that is, the risk of concluding that no difference exists between groups, although a true difference exists. Whether or not there is a lower limit of studies and/or patients enrolled to make a CR useful to do obviously depends on the effect size of a given research question. Presumably, the greater the effect size the lower the number of cumulative patients needed. The number of CRs in each subcategory of nutrition topic was too small to make meaningful analyses of whether a particular topic is more likely or not to reach conclusiveness. We suspect that CRs partly follow the ‘fashionable’ topics of research in clinical nutrition, thus, it might be asked why these specific 87 research topics had been taken as questions for CRs, rather than other topics. We conclude that in the field of Nutrition, CRs are a useful and potent clinical tool that allows, in 64.4% of the cases, to reach a valid clinical conclusion, although emphasizing the strength of the available evidence.
& 2014 Macmillan Publishers Limited
CONFLICT OF INTEREST The authors declare no conflict of interest.
REFERENCES 1 Mandel D, Littner Y, Mimouni FB, Lubetzky R. Conclusiveness of the Cochrane Neonatal Reviews: a systematic analysis. Acta Paediatr 2006; 95: 1209–1212. 2 Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–634. 3 Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000; 53: 1119–1129. 4 Littner Y, Mimouni FB, Dollberg S, Mandel D. Negative results and impact factor: a lesson from neonatology. Arch Pediatr Adolesc Med 2005; 159: 1036–1037. 5 Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997; 350: 326–329.
European Journal of Clinical Nutrition (2014) 143 – 145
REVIEW
Conclusiveness of the Cochrane Reviews in Nutrition: a systematic analysis S Cohen1,2, D Mandel2,3, FB Mimouni2,4, R Marom2,3 and R Lubetzky2,4 BACKGROUND/OBJECTIVE: To assess the conclusiveness of the Cochrane Reviews (CRs) in the field of Nutrition, we tested the hypotheses that: (1) the majority of CRs is inconclusive; (2) the majority of CRs recognizes the need for further and better studies and (3) the ability to reach a conclusion is dependent on the number of studies performed and number of patients enrolled. SUBJECTS/METHODS: We selected all 87 CRs in the field of Nutrition available in Cochrane library. Each CR was analyzed for the number of randomized clinical trials (RCTs) found, number of RCTs included for analysis, number of patients enrolled, the stated need for further studies and the reason(s) for it and the conclusiveness of the CR. RESULTS: Fifty-six out of eighty-seven CRs (64.4%) were conclusive. The average number of available articles, the percentage of articles included, the average number of RCT’s retained in the analyses and the total cumulative number of patients enrolled in the studies retained for analysis were significantly higher in conclusive CRs than in non-conclusive ones. The majority of inconclusive CRs (70.9%) recognized the need for further studies, a percentage not significantly different from that found in conclusive ones (58.9%, P ¼ 0.26). The percentage of conclusive CRs was not affected by year of publication. CONCLUSIONS: The majority of CRs in Nutrition is conclusive, but most of them emphasize the need for further studies. The ability for a CR to reach a conclusion is affected by the cumulative patient sample size and number of RCT’s included in the analysis. European Journal of Clinical Nutrition (2014) 68, 143–145; doi:10.1038/ejcn.2013.252; published online 11 December 2013 Keywords: nutrition; meta-analysis; randomized clinical trial
INTRODUCTION The Cochrane Collaboration is an international, independent, not-for-profit organization of over 28 000 contributors from more than 100 countries (http://www.cochrane.org). Its stated aims are to ‘making up-to-date, accurate information about the effects of health care readily available worldwide’ (http://www.cochrane.org). The contributors of the Cochrane Reviews (CR) work together to produce systematic reviews of health care interventions, which are then published online in the Cochrane Library. An intended aim of the CRs is to ‘help providers, practitioners and patients make informed decisions about health care’ based on ‘comprehensive, reliable and relevant source of evidence on which to base these decisions’ (http://www.cochrane.org). The CRs constitute a large database of systematic reviews and meta-analyses, and have become a key resource in evidencebased medicine (http://en.wikipedia.org/wiki/Cochrane_Library). From a few reviews published in 1995, at the inception of the project, there are currently more than 5000 reviews available (http://www.cochrane.org). The CRs can be withdrawn from the active database when they become out-of-date or are replaced by new CRs in a similar subject area (http://www.cochrane.org). On 7 August 2012, the database comprised 5131 published reviews, 2235 protocols, of which 29 were new reviews, 29 updated reviews, 68 new protocols and 10 updated protocols, at the same time there was 1 withdrawn review and 5 withdrawn protocols. Typically, a given CR provides a systematic analysis of available clinical trials, and ends up in a specific recommendation on the
efficacy of a particular medical intervention (for example, surgical or pharmaceutical) (http://www.cochrane.org). Often, because the CR did not find any strong evidence for the efficacy of a particular treatment or strategy, it ends up recommending that ‘additional medical studies should be performed before a final conclusion can be reached’ (http://www.cochrane.org). Therefore, we conducted the present study to systematically analyze the conclusions and recommendations of all CRs related to the field of Nutrition available until 1 July 2012. We aimed to verify whether the reviews are conclusive or not from a clinical practice standpoint. For this purpose, we defined conclusiveness as the ability or not to conclude that a specific treatment was beneficial.1 We tested the hypotheses that: (1) the majority of CRs is inconclusive; (2) the majority of CRs recognizes the need for further and better studies and (3) the ability to reach a conclusion is dependent on both the number of studies performed and number of patients enrolled. A ‘time effect’ was also considered, as we hypothesized that newer CRs are more likely to be conclusive than older ones. MATERIALS AND METHODS We selected all 87 CRs in Nutrition available in the Cochrane library (http://www.cochrane.org) on 15 June 2012, starting from 1999. In order to avoid interobserver variability, only one author (SC) took charge of the CRs analysis. The following were recorded: number of randomized clinical trials (RCTs) retrieved by the CR authors on the topic, number of RCTs retained in the final analysis (that is, that fulfilled the criteria for inclusion in the CR),
1 Department of Pediatric Gastroenterology and Nutrition, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel; 2Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel; 3Department of Neonatology, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel and 4Department of Pediatrics, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel. Correspondence: Dr S Cohen, Department of Pediatric Gastroenterology and Nutrition, Tel Aviv Sourasky Medical Center, 6 Weizman Street, Tel Aviv 64239, Israel. E-mail: [email protected] Received 4 July 2013; revised 25 October 2013; accepted 1 November 2013; published online 11 December 2013
Conclusiveness of the Cochrane Reviews in Nutrition S Cohen et al
144 total cumulative number of patients enrolled in the included studies and type of conclusion reached by the reviewers. The above-mentioned information was retrieved from the complete review itself. We classified each CR in the following manner: (1) one strategy or drug is ‘better’ than the alternative; (2) there are no significant differences in strategies or drugs; (3) the quality of the studies is inadequate, thus no decision can be reached; (4) there are not enough data, thus no decision can be reached. The first two categories were defined by us as ‘conclusive’, and the last two as ‘inconclusive’.1 Importantly, conclusiveness was determined on the basis of the main outcome (as opposed to secondary outcomes). The need for additional studies and its reason(s) were recorded, whenever stated. In particular, we noted whether the stated reason was that the sample size or the number of studies was too small, there were not enough data in specific subgroups of patients or there were not enough data for analysis of rare side effects. The Minitab version 16 (State College, PA, USA) was used for statistical analyses. We used Kruskal–Wallis tests to test differences between ‘conclusive’ and ‘inconclusive’ studies, and linear regression analysis to study the correlation between number of studies and sample size, and between percentage of conclusive reviews and year of publication. Results are expressed as mean±s.d., or median and range. A P-value of o0.05 was considered significant.
RESULTS Eighty-seven CRs related to Nutrition were retrieved. Thirty-nine of them (45%) were related to neonatal nutrition topics, and 16 (18%) were related to the rest of the pediatric literature (chronic illness 15 CRs, and cancer 1 CR). There were 32 adult CRs (37%), 5 relating to pregnancy, 4 to trauma or surgery, 2 to cancer, 2 to IBD and 19 to chronic diseases or infections. Table 1 depicts the mean±s.d., median and range of number of RCTs examined, number of RCTs included for analysis, total cumulative number of patients enrolled, stated need or not for further and better studies and type of conclusion reached by the reviewers. We found that 56 CRs (64.4%) were conclusive and 31 (35.6%) inconclusive. In 23 (41.1%) of the 56 ‘conclusive’ studies, one strategy or drug was ‘better than the alternative’ and in the 33 (58.9%) remaining ones there were no significant differences between the two strategies or drugs. In 9 (29.1%) of the 31 studies classified as inconclusive, the reason for inconclusiveness was because the studies were of insufficient quality, and in 22 (70.9%) others because there were insufficient data. Table 1 depicts the comparison between conclusive and inconclusive reviews. The average number of available articles was significantly higher in conclusive reviews (P ¼ 0.002). The percentage of articles included in conclusive studies was also significantly higher than in inconclusive ones (P ¼ 0.014). Thus, the average number of RCT’s retained in the analyses was significantly higher in conclusive CRs (2.6 times higher) (P ¼ 0.001). The total cumulative number of patients enrolled in the studies retained for analysis was also significantly higher (median 11 times higher) than in the conclusive CRs (Po0.0001). The majority of inconclusive CRs (70.9%) recognized the need for further studies, a percentage not significantly different from that found in conclusive ones (58.9%, P ¼ 0.26). In 41% of the conclusive studies there was no reason given for the need for further studies, compared with 29% of the inconclusive studies. Whenever a reason was given for the need for additional studies, there were no differences between conclusive studies and inconclusive studies. In majority of the cases, the reason given was not enough studies or too small a sample size (77.7% of conclusive studies and 86.3% of inconclusive studies, P ¼ 0.40 by w2 analysis). In linear regression analysis, there was no correlation between the percentage of conclusive reviews and the year of publication. DISCUSSION We found that a relatively small majority (64.4%) of CRs in Nutrition were conclusive, contrary to what we hypothesized. In a previous study in a very different field of Medicine (that is, European Journal of Clinical Nutrition (2014) 143 – 145
Table 1.
Study characteristics of the CRs in Nutrition P
‘Conclusive’ CRs (n ¼ 56)
‘Inconclusive’ CRs (n ¼ 31)
40.7±56.0 (19, 1–260)
16.2±20.2 (12, 0–110)
0.002
Percent of included articles mean±s.d. (median, range)
46.6±25.7 (45.9, 1.81–100)
30.8±32.0 (19.6, 0–100)
0.014
Number of retained articles mean±s.d. (median, range)
12.1±12.8 (8, 1–62)
4.7±7.4 (1, 0–27)
0.001
1930±6290 (49, 0–31 833) 70.9%
o0.0001
Available articles mean±s.d. (median, range)
Total number of patients enrolled mean±s.d. 5577±21 626 (median, range) (538, 49–157 628) Percent of reviews where 58.9% further studies are needed
0.26
Abbreviation: CRs, Cochrane Reviews. Data are expressed as mean±s.d., median, range.
Neonatology), we found very similar results, in that 67.7% of Cochrane Neonatal Reviews fulfilled our criteria for being conclusive, whereas only 32.3% were inconclusive.1 Thus, from the standpoint of the practicing nutritionist/dietitian expert, a CR on a specific topic relevant to his/her field of expertise has a good likelihood of being conclusive. This fact cannot necessarily be translated into clinical decision making. Indeed, many factors may potentially limit the clinical applicability of meta-analyses, and many of them have already been discussed extensively in the Medical literature (http://en.wikipedia.org/wiki/ Cochrane_Library).2 For instance, the publication bias2,3 relates to the fact that studies with positive results are more often published than those with negative results. They even have a better chance of being published in a journal of higher impact factor than those with negative results.4 Another example of bias is the location bias, which relates to the fact that not all types of populations are similar, and therefore studies may have a certain effect in a specific population and a very different one in another population. The language bias provide a specific advantage to articles published in English,2,5 which are more widely read and understood than articles published in a less common language. Another type of bias is the citation bias, which relates to the fact that articles with negative results, less frequently quoted, might be missed during the search phase of the meta-analysis. Finally, a multiple publication bias is created when authors report more than once a positive study, increasing its chance of being found in the search.2 It may appear paradoxical that the need for further studies was very similar in conclusive (58.9%) and inconclusive (70.9%) ones. It appears that often, the reason for this need is not stated in the study, and whenever stated is very similar in conclusive and inconclusive studies (most of the time ‘not enough studies’ or ‘too small a sample size’). Rarely, the need for additional studies is ‘justified’ by the authors of the CR by the need to study some specific subgroups of patients, or by the fact that in specific subgroups there are not enough patients to reach valid conclusions. We believe that any conclusion or advice regarding the need of further studies given at the end of a CR needs more detailed information, for example, on the kind of studies, which should be done or any strategies to overcome the weakness and shortcomings of the available studies. Consistent with our hypothesis, we found that conclusiveness was dependent on both the number of studies retained in the analysis and the cumulative number of patients enrolled. This is logical, in that meta-analyses are precisely performed in order to & 2014 Macmillan Publishers Limited
Conclusiveness of the Cochrane Reviews in Nutrition S Cohen et al
145 ‘increase’ the sample size and reduce the type II error, that is, the risk of concluding that no difference exists between groups, although a true difference exists. Whether or not there is a lower limit of studies and/or patients enrolled to make a CR useful to do obviously depends on the effect size of a given research question. Presumably, the greater the effect size the lower the number of cumulative patients needed. The number of CRs in each subcategory of nutrition topic was too small to make meaningful analyses of whether a particular topic is more likely or not to reach conclusiveness. We suspect that CRs partly follow the ‘fashionable’ topics of research in clinical nutrition, thus, it might be asked why these specific 87 research topics had been taken as questions for CRs, rather than other topics. We conclude that in the field of Nutrition, CRs are a useful and potent clinical tool that allows, in 64.4% of the cases, to reach a valid clinical conclusion, although emphasizing the strength of the available evidence.
& 2014 Macmillan Publishers Limited
CONFLICT OF INTEREST The authors declare no conflict of interest.
REFERENCES 1 Mandel D, Littner Y, Mimouni FB, Lubetzky R. Conclusiveness of the Cochrane Neonatal Reviews: a systematic analysis. Acta Paediatr 2006; 95: 1209–1212. 2 Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–634. 3 Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000; 53: 1119–1129. 4 Littner Y, Mimouni FB, Dollberg S, Mandel D. Negative results and impact factor: a lesson from neonatology. Arch Pediatr Adolesc Med 2005; 159: 1036–1037. 5 Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997; 350: 326–329.
European Journal of Clinical Nutrition (2014) 143 – 145
