LETTER TO THE EDITOR Meta-analyses in Surgical Publications: How Qualitative Is This Evidence-based Information? To the Editor: vidence-based information constitutes both the target and the success of every ambitious contemporary study, as well as the necessary presupposition for performing evidence-based medicine. One valuable tool for the evaluation of the existing evidence is the meta-analysis, which is a systematic method that takes data from a number of independent studies and integrates them using statistical analysis. Meta-analyses are considered to provide level I-II evidence.1 Although bias is minimized by the use of explicit statistical methods, the overall suboptimal quality of meta-analyses frequently limits the accuracy and reliability of the conclusions. Indeed, there is considerable evidence that key information is often poorly addressed, and that data reporting does not allow for an accurate assessment of the strengths and weaknesses of the investigation.2 These findings led to the development of the QUOROM (QUality Of Reporting Of Meta-analyses) statement3 on standards for meta-analyses of clinical randomized controlled trials. In this study, we aimed to assess the quality of information delivered by metaanalyses by means of QUOROM statement in all meta-analyses published in Annals of Surgery, the highest ranked surgical journal, from March 2000 to March 2011. An Excel database encompassing 29 parameters was constructed on the basis of the QUOROM statement. Individual publications of each meta-analysis were also assessed. Thirty-one consecutive meta-analyses were included in this study (Table 1). The number of meta-analyses fulfilling each of the parameters considered was as follows: information obtained from more than 2 databases 23/31; language of publication exclusively English 25/31; defined population, intervention, and principal outcomes 31/31; study design encompassing review of randomized controlled trials 10/31; quality assessment of contributing publications 10/31; handling of missing data 10/31; assessment of statistical heterogeneity 30/31; subgroup analysis
E
Disclosure: The authors declare no conflicts of interest. C 2013 Wolters Kluwer Health, Inc. All Copyright rights reserved. ISSN: 0003-4932/13/26105-e0129 DOI: 10.1097/SLA.0000000000000440
23/31; assessment of publication bias 26/31; agreement on selection and validity assessment 22/31; simple summary results 28/31; data available to calculate effect size and confidence interval 27/31; key findings summarized 30/31; clinical inferences based on internal and external validity 24/31; description of potential biases in the review process 23/31; and future research agenda suggested 18/31. Our study demonstrated a trend toward greater compliance with specific QUOROM guidelines on years considered, inclusion and exclusion criteria, definition of the selected
population, information on the studied type of intervention, and principal outcomes. Good compliance with QUOROM guidelines was also noted with information on principal measures of effect (eg, odds ratio, weighted averages etc.) and statistical methods of combining results. Statistical heterogeneity was widely calculated and reported. Ninety-seven percent of the reviewed studies provided a meta-analysis profile summarizing trial flow according to QUOROM guidelines. However, considerable variability was noticed in QUOROM quality assessment
TABLE 1. First Author, Publication Year, Short Title, and Number of Contributing Publications for Each of the Studies Reviewed First Author Publication Year Hodgson 2000 EU Hernia Trialists Collaboration 2002 Sewnath 2002 Vincent 2003 Hall 2004 Mulier 2005 Aziz 2006 Lovegrove 2006 Andersson 2007 Diener 2007 Sanabria 2007 Abulkhir 2008 Diener 2008 Ferreyra 2008 Hsia 2008 H¨uer 2008 Karanikolas 2008 Nanidis 2008 Petrov 2008 Treadwell 2008 Campos 2009 Slim 2009 Zhao 2009 Diener 2010 Lanitis 2010 Lansdale 2010 Maeso 2010 Mingtai 2010 Ahmad 2011 Jay 2011 Pontiroli 2011
Annals of Surgery r Volume 261, Number 5, May 2015
Short Title
Included Studies
Ideal Method of Abdominal Fascial Closure Repair of Groin Hernia With Synthetic Mesh
13 58
Preoperative Biliary Drainage for Tumors Causing Obstructive Jaundice Hypoalbuminemia in Acute Illness Laparoscopic Versus Open Pyloromyotomy Local Recurrence After Hepatic Radiofrequency Coagulation Laparoscopic vs Open Appendectomy in Children Hand Sewn vs Stapled Ileal Pouch Anal Anastomosis Following Proctocolectomy Nonsurgical Treatment of Appendiceal Abscess or Phlegmon Pylorus Preserving vs Classic Pancreaticoduodenectomy for Periampullary and Pancreatic Carcinoma Prophylactic Antibiotics for Mesh Inguinal Hernioplasty Preoperative Portal Vein Embolization for Major Liver Resection Duodenum Preserving Pancreas Head Resection vs Pancreatoduodenectomy in Chronic Pancreatitis Positive Airway Pressure After Abdominal Surgery Recombinant Activated Factor VII in Patients Without Hemophilia Defunctioning Stroma in Low Rectal Cancer Surgery Prophylactic Dexamethasone on Nausea and Vomiting After Laparoscopic Cholecystectomy Laparoscopic Versus Open Donor Nephrectomy in Renal Transplantation ERCP vs Conservative Management in Acute Biliary Pancreatitis/Cholangitis Bariatric Surgery for Pediatric Patients Endoscopic and Surgical Treatment of Achalasia Mechanical Bowel Preparation Before Colorectal Surgery Open Mesh Techniques for Inguinal Hernia Repair Elective Midline Laparotomy Closure Skin-Sparing Mastectomy vs Non-Skin-Sparing Mastectomy for Breast Cancer Neonatal Endosurgical Congenital Diaphragmatic Hernia Repair Da Vinci Surgical System vs Laparoscopy in Abdominal Surgery Vypro II Mesh for Inguinal Hernia Repair Surgical Scalpel or Diathermy in Abdominal Skin Incisions Ischemic Cholangiopathy After controlled donation After Cardiac Death Liver Transplantation Mortality Prevention in Morbid Obesity Through Bariatric Surgery
23 90 8 95 23 21 61 6 6 37 4 9 22 27 17 73 3 19 105 14 10 19 9 3 31 10 11 11 8
www.annalsofsurgery.com | e129
Copyright © 2013 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
REFERENCES
FIGURE 1. Compliance with the QUOROM statement guidelines of each study included, concerning some quality parameters: Review of only randomized controlled trials, quality assessment, information on data abstraction, handling of missing data, agreement on selection/validity assessment, and future research agenda. measures (Fig. 1). Only one third of the metaanalyses reviewed randomized controlled trials, the study design with the highest level of evidence. Most importantly, the quality of the contributing publications was assessed in only one third of the reviewed studies, using, for example, either the Jadad Quality Scale4 or the Newcastle-Ottawa Scale.5 This represents a potentially significant shortcoming of meta-analyses, given that inclusion of studies of unknown quality may influence the reported observations. The possibility that the summary measures of a particular intervention approach the truth depends on the quality, methodological characteristics, and risk of bias of the contributing publications.6,7 It is essential to evaluate the quality and risk of bias of each single publication, and to clearly describe the methods used for this purpose. An additional issue that merits consideration is that of missing data. Hardly one third of the reviewed studies reported their handling of missing data. The absence of information from particular studies may jeopardize the validity of results. Studies with statistically significant results tend to be published more rapidly and more frequently than those with no observed differences—resulting in “missing studies.”8 Moreover, not all eligible studies may report results on the particular outcome being considered—leading to “missing outcomes.” Both “missing studies” and “missing outcomes” can result in selection bias,8 a fact that further highlights the need for clear reporting on the handling of missing data. e130 | www.annalsofsurgery.com
1. Guyatt G, Rennie D, Meade MO, et al. Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. New York, NY: McGraw-Hill Medical; 2008. 2. Dixon E, Hameed M, Sutherland F, et al. Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann Surg. 2005;241:450-459. 3. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–1900. 4. Clark HD, Wells GA, Huet C, et al. Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials. 1999;20:448–452. 5. Stang A. Critical evaluation of the NewcastleOttawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25:603–605. 6. Gluud LL. Bias in clinical intervention research. Am J Epidemiol. 2006;163:493–501. 7. Pildal J, Hrobjartsson A, Jorgensen KJ, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36:847–857. 8. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6:e1000100.
CONCLUSIONS This review demonstrates that although very good compliance was observed for particular QUOROM guidelines, a remarkable number of studies failed to follow important QUOROM guidelines. These guidelines need to be addressed by future reviewers in an effort to improve the validity and reproducibility of the manuscripts considered for publication. Moreover, this study implies that despite the QUOROM publication, results of current meta-analyses and systematic reviews are still subject to validity issues and should be interpreted with caution.
ACKNOWLEDGMENTS The authors want to thank Dr Athanasios Katsargyris, Mr Savvas Alaeddine, and Prof Ernesto P. Molmenti for their contribution to this work and Prof Markus Neuh¨auser for his advice and support in statistical issues. Georgios C. Sotiropoulos, MD, FACS, FEBS Department of General, Visceral and Transplantation Surgery University Hospital Essen Essen, Germany Christos Dervenis, MD, FRCS First Department of Surgery Agia Olga General Hospital Athens, Greece [email protected]
Reply:
T
he past decade has seen the development and diffusion of many recommended standards for different methodological types of peer-reviewed publications to ensure the validity of the articles and assist readers and reviewers in the proper evaluation of the presented information. The example best known to most clinicians is the CONSORT statement,1 which applies to traditional randomized trials. Similarly, the STROBE statement2 was developed for the reporting of observational studies. Perhaps, less well known is the existence of similar guidelines for the presentation of meta-analyses: MOOSE for meta-analyses of observational studies3 and QUOROM for meta-analyses of randomized trials.4 QUOROM was near simultaneously published in multiple journals, including the British Journal of Surgery, although no specific mention of recommendations with regard to surgical trials was included. The QUOROM statement consists essentially of a checklist (18 items) and a flow diagram; it is an evaluative tool for metaanalyses. Although most often used qualitatively, it has also been used by some to attribute a score to individual publications.5 Disclosure: No funding was received in support of the writing of this editorial, and there are no conflicts of interest. DOI: 10.1097/SLA.0000000000000441
C 2013 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2013 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
In 2008, Knobloch et al6 published a commentary about a 2003 study that had found the use of QUOROM lacking in the reporting of surgical trials.7 Their letter reviewed 8 meta-analyses of randomized trials published in the Annals of Surgery alone between 2006 and 2007 and suggested a persistently low reference and adherence to QUOROM: only one study mentioned the QUOROM statement (12.5%) and only 2 published a workflow chart (25%). In response, the Annals of Surgery strengthened its statement of compliance with QUOROM in its instructions to authors. In this issue, Sotiropoulos and Dervenis8 report a more detailed audit of 31 consecutive meta-analyses published in the Annals of Surgery from 2000 to 2011. It is encouraging to note that there has been an increase in the overall compliance with QUOROM. Furthermore, many aspects of the QUOROM checklist seem to have been reported including general information and measures of effect. However, items related to “quality assessment measures” showed variability and infrequent reporting (only one third of studies). In some cases, this might have been anticipated: one cannot expect to identify or include a large number of randomized trials (which are specifically targeted by the QUOROM statement) if only less than 10% of mainstream surgical works use a randomized design.9 It might have been more appropriate for some of these reviewed meta-analyses to use the MOOSE recommendations. Sotiropoulos and Dervenis also comment on the lack of use of a formal study quality “scoring system” such as the Jadad quality scale; however, in its most recent update, The Cochrane Collaboration discourages the use of such a scoring system,10 favoring the semiquantitative Cochrane Risk Bias tool.11 Nevertheless, as highlighted by the authors, the absence of a formal assessment of the risk of bias of a study and its subsequent inclusion in a summary analysis clearly open the door to potentially biased assessments and resulting erroneous meta-analytical conclusions. This speaks to the potential illusion that can be created by the quantitative part of a meta-analysis, as summarized by Ioannidis: “simply putting problematic data together does not overcome their (original studies) problems,”12 perhaps better known as “garbage in . . . garbage out (GIGO).” Finally, Sotiropoulos and Dervenis also raise the issue of missing data—either at the study level (eg, excluded patients, dif-
ferential outcome validity) or in the literature by ignoring “gray literature” and possibly perpetuating publication bias. Many of these stated weaknesses are not unique to surgical meta-analyses and have in fact spearheaded the development of a new reporting standard that is replacing QUOROM: the PRISMA statement.13 The PRISMA statement comprises a 4-phase flow diagram and a checklist expanded to 27 items; unlike QUOROM, it can be more broadly applied to systematic reviews and meta-analyses. Several aspects may be particularly attractive to surgery: PRISMA does not exclusively apply to meta-analyses of randomized trials, which remain the exception in the surgical literature. It can also be used to target reviews of the evaluation of interventions, which are a common topic in the surgical literature. PRISMA recommends to authors a PICOS (Population, Intervention, Comparator, Outcome, Study design) approach in the development of their clinical question that strengthens the way they will plan gathering information about the intervention and outcomes, traditional surgical study reporting weak points. Also, the evaluation of bias requires assessments not just at the study level but also at the outcome level. The latter would be particularly pertinent in surgical trials not only where the design may be strong (eg, randomization scheme, on-table randomization) but also where the recording of certain outcomes may be inconsistent (eg, ad hoc or not validated scheme used to record complications). PRISMA also suggests that reviewers search for the presence of a registered protocol (mostly for randomized trials) to avoid the potential for bias of selective outcome reporting or an opportunistic late change in what had been the originally planned primary outcome or timing of analysis.14,15 In summary, risk of bias in the reporting of evidence generated from metaanalyses represents an increasing challenge, especially in surgical meta-analyses and despite QUOROM; this challenge is amplified by the ever-increasing number of metaanalyses being submitted to journals. Annals of Surgery may help meet this challenge by strengthening the recommendations to authors and reviewers as pertains to the new PRISMA statement. Jeffrey S. Barkun, MD Department of Surgery McGill University/Royal Victoria Hospital
C 2013 Wolters Kluwer Health, Inc. All rights reserved.
Montreal, Quebec, Canada [email protected]
REFERENCES 1. Moher D, Schulz KF, Altman D, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285:1987–1991. 2. STROBE Statement. STrengthening the Reporting of Observational studies in Epidemiology. March 30, 2008. Available at: http://www .strobe-statement.org. Accessed September 25, 2013. 3. Stroup DF, Berlin JA, Morton SC, et al. Metaanalysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–2012. 4. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–1900. 5. Jin Wena, Yu Renb, Li Wang, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol. 2008;61: 770–775. 6. Knobloch K, Gohritz A, Vogt PM. CONSORT and QUOROM statements revisited: standards of reporting of randomized controlled trials in general surgery. Ann Surg. 2008;248:1106–1107. 7. Balasubramanian SP, Wiener M, Alshameeri Z, et al. Standards of reporting of randomized controlled trials in general surgery. Ann Surg. 2006;244:663–667. 8. Sotiropoulos GC, Dervenis C. Meta-analyses in surgical publications: how qualitative is this evidence-based information? Ann Surg. 2015;261:e129–e130. 9. Chang DC, Matsen SL, Simpkins CE. Why should surgeons care about clinical research methodology? J Am Coll Surg. 2006;203:827–830. 10. The Cochrane Collaboration. Chapter 8: Assessing risk of bias in included studies. In: Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. Available at: http://www.cochrane.org/sites/default/files/uploads/ handbook/Handbook510pdf_Ch08_RiskOfBias.pdf. Accessed September 27, 2013. 11. The Cochrane Collaboration. Assessment of study quality. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. Updated March 2011. 12. Ioannidis JP. Meta-analysis in public health: potentials and problems. Ital J Public Health. 2006;3:9–14. 13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6:e1000100. 14. Hannink G, Gooszen HG, Rovers MM. Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann Surg. 2013;257:818–823. 15. Singh D. Merck withdraws arthritis drug worldwide. BMJ. 2004;329:816.
www.annalsofsurgery.com | e131
Copyright © 2013 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E
Disclosure: The authors declare no conflicts of interest. C 2013 Wolters Kluwer Health, Inc. All Copyright rights reserved. ISSN: 0003-4932/13/26105-e0129 DOI: 10.1097/SLA.0000000000000440
23/31; assessment of publication bias 26/31; agreement on selection and validity assessment 22/31; simple summary results 28/31; data available to calculate effect size and confidence interval 27/31; key findings summarized 30/31; clinical inferences based on internal and external validity 24/31; description of potential biases in the review process 23/31; and future research agenda suggested 18/31. Our study demonstrated a trend toward greater compliance with specific QUOROM guidelines on years considered, inclusion and exclusion criteria, definition of the selected
population, information on the studied type of intervention, and principal outcomes. Good compliance with QUOROM guidelines was also noted with information on principal measures of effect (eg, odds ratio, weighted averages etc.) and statistical methods of combining results. Statistical heterogeneity was widely calculated and reported. Ninety-seven percent of the reviewed studies provided a meta-analysis profile summarizing trial flow according to QUOROM guidelines. However, considerable variability was noticed in QUOROM quality assessment
TABLE 1. First Author, Publication Year, Short Title, and Number of Contributing Publications for Each of the Studies Reviewed First Author Publication Year Hodgson 2000 EU Hernia Trialists Collaboration 2002 Sewnath 2002 Vincent 2003 Hall 2004 Mulier 2005 Aziz 2006 Lovegrove 2006 Andersson 2007 Diener 2007 Sanabria 2007 Abulkhir 2008 Diener 2008 Ferreyra 2008 Hsia 2008 H¨uer 2008 Karanikolas 2008 Nanidis 2008 Petrov 2008 Treadwell 2008 Campos 2009 Slim 2009 Zhao 2009 Diener 2010 Lanitis 2010 Lansdale 2010 Maeso 2010 Mingtai 2010 Ahmad 2011 Jay 2011 Pontiroli 2011
Annals of Surgery r Volume 261, Number 5, May 2015
Short Title
Included Studies
Ideal Method of Abdominal Fascial Closure Repair of Groin Hernia With Synthetic Mesh
13 58
Preoperative Biliary Drainage for Tumors Causing Obstructive Jaundice Hypoalbuminemia in Acute Illness Laparoscopic Versus Open Pyloromyotomy Local Recurrence After Hepatic Radiofrequency Coagulation Laparoscopic vs Open Appendectomy in Children Hand Sewn vs Stapled Ileal Pouch Anal Anastomosis Following Proctocolectomy Nonsurgical Treatment of Appendiceal Abscess or Phlegmon Pylorus Preserving vs Classic Pancreaticoduodenectomy for Periampullary and Pancreatic Carcinoma Prophylactic Antibiotics for Mesh Inguinal Hernioplasty Preoperative Portal Vein Embolization for Major Liver Resection Duodenum Preserving Pancreas Head Resection vs Pancreatoduodenectomy in Chronic Pancreatitis Positive Airway Pressure After Abdominal Surgery Recombinant Activated Factor VII in Patients Without Hemophilia Defunctioning Stroma in Low Rectal Cancer Surgery Prophylactic Dexamethasone on Nausea and Vomiting After Laparoscopic Cholecystectomy Laparoscopic Versus Open Donor Nephrectomy in Renal Transplantation ERCP vs Conservative Management in Acute Biliary Pancreatitis/Cholangitis Bariatric Surgery for Pediatric Patients Endoscopic and Surgical Treatment of Achalasia Mechanical Bowel Preparation Before Colorectal Surgery Open Mesh Techniques for Inguinal Hernia Repair Elective Midline Laparotomy Closure Skin-Sparing Mastectomy vs Non-Skin-Sparing Mastectomy for Breast Cancer Neonatal Endosurgical Congenital Diaphragmatic Hernia Repair Da Vinci Surgical System vs Laparoscopy in Abdominal Surgery Vypro II Mesh for Inguinal Hernia Repair Surgical Scalpel or Diathermy in Abdominal Skin Incisions Ischemic Cholangiopathy After controlled donation After Cardiac Death Liver Transplantation Mortality Prevention in Morbid Obesity Through Bariatric Surgery
23 90 8 95 23 21 61 6 6 37 4 9 22 27 17 73 3 19 105 14 10 19 9 3 31 10 11 11 8
www.annalsofsurgery.com | e129
Copyright © 2013 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
REFERENCES
FIGURE 1. Compliance with the QUOROM statement guidelines of each study included, concerning some quality parameters: Review of only randomized controlled trials, quality assessment, information on data abstraction, handling of missing data, agreement on selection/validity assessment, and future research agenda. measures (Fig. 1). Only one third of the metaanalyses reviewed randomized controlled trials, the study design with the highest level of evidence. Most importantly, the quality of the contributing publications was assessed in only one third of the reviewed studies, using, for example, either the Jadad Quality Scale4 or the Newcastle-Ottawa Scale.5 This represents a potentially significant shortcoming of meta-analyses, given that inclusion of studies of unknown quality may influence the reported observations. The possibility that the summary measures of a particular intervention approach the truth depends on the quality, methodological characteristics, and risk of bias of the contributing publications.6,7 It is essential to evaluate the quality and risk of bias of each single publication, and to clearly describe the methods used for this purpose. An additional issue that merits consideration is that of missing data. Hardly one third of the reviewed studies reported their handling of missing data. The absence of information from particular studies may jeopardize the validity of results. Studies with statistically significant results tend to be published more rapidly and more frequently than those with no observed differences—resulting in “missing studies.”8 Moreover, not all eligible studies may report results on the particular outcome being considered—leading to “missing outcomes.” Both “missing studies” and “missing outcomes” can result in selection bias,8 a fact that further highlights the need for clear reporting on the handling of missing data. e130 | www.annalsofsurgery.com
1. Guyatt G, Rennie D, Meade MO, et al. Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. New York, NY: McGraw-Hill Medical; 2008. 2. Dixon E, Hameed M, Sutherland F, et al. Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann Surg. 2005;241:450-459. 3. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–1900. 4. Clark HD, Wells GA, Huet C, et al. Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials. 1999;20:448–452. 5. Stang A. Critical evaluation of the NewcastleOttawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25:603–605. 6. Gluud LL. Bias in clinical intervention research. Am J Epidemiol. 2006;163:493–501. 7. Pildal J, Hrobjartsson A, Jorgensen KJ, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36:847–857. 8. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6:e1000100.
CONCLUSIONS This review demonstrates that although very good compliance was observed for particular QUOROM guidelines, a remarkable number of studies failed to follow important QUOROM guidelines. These guidelines need to be addressed by future reviewers in an effort to improve the validity and reproducibility of the manuscripts considered for publication. Moreover, this study implies that despite the QUOROM publication, results of current meta-analyses and systematic reviews are still subject to validity issues and should be interpreted with caution.
ACKNOWLEDGMENTS The authors want to thank Dr Athanasios Katsargyris, Mr Savvas Alaeddine, and Prof Ernesto P. Molmenti for their contribution to this work and Prof Markus Neuh¨auser for his advice and support in statistical issues. Georgios C. Sotiropoulos, MD, FACS, FEBS Department of General, Visceral and Transplantation Surgery University Hospital Essen Essen, Germany Christos Dervenis, MD, FRCS First Department of Surgery Agia Olga General Hospital Athens, Greece [email protected]
Reply:
T
he past decade has seen the development and diffusion of many recommended standards for different methodological types of peer-reviewed publications to ensure the validity of the articles and assist readers and reviewers in the proper evaluation of the presented information. The example best known to most clinicians is the CONSORT statement,1 which applies to traditional randomized trials. Similarly, the STROBE statement2 was developed for the reporting of observational studies. Perhaps, less well known is the existence of similar guidelines for the presentation of meta-analyses: MOOSE for meta-analyses of observational studies3 and QUOROM for meta-analyses of randomized trials.4 QUOROM was near simultaneously published in multiple journals, including the British Journal of Surgery, although no specific mention of recommendations with regard to surgical trials was included. The QUOROM statement consists essentially of a checklist (18 items) and a flow diagram; it is an evaluative tool for metaanalyses. Although most often used qualitatively, it has also been used by some to attribute a score to individual publications.5 Disclosure: No funding was received in support of the writing of this editorial, and there are no conflicts of interest. DOI: 10.1097/SLA.0000000000000441
C 2013 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2013 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery r Volume 261, Number 5, May 2015
In 2008, Knobloch et al6 published a commentary about a 2003 study that had found the use of QUOROM lacking in the reporting of surgical trials.7 Their letter reviewed 8 meta-analyses of randomized trials published in the Annals of Surgery alone between 2006 and 2007 and suggested a persistently low reference and adherence to QUOROM: only one study mentioned the QUOROM statement (12.5%) and only 2 published a workflow chart (25%). In response, the Annals of Surgery strengthened its statement of compliance with QUOROM in its instructions to authors. In this issue, Sotiropoulos and Dervenis8 report a more detailed audit of 31 consecutive meta-analyses published in the Annals of Surgery from 2000 to 2011. It is encouraging to note that there has been an increase in the overall compliance with QUOROM. Furthermore, many aspects of the QUOROM checklist seem to have been reported including general information and measures of effect. However, items related to “quality assessment measures” showed variability and infrequent reporting (only one third of studies). In some cases, this might have been anticipated: one cannot expect to identify or include a large number of randomized trials (which are specifically targeted by the QUOROM statement) if only less than 10% of mainstream surgical works use a randomized design.9 It might have been more appropriate for some of these reviewed meta-analyses to use the MOOSE recommendations. Sotiropoulos and Dervenis also comment on the lack of use of a formal study quality “scoring system” such as the Jadad quality scale; however, in its most recent update, The Cochrane Collaboration discourages the use of such a scoring system,10 favoring the semiquantitative Cochrane Risk Bias tool.11 Nevertheless, as highlighted by the authors, the absence of a formal assessment of the risk of bias of a study and its subsequent inclusion in a summary analysis clearly open the door to potentially biased assessments and resulting erroneous meta-analytical conclusions. This speaks to the potential illusion that can be created by the quantitative part of a meta-analysis, as summarized by Ioannidis: “simply putting problematic data together does not overcome their (original studies) problems,”12 perhaps better known as “garbage in . . . garbage out (GIGO).” Finally, Sotiropoulos and Dervenis also raise the issue of missing data—either at the study level (eg, excluded patients, dif-
ferential outcome validity) or in the literature by ignoring “gray literature” and possibly perpetuating publication bias. Many of these stated weaknesses are not unique to surgical meta-analyses and have in fact spearheaded the development of a new reporting standard that is replacing QUOROM: the PRISMA statement.13 The PRISMA statement comprises a 4-phase flow diagram and a checklist expanded to 27 items; unlike QUOROM, it can be more broadly applied to systematic reviews and meta-analyses. Several aspects may be particularly attractive to surgery: PRISMA does not exclusively apply to meta-analyses of randomized trials, which remain the exception in the surgical literature. It can also be used to target reviews of the evaluation of interventions, which are a common topic in the surgical literature. PRISMA recommends to authors a PICOS (Population, Intervention, Comparator, Outcome, Study design) approach in the development of their clinical question that strengthens the way they will plan gathering information about the intervention and outcomes, traditional surgical study reporting weak points. Also, the evaluation of bias requires assessments not just at the study level but also at the outcome level. The latter would be particularly pertinent in surgical trials not only where the design may be strong (eg, randomization scheme, on-table randomization) but also where the recording of certain outcomes may be inconsistent (eg, ad hoc or not validated scheme used to record complications). PRISMA also suggests that reviewers search for the presence of a registered protocol (mostly for randomized trials) to avoid the potential for bias of selective outcome reporting or an opportunistic late change in what had been the originally planned primary outcome or timing of analysis.14,15 In summary, risk of bias in the reporting of evidence generated from metaanalyses represents an increasing challenge, especially in surgical meta-analyses and despite QUOROM; this challenge is amplified by the ever-increasing number of metaanalyses being submitted to journals. Annals of Surgery may help meet this challenge by strengthening the recommendations to authors and reviewers as pertains to the new PRISMA statement. Jeffrey S. Barkun, MD Department of Surgery McGill University/Royal Victoria Hospital
C 2013 Wolters Kluwer Health, Inc. All rights reserved.
Montreal, Quebec, Canada [email protected]
REFERENCES 1. Moher D, Schulz KF, Altman D, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285:1987–1991. 2. STROBE Statement. STrengthening the Reporting of Observational studies in Epidemiology. March 30, 2008. Available at: http://www .strobe-statement.org. Accessed September 25, 2013. 3. Stroup DF, Berlin JA, Morton SC, et al. Metaanalysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–2012. 4. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–1900. 5. Jin Wena, Yu Renb, Li Wang, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol. 2008;61: 770–775. 6. Knobloch K, Gohritz A, Vogt PM. CONSORT and QUOROM statements revisited: standards of reporting of randomized controlled trials in general surgery. Ann Surg. 2008;248:1106–1107. 7. Balasubramanian SP, Wiener M, Alshameeri Z, et al. Standards of reporting of randomized controlled trials in general surgery. Ann Surg. 2006;244:663–667. 8. Sotiropoulos GC, Dervenis C. Meta-analyses in surgical publications: how qualitative is this evidence-based information? Ann Surg. 2015;261:e129–e130. 9. Chang DC, Matsen SL, Simpkins CE. Why should surgeons care about clinical research methodology? J Am Coll Surg. 2006;203:827–830. 10. The Cochrane Collaboration. Chapter 8: Assessing risk of bias in included studies. In: Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. Available at: http://www.cochrane.org/sites/default/files/uploads/ handbook/Handbook510pdf_Ch08_RiskOfBias.pdf. Accessed September 27, 2013. 11. The Cochrane Collaboration. Assessment of study quality. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. Updated March 2011. 12. Ioannidis JP. Meta-analysis in public health: potentials and problems. Ital J Public Health. 2006;3:9–14. 13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6:e1000100. 14. Hannink G, Gooszen HG, Rovers MM. Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann Surg. 2013;257:818–823. 15. Singh D. Merck withdraws arthritis drug worldwide. BMJ. 2004;329:816.
www.annalsofsurgery.com | e131
Copyright © 2013 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
