Article Alerts Received 30 December 2009,
Accepted 28 January 2010
Published online in Wiley Interscience
(www.interscience.wiley.com) DOI: 10.1002/jrsm.7
Article alerts: Introduction and items from 2009, part I Adam R. Hafdahl∗ † This ‘Article Alerts’ feature is intended to apprise readers of recent methodological work in research synthesis and compile previous such contributions from various outlets. In this first installment we introduce the feature by commenting on its main aims and distinguishing between the print and archive versions. The feature’s content and process are also described, including encouragement of interactive contributions from readers. The current installment’s 100 items, a subset of relevant work published in 2009, are categorized by type of contribution and supplemented with suggested keywords. Copyright © 2010 John Wiley & Sons, Ltd. Keywords: bibliography; research synthesis; systematic review; meta-analysis; methodology; literature search
Welcome to the first installment of this feature section. Its primary purpose is to alert this journal’s readers to recent publications about the methodology of research synthesis, which are scattered among hundreds of outlets in numerous disciplines. A secondary purpose is to provide a fairly comprehensive compilation of relevant methodological contributions in articles, books, chapters, dissertations, conference papers, technical reports, and other outlets. We hope that this effort to consolidate a wide range of scholarly work from diverse sources proves valuable and stimulating to several audiences: research synthesists who apply these methods in a variety of substantive domains; statisticians and other methodologists who develop, evaluate, or disseminate new or existing techniques; researchers advancing substantive theory and applications; instructors of relevant courses, workshops, or other educational offerings; and policy makers, practitioners, patients, and other stakeholders in endeavors that involve, influence, or are influenced by research synthesis. Complementary print and archive versions will serve the above two purposes. The print version will mainly include relevant articles published in scholarly journals during the previous or current calendar year. In addition to a reference citation for each such item, taxonomic or content-descriptive information will be provided for some items, such as categories or keywords. This print version will appear in at least half of the journal’s issues each year and may occasionally include items that are older or from other types of outlets, such as when work on a special topic is highlighted. The archive version, which readers can access via the journal’s Web site (http://www3.interscience.wiley.com/journal/122342604/home), will be more comprehensive, including not only the print version’s items but also a regularly updated collection of both published and unpublished work related to research synthesis methodology. This archive version may include additional content-descriptive or taxonomic information. Content and process The current issue’s installment of the print version appears in the next section. First, however, some specific remarks about this feature’s content and process are warranted. Relevance and inclusion criteria. As anyone who has tried to identify, retrieve, or organize scholarly work on research synthesis methodology can attest to, operationally defining ‘relevance’ and establishing inclusion/exclusion criteria are complicated by the diverse contributions in this area. A salient feature of research synthesis that substantially broadens its scope of relevant methodology is its role as intermediary between authors of primary studies and consumers of research: Some pertinent activities fall squarely within the domain of research synthesis proper, such as identifying and retrieving reports of primary studies, appraising and describing (a subset of) these reports, and aggregating selected reports’ results, but many who use or study research synthesis are keenly interested in what comes before (e.g. aspects of primary studies’ design, conduct, reporting, and dissemination) or after (e.g. knowledge translation and implementation in evidence-based arenas). Although definitions and criteria may evolve as this feature matures, in the print version we plan to emphasize techniques and concepts pertaining most directly to research synthesis proper; the more comprehensive archive version will additionally include work on a variety of associated topics. Both components of this feature may cover a broader array of work than that represented
Copyright © 2010 John Wiley & Sons, Ltd.
81
Department of Mathematics, Washington University in St. Louis, MO, U.S.A. ∗ Correspondence to: Adam R. Hafdahl, ARCH Statistical Consulting, LLC, P.O. Box 2282, St. Louis, MO 63109, U.S.A. † E-mail: [email protected]
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL by the journal’s content. To be clear, inclusion of a given item in the print or archive version (or its exclusion therefrom) is not meant to reflect on the work’s quality, importance, utility to a particular audience, or any other characteristic besides its judged relevance. Search strategies. This feature is intended to be fairly exhaustive in its coverage of relevant work. Owing to the scope of this task and limitations on available resources, however, some portion of pertinent items will surely be missed—more so for the time- and space-limited print version than the more dynamic, cumulative, and comprehensive archive version. Several search strategies have been used to build the current feature editor’s collection of over 6000 items and will continue to play a role in updating this feature’s print and archive versions. These include searching bibliographic databases (e.g. [Social] Science[s] Citation Index, PsycINFO, Medline, ERIC, Dissertation Abstracts International, Current Index to Statistics), perusing reference lists and citation indexes based on relevant items, browsing journals that often publish relevant items, and seeking out fugitive literature (e.g. contacting authors, searching the Internet, perusing conference proceedings).‡ Although characteristics besides an item’s relevance may influence its inclusion via its accessibility, contributions from readers may limit such omissions. Arrangement of items. Organization of this feature’s content will also evolve. Early installments of the print version will most likely consist essentially of a list of items divided into relatively few convenient categories (e.g. by type of contribution or phase of systematic review), perhaps with keywords or annotations. As the feature matures, a more articulated organizational scheme may be used. The archive version will be more amenable to a refined descriptive or taxonomic approach to highlighting important features of the included works, such as by using a controlled vocabulary (e.g. hierarchical subject headings). Interactive contributions. We encourage readers to contribute to this feature in several ways. An easy and direct example is to notify the feature editor about items to consider for inclusion, ranging from individual items to sizeable collections (e.g. in personal bibliographies). Especially valuable would be fugitive literature and suggested items on topics that seem underrepresented in the print or archive versions; such lacunae may reflect the feature editor’s limited familiarity with particular areas of relevant work. Another way to contribute is to provide constructive feedback about the feature’s content or arrangement, such as proposing topics of interest, recommending fruitful search strategies (e.g. effective search terms, pertinent journals, or professional meetings), or sharing ideas about how to organize the print or archive versions or supplement their entries with useful information. We also welcome suggestions for special topics to highlight in the print version as a focused collection of both current and older items; examples might include work on a novel or otherwise noteworthy methodological problem, an emerging domain of application, or methods that share some common principle or theme (e.g. synthesis of qualitative research, measurement issues, Bayesian approaches).
Items for current installment Each of the 100 items listed below was published in a scholarly journal during 2009, though an online version may have appeared earlier. Not all relevant publications from 2009 are included: Well over 100 more will appear in future installments of this feature section. The rather broad categories reflect the work’s primary type of contribution, which was difficult to classify for some items. Suggested keywords are listed in brackets after each item’s reference citation; those following a semicolon represent less central aspects of the work. Consecutive numbering of items will continue across subsequent issues and volumes. Proposal of novel or refined method: Primarily non-statistical Authors of the items in this first category proposed a novel procedure or strategy for a largely non-statistical aspect of research synthesis or a closely related task (e.g. reporting primary studies, appraising reviews). In cases where the proposed method encompasses statistical methods, the contribution as a whole extends substantially beyond statistical considerations. Many of these items include worked examples or real-data illustrations. 1. Dieste O, Grimán A, Juristo N. Developing search strategies for detecting relevant experiments. Empirical Software Engineering 2009; 14:513–539 [literature search, method comparison; primary-study design]. 2. Hawkins N, Scott DA, Woods B. How far do you go? Efficient searching for indirect evidence. Medical Decision Making 2009; 29:273–281 [network meta-analysis, literature search]. 3. Howard GS, Lau MY, Maxwell SE, Venter A, Lundy R, Sweeny RM. Do research literatures give correct answers? Review of General Psychology 2009; 13:116–121 [publication bias, significance testing, replication]. 4. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine 2009; 6(7) [guidance for reporting, overview of research synthesis]. 5. Little J, Higgins JPT, Ioannidis, JPA, Moher D, Gagnon F, von Elm E et al. Strengthening the reporting of genetic association studies (STREGA): an extension of the STROBE statement. European Journal of Epidemiology 2009; 24:37–55 [genetic association, guidance for reporting; primary-study design].
‡ Other
82
bibliographic compilations with substantial content on research synthesis methodology include those maintained by the Cochrane Collaboration (http://cmr.cochrane.org), the Oregon Health and Science University (http://www.citeulike.org/user/SRCMethodsLibrary), and the Meta-analysis Unit at the University of Murcia (http://www.um.es/metaanalysis/search.php). Readers are encouraged to consult those resources, whose contents are not currently searched for this feature section but may be incorporated in the future.
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL 6. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 2009; 6(7) [guidance for reporting; review quality]. 7. Panesar SS, Rao C, Vecht JA, Mirza SB, Netuveli G, Morris R et al. Development of the Veritas plot and its application in cardiac surgery: an evidence-synthesis graphic tool for the clinician to assess multiple meta-analyses reporting on a common outcome. Canadian Journal of Surgery 2009; 52:E137–E145 [review quality, primary-study design, appraising reviews, graphics; evidence-based medicine, heterogeneity, publication bias]. 8. Ryan RE, Kaufman CA, Hill SJ. Building blocks for meta-synthesis: Data integration tables for summarising, mapping, and synthesising evidence on interventions for communicating with health consumers. BMC Medical Research Methodology 2009; 9(16) [review of reviews, dissemination, implications for individuals, knowledge transfer, guidance for reviewing; Cochrane Collaboration, policy implications, review quality]. 9. Sutton AJ, Cooper NJ, Jones DR. Evidence synthesis as the key to more coherent and efficient research. BMC Medical Research Methodology 2009; 9(29) [informing primary research, power, heterogeneity, cumulative meta-analysis; individual patient data, network meta-analysis, generalized synthesis of evidence]. 10. Wells K, Littell JH. Study quality assessment in systematic reviews of research on intervention effects. Research on Social Work Practice 2009; 19:52–62 [primary-study quality, threat to validity, inclusion/exclusion, primary-study design; publication bias, moderator].
Proposal of novel or refined method: Statistical In contrast to methods proposed by the previous category’s authors, those below primarily involve statistical techniques, and their authors often included evaluations of the proposed method and competing approaches—often by Monte Carlo simulation. Much of this work pertains to the data-analysis phase of a quantitative review (i.e. meta-analysis), but some authors addressed broader issues as well (e.g. data repositories, updating reviews).
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
83
11. Bonett DG. Meta-analytic interval estimation for standardized and unstandardized mean differences. Psychological Methods 2009; 14:225–238 [interval estimation, standardized mean difference, random effects, heterogeneity, moderator, Monte Carlo simulation]. 12. Cheung MW-L, Chan W. A two-stage approach to synthesizing covariance matrices in meta-analytic structural equation modeling. Structural Equation Modeling 2009; 16:28–53 [explanatory model, multivariate technique, correlation, Monte Carlo simulation; heterogeneity, software]. 13. Chu H, Nie L, Cole SR, Poole C. Meta-analysis of diagnostic accuracy studies accounting for disease prevalence: alternative parameterizations and model selection. Statistics in Medicine 2009; 28:2384–2399 [diagnostic test accuracy, multivariate technique, random effects, Monte Carlo simulation; heterogeneity]. 14. Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ. Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Statistics in Medicine 2009; 28:1861–1881 [network meta-analysis, heterogeneity, moderator, Bayesian approach]. 15. Costafreda SG, David AS, Brammer MJ. A parametric approach to voxel-based meta-analysis. Neuroimage 2009; 46:115–122 [neuroimaging, heterogeneity, random effects, Monte Carlo simulation; power]. 16. Fibrinogen Studies Collaboration. Correcting for multivariate measurement error by regression calibration in meta-analyses of epidemiological studies. Statistics in Medicine 2009; 28:1067–1092 [individual patient data, longitudinal data, heterogeneity, survival analysis, multivariate technique, random effects; threat to validity, multi-site study]. 17. Hafdahl AR. Improved Fisher z estimators for univariate random-effects meta-analysis of correlations. British Journal of Mathematical and Statistical Psychology 2009; 62:233–261 [random effects, correlation; interval estimation, method comparison]. 18. Hansen RA, Moore CG, Dusetzina SB, Leinwand BI, Gartlehner G, Gaynes BN. Controlling for drug dose in systematic review and meta-analysis: a case study of the effect of antidepressant dose. Medical Decision Making 2009; 29:91–103 [heterogeneity, moderator, coding; dose-response data]. 19. Lai Y, Eckenrode SE, She J-X. A statistical framework for integrating two microarray data sets in differential expression analysis. BMC Bioinformatics 2009; 10(Suppl. 1):S23 [microarray, heterogeneity, Monte Carlo simulation; threat to validity, assumption violation]. 20. Lajeunesse MJ. Meta-analysis and the comparative phylogenetic method. American Naturalist 2009; 174:369–381 [dependence, moderator, heterogeneity; missing data]. 21. Moeltner K, Woodward R. Meta-functional benefit transfer for wetland valuation: making the most of small samples. Environmental and Resource Economics 2009; 42:89–108 [benefit transfer, Bayesian approach, sample size; assumption violation]. 22. Ochsner SA, Steffen DL, Hilsenbeck SG, Chen ES, Watkins C, McKenna NJ. GEMS (Gene Expression Metasignatures), a web resource for querying meta-analysis of expression microarray datasets: 17-estradiol in MCF-7 cells. Cancer Research 2009; 69:23–26 [microarray; data repository, combining p values]. 23. Salanti G, Ioannidis JPA. Synthesis of observational studies should consider credibility ceilings. Journal of Clinical Epidemiology 2009; 62:115–122 [primary-study design, sensitivity analysis; threat to validity, random effects]. 24. Sutton AJ, Donegan S, Takwoingi Y, Garner P, Gamble C, Donald A. An encouraging assessment of methods to inform priorities for updating systematic reviews. Journal of Clinical Epidemiology 2009; 62:241–251 [updating reviews, random effects, method comparison; Cochrane Collaboration, binary outcome].
A. R. HAFDAHL 25. Tian L, Cai T, Pfeffer MA, Piankov N, Cremieux P-Y, Wei LJ. Exact and efficient inference procedure for meta-analysis and its application to the analysis of independent 2 × 2 tables with all available data but without artificial continuity correction. Biostatistics 2009; 10:275–281 [binary outcome, interval estimation, Monte Carlo simulation]. 26. Turner RM, Spiegelhalter DJ, Smith GCS, Thompson SG. Bias modelling in evidence synthesis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:21–47 [primary-study quality, threat to validity, generalized synthesis of evidence, policy implications, heterogeneity, sensitivity analysis]. 27. Welton NJ, Ades AE, Carlin JB, Altman DG, Sterne JAC. Models for potentially biased evidence in meta-analysis using empirically based priors. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:119–136 [primary-study quality, Bayesian approach, heterogeneity, discordant reviews, random effects, sensitivity analysis, moderator]. 28. Welton NJ, Caldwell DM, Adamopoulos E, Vedhara K. Mixed treatment comparison meta-analysis of complex interventions: Psychological interventions in coronary heart disease. American Journal of Epidemiology 2009; 169:1158–1165 [network meta-analysis, heterogeneity, random effects, moderator, Bayesian approach; binary outcome, standardized mean difference]. 29. Won S, Morris N, Lu Q, Elston RC. Choosing an optimal method to combine P-values. Statistics in Medicine 2009; 28: 1537–1553 [combining p values, power, Monte Carlo simulation; genetic association].
Broad review of synthesis methods Authors of items in this category described research synthesis methods in a fairly broad sense, often with particular attention to a specific research domain or type of data (e.g. from studies of diagnostic test accuracy). They typically covered most major phases of review and often addressed the larger context in which syntheses are undertaken (e.g. evidence-based practice). A few authors focused specifically on meta-analysis or on qualitative methods, and some included real-data illustrations.
84
30. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology 2009; 9(59) [overview of research synthesis, qualitative research]. 31. Bath PMW, Gray LJ. Systematic reviews as a tool for planning and interpreting trials. International Journal of Stroke 2009; 4: 23–27 [overview of research synthesis, informing primary research; cumulative meta-analysis, individual patient data, moderator, categorical data]. 32. Charrois TL, Durec T, Tsuyuki RT. Systematic reviews of pharmacy practice research: methodologic issues in searching, evaluating, interpreting, and disseminating results. Annals of Pharmacotherapy 2009; 43:118–122 [overview of research synthesis, evidence-based practice, dissemination]. 33. Chatzisarantis NLD, Stoica A. A primer on the understanding of meta-analysis. Psychology of Sport and Exercise 2009; 10:498–501 [overview of meta-analysis, validity generalization, heterogeneity, moderator]. 34. Coleman CI, Talati R, White CM. A clinician’s perspective on rating the strength of evidence in a systematic review. Pharmacotherapy 2009; 29:1017–1029 [evidence-based practice, strength of evidence, appraising reviews; publication bias, implications for individuals]. 35. Hamada C. The role of meta-analysis in cancer clinical trials. International Journal of Clinical Oncology 2009; 14:90–94 [overview of meta-analysis, sample size, heterogeneity, moderator, large study; evidence-based medicine, individual patient data, publication bias]. 36. Hawke F, Burns J, Landorf KB. Evidence-based podiatric medicine: importance of systematic reviews in clinical practice. Journal of the American Podiatric Medical Association 2009; 99:260–266 [overview of research synthesis, evidence-based practice, Cochrane Collaboration; knowledge transfer, primary-study quality]. 37. Johnson NP, Khan KS. Gynaecologists blaze the trail in primary studies and systematic reviews of diagnostic test accuracy. Australian and New Zealand Journal of Obstetrics and Gynaecology 2009; 49:71–76 [diagnostic test accuracy, primary-study quality, overview of research synthesis, Cochrane Collaboration]. 38. Manchikanti L, Benyamin R, Helm SII, Hirsch JA. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 3: systematic reviews and meta-analyses of randomized trials. Pain Physician 2009; 12:35–72 [overview of research synthesis, review quality, appraising reviews, threat to validity, strength of evidence, guidance for reporting]. 39. Manchikanti L, Datta S, Smith HS, Hirsch JA. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 6. Systematic reviews and meta-analyses of observational studies. Pain Physician 2009; 12:819–850 [overview of research synthesis, primary-study design, primary-study quality, threat to validity, guidance for reviewing, guidance for reporting; evidence-based practice, implications for individuals, review quality]. 40. Perera R, Heneghan C. Interpreting meta-analyses in systematic reviews. Annals of Internal Medicine 2009; 150:JC2-2–JC2-3 [overview of meta-analysis, graphics, heterogeneity]. 41. Ressing M, Blettner M, Klug SJ. Systematic literature reviews and meta-analyses: part 6 of a series on evaluation of scientific publications. Deutsches Ärzteblatt International 2009; 106:456–463 [overview of research synthesis; individual patient data]. 42. Sagoo GS, Little J, Higgins JPT. Systematic reviews of genetic association studies. PLoS Medicine 2009; 6(3) [genetic association, overview of research synthesis; threat to validity, guidance for reporting]. 43. Simunovic N, Sprague S, Bhandari M. Methodological issues in systematic reviews and meta-analyses of observational studies in orthopaedic research. Journal of Bone and Joint Surgery: American Volume 2009; 91(Suppl. 3):87–94 [overview of research synthesis, primary-study design, threat to validity].
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL 44. Suri H, Clarke D. Advancements in research synthesis methods: from a methodologically inclusive perspective. Review of Educational Research 2009; 79:395–430 [overview of research synthesis, qualitative research]. 45. Thorne S. The role of qualitative research within an evidence-based context: can metasynthesis be the answer? International Journal of Nursing Studies 2009; 46:569–575 [qualitative research; evidence-based practice]. 46. Turkelson C, Jacobs JJ. Role of technology assessment in orthopaedics. Clinical Orthopaedics and Related Research 2009; 467:2570–2576 [overview of research synthesis, technology assessment]. 47. Virgili G, Conti AA, Murro V, Gensini GF, Gusinu R. Systematic reviews of diagnostic test accuracy and the Cochrane collaboration. Internal and Emergency Medicine 2009; 4:255–258 [diagnostic test accuracy, Cochrane Collaboration, primarystudy quality]. 48. Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. American Journal of Gastroenterology 2009; 104: 1086–1092 [overview of research synthesis, appraising reviews, threat to validity, primary-study quality, heterogeneity, implications for individuals]
Exposition of specific method or issue In contrast to items in the previous category, those below tended to focus on a fairly specific existing technique or issue (e.g. choice of effect-size index), often including more detail than a broader review. Although many of these authors offered recommendations on the basis of their review of relevant methodology, their emphasis was not on introducing new methods.
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
85
49. Baguley T. Standardized or simple effect size: what should be reported? British Journal of Psychology 2009; 100:603–617 [effect size; standardized mean difference, correlation, primary-study design]. 50. Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI. Understanding heterogeneity in meta-analysis: the role of metaregression. International Journal of Clinical Practice 2009; 63:1426–1434 [heterogeneity, moderator; graphics, threat to validity]. 51. Chappell FM, Raab GM, Wardlaw JM. When are summary ROC curves appropriate for diagnostic meta-analyses? Statistics in Medicine 2009; 28:2653–2668 [diagnostic test accuracy, heterogeneity, random effects, multivariate technique, method comparison, guidance for reviewing; sample size, software]. 52. Cummings P. The relative merits of risk ratios and odds ratios. Archives of Pediatrics and Adolescent Medicine 2009; 163:438–445 [effect size, binary outcome; heterogeneity, moderator]. 53. Edwards SJ, Clarke MJ, Wordsworth S, Borrill J. Indirect comparisons of treatments based on systematic reviews of randomised controlled trials. International Journal of Clinical Practice 2009; 63:841–854 [network meta-analysis, assessment of review practice, method comparison]. 54. Elamin MB, Flynn DN, Bassler D, Briel M, Alonso-Coello P, Karanicolas PJ et al. Choice of data extraction tools for systematic reviews depends on resources and review complexity. Journal of Clinical Epidemiology 2009; 62:506–510 [data management, coding, software]. 55. Feingold A. Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis. Psychological Methods 2009; 14:43–53 [effect size, standardized mean difference, primary-study design]. 56. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:137–159 [heterogeneity, random effects, Bayesian approach, interval estimation, primary-study quality, threat to validity; sensitivity analysis, moderator, software]. 57. Howard GS, Hill TL, Maxwell SE, Baptista TM, Farias MH, Coelho C et al. What’s wrong with research literatures? And how to make them right. Review of General Psychology 2009; 13:146–166 [publication bias, replication, significance testing, Bayesian approach; interval estimation, registry of trials]. 58. Hutton JL. Number needed to treat and number needed to harm are not the best way to report and assess the results of randomised clinical trials. British Journal of Haematology 2009; 146:27–30 [number needed to treat, binary outcome, practical significance]. 59. Keus F, Wetterslev J, Gluud C, Gooszen HG, van Laarhoven CJHM. Robustness assessments are needed to reduce bias in meta-analyses that include zero-event randomized trials. American Journal of Gastroenterology 2009; 104:546–551 [effect size, binary outcome, method comparison]. 60. Pignon J-P, Auperin A, Borget I, Hill C. Role of meta-analyses and of large randomized trials in the study of cancer treatments. Lung Cancer 2009; 65:9–12 [large study, moderator, informing primary research; individual patient data]. 61. Rosenberger RS, Johnston RJ. Selection effects in meta-analysis and benefit transfer: avoiding unintended consequences. Land Economics 2009; 85:410–428 [economic valuation, threat to validity, moderator; policy implications]. 62. Rücker G, Schwarzer G, Carpenter J, Olkin I. Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Statistics in Medicine 2009; 28:721–738 [binary outcome, effect size, random effects, method comparison, Monte Carlo simulation; heterogeneity]. 63. Schmidt FL, Oh I-S, Hayes TL. Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology 2009; 62:97–128 [random effects, heterogeneity; correlation, standardized mean difference]. 64. Scifres CM, Iams JD, Klebanoff M, Macones GA. Metaanalysis vs. large clinical trials: which should guide our management? American Journal of Obstetrics and Gynecology 2009; 200:484.e1–484.e5 [large study, power, individual patient data, heterogeneity; publication bias, significance testing, primary-study quality].
A. R. HAFDAHL 65. Shi Q Sargent DJ. Meta-analysis for the evaluation of surrogate endpoints in cancer clinical trials. International Journal of Clinical Oncology 2009; 14:102–111 [surrogate endpoint, primary-study design, individual patient data, heterogeneity]. 66. Wager TD, Lindquist MA, Nichols TE, Kober H, Van Snellenberg JX. Evaluating the consistency and specificity of neuroimaging data using meta-analysis. Neuroimage 2009; 45:S210–S221 [neuroimaging, significance testing, graphics, resampling; categorical data, power, heterogeneity, publication bias].
Evaluation of method A major contribution of the following items is the evaluation of one or more methods for research synthesis or a related task. This often entailed analytic critique, Monte Carlo simulation, or application to one or more real-world data sets. Some of these authors also proposed a novel or refined method, but not as the main focus. 67. Bax L, Ikeda N, Fukui N, Yaju Y, Tsuruta H, Moons KGM. More than numbers: the power of graphs in meta-analysis. American Journal of Epidemiology 2009; 169:249–255 [graphics, heterogeneity, publication bias, method comparison]. 68. Borm GF, den Heijer M, Zielhuis GA. Publication bias was not a good reason to discourage trials with low power. Journal of Clinical Epidemiology 2009; 62:47–53 [publication bias, power, Monte Carlo simulation; inclusion/exclusion]. 69. Borm GF, Donders ART. Updating meta-analyses leads to larger type I errors than publication bias. Journal of Clinical Epidemiology 2009; 62:825–830 [updating reviews, cumulative meta-analysis, publication bias, Monte Carlo simulation; heterogeneity]. 70. Hoogendam A, Robbé PFD, Stalenhoef AFH, Overbeke JPM. Evaluation of PubMed filters used for evidence-based searching: validation using relative recall. Journal of the Medical Library Association 2009; 97:186–193 [literature search, evidence-based medicine, Cochrane Collaboration]. 71. Jackson D, Bowden J. A re-evaluation of the ‘quantile approximation method’ for random effects meta-analysis. Statistics in Medicine 2009; 28:338–348 [random effects, Monte Carlo simulation; assumption violation]. 72. Moreno SG, Sutton AJ, Ades AE, Stanley TD, Abrams KR, Peters JL et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology 2009; 9(2) [publication bias, heterogeneity, random effects, Monte Carlo simulation; decision making]. 73. Nugent WR. Construct validity invariance and discrepancies in meta-analytic effect sizes based on different measures: a simulation study. Educational and Psychological Measurement 2009; 69:62–78 [effect size, assumption violation, threat to validity, standardized mean difference, correlation, Monte Carlo simulation]. 74. Nugent WR. Meta-analysis as a research synthesis methodology: Cause for concern. Journal of Social Service Research 2009; 35:181–192 [effect size, standardized mean difference, threat to validity; heterogeneity]. 75. O’Regan C, Ghement I, Eyawo O, Guyatt GH, Mills EJ. Incorporating multiple interventions in meta-analysis: an evaluation of the mixed treatment comparison with the adjusted indirect comparison. Trials 2009; 10(86) [network meta-analysis, Bayesian approach, method comparison]. 76. Riley RD. Multivariate meta-analysis: the effect of ignoring within-study correlation. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:789–811 [multivariate technique, random effects, missing data, Monte Carlo simulation, method comparison; individual patient data, Bayesian approach]. 77. Romano JL, Kromrey JD. What are the consequences if the assumption of independent observations is violated in reliability generalization meta-analysis studies? Educational and Psychological Measurement 2009; 69:404–428 [reliability generalization, dependence, Monte Carlo simulation, interval estimation]. 78. Salanti G, Marinho V, Higgins JPT. A case study of multiple-treatments meta-analysis demonstrates that covariates should be considered. Journal of Clinical Epidemiology 2009; 62:857–864 [network meta-analysis, multivariate technique, moderator; assumption violation, Bayesian approach]. 79. Salimi-Khorshidi G, Smith SM, Keltner JR, Wager TD, Nichols TE. Meta-analysis of neuroimaging data: a comparison of imagebased and coordinate-based pooling of studies. Neuroimage 2009; 45:810–823 [neuroimaging, random effects, missing data, multilevel model, method comparison; power, combining p values]. 80. Thorlund K, Devereaux PJ, Wetterslev J, Guyatt G, Ioannidis JPA, Thabane L et al. Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses? International Journal of Epidemiology 2009; 38:276–286 [cumulative metaanalysis, sample size, power, sequential testing; heterogeneity, binary outcome]. 81. van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. Journal of Clinical Epidemiology 2009; 62:838–844 [literature search, Cochrane Collaboration, grey literature, primary-study quality; publication bias, registry of trials, sponsorship bias].
Evaluation of substantive application(s) Items in this category primarily involved evaluating one or more reported applications of research synthesis or related methods, often with an emphasis on the quality of implementation (rather than the methods’ inherent properties). Some authors focused on specific phases of review or particular threats to validity.
86
82. Al Faleh K, Al-Omran M. Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews. BMC Pediatrics 2009; 9(38) [review quality, assessment of review practice; guidance for reporting]. Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL 83. Brok J, Thorlund K, Wetterslev J, Gluud C. Apparently conclusive meta-analyses may be inconclusive—trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal metaanalyses. International Journal of Epidemiology 2009; 38:287–298 [cumulative meta-analysis, sample size, power, sequential testing, assessment of review practice; heterogeneity, Cochrane Collaboration]. 84. Dieckmann NF, Malle BF, Bodner TE. An empirical assessment of meta-analytic practice. Review of General Psychology 2009; 13:101–115 [review quality, assessment of review practice, dissemination, threat to validity; guidance for reporting, overview of research synthesis, sensitivity analysis]. 85. Diener MK, Wolff RF, von Elm E, Rahbari NN, Mavergames C, Knaebel H-P et al. Can decision making in general surgery be based on evidence? An empirical study of Cochrane Reviews. Surgery 2009; 146:444–461 [guidance for practice, strength of evidence, Cochrane Collaboration]. 86. Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results—guidance for future prognosis reviews. Journal of Clinical Epidemiology 2009; 62:781–796 [assessment of review practice, review quality, guidance for reviewing, prognostic studies]. 87. Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. Journal of the American Medical Association 2009; 302:977–984 [registry of trials, reporting bias; publication bias]. 88. Mokkink LB, Terwee CB, Stratford PW, Alonso J, Patrick DL, Riphagen I et al. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research 2009; 18:313–333 [review quality, assessment of review practice, primary-study quality, dissemination]. 89. Mullan RJ, Flynn DN, Carlberg B, Tleyjeh IM, Kamath C, LaBella ML et al. Systematic reviewers commonly contact study authors but do so with limited rigor. Journal of Clinical Epidemiology 2009; 62:138–142 [literature search; missing data]. 90. Rosén M. The aprotinin saga and the risks of conducting meta-analyses on small randomised controlled trials—a critique of a Cochrane review. BMC Health Services Research 2009; 9(34) [assessment of review practice, primary-study quality, primary-study design]. 91. Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. Journal of Clinical Epidemiology 2009; 62:380–386 [publication bias, Cochrane Collaboration, review quality; grey literature]. 92. Vavken P, Dorotka R. A systematic review of conflicting meta-analyses in orthopaedic surgery. Clinical Orthopaedics and Related Research 2009; 467:2723–2735 [discordant reviews, assessment of review practice, review quality; primary-study quality, updating reviews]. 93. Wells E. Uses of meta-analysis in criminal justice research: a quantitative review. Justice Quarterly 2009; 26:268–294 [assessment of review practice, Campbell Collaboration; narrative review, significance testing, dependence]. 94. Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association 2009; 97:21–29 [literature search, guidance for reporting, Cochrane Collaboration, assessment of review practice]. Other type of contribution Each of the following items makes a unique type of methodological contribution relevant to research synthesis that is not readily classified into one of the preceding categories. 95. Cafri G, Kromrey JD, Brannick MT. A SAS macro for statistical power calculations in meta-analysis. Behavior Research Methods 2009; 41:35–46 [power, software, heterogeneity, random effects, moderator]. 96. Clarke M. Can you believe what you read in the papers? Trials 2009; 10(55) [threat to validity, primary-study quality]. 97. Kontopantelis E, Reeves D. MetaEasy: a meta-analysis add-in for Microsoft Excel. Journal of Statistical Software 2009; 30(7) [software; effect size, graphics, heterogeneity, random effects, Bayesian approach]. 98. Moher D. Guidelines for reporting health care research: advancing the clarity and transparency of scientific reporting. Journal Canadien d’Anesthesie 2009; 56:96–101 [dissemination, guidance for reporting; primary-study quality]. 99. Riley RD, Sauerbrei W, Altman DG. Prognostic markers in cancer: the evolution of evidence from single studies to metaanalysis, and beyond. British Journal of Cancer 2009; 100:1219–1229 [prognostic studies, primary-study quality; individual patient data, multi-site study]. 100. Senn SJ. Overstating the evidence—double counting in meta-analysis and related problems. BMC Medical Research Methodology 2009; 9(10) [review quality, threat to validity, assessment of review practice, dependence, missing data; guidance for reporting].
87
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
Accepted 28 January 2010
Published online in Wiley Interscience
(www.interscience.wiley.com) DOI: 10.1002/jrsm.7
Article alerts: Introduction and items from 2009, part I Adam R. Hafdahl∗ † This ‘Article Alerts’ feature is intended to apprise readers of recent methodological work in research synthesis and compile previous such contributions from various outlets. In this first installment we introduce the feature by commenting on its main aims and distinguishing between the print and archive versions. The feature’s content and process are also described, including encouragement of interactive contributions from readers. The current installment’s 100 items, a subset of relevant work published in 2009, are categorized by type of contribution and supplemented with suggested keywords. Copyright © 2010 John Wiley & Sons, Ltd. Keywords: bibliography; research synthesis; systematic review; meta-analysis; methodology; literature search
Welcome to the first installment of this feature section. Its primary purpose is to alert this journal’s readers to recent publications about the methodology of research synthesis, which are scattered among hundreds of outlets in numerous disciplines. A secondary purpose is to provide a fairly comprehensive compilation of relevant methodological contributions in articles, books, chapters, dissertations, conference papers, technical reports, and other outlets. We hope that this effort to consolidate a wide range of scholarly work from diverse sources proves valuable and stimulating to several audiences: research synthesists who apply these methods in a variety of substantive domains; statisticians and other methodologists who develop, evaluate, or disseminate new or existing techniques; researchers advancing substantive theory and applications; instructors of relevant courses, workshops, or other educational offerings; and policy makers, practitioners, patients, and other stakeholders in endeavors that involve, influence, or are influenced by research synthesis. Complementary print and archive versions will serve the above two purposes. The print version will mainly include relevant articles published in scholarly journals during the previous or current calendar year. In addition to a reference citation for each such item, taxonomic or content-descriptive information will be provided for some items, such as categories or keywords. This print version will appear in at least half of the journal’s issues each year and may occasionally include items that are older or from other types of outlets, such as when work on a special topic is highlighted. The archive version, which readers can access via the journal’s Web site (http://www3.interscience.wiley.com/journal/122342604/home), will be more comprehensive, including not only the print version’s items but also a regularly updated collection of both published and unpublished work related to research synthesis methodology. This archive version may include additional content-descriptive or taxonomic information. Content and process The current issue’s installment of the print version appears in the next section. First, however, some specific remarks about this feature’s content and process are warranted. Relevance and inclusion criteria. As anyone who has tried to identify, retrieve, or organize scholarly work on research synthesis methodology can attest to, operationally defining ‘relevance’ and establishing inclusion/exclusion criteria are complicated by the diverse contributions in this area. A salient feature of research synthesis that substantially broadens its scope of relevant methodology is its role as intermediary between authors of primary studies and consumers of research: Some pertinent activities fall squarely within the domain of research synthesis proper, such as identifying and retrieving reports of primary studies, appraising and describing (a subset of) these reports, and aggregating selected reports’ results, but many who use or study research synthesis are keenly interested in what comes before (e.g. aspects of primary studies’ design, conduct, reporting, and dissemination) or after (e.g. knowledge translation and implementation in evidence-based arenas). Although definitions and criteria may evolve as this feature matures, in the print version we plan to emphasize techniques and concepts pertaining most directly to research synthesis proper; the more comprehensive archive version will additionally include work on a variety of associated topics. Both components of this feature may cover a broader array of work than that represented
Copyright © 2010 John Wiley & Sons, Ltd.
81
Department of Mathematics, Washington University in St. Louis, MO, U.S.A. ∗ Correspondence to: Adam R. Hafdahl, ARCH Statistical Consulting, LLC, P.O. Box 2282, St. Louis, MO 63109, U.S.A. † E-mail: [email protected]
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL by the journal’s content. To be clear, inclusion of a given item in the print or archive version (or its exclusion therefrom) is not meant to reflect on the work’s quality, importance, utility to a particular audience, or any other characteristic besides its judged relevance. Search strategies. This feature is intended to be fairly exhaustive in its coverage of relevant work. Owing to the scope of this task and limitations on available resources, however, some portion of pertinent items will surely be missed—more so for the time- and space-limited print version than the more dynamic, cumulative, and comprehensive archive version. Several search strategies have been used to build the current feature editor’s collection of over 6000 items and will continue to play a role in updating this feature’s print and archive versions. These include searching bibliographic databases (e.g. [Social] Science[s] Citation Index, PsycINFO, Medline, ERIC, Dissertation Abstracts International, Current Index to Statistics), perusing reference lists and citation indexes based on relevant items, browsing journals that often publish relevant items, and seeking out fugitive literature (e.g. contacting authors, searching the Internet, perusing conference proceedings).‡ Although characteristics besides an item’s relevance may influence its inclusion via its accessibility, contributions from readers may limit such omissions. Arrangement of items. Organization of this feature’s content will also evolve. Early installments of the print version will most likely consist essentially of a list of items divided into relatively few convenient categories (e.g. by type of contribution or phase of systematic review), perhaps with keywords or annotations. As the feature matures, a more articulated organizational scheme may be used. The archive version will be more amenable to a refined descriptive or taxonomic approach to highlighting important features of the included works, such as by using a controlled vocabulary (e.g. hierarchical subject headings). Interactive contributions. We encourage readers to contribute to this feature in several ways. An easy and direct example is to notify the feature editor about items to consider for inclusion, ranging from individual items to sizeable collections (e.g. in personal bibliographies). Especially valuable would be fugitive literature and suggested items on topics that seem underrepresented in the print or archive versions; such lacunae may reflect the feature editor’s limited familiarity with particular areas of relevant work. Another way to contribute is to provide constructive feedback about the feature’s content or arrangement, such as proposing topics of interest, recommending fruitful search strategies (e.g. effective search terms, pertinent journals, or professional meetings), or sharing ideas about how to organize the print or archive versions or supplement their entries with useful information. We also welcome suggestions for special topics to highlight in the print version as a focused collection of both current and older items; examples might include work on a novel or otherwise noteworthy methodological problem, an emerging domain of application, or methods that share some common principle or theme (e.g. synthesis of qualitative research, measurement issues, Bayesian approaches).
Items for current installment Each of the 100 items listed below was published in a scholarly journal during 2009, though an online version may have appeared earlier. Not all relevant publications from 2009 are included: Well over 100 more will appear in future installments of this feature section. The rather broad categories reflect the work’s primary type of contribution, which was difficult to classify for some items. Suggested keywords are listed in brackets after each item’s reference citation; those following a semicolon represent less central aspects of the work. Consecutive numbering of items will continue across subsequent issues and volumes. Proposal of novel or refined method: Primarily non-statistical Authors of the items in this first category proposed a novel procedure or strategy for a largely non-statistical aspect of research synthesis or a closely related task (e.g. reporting primary studies, appraising reviews). In cases where the proposed method encompasses statistical methods, the contribution as a whole extends substantially beyond statistical considerations. Many of these items include worked examples or real-data illustrations. 1. Dieste O, Grimán A, Juristo N. Developing search strategies for detecting relevant experiments. Empirical Software Engineering 2009; 14:513–539 [literature search, method comparison; primary-study design]. 2. Hawkins N, Scott DA, Woods B. How far do you go? Efficient searching for indirect evidence. Medical Decision Making 2009; 29:273–281 [network meta-analysis, literature search]. 3. Howard GS, Lau MY, Maxwell SE, Venter A, Lundy R, Sweeny RM. Do research literatures give correct answers? Review of General Psychology 2009; 13:116–121 [publication bias, significance testing, replication]. 4. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine 2009; 6(7) [guidance for reporting, overview of research synthesis]. 5. Little J, Higgins JPT, Ioannidis, JPA, Moher D, Gagnon F, von Elm E et al. Strengthening the reporting of genetic association studies (STREGA): an extension of the STROBE statement. European Journal of Epidemiology 2009; 24:37–55 [genetic association, guidance for reporting; primary-study design].
‡ Other
82
bibliographic compilations with substantial content on research synthesis methodology include those maintained by the Cochrane Collaboration (http://cmr.cochrane.org), the Oregon Health and Science University (http://www.citeulike.org/user/SRCMethodsLibrary), and the Meta-analysis Unit at the University of Murcia (http://www.um.es/metaanalysis/search.php). Readers are encouraged to consult those resources, whose contents are not currently searched for this feature section but may be incorporated in the future.
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL 6. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 2009; 6(7) [guidance for reporting; review quality]. 7. Panesar SS, Rao C, Vecht JA, Mirza SB, Netuveli G, Morris R et al. Development of the Veritas plot and its application in cardiac surgery: an evidence-synthesis graphic tool for the clinician to assess multiple meta-analyses reporting on a common outcome. Canadian Journal of Surgery 2009; 52:E137–E145 [review quality, primary-study design, appraising reviews, graphics; evidence-based medicine, heterogeneity, publication bias]. 8. Ryan RE, Kaufman CA, Hill SJ. Building blocks for meta-synthesis: Data integration tables for summarising, mapping, and synthesising evidence on interventions for communicating with health consumers. BMC Medical Research Methodology 2009; 9(16) [review of reviews, dissemination, implications for individuals, knowledge transfer, guidance for reviewing; Cochrane Collaboration, policy implications, review quality]. 9. Sutton AJ, Cooper NJ, Jones DR. Evidence synthesis as the key to more coherent and efficient research. BMC Medical Research Methodology 2009; 9(29) [informing primary research, power, heterogeneity, cumulative meta-analysis; individual patient data, network meta-analysis, generalized synthesis of evidence]. 10. Wells K, Littell JH. Study quality assessment in systematic reviews of research on intervention effects. Research on Social Work Practice 2009; 19:52–62 [primary-study quality, threat to validity, inclusion/exclusion, primary-study design; publication bias, moderator].
Proposal of novel or refined method: Statistical In contrast to methods proposed by the previous category’s authors, those below primarily involve statistical techniques, and their authors often included evaluations of the proposed method and competing approaches—often by Monte Carlo simulation. Much of this work pertains to the data-analysis phase of a quantitative review (i.e. meta-analysis), but some authors addressed broader issues as well (e.g. data repositories, updating reviews).
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
83
11. Bonett DG. Meta-analytic interval estimation for standardized and unstandardized mean differences. Psychological Methods 2009; 14:225–238 [interval estimation, standardized mean difference, random effects, heterogeneity, moderator, Monte Carlo simulation]. 12. Cheung MW-L, Chan W. A two-stage approach to synthesizing covariance matrices in meta-analytic structural equation modeling. Structural Equation Modeling 2009; 16:28–53 [explanatory model, multivariate technique, correlation, Monte Carlo simulation; heterogeneity, software]. 13. Chu H, Nie L, Cole SR, Poole C. Meta-analysis of diagnostic accuracy studies accounting for disease prevalence: alternative parameterizations and model selection. Statistics in Medicine 2009; 28:2384–2399 [diagnostic test accuracy, multivariate technique, random effects, Monte Carlo simulation; heterogeneity]. 14. Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ. Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Statistics in Medicine 2009; 28:1861–1881 [network meta-analysis, heterogeneity, moderator, Bayesian approach]. 15. Costafreda SG, David AS, Brammer MJ. A parametric approach to voxel-based meta-analysis. Neuroimage 2009; 46:115–122 [neuroimaging, heterogeneity, random effects, Monte Carlo simulation; power]. 16. Fibrinogen Studies Collaboration. Correcting for multivariate measurement error by regression calibration in meta-analyses of epidemiological studies. Statistics in Medicine 2009; 28:1067–1092 [individual patient data, longitudinal data, heterogeneity, survival analysis, multivariate technique, random effects; threat to validity, multi-site study]. 17. Hafdahl AR. Improved Fisher z estimators for univariate random-effects meta-analysis of correlations. British Journal of Mathematical and Statistical Psychology 2009; 62:233–261 [random effects, correlation; interval estimation, method comparison]. 18. Hansen RA, Moore CG, Dusetzina SB, Leinwand BI, Gartlehner G, Gaynes BN. Controlling for drug dose in systematic review and meta-analysis: a case study of the effect of antidepressant dose. Medical Decision Making 2009; 29:91–103 [heterogeneity, moderator, coding; dose-response data]. 19. Lai Y, Eckenrode SE, She J-X. A statistical framework for integrating two microarray data sets in differential expression analysis. BMC Bioinformatics 2009; 10(Suppl. 1):S23 [microarray, heterogeneity, Monte Carlo simulation; threat to validity, assumption violation]. 20. Lajeunesse MJ. Meta-analysis and the comparative phylogenetic method. American Naturalist 2009; 174:369–381 [dependence, moderator, heterogeneity; missing data]. 21. Moeltner K, Woodward R. Meta-functional benefit transfer for wetland valuation: making the most of small samples. Environmental and Resource Economics 2009; 42:89–108 [benefit transfer, Bayesian approach, sample size; assumption violation]. 22. Ochsner SA, Steffen DL, Hilsenbeck SG, Chen ES, Watkins C, McKenna NJ. GEMS (Gene Expression Metasignatures), a web resource for querying meta-analysis of expression microarray datasets: 17-estradiol in MCF-7 cells. Cancer Research 2009; 69:23–26 [microarray; data repository, combining p values]. 23. Salanti G, Ioannidis JPA. Synthesis of observational studies should consider credibility ceilings. Journal of Clinical Epidemiology 2009; 62:115–122 [primary-study design, sensitivity analysis; threat to validity, random effects]. 24. Sutton AJ, Donegan S, Takwoingi Y, Garner P, Gamble C, Donald A. An encouraging assessment of methods to inform priorities for updating systematic reviews. Journal of Clinical Epidemiology 2009; 62:241–251 [updating reviews, random effects, method comparison; Cochrane Collaboration, binary outcome].
A. R. HAFDAHL 25. Tian L, Cai T, Pfeffer MA, Piankov N, Cremieux P-Y, Wei LJ. Exact and efficient inference procedure for meta-analysis and its application to the analysis of independent 2 × 2 tables with all available data but without artificial continuity correction. Biostatistics 2009; 10:275–281 [binary outcome, interval estimation, Monte Carlo simulation]. 26. Turner RM, Spiegelhalter DJ, Smith GCS, Thompson SG. Bias modelling in evidence synthesis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:21–47 [primary-study quality, threat to validity, generalized synthesis of evidence, policy implications, heterogeneity, sensitivity analysis]. 27. Welton NJ, Ades AE, Carlin JB, Altman DG, Sterne JAC. Models for potentially biased evidence in meta-analysis using empirically based priors. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:119–136 [primary-study quality, Bayesian approach, heterogeneity, discordant reviews, random effects, sensitivity analysis, moderator]. 28. Welton NJ, Caldwell DM, Adamopoulos E, Vedhara K. Mixed treatment comparison meta-analysis of complex interventions: Psychological interventions in coronary heart disease. American Journal of Epidemiology 2009; 169:1158–1165 [network meta-analysis, heterogeneity, random effects, moderator, Bayesian approach; binary outcome, standardized mean difference]. 29. Won S, Morris N, Lu Q, Elston RC. Choosing an optimal method to combine P-values. Statistics in Medicine 2009; 28: 1537–1553 [combining p values, power, Monte Carlo simulation; genetic association].
Broad review of synthesis methods Authors of items in this category described research synthesis methods in a fairly broad sense, often with particular attention to a specific research domain or type of data (e.g. from studies of diagnostic test accuracy). They typically covered most major phases of review and often addressed the larger context in which syntheses are undertaken (e.g. evidence-based practice). A few authors focused specifically on meta-analysis or on qualitative methods, and some included real-data illustrations.
84
30. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology 2009; 9(59) [overview of research synthesis, qualitative research]. 31. Bath PMW, Gray LJ. Systematic reviews as a tool for planning and interpreting trials. International Journal of Stroke 2009; 4: 23–27 [overview of research synthesis, informing primary research; cumulative meta-analysis, individual patient data, moderator, categorical data]. 32. Charrois TL, Durec T, Tsuyuki RT. Systematic reviews of pharmacy practice research: methodologic issues in searching, evaluating, interpreting, and disseminating results. Annals of Pharmacotherapy 2009; 43:118–122 [overview of research synthesis, evidence-based practice, dissemination]. 33. Chatzisarantis NLD, Stoica A. A primer on the understanding of meta-analysis. Psychology of Sport and Exercise 2009; 10:498–501 [overview of meta-analysis, validity generalization, heterogeneity, moderator]. 34. Coleman CI, Talati R, White CM. A clinician’s perspective on rating the strength of evidence in a systematic review. Pharmacotherapy 2009; 29:1017–1029 [evidence-based practice, strength of evidence, appraising reviews; publication bias, implications for individuals]. 35. Hamada C. The role of meta-analysis in cancer clinical trials. International Journal of Clinical Oncology 2009; 14:90–94 [overview of meta-analysis, sample size, heterogeneity, moderator, large study; evidence-based medicine, individual patient data, publication bias]. 36. Hawke F, Burns J, Landorf KB. Evidence-based podiatric medicine: importance of systematic reviews in clinical practice. Journal of the American Podiatric Medical Association 2009; 99:260–266 [overview of research synthesis, evidence-based practice, Cochrane Collaboration; knowledge transfer, primary-study quality]. 37. Johnson NP, Khan KS. Gynaecologists blaze the trail in primary studies and systematic reviews of diagnostic test accuracy. Australian and New Zealand Journal of Obstetrics and Gynaecology 2009; 49:71–76 [diagnostic test accuracy, primary-study quality, overview of research synthesis, Cochrane Collaboration]. 38. Manchikanti L, Benyamin R, Helm SII, Hirsch JA. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 3: systematic reviews and meta-analyses of randomized trials. Pain Physician 2009; 12:35–72 [overview of research synthesis, review quality, appraising reviews, threat to validity, strength of evidence, guidance for reporting]. 39. Manchikanti L, Datta S, Smith HS, Hirsch JA. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 6. Systematic reviews and meta-analyses of observational studies. Pain Physician 2009; 12:819–850 [overview of research synthesis, primary-study design, primary-study quality, threat to validity, guidance for reviewing, guidance for reporting; evidence-based practice, implications for individuals, review quality]. 40. Perera R, Heneghan C. Interpreting meta-analyses in systematic reviews. Annals of Internal Medicine 2009; 150:JC2-2–JC2-3 [overview of meta-analysis, graphics, heterogeneity]. 41. Ressing M, Blettner M, Klug SJ. Systematic literature reviews and meta-analyses: part 6 of a series on evaluation of scientific publications. Deutsches Ärzteblatt International 2009; 106:456–463 [overview of research synthesis; individual patient data]. 42. Sagoo GS, Little J, Higgins JPT. Systematic reviews of genetic association studies. PLoS Medicine 2009; 6(3) [genetic association, overview of research synthesis; threat to validity, guidance for reporting]. 43. Simunovic N, Sprague S, Bhandari M. Methodological issues in systematic reviews and meta-analyses of observational studies in orthopaedic research. Journal of Bone and Joint Surgery: American Volume 2009; 91(Suppl. 3):87–94 [overview of research synthesis, primary-study design, threat to validity].
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL 44. Suri H, Clarke D. Advancements in research synthesis methods: from a methodologically inclusive perspective. Review of Educational Research 2009; 79:395–430 [overview of research synthesis, qualitative research]. 45. Thorne S. The role of qualitative research within an evidence-based context: can metasynthesis be the answer? International Journal of Nursing Studies 2009; 46:569–575 [qualitative research; evidence-based practice]. 46. Turkelson C, Jacobs JJ. Role of technology assessment in orthopaedics. Clinical Orthopaedics and Related Research 2009; 467:2570–2576 [overview of research synthesis, technology assessment]. 47. Virgili G, Conti AA, Murro V, Gensini GF, Gusinu R. Systematic reviews of diagnostic test accuracy and the Cochrane collaboration. Internal and Emergency Medicine 2009; 4:255–258 [diagnostic test accuracy, Cochrane Collaboration, primarystudy quality]. 48. Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. American Journal of Gastroenterology 2009; 104: 1086–1092 [overview of research synthesis, appraising reviews, threat to validity, primary-study quality, heterogeneity, implications for individuals]
Exposition of specific method or issue In contrast to items in the previous category, those below tended to focus on a fairly specific existing technique or issue (e.g. choice of effect-size index), often including more detail than a broader review. Although many of these authors offered recommendations on the basis of their review of relevant methodology, their emphasis was not on introducing new methods.
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
85
49. Baguley T. Standardized or simple effect size: what should be reported? British Journal of Psychology 2009; 100:603–617 [effect size; standardized mean difference, correlation, primary-study design]. 50. Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI. Understanding heterogeneity in meta-analysis: the role of metaregression. International Journal of Clinical Practice 2009; 63:1426–1434 [heterogeneity, moderator; graphics, threat to validity]. 51. Chappell FM, Raab GM, Wardlaw JM. When are summary ROC curves appropriate for diagnostic meta-analyses? Statistics in Medicine 2009; 28:2653–2668 [diagnostic test accuracy, heterogeneity, random effects, multivariate technique, method comparison, guidance for reviewing; sample size, software]. 52. Cummings P. The relative merits of risk ratios and odds ratios. Archives of Pediatrics and Adolescent Medicine 2009; 163:438–445 [effect size, binary outcome; heterogeneity, moderator]. 53. Edwards SJ, Clarke MJ, Wordsworth S, Borrill J. Indirect comparisons of treatments based on systematic reviews of randomised controlled trials. International Journal of Clinical Practice 2009; 63:841–854 [network meta-analysis, assessment of review practice, method comparison]. 54. Elamin MB, Flynn DN, Bassler D, Briel M, Alonso-Coello P, Karanicolas PJ et al. Choice of data extraction tools for systematic reviews depends on resources and review complexity. Journal of Clinical Epidemiology 2009; 62:506–510 [data management, coding, software]. 55. Feingold A. Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis. Psychological Methods 2009; 14:43–53 [effect size, standardized mean difference, primary-study design]. 56. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:137–159 [heterogeneity, random effects, Bayesian approach, interval estimation, primary-study quality, threat to validity; sensitivity analysis, moderator, software]. 57. Howard GS, Hill TL, Maxwell SE, Baptista TM, Farias MH, Coelho C et al. What’s wrong with research literatures? And how to make them right. Review of General Psychology 2009; 13:146–166 [publication bias, replication, significance testing, Bayesian approach; interval estimation, registry of trials]. 58. Hutton JL. Number needed to treat and number needed to harm are not the best way to report and assess the results of randomised clinical trials. British Journal of Haematology 2009; 146:27–30 [number needed to treat, binary outcome, practical significance]. 59. Keus F, Wetterslev J, Gluud C, Gooszen HG, van Laarhoven CJHM. Robustness assessments are needed to reduce bias in meta-analyses that include zero-event randomized trials. American Journal of Gastroenterology 2009; 104:546–551 [effect size, binary outcome, method comparison]. 60. Pignon J-P, Auperin A, Borget I, Hill C. Role of meta-analyses and of large randomized trials in the study of cancer treatments. Lung Cancer 2009; 65:9–12 [large study, moderator, informing primary research; individual patient data]. 61. Rosenberger RS, Johnston RJ. Selection effects in meta-analysis and benefit transfer: avoiding unintended consequences. Land Economics 2009; 85:410–428 [economic valuation, threat to validity, moderator; policy implications]. 62. Rücker G, Schwarzer G, Carpenter J, Olkin I. Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Statistics in Medicine 2009; 28:721–738 [binary outcome, effect size, random effects, method comparison, Monte Carlo simulation; heterogeneity]. 63. Schmidt FL, Oh I-S, Hayes TL. Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology 2009; 62:97–128 [random effects, heterogeneity; correlation, standardized mean difference]. 64. Scifres CM, Iams JD, Klebanoff M, Macones GA. Metaanalysis vs. large clinical trials: which should guide our management? American Journal of Obstetrics and Gynecology 2009; 200:484.e1–484.e5 [large study, power, individual patient data, heterogeneity; publication bias, significance testing, primary-study quality].
A. R. HAFDAHL 65. Shi Q Sargent DJ. Meta-analysis for the evaluation of surrogate endpoints in cancer clinical trials. International Journal of Clinical Oncology 2009; 14:102–111 [surrogate endpoint, primary-study design, individual patient data, heterogeneity]. 66. Wager TD, Lindquist MA, Nichols TE, Kober H, Van Snellenberg JX. Evaluating the consistency and specificity of neuroimaging data using meta-analysis. Neuroimage 2009; 45:S210–S221 [neuroimaging, significance testing, graphics, resampling; categorical data, power, heterogeneity, publication bias].
Evaluation of method A major contribution of the following items is the evaluation of one or more methods for research synthesis or a related task. This often entailed analytic critique, Monte Carlo simulation, or application to one or more real-world data sets. Some of these authors also proposed a novel or refined method, but not as the main focus. 67. Bax L, Ikeda N, Fukui N, Yaju Y, Tsuruta H, Moons KGM. More than numbers: the power of graphs in meta-analysis. American Journal of Epidemiology 2009; 169:249–255 [graphics, heterogeneity, publication bias, method comparison]. 68. Borm GF, den Heijer M, Zielhuis GA. Publication bias was not a good reason to discourage trials with low power. Journal of Clinical Epidemiology 2009; 62:47–53 [publication bias, power, Monte Carlo simulation; inclusion/exclusion]. 69. Borm GF, Donders ART. Updating meta-analyses leads to larger type I errors than publication bias. Journal of Clinical Epidemiology 2009; 62:825–830 [updating reviews, cumulative meta-analysis, publication bias, Monte Carlo simulation; heterogeneity]. 70. Hoogendam A, Robbé PFD, Stalenhoef AFH, Overbeke JPM. Evaluation of PubMed filters used for evidence-based searching: validation using relative recall. Journal of the Medical Library Association 2009; 97:186–193 [literature search, evidence-based medicine, Cochrane Collaboration]. 71. Jackson D, Bowden J. A re-evaluation of the ‘quantile approximation method’ for random effects meta-analysis. Statistics in Medicine 2009; 28:338–348 [random effects, Monte Carlo simulation; assumption violation]. 72. Moreno SG, Sutton AJ, Ades AE, Stanley TD, Abrams KR, Peters JL et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology 2009; 9(2) [publication bias, heterogeneity, random effects, Monte Carlo simulation; decision making]. 73. Nugent WR. Construct validity invariance and discrepancies in meta-analytic effect sizes based on different measures: a simulation study. Educational and Psychological Measurement 2009; 69:62–78 [effect size, assumption violation, threat to validity, standardized mean difference, correlation, Monte Carlo simulation]. 74. Nugent WR. Meta-analysis as a research synthesis methodology: Cause for concern. Journal of Social Service Research 2009; 35:181–192 [effect size, standardized mean difference, threat to validity; heterogeneity]. 75. O’Regan C, Ghement I, Eyawo O, Guyatt GH, Mills EJ. Incorporating multiple interventions in meta-analysis: an evaluation of the mixed treatment comparison with the adjusted indirect comparison. Trials 2009; 10(86) [network meta-analysis, Bayesian approach, method comparison]. 76. Riley RD. Multivariate meta-analysis: the effect of ignoring within-study correlation. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172:789–811 [multivariate technique, random effects, missing data, Monte Carlo simulation, method comparison; individual patient data, Bayesian approach]. 77. Romano JL, Kromrey JD. What are the consequences if the assumption of independent observations is violated in reliability generalization meta-analysis studies? Educational and Psychological Measurement 2009; 69:404–428 [reliability generalization, dependence, Monte Carlo simulation, interval estimation]. 78. Salanti G, Marinho V, Higgins JPT. A case study of multiple-treatments meta-analysis demonstrates that covariates should be considered. Journal of Clinical Epidemiology 2009; 62:857–864 [network meta-analysis, multivariate technique, moderator; assumption violation, Bayesian approach]. 79. Salimi-Khorshidi G, Smith SM, Keltner JR, Wager TD, Nichols TE. Meta-analysis of neuroimaging data: a comparison of imagebased and coordinate-based pooling of studies. Neuroimage 2009; 45:810–823 [neuroimaging, random effects, missing data, multilevel model, method comparison; power, combining p values]. 80. Thorlund K, Devereaux PJ, Wetterslev J, Guyatt G, Ioannidis JPA, Thabane L et al. Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses? International Journal of Epidemiology 2009; 38:276–286 [cumulative metaanalysis, sample size, power, sequential testing; heterogeneity, binary outcome]. 81. van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. Journal of Clinical Epidemiology 2009; 62:838–844 [literature search, Cochrane Collaboration, grey literature, primary-study quality; publication bias, registry of trials, sponsorship bias].
Evaluation of substantive application(s) Items in this category primarily involved evaluating one or more reported applications of research synthesis or related methods, often with an emphasis on the quality of implementation (rather than the methods’ inherent properties). Some authors focused on specific phases of review or particular threats to validity.
86
82. Al Faleh K, Al-Omran M. Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews. BMC Pediatrics 2009; 9(38) [review quality, assessment of review practice; guidance for reporting]. Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
A. R. HAFDAHL 83. Brok J, Thorlund K, Wetterslev J, Gluud C. Apparently conclusive meta-analyses may be inconclusive—trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal metaanalyses. International Journal of Epidemiology 2009; 38:287–298 [cumulative meta-analysis, sample size, power, sequential testing, assessment of review practice; heterogeneity, Cochrane Collaboration]. 84. Dieckmann NF, Malle BF, Bodner TE. An empirical assessment of meta-analytic practice. Review of General Psychology 2009; 13:101–115 [review quality, assessment of review practice, dissemination, threat to validity; guidance for reporting, overview of research synthesis, sensitivity analysis]. 85. Diener MK, Wolff RF, von Elm E, Rahbari NN, Mavergames C, Knaebel H-P et al. Can decision making in general surgery be based on evidence? An empirical study of Cochrane Reviews. Surgery 2009; 146:444–461 [guidance for practice, strength of evidence, Cochrane Collaboration]. 86. Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results—guidance for future prognosis reviews. Journal of Clinical Epidemiology 2009; 62:781–796 [assessment of review practice, review quality, guidance for reviewing, prognostic studies]. 87. Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. Journal of the American Medical Association 2009; 302:977–984 [registry of trials, reporting bias; publication bias]. 88. Mokkink LB, Terwee CB, Stratford PW, Alonso J, Patrick DL, Riphagen I et al. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research 2009; 18:313–333 [review quality, assessment of review practice, primary-study quality, dissemination]. 89. Mullan RJ, Flynn DN, Carlberg B, Tleyjeh IM, Kamath C, LaBella ML et al. Systematic reviewers commonly contact study authors but do so with limited rigor. Journal of Clinical Epidemiology 2009; 62:138–142 [literature search; missing data]. 90. Rosén M. The aprotinin saga and the risks of conducting meta-analyses on small randomised controlled trials—a critique of a Cochrane review. BMC Health Services Research 2009; 9(34) [assessment of review practice, primary-study quality, primary-study design]. 91. Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. Journal of Clinical Epidemiology 2009; 62:380–386 [publication bias, Cochrane Collaboration, review quality; grey literature]. 92. Vavken P, Dorotka R. A systematic review of conflicting meta-analyses in orthopaedic surgery. Clinical Orthopaedics and Related Research 2009; 467:2723–2735 [discordant reviews, assessment of review practice, review quality; primary-study quality, updating reviews]. 93. Wells E. Uses of meta-analysis in criminal justice research: a quantitative review. Justice Quarterly 2009; 26:268–294 [assessment of review practice, Campbell Collaboration; narrative review, significance testing, dependence]. 94. Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association 2009; 97:21–29 [literature search, guidance for reporting, Cochrane Collaboration, assessment of review practice]. Other type of contribution Each of the following items makes a unique type of methodological contribution relevant to research synthesis that is not readily classified into one of the preceding categories. 95. Cafri G, Kromrey JD, Brannick MT. A SAS macro for statistical power calculations in meta-analysis. Behavior Research Methods 2009; 41:35–46 [power, software, heterogeneity, random effects, moderator]. 96. Clarke M. Can you believe what you read in the papers? Trials 2009; 10(55) [threat to validity, primary-study quality]. 97. Kontopantelis E, Reeves D. MetaEasy: a meta-analysis add-in for Microsoft Excel. Journal of Statistical Software 2009; 30(7) [software; effect size, graphics, heterogeneity, random effects, Bayesian approach]. 98. Moher D. Guidelines for reporting health care research: advancing the clarity and transparency of scientific reporting. Journal Canadien d’Anesthesie 2009; 56:96–101 [dissemination, guidance for reporting; primary-study quality]. 99. Riley RD, Sauerbrei W, Altman DG. Prognostic markers in cancer: the evolution of evidence from single studies to metaanalysis, and beyond. British Journal of Cancer 2009; 100:1219–1229 [prognostic studies, primary-study quality; individual patient data, multi-site study]. 100. Senn SJ. Overstating the evidence—double counting in meta-analysis and related problems. BMC Medical Research Methodology 2009; 9(10) [review quality, threat to validity, assessment of review practice, dependence, missing data; guidance for reporting].
87
Copyright © 2010 John Wiley & Sons, Ltd.
Res. Syn. Meth. 2010, 1 81--87
