FEATURE ARTICLE
Publication Bias & Small-study Effects in Pediatric Dentistry Meta-analyses Spyridon N. Papageorgioua,b,c, Dionysia Dimitrakid, Trilby Coolidgee, and Nikolaos Kotsanosd a
Department of Orthodontics, School of Dentistry, University of Bonn, Bonn, Germany Department of Oral Technology, School of Dentistry, University of Bonn, Bonn, Germany c Clinical Research Unit 208, University of Bonn, Bonn, Germany d Department of Paediatric Dentistry, School of Dentistry, Aristotle University of Thessaloniki, Thessaloniki, Greece e Department of Oral Health Sciences, School of Dentistry, University of Washington, Seattle, WA, USA b
Abstract
Objectives: The aim of this study was to examine the presence and extent of publication bias and small-study effects in meta-analyses (MAs) investigating pediatric dentistry-related subjects. Methods: Following a literature search, 46 MAs including 882 studies were analyzed qualitatively. Of these, 39 provided enough data to be re-analyzed. Publication bias was assessed with the following methods: contourenhanced funnel plots, Begg and Mazumdar’s rank correlation and Egger’s linear regression tests, Rosenthal’s failsafe N, and Duval and Tweedie’s ‘‘trim and fill’’ procedure. Results: Only a few MAs adequately assessed the existence and effect of publication bias. Inspection of the funnel plots indicated asymmetry, which was confirmed by Begg–Mazumdar’s test in 18% and by Egger’s test in 33% of the MAs. According to Rosenthal’s criterion, 80% of the MAs were robust, while adjusted effects with unpublished studies differed from little to great from the unadjusted ones. Pooling of the Egger’s intercepts indicated that evidence of asymmetry was found in the pediatric dental literature, which was accentuated in dental journals and in diagnostic MAs. Since indications of small-study effects and publication bias in pediatric dentistry were found, the influence of small or missing trials on estimated treatment effects should be routinely assessed in future MAs. Keywords: Pediatric dentistry, Meta-analytical study, Meta-analysis, Publication bias, File-drawer problem, Funnel plot.
Corresponding author. Department of Orthodontics, School of Dentistry, University of Bonn, 53111 Bonn, Germany. Tel.: +49 (0)228 287 22449; fax: +49 (0)228 287 22588; E-mail: [email protected].
Funding and conflict of interest: The authors declare no conflict of interest.
J Evid Base Dent Pract 2015;15:8-24 1532-3382/$36.00 Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jebdp.2014.09.001
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
INTRODUCTION There are many sources of systematic errors in biomedical research and publication bias is just one type of a group of biases termed reporting bias, which is the selective reporting or suppressing of information.1 There is quite a lot of evidence that these biases exist, and it may be safely assumed that most systematic reviews will be subject to reporting bias to some extent.2 Publication bias (also known as the ‘‘file-drawer problem’’) has been identified in many fields of research, including dentistry,3–5 and can be considered as one of the major drawbacks of metaanalyses (MAs) compromising their validity. Publication bias consists of the fact that studies with non-significant results might be published only after some time or not at all. On the other hand, significant/positive results might have a better chance of being published, are published earlier, or published in journals with higher impact factors. Analysis of research publications in five orthodontic journals indicated that studies with significant results were more likely to be accepted for publication.3 This trend was also observed for other dental specialties and was independent of the journal’s Impact Factor. Scholey6 reported in his thesis that less than half of the abstracts presented at leading dental conferences proceeded to full publication. Empirical evidence indicates that results showing non-significant differences have greater chances to remain unpublished, and this could be interpreted as a possible indication of publication bias.7 It has been frequently noted that small trials tend to report greater treatment benefits than larger trials.8,9 For example, on the subject of pulpal exposure after one- vs. two-step incomplete caries removal, the trial of Lula et al.10 including 36 teeth reported a ‘‘strong’’ odds ratio of 0.09 (exposure odds decreased by 91%), while the subsequent trial of Bjørndal et al,11 published the following year including 292 teeth, reported a considerably ‘‘weaker’’ odds ratio of 0.49 (exposure odds decreased by 51%). Although publication bias is often regarded as the main reason for small-study effects,7,12 other factors may also exist,13,14 such as selective outcome reporting, a mathematical artifact due to discordant trial sizes, or random error.15–17 Such ‘‘small-study effects,’’ as they are termed, can result from a combination of lower methodological quality of small trials and reporting biases (including publication bias).8,18,19 However, if the small trials have implemented more careful patient selection and intervention procedures, their results could reflect the actual clinical heterogeneity.20 Systematic reviews and MAs are considered to provide the best quality of evidence, as they increase the power and precision of the included studies, identify heterogeneities across existing studies and might even answer questions that weren’t asked in the original studies. The validity of systematic reviews and MAs is associated with their methodological quality and the unbiased disseminaVolume 15, Number 1
tion of the results of included trials. As however these qualitative/quantitative reviews rely on published material, publication bias can be logically considered as one of the major drawbacks of meta-analyses compromising their validity. If the sample of identified studies is biased, then the validity of the MA is threatened, no matter how systematic and thorough its methodology is. For example, two recent methodological assessments of MAs in the field of orthodontics21,22 found that only one third of them formally considered publication bias, although indications of small-study effects exist. Thus, it is important to continue to consider the possibility that publication bias and small-study effects may impact the accurate interpretation of MAs. The aim of this study was to assess these two biases in a set of MAs in pediatric dentistry through analysis of the studies composing each MA and meta-analytic techniques to explore the extent that small-study effects and the potential for publication bias appear to occur in them.
MATERIALS AND METHODS Selection of MAs and corresponding trials A comprehensive search (Supplementary Table I) of the literature for dentistry-related MAs was previously conducted for a broad meta-epidemiological study.23 This database was augmented by manual searches of MEDLINE via PubMed and Google Scholar up to the second week of March 2013 in order to keep it up to date. No restrictions were applied concerning language, publication date or publication status. MAs were eligible for inclusion if they reported data for any group compared with another group (i.e. placebo, sham, control or other group). From that database, we then selected all MAs in the field of pediatric dentistry with 7 or more included trials. Although no firm guidelines exist, and previous research has included MAs with a minimum of 5–10,20,24,25 a minimum of 7 included studies per MA was deemed as adequate for the assessment of funnel plot asymmetry, due to the small number of eligible MAs. The reports of all trials from the included MAs were obtained and the corresponding original data were re-analyzed. When trial data were not provided by the paper, and retrieval attempts failed, communication with the authors of MAs/trials was attempted.
Data synthesis All analyses were conducted using both the reported effect size metrics (i.e. odds ratios, risk ratio, mean differences, standardized mean differences, etc.) and fixedeffect (Mantel–Haenszel method) or random-effects model (DerSimonian and Laird method), depending on the original paper. The size and impact of the between-study heterogeneity (inconsistency) were measured with the I2 statistic and its 95% confidence 9
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
interval (CI) according to the non-central c2 approximation of Q. A value of 0% indicates no observed heterogeneity, 25% low, 50% moderate and 75% high heterogeneity.26 The extent and impact of small-study effects and/or publication bias were assessed by various means15,27: (a) we drew contour-enhanced funnel plots, which is the most common assessment method for the existence of publication bias in a MA,28 (b) we performed Begg and Mazumdar’s rank correlation test29 and Egger’s linear regression test20 to quantify the bias captured by the funnel plot, and (c) we used Rosenthal’s failsafe N and the Duval and Tweedie’s ‘‘trim and fill’’ method to check the robustness of the results to the possible existence of publication bias.7,30,31 Details on these methods are given in the Supplement. In brief, in the absence of bias, a plot of individual study results to their standard errors should show a random dispersal of the studies in the form of a funnel. An asymmetric plot can indicate that studies with non-significant results were not published, leading to an absence of points in that area of the graph. The two regression tests (Begg–Mazumdar’s and Egger’s test) try to capture the degree of asymmetry of the funnel plot, by regressing a measure of the observed effect to the study’s precision. In case of asymmetry, the regression line would be inclined, either to the left or to the right. The trim and fill approach attempts to find missing studies to make the funnel plot symmetrical and adjust the observed effect accordingly. In the absence of publication bias, no studies are added to the funnel plot. Journal impact factors (IFs) were acquired from ISI Journal Citation Reports.32 H-indices for journals were acquired from SCImago Journal & Country Rank.33 The intercepts of each MA from the Egger’s test were meta-analyzed across studies and associations with influencing characteristics were investigated with mixed-effects subgroup analyses/ random-effects meta-regression.34 All reported P-values are two-sided with level of significance (a) at 0.05 for the null-effect tests and at 0.10 for the regression-based asymmetry tests.25,35 All analyses were performed in Stata version 12 (StataCorp, College Station, TX, USA) with the macros metan, metareg, heterogi, metabias, confunnel and metatrim.
RESULTS The selection and inclusion of studies are summarized in Figure 1. From the original dataset of 81 published MA papers, we included meta-analytical syntheses of seven or more included studies. One such MA could also include subgroup analyses, sensitivity analyses or metaregressions, but was still counted as one MA. From each paper only one MA synthesis was included, in order to deal with study overlap and data correlation, choosing the largest, or in case of equal numbers of studies the one with less heterogeneity. Exclusion was made for Johnson and Little,36 which contributed with two distinct MAs 10
Figure 1. Flow chart describing the identification and selection procedures of included meta-analyses. of different type and included studies. A total of 46 metaanalytical syntheses, reported in 45 articles, were eligible to be included in our evaluation. However, data from primary studies were not provided in seven of these MAs37–43 and were excluded from current re-analysis. Thus, a total of 39 MAs were re-analyzed. An overview of selected characteristics of the 46 included studies originally identified is presented in Table 1. These 46 MAs included 1000 studies of various designs and over 48 million patients (due to including large case-control studies). The median number of studies included per MA was 12.5 (range: 7–133) and the median number of patients per MA was 8994 (range: 297–13,545,026). Twenty-eight MAs included subgroup analyses or meta-regression analysis to identify sources of heterogeneity (Supplementary Table II). Twenty-six MAs assessed various interventions, while the other 20 were of diagnostic nature (two of which assessed genetic associations with conventional MA procedures). Only some included MAs assessed the existence or impact of publication bias on their results, as is seen in Table 2. Almost half (48%) of the included MAs used a funnel plot to evaluate the effects of small studies, although the plot was not always provided in the published report. Statistical tests for funnel plot asymmetry were used in some articles, with Egger’s linear regression test being the most commonly used (n ¼ 14; 30%) and Begg and Mazumdar’s rank correlation test following (n ¼ 6; 13%). Five reports (11%) assessed the robustness of the MA results to publication bias with the failsafe N. Through the various methods described, evidence of publication bias was reported in 11 (24%) of the included MAs. The funnel plots with contours to display areas of significance and non - significance for the 36 re-analyzed March 2015
Meta-analysis
Outcome
Cleft lip with or without palate Badovinac et al44 Non-syndromic CL/P risk
Experimental group Folic acid supplementation during pregnancy Maternal multivitamin consumption Maternal smoking in pregnancy Maternal age (20 yrs)
Diagnostic
NR
15
NRS
Control
Intervention
NR
22
NRS/RCT
Infants’ MTHFR C677T homozygous TT genotype Tobacco smoking during pregnancy Homogenous 677 TT polymorphism infants TGFA C2 allele polymorphism Exposure to high levels of chlorination disinfection by-products Paternal occupational exposure to pesticides Mothers’ MTHFR C677T hetero-/ homozygous CT/TT genotype Maternal age (20 yrs)
Diagnostic
Patientsa 575,407
Effect size
RE
OR
FE
OR
8,089,128
RE
OR
13,545,026c
RE
OR
65,501
RE
OR
10,907,789
RE
OR
1,111,929
FE
OR
NRS (CC)
7656
RE
OR
NRS (CC)
1003
FE
OR
3,510,909
RE
OR
NRS (CC)
255,555
RE
OR
8
NRS (CC)
20,113
RE
OR
NR
8
NRS
9,408,686
NR; considered FE RE
OR
FE
Cohen’s d
Goh et al45
Cleft palate risk
Hackshaw et al46
CL/P risk
Herkrath et al47
CL/P risk
Johnson and Little36 – supplements Johnson and Little36 – genetic
CL/P risk
Little et al48
CL/P risk
Luo et al49
CL/P risk
Mitchell50
Non-syndromic CL/P risk
Nieuwenhuijsen et al51
CL/P risk
Romitti et al42
Oral cleft risk
Verkleij-Hagoort et al52
CL/P risk
Vieira et al53
CL/P risk
Wyszynski et al54
CL/P risk
Tobacco smoking during first pregnancy semester
Control
Diagnostic
NR
11
NRS
109,831
Dental caries Costa55
Dental caries
Toothpaste with high fluoride concentration (500–5000 ppm)
Toothpaste with low fluoride concentration (0–1450 ppm)
Intervention
NR
11
RCT
9232
CL/P risk
Combination method
1184b
OR
(Continued )
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Volume 15, Number 1
TABLE 1. Summary of the 46 included meta-analyses
11
12
TABLE 1. (Continued ) Meta-analysis Derks et al56
Outcome
Experimental group
Comparison group
MA type
Conflict of interest
Study arms
Type of studies
Patientsa
Combination method
Effect size
Fluoridereleasing bonding material
Control
Intervention
NR
7
RCT
297
FE
Other; reanalyzed as OR
Female patients
Male patients
Diagnostic
NR
29
NRS
8777
RE
MD
Obese children according to BMI
Diagnostic
None
16
NRS
32,639
RE
SMD
Helfenstein and Steiner58 Huang et al59
Dental caries
Intervention
NR
Control
Intervention
NR
Hujoel60
Dental caries
Fluoride varnish Fluoride varnish Vitamin D supplementation Pit and fissure sealants Fluoride gel Fluoride varnish Fluoride mouthrinse Fluoride toothpaste Topical fluoride Fluoride toothpaste
Children with normal weight according to BMI Control
Control/ placebo Control
Intervention
None
38
Intervention
NR
75
NRS
Intervention Intervention
NR NR
14 7
RCT RCT
Intervention
NR
34
Intervention
NR
Control/ placebo Fluoride gel/ mouthrinse Fluoride toothpaste alone
Intervention
Haugejorden37 Hayden et al57
Llodra et al
61
Dental caries
Dental caries
8
NRS/RCT
1841
RE
MD
8
RCT
3488
RE
OR
NRS/RCT
2827
RE
RR
FE
RR
4492 2278
RE RE
SMD SMD
RCT
14,663
RE
SMD
70
RCT
42,300
RE
SMD
NR
133
RCT
65,179
RE
Custom (PF)
Intervention
NR
9
RCT
3801
RE
Custom (PF)
Intervention
NR
9
RCT
4026
RE
Custom (PF)
Control
Intervention
NR
8
NRS
(4024)
RE
RR
Control
Intervention
External funding; not commercial External and internal funding; not commercial
19
RCT
8263
FE
Custom (PF)
85
RCT
73,684
RE
Custom (PF)
NR
62
Marinho et al Marinho et al63
Dental caries Dental caries
Marinho et al64
Dental caries
65
Dental caries
66
Dental caries
67
Marinho et al
Dental caries
Marinho et al68
Dental caries
Mejare et al69
Dental caries
van Rijkom et al70
Dental caries
Walsh et al71
Dental caries
Fluoride toothpaste
Placebo/other fluoride toothpaste with lower concentration
Intervention
Dental fluorosis
Optimal water fluoridation (0.8–1.2 ppm)
Sub-optimal water fluoridation (#0.4 ppm)
Intervention
NR
10
RCT
15,277
RE
RR (also OR & RD)
Dental fluorosis
Long duration of fluoride exposure (>23–27 mos)
Short duration of fluoride exposure (#23–27 mos)
Diagnostic
NR
10
NRS
1711
RE
OR
Marinho et al Marinho et al
March 2015
Fluorosis Australian National Health and Medical Research Council72 Bardsen38
Fluoride toothpaste plus others (varnish, gel or rinse) Pit and fissure sealants Fluoride gel
Placebo Control/ placebo Control/ placebo Placebo
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Caries inhibition (prevented fraction) during treatment with fixed appliances Dental caries prevention with the use of fluoride toothpaste Dental caries
Dental fluorosis
Ismail and Bandekar39
Dental fluorosis
Restorative Fallahinejad Ghajari et al73
Fallahinejad Ghajari et al74
Randall et al75
Pulpotomy success in primary molars Pulpotomy success (radiographic) in primary molars Failure of restorations
Schwendicke et al76
Any complication
Xiao-Yan et al77
Failure of restorations
Other Fraz~ao et al78
Malocclusion risk
Reconstituted infant formula consumption Fluoride supplementation
Breast milk/cow’s milk Control/ sporadical users
Intervention
None
18
NRS/RCT
8454
RE
OR
Diagnostic
NR
19
NRS
5635
‘‘Generalized variance’’ reported; reanalyzed as RE
OR
Mineral trioxide aggregate
Formocresol
Intervention
NR
9
RCT
530
RE
OR
Formocresol
Ferric sulfate
Intervention
NR
18
RCT
561
FE
OR
Preformed metal crowns One- or two-step incomplete caries removal Atraumatic restorative treatment using high-viscosity glass-ionomer cement
Amalgam fillings
Intervention
NR
10
NRS
(4410)
FE
OR
Complete caries removal Amalgam restorations without glass-ionomer cement as base or liner
Intervention
None
9
RCT
RE
OR
Intervention
None
36
RCT
(9211)
FE
RR (also OR & RD)
31,827
FE
OR OR
76,536
1257 (1628)
Deciduous dentition Overjet > 3 mm
NR
7
NRS
Traumatic dental injury
Permanent dentition Overjet # 3 mm
Diagnostic
Nguyen et al79
Diagnostic
NR
9
NRS
Polder et al40
Dental agenesis
Female patients
Male patients
Diagnostic
NR
28
NRS
Themessl-Huber et al80
Child dental anxiety
–
Diagnostic
NR
32
NRS/RCT
9782
Weinberg et al41
Maximum head width (Eu–Eu)
Parental dental anxiety Unaffected parents of CL/P children
NR; considered FE NR; considered FE RE
Control Parents
Diagnostic
NR
7
NRS (CC)
1778
FE
NR
RR
Correlation coefficient Cohen’s d
BMI: body mass index; NRS (CC): non-randomized case–control study; CL/P: cleft lip with/without palate; FE: fixed-effect model; MD: mean difference; mos: months; NNT: number needed to treat; NR: not reported; NRS: non-randomized study; OR: odds ratio; PF: prevented fraction; ppm: parts per million; RCT: randomized controlled trial; RD: risk difference; RE: random-effects model; RR: risk ratio; SG: subgroup analysis; SMD: standardized mean difference; yrs: years. a
Numbers in parentheses indicated number of teeth or number of restorations.
b
Only total events number reported.
c
As reported at the original table for all age-groups.
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Volume 15, Number 1
Berg et al29
13
14
TABLE 2. Reported assessments of publication bias in the original 46 meta-analyses Begg– Mazumdar’s test
Egger’s test
Fail-safe N
O O
O O O
Not evident Not evident Not evident Not evident Evident from Egger’s test
O
Evident from Egger’s test
O
O O
O (not provided)
O (Rosenthal’s)
NA Not evident NA Evident; test results not considered rigorous, due to the small number of studies NA Not evident NA NA
O O
a
Helfenstein and Steiner58
O
Huang et al59 Hujoel60
O O
O O
Llodra et al61 Marinho et al62 Marinho et al63
O O O
O O
Marinho et al64
O
O
Meta-analysis
Funnel plot
Cleft lip with or without palate O Badovinac et al44 Goh et al45 O (not provided) Hackshaw et al46 O (not provided) Herkrath et al47 Johnson and Little36 – supplements Johnson and Little36 – genetic Little et al48 Luo et al49 O (not provided) Mitchell50 Nieuwenhuijsen et al51 O (not provided)
March 2015
Funnel plot used to see if studies homogenous NA NA The authors report that the funnel plot ‘‘confirms a low level of publication bias’’ Unlikely that under-reporting of non-significant results can influence the results Evident from funnel plot asymmetry and Egger’s test Evident from funnel plot asymmetry, Egger’s test and identification of unpublished trial results Evident from funnel plot asymmetry Evident from funnel plot asymmetry and Egger’s test Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Romitti et al42 Verkleij-Hagoort et al52 Vieira et al53 Wyszynski et al54 Dental caries Costa55 Derks et al56 Haugejorden37 Hayden et al57
Evidence
O
O
Marinho et al66
O
O
Marinho et al67 Marinho et al68 Mejare et al69 van Rijkom et al70 Walsh et al71 Fluorosis Australian National Health and Medical Research Council72 Bardsen38 Berg et al29
b b O
b b
Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident NA NA NA Evident from funnel plot asymmetry NA
NA
O
O
Ismail and Bandekar39 Restorative Fallahinejad Ghajari et al73 Fallahinejad Ghajari et al74 Randall et al75 Schwendicke et al76 Xiao-Yan et al77
NA Evident from funnel plot asymmetry and Egger’s test; test’s power considered limited NA
O O O O
O O
O
O
O (Rosenthal’s) O (Orwin’s)
Other Fraz~ao et al78 Nguyen et al79 Polder et al40 Themessl-Huber et al80 Weinberg et al41
O: included; : missing; NA: non applicable. Some regression test for asymmetry is reported to be performed, but no additional data or results are reported. b Funnel plot and Egger’s test were planned, but not performed on the basis of insufficient data. a
Not evident Not evident NA Evident from funnel plot asymmetry Egger’s test borderline significant, but it is unclear whether publication bias exists, as the authors plotted multiple datasets from each study NA NA NA Evident from Egger’s test NA
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Volume 15, Number 1
Marinho et al65
15
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Figure 2. Examples of four funnel plots. a) Symmetric funnel plot,46 b) asymmetric funnel plot without missing studies, as no studies exist with negative effect estimates,54 c) asymmetric funnel plot with missing studies in areas of statistical significance, indicating that asymmetry is due to reasons other than publication bias,51 d) asymmetric funnel plot with missing studies in areas of statistical non-significance, possibly indicating publication bias.53 Green areas indicate where studies could be missing.
MAs are presented in Supplementary Figure Ia and Ib. The inter-rater agreement of the funnel plot assessment – measured by Cohen’s kappa – between the two authors was almost perfect (kappa: 0.836; asymptomatic standard error: 0.115). For 22 funnel plots, the scatter of effects and the regression estimates indicated asymmetry (Supplementary Figure Ia and Ib [A, B, C, D, J, K, L, M, N, O, R, S, V, W, Y, Z, ZA, ZB, ZC, ZD, ZF and ZG]]). In 14 funnel plots, the contours suggested missing trials in areas of non – significance (Supplementary Figure Ia and Ib [B, C, D, L, N, O, S, W, Y, Z, ZB, ZD, ZF and ZG]). Four funnel plots are provided in Figure 2 as an example of symmetry, asymmetry without bias, asymmetry with indications of bias other than publication bias and asymmetry with indications of publication bias. 16
The regression-based tests for funnel plot asymmetry, along with the tests for robustness are presented in Table 3. Regarding the Begg–Mazumdar rank correlation test, seven significant positive correlations were found among the included MAs. Regarding the Egger’s linear regression test, the intercepts ranged from 6.01 to 5.89 with an average intercept of 0.23 (95% CI: 0.88 to 0.42). Significant deviations of the intercept were identified for 13 MAs, indicating plot correlation of effect sizes with the SEs. As an example, the forest plots for the normal metaanalytical synthesis (Supplementary Figures IIa, IIIa, IVa, Va) and the cumulative meta-analytical synthesis, adding one study with each line (Supplementary Figures IIb, IIIb, IVb, Vb), of four MAs59,72,75,76 with indications of asymmetry are provided (Supplementary Figures II–V). March 2015
Begg and Mazumdar’s test Meta-analysis
Study arms
Cleft lip with or without palate Badovinac et al44 12 Goh et al45 11 Hackshaw et al46 38 Herkrath et al47 16 36 13 Johnson and Little – supplements 36 Johnson and Little – 22 genetic 48 Little et al 15 15 Luo et al49 Mitchell50 15 Nieuwenhuijsen 8 51 et al Verkleij-Hagoort 8 et al52 53 Vieira et al 8 Wyszynski et al54 11 Dental caries Costa55 11 7 Derks et al56 57 Hayden et al 16 Helfenstein and 8 Steiner58 59 Huang et al 8 38 Hujoel60 Llodra et al61 75 Marinho et al62 14 Marinho et al63 7 64 34 Marinho et al 65 Marinho et al 70 Marinho et al66 133 67 Marinho et al 9 Marinho et al68 9 8 Mejare et al69 70 van Rijkom et al 19 71 Walsh et al 85 Fluorosis Australian National 10 Health and Medical Research Council72 Restorative Fallahinejad Ghajari 9 et al73 Fallahinejad Ghajari 18 et al74 75 Randall et al 10 Schwendicke et al76 8 Xiao-Yan et al77 36 Other Fraz~ao et al78 7 Nguyen et al79 9 Themessl-Huber 32 80 et al Summary Mean
I2 34 0 43 57 29
Random Fixed Random Random Random
0.537 0.756 0.763 0.558 0.428
0.27 (1.88, 2.43) 0.55 (0.86, 1.96) 0.11 (0.79, 0.57) 0.18 (1.50, 1.14) 0.95 (1.48, 3.38)
0.782 0.4 0.736 0.773 0.408
116 203 935 10 NA
O O O –
0.75 (0.65, 0.87) 0.63 (0.54, 0.73) 1.28 (1.20, 1.36) 1.12 (0.97, 1.29) 1.13 (0.90, 1.42)
2 0 0 0 1
61
Random
0.735
0.00 (1.62, 1.61)
0.995
191
O
0.75 (0.64, 0.88)
0
28 37 24 7
Fixed Random Fixed Random
0.767 0.428 0.921 0.019
0.87 (0.13, 1.87) 0.68 (1.02, 2.37) 0.34 (1.12, 1.81) 1.06 (0.04, 2.15)
0.083 0.403 0.621 0.057
140 – 28 NA
O – –
1.32 (1.24, 1.40) 1.22 (0.97, 1.55) 1.45 (1.17, 1.79) 0.98 (0.87, 1.11)
3 1 0 2
1.29 (1.22, 1.37) 1.19 (0.93, 1.52) Same 0.96 (0.80, 1.16)
0
Random
0.711
1.45 (0.65, 3.55)
0.142
NA
–
1.01 (0.87, 1.17)
3
0.93 (0.80, 1.09)
0 44
Fixed Random
0.902 0.276
0.02 (0.93, 0.98) 1.66 (0.49, 2.83)
0.953 0.011
NA 91
– O
1.06 (0.97, 1.16) 1.43 (1.21, 1.68)
0 6
Same 1.18 (0.99, 1.41)a
80 0 73 82
Fixed Fixed Random Random
0.155 0.764 0.822 0.386
2.06 (4.98, 0.86) 1.05 (1.85, 3.94) 0.01 (2.30, 2.27) 2.69 (11.28, 5.89)
0.145 0.395 0.989 0.472
107 5 56 175
O O
0.36 (0.49, 0.23) 0.74 (0.60, 0.92) 0.10 (0.001, 0.21) 0.38 (0.19, 0.57)
0 2 1 0
Same 0.71 (0.58, 0.87) 0.00 (0.002, 0.20)a Same
77 54 93 55 77 63 74 83 78 33 92 53 86
Random Random Fixed Random Random Random Random Random Random Random Random Fixed Random
0.108 0.555
Publication Bias & Small-study Effects in Pediatric Dentistry Meta-analyses Spyridon N. Papageorgioua,b,c, Dionysia Dimitrakid, Trilby Coolidgee, and Nikolaos Kotsanosd a
Department of Orthodontics, School of Dentistry, University of Bonn, Bonn, Germany Department of Oral Technology, School of Dentistry, University of Bonn, Bonn, Germany c Clinical Research Unit 208, University of Bonn, Bonn, Germany d Department of Paediatric Dentistry, School of Dentistry, Aristotle University of Thessaloniki, Thessaloniki, Greece e Department of Oral Health Sciences, School of Dentistry, University of Washington, Seattle, WA, USA b
Abstract
Objectives: The aim of this study was to examine the presence and extent of publication bias and small-study effects in meta-analyses (MAs) investigating pediatric dentistry-related subjects. Methods: Following a literature search, 46 MAs including 882 studies were analyzed qualitatively. Of these, 39 provided enough data to be re-analyzed. Publication bias was assessed with the following methods: contourenhanced funnel plots, Begg and Mazumdar’s rank correlation and Egger’s linear regression tests, Rosenthal’s failsafe N, and Duval and Tweedie’s ‘‘trim and fill’’ procedure. Results: Only a few MAs adequately assessed the existence and effect of publication bias. Inspection of the funnel plots indicated asymmetry, which was confirmed by Begg–Mazumdar’s test in 18% and by Egger’s test in 33% of the MAs. According to Rosenthal’s criterion, 80% of the MAs were robust, while adjusted effects with unpublished studies differed from little to great from the unadjusted ones. Pooling of the Egger’s intercepts indicated that evidence of asymmetry was found in the pediatric dental literature, which was accentuated in dental journals and in diagnostic MAs. Since indications of small-study effects and publication bias in pediatric dentistry were found, the influence of small or missing trials on estimated treatment effects should be routinely assessed in future MAs. Keywords: Pediatric dentistry, Meta-analytical study, Meta-analysis, Publication bias, File-drawer problem, Funnel plot.
Corresponding author. Department of Orthodontics, School of Dentistry, University of Bonn, 53111 Bonn, Germany. Tel.: +49 (0)228 287 22449; fax: +49 (0)228 287 22588; E-mail: [email protected].
Funding and conflict of interest: The authors declare no conflict of interest.
J Evid Base Dent Pract 2015;15:8-24 1532-3382/$36.00 Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jebdp.2014.09.001
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
INTRODUCTION There are many sources of systematic errors in biomedical research and publication bias is just one type of a group of biases termed reporting bias, which is the selective reporting or suppressing of information.1 There is quite a lot of evidence that these biases exist, and it may be safely assumed that most systematic reviews will be subject to reporting bias to some extent.2 Publication bias (also known as the ‘‘file-drawer problem’’) has been identified in many fields of research, including dentistry,3–5 and can be considered as one of the major drawbacks of metaanalyses (MAs) compromising their validity. Publication bias consists of the fact that studies with non-significant results might be published only after some time or not at all. On the other hand, significant/positive results might have a better chance of being published, are published earlier, or published in journals with higher impact factors. Analysis of research publications in five orthodontic journals indicated that studies with significant results were more likely to be accepted for publication.3 This trend was also observed for other dental specialties and was independent of the journal’s Impact Factor. Scholey6 reported in his thesis that less than half of the abstracts presented at leading dental conferences proceeded to full publication. Empirical evidence indicates that results showing non-significant differences have greater chances to remain unpublished, and this could be interpreted as a possible indication of publication bias.7 It has been frequently noted that small trials tend to report greater treatment benefits than larger trials.8,9 For example, on the subject of pulpal exposure after one- vs. two-step incomplete caries removal, the trial of Lula et al.10 including 36 teeth reported a ‘‘strong’’ odds ratio of 0.09 (exposure odds decreased by 91%), while the subsequent trial of Bjørndal et al,11 published the following year including 292 teeth, reported a considerably ‘‘weaker’’ odds ratio of 0.49 (exposure odds decreased by 51%). Although publication bias is often regarded as the main reason for small-study effects,7,12 other factors may also exist,13,14 such as selective outcome reporting, a mathematical artifact due to discordant trial sizes, or random error.15–17 Such ‘‘small-study effects,’’ as they are termed, can result from a combination of lower methodological quality of small trials and reporting biases (including publication bias).8,18,19 However, if the small trials have implemented more careful patient selection and intervention procedures, their results could reflect the actual clinical heterogeneity.20 Systematic reviews and MAs are considered to provide the best quality of evidence, as they increase the power and precision of the included studies, identify heterogeneities across existing studies and might even answer questions that weren’t asked in the original studies. The validity of systematic reviews and MAs is associated with their methodological quality and the unbiased disseminaVolume 15, Number 1
tion of the results of included trials. As however these qualitative/quantitative reviews rely on published material, publication bias can be logically considered as one of the major drawbacks of meta-analyses compromising their validity. If the sample of identified studies is biased, then the validity of the MA is threatened, no matter how systematic and thorough its methodology is. For example, two recent methodological assessments of MAs in the field of orthodontics21,22 found that only one third of them formally considered publication bias, although indications of small-study effects exist. Thus, it is important to continue to consider the possibility that publication bias and small-study effects may impact the accurate interpretation of MAs. The aim of this study was to assess these two biases in a set of MAs in pediatric dentistry through analysis of the studies composing each MA and meta-analytic techniques to explore the extent that small-study effects and the potential for publication bias appear to occur in them.
MATERIALS AND METHODS Selection of MAs and corresponding trials A comprehensive search (Supplementary Table I) of the literature for dentistry-related MAs was previously conducted for a broad meta-epidemiological study.23 This database was augmented by manual searches of MEDLINE via PubMed and Google Scholar up to the second week of March 2013 in order to keep it up to date. No restrictions were applied concerning language, publication date or publication status. MAs were eligible for inclusion if they reported data for any group compared with another group (i.e. placebo, sham, control or other group). From that database, we then selected all MAs in the field of pediatric dentistry with 7 or more included trials. Although no firm guidelines exist, and previous research has included MAs with a minimum of 5–10,20,24,25 a minimum of 7 included studies per MA was deemed as adequate for the assessment of funnel plot asymmetry, due to the small number of eligible MAs. The reports of all trials from the included MAs were obtained and the corresponding original data were re-analyzed. When trial data were not provided by the paper, and retrieval attempts failed, communication with the authors of MAs/trials was attempted.
Data synthesis All analyses were conducted using both the reported effect size metrics (i.e. odds ratios, risk ratio, mean differences, standardized mean differences, etc.) and fixedeffect (Mantel–Haenszel method) or random-effects model (DerSimonian and Laird method), depending on the original paper. The size and impact of the between-study heterogeneity (inconsistency) were measured with the I2 statistic and its 95% confidence 9
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
interval (CI) according to the non-central c2 approximation of Q. A value of 0% indicates no observed heterogeneity, 25% low, 50% moderate and 75% high heterogeneity.26 The extent and impact of small-study effects and/or publication bias were assessed by various means15,27: (a) we drew contour-enhanced funnel plots, which is the most common assessment method for the existence of publication bias in a MA,28 (b) we performed Begg and Mazumdar’s rank correlation test29 and Egger’s linear regression test20 to quantify the bias captured by the funnel plot, and (c) we used Rosenthal’s failsafe N and the Duval and Tweedie’s ‘‘trim and fill’’ method to check the robustness of the results to the possible existence of publication bias.7,30,31 Details on these methods are given in the Supplement. In brief, in the absence of bias, a plot of individual study results to their standard errors should show a random dispersal of the studies in the form of a funnel. An asymmetric plot can indicate that studies with non-significant results were not published, leading to an absence of points in that area of the graph. The two regression tests (Begg–Mazumdar’s and Egger’s test) try to capture the degree of asymmetry of the funnel plot, by regressing a measure of the observed effect to the study’s precision. In case of asymmetry, the regression line would be inclined, either to the left or to the right. The trim and fill approach attempts to find missing studies to make the funnel plot symmetrical and adjust the observed effect accordingly. In the absence of publication bias, no studies are added to the funnel plot. Journal impact factors (IFs) were acquired from ISI Journal Citation Reports.32 H-indices for journals were acquired from SCImago Journal & Country Rank.33 The intercepts of each MA from the Egger’s test were meta-analyzed across studies and associations with influencing characteristics were investigated with mixed-effects subgroup analyses/ random-effects meta-regression.34 All reported P-values are two-sided with level of significance (a) at 0.05 for the null-effect tests and at 0.10 for the regression-based asymmetry tests.25,35 All analyses were performed in Stata version 12 (StataCorp, College Station, TX, USA) with the macros metan, metareg, heterogi, metabias, confunnel and metatrim.
RESULTS The selection and inclusion of studies are summarized in Figure 1. From the original dataset of 81 published MA papers, we included meta-analytical syntheses of seven or more included studies. One such MA could also include subgroup analyses, sensitivity analyses or metaregressions, but was still counted as one MA. From each paper only one MA synthesis was included, in order to deal with study overlap and data correlation, choosing the largest, or in case of equal numbers of studies the one with less heterogeneity. Exclusion was made for Johnson and Little,36 which contributed with two distinct MAs 10
Figure 1. Flow chart describing the identification and selection procedures of included meta-analyses. of different type and included studies. A total of 46 metaanalytical syntheses, reported in 45 articles, were eligible to be included in our evaluation. However, data from primary studies were not provided in seven of these MAs37–43 and were excluded from current re-analysis. Thus, a total of 39 MAs were re-analyzed. An overview of selected characteristics of the 46 included studies originally identified is presented in Table 1. These 46 MAs included 1000 studies of various designs and over 48 million patients (due to including large case-control studies). The median number of studies included per MA was 12.5 (range: 7–133) and the median number of patients per MA was 8994 (range: 297–13,545,026). Twenty-eight MAs included subgroup analyses or meta-regression analysis to identify sources of heterogeneity (Supplementary Table II). Twenty-six MAs assessed various interventions, while the other 20 were of diagnostic nature (two of which assessed genetic associations with conventional MA procedures). Only some included MAs assessed the existence or impact of publication bias on their results, as is seen in Table 2. Almost half (48%) of the included MAs used a funnel plot to evaluate the effects of small studies, although the plot was not always provided in the published report. Statistical tests for funnel plot asymmetry were used in some articles, with Egger’s linear regression test being the most commonly used (n ¼ 14; 30%) and Begg and Mazumdar’s rank correlation test following (n ¼ 6; 13%). Five reports (11%) assessed the robustness of the MA results to publication bias with the failsafe N. Through the various methods described, evidence of publication bias was reported in 11 (24%) of the included MAs. The funnel plots with contours to display areas of significance and non - significance for the 36 re-analyzed March 2015
Meta-analysis
Outcome
Cleft lip with or without palate Badovinac et al44 Non-syndromic CL/P risk
Experimental group Folic acid supplementation during pregnancy Maternal multivitamin consumption Maternal smoking in pregnancy Maternal age (20 yrs)
Diagnostic
NR
15
NRS
Control
Intervention
NR
22
NRS/RCT
Infants’ MTHFR C677T homozygous TT genotype Tobacco smoking during pregnancy Homogenous 677 TT polymorphism infants TGFA C2 allele polymorphism Exposure to high levels of chlorination disinfection by-products Paternal occupational exposure to pesticides Mothers’ MTHFR C677T hetero-/ homozygous CT/TT genotype Maternal age (20 yrs)
Diagnostic
Patientsa 575,407
Effect size
RE
OR
FE
OR
8,089,128
RE
OR
13,545,026c
RE
OR
65,501
RE
OR
10,907,789
RE
OR
1,111,929
FE
OR
NRS (CC)
7656
RE
OR
NRS (CC)
1003
FE
OR
3,510,909
RE
OR
NRS (CC)
255,555
RE
OR
8
NRS (CC)
20,113
RE
OR
NR
8
NRS
9,408,686
NR; considered FE RE
OR
FE
Cohen’s d
Goh et al45
Cleft palate risk
Hackshaw et al46
CL/P risk
Herkrath et al47
CL/P risk
Johnson and Little36 – supplements Johnson and Little36 – genetic
CL/P risk
Little et al48
CL/P risk
Luo et al49
CL/P risk
Mitchell50
Non-syndromic CL/P risk
Nieuwenhuijsen et al51
CL/P risk
Romitti et al42
Oral cleft risk
Verkleij-Hagoort et al52
CL/P risk
Vieira et al53
CL/P risk
Wyszynski et al54
CL/P risk
Tobacco smoking during first pregnancy semester
Control
Diagnostic
NR
11
NRS
109,831
Dental caries Costa55
Dental caries
Toothpaste with high fluoride concentration (500–5000 ppm)
Toothpaste with low fluoride concentration (0–1450 ppm)
Intervention
NR
11
RCT
9232
CL/P risk
Combination method
1184b
OR
(Continued )
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Volume 15, Number 1
TABLE 1. Summary of the 46 included meta-analyses
11
12
TABLE 1. (Continued ) Meta-analysis Derks et al56
Outcome
Experimental group
Comparison group
MA type
Conflict of interest
Study arms
Type of studies
Patientsa
Combination method
Effect size
Fluoridereleasing bonding material
Control
Intervention
NR
7
RCT
297
FE
Other; reanalyzed as OR
Female patients
Male patients
Diagnostic
NR
29
NRS
8777
RE
MD
Obese children according to BMI
Diagnostic
None
16
NRS
32,639
RE
SMD
Helfenstein and Steiner58 Huang et al59
Dental caries
Intervention
NR
Control
Intervention
NR
Hujoel60
Dental caries
Fluoride varnish Fluoride varnish Vitamin D supplementation Pit and fissure sealants Fluoride gel Fluoride varnish Fluoride mouthrinse Fluoride toothpaste Topical fluoride Fluoride toothpaste
Children with normal weight according to BMI Control
Control/ placebo Control
Intervention
None
38
Intervention
NR
75
NRS
Intervention Intervention
NR NR
14 7
RCT RCT
Intervention
NR
34
Intervention
NR
Control/ placebo Fluoride gel/ mouthrinse Fluoride toothpaste alone
Intervention
Haugejorden37 Hayden et al57
Llodra et al
61
Dental caries
Dental caries
8
NRS/RCT
1841
RE
MD
8
RCT
3488
RE
OR
NRS/RCT
2827
RE
RR
FE
RR
4492 2278
RE RE
SMD SMD
RCT
14,663
RE
SMD
70
RCT
42,300
RE
SMD
NR
133
RCT
65,179
RE
Custom (PF)
Intervention
NR
9
RCT
3801
RE
Custom (PF)
Intervention
NR
9
RCT
4026
RE
Custom (PF)
Control
Intervention
NR
8
NRS
(4024)
RE
RR
Control
Intervention
External funding; not commercial External and internal funding; not commercial
19
RCT
8263
FE
Custom (PF)
85
RCT
73,684
RE
Custom (PF)
NR
62
Marinho et al Marinho et al63
Dental caries Dental caries
Marinho et al64
Dental caries
65
Dental caries
66
Dental caries
67
Marinho et al
Dental caries
Marinho et al68
Dental caries
Mejare et al69
Dental caries
van Rijkom et al70
Dental caries
Walsh et al71
Dental caries
Fluoride toothpaste
Placebo/other fluoride toothpaste with lower concentration
Intervention
Dental fluorosis
Optimal water fluoridation (0.8–1.2 ppm)
Sub-optimal water fluoridation (#0.4 ppm)
Intervention
NR
10
RCT
15,277
RE
RR (also OR & RD)
Dental fluorosis
Long duration of fluoride exposure (>23–27 mos)
Short duration of fluoride exposure (#23–27 mos)
Diagnostic
NR
10
NRS
1711
RE
OR
Marinho et al Marinho et al
March 2015
Fluorosis Australian National Health and Medical Research Council72 Bardsen38
Fluoride toothpaste plus others (varnish, gel or rinse) Pit and fissure sealants Fluoride gel
Placebo Control/ placebo Control/ placebo Placebo
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Caries inhibition (prevented fraction) during treatment with fixed appliances Dental caries prevention with the use of fluoride toothpaste Dental caries
Dental fluorosis
Ismail and Bandekar39
Dental fluorosis
Restorative Fallahinejad Ghajari et al73
Fallahinejad Ghajari et al74
Randall et al75
Pulpotomy success in primary molars Pulpotomy success (radiographic) in primary molars Failure of restorations
Schwendicke et al76
Any complication
Xiao-Yan et al77
Failure of restorations
Other Fraz~ao et al78
Malocclusion risk
Reconstituted infant formula consumption Fluoride supplementation
Breast milk/cow’s milk Control/ sporadical users
Intervention
None
18
NRS/RCT
8454
RE
OR
Diagnostic
NR
19
NRS
5635
‘‘Generalized variance’’ reported; reanalyzed as RE
OR
Mineral trioxide aggregate
Formocresol
Intervention
NR
9
RCT
530
RE
OR
Formocresol
Ferric sulfate
Intervention
NR
18
RCT
561
FE
OR
Preformed metal crowns One- or two-step incomplete caries removal Atraumatic restorative treatment using high-viscosity glass-ionomer cement
Amalgam fillings
Intervention
NR
10
NRS
(4410)
FE
OR
Complete caries removal Amalgam restorations without glass-ionomer cement as base or liner
Intervention
None
9
RCT
RE
OR
Intervention
None
36
RCT
(9211)
FE
RR (also OR & RD)
31,827
FE
OR OR
76,536
1257 (1628)
Deciduous dentition Overjet > 3 mm
NR
7
NRS
Traumatic dental injury
Permanent dentition Overjet # 3 mm
Diagnostic
Nguyen et al79
Diagnostic
NR
9
NRS
Polder et al40
Dental agenesis
Female patients
Male patients
Diagnostic
NR
28
NRS
Themessl-Huber et al80
Child dental anxiety
–
Diagnostic
NR
32
NRS/RCT
9782
Weinberg et al41
Maximum head width (Eu–Eu)
Parental dental anxiety Unaffected parents of CL/P children
NR; considered FE NR; considered FE RE
Control Parents
Diagnostic
NR
7
NRS (CC)
1778
FE
NR
RR
Correlation coefficient Cohen’s d
BMI: body mass index; NRS (CC): non-randomized case–control study; CL/P: cleft lip with/without palate; FE: fixed-effect model; MD: mean difference; mos: months; NNT: number needed to treat; NR: not reported; NRS: non-randomized study; OR: odds ratio; PF: prevented fraction; ppm: parts per million; RCT: randomized controlled trial; RD: risk difference; RE: random-effects model; RR: risk ratio; SG: subgroup analysis; SMD: standardized mean difference; yrs: years. a
Numbers in parentheses indicated number of teeth or number of restorations.
b
Only total events number reported.
c
As reported at the original table for all age-groups.
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Volume 15, Number 1
Berg et al29
13
14
TABLE 2. Reported assessments of publication bias in the original 46 meta-analyses Begg– Mazumdar’s test
Egger’s test
Fail-safe N
O O
O O O
Not evident Not evident Not evident Not evident Evident from Egger’s test
O
Evident from Egger’s test
O
O O
O (not provided)
O (Rosenthal’s)
NA Not evident NA Evident; test results not considered rigorous, due to the small number of studies NA Not evident NA NA
O O
a
Helfenstein and Steiner58
O
Huang et al59 Hujoel60
O O
O O
Llodra et al61 Marinho et al62 Marinho et al63
O O O
O O
Marinho et al64
O
O
Meta-analysis
Funnel plot
Cleft lip with or without palate O Badovinac et al44 Goh et al45 O (not provided) Hackshaw et al46 O (not provided) Herkrath et al47 Johnson and Little36 – supplements Johnson and Little36 – genetic Little et al48 Luo et al49 O (not provided) Mitchell50 Nieuwenhuijsen et al51 O (not provided)
March 2015
Funnel plot used to see if studies homogenous NA NA The authors report that the funnel plot ‘‘confirms a low level of publication bias’’ Unlikely that under-reporting of non-significant results can influence the results Evident from funnel plot asymmetry and Egger’s test Evident from funnel plot asymmetry, Egger’s test and identification of unpublished trial results Evident from funnel plot asymmetry Evident from funnel plot asymmetry and Egger’s test Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Romitti et al42 Verkleij-Hagoort et al52 Vieira et al53 Wyszynski et al54 Dental caries Costa55 Derks et al56 Haugejorden37 Hayden et al57
Evidence
O
O
Marinho et al66
O
O
Marinho et al67 Marinho et al68 Mejare et al69 van Rijkom et al70 Walsh et al71 Fluorosis Australian National Health and Medical Research Council72 Bardsen38 Berg et al29
b b O
b b
Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident Funnel plot asymmetric, but Egger’s test non-significant and publication bias was judged as not evident NA NA NA Evident from funnel plot asymmetry NA
NA
O
O
Ismail and Bandekar39 Restorative Fallahinejad Ghajari et al73 Fallahinejad Ghajari et al74 Randall et al75 Schwendicke et al76 Xiao-Yan et al77
NA Evident from funnel plot asymmetry and Egger’s test; test’s power considered limited NA
O O O O
O O
O
O
O (Rosenthal’s) O (Orwin’s)
Other Fraz~ao et al78 Nguyen et al79 Polder et al40 Themessl-Huber et al80 Weinberg et al41
O: included; : missing; NA: non applicable. Some regression test for asymmetry is reported to be performed, but no additional data or results are reported. b Funnel plot and Egger’s test were planned, but not performed on the basis of insufficient data. a
Not evident Not evident NA Evident from funnel plot asymmetry Egger’s test borderline significant, but it is unclear whether publication bias exists, as the authors plotted multiple datasets from each study NA NA NA Evident from Egger’s test NA
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Volume 15, Number 1
Marinho et al65
15
JOURNAL OF EVIDENCE-BASED DENTAL PRACTICE
Figure 2. Examples of four funnel plots. a) Symmetric funnel plot,46 b) asymmetric funnel plot without missing studies, as no studies exist with negative effect estimates,54 c) asymmetric funnel plot with missing studies in areas of statistical significance, indicating that asymmetry is due to reasons other than publication bias,51 d) asymmetric funnel plot with missing studies in areas of statistical non-significance, possibly indicating publication bias.53 Green areas indicate where studies could be missing.
MAs are presented in Supplementary Figure Ia and Ib. The inter-rater agreement of the funnel plot assessment – measured by Cohen’s kappa – between the two authors was almost perfect (kappa: 0.836; asymptomatic standard error: 0.115). For 22 funnel plots, the scatter of effects and the regression estimates indicated asymmetry (Supplementary Figure Ia and Ib [A, B, C, D, J, K, L, M, N, O, R, S, V, W, Y, Z, ZA, ZB, ZC, ZD, ZF and ZG]]). In 14 funnel plots, the contours suggested missing trials in areas of non – significance (Supplementary Figure Ia and Ib [B, C, D, L, N, O, S, W, Y, Z, ZB, ZD, ZF and ZG]). Four funnel plots are provided in Figure 2 as an example of symmetry, asymmetry without bias, asymmetry with indications of bias other than publication bias and asymmetry with indications of publication bias. 16
The regression-based tests for funnel plot asymmetry, along with the tests for robustness are presented in Table 3. Regarding the Begg–Mazumdar rank correlation test, seven significant positive correlations were found among the included MAs. Regarding the Egger’s linear regression test, the intercepts ranged from 6.01 to 5.89 with an average intercept of 0.23 (95% CI: 0.88 to 0.42). Significant deviations of the intercept were identified for 13 MAs, indicating plot correlation of effect sizes with the SEs. As an example, the forest plots for the normal metaanalytical synthesis (Supplementary Figures IIa, IIIa, IVa, Va) and the cumulative meta-analytical synthesis, adding one study with each line (Supplementary Figures IIb, IIIb, IVb, Vb), of four MAs59,72,75,76 with indications of asymmetry are provided (Supplementary Figures II–V). March 2015
Begg and Mazumdar’s test Meta-analysis
Study arms
Cleft lip with or without palate Badovinac et al44 12 Goh et al45 11 Hackshaw et al46 38 Herkrath et al47 16 36 13 Johnson and Little – supplements 36 Johnson and Little – 22 genetic 48 Little et al 15 15 Luo et al49 Mitchell50 15 Nieuwenhuijsen 8 51 et al Verkleij-Hagoort 8 et al52 53 Vieira et al 8 Wyszynski et al54 11 Dental caries Costa55 11 7 Derks et al56 57 Hayden et al 16 Helfenstein and 8 Steiner58 59 Huang et al 8 38 Hujoel60 Llodra et al61 75 Marinho et al62 14 Marinho et al63 7 64 34 Marinho et al 65 Marinho et al 70 Marinho et al66 133 67 Marinho et al 9 Marinho et al68 9 8 Mejare et al69 70 van Rijkom et al 19 71 Walsh et al 85 Fluorosis Australian National 10 Health and Medical Research Council72 Restorative Fallahinejad Ghajari 9 et al73 Fallahinejad Ghajari 18 et al74 75 Randall et al 10 Schwendicke et al76 8 Xiao-Yan et al77 36 Other Fraz~ao et al78 7 Nguyen et al79 9 Themessl-Huber 32 80 et al Summary Mean
I2 34 0 43 57 29
Random Fixed Random Random Random
0.537 0.756 0.763 0.558 0.428
0.27 (1.88, 2.43) 0.55 (0.86, 1.96) 0.11 (0.79, 0.57) 0.18 (1.50, 1.14) 0.95 (1.48, 3.38)
0.782 0.4 0.736 0.773 0.408
116 203 935 10 NA
O O O –
0.75 (0.65, 0.87) 0.63 (0.54, 0.73) 1.28 (1.20, 1.36) 1.12 (0.97, 1.29) 1.13 (0.90, 1.42)
2 0 0 0 1
61
Random
0.735
0.00 (1.62, 1.61)
0.995
191
O
0.75 (0.64, 0.88)
0
28 37 24 7
Fixed Random Fixed Random
0.767 0.428 0.921 0.019
0.87 (0.13, 1.87) 0.68 (1.02, 2.37) 0.34 (1.12, 1.81) 1.06 (0.04, 2.15)
0.083 0.403 0.621 0.057
140 – 28 NA
O – –
1.32 (1.24, 1.40) 1.22 (0.97, 1.55) 1.45 (1.17, 1.79) 0.98 (0.87, 1.11)
3 1 0 2
1.29 (1.22, 1.37) 1.19 (0.93, 1.52) Same 0.96 (0.80, 1.16)
0
Random
0.711
1.45 (0.65, 3.55)
0.142
NA
–
1.01 (0.87, 1.17)
3
0.93 (0.80, 1.09)
0 44
Fixed Random
0.902 0.276
0.02 (0.93, 0.98) 1.66 (0.49, 2.83)
0.953 0.011
NA 91
– O
1.06 (0.97, 1.16) 1.43 (1.21, 1.68)
0 6
Same 1.18 (0.99, 1.41)a
80 0 73 82
Fixed Fixed Random Random
0.155 0.764 0.822 0.386
2.06 (4.98, 0.86) 1.05 (1.85, 3.94) 0.01 (2.30, 2.27) 2.69 (11.28, 5.89)
0.145 0.395 0.989 0.472
107 5 56 175
O O
0.36 (0.49, 0.23) 0.74 (0.60, 0.92) 0.10 (0.001, 0.21) 0.38 (0.19, 0.57)
0 2 1 0
Same 0.71 (0.58, 0.87) 0.00 (0.002, 0.20)a Same
77 54 93 55 77 63 74 83 78 33 92 53 86
Random Random Fixed Random Random Random Random Random Random Random Random Fixed Random
0.108 0.555
