Research Article Received 7 February 2014,
Accepted 20 August 2014
Published online in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/sim.6301
The Peto odds ratio viewed as a new effect measure A. Catharina Brockhaus,a,b*† Ralf Bendera and Guido Skipkaa Meta-analysis has generally been accepted as a fundamental tool for combining effect estimates from several studies. For binary studies with rare events, the Peto odds ratio (POR) method has become the relative effect estimator of choice. However, the POR leads to biased estimates for the OR when treatment effects are large or the group size ratio is not balanced. The aim of this work is to derive the limit of the POR estimator for increasing sample size, to investigate whether the POR limit is equal to the true OR and, if this is not the case, in which situations the POR limit is sufficiently close to the OR. It was found that the derived limit of the expected POR is not equivalent to the OR, because it depends on the group size ratio. Thus, the POR represents a different effect measure. We investigated in which situations the POR is reasonably close to the OR and found that this depends only slightly on the baseline risk within the range (0.001; 0.1) yet substantially on the group size ratio and the effect size itself. We derived the maximum effect size of the POR for different group size ratios and tolerated amounts of bias, for which the POR method results in an acceptable estimator of the OR. We conclude that the limit of the expected POR can be regarded as a new effect measure, which can be used in the presented situations as a valid estimate of the true OR. Copyright © 2014 John Wiley & Sons, Ltd. Keywords:
Peto odds ratio; limit; effect measure; rare event; meta-analysis
1. Introduction Meta-analysis is a fundamental and generally accepted tool within the framework of systematic reviews. A meta-analysis combines estimates of the true effect of several studies. In the case of binary data and rare events, the Peto odds ratio (POR) method [1], also called one-step method [2], is currently the estimation method of choice [3, 4]. However, there are situations with rare events, where the use of POR results in biased estimates of the true underlying odds ratio (OR). Greenland and Salvan [2] showed that the POR method can be extremely biased when the study design is highly unbalanced. Sweeting et al. [5] conducted a simulation study comparing several meta-analysis methods on sparse data and discovered also that the POR method performs poorly in unbalanced designs. The most extensive simulation study to compare several fixed-effect methods for pooling ORs of sparse data was performed by Bradburn et al. [6]. They showed that the POR method only leads to unbiased estimates when events are rare, treatment effects are small or moderate, and the numbers of treated and controlled participants are similar (balanced study design). In practice, this standard situation of rare events, small treatment effects, and a balanced design is not always met. Especially, in the case of rare adverse events, the treatment effects might not be small. In each simulation study described earlier, the estimator of the POR method is compared with the true value of the effect measure OR or risk ratio (RR), assuming that the POR estimator converges toward it. If this assumption holds, the POR method should result in an asymptotically unbiased estimator of OR or RR independent of the true effect size. The aforementioned simulation studies suggest that this is not the case, but to the best of our knowledge, there are yet no investigations of the limit of the POR estimator for increasing sample size. In the following, we consider the true OR in the situation of one study with two parallel groups (presentable in one 2 × 2 table) without any confounder adjustment and p (1−p ) by using the common definition OR = p1 (1−p2 ) , where p1 and p2 are the probabilities that the event will 2 1 occur in the treatment and control group, respectively. a Department of Medical Biometry, Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany b Institute of Health Economics and Clinical Epidemiology, University Hospital Cologne, Cologne, Germany *Correspondence
to: Anne Catharina Brockhaus, Department of Medical Biometry, Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany. † E-mail: [email protected]
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
If the limit of the POR estimator is not equal to the true OR, it is also of interest for which baseline risks, effect sizes, and ratios of group sizes the POR limit is sufficiently close to the true OR. The aim of this work is to derive the limit of the POR estimator for increasing sample size, to investigate whether the POR limit is equal to the true OR and, if this is not the case, in which situations the POR limit is sufficiently close to the OR.
2. Methods 2.1. The Peto odds ratio method The POR was introduced by Yusuf et al. [1] in 1985, and it gained acceptance as an effect estimator for the real underlying OR in the data situation of rare events [3, 7]. The pooled log-OR estimate according to the POR method is given by the following: ∑s ̂ = log(POR)
(Oi − Ei ) , ∑s i=1 Vi
i=1
where O is the observed number; E is the expected number of events among the treated patients, under the null hypothesis of no treatment difference; V is the estimated variance of their difference; and i is the study indicator. In the following, we consider the situation of one study with two parallel groups, where the data can be presented in one 2 × 2 table (Table I). Table I. A 2 × 2 table for one study.
Treatment Control
Event
Nonevent
Total
a c a+c
b d b+d
n1 n2 N
In this study, a of the n1 patients in the treated group and c of the n2 patients in the control group experience an event and b of the n1 patients in the treated group and d of the n2 patients in the control group do not experience an event. N = n1 +n2 is considered as the total number of patients in the treatment and control group. In this case, the POR estimator is given by the following: ) ( ⎛ a − n1 (a+c) ⎞ (aN − an1 − cn1 )N 2 (N − 1) N ⎜ ⎟ ̂ , POR = exp n n (a+c)(b+d) = exp ⎜ 12 ⎟ Nn1 n2 (a + c)(b + d) 2 ⎝ N (N−1) ⎠
(1)
where O = a is the observed number of events in the treatment group, E = (a + b)(a + c)∕N is the expected number of events in the treatment group under the assumption of no treatment effect, and V = ̂ All evaluations and presented (a+b)(a+c)(c+d)(b+d)∕N 2 (N −1) is the estimated variance of log(POR). figures were performed with the statistical software R for Windows, version 2.14 [8]. 2.2. Derivation of the limit of the expected Peto odds ratio ̂ for N → ∞. We started the derivation We derived the theoretical effect measure POR as limit of E(POR) by replacing the observed frequencies in Equation 1 with their expected values E(a) = p1 n1 and E(c) = p2 n2 . This procedure implicitly uses the delta method by approximating the expectation: E(POR(a, c)) ∼ POR(E(a), E(c)). By using this approximate procedure, the limit of the expectation of POR is given by the following: ( ̂ ≈ lim POR(E(a), ̂ lim E(POR) E(c)) = lim exp
N→∞
N→∞
N→∞
(p1 n1 N − p1 n21 − p2 n1 n2 )(N 2 − N)
)
(n1 n2 (p1 n1 + p2 n2 )((n1 − p1 n1 ) + (n2 − p2 n2 )) (2)
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
By reparameterizing the group sizes as n1 = 𝜆n, n2 = (1 − 𝜆)n, and N = n1 + n2 , Equation 2 can be expressed as the following: ̂ ≈ lim POR(E(a), ̂ E(c)) = lim E(POR)
N→∞
N→∞
( = lim exp N→∞
( = lim exp N→∞
( = lim exp N→∞
( = exp
(N(N − 1)(p1 𝜆N 2 − p1 𝜆2 N 2 − p2 𝜆(1 − 𝜆)N 2 ) 𝜆(1 − 𝜆)N 2 (p1 𝜆N + p2 (1 − 𝜆)N)N(1 − p1 𝜆 − p2 (1 − 𝜆))
)
p1 𝜆 − p1 𝜆2 − p2 𝜆(1 − 𝜆) N 3 (N − 1) ⋅ 𝜆(1 − 𝜆)(p1 𝜆 + p2 (1 − 𝜆))(1 − p1 𝜆 − p2 (1 − 𝜆)) N4 p1 𝜆(1 − 𝜆) − p2 𝜆(1 − 𝜆) N−1 ⋅ N 𝜆(1 − 𝜆)(p1 𝜆 + p2 (1 − 𝜆))(1 − p1 𝜆 − p2 (1 − 𝜆))
p1 − p2 (p1 𝜆 + p2 (1 − 𝜆))(1 − (p1 𝜆 + p2 (1 − 𝜆)))
)
)
)
This can be written as the following: ̂ ≈ lim POR(E(a), ̂ lim E(POR) E(c)) = exp
N→∞
N→∞
(
(p1 − p2 ) p(1 − p)
) ,
(3)
where p = p1 𝜆 + p2 (1 − 𝜆) is the group size-weighted average of the group event rates. It is clearly seen that the limit depends on the difference of the event rates in both treatment groups, standardized by the weighted average of the event rates in both groups. The weights consist of the group sizes and thus depend on the group size ratio z = n1 ∕n2 . Hence, the weights can be expressed as 𝜆 = z∕(1 + z). The group size ratio can also be expressed as z = 𝜆∕(1 − 𝜆), which is the odds of 𝜆. The weights for the most common group size ratios are presented in the following Table II. Table II. Weights 𝜆 for the most common group size ratios n1 :n2 . Group size ratio n1 :n2
1 1 1 1 2 3 4
∶ ∶ ∶ ∶ ∶ ∶ ∶
1 2 3 4 1 1 1
Weight 𝜆
1∕2 1∕3 1∕4 1∕5 2∕3 3∕4 4∕5
̂ Equation 3 is the derivation of the limit of the POR of the expected event numbers POR(E(a), E(c)) and approximates the expected POR. By means of simulations, we verified that the use of this approximation has no relevant impact on the limit for N to infinity. We considered all of the group size ratios presented in Table II and three values of the baseline risks p1 and p2 , ranging from very rare events to more common events, and generated 100,000 data sets for each parameter combination. We confirmed that the mean of the POR estimates converges toward the derived limit for N to infinity.
3. Comparison of Peto odds ratio with odds ratio and risk ratio From the limit of the expected POR, it can be concluded that the POR viewed as effect measure is not equivalent to the effect measures OR or RR. In contrast to the POR, OR and RR are independent Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
of the group size ratio. We explored these differences by plotting all three effect measures for different risk combinations in the case of a balanced design. Because the OR and RR are very close and their lines would overlap in all the plots, we only present the comparison of POR and OR in Figure 1. These plots show that even for small event rates, the limit of the expected POR is not equal to the OR. In the following, we assess for which effect size, group size ratio, and baseline risk the POR is sufficiently close to the true OR so that the use of the POR method to estimate the OR is reasonable and not strongly biased. We plotted the logarithm of the true POR against the logarithm of the ratio POR∕OR for different baseline risks p1 and different group size ratios and effect sizes between 0.02 and 50 (Figures 2–4). Figures of other group size ratios are presented in Appendix A.1. For the aforementioned Figures 2–4, we defined the relative deviation boundaries at 1.2 and 1.2−1 (represented as gray dotted lines). This means that for all the values of log(POR), where the ratio log(POR∕OR) stays between those boundaries, the POR method results in a valid estimation of the true OR. All of the presented graphs cross the boundary on the left side and the right side of the null. The more unbalanced the group size ratio is, the less monotone the graph becomes. Thus, the graph crosses a second time both boundaries (compare Figures 3, A.3, and A.4). This second area, in which the POR results in a valid estimation of the true OR is far away from the first one and very small, because the graph is strictly monotonically increasing or strictly monotonically decreasing at this second crossing point. Because this second region is difficult to tabulate and difficult to handle in a practical data setting, we passed on regarding these second and very small areas. Besides the 1.2 times deviation boundary, we further considered different upper boundaries between 1.01 and 1.3 and defined the equivalent lower
Figure 1. Logarithm of the effect measures Peto odds ratio (POR) and OR in a balanced study with different baseline risks in the treatment group p1 and different baseline risks in the control group p2 .
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure 2. Plot of log(POR/OR) versus log(POR) in a balanced design (1 ∶ 1 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure 3. Plot of log(POR/OR) versus log(POR) in an unbalanced design (1 ∶ 2 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure 4. Plot of log(POR/OR) versus log(POR) in an unbalanced design (1 ∶ 3 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure 5. Maximum effect size for which the Peto odds ratio (POR) method can be considered as valid estimator for OR in dependence on the allowed relative deviation boundary; balanced design (1 ∶ 1 ratio); p1 the baseline risk in the treatment group. Table III. Ranges of Peto odds ratios (PORs) for which the limit of the POR estimator is sufficiently close to the OR for different relative deviation boundaries, different group size ratios, and a baseline risk of 0.001. Group size ratio Boundary 1.01 1.05 1.10 1.15 1.20 1.25 1.30
POR
4 ∶ 1
3 ∶ 1
2 ∶ 1
1 ∶ 1
1 ∶ 2
1 ∶ 3
1 ∶ 4
Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit
0.83 1.19 0.64 1.43 0.52 1.61 0.44 1.75 0.38 1.86 0.33 1.94 0.29 2.03
0.81 1.20 0.61 1.47 0.48 1.67 0.39 1.83 0.33 1.94 0.28 2.06 0.24 2.15
0.77 1.25 0.51 1.58 0.32 1.83 0.09 2.02 0.09 2.17 0.08 2.30 0.08 2.42
0.62 1.62 0.45 2.24 0.37 2.69 0.33 3.03 0.30 3.29 0.28 3.53 0.27 3.73
0.80 1.30 0.63 1.96 0.55 3.07 0.50 10.72 0.46 11.24 0.43 11.69 0.41 12.09
0.83 1.23 0.68 1.65 0.60 2.10 0.55 2.56 0.51 3.07 0.49 3.61 0.47 4.24
0.84 1.20 0.70 1.56 0.62 1.93 0.57 2.28 0.54 2.65 0.51 3.02 0.49 3.42
boundaries as (upper boundary)−1 . In a next step, we determined the maximum effect sizes of the POR for which the ratio of POR to OR, depending on the group size ratio, lies within each of the previously chosen boundaries. We identified the first crossing point of the graph with the upper or lower boundary on the left and on the right side of the null effect. We regarded these crossing points as the maximum effect sizes for which the ratio of POR to OR lies within the corresponding boundaries. These maximum effect sizes are illustrated in Figure 5 for a balanced design. The lower line represents the logarithm of the true effect sizes log(POR) < 0, and the upper line represents the logarithm of the true effect sizes log(POR) > 0. Thus, the POR method results in a valid estimation of the true OR, if the logarithm of the true POR lies between the lower and the upper line. We constructed respective plots for different baseline risks between 0.001 and 0.1 and for different group size ratios (Appendix A.2). Because the results for different baseline risks were very similar, we only present the plot for a baseline risk of p1 = 0.001. Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
In Table III, we summarized for the baseline risk p1 = 0.001 different relative deviation boundaries and several common group size ratios, up to which effect size the POR method results in a reliable estimator of the OR. For example, in a study with a balanced design and an accepted deviation boundary of 1.10, the POR method will result in a reliable estimation of the true OR, if the true POR lies between 0.37 and 2.69. The maximum effect sizes of the 1 ∶ 2 ratio are very different for the smaller (1.01, 1.05, and 1.10) than for the larger (1.15, 1.20, 1.25, and 1.30) chosen deviation boundaries. This situation can be easily understood by regarding the shape of the plot of log(POR∕OR) versus log(POR) (Figure 3). A small boundary (represented by the gray dotted line) crosses the graph on the left side of the bump. By choosing a larger boundary, the upper boundary does not cross the graph at all, and the lower boundary crosses it only after the bump. Therefore, the tolerated effect size is much larger. This situation is also summarized in Figure A.5. This observed phenomenon can be equally explained for the 2 ∶ 1 ratio (Figures A.2 and A.8).
4. Example We illustrate the usage of the main results presented in Table III with a practical example from a published study. Tang et al. [9] conducted a randomized controlled trial to determine the impact of laparoscopically assisted colectomy, compared with conventional open surgery, for colorectal cancer on systemic immunity. They presented the complications as the total number per treatment group, which occurred rarely for most of these outcomes. In the first step, we extract binary outcomes with rare events from this study and calculate the POR estimate, because this is currently the estimation method of choice in the case of rare events [3]. As a further illustration, we also calculated the OR estimate, and in the case of zero events, we used the 0.5 correction, which adds 0.5 to all cells of the study results table where the problem occurs. The extracted data, as well as both calculated estimators, are presented in Table IV. These simple calculations show that the POR and OR estimates are very close and even the same for some outcomes but very different for others. Because we do not know the true OR, it is impossible to say which estimator is closer to the truth. The derived Table III might help us with the decision whether the use of the POR estimate leads to an adequate estimate of the true but unknown OR. We need to decide which deviation from this true OR we would still accept as an adequate estimate. Let us be rather strict in this example and choose a deviation boundary of 1.05. In the next step, we need to check whether our calculated POR estimate lies between the maximal effect sizes in terms of the true POR, given in Table III. For a deviation boundary of 1.05 and a balanced design, POR values between the lower limit of 0.45 and the upper limit of 2.24 will result in an adequate estimation of the true OR. Whether the POR method lies between the maximal effect sizes for our chosen boundary of 1.05 is presented in Table IV. The POR as an estimation of the true OR is adequate for the outcomes of the complications ureteric injury, cardiac insufficiency, anastomotic leak, and intra-abdominal abscess. The POR estimate of the other outcomes do not lie between the given limits; therefore, the use of the POR as an estimation method of the true OR will lead in these cases to
Table IV. Number of complications by treatment; values extracted from the study by Tang et al. [9]. Laparoscopic Surgery n1 = 118
Open Surgery n2 = 118
OR estimator
POR estimator
1.05 boundary
1 1 1
1 1 0
1.00 1.00 3.03
1.00 1.00 7.39
Yes Yes No
2 2 0 1 0
1 1 2 0 1
2.02 2.02 0.20 3.03 0.33
1.96 1.96 0.13 7.39 0.14
Yes Yes No No No
Intraoperative Ureteric injury Cardiac insufficiency Surgical emphysema Postoperative Anastomotic leak Intra-abdominal abscess Pulmonary complications Deep vein thrombosis Urinary tract infection POR, Peto odds ratio.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
a bias of more than 1.05 times the value of the true OR. For these outcomes, the POR method might not be the best choice.
5. Discussion ̂ We derived the theoretical effect measure POR as limit of POR(E(a), E(c)) for N → ∞, depending ̂ Because of this dependency, on the group size ratio, which is approximately equal to the E(POR). it can be concluded that the POR is not equal to OR or RR, which both are independent of the group size ratio. We explored the absolute differences of the three effect measures POR, RR, and OR for different baseline risks. These plots support the finding that even for small event rates, the limit of the expected POR is not equivalent to the OR or RR. Therefore, the estimator of the POR method does not converge toward the OR or RR but toward a different effect measure, which can be viewed as a new theoretical effect measure POR. The dependency of this effect measure on the group size ratio makes its interpretation difficult and limits its practical usefulness. We also assessed for which effect size, group size ratio, and baseline risk the POR is sufficiently close to the OR so that the use of the POR method as estimator for the OR is acceptable. We considered several relative deviation boundaries between 1.01 and 1.30 for the comparison of POR and OR. For each boundary, we derived the maximal effect size for which the corresponding ratio POR∕OR lies within the given boundary. Because the plots for baseline risks between 0.001 and 0.1 are very similar, we concluded that the baseline risk – within this range – has a small and negligible impact on the ratio POR∕OR and therefore on the validity of the POR as an estimator of the OR. However, the group size ratio has a huge impact because any imbalance of more than a 1 ∶ 2 ratio leads to a strong relative deviation between POR and OR. We provided guidelines in the form of tables with deviation, which shows situations where the use of the POR as an estimation of the OR leads to acceptable deviations, but it also shows situations where the use of the POR leads to possibly unacceptable deviations from the OR. In situations, where the POR leads to possibly unacceptable deviations, further research is needed to compare the POR estimator with other available estimators. For studies with zero events, the estimated OR is also biased because of the continuity adjustment. Therefore, it is very important to explore which of the available biased estimators is the most adequate one. The same comparison should be investigated separately for the baseline risks between 0.1 and 0.001, in case of lack of zero events in both groups. Besides the regarded bias, it might also be of interest to consider the coverage of the CIs of the POR compared with the OR. This was already explored by Bradburn et al. [6] in a meta-analytical context. They showed that the POR method gave the most appropriate coverage compared to the inverse variance or the Mantel–Haenszel method but performed unacceptably for large effect sizes (RR = 0.2). We were able to replicate these findings and confirmed that the coverage depends on the effect size. Furthermore, we regarded the coverage for only one study with different sample sizes for a given baseline risk between 0.001 and 0.1 and different effect sizes. We discovered that the coverage also depends on the sample size and differs between studies with rare events and studies with no events. Nevertheless, the coverage of the POR seems consistently more appropriate than the coverage of the OR, as long as the effect size is not too large. This is because for large effect sizes, the bias of the POR estimator is unacceptably large as well. Even though the CI of the POR is consistently smaller than the width of the CI of the OR, in the case of large effect sizes, the CI does not overlap the true OR anymore because of the bias. The presented work has the following limitations: In practice, neither the true effect size in terms of OR nor the theoretical limit of POR are available. The use of the presented tables as a guideline to decide whether the POR method is a valid method to estimate the OR when only the corresponding estimates are available ignores the estimation uncertainty. Furthermore, our guidelines are dependent on the chosen deviation boundary, which can vary strongly for different tolerated deviations. These boundaries limit the bias of the POR method as an estimate of the OR, but further research is needed to compare the POR method with other available methods, especially in those cases where the ratio of POR to OR does not lie within any meaningful deviation boundary. These guidelines also depend strongly on the group size ratio. However, this is not a limitation of our work but of the POR itself and therefore not remediable. Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
6. Conclusion In conclusion, we demonstrated that the POR can be viewed as a new effect measure by derivation of the limit of the POR of the expected event numbers as function of the group size ratio, which can also be regarded as the expected POR estimate. We compared this limit with the OR and demonstrated that the POR does not converge toward the OR or RR with increasing sample size. Furthermore, we investigated in which situations the limit of the expected POR is reasonably close to the OR so that one can use the POR method as an estimator of the OR. We found that the validity of the POR method as an estimator of the OR depends only slightly on the baseline risk within the range 0.001–0.1. However, the group size ratio and the effect size itself have a huge impact on the validity. With a chosen relative deviation boundary, which one is willing to accept, the presented results are helpful to decide whether the POR method is useful to estimate the true OR in specific practical situations.
Appendix A A.1. Deviation of the Peto odds ration from the odds ratio
Figure A.1. Plot of log(POR/OR) versus log(POR) in an unbalanced design (1 ∶ 4 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure A.2. Plot of log(POR/OR) versus log(POR) in an unbalanced design (2 ∶ 1 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 . Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure A.3. Plot of log(POR/OR) versus log(POR) in an unbalanced design (3 ∶ 1 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure A.4. Plot of log(POR/OR) versus log(POR) in an unbalanced design (4 ∶ 1 ratio) for different baseline risks in the treatment group p1; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
A.2. Maximum effect size for which the Peto odds ratio method can be considered as valid estimator for odds ratio in dependence on the allowed relative deviation boundary
Figure A.5. Unbalanced design (1 ∶ 2 ratio); p1 the baseline risk in the treatment group.
Figure A.6. Unbalanced design (1 ∶ 3 ratio); p1 the baseline risk in the treatment group.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure A.7. Unbalanced design (1 ∶ 4 ratio); p1 the baseline risk in the treatment group.
Figure A.8. Unbalanced design (2 ∶ 1 ratio); p1 the baseline risk in the treatment group.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure A.9. Unbalanced design (3 ∶ 1 ratio); p1 the baseline risk in the treatment group.
Figure A.10. Unbalanced design (4 ∶ 1 ratio); p1 the baseline risk in the treatment group.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Acknowledgements We thank Anke Schulz for all the helpful and vivid discussions. We thank Gerta Rücker and the reviewers and editors for their helpful and constructive comments, which greatly improved this work.
References 1. Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Progress in Cardiovascular Diseases 1985; 27(5):335–371. 2. Greenland S, Salvan A. Bias in the one-step method for pooling study results. Statistics in Medicine 1990; 9(3):247–252. 3. Higgins JPT, Deeks JJ, Altman DGe. Chapter 16: special topics in statistics. In Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011), Higgins JPT, Green S (eds). The Cochrane Collaboration, 2011. (Available from: www.cochrane-handbook.org [accessed November 2013]). 4. Nikodem N, Siedmiogordzki K, Borowiack E, Zapalska A, Walczak J. Systematic review of statistical methods of metaanalysis and indirect comparison potentially available to use in systematic reviews. eBayesMet 2010. (Available from: http://www.ebayesmet.org [accessed 6 May 2014]). 5. Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23(9):1351–1375. 6. Bradburn MJ, Deeks JJ, Berlin JA, Localio AR. Much ado about nothing: a comparison of the performance of metaanalytical methods with rare events. Statistics in Medicine 2007; 26(1):53–77. 7. Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: analysing data and undertaking meta-analyses. In Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. (Available from: www.cochrane-handbook.org [accessed November 2013]). 8. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing: Vienna, Austria, 2011. Available from: http://www.R-project.org. 9. Tang C-L, Eu K-W, Tai B-C, Soh JGS, Machin D, Seow-Choen F. Randomized clinical trial of the effect of open versus laparoscopically assisted colectomy on systemic immunity in patients with colorectal cancer. British Journal of Surgery 2001; 88:801–807.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
Accepted 20 August 2014
Published online in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/sim.6301
The Peto odds ratio viewed as a new effect measure A. Catharina Brockhaus,a,b*† Ralf Bendera and Guido Skipkaa Meta-analysis has generally been accepted as a fundamental tool for combining effect estimates from several studies. For binary studies with rare events, the Peto odds ratio (POR) method has become the relative effect estimator of choice. However, the POR leads to biased estimates for the OR when treatment effects are large or the group size ratio is not balanced. The aim of this work is to derive the limit of the POR estimator for increasing sample size, to investigate whether the POR limit is equal to the true OR and, if this is not the case, in which situations the POR limit is sufficiently close to the OR. It was found that the derived limit of the expected POR is not equivalent to the OR, because it depends on the group size ratio. Thus, the POR represents a different effect measure. We investigated in which situations the POR is reasonably close to the OR and found that this depends only slightly on the baseline risk within the range (0.001; 0.1) yet substantially on the group size ratio and the effect size itself. We derived the maximum effect size of the POR for different group size ratios and tolerated amounts of bias, for which the POR method results in an acceptable estimator of the OR. We conclude that the limit of the expected POR can be regarded as a new effect measure, which can be used in the presented situations as a valid estimate of the true OR. Copyright © 2014 John Wiley & Sons, Ltd. Keywords:
Peto odds ratio; limit; effect measure; rare event; meta-analysis
1. Introduction Meta-analysis is a fundamental and generally accepted tool within the framework of systematic reviews. A meta-analysis combines estimates of the true effect of several studies. In the case of binary data and rare events, the Peto odds ratio (POR) method [1], also called one-step method [2], is currently the estimation method of choice [3, 4]. However, there are situations with rare events, where the use of POR results in biased estimates of the true underlying odds ratio (OR). Greenland and Salvan [2] showed that the POR method can be extremely biased when the study design is highly unbalanced. Sweeting et al. [5] conducted a simulation study comparing several meta-analysis methods on sparse data and discovered also that the POR method performs poorly in unbalanced designs. The most extensive simulation study to compare several fixed-effect methods for pooling ORs of sparse data was performed by Bradburn et al. [6]. They showed that the POR method only leads to unbiased estimates when events are rare, treatment effects are small or moderate, and the numbers of treated and controlled participants are similar (balanced study design). In practice, this standard situation of rare events, small treatment effects, and a balanced design is not always met. Especially, in the case of rare adverse events, the treatment effects might not be small. In each simulation study described earlier, the estimator of the POR method is compared with the true value of the effect measure OR or risk ratio (RR), assuming that the POR estimator converges toward it. If this assumption holds, the POR method should result in an asymptotically unbiased estimator of OR or RR independent of the true effect size. The aforementioned simulation studies suggest that this is not the case, but to the best of our knowledge, there are yet no investigations of the limit of the POR estimator for increasing sample size. In the following, we consider the true OR in the situation of one study with two parallel groups (presentable in one 2 × 2 table) without any confounder adjustment and p (1−p ) by using the common definition OR = p1 (1−p2 ) , where p1 and p2 are the probabilities that the event will 2 1 occur in the treatment and control group, respectively. a Department of Medical Biometry, Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany b Institute of Health Economics and Clinical Epidemiology, University Hospital Cologne, Cologne, Germany *Correspondence
to: Anne Catharina Brockhaus, Department of Medical Biometry, Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany. † E-mail: [email protected]
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
If the limit of the POR estimator is not equal to the true OR, it is also of interest for which baseline risks, effect sizes, and ratios of group sizes the POR limit is sufficiently close to the true OR. The aim of this work is to derive the limit of the POR estimator for increasing sample size, to investigate whether the POR limit is equal to the true OR and, if this is not the case, in which situations the POR limit is sufficiently close to the OR.
2. Methods 2.1. The Peto odds ratio method The POR was introduced by Yusuf et al. [1] in 1985, and it gained acceptance as an effect estimator for the real underlying OR in the data situation of rare events [3, 7]. The pooled log-OR estimate according to the POR method is given by the following: ∑s ̂ = log(POR)
(Oi − Ei ) , ∑s i=1 Vi
i=1
where O is the observed number; E is the expected number of events among the treated patients, under the null hypothesis of no treatment difference; V is the estimated variance of their difference; and i is the study indicator. In the following, we consider the situation of one study with two parallel groups, where the data can be presented in one 2 × 2 table (Table I). Table I. A 2 × 2 table for one study.
Treatment Control
Event
Nonevent
Total
a c a+c
b d b+d
n1 n2 N
In this study, a of the n1 patients in the treated group and c of the n2 patients in the control group experience an event and b of the n1 patients in the treated group and d of the n2 patients in the control group do not experience an event. N = n1 +n2 is considered as the total number of patients in the treatment and control group. In this case, the POR estimator is given by the following: ) ( ⎛ a − n1 (a+c) ⎞ (aN − an1 − cn1 )N 2 (N − 1) N ⎜ ⎟ ̂ , POR = exp n n (a+c)(b+d) = exp ⎜ 12 ⎟ Nn1 n2 (a + c)(b + d) 2 ⎝ N (N−1) ⎠
(1)
where O = a is the observed number of events in the treatment group, E = (a + b)(a + c)∕N is the expected number of events in the treatment group under the assumption of no treatment effect, and V = ̂ All evaluations and presented (a+b)(a+c)(c+d)(b+d)∕N 2 (N −1) is the estimated variance of log(POR). figures were performed with the statistical software R for Windows, version 2.14 [8]. 2.2. Derivation of the limit of the expected Peto odds ratio ̂ for N → ∞. We started the derivation We derived the theoretical effect measure POR as limit of E(POR) by replacing the observed frequencies in Equation 1 with their expected values E(a) = p1 n1 and E(c) = p2 n2 . This procedure implicitly uses the delta method by approximating the expectation: E(POR(a, c)) ∼ POR(E(a), E(c)). By using this approximate procedure, the limit of the expectation of POR is given by the following: ( ̂ ≈ lim POR(E(a), ̂ lim E(POR) E(c)) = lim exp
N→∞
N→∞
N→∞
(p1 n1 N − p1 n21 − p2 n1 n2 )(N 2 − N)
)
(n1 n2 (p1 n1 + p2 n2 )((n1 − p1 n1 ) + (n2 − p2 n2 )) (2)
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
By reparameterizing the group sizes as n1 = 𝜆n, n2 = (1 − 𝜆)n, and N = n1 + n2 , Equation 2 can be expressed as the following: ̂ ≈ lim POR(E(a), ̂ E(c)) = lim E(POR)
N→∞
N→∞
( = lim exp N→∞
( = lim exp N→∞
( = lim exp N→∞
( = exp
(N(N − 1)(p1 𝜆N 2 − p1 𝜆2 N 2 − p2 𝜆(1 − 𝜆)N 2 ) 𝜆(1 − 𝜆)N 2 (p1 𝜆N + p2 (1 − 𝜆)N)N(1 − p1 𝜆 − p2 (1 − 𝜆))
)
p1 𝜆 − p1 𝜆2 − p2 𝜆(1 − 𝜆) N 3 (N − 1) ⋅ 𝜆(1 − 𝜆)(p1 𝜆 + p2 (1 − 𝜆))(1 − p1 𝜆 − p2 (1 − 𝜆)) N4 p1 𝜆(1 − 𝜆) − p2 𝜆(1 − 𝜆) N−1 ⋅ N 𝜆(1 − 𝜆)(p1 𝜆 + p2 (1 − 𝜆))(1 − p1 𝜆 − p2 (1 − 𝜆))
p1 − p2 (p1 𝜆 + p2 (1 − 𝜆))(1 − (p1 𝜆 + p2 (1 − 𝜆)))
)
)
)
This can be written as the following: ̂ ≈ lim POR(E(a), ̂ lim E(POR) E(c)) = exp
N→∞
N→∞
(
(p1 − p2 ) p(1 − p)
) ,
(3)
where p = p1 𝜆 + p2 (1 − 𝜆) is the group size-weighted average of the group event rates. It is clearly seen that the limit depends on the difference of the event rates in both treatment groups, standardized by the weighted average of the event rates in both groups. The weights consist of the group sizes and thus depend on the group size ratio z = n1 ∕n2 . Hence, the weights can be expressed as 𝜆 = z∕(1 + z). The group size ratio can also be expressed as z = 𝜆∕(1 − 𝜆), which is the odds of 𝜆. The weights for the most common group size ratios are presented in the following Table II. Table II. Weights 𝜆 for the most common group size ratios n1 :n2 . Group size ratio n1 :n2
1 1 1 1 2 3 4
∶ ∶ ∶ ∶ ∶ ∶ ∶
1 2 3 4 1 1 1
Weight 𝜆
1∕2 1∕3 1∕4 1∕5 2∕3 3∕4 4∕5
̂ Equation 3 is the derivation of the limit of the POR of the expected event numbers POR(E(a), E(c)) and approximates the expected POR. By means of simulations, we verified that the use of this approximation has no relevant impact on the limit for N to infinity. We considered all of the group size ratios presented in Table II and three values of the baseline risks p1 and p2 , ranging from very rare events to more common events, and generated 100,000 data sets for each parameter combination. We confirmed that the mean of the POR estimates converges toward the derived limit for N to infinity.
3. Comparison of Peto odds ratio with odds ratio and risk ratio From the limit of the expected POR, it can be concluded that the POR viewed as effect measure is not equivalent to the effect measures OR or RR. In contrast to the POR, OR and RR are independent Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
of the group size ratio. We explored these differences by plotting all three effect measures for different risk combinations in the case of a balanced design. Because the OR and RR are very close and their lines would overlap in all the plots, we only present the comparison of POR and OR in Figure 1. These plots show that even for small event rates, the limit of the expected POR is not equal to the OR. In the following, we assess for which effect size, group size ratio, and baseline risk the POR is sufficiently close to the true OR so that the use of the POR method to estimate the OR is reasonable and not strongly biased. We plotted the logarithm of the true POR against the logarithm of the ratio POR∕OR for different baseline risks p1 and different group size ratios and effect sizes between 0.02 and 50 (Figures 2–4). Figures of other group size ratios are presented in Appendix A.1. For the aforementioned Figures 2–4, we defined the relative deviation boundaries at 1.2 and 1.2−1 (represented as gray dotted lines). This means that for all the values of log(POR), where the ratio log(POR∕OR) stays between those boundaries, the POR method results in a valid estimation of the true OR. All of the presented graphs cross the boundary on the left side and the right side of the null. The more unbalanced the group size ratio is, the less monotone the graph becomes. Thus, the graph crosses a second time both boundaries (compare Figures 3, A.3, and A.4). This second area, in which the POR results in a valid estimation of the true OR is far away from the first one and very small, because the graph is strictly monotonically increasing or strictly monotonically decreasing at this second crossing point. Because this second region is difficult to tabulate and difficult to handle in a practical data setting, we passed on regarding these second and very small areas. Besides the 1.2 times deviation boundary, we further considered different upper boundaries between 1.01 and 1.3 and defined the equivalent lower
Figure 1. Logarithm of the effect measures Peto odds ratio (POR) and OR in a balanced study with different baseline risks in the treatment group p1 and different baseline risks in the control group p2 .
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure 2. Plot of log(POR/OR) versus log(POR) in a balanced design (1 ∶ 1 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure 3. Plot of log(POR/OR) versus log(POR) in an unbalanced design (1 ∶ 2 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure 4. Plot of log(POR/OR) versus log(POR) in an unbalanced design (1 ∶ 3 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure 5. Maximum effect size for which the Peto odds ratio (POR) method can be considered as valid estimator for OR in dependence on the allowed relative deviation boundary; balanced design (1 ∶ 1 ratio); p1 the baseline risk in the treatment group. Table III. Ranges of Peto odds ratios (PORs) for which the limit of the POR estimator is sufficiently close to the OR for different relative deviation boundaries, different group size ratios, and a baseline risk of 0.001. Group size ratio Boundary 1.01 1.05 1.10 1.15 1.20 1.25 1.30
POR
4 ∶ 1
3 ∶ 1
2 ∶ 1
1 ∶ 1
1 ∶ 2
1 ∶ 3
1 ∶ 4
Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit Lower limit Upper limit
0.83 1.19 0.64 1.43 0.52 1.61 0.44 1.75 0.38 1.86 0.33 1.94 0.29 2.03
0.81 1.20 0.61 1.47 0.48 1.67 0.39 1.83 0.33 1.94 0.28 2.06 0.24 2.15
0.77 1.25 0.51 1.58 0.32 1.83 0.09 2.02 0.09 2.17 0.08 2.30 0.08 2.42
0.62 1.62 0.45 2.24 0.37 2.69 0.33 3.03 0.30 3.29 0.28 3.53 0.27 3.73
0.80 1.30 0.63 1.96 0.55 3.07 0.50 10.72 0.46 11.24 0.43 11.69 0.41 12.09
0.83 1.23 0.68 1.65 0.60 2.10 0.55 2.56 0.51 3.07 0.49 3.61 0.47 4.24
0.84 1.20 0.70 1.56 0.62 1.93 0.57 2.28 0.54 2.65 0.51 3.02 0.49 3.42
boundaries as (upper boundary)−1 . In a next step, we determined the maximum effect sizes of the POR for which the ratio of POR to OR, depending on the group size ratio, lies within each of the previously chosen boundaries. We identified the first crossing point of the graph with the upper or lower boundary on the left and on the right side of the null effect. We regarded these crossing points as the maximum effect sizes for which the ratio of POR to OR lies within the corresponding boundaries. These maximum effect sizes are illustrated in Figure 5 for a balanced design. The lower line represents the logarithm of the true effect sizes log(POR) < 0, and the upper line represents the logarithm of the true effect sizes log(POR) > 0. Thus, the POR method results in a valid estimation of the true OR, if the logarithm of the true POR lies between the lower and the upper line. We constructed respective plots for different baseline risks between 0.001 and 0.1 and for different group size ratios (Appendix A.2). Because the results for different baseline risks were very similar, we only present the plot for a baseline risk of p1 = 0.001. Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
In Table III, we summarized for the baseline risk p1 = 0.001 different relative deviation boundaries and several common group size ratios, up to which effect size the POR method results in a reliable estimator of the OR. For example, in a study with a balanced design and an accepted deviation boundary of 1.10, the POR method will result in a reliable estimation of the true OR, if the true POR lies between 0.37 and 2.69. The maximum effect sizes of the 1 ∶ 2 ratio are very different for the smaller (1.01, 1.05, and 1.10) than for the larger (1.15, 1.20, 1.25, and 1.30) chosen deviation boundaries. This situation can be easily understood by regarding the shape of the plot of log(POR∕OR) versus log(POR) (Figure 3). A small boundary (represented by the gray dotted line) crosses the graph on the left side of the bump. By choosing a larger boundary, the upper boundary does not cross the graph at all, and the lower boundary crosses it only after the bump. Therefore, the tolerated effect size is much larger. This situation is also summarized in Figure A.5. This observed phenomenon can be equally explained for the 2 ∶ 1 ratio (Figures A.2 and A.8).
4. Example We illustrate the usage of the main results presented in Table III with a practical example from a published study. Tang et al. [9] conducted a randomized controlled trial to determine the impact of laparoscopically assisted colectomy, compared with conventional open surgery, for colorectal cancer on systemic immunity. They presented the complications as the total number per treatment group, which occurred rarely for most of these outcomes. In the first step, we extract binary outcomes with rare events from this study and calculate the POR estimate, because this is currently the estimation method of choice in the case of rare events [3]. As a further illustration, we also calculated the OR estimate, and in the case of zero events, we used the 0.5 correction, which adds 0.5 to all cells of the study results table where the problem occurs. The extracted data, as well as both calculated estimators, are presented in Table IV. These simple calculations show that the POR and OR estimates are very close and even the same for some outcomes but very different for others. Because we do not know the true OR, it is impossible to say which estimator is closer to the truth. The derived Table III might help us with the decision whether the use of the POR estimate leads to an adequate estimate of the true but unknown OR. We need to decide which deviation from this true OR we would still accept as an adequate estimate. Let us be rather strict in this example and choose a deviation boundary of 1.05. In the next step, we need to check whether our calculated POR estimate lies between the maximal effect sizes in terms of the true POR, given in Table III. For a deviation boundary of 1.05 and a balanced design, POR values between the lower limit of 0.45 and the upper limit of 2.24 will result in an adequate estimation of the true OR. Whether the POR method lies between the maximal effect sizes for our chosen boundary of 1.05 is presented in Table IV. The POR as an estimation of the true OR is adequate for the outcomes of the complications ureteric injury, cardiac insufficiency, anastomotic leak, and intra-abdominal abscess. The POR estimate of the other outcomes do not lie between the given limits; therefore, the use of the POR as an estimation method of the true OR will lead in these cases to
Table IV. Number of complications by treatment; values extracted from the study by Tang et al. [9]. Laparoscopic Surgery n1 = 118
Open Surgery n2 = 118
OR estimator
POR estimator
1.05 boundary
1 1 1
1 1 0
1.00 1.00 3.03
1.00 1.00 7.39
Yes Yes No
2 2 0 1 0
1 1 2 0 1
2.02 2.02 0.20 3.03 0.33
1.96 1.96 0.13 7.39 0.14
Yes Yes No No No
Intraoperative Ureteric injury Cardiac insufficiency Surgical emphysema Postoperative Anastomotic leak Intra-abdominal abscess Pulmonary complications Deep vein thrombosis Urinary tract infection POR, Peto odds ratio.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
a bias of more than 1.05 times the value of the true OR. For these outcomes, the POR method might not be the best choice.
5. Discussion ̂ We derived the theoretical effect measure POR as limit of POR(E(a), E(c)) for N → ∞, depending ̂ Because of this dependency, on the group size ratio, which is approximately equal to the E(POR). it can be concluded that the POR is not equal to OR or RR, which both are independent of the group size ratio. We explored the absolute differences of the three effect measures POR, RR, and OR for different baseline risks. These plots support the finding that even for small event rates, the limit of the expected POR is not equivalent to the OR or RR. Therefore, the estimator of the POR method does not converge toward the OR or RR but toward a different effect measure, which can be viewed as a new theoretical effect measure POR. The dependency of this effect measure on the group size ratio makes its interpretation difficult and limits its practical usefulness. We also assessed for which effect size, group size ratio, and baseline risk the POR is sufficiently close to the OR so that the use of the POR method as estimator for the OR is acceptable. We considered several relative deviation boundaries between 1.01 and 1.30 for the comparison of POR and OR. For each boundary, we derived the maximal effect size for which the corresponding ratio POR∕OR lies within the given boundary. Because the plots for baseline risks between 0.001 and 0.1 are very similar, we concluded that the baseline risk – within this range – has a small and negligible impact on the ratio POR∕OR and therefore on the validity of the POR as an estimator of the OR. However, the group size ratio has a huge impact because any imbalance of more than a 1 ∶ 2 ratio leads to a strong relative deviation between POR and OR. We provided guidelines in the form of tables with deviation, which shows situations where the use of the POR as an estimation of the OR leads to acceptable deviations, but it also shows situations where the use of the POR leads to possibly unacceptable deviations from the OR. In situations, where the POR leads to possibly unacceptable deviations, further research is needed to compare the POR estimator with other available estimators. For studies with zero events, the estimated OR is also biased because of the continuity adjustment. Therefore, it is very important to explore which of the available biased estimators is the most adequate one. The same comparison should be investigated separately for the baseline risks between 0.1 and 0.001, in case of lack of zero events in both groups. Besides the regarded bias, it might also be of interest to consider the coverage of the CIs of the POR compared with the OR. This was already explored by Bradburn et al. [6] in a meta-analytical context. They showed that the POR method gave the most appropriate coverage compared to the inverse variance or the Mantel–Haenszel method but performed unacceptably for large effect sizes (RR = 0.2). We were able to replicate these findings and confirmed that the coverage depends on the effect size. Furthermore, we regarded the coverage for only one study with different sample sizes for a given baseline risk between 0.001 and 0.1 and different effect sizes. We discovered that the coverage also depends on the sample size and differs between studies with rare events and studies with no events. Nevertheless, the coverage of the POR seems consistently more appropriate than the coverage of the OR, as long as the effect size is not too large. This is because for large effect sizes, the bias of the POR estimator is unacceptably large as well. Even though the CI of the POR is consistently smaller than the width of the CI of the OR, in the case of large effect sizes, the CI does not overlap the true OR anymore because of the bias. The presented work has the following limitations: In practice, neither the true effect size in terms of OR nor the theoretical limit of POR are available. The use of the presented tables as a guideline to decide whether the POR method is a valid method to estimate the OR when only the corresponding estimates are available ignores the estimation uncertainty. Furthermore, our guidelines are dependent on the chosen deviation boundary, which can vary strongly for different tolerated deviations. These boundaries limit the bias of the POR method as an estimate of the OR, but further research is needed to compare the POR method with other available methods, especially in those cases where the ratio of POR to OR does not lie within any meaningful deviation boundary. These guidelines also depend strongly on the group size ratio. However, this is not a limitation of our work but of the POR itself and therefore not remediable. Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
6. Conclusion In conclusion, we demonstrated that the POR can be viewed as a new effect measure by derivation of the limit of the POR of the expected event numbers as function of the group size ratio, which can also be regarded as the expected POR estimate. We compared this limit with the OR and demonstrated that the POR does not converge toward the OR or RR with increasing sample size. Furthermore, we investigated in which situations the limit of the expected POR is reasonably close to the OR so that one can use the POR method as an estimator of the OR. We found that the validity of the POR method as an estimator of the OR depends only slightly on the baseline risk within the range 0.001–0.1. However, the group size ratio and the effect size itself have a huge impact on the validity. With a chosen relative deviation boundary, which one is willing to accept, the presented results are helpful to decide whether the POR method is useful to estimate the true OR in specific practical situations.
Appendix A A.1. Deviation of the Peto odds ration from the odds ratio
Figure A.1. Plot of log(POR/OR) versus log(POR) in an unbalanced design (1 ∶ 4 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure A.2. Plot of log(POR/OR) versus log(POR) in an unbalanced design (2 ∶ 1 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 . Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure A.3. Plot of log(POR/OR) versus log(POR) in an unbalanced design (3 ∶ 1 ratio) for different baseline risks in the treatment group p1 ; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Figure A.4. Plot of log(POR/OR) versus log(POR) in an unbalanced design (4 ∶ 1 ratio) for different baseline risks in the treatment group p1; the gray dotted lines represent the defined relative deviation boundaries 1.2 and 1.2−1 .
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
A.2. Maximum effect size for which the Peto odds ratio method can be considered as valid estimator for odds ratio in dependence on the allowed relative deviation boundary
Figure A.5. Unbalanced design (1 ∶ 2 ratio); p1 the baseline risk in the treatment group.
Figure A.6. Unbalanced design (1 ∶ 3 ratio); p1 the baseline risk in the treatment group.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure A.7. Unbalanced design (1 ∶ 4 ratio); p1 the baseline risk in the treatment group.
Figure A.8. Unbalanced design (2 ∶ 1 ratio); p1 the baseline risk in the treatment group.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Figure A.9. Unbalanced design (3 ∶ 1 ratio); p1 the baseline risk in the treatment group.
Figure A.10. Unbalanced design (4 ∶ 1 ratio); p1 the baseline risk in the treatment group.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
A. C. BROCKHAUS, R. BENDER AND G. SKIPKA
Acknowledgements We thank Anke Schulz for all the helpful and vivid discussions. We thank Gerta Rücker and the reviewers and editors for their helpful and constructive comments, which greatly improved this work.
References 1. Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Progress in Cardiovascular Diseases 1985; 27(5):335–371. 2. Greenland S, Salvan A. Bias in the one-step method for pooling study results. Statistics in Medicine 1990; 9(3):247–252. 3. Higgins JPT, Deeks JJ, Altman DGe. Chapter 16: special topics in statistics. In Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011), Higgins JPT, Green S (eds). The Cochrane Collaboration, 2011. (Available from: www.cochrane-handbook.org [accessed November 2013]). 4. Nikodem N, Siedmiogordzki K, Borowiack E, Zapalska A, Walczak J. Systematic review of statistical methods of metaanalysis and indirect comparison potentially available to use in systematic reviews. eBayesMet 2010. (Available from: http://www.ebayesmet.org [accessed 6 May 2014]). 5. Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23(9):1351–1375. 6. Bradburn MJ, Deeks JJ, Berlin JA, Localio AR. Much ado about nothing: a comparison of the performance of metaanalytical methods with rare events. Statistics in Medicine 2007; 26(1):53–77. 7. Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: analysing data and undertaking meta-analyses. In Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. (Available from: www.cochrane-handbook.org [accessed November 2013]). 8. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing: Vienna, Austria, 2011. Available from: http://www.R-project.org. 9. Tang C-L, Eu K-W, Tai B-C, Soh JGS, Machin D, Seow-Choen F. Randomized clinical trial of the effect of open versus laparoscopically assisted colectomy on systemic immunity in patients with colorectal cancer. British Journal of Surgery 2001; 88:801–807.
Copyright © 2014 John Wiley & Sons, Ltd.
Statist. Med. 2014
