Schizophrenia Bulletin vol. 41 no. 1 pp. 20–21, 2015 doi:10.1093/schbul/sbu139 Advance Access publication October 16, 2014
Invited Commentary
Methodological Comments: McFarlane et al
Helena Chmura Kraemer* Department of Psychiatry and Behavioral Sciences, Stanford University (Emerita), Palo Alto, CA
Does the McFarlane et al study1 provide a model for practical trials? Unfortunately, not. The methodological problems are here reviewed and used as a springboard to set out methodological criteria that might define such a model.
reasonable doubt that T is better than C. Here, the conclusion can only continue to be that T may be better than C. What characterizes an ideal practical trial? 1. Clinical equipoise: Reasonable doubt of what the right answer is and a design that will disturb such equipoise.2 2. A patient sample representative of the population clinicians need to deal with. 3. Credible C, a treatment clinicians might choose if T were not available. 4. Protocols for T and C consistent with what clinicians could do in practice. 5. A single primary outcome that reflects the benefit to harm balance in individual patients (avoiding multiple testing).3–5 6. If an observational trial, use of propensity6–8 or other such analyses to remove as much of the bias as possible. If a RCT, an approach consistent with RCT methodology. Any crucial mathematical assumptions must be empirically supported. 7. Adequate power to detect clinically significant treatment effects. 8. Emphasis on clinically interpretable effect sizes, not on P values. 9. After the primary analysis is done, exploratory moderator analysis to identify the subgroups, if any, for whom T > C, T ≈ C, T < C.9–11
Key words: practical trials/methodology/criteria A “practical trial” is one that seeks to answer those questions clinicians need answered to make the best possible clinical decisions for patients. Does the McFarlane et al study1 provide a model for practical trials? Unfortunately, not. The authors themselves give warning that the results are likely to be invalid except under special circumstances.1 A reminder of the design: A baseline variable is used to identify the “high risk” and “low risk” patients, with “high risk” assigned to treatment (T) and “low risk” to control (C). Two straight lines with the same slope are fitted to the observed outcomes and are used to extrapolate what the responses of “low risk” to T and “high risk” to C would have been. The vertical distance between these 2 parallel lines indicates the treatment effect. Figure 1a illustrates those circumstances (parallel linear trajectories) on which correct inferences are based. However, in figure 1b, the T and C have the same nonlinear trajectory and linear extrapolation leads to concluding that T is superior to C (T > C) when T and C are clinically equivalent (T ≈ C). In figure 1c, the lines are not parallel, and here extrapolation leads to concluding that T ≈ C, when in fact T > C for some and T < C for others. Nonsignificant tests of linearity and interaction do not support the contention that these assumptions hold: Absence of proof (nonsignificance) is not proof of absence. Nonsignificant results generally mean inadequate power to detect nonlinearity or interaction. At initiation of any randomized clinical trial (RCT), there is rationale and justification to believe that T may be better than C. The point of a RCT is to prove beyond
In short, the ideal practical trial is designed, not to avoid the rigors of standard RCTs, but to enhance methodology to provide practical guidance to clinicians in order to reduce the burden of health problems. Acknowledgment The authors have declared that there are no conflicts of interest in relation to the subject of this study.
© The Author 2014. Published by Oxford University Press on behalf of the Maryland Psychiatric Research Center. All rights reserved. For permissions, please email: [email protected]
20
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ at University of Birmingham on May 31, 2015
*To whom correspondence should be addressed; Department of Psychiatry and Behavioral Sciences, Stanford University (Emerita), 1116 Forest Avenue, Palo Alto, CA 94301, US; tel: 650-328-7564; e-mail: [email protected]
Methodological Comments
References
Fig. 1. Three hypothetical situations with solid lines indicating the true outcome curves for T and C as a function of the selected baseline variable, the dotted lines indicate extrapolations that would be used with the parallel line assumption. The vertical axis is the outcome and the horizontal axis the baseline factor used for assignment (cutpoint here at 50). Only the right side (baseline over 50 here) of the T response curve and the left side of the C response curve are actually observed. The vertical separation of the dotted lines indicates the estimated treatment effect size using this design. The separation between the solid lines indicates the true treatment effect, which may not be the same for patients with differing baselines.
21
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ at University of Birmingham on May 31, 2015
1. McFarlane WR, Levin B, Travis L, et al. Clinical and functional outcomes after 2 years in the early detection and intervention for the prevention of psychosis multisite effectiveness trial. Schizophr Bull. [first published online July 26, 2014]. doi:10.1093/schbul/sbu108 2. Freedman B. Equipoise and the ethics of clinical research. N Engl J Med. 1987;317:141–145. 3. Lieberman JA, Stroup TS, McEvoy JP, et al.; Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) Investigators. Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. N Engl J Med. 2005;353:1209–1223. 4. Kraemer HC. Discovering, comparing, and combining moderators of treatment on outcome after randomized clinical trials: a parametric approach. Stat Med. 2013;32:1964–1973. 5. Wallace ML, Frank E, Kraemer HC. A novel approach for developing and interpreting treatment moderator profiles in randomized clinical trials. JAMA Psychiatry. 2013;70:1241–1247. 6. Jo B, Stuart EA. On the use of propensity scores in principal causal effect estimation. Stat Med. 2009;28:2857–2875. 7. VanderWeele T. The use of propensity score methods in psychiatric research. Int J Methods Psychiatr Res. 2006;15:95–103. 8. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55. 9. Kraemer HC, Frank E, Kupfer DJ. Moderators of treatment outcomes: clinical, research, and policy importance. J Am Med Assoc. 2006;296:1–4. 10. Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiatry. 2002;59:877–883. 11. Kraemer HC, Stice E, Kazdin A, Offord D, Kupfer D. How do risk factors work together? Mediators, moderators, and independent, overlapping, and proxy risk factors. Am J Psychiatry. 2001;158:848–856.
Invited Commentary
Methodological Comments: McFarlane et al
Helena Chmura Kraemer* Department of Psychiatry and Behavioral Sciences, Stanford University (Emerita), Palo Alto, CA
Does the McFarlane et al study1 provide a model for practical trials? Unfortunately, not. The methodological problems are here reviewed and used as a springboard to set out methodological criteria that might define such a model.
reasonable doubt that T is better than C. Here, the conclusion can only continue to be that T may be better than C. What characterizes an ideal practical trial? 1. Clinical equipoise: Reasonable doubt of what the right answer is and a design that will disturb such equipoise.2 2. A patient sample representative of the population clinicians need to deal with. 3. Credible C, a treatment clinicians might choose if T were not available. 4. Protocols for T and C consistent with what clinicians could do in practice. 5. A single primary outcome that reflects the benefit to harm balance in individual patients (avoiding multiple testing).3–5 6. If an observational trial, use of propensity6–8 or other such analyses to remove as much of the bias as possible. If a RCT, an approach consistent with RCT methodology. Any crucial mathematical assumptions must be empirically supported. 7. Adequate power to detect clinically significant treatment effects. 8. Emphasis on clinically interpretable effect sizes, not on P values. 9. After the primary analysis is done, exploratory moderator analysis to identify the subgroups, if any, for whom T > C, T ≈ C, T < C.9–11
Key words: practical trials/methodology/criteria A “practical trial” is one that seeks to answer those questions clinicians need answered to make the best possible clinical decisions for patients. Does the McFarlane et al study1 provide a model for practical trials? Unfortunately, not. The authors themselves give warning that the results are likely to be invalid except under special circumstances.1 A reminder of the design: A baseline variable is used to identify the “high risk” and “low risk” patients, with “high risk” assigned to treatment (T) and “low risk” to control (C). Two straight lines with the same slope are fitted to the observed outcomes and are used to extrapolate what the responses of “low risk” to T and “high risk” to C would have been. The vertical distance between these 2 parallel lines indicates the treatment effect. Figure 1a illustrates those circumstances (parallel linear trajectories) on which correct inferences are based. However, in figure 1b, the T and C have the same nonlinear trajectory and linear extrapolation leads to concluding that T is superior to C (T > C) when T and C are clinically equivalent (T ≈ C). In figure 1c, the lines are not parallel, and here extrapolation leads to concluding that T ≈ C, when in fact T > C for some and T < C for others. Nonsignificant tests of linearity and interaction do not support the contention that these assumptions hold: Absence of proof (nonsignificance) is not proof of absence. Nonsignificant results generally mean inadequate power to detect nonlinearity or interaction. At initiation of any randomized clinical trial (RCT), there is rationale and justification to believe that T may be better than C. The point of a RCT is to prove beyond
In short, the ideal practical trial is designed, not to avoid the rigors of standard RCTs, but to enhance methodology to provide practical guidance to clinicians in order to reduce the burden of health problems. Acknowledgment The authors have declared that there are no conflicts of interest in relation to the subject of this study.
© The Author 2014. Published by Oxford University Press on behalf of the Maryland Psychiatric Research Center. All rights reserved. For permissions, please email: [email protected]
20
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ at University of Birmingham on May 31, 2015
*To whom correspondence should be addressed; Department of Psychiatry and Behavioral Sciences, Stanford University (Emerita), 1116 Forest Avenue, Palo Alto, CA 94301, US; tel: 650-328-7564; e-mail: [email protected]
Methodological Comments
References
Fig. 1. Three hypothetical situations with solid lines indicating the true outcome curves for T and C as a function of the selected baseline variable, the dotted lines indicate extrapolations that would be used with the parallel line assumption. The vertical axis is the outcome and the horizontal axis the baseline factor used for assignment (cutpoint here at 50). Only the right side (baseline over 50 here) of the T response curve and the left side of the C response curve are actually observed. The vertical separation of the dotted lines indicates the estimated treatment effect size using this design. The separation between the solid lines indicates the true treatment effect, which may not be the same for patients with differing baselines.
21
Downloaded from http://schizophreniabulletin.oxfordjournals.org/ at University of Birmingham on May 31, 2015
1. McFarlane WR, Levin B, Travis L, et al. Clinical and functional outcomes after 2 years in the early detection and intervention for the prevention of psychosis multisite effectiveness trial. Schizophr Bull. [first published online July 26, 2014]. doi:10.1093/schbul/sbu108 2. Freedman B. Equipoise and the ethics of clinical research. N Engl J Med. 1987;317:141–145. 3. Lieberman JA, Stroup TS, McEvoy JP, et al.; Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) Investigators. Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. N Engl J Med. 2005;353:1209–1223. 4. Kraemer HC. Discovering, comparing, and combining moderators of treatment on outcome after randomized clinical trials: a parametric approach. Stat Med. 2013;32:1964–1973. 5. Wallace ML, Frank E, Kraemer HC. A novel approach for developing and interpreting treatment moderator profiles in randomized clinical trials. JAMA Psychiatry. 2013;70:1241–1247. 6. Jo B, Stuart EA. On the use of propensity scores in principal causal effect estimation. Stat Med. 2009;28:2857–2875. 7. VanderWeele T. The use of propensity score methods in psychiatric research. Int J Methods Psychiatr Res. 2006;15:95–103. 8. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55. 9. Kraemer HC, Frank E, Kupfer DJ. Moderators of treatment outcomes: clinical, research, and policy importance. J Am Med Assoc. 2006;296:1–4. 10. Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiatry. 2002;59:877–883. 11. Kraemer HC, Stice E, Kazdin A, Offord D, Kupfer D. How do risk factors work together? Mediators, moderators, and independent, overlapping, and proxy risk factors. Am J Psychiatry. 2001;158:848–856.
