LETTER TO Effect of the World Health Organization Checklist on Patient Outcomes: A Stepped Wedge Cluster Randomized Controlled Trial To the Editor:
W
e read with great interest the article by Haugen et al.1 The authors report the results of a stepped wedge cluster randomized controlled trial examining the effects of the World Health Organization checklist on patient outcomes. On the basis of the results, the authors conclude that implementation of the World Health Organization Surgical Safety Checklist (SSC) was associated with a robust reduction in morbidity and length of in-hospital stay and some reduction in mortality. The stepped wedge design is a type of crossover design in which different clusters switch treatments in only 1 direction at different time points.1 Typically, all clusters start in the control condition. Then, the clusters switch to the intervention at consecutive time points, where the time of the switch is randomized for every cluster. Eventually, all clusters will have switched from one condition to the other.2 This is the study design in question in the authors’ study. Although we recognize the importance of this work and applaud the tremendous effort it represents, we have concerns regarding a number of the methodological means employed in the study. We commend the authors for the sample size and power calculations performed. Although the use of this study design is at a relatively early stage in the health care setting,2 we note that the stepped wedge design is increasingly being used in cluster randomized trials. Indeed recent reviews have noted that the reporting of stepped wedge cluster randomized trials needs to be improved, especially the reporting of sample size and power calculations.3,4 However, we have a number of specific concerns. We contend the absence of blinding in this study is an excellent opportunity lost to more reliably estimate the size of the effects of checklist introduction. Clinical teams were fully aware that Disclosure: The authors declare no conflicts of interest. Copyright ß 2015 Wolters Kluwer Health, Inc. All rights reserved. ISSN: 0003-4932/14/26105-0821 DOI: 10.1097/SLA.0000000000001012
e24 | www.annalsofsurgery.com
THE
EDITOR
they were participants in a study of their own behavior. Surgical practice may well have been altered by this research context rather than by the checklist. This may well be an elegant demonstration of the Hawthorne effect described in studies. We respectfully suggest that clinicians could have been kept unaware of the study, and the checklist could have been introduced as a matter of hospital policy in precisely the ways intended for subsequent routine use. The reporting of the data is also an area of contention within this study. The authors describe data from the postintervention cases (n ¼ 3083) were handled as the total intervention group and included cases with noncompliance (intention to treat), partial compliance, and full compliance to the SSC. However, to investigate the SSC effects of full compliance, data from these cases (n ¼ 2263) were handled separately in the analysis. This analysis is not an intentionto-treat analysis and could be described as misleading and also ascribed as a bias within this study. The authors also comment that any difference in complication rates and procedure complexity in each surgical specialty between pre- and postintervention was controlled for in the regression model with interactions. However, we argue that the diversity of specialities involved and the range of operations ranging in complexity and nature make it almost impossible to accurately correct for such vastly different data. We suggest that the authors do not ascribe enough gravity to this potential large bias within the study. N. Lynch, MCh M. Kerin, MCh, FRCSI Discipline of Surgery National University of Ireland Galway, Ireland [email protected]
REFERENCES 1. Haugen AS, Softeland E, Almeland SK, et al. Effect of the World Health Organization checklist on patient outcomes: a stepped wedge cluster randomized controlled trial. Ann Surg. 2015;261:821–828. 2. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6:54. 3. Woertman W, de Hoop E, Moerbeek M, et al. Stepped wedge designs could reduce the required sample size in cluster randomized trials. J Clin Epidemiol. 2013;66:752–758. 4. Mdege ND, Man MS, Taylor Nee Brown CA, et al. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. J Clin Epidemiol. 2011;64:936– 948.
Reply:
W
e thank Mr Lynch and Professor Kerin for their interest and comments regarding our recently published study.1 In our reply, we would like to address the methodological concerns raised. First, Lynch and Kerin suggest that the size of the effect of the World Health Organization Surgical Safety Checklist (WHO SSC) would have been more reliably estimated if the clinical teams had been blinded to the ongoing study—that is, that we obtained a Hawthorne effect as our clinical teams were fully aware of the study. Our study was implemented as part of a quality service improvement project across the wider Health Trust, including all study hospitals. It would have been impossible to set up a study of this size without people knowing it was going on. In fact, the study would have likely been ethically indefensible on such ground and hence not approved. We did take care to address any biases: in line with blinding recommendations for stepped-wedge cluster randomized controlled trials (CRCTs) and to reduce information bias,2 the clinical teams were not informed that the complication codes or length of stay represented the study outcomes. Health care personnel in postanesthesia care/recovery units and in the wards were blinded to the study and its outcomes. Perhaps, as a result, the overall WHO SSC compliance was 73% (2263/3083), which suggests that full implementation (compliance at 100%) is hard to achieve and is indeed an argument against a Hawthorne effect. The novelty of the WHO SSC would arguably also diminish over time, further limiting such bias. Second, Lynch and Kerin describe our data handling and analysis of the postintervention cases as potentially misleading. To clarify, we analyzed the data from pre- to postintervention cases as recommended for stepped-wedge CRCTs.3 From preintervention (n ¼ 2212) to postintervention (n ¼ 3082), complications decreased from 19.9% to 12.4% (P < 0.001). We found a reduction in complications from 19.9% to 11.5% (P < 0.001) when all 3 parts of the WHO SSC had been used (n ¼ 2263).1 This is a transparent way to present the results that underline the importance of fully utilizing all 3 parts of the SSC—that is, the Sign in, Time out, and Sign out parts. This is the full picture of the trial—which the readers can now interpret. Our own view is that better/fuller completion of the SSC leads to more improvement.
Disclosure: The authors declare no conflicts of interest. DOI: 10.1097/SLA.0000000000001024
Annals of Surgery Volume 263, Number 2, February 2016
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
W
e read with great interest the article by Haugen et al.1 The authors report the results of a stepped wedge cluster randomized controlled trial examining the effects of the World Health Organization checklist on patient outcomes. On the basis of the results, the authors conclude that implementation of the World Health Organization Surgical Safety Checklist (SSC) was associated with a robust reduction in morbidity and length of in-hospital stay and some reduction in mortality. The stepped wedge design is a type of crossover design in which different clusters switch treatments in only 1 direction at different time points.1 Typically, all clusters start in the control condition. Then, the clusters switch to the intervention at consecutive time points, where the time of the switch is randomized for every cluster. Eventually, all clusters will have switched from one condition to the other.2 This is the study design in question in the authors’ study. Although we recognize the importance of this work and applaud the tremendous effort it represents, we have concerns regarding a number of the methodological means employed in the study. We commend the authors for the sample size and power calculations performed. Although the use of this study design is at a relatively early stage in the health care setting,2 we note that the stepped wedge design is increasingly being used in cluster randomized trials. Indeed recent reviews have noted that the reporting of stepped wedge cluster randomized trials needs to be improved, especially the reporting of sample size and power calculations.3,4 However, we have a number of specific concerns. We contend the absence of blinding in this study is an excellent opportunity lost to more reliably estimate the size of the effects of checklist introduction. Clinical teams were fully aware that Disclosure: The authors declare no conflicts of interest. Copyright ß 2015 Wolters Kluwer Health, Inc. All rights reserved. ISSN: 0003-4932/14/26105-0821 DOI: 10.1097/SLA.0000000000001012
e24 | www.annalsofsurgery.com
THE
EDITOR
they were participants in a study of their own behavior. Surgical practice may well have been altered by this research context rather than by the checklist. This may well be an elegant demonstration of the Hawthorne effect described in studies. We respectfully suggest that clinicians could have been kept unaware of the study, and the checklist could have been introduced as a matter of hospital policy in precisely the ways intended for subsequent routine use. The reporting of the data is also an area of contention within this study. The authors describe data from the postintervention cases (n ¼ 3083) were handled as the total intervention group and included cases with noncompliance (intention to treat), partial compliance, and full compliance to the SSC. However, to investigate the SSC effects of full compliance, data from these cases (n ¼ 2263) were handled separately in the analysis. This analysis is not an intentionto-treat analysis and could be described as misleading and also ascribed as a bias within this study. The authors also comment that any difference in complication rates and procedure complexity in each surgical specialty between pre- and postintervention was controlled for in the regression model with interactions. However, we argue that the diversity of specialities involved and the range of operations ranging in complexity and nature make it almost impossible to accurately correct for such vastly different data. We suggest that the authors do not ascribe enough gravity to this potential large bias within the study. N. Lynch, MCh M. Kerin, MCh, FRCSI Discipline of Surgery National University of Ireland Galway, Ireland [email protected]
REFERENCES 1. Haugen AS, Softeland E, Almeland SK, et al. Effect of the World Health Organization checklist on patient outcomes: a stepped wedge cluster randomized controlled trial. Ann Surg. 2015;261:821–828. 2. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6:54. 3. Woertman W, de Hoop E, Moerbeek M, et al. Stepped wedge designs could reduce the required sample size in cluster randomized trials. J Clin Epidemiol. 2013;66:752–758. 4. Mdege ND, Man MS, Taylor Nee Brown CA, et al. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. J Clin Epidemiol. 2011;64:936– 948.
Reply:
W
e thank Mr Lynch and Professor Kerin for their interest and comments regarding our recently published study.1 In our reply, we would like to address the methodological concerns raised. First, Lynch and Kerin suggest that the size of the effect of the World Health Organization Surgical Safety Checklist (WHO SSC) would have been more reliably estimated if the clinical teams had been blinded to the ongoing study—that is, that we obtained a Hawthorne effect as our clinical teams were fully aware of the study. Our study was implemented as part of a quality service improvement project across the wider Health Trust, including all study hospitals. It would have been impossible to set up a study of this size without people knowing it was going on. In fact, the study would have likely been ethically indefensible on such ground and hence not approved. We did take care to address any biases: in line with blinding recommendations for stepped-wedge cluster randomized controlled trials (CRCTs) and to reduce information bias,2 the clinical teams were not informed that the complication codes or length of stay represented the study outcomes. Health care personnel in postanesthesia care/recovery units and in the wards were blinded to the study and its outcomes. Perhaps, as a result, the overall WHO SSC compliance was 73% (2263/3083), which suggests that full implementation (compliance at 100%) is hard to achieve and is indeed an argument against a Hawthorne effect. The novelty of the WHO SSC would arguably also diminish over time, further limiting such bias. Second, Lynch and Kerin describe our data handling and analysis of the postintervention cases as potentially misleading. To clarify, we analyzed the data from pre- to postintervention cases as recommended for stepped-wedge CRCTs.3 From preintervention (n ¼ 2212) to postintervention (n ¼ 3082), complications decreased from 19.9% to 12.4% (P < 0.001). We found a reduction in complications from 19.9% to 11.5% (P < 0.001) when all 3 parts of the WHO SSC had been used (n ¼ 2263).1 This is a transparent way to present the results that underline the importance of fully utilizing all 3 parts of the SSC—that is, the Sign in, Time out, and Sign out parts. This is the full picture of the trial—which the readers can now interpret. Our own view is that better/fuller completion of the SSC leads to more improvement.
Disclosure: The authors declare no conflicts of interest. DOI: 10.1097/SLA.0000000000001024
Annals of Surgery Volume 263, Number 2, February 2016
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
