ORIGINAL ARTICLE
Robotic Sacrocolpopexy Performance and Cumulative Summation Analysis Erinn M. Myers, MD,* Elizabeth J. Geller, MD,* AnnaMarie Connolly, MD,* James Michael Bowling, PhD,Þ and Catherine A. Matthews, MD*
Objectives: This study aimed to apply Cumulative Summation (CUSUM) analysis as a tool to monitor robotic sacrocolpopexy (RSCP) proficiency over time. Methods: A retrospective analysis of all women who underwent RSCP between September of 2008 and December of 2011 at the University of North Carolina at Chapel Hill. The performance for 2 attending surgeons was analyzed sequentially over time. Intraoperative complications such as genitourinary or gastrointestinal tract injury, conversion to laparotomy, pulmonary embolus, hemorrhage, and blood transfusion, were identified by International Classification of Diseases, Ninth Revision and Current Procedural Terminology codes. A successful outcome was defined as no intraoperative complications. The target value of success was set at less than 10% complications. CUSUM analysis was then sequentially applied to all RSCP cases for 2 attending surgeons. Results: Over 27 months, 169 RSCPs were performed. The first surgeon performed 107 RSCPs and the second surgeon performed 62 RSCPs with 8 (7.4%) and 3 (4.9%) intraoperative complications, respectively. Total complications included 7 (4.1%) cystotomies, 2 (1.2%) vaginal lacerations, 1 (0.6%) blood transfusion, and 1 (0.6%) bowel perforation. A CUSUM graph was created for each surgeon. Conclusions: CUSUM analysis was successfully applied to monitor RSCP proficiency. Such testing of individual successive procedural outcomes with CUSUM may offer an objective tool to aid in physician self-assessment. Key Words: Cumulative Summation (CUSUM), robotic sacrocolpopexy, surgical proficiency (Female Pelvic Med Reconstr Surg 2014;20: 83Y86)
E
stablishing surgical competence and monitoring surgeon performance are essential to patient safety, but can prove challenging. Surrogate markers of competence have included setting a predetermined number of cases, time taken for task completion, and subjective assessment by a mentor.1 However, these criteria cannot be easily standardized. All surgeons may not achieve adequate performance with a fixed sample size. Operative time can vary depending on the complexity of the individual patient’s anatomy and the procedure performed. Finally, assessment by a mentor is subject to bias and may not be consistently reproducible.1Y3 Regardless of which evaluation method is selected to establish proficiency, continued
From the *Department of Obstetrics and Gynecology, and †Gillings School of International Public Health, University of North Carolina, Chapel Hill, NC. Reprints: Erinn M. Myers, MD, Department of Obstetrics and Gynecology, University of North Carolina, 3023 Old Clinic Bldg, Campus Box 7570, Chapel Hill, NC 27599. E-mail: [email protected]. Dr Catherine A. Matthews is a consultant of Intuitive Surgical and has received research support from the American Medical Systems. Dr Elizabeth J. Geller has received honoraria from Intuitive Surgical. The other authors have declared they have no conflicts of interest. Copyright * 2014 by Lippincott Williams & Wilkins DOI: 10.1097/SPV.0000000000000044
Female Pelvic Medicine & Reconstructive Surgery
&
evaluation of performance over time is necessary to assure that performance goals and patient safety are maintained. Throughout their careers, physicians are similarly required to evaluate their clinical funds of knowledge and skills over time by participating in specialty-specific, continuous professional development, and educational programs known as maintenance of certification (MOC). These programs are endorsed by the American Board of Medical Specialties,4,5 which outlines that MOC is a 4-part process insuring that physicians (1) obtain professional standing with a valid and unrestricted medical license, (2) commit to lifelong learning and periodic self-assessment, (3) demonstrate cognitive expertise by completing written examinations, and (4) exemplify quality performance in practice.4,5 In compliance with these guidelines, the American Board of Obstetrics and Gynecology requires that all certified obstetrician-gynecologists enroll in the annual MOC process that addresses all 4 components during a 6-year cycle.6 Self-assessment, part 2, is of particular interest as it allows physicians to assess and improve their personal performance over time.7 Although self-assessment is part of MOC, it can be subject to individual bias with weak to no associations found between physicians’ self-assessment and external evaluation.8 Cumulative Summation (CUSUM) is a mathematical model that provides surgeons with a real-time, self-assessment tool to examine individual outcomes after each procedure.9 CUSUM was originally described during World War II and was developed to establish quality control in munitions production lines.10 Sequential testing is instituted to ensure that successive products or outcomes are within predetermined boundary limits. When this is applied to surgical outcomes, desired performance goals are defined and then observed performance is measured and compared to the desired performance goal using the CUSUM equation. The CUSUM graph produced from the data will display any observed outcomes that fall outside the acceptable range, thus providing a visual graph to monitor performance. The equation can be recalculated after each procedure for realtime performance monitoring for individual surgeons.9 CUSUM analysis has previously been reported in the literature to describe the learning curve for procedural skills such as endotracheal intubation, fetoscopic laser ablation, robotic hysterectomy, and laparoscopic sacrocolpopexy.11Y14 Our objective was to apply CUSUM analysis to a single procedure, robotic sacrocolpopexy (RSCP), to monitor maintenance of surgical skill proficiency.
MATERIALS AND METHODS Institutional review board approval was obtained at the University of North Carolina at Chapel Hill for this analysis. All women who underwent RSCP between September 2008 and December 2011 were identified by the Current Procedural Terminology code 57425. All cases performed by the 2 attending surgeons were included. Intraoperative complications, defined by the International Classification of Diseases, Ninth
Volume 20, Number 2, March/April 2014
www.fpmrs.net
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
83
Myers et al
Female Pelvic Medicine & Reconstructive Surgery
Revision and Current Procedural Terminology codes included genitourinary or gastrointestinal tract injury, conversion to laparotomy, pulmonary embolus, hemorrhage, and blood transfusion. These complications were confirmed by cross-referencing the Operating Room Robotic Database, Medical Information Management Database, Transfusion Database, and each operative note and discharge summary in the electronic medical record. Demographic data such as medical and surgical history and preoperative and postoperative pelvic organ prolapse quantification examinations were not collected for this analysis. We have previously described the general surgical steps that we use for RSCP, including port placement, robotic docking, and specific instrumentation.15 In general, the anterior and posterior vaginal dissections were extended down to the bladder trigone and the perineal body, respectively. During the study period, both surgeons used IntePro Y-mesh (American Medical Systems, Inc, Minnetonka, Minn) and attached the mesh using 6 to 8 stitches on the anterior and posterior vaginal walls and 2 to 3 sutures in the anterior longitudinal sacral ligament. Surgeon 1 used 2-0 Ethibond (Ethicon, Somerville, NJ) and surgeon 2 used 2-0 Gore-Tex (W.L. Gore and Associates, Flagstaff, Ariz) to secure the mesh to the vagina and anterior longitudinal ligament. Surgeon 1 incised the peritoneum along the right pelvic sidewall from the sacrum to the cul-de-sac and then after mesh placement reperitonealized the mesh. Surgeon 2 created a peritoneal tunnel for the mesh. Concomitant procedures and their associated complications were recorded but not included in the CUSUM analysis (Table 1). CUSUM analysis was sequentially applied to all surgical cases for 2 attending surgeons. The CUSUM equation or cumulative sum used for this series of observations was as follows: Sn = (X0jXi)1 + (X0jXi)2 + (X0jXi)n.16 The CUSUM equation variables used were the following: -X0, target value set for the desired level of performance (complication rate of G10%, or, alternatively, 9 of 10 cases without complications for a numerical value of 0.9); -Xi, actual value assigned to each operative case performed with Xi = 1 for success (no complication) and Xi = 0 for failure (intraoperative complication); -Sn, cumulative sum of (X0 j Xi)n for the series of performances, in this setting, operative cases. The target value of success was set at less than 10% complications, based on previously reported intraoperative complication rates for robotic, laparoscopic, and abdominal sacrocolpexy.17Y19 The rates of intraoperative complications such as conversion to laparotomy,
&
Volume 20, Number 2, March/April 2014
transfusion, cystotomy, enterotomy, ureteral injury, pulmonary embolus were found to be cumulatively less than but approaching 10%. Applying the previously mentioned equation, a sequence of 4 RSCPs including a cystotomy with the first, no complications in cases 2 and 3, and a bowel injury in the fourth case would be as follows: Sn ¼ ð0:9 0Þ þ ð0:9 1Þ þ ð0:9 0Þ ¼ ð0:9Þ þ ð0:1Þ þ ð0:9Þ ¼ 1:6
Cumulative sums for each procedure performed are plotted with each sequential procedure on the x axis and cumulative sum value on the y axis.
RESULTS Over 27 months, 169 RSCPs were performed. The first surgeon has been performing RSCPs for 7 years, whereas the second surgeon has been performing RSCPs for 5 years. During the study period, the first surgeon performed 107 RSCPs and the second surgeon performed 62 RSCPs with 8 (7.4%) and 3 (4.8%) intraoperative complications, respectively. Total complications related to RSCP included 7 (4.1%) cystotomies, 2 (1.2%) vaginal lacerations, 1 (0.6%) blood transfusion, and 1 (0.6%) bowel perforation. There were 6 (3.6%) bladder injuries due to trocar perforation at the time of midurethral sling placement. Although mean body mass index of patients undergoing surgery in this series did not differ based on operating surgeon (31 [10.3] vs 30.2 [9.7], P = 0.62), there was an observed difference in estimated blood loss (91 [90.7] vs 58 [57.5], P = 0.012). CUSUM graphs were created for each surgeon (Figs. 1 and 2). The upward peaks correlate with the complications related to RSCP observed for each surgeon. Surgeon 2 had 1 complication at the beginning of the series, but sequential successes brought the plotted graph below the alert line. Both graphs display a downward slope within the acceptable performance zone, demonstrating overall maintenance of proficiency over time.
DISCUSSION Traditionally, surgical proficiency has been described in relation to the number of cases performed and or operative time. Reported performance measures specifically for RSCP in the literature include procedure number and operative time but these can be misleading secondary to concomitant procedures, surgeon experience, complexity of surgical cases, and involvement of trainees. Akl et al described operative time as a proficiency measure in a case series of RSCP. In this study, mean
TABLE 1. Concomitant Surgeries in Addition to Sacrocolpopexy
Robotic-assisted hysterectomy Total hysterectomy Supracervical hysterectomy Bilateral salpingo-oophorectomy Midurethral sling Posterior repair Lysis of adhesions Perineoplasty Anterior repair Rectopexy Cystoscopy
Surgeon 1 (n = 107 RSCP)
Surgeon 2 (n = 62 RSCP)
P*
58 (54.2) 58 (54.2) 0 (0) 33 (30.8) 51 (47.7) 18 (16.8) 12 (11.2) 4 (3.7) 1 (0.9) 0 (0) 55 (51.4)
28 (45.2) 3 (4.8) 25 (40.3) 7 (11.3) 27 (43.5) 11 (17.7) 5 (8.1) 31 (50) 2 (3.2) 3 (4.8) 41 (66.1)
0.257 G0.001 G0.001 0.004 0.605 0.879 0.512 G0.001 0.555 0.048 0.062
Data presented as n (%). *W2 or Fisher exact test.
84
www.fpmrs.net
* 2014 Lippincott Williams & Wilkins
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 20, Number 2, March/April 2014
RSCP Performance and CUSUM Analysis
FIGURE 1. CUSUM for surgeon 1. Total number of RSCP procedures, 107. Total number of complications, 8.
operative time was 197.9 (66.8) minutes, which decreased by 25% after the first 10 cases, with the last 30 cases having a mean of 167.3 minutes.18 Geller et al reported a retrospective case series of 147 patients in which operative time was used to determine that 10 cases were needed to find a significant decrease in time needed for critical steps of the procedure and overall surgical time. The reported total median time for RSCP was 83 (33Y179) minutes. When the first 10 cases were compared to the remaining cases, the total RSCP time decreased from 129 (90Y179) to 80 (30Y148) minutes (P e 0.001).20 According to Dagash et al,2 the absolute number of procedures performed is generally not useful in defining surgical proficiency. As such, CUSUM analysis has been successfully used to understand and describe individual learning curves for various procedures other than sacrocolpopexy. Komatsu et al used CUSUM analysis to describe the learning curves for 15 interns performing orotracheal intubation; 9 interns achieved proficiency after 26 (range, 15Y42) cases. However, the remaining 6 interns performed 28 to 53 intubations, but did not achieve proficiency.12 Dessolle et al used CUSUM analysis to describe proficiency of trainees performing embryo transfer.
Although prior data had established 50 cases as needed to achieve proficiency,21 Dessolle et al22 found that the number of cases need to achieve proficiency varied by learner, with a range of 11 to 99 cases. Similarly, Woelk et al14 determined that 91 robotic hysterectomies were required to become proficient, based on CUSUM analysis of intraoperative complications. CUSUM analysis is a well-described quality control tool that has been successfully used to describe procedural learning curves for individual learners.13,14,22 Once proficiency is established, CUSUM can be used to monitor performance over time. If surgical complication rates are low, the graph will display a downward slope and performance will be deemed acceptable. However, if complication rates are high, the slope of the graph will trend in an upward direction.9 If this upward slope crosses the alert line, providers should investigate potential reasons for such performance. In this study, CUSUM was used to monitor maintenance of proficiency rather than to describe a learning curve. This is in contrast to the study by Claerhout et al11 that described an adequate learning curve for laparoscopic sacrocolpopexy after 60 cases. Although operative complications were observed in this study, the CUSUM graphing demonstrates that they were
FIGURE 2. CUSUM for surgeon 2. Total number of RSCP procedures, 62. Total number of complications, 3. * 2014 Lippincott Williams & Wilkins
www.fpmrs.net
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
85
Female Pelvic Medicine & Reconstructive Surgery
Myers et al
outweighed by the number of ‘‘successful’’ outcomes. As such, surgeon performance did not cross the ‘‘alert line.’’ When using the CUSUM equation, it is important to note that once a surgeon performs a series of sequentially successful procedures free from complication (‘‘successful outcome’’), the CUSUM graph line may lay so low in the ‘‘acceptable performance zone’’ of the graph as to require multiple adverse outcomes to bring it back up to the ‘‘alert line.’’ Therefore, as performance improves and monitoring continues over time, the complications or adverse outcomes could be redefined to create more stringent definitions of acceptable performance. Alternatively, it may be more reasonable to use this tool for a predetermined amount of time each year, for example, an annual 2-month assessment, to prevent cumulative successes from skewing the data. In addition, the CUSUM graph should not be inappropriately analyzed or used to assign blame. Rather, its designed use is as a quality assurance tool to allow providers to understand when performance is and is not in an acceptable zone. If the performance is not acceptable, it is important for providers to evaluate all possible contributing factors to the poor outcomes observed including surgical technique, systems error, learner involvement, surgical pathology, and patient populations.9,23 A strength of this study includes the use of patient-centered outcomes in monitoring surgeon proficiency. All intraoperative complications were confirmed by review of each operative report and discharge summary. We have demonstrated that CUSUM analysis can be successfully used to monitor proficiency for RSCP in a way that is clinically relevant. Another study strength was the application of CUSUM analysis to a large number of surgical cases over several years, ensuring a thorough analysis of performance. The patient population studied was from 2008 to 2011 and does not include the same patient population studied in Geller et al.20 This study provides a model for monitoring surgical proficiency over time that can be followed by both the individual surgeon and the system in which s/he works. Although the surgical approach studied was RSCP, this model can also be applied to other routes of colpopexy or other surgical procedures to study the respective performance for individual surgeons. We do not intend the article to imply that 1 surgical procedure may be ‘‘safer’’ than another, but instead it should describe a method to measure a surgeon’s performance regardless of the procedure. A limitation of this study is that intraoperative RSCP complications were monitored for only 2 surgeons. Another limitation of this work was that a risk-adjusted CUSUM was not performed and all intraoperative complications were considered equally. Lastly, although concomitant procedures were performed, including robotic hysterectomy and midurethral slings, complications from these procedures were not analyzed in an effort to focus solely on RSCP proficiency. Future work could include analyses of these variables. CUSUM analysis was successfully applied to monitor RSCP proficiency over time. This statistical method was easily applied to our clinical practice and could be adopted by surgeons to use as a self-assessment tool.
&
Volume 20, Number 2, March/April 2014
3. Warf BC, Donnelly MB, Schwartz RW, et al. Interpreting the judgment of surgical faculty regarding resident competence. J Surg Res 1999; 86:29Y35. 4. American Board of Medical Specialties. Available at: http://www.abms. org/maintenance_of_certification/abms_moc.aspx. Accessed May 13, 2013. 5. Iglehart JK, Baron RB. Ensuring physicians’ competenceVis maintenance of certification the answer? N Engl J Med 2012;367: 2543Y2549. 6. American Board of Obstetrics and Gynecology. Available at: http://abog.org/valueofmoc.asp. Accessed May 13, 2013. 7. Lowe MM, Aparicio A, Galbraith R, et al. The future of continuing medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009;135(Suppl 3):69SY75S. 8. Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094Y1102. 9. Rogers CA, Reeves BC, Caputo M, et al. Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 2004;128:811Y819. 10. Siegmund D. Sequential Analysis, Tests, and Confidence Intervals. New York, NY: Springer; 1985. 11. Claerhout F, Roovers JP, Lewi P, et al. Implementation of laparoscopic sacrocolpopexyVa single centre’s experience. Int Urogynecol J Pelvic Floor Dysfunct 2009;20:1119Y1125. 12. Komatsu R, Kasuya Y, Yogo H, et al. Learning curves for bag-and-mask ventilation and orotracheal intubation: an application of the cumulative sum method. Anesthesiology 2010;112:1525Y1531. 13. Papanna R, Biau DJ, Mann LK, et al. Use of the Learning Curve-Cumulative Summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: fetoscopic laser ablation as a model. Am J Obstet Gynecol 2011;204:218.e1Y218.e9. 14. Woelk JL, Casiano ER, Weaver AL, et al. The learning curve of robotic hysterectomy. Obstet Gynecol 2013;121:87Y95. 15. Parnell BA, Matthews CA. Robotic-assisted techniques and outcomes in the realm of pelvic reconstructive surgery. Clin Obstet Gynecol 2011;54:412Y419. 16. Van Rij AM, McDonald JR, Pettigrew RA, et al. CUSUM as an aid to early assessment of the surgical trainee. Br J Surg 1995;82:1500Y1503. 17. Nygaard IE, McCreery R, Brubaker L, et al. Abdominal sacrocolpopexy: a comprehensive review. Obstet Gynecol 2004;104: 805Y823. 18. Akl MN, Long JB, Giles DL, et al. Robotic-assisted sacrocolpopexy: technique and learning curve. Surg Endosc 2009;23:2390Y2394. 19. Mustafa S, Amit A, Filmar S, et al. Implementation of laparoscopic sacrocolpopexy: establishment of a learning curve and short-term outcomes. Arch Gynecol Obstet 2012;286:983Y988. 20. Geller EJ, Lin FC, Matthews CA. Analysis of robotic performance times to improve operative efficiency. J Minim Invasive Gynecol 2013;20: 43Y48.
REFERENCES
21. Papageorgiou TC, Hearns-Stokes RM, et al. Training of providers in embryo transfer: what is the minimum number of transfers required for proficiency? Hum Reprod 2001;16:1415Y1419.
1. Schreuder HW, Wolswijk R, Zweemer RP, et al. Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 2012;119:137Y149.
22. Dessolle L, Freour T, Barriere P, et al. How soon can I be proficient in embryo transfer? Lessons from the Cumulative Summation test for learning curve (LC-CUSUM). Hum Reprod 2010;25:380Y386.
2. Dagash H, Chowdhury M, Pierro A. When can I be proficient in laparoscopic surgery? A systematic review of the evidence. J Pediatr Surg 2003;38:720Y724.
23. Williams AK, Chalasani V, Martinez CH, et al. Cumulative Summation graphs are a useful tool for monitoring positive surgical margin rates in robot-assisted radical prostatectomy. BJU Int 2011;107:1648Y1652.
86
www.fpmrs.net
* 2014 Lippincott Williams & Wilkins
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Robotic Sacrocolpopexy Performance and Cumulative Summation Analysis Erinn M. Myers, MD,* Elizabeth J. Geller, MD,* AnnaMarie Connolly, MD,* James Michael Bowling, PhD,Þ and Catherine A. Matthews, MD*
Objectives: This study aimed to apply Cumulative Summation (CUSUM) analysis as a tool to monitor robotic sacrocolpopexy (RSCP) proficiency over time. Methods: A retrospective analysis of all women who underwent RSCP between September of 2008 and December of 2011 at the University of North Carolina at Chapel Hill. The performance for 2 attending surgeons was analyzed sequentially over time. Intraoperative complications such as genitourinary or gastrointestinal tract injury, conversion to laparotomy, pulmonary embolus, hemorrhage, and blood transfusion, were identified by International Classification of Diseases, Ninth Revision and Current Procedural Terminology codes. A successful outcome was defined as no intraoperative complications. The target value of success was set at less than 10% complications. CUSUM analysis was then sequentially applied to all RSCP cases for 2 attending surgeons. Results: Over 27 months, 169 RSCPs were performed. The first surgeon performed 107 RSCPs and the second surgeon performed 62 RSCPs with 8 (7.4%) and 3 (4.9%) intraoperative complications, respectively. Total complications included 7 (4.1%) cystotomies, 2 (1.2%) vaginal lacerations, 1 (0.6%) blood transfusion, and 1 (0.6%) bowel perforation. A CUSUM graph was created for each surgeon. Conclusions: CUSUM analysis was successfully applied to monitor RSCP proficiency. Such testing of individual successive procedural outcomes with CUSUM may offer an objective tool to aid in physician self-assessment. Key Words: Cumulative Summation (CUSUM), robotic sacrocolpopexy, surgical proficiency (Female Pelvic Med Reconstr Surg 2014;20: 83Y86)
E
stablishing surgical competence and monitoring surgeon performance are essential to patient safety, but can prove challenging. Surrogate markers of competence have included setting a predetermined number of cases, time taken for task completion, and subjective assessment by a mentor.1 However, these criteria cannot be easily standardized. All surgeons may not achieve adequate performance with a fixed sample size. Operative time can vary depending on the complexity of the individual patient’s anatomy and the procedure performed. Finally, assessment by a mentor is subject to bias and may not be consistently reproducible.1Y3 Regardless of which evaluation method is selected to establish proficiency, continued
From the *Department of Obstetrics and Gynecology, and †Gillings School of International Public Health, University of North Carolina, Chapel Hill, NC. Reprints: Erinn M. Myers, MD, Department of Obstetrics and Gynecology, University of North Carolina, 3023 Old Clinic Bldg, Campus Box 7570, Chapel Hill, NC 27599. E-mail: [email protected]. Dr Catherine A. Matthews is a consultant of Intuitive Surgical and has received research support from the American Medical Systems. Dr Elizabeth J. Geller has received honoraria from Intuitive Surgical. The other authors have declared they have no conflicts of interest. Copyright * 2014 by Lippincott Williams & Wilkins DOI: 10.1097/SPV.0000000000000044
Female Pelvic Medicine & Reconstructive Surgery
&
evaluation of performance over time is necessary to assure that performance goals and patient safety are maintained. Throughout their careers, physicians are similarly required to evaluate their clinical funds of knowledge and skills over time by participating in specialty-specific, continuous professional development, and educational programs known as maintenance of certification (MOC). These programs are endorsed by the American Board of Medical Specialties,4,5 which outlines that MOC is a 4-part process insuring that physicians (1) obtain professional standing with a valid and unrestricted medical license, (2) commit to lifelong learning and periodic self-assessment, (3) demonstrate cognitive expertise by completing written examinations, and (4) exemplify quality performance in practice.4,5 In compliance with these guidelines, the American Board of Obstetrics and Gynecology requires that all certified obstetrician-gynecologists enroll in the annual MOC process that addresses all 4 components during a 6-year cycle.6 Self-assessment, part 2, is of particular interest as it allows physicians to assess and improve their personal performance over time.7 Although self-assessment is part of MOC, it can be subject to individual bias with weak to no associations found between physicians’ self-assessment and external evaluation.8 Cumulative Summation (CUSUM) is a mathematical model that provides surgeons with a real-time, self-assessment tool to examine individual outcomes after each procedure.9 CUSUM was originally described during World War II and was developed to establish quality control in munitions production lines.10 Sequential testing is instituted to ensure that successive products or outcomes are within predetermined boundary limits. When this is applied to surgical outcomes, desired performance goals are defined and then observed performance is measured and compared to the desired performance goal using the CUSUM equation. The CUSUM graph produced from the data will display any observed outcomes that fall outside the acceptable range, thus providing a visual graph to monitor performance. The equation can be recalculated after each procedure for realtime performance monitoring for individual surgeons.9 CUSUM analysis has previously been reported in the literature to describe the learning curve for procedural skills such as endotracheal intubation, fetoscopic laser ablation, robotic hysterectomy, and laparoscopic sacrocolpopexy.11Y14 Our objective was to apply CUSUM analysis to a single procedure, robotic sacrocolpopexy (RSCP), to monitor maintenance of surgical skill proficiency.
MATERIALS AND METHODS Institutional review board approval was obtained at the University of North Carolina at Chapel Hill for this analysis. All women who underwent RSCP between September 2008 and December 2011 were identified by the Current Procedural Terminology code 57425. All cases performed by the 2 attending surgeons were included. Intraoperative complications, defined by the International Classification of Diseases, Ninth
Volume 20, Number 2, March/April 2014
www.fpmrs.net
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
83
Myers et al
Female Pelvic Medicine & Reconstructive Surgery
Revision and Current Procedural Terminology codes included genitourinary or gastrointestinal tract injury, conversion to laparotomy, pulmonary embolus, hemorrhage, and blood transfusion. These complications were confirmed by cross-referencing the Operating Room Robotic Database, Medical Information Management Database, Transfusion Database, and each operative note and discharge summary in the electronic medical record. Demographic data such as medical and surgical history and preoperative and postoperative pelvic organ prolapse quantification examinations were not collected for this analysis. We have previously described the general surgical steps that we use for RSCP, including port placement, robotic docking, and specific instrumentation.15 In general, the anterior and posterior vaginal dissections were extended down to the bladder trigone and the perineal body, respectively. During the study period, both surgeons used IntePro Y-mesh (American Medical Systems, Inc, Minnetonka, Minn) and attached the mesh using 6 to 8 stitches on the anterior and posterior vaginal walls and 2 to 3 sutures in the anterior longitudinal sacral ligament. Surgeon 1 used 2-0 Ethibond (Ethicon, Somerville, NJ) and surgeon 2 used 2-0 Gore-Tex (W.L. Gore and Associates, Flagstaff, Ariz) to secure the mesh to the vagina and anterior longitudinal ligament. Surgeon 1 incised the peritoneum along the right pelvic sidewall from the sacrum to the cul-de-sac and then after mesh placement reperitonealized the mesh. Surgeon 2 created a peritoneal tunnel for the mesh. Concomitant procedures and their associated complications were recorded but not included in the CUSUM analysis (Table 1). CUSUM analysis was sequentially applied to all surgical cases for 2 attending surgeons. The CUSUM equation or cumulative sum used for this series of observations was as follows: Sn = (X0jXi)1 + (X0jXi)2 + (X0jXi)n.16 The CUSUM equation variables used were the following: -X0, target value set for the desired level of performance (complication rate of G10%, or, alternatively, 9 of 10 cases without complications for a numerical value of 0.9); -Xi, actual value assigned to each operative case performed with Xi = 1 for success (no complication) and Xi = 0 for failure (intraoperative complication); -Sn, cumulative sum of (X0 j Xi)n for the series of performances, in this setting, operative cases. The target value of success was set at less than 10% complications, based on previously reported intraoperative complication rates for robotic, laparoscopic, and abdominal sacrocolpexy.17Y19 The rates of intraoperative complications such as conversion to laparotomy,
&
Volume 20, Number 2, March/April 2014
transfusion, cystotomy, enterotomy, ureteral injury, pulmonary embolus were found to be cumulatively less than but approaching 10%. Applying the previously mentioned equation, a sequence of 4 RSCPs including a cystotomy with the first, no complications in cases 2 and 3, and a bowel injury in the fourth case would be as follows: Sn ¼ ð0:9 0Þ þ ð0:9 1Þ þ ð0:9 0Þ ¼ ð0:9Þ þ ð0:1Þ þ ð0:9Þ ¼ 1:6
Cumulative sums for each procedure performed are plotted with each sequential procedure on the x axis and cumulative sum value on the y axis.
RESULTS Over 27 months, 169 RSCPs were performed. The first surgeon has been performing RSCPs for 7 years, whereas the second surgeon has been performing RSCPs for 5 years. During the study period, the first surgeon performed 107 RSCPs and the second surgeon performed 62 RSCPs with 8 (7.4%) and 3 (4.8%) intraoperative complications, respectively. Total complications related to RSCP included 7 (4.1%) cystotomies, 2 (1.2%) vaginal lacerations, 1 (0.6%) blood transfusion, and 1 (0.6%) bowel perforation. There were 6 (3.6%) bladder injuries due to trocar perforation at the time of midurethral sling placement. Although mean body mass index of patients undergoing surgery in this series did not differ based on operating surgeon (31 [10.3] vs 30.2 [9.7], P = 0.62), there was an observed difference in estimated blood loss (91 [90.7] vs 58 [57.5], P = 0.012). CUSUM graphs were created for each surgeon (Figs. 1 and 2). The upward peaks correlate with the complications related to RSCP observed for each surgeon. Surgeon 2 had 1 complication at the beginning of the series, but sequential successes brought the plotted graph below the alert line. Both graphs display a downward slope within the acceptable performance zone, demonstrating overall maintenance of proficiency over time.
DISCUSSION Traditionally, surgical proficiency has been described in relation to the number of cases performed and or operative time. Reported performance measures specifically for RSCP in the literature include procedure number and operative time but these can be misleading secondary to concomitant procedures, surgeon experience, complexity of surgical cases, and involvement of trainees. Akl et al described operative time as a proficiency measure in a case series of RSCP. In this study, mean
TABLE 1. Concomitant Surgeries in Addition to Sacrocolpopexy
Robotic-assisted hysterectomy Total hysterectomy Supracervical hysterectomy Bilateral salpingo-oophorectomy Midurethral sling Posterior repair Lysis of adhesions Perineoplasty Anterior repair Rectopexy Cystoscopy
Surgeon 1 (n = 107 RSCP)
Surgeon 2 (n = 62 RSCP)
P*
58 (54.2) 58 (54.2) 0 (0) 33 (30.8) 51 (47.7) 18 (16.8) 12 (11.2) 4 (3.7) 1 (0.9) 0 (0) 55 (51.4)
28 (45.2) 3 (4.8) 25 (40.3) 7 (11.3) 27 (43.5) 11 (17.7) 5 (8.1) 31 (50) 2 (3.2) 3 (4.8) 41 (66.1)
0.257 G0.001 G0.001 0.004 0.605 0.879 0.512 G0.001 0.555 0.048 0.062
Data presented as n (%). *W2 or Fisher exact test.
84
www.fpmrs.net
* 2014 Lippincott Williams & Wilkins
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 20, Number 2, March/April 2014
RSCP Performance and CUSUM Analysis
FIGURE 1. CUSUM for surgeon 1. Total number of RSCP procedures, 107. Total number of complications, 8.
operative time was 197.9 (66.8) minutes, which decreased by 25% after the first 10 cases, with the last 30 cases having a mean of 167.3 minutes.18 Geller et al reported a retrospective case series of 147 patients in which operative time was used to determine that 10 cases were needed to find a significant decrease in time needed for critical steps of the procedure and overall surgical time. The reported total median time for RSCP was 83 (33Y179) minutes. When the first 10 cases were compared to the remaining cases, the total RSCP time decreased from 129 (90Y179) to 80 (30Y148) minutes (P e 0.001).20 According to Dagash et al,2 the absolute number of procedures performed is generally not useful in defining surgical proficiency. As such, CUSUM analysis has been successfully used to understand and describe individual learning curves for various procedures other than sacrocolpopexy. Komatsu et al used CUSUM analysis to describe the learning curves for 15 interns performing orotracheal intubation; 9 interns achieved proficiency after 26 (range, 15Y42) cases. However, the remaining 6 interns performed 28 to 53 intubations, but did not achieve proficiency.12 Dessolle et al used CUSUM analysis to describe proficiency of trainees performing embryo transfer.
Although prior data had established 50 cases as needed to achieve proficiency,21 Dessolle et al22 found that the number of cases need to achieve proficiency varied by learner, with a range of 11 to 99 cases. Similarly, Woelk et al14 determined that 91 robotic hysterectomies were required to become proficient, based on CUSUM analysis of intraoperative complications. CUSUM analysis is a well-described quality control tool that has been successfully used to describe procedural learning curves for individual learners.13,14,22 Once proficiency is established, CUSUM can be used to monitor performance over time. If surgical complication rates are low, the graph will display a downward slope and performance will be deemed acceptable. However, if complication rates are high, the slope of the graph will trend in an upward direction.9 If this upward slope crosses the alert line, providers should investigate potential reasons for such performance. In this study, CUSUM was used to monitor maintenance of proficiency rather than to describe a learning curve. This is in contrast to the study by Claerhout et al11 that described an adequate learning curve for laparoscopic sacrocolpopexy after 60 cases. Although operative complications were observed in this study, the CUSUM graphing demonstrates that they were
FIGURE 2. CUSUM for surgeon 2. Total number of RSCP procedures, 62. Total number of complications, 3. * 2014 Lippincott Williams & Wilkins
www.fpmrs.net
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
85
Female Pelvic Medicine & Reconstructive Surgery
Myers et al
outweighed by the number of ‘‘successful’’ outcomes. As such, surgeon performance did not cross the ‘‘alert line.’’ When using the CUSUM equation, it is important to note that once a surgeon performs a series of sequentially successful procedures free from complication (‘‘successful outcome’’), the CUSUM graph line may lay so low in the ‘‘acceptable performance zone’’ of the graph as to require multiple adverse outcomes to bring it back up to the ‘‘alert line.’’ Therefore, as performance improves and monitoring continues over time, the complications or adverse outcomes could be redefined to create more stringent definitions of acceptable performance. Alternatively, it may be more reasonable to use this tool for a predetermined amount of time each year, for example, an annual 2-month assessment, to prevent cumulative successes from skewing the data. In addition, the CUSUM graph should not be inappropriately analyzed or used to assign blame. Rather, its designed use is as a quality assurance tool to allow providers to understand when performance is and is not in an acceptable zone. If the performance is not acceptable, it is important for providers to evaluate all possible contributing factors to the poor outcomes observed including surgical technique, systems error, learner involvement, surgical pathology, and patient populations.9,23 A strength of this study includes the use of patient-centered outcomes in monitoring surgeon proficiency. All intraoperative complications were confirmed by review of each operative report and discharge summary. We have demonstrated that CUSUM analysis can be successfully used to monitor proficiency for RSCP in a way that is clinically relevant. Another study strength was the application of CUSUM analysis to a large number of surgical cases over several years, ensuring a thorough analysis of performance. The patient population studied was from 2008 to 2011 and does not include the same patient population studied in Geller et al.20 This study provides a model for monitoring surgical proficiency over time that can be followed by both the individual surgeon and the system in which s/he works. Although the surgical approach studied was RSCP, this model can also be applied to other routes of colpopexy or other surgical procedures to study the respective performance for individual surgeons. We do not intend the article to imply that 1 surgical procedure may be ‘‘safer’’ than another, but instead it should describe a method to measure a surgeon’s performance regardless of the procedure. A limitation of this study is that intraoperative RSCP complications were monitored for only 2 surgeons. Another limitation of this work was that a risk-adjusted CUSUM was not performed and all intraoperative complications were considered equally. Lastly, although concomitant procedures were performed, including robotic hysterectomy and midurethral slings, complications from these procedures were not analyzed in an effort to focus solely on RSCP proficiency. Future work could include analyses of these variables. CUSUM analysis was successfully applied to monitor RSCP proficiency over time. This statistical method was easily applied to our clinical practice and could be adopted by surgeons to use as a self-assessment tool.
&
Volume 20, Number 2, March/April 2014
3. Warf BC, Donnelly MB, Schwartz RW, et al. Interpreting the judgment of surgical faculty regarding resident competence. J Surg Res 1999; 86:29Y35. 4. American Board of Medical Specialties. Available at: http://www.abms. org/maintenance_of_certification/abms_moc.aspx. Accessed May 13, 2013. 5. Iglehart JK, Baron RB. Ensuring physicians’ competenceVis maintenance of certification the answer? N Engl J Med 2012;367: 2543Y2549. 6. American Board of Obstetrics and Gynecology. Available at: http://abog.org/valueofmoc.asp. Accessed May 13, 2013. 7. Lowe MM, Aparicio A, Galbraith R, et al. The future of continuing medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009;135(Suppl 3):69SY75S. 8. Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094Y1102. 9. Rogers CA, Reeves BC, Caputo M, et al. Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 2004;128:811Y819. 10. Siegmund D. Sequential Analysis, Tests, and Confidence Intervals. New York, NY: Springer; 1985. 11. Claerhout F, Roovers JP, Lewi P, et al. Implementation of laparoscopic sacrocolpopexyVa single centre’s experience. Int Urogynecol J Pelvic Floor Dysfunct 2009;20:1119Y1125. 12. Komatsu R, Kasuya Y, Yogo H, et al. Learning curves for bag-and-mask ventilation and orotracheal intubation: an application of the cumulative sum method. Anesthesiology 2010;112:1525Y1531. 13. Papanna R, Biau DJ, Mann LK, et al. Use of the Learning Curve-Cumulative Summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: fetoscopic laser ablation as a model. Am J Obstet Gynecol 2011;204:218.e1Y218.e9. 14. Woelk JL, Casiano ER, Weaver AL, et al. The learning curve of robotic hysterectomy. Obstet Gynecol 2013;121:87Y95. 15. Parnell BA, Matthews CA. Robotic-assisted techniques and outcomes in the realm of pelvic reconstructive surgery. Clin Obstet Gynecol 2011;54:412Y419. 16. Van Rij AM, McDonald JR, Pettigrew RA, et al. CUSUM as an aid to early assessment of the surgical trainee. Br J Surg 1995;82:1500Y1503. 17. Nygaard IE, McCreery R, Brubaker L, et al. Abdominal sacrocolpopexy: a comprehensive review. Obstet Gynecol 2004;104: 805Y823. 18. Akl MN, Long JB, Giles DL, et al. Robotic-assisted sacrocolpopexy: technique and learning curve. Surg Endosc 2009;23:2390Y2394. 19. Mustafa S, Amit A, Filmar S, et al. Implementation of laparoscopic sacrocolpopexy: establishment of a learning curve and short-term outcomes. Arch Gynecol Obstet 2012;286:983Y988. 20. Geller EJ, Lin FC, Matthews CA. Analysis of robotic performance times to improve operative efficiency. J Minim Invasive Gynecol 2013;20: 43Y48.
REFERENCES
21. Papageorgiou TC, Hearns-Stokes RM, et al. Training of providers in embryo transfer: what is the minimum number of transfers required for proficiency? Hum Reprod 2001;16:1415Y1419.
1. Schreuder HW, Wolswijk R, Zweemer RP, et al. Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 2012;119:137Y149.
22. Dessolle L, Freour T, Barriere P, et al. How soon can I be proficient in embryo transfer? Lessons from the Cumulative Summation test for learning curve (LC-CUSUM). Hum Reprod 2010;25:380Y386.
2. Dagash H, Chowdhury M, Pierro A. When can I be proficient in laparoscopic surgery? A systematic review of the evidence. J Pediatr Surg 2003;38:720Y724.
23. Williams AK, Chalasani V, Martinez CH, et al. Cumulative Summation graphs are a useful tool for monitoring positive surgical margin rates in robot-assisted radical prostatectomy. BJU Int 2011;107:1648Y1652.
86
www.fpmrs.net
* 2014 Lippincott Williams & Wilkins
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
