ORIGINAL ARTICLE
Development and Validation of a Ureteral Anastomosis Simulation Model for Surgical Training Elena Tunitsky, MD,* Alana Murphy, MD,Þ Matthew D. Barber, MD, MHS,* Matthew Simmons, MD, PhD,Þ and J. Eric Jelovsek, MD, MMEd*
Objective: To develop and validate a new ureteral anastomosis simulation model.
Methods: We designed a training model to simulate the task of ureteral anastomosis required for ureteroneocystostomy that is suitable for robotic and laparoscopic approaches. Face validity was measured using questions related to surgical authenticity and educational value of the model. Construct validity was measured by comparing scores using Global Operative Assessment of Laparoscopic Skills Scale (GOALS) scale between ‘‘procedure experts,’’ ‘‘robotic experts,’’ and ‘‘trainees’’ groups. One-way analysis of variance was used to compare differences in the scores and operating times between the 3 groups. Associations between previous surgical experience and performance scores were measured using the Spearman rho correlation coefficient. Results: Four urologists experienced with robotically assisted ureteroneocystostomies were included in the procedure experts group. The robotic experts group consisted of 5 gynecologists experienced in robotic surgery. The trainees group consisted of 12 urology and gynecology upper-level residents and fellows. All experts agreed or strongly agreed that the model was authentic to the live procedure and a useful training tool. Mean (SD) total GOALS scores were significantly better for the procedure experts group compared to the robotic experts group and to the trainees group (P=0.02 vs P=0.004, respectively). The robotic experts group’s GOALS scores were also significantly higher than that of the trainees group (P=0.05). There were no differences in mean times required to complete the procedure. Surgical experience moderately correlated with scores on all 3 assessment scales. Conclusions: Superior performance on the model by more experienced surgeons demonstrates evidence of construct validity. This authentic and useful model allows surgeons to learn and practice the ureteral anastomosis portion of the ureteral reimplantation surgeries before operating on a live patient. Key Words: ureteral reimplantation, ureteral anastomosis, robotics, simulation model, education (Female Pelvic Med Reconstr Surg 2013;19: 346Y351)
U
reteral reimplantation using a minimally invasive approach is becoming more prevalent. Laparoscopic and robotically assisted approaches have been described for various ureteral reimplantation techniques, and functional outcomes seem to be comparable to those of open procedures.1Y3 Indications for a ureteral reimplantation with or without a Boari flap or Psoas hitch From the *Urogynecology and Reconstructive Pelvic Surgery, Obstetrics, Gynecology and Women’s Health Institute, and †Glickman Urologic Institute, Cleveland Clinic, Cleveland, OH. Reprints: Elena Tunitsky, MD, Obstetrics, Gynecology and Women’s Health Institute, Cleveland Clinic 9500 Euclid Ave, Cleveland, OH 44195. E-mail: [email protected]. The authors have declared they have no conflicts of interest. Supplemental digital contents are available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.fpmrs.net). Copyright * 2013 by Lippincott Williams & Wilkins DOI: 10.1097/SPV.0b013e3182a331bf
346
www.fpmrs.net
in the adult patient are 3-fold: ureteral stricture, distal ureteral malignancy, and iatrogenic ureteral injury. Most iatrogenic ureteral injuries occur at the time of gynecologic surgery, with an estimated rate of 1 to 13 per 1000 surgeries.4 Despite a variety of indications for ureteral reimplantation, the procedure is uncommon. The infrequent nature of ureteral reimplantation is a significant barrier for urology and urogynecology trainees to obtaining the proficient skills necessary for performing these procedures using a minimally invasive approach. In general, complex reconstructive procedures using minimally invasive techniques require an advanced psychomotor skill set learned through repetitive practice. Teaching these skills to surgical trainees is becoming more challenging in the current educational environment of shorter workweeks, emphasis on operating room efficiency, and cost containment. Owing to these constraints, simulation-based training is taking on a vital role as a teaching and assessment tool in surgical education. A recent meta-analysis by Cook et al5 compared technologyenhanced simulation training to no intervention and demonstrated that simulation training was consistently associated with improved outcomes in knowledge, skills, and behaviors. This review concluded that there is no longer a need to compare simulation-based learning to traditional nonsimulation-based learning and that future studies should continue to develop simulators specific to their specialty and learn how to efficiently use them in the clinical setting. The objective of this study was to create a novel costefficient surgical model for learning and practicing the task of robotic-assisted ureteral anastomosis required for ureteral reimplantation surgeries and present evidence regarding its validity as a training and assessment tool.
MATERIALS AND METHODS Institutional review board approval was received. This was a cross-sectional study design. The study proceeded in 3 phases: (1) model construction, (2) validity testing, and (3) setting standards for a passing performance when using the model as an assessment tool.
Model Construction The model design is demonstrated in Figure 1. Construction of the model included drilling 2 small holes in 2 of the corners of the plastic container (15 11 3 inches). Twine was then threaded through the holes and alligator clips were then attached to the twine using electric tape. A nontoxic waterbased hydrogel material (LifeLike BioTissue, London, Ontario, Canada) that can be easily stored in a container with water at a room temperature was used to replicate the visual and tactile sensation of human ureter and bladder. These materials can be stored indefinitely and can be reused for simulation approximately 10 times. The simulated ureter measured 0.5-mm wall thickness, 6 mm in diameter, and 15 cm in length. A simulated bladder was created using a rectangular 12 15-cm piece of hydrogel with a thickness of 4.40 mm. Using scissors, an
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
FIGURE 1. Simulation training model for ureteral reimplantation. A, Storage container 15 11 3 inches (approximately US $5); large bag clip (approximately US $3) attached with Velcro adhesive tape (approximately US $4) (B); alligator clips 2 (approximately US $3) (C); twine (approximately US $5) (D); ureteral 6-F JJ stent (E). **The cost does not include ureteral stent and suture.
opening of approximately 1 cm in diameter was made in the ‘‘bladder’’ sheet to simulate a cystotomy and a 6-F double-J ureteral stent was inserted between the ‘‘ureter’’ and the ‘‘bladder.’’ The stent and the bladder were secured using the bag clip. The ureter with the stent were then secured with an alligator clip and adjusted to an appropriate tension using the twine. Once the initial model was constructed, simulation materials of varied dimensions were demonstrated to the expert urologists; and the materials were selected based on their consensus. Recommended modifications were that the type and length of suture were standardized including the following: 4.0 polyglactin, dyed suture on RB-1 needle 6 inches in length. In addition, the decision was made to apply mineral oil at the suture sites to mimic the aqueous environment naturally present in the pelvis and to decrease the stiffness of the materials.
Model Acceptability and Validity The model was developed to be used for teaching and assessment. When using a simulation model for assessment, it is critical to measure the model’s psychometric properties. These important aspects of any educational assessment tool are summarized by the contemporary framework proposed by van der Vleuten and Shuwirth’s utility equation: Utility = Validity Reliability Acceptability Educational impact Costeffectiveness.6,7 This framework was used in this study to ensure that important aspects of assessment were considered in model development. Eligible participants approached included staff and trainees in urology and obstetrics and gynecology at a large referral tertiary care center. Inclusion criteria were * 2013 Lippincott Williams & Wilkins
Simulation Model for Ureteral Anastomosis
participants who were familiar with the robotic console and who perform laparoscopic or robotically assisted procedures that require suturing as part of their training or practice. The participants were categorized into 3 groups: ‘‘procedure experts,’’ ‘‘robotic experts,’’ and ‘‘trainees.’’ The procedure experts group consisted of faculty in urology who performed at least 10 robotically assisted ureteral reimplantation procedures. The robotic experts group included fellowship-trained gynecologic surgeons with a high annual volume of robotically assisted surgeries including hysterectomies, myomectomies, and sacrocolpopexies but did not perform robotically assisted ureteral reimplantation procedures as part of their practice. The trainees group included all urology residents in their fourth year of training as well as urology and urogynecology fellows. All experts were approached individually for participation, whereas trainees were solicited via e-mail from their program directors and were offered a US $10 coffee gift card for their participation. The study was exempt by the institutional review board as an educational project. Per institutional review board instructions, in lieu of the informed consent, the participants were given a detailed information sheet describing the study that did not require a signature, All participants completed a questionnaire on demographics and surgical experience and were asked to perform a ureteral neocystostomy using the simulated model. The procedure was briefly reviewed with each participant, and they were given a choice of suturing approach: interrupted, continuous suturing in a full circle, or continuous suturing in 2 hemicircles. The fourth robotic arm was available for retraction; alternatively, a participant could ask the instructor for assistance in retraction. An example of the procedure performed by a procedure expert and by a trainee is illustrated in Figure 2. Supplemental video illustrates the ureteral anastamosis performed on the model (Supplemental Digital Content, http://links.lww.com/FPMRS/A15). After completing the procedure, participants were asked to assess the realism, the use of the model and its perceived acceptability and educational impact by responding to a questionnaire using a 5-item Likert response scale. In addition, they were asked to estimate the appropriate cost for the model. The primary outcome of the study was total score using the valid and reliable Global Operative Assessment of Laparoscopic Skills (GOALS) scale.8 The GOALS scale assesses performance essential to laparoscopy in 5 domains: depth perception, bimanual dexterity, efficiency, tissue handling, and autonomy, with a maximum of 5 points for each domain (total maximum of 25 points). Two additional valid and reliable scales were used as secondary outcome measures of psychomotor skill including the modified Surgical Skills Index (SSI)9 and the modified Global Rating Scale of Operative Performance (GRS).10 All 3 scales were used to rate the performance of anastomotic suturing using live observation by 1 of 2 investigators (E.T. or A.M.). The investigators involved in the assessment are fellows in female pelvic and reconstructive surgery in the departments of gynecology and urology, respectively, with adequate experience in laparoscopic and robotic surgery to use validated assessment scales. An additional secondary outcome included total time of performance.
Setting the Pass Mark The distributions of scores were plotted using the total GOALS scale scores, and the Contrasting Groups method of standard setting was applied.11,12 For the purpose of this portion of the analysis, the procedure experts and the robotic experts were combined into an ‘‘Expert’’ group. The passing score was set at the intersection of the distribution of the 2 groups, www.fpmrs.net
347
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Tunitsky et al
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
FIGURE 2. Examples of ureteroneocystostomy performed using the model. A, Procedure performed by a trainee. B, Procedure performed by a procedure expert.
assuming false-negative and false-positive errors were of equal weight. The established passing scores were the scores that best discriminated between the groups. The Contrasting Groups method has been previously applied to assessing laparoscopy skills and is the basis for the widely used cutoff values used in the Fundamentals of Laparoscopy Skills (FLS) curriculum required of all graduating general surgery residents.13
Statistical Analysis One-way analysis of variance was used to compare differences in the scale scores and operating times between the 3 groups: procedure experts, robotic experts, and trainees. Associations between past surgical experience of participants and their performance scores were measured using the Spearman rho correlation coefficient. Significance was considered at P = 0.05. Sample size determination was informed based on previous studies using the GOALS scale.14 Assuming a common peritem standard deviation of 0.8 points and a mean per item score of 2.5 for the trainees group and mean score of 3.5 for the robotic experts group and 4.5 for the procedure experts group for each of the 5 GOALS domains, 20 participants who included
4 in the procedure experts group, 4 in the robotic experts group, and 12 in the trainees group would achieve more than 90% power to detect differences among the means of the total GOALS score with a 0.05 significance level.
RESULTS Participants’ demographics and surgical experience is described in Table 1.
Face Validity Testing Most of the participants (all experts) ‘‘agreed’’ or ‘‘strongly agreed’’ that the model is realistic and useful (Table 2). Several participants commented that the ureteral material was more fragile and less pliable, making the simulation exercise more difficult compared to live surgery. When asked about the cost of the model, most of the participants (15 [71%]) estimated the price of the model to be between US $50 and US $200 (Table 2).
Construct Validity Participants in the procedure experts group performed significantly better and received higher overall performance scores
TABLE 1. Participants’ Demographics and Surgical Experience
Number of participants Age, yrs Sex (male/female) Surgical experience beyond training, yrs Total number of open ureteroneocystostomies performed Total number of robotic/laparoscopic ureteroneocystostomies performed Number of laparoscopic surgeries requiring suturing performed annually Number of robotic surgeries requiring suturing performed annually
Procedure Experts
Robotic Experts
Trainees
4
5
12
38 (33Y50) 4/0 5 (3Y8) 23 (8Y43) 18 (13Y25) 33 (8Y50) 50 (33Y50)
38 (33Y58) 1/4 3 (1Y20) 0 (0Y3) 0 (0Y8) 38 (8Y50) 33 (18Y50)
33 (28Y45) 7/5 0 3 (0Y50) 3 (0Y18) 5 (0Y18) 0 (3Y43)
All statistics are reported as median (range).
348
www.fpmrs.net
* 2013 Lippincott Williams & Wilkins
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Simulation Model for Ureteral Anastomosis
Volume 19, Number 6, November/December 2013
TABLE 2. Responses to the Postprocedure Questionnaire 1 Strongly Disagree
2 Disagree
3 Neither Agree Nor Disagree
4 Agree
1. The simulated material closely approximates the texture and behavior of the real ureter and bladder Total (%) 1 (4.8) 5 (23.8) 13 (61.9) 2. The model setup appears appropriate for the ureteral reimplantation procedures, as it closely approximates the live surgery Total (%) 3 (14.3) 14 (66.7) 3. The training model appears useful for improving the suturing technique required for this procedure Total (%) 1 (4.8) 7 (33.3) 4. The training model appears useful for teaching and learning the procedure Total (%) 10 (47.6) 5. The training model appears useful for assessing the learner’s ability to perform this procedure before live performance Total (%) 1 (4.7) 5 (23.8) 9 (42.9) Acceptable price range of this training model (US$) G50 51Y100 101Y200 201Y300 Total (%) 3 (14.3) 10 (47.6) 5 (23.8) 3 (14.3)
than the participants in the robotic experts and trainees groups using each of the 3 scales (Table 3). The robotic experts group demonstrated significantly better performance than the trainees group on the GOALS scale but not on the modified SSI and GRS scales. When evaluating the individual items of the GOALS scale, the procedure experts group performed significantly better than the trainees group across all 5 domains. In contrast, the robotic experts group performed significantly better than the trainees group in only 3 of the 5 domains (depth perception, bimanual dexterity, and efficiency), whereas their performance was similar in the tissue handling and autonomy domains. Likewise, the robotic experts group scored significantly lower in these 2 domains than the procedure experts group but scored similarly in the remaining 3 domains (Table 3). Past surgical experience with the ureteral reimplantation procedures (open or robotically assisted) as well as annual volume of robotic procedures requiring suturing strongly or moderately correlated with better scores on all 3 assessment scales. However, annual volume of laparoscopic procedures did not significantly correlate with performance (Table 4). Additionally, median (range) time for the procedure experts group was 11.7 minutes (7.7Y14 minutes); for the robotic experts group, 13 minutes (10.8Y18 minutes); and for the trainees group, 14.1 minutes (10.2Y23 minutes); however,
5 Strongly Agree 2 (9.5) 4 (19) 13 (61.9) 11 (52.4) 6 (28.6) 9300
this was not significantly different between the groups (Table 3). An example of the completed procedure performed by a procedure expert and by a trainee is illustrated in Figure 2.
The Pass Mark Figure 3 demonstrates how the cutoff scores were determined for the GOALS scale using the Contrasting Groups approach. Overall, surgeons who achieved absolute GOALS score of 20 (80%) passed from a trainee’s level of performance to an expert’s level.
DISCUSSION Simulation training offers the advantages of repetitive practice in a safe environment without compromising patient care. This model allows trainees to focus on developing competence with the suturing task of anastomosis, one of the more complicated and time consuming aspects of the procedure. Suturing always presents a challenge to an inexperienced laparoscopic or robotic surgeon, yet it is the most repetitive and least variable task making it perfect for simulation training. Once a surgeon becomes competent at suturing using the inanimate model, they should be able to focus on the more intricate steps of the procedure such as ureterolysis, stent placement, flap creation, and overall decision making.
TABLE 3. Performance of Procedure Experts, Robotic Experts, and Trainees Using GOALS, SSI, and GRS Assessment Tools
Time, mean SD), min GOALS domains Depth perception Bimanual dexterity Efficiency Tissue handling Autonomy Total GOALS score (maximum 35) Total SSI score (maximum 28) Total GRS score (maximum 30)
Procedure Experts (PE)
Robotic Experts (RE)
Trainees (T)
P PE vs RE
P PE vs T
P RE vs T
11.25 (1.8)
12.62 (1.60)
14.73 (1.03)
0.57
0.11
0.30
5 (0.25) 5 (0.28) 4.4 (0.29) 3.2 (0.46) 3.6 (0.50) 21.2 (1.47) 23.4 (1.7) 26 (1.75)
4.1 (0.16) 3.4 (0.18) 3.1 (0.19) 2.8 (0.30) 3.5 (0.32) 17.3 (0.94) 21 (1.09) 21.3 (1.13)
1 1 0.35 0.01 0.05 0.02 0.03 0.05
0.04 0.004 0.001 0.01 0.034 0.004 0.008 0.002
0.02 0.001 0.004 0.54 0.88 0.05 0.30 0.06
5 (0.28) 5 (0.3) 4.75 (0.33) 4.75 (0.52) 5 (0.56) 24 (1.6) 28 (1.9) 30 (1.95)
All data are presented as mean (SD).
* 2013 Lippincott Williams & Wilkins
www.fpmrs.net
349
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Tunitsky et al
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
TABLE 4. Performance Using GOALS, SSI and GRS Assessment Tools by Surgical Experience GOALS
GRS
SSI
Surgical Experience
Rho
P
Rho
P
Rho
P
Total number of open ureteroneocystostomies Total number of laparoscopic or robotic ureteroneocystostomies Annual laparoscopic procedures requiring suturing Annual robotic procedures requiring suturing
0.49 0.53 0.40 0.74
0.02 0.01 0.07 0.0001
0.49 0.54 0.27 0.63
0.02 0.01 0.23 0.002
0.46 0.55 0.39 0.71
0.04 0.01 0.08 0.0003
Rho, Spearman rho correlation coefficient.
Our model involves simple design and construction, using inexpensive components that are easily obtained at a hardware store. Whereas the simulation material used in our model is commercially available, one may use other materials such as Penrose or plastic tubing. The benefit of the hydrogel material was its close resemblance in texture and size to the live ureter and bladder as reflected in the responses from participants. Overall, the participants rated the training model as authentic to the live procedure and also useful for training, and several trainees commented that they would have found it useful to practice on the model before the live surgery, lending support for the model’s usefulness and face validity. Most of the participants estimated the price of the model to be between US $50 and US $200. The actual price of the model of US $150 is in the range acceptable to the participants (US $50 for the one-time setup and US $100 for the simulated material set, which can be used approximately 10 times). We were able to demonstrate evidence of the models’ construct validity using valid and reliable assessment scales. The GOALS assessment scale was developed specifically for laparoscopic training8 and has been used in studies describing robotic models.15 The GOALS scores for performance on our model correlated with surgeon’s expertise, as designated by the group assignment. The performance also correlated with surgical experience the specific procedure and with the surgeon’s annual robotic volume. We included a group of gynecologic expert robotic surgeons who did not have experience with ureteral reimplantation procedures to elucidate if our model was specific to the ureteral reimplantation or if it solely tested only skills for robotic suturing. Whereas the robotic experts’ total GOALS scores were significantly better than those of the trainees, there was no significant difference in the domains of, tissue handling and autonomy, which supports that the model was able to discriminate between surgeons experienced with the specific procedure and those experienced in robotic surgery. This lends support to the notion that while overall robotic experience correlates with better overall performance on the model, the knowledge of ureteral surgery is as important factor that distinguishes a robotic surgeon from an expert urology robotic surgeon. This is also why we did not detect a difference in performance using the modified GRS and SSI scales.9,10 The domains of GRS and SSI scales that we tested in study address skills that reflect familiarity with the specific procedure more than a general surgical skill level (GRS: respect for tissue, time and motion, instrument handling, flow of operation, use of assistance, and knowledge of specific procedure; SSI: maintenance of visibility, use of assistants, tissue handling, knot tying/ligation, procedure completion, time and motion, flow of operation and forward planning, and knowledge of specific procedure). Not surprisingly, the robotic experts who despite having superior surgical skills were not familiar
350
www.fpmrs.net
with ureteral procedures scored similarly to the trainees using these domains. When evaluating the performance by laparoscopic volume, not surprisingly, we did not see a significant correlation. Among the participants, the gynecology experts and the trainees had high annual laparoscopic volume but were least familiar with the procedure. Thus, the lack of knowledge or experience performing the specific procedure counteracted the annual laparoscopic experience. It is possible that if our cohort included expert laparoscopic urologists who were not expert robotic surgeons, we would have been able to comment on correlation between laparoscopic skills and the performance on the model. The overall time required to complete the task was inversely proportional to the experience of the participant. This difference, however, was not significant. The trainee time varied the most and did not correlate with the educational level. In fact, some of the more experienced fellows who have scored similarly to the experts on the performance scale took the longest time to complete the task. One possible explanation for this finding is that when a simulated procedure is performed by surgeons who are familiar with ureteral surgery, they are likely focused on achieving the best possible result, one that is most authentic to the live procedure. In contrast, those unfamiliar with the surgery, not having a frame of reference, may be more concerned with completing the task faster, disregarding the final outcome. In addition, we left the choice of suturing technique up to the participants. One of the procedure experts, 4 of 5 robotic experts, and 4 of the 12 trainees (all undergoing fellowship training) elected to complete the task using
FIGURE 3. Passing mark. * 2013 Lippincott Williams & Wilkins
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
interrupted sutures. This likely increased their time, irrespective of the result. Unfortunately, we did not evaluate the final success of the anastomosis by testing for leak owing to the limitations of the model; however, we suspect that the quality of anastomosis would not always correlate with lower times. In developing and evaluating our simulation model, we followed the framework that assesses use of an educational tool by addressing its validity, reliability, acceptability, educational impact, and cost-effectiveness.6,7 Appropriate face and construct validity testing was conducted. Whereas reliability was not directly addressed, validated and reliable scales were used to assess construct validity. Although the educational impact and cost-effectiveness were addressed as part of the questionnaire assessing face validity, these parameters would be better addressed once the model is implemented into the surgical training curriculum and correlation is made between the practice using the model and improvement in live performance. We acknowledge that an important limitation of our study was that the evaluation of performance was not blinded, as we did not have recording capabilities in our training robotic tower. However, the GOALS scale has previously demonstrated good inter-rater reliability when it was initially introduced by Vassiliou et al.8 Thus, whereas masking the operating surgeon to the evaluator is a limitation, we feel that our results would have been similar if masking was possible. Furthermore, the model does not simulate all of the important aspects of ureteral reimplantation such as ureterolysis, introduction of a stent, bladder mobilization, creation of a psoas hitch or a Boari flap. Future models should explore incorporating some of these tasks into the simulator. In conclusion, this novel training model for ureteral reimplantation is easy to construct, affordable, and has demonstrated evidence of face, content, and construct validity. It is a valuable education tool that would allow trainees to achieve competency with a complex step of an infrequently performed procedure before attempting this surgery on a live patient. REFERENCES 1. Kozinn SI, Canes D, Sorcini A, et al. Robotic versus open distal ureteral reconstruction and reimplantation for benign stricture disease. J Endourol 2012;26:147Y151. 2. Rassweiler JJ, Gozen AS, Erdogru T, et al. Ureteral reimplantation for management of ureteral strictures: a retrospective comparison
* 2013 Lippincott Williams & Wilkins
Simulation Model for Ureteral Anastomosis
of laparoscopic and open techniques. Eur Urol 2007;51:512Y522; discussion 522Y523. 3. Uberoi J, Harnisch B, Sethi AS, et al. Robot-assisted laparoscopic distal ureterectomy and ureteral reimplantation with psoas hitch. J Endourol 2007;21:368Y373; discussion 372Y373. 4. Gilmour DT, Das S, Flowerdew G. Rates of urinary tract injury from gynecologic surgery and the role of intraoperative cystoscopy. Obstet Gynecol 2006;107:1366Y1372. 5. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011;306:978Y988. 6. Jansen JJ, Scherpbier AJ, Metz JC, et al. Performance-based assessment in continuing medical education for general practitioners: construct validity. Med Educ 1996;30:339Y344. 7. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ 2005;39:309Y317. 8. Vassiliou MC, Feldman LS, Andrew CG, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005;190:107Y113. 9. Chen CC, Korn A, Klingele C, et al. Objective assessment of vaginal surgical skills. Am J Obstet Gynecol 2010;203:79.e1Y79.e8. 10. Reznick R, Regehr G, MacRae H, et al. Testing technical skill via an innovative ‘‘bench station’’ examination. Am J Surg 1997;173:226Y230. 11. Burrows PJ, Bingham L, Brailovsky CA. A modified contrasting groups method used for setting the passmark in a small scale standardised patient examination. Adv Health Sci Educ Theory Pract 1999;4:145Y154. 12. Clauser BE, Clyman SG. A contrasting-groups approach to standard setting for performance assessments of clinical skills. Acad Med 1994;69:S42YS44. 13. Fraser SA, Klassen DR, Feldman LS, et al. Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 2003;17:964Y967. 14. Gumbs AA, Hogle NJ, Fowler DL. Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills. J Am Coll Surg 2007;204:308Y313. 15. Hung AJ, Ng CK, Patil MB, et al. Validation of a novel robotic-assisted partial nephrectomy surgical training model. BJU Int 2012;110:870Y874.
www.fpmrs.net
351
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Development and Validation of a Ureteral Anastomosis Simulation Model for Surgical Training Elena Tunitsky, MD,* Alana Murphy, MD,Þ Matthew D. Barber, MD, MHS,* Matthew Simmons, MD, PhD,Þ and J. Eric Jelovsek, MD, MMEd*
Objective: To develop and validate a new ureteral anastomosis simulation model.
Methods: We designed a training model to simulate the task of ureteral anastomosis required for ureteroneocystostomy that is suitable for robotic and laparoscopic approaches. Face validity was measured using questions related to surgical authenticity and educational value of the model. Construct validity was measured by comparing scores using Global Operative Assessment of Laparoscopic Skills Scale (GOALS) scale between ‘‘procedure experts,’’ ‘‘robotic experts,’’ and ‘‘trainees’’ groups. One-way analysis of variance was used to compare differences in the scores and operating times between the 3 groups. Associations between previous surgical experience and performance scores were measured using the Spearman rho correlation coefficient. Results: Four urologists experienced with robotically assisted ureteroneocystostomies were included in the procedure experts group. The robotic experts group consisted of 5 gynecologists experienced in robotic surgery. The trainees group consisted of 12 urology and gynecology upper-level residents and fellows. All experts agreed or strongly agreed that the model was authentic to the live procedure and a useful training tool. Mean (SD) total GOALS scores were significantly better for the procedure experts group compared to the robotic experts group and to the trainees group (P=0.02 vs P=0.004, respectively). The robotic experts group’s GOALS scores were also significantly higher than that of the trainees group (P=0.05). There were no differences in mean times required to complete the procedure. Surgical experience moderately correlated with scores on all 3 assessment scales. Conclusions: Superior performance on the model by more experienced surgeons demonstrates evidence of construct validity. This authentic and useful model allows surgeons to learn and practice the ureteral anastomosis portion of the ureteral reimplantation surgeries before operating on a live patient. Key Words: ureteral reimplantation, ureteral anastomosis, robotics, simulation model, education (Female Pelvic Med Reconstr Surg 2013;19: 346Y351)
U
reteral reimplantation using a minimally invasive approach is becoming more prevalent. Laparoscopic and robotically assisted approaches have been described for various ureteral reimplantation techniques, and functional outcomes seem to be comparable to those of open procedures.1Y3 Indications for a ureteral reimplantation with or without a Boari flap or Psoas hitch From the *Urogynecology and Reconstructive Pelvic Surgery, Obstetrics, Gynecology and Women’s Health Institute, and †Glickman Urologic Institute, Cleveland Clinic, Cleveland, OH. Reprints: Elena Tunitsky, MD, Obstetrics, Gynecology and Women’s Health Institute, Cleveland Clinic 9500 Euclid Ave, Cleveland, OH 44195. E-mail: [email protected]. The authors have declared they have no conflicts of interest. Supplemental digital contents are available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.fpmrs.net). Copyright * 2013 by Lippincott Williams & Wilkins DOI: 10.1097/SPV.0b013e3182a331bf
346
www.fpmrs.net
in the adult patient are 3-fold: ureteral stricture, distal ureteral malignancy, and iatrogenic ureteral injury. Most iatrogenic ureteral injuries occur at the time of gynecologic surgery, with an estimated rate of 1 to 13 per 1000 surgeries.4 Despite a variety of indications for ureteral reimplantation, the procedure is uncommon. The infrequent nature of ureteral reimplantation is a significant barrier for urology and urogynecology trainees to obtaining the proficient skills necessary for performing these procedures using a minimally invasive approach. In general, complex reconstructive procedures using minimally invasive techniques require an advanced psychomotor skill set learned through repetitive practice. Teaching these skills to surgical trainees is becoming more challenging in the current educational environment of shorter workweeks, emphasis on operating room efficiency, and cost containment. Owing to these constraints, simulation-based training is taking on a vital role as a teaching and assessment tool in surgical education. A recent meta-analysis by Cook et al5 compared technologyenhanced simulation training to no intervention and demonstrated that simulation training was consistently associated with improved outcomes in knowledge, skills, and behaviors. This review concluded that there is no longer a need to compare simulation-based learning to traditional nonsimulation-based learning and that future studies should continue to develop simulators specific to their specialty and learn how to efficiently use them in the clinical setting. The objective of this study was to create a novel costefficient surgical model for learning and practicing the task of robotic-assisted ureteral anastomosis required for ureteral reimplantation surgeries and present evidence regarding its validity as a training and assessment tool.
MATERIALS AND METHODS Institutional review board approval was received. This was a cross-sectional study design. The study proceeded in 3 phases: (1) model construction, (2) validity testing, and (3) setting standards for a passing performance when using the model as an assessment tool.
Model Construction The model design is demonstrated in Figure 1. Construction of the model included drilling 2 small holes in 2 of the corners of the plastic container (15 11 3 inches). Twine was then threaded through the holes and alligator clips were then attached to the twine using electric tape. A nontoxic waterbased hydrogel material (LifeLike BioTissue, London, Ontario, Canada) that can be easily stored in a container with water at a room temperature was used to replicate the visual and tactile sensation of human ureter and bladder. These materials can be stored indefinitely and can be reused for simulation approximately 10 times. The simulated ureter measured 0.5-mm wall thickness, 6 mm in diameter, and 15 cm in length. A simulated bladder was created using a rectangular 12 15-cm piece of hydrogel with a thickness of 4.40 mm. Using scissors, an
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
FIGURE 1. Simulation training model for ureteral reimplantation. A, Storage container 15 11 3 inches (approximately US $5); large bag clip (approximately US $3) attached with Velcro adhesive tape (approximately US $4) (B); alligator clips 2 (approximately US $3) (C); twine (approximately US $5) (D); ureteral 6-F JJ stent (E). **The cost does not include ureteral stent and suture.
opening of approximately 1 cm in diameter was made in the ‘‘bladder’’ sheet to simulate a cystotomy and a 6-F double-J ureteral stent was inserted between the ‘‘ureter’’ and the ‘‘bladder.’’ The stent and the bladder were secured using the bag clip. The ureter with the stent were then secured with an alligator clip and adjusted to an appropriate tension using the twine. Once the initial model was constructed, simulation materials of varied dimensions were demonstrated to the expert urologists; and the materials were selected based on their consensus. Recommended modifications were that the type and length of suture were standardized including the following: 4.0 polyglactin, dyed suture on RB-1 needle 6 inches in length. In addition, the decision was made to apply mineral oil at the suture sites to mimic the aqueous environment naturally present in the pelvis and to decrease the stiffness of the materials.
Model Acceptability and Validity The model was developed to be used for teaching and assessment. When using a simulation model for assessment, it is critical to measure the model’s psychometric properties. These important aspects of any educational assessment tool are summarized by the contemporary framework proposed by van der Vleuten and Shuwirth’s utility equation: Utility = Validity Reliability Acceptability Educational impact Costeffectiveness.6,7 This framework was used in this study to ensure that important aspects of assessment were considered in model development. Eligible participants approached included staff and trainees in urology and obstetrics and gynecology at a large referral tertiary care center. Inclusion criteria were * 2013 Lippincott Williams & Wilkins
Simulation Model for Ureteral Anastomosis
participants who were familiar with the robotic console and who perform laparoscopic or robotically assisted procedures that require suturing as part of their training or practice. The participants were categorized into 3 groups: ‘‘procedure experts,’’ ‘‘robotic experts,’’ and ‘‘trainees.’’ The procedure experts group consisted of faculty in urology who performed at least 10 robotically assisted ureteral reimplantation procedures. The robotic experts group included fellowship-trained gynecologic surgeons with a high annual volume of robotically assisted surgeries including hysterectomies, myomectomies, and sacrocolpopexies but did not perform robotically assisted ureteral reimplantation procedures as part of their practice. The trainees group included all urology residents in their fourth year of training as well as urology and urogynecology fellows. All experts were approached individually for participation, whereas trainees were solicited via e-mail from their program directors and were offered a US $10 coffee gift card for their participation. The study was exempt by the institutional review board as an educational project. Per institutional review board instructions, in lieu of the informed consent, the participants were given a detailed information sheet describing the study that did not require a signature, All participants completed a questionnaire on demographics and surgical experience and were asked to perform a ureteral neocystostomy using the simulated model. The procedure was briefly reviewed with each participant, and they were given a choice of suturing approach: interrupted, continuous suturing in a full circle, or continuous suturing in 2 hemicircles. The fourth robotic arm was available for retraction; alternatively, a participant could ask the instructor for assistance in retraction. An example of the procedure performed by a procedure expert and by a trainee is illustrated in Figure 2. Supplemental video illustrates the ureteral anastamosis performed on the model (Supplemental Digital Content, http://links.lww.com/FPMRS/A15). After completing the procedure, participants were asked to assess the realism, the use of the model and its perceived acceptability and educational impact by responding to a questionnaire using a 5-item Likert response scale. In addition, they were asked to estimate the appropriate cost for the model. The primary outcome of the study was total score using the valid and reliable Global Operative Assessment of Laparoscopic Skills (GOALS) scale.8 The GOALS scale assesses performance essential to laparoscopy in 5 domains: depth perception, bimanual dexterity, efficiency, tissue handling, and autonomy, with a maximum of 5 points for each domain (total maximum of 25 points). Two additional valid and reliable scales were used as secondary outcome measures of psychomotor skill including the modified Surgical Skills Index (SSI)9 and the modified Global Rating Scale of Operative Performance (GRS).10 All 3 scales were used to rate the performance of anastomotic suturing using live observation by 1 of 2 investigators (E.T. or A.M.). The investigators involved in the assessment are fellows in female pelvic and reconstructive surgery in the departments of gynecology and urology, respectively, with adequate experience in laparoscopic and robotic surgery to use validated assessment scales. An additional secondary outcome included total time of performance.
Setting the Pass Mark The distributions of scores were plotted using the total GOALS scale scores, and the Contrasting Groups method of standard setting was applied.11,12 For the purpose of this portion of the analysis, the procedure experts and the robotic experts were combined into an ‘‘Expert’’ group. The passing score was set at the intersection of the distribution of the 2 groups, www.fpmrs.net
347
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Tunitsky et al
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
FIGURE 2. Examples of ureteroneocystostomy performed using the model. A, Procedure performed by a trainee. B, Procedure performed by a procedure expert.
assuming false-negative and false-positive errors were of equal weight. The established passing scores were the scores that best discriminated between the groups. The Contrasting Groups method has been previously applied to assessing laparoscopy skills and is the basis for the widely used cutoff values used in the Fundamentals of Laparoscopy Skills (FLS) curriculum required of all graduating general surgery residents.13
Statistical Analysis One-way analysis of variance was used to compare differences in the scale scores and operating times between the 3 groups: procedure experts, robotic experts, and trainees. Associations between past surgical experience of participants and their performance scores were measured using the Spearman rho correlation coefficient. Significance was considered at P = 0.05. Sample size determination was informed based on previous studies using the GOALS scale.14 Assuming a common peritem standard deviation of 0.8 points and a mean per item score of 2.5 for the trainees group and mean score of 3.5 for the robotic experts group and 4.5 for the procedure experts group for each of the 5 GOALS domains, 20 participants who included
4 in the procedure experts group, 4 in the robotic experts group, and 12 in the trainees group would achieve more than 90% power to detect differences among the means of the total GOALS score with a 0.05 significance level.
RESULTS Participants’ demographics and surgical experience is described in Table 1.
Face Validity Testing Most of the participants (all experts) ‘‘agreed’’ or ‘‘strongly agreed’’ that the model is realistic and useful (Table 2). Several participants commented that the ureteral material was more fragile and less pliable, making the simulation exercise more difficult compared to live surgery. When asked about the cost of the model, most of the participants (15 [71%]) estimated the price of the model to be between US $50 and US $200 (Table 2).
Construct Validity Participants in the procedure experts group performed significantly better and received higher overall performance scores
TABLE 1. Participants’ Demographics and Surgical Experience
Number of participants Age, yrs Sex (male/female) Surgical experience beyond training, yrs Total number of open ureteroneocystostomies performed Total number of robotic/laparoscopic ureteroneocystostomies performed Number of laparoscopic surgeries requiring suturing performed annually Number of robotic surgeries requiring suturing performed annually
Procedure Experts
Robotic Experts
Trainees
4
5
12
38 (33Y50) 4/0 5 (3Y8) 23 (8Y43) 18 (13Y25) 33 (8Y50) 50 (33Y50)
38 (33Y58) 1/4 3 (1Y20) 0 (0Y3) 0 (0Y8) 38 (8Y50) 33 (18Y50)
33 (28Y45) 7/5 0 3 (0Y50) 3 (0Y18) 5 (0Y18) 0 (3Y43)
All statistics are reported as median (range).
348
www.fpmrs.net
* 2013 Lippincott Williams & Wilkins
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Simulation Model for Ureteral Anastomosis
Volume 19, Number 6, November/December 2013
TABLE 2. Responses to the Postprocedure Questionnaire 1 Strongly Disagree
2 Disagree
3 Neither Agree Nor Disagree
4 Agree
1. The simulated material closely approximates the texture and behavior of the real ureter and bladder Total (%) 1 (4.8) 5 (23.8) 13 (61.9) 2. The model setup appears appropriate for the ureteral reimplantation procedures, as it closely approximates the live surgery Total (%) 3 (14.3) 14 (66.7) 3. The training model appears useful for improving the suturing technique required for this procedure Total (%) 1 (4.8) 7 (33.3) 4. The training model appears useful for teaching and learning the procedure Total (%) 10 (47.6) 5. The training model appears useful for assessing the learner’s ability to perform this procedure before live performance Total (%) 1 (4.7) 5 (23.8) 9 (42.9) Acceptable price range of this training model (US$) G50 51Y100 101Y200 201Y300 Total (%) 3 (14.3) 10 (47.6) 5 (23.8) 3 (14.3)
than the participants in the robotic experts and trainees groups using each of the 3 scales (Table 3). The robotic experts group demonstrated significantly better performance than the trainees group on the GOALS scale but not on the modified SSI and GRS scales. When evaluating the individual items of the GOALS scale, the procedure experts group performed significantly better than the trainees group across all 5 domains. In contrast, the robotic experts group performed significantly better than the trainees group in only 3 of the 5 domains (depth perception, bimanual dexterity, and efficiency), whereas their performance was similar in the tissue handling and autonomy domains. Likewise, the robotic experts group scored significantly lower in these 2 domains than the procedure experts group but scored similarly in the remaining 3 domains (Table 3). Past surgical experience with the ureteral reimplantation procedures (open or robotically assisted) as well as annual volume of robotic procedures requiring suturing strongly or moderately correlated with better scores on all 3 assessment scales. However, annual volume of laparoscopic procedures did not significantly correlate with performance (Table 4). Additionally, median (range) time for the procedure experts group was 11.7 minutes (7.7Y14 minutes); for the robotic experts group, 13 minutes (10.8Y18 minutes); and for the trainees group, 14.1 minutes (10.2Y23 minutes); however,
5 Strongly Agree 2 (9.5) 4 (19) 13 (61.9) 11 (52.4) 6 (28.6) 9300
this was not significantly different between the groups (Table 3). An example of the completed procedure performed by a procedure expert and by a trainee is illustrated in Figure 2.
The Pass Mark Figure 3 demonstrates how the cutoff scores were determined for the GOALS scale using the Contrasting Groups approach. Overall, surgeons who achieved absolute GOALS score of 20 (80%) passed from a trainee’s level of performance to an expert’s level.
DISCUSSION Simulation training offers the advantages of repetitive practice in a safe environment without compromising patient care. This model allows trainees to focus on developing competence with the suturing task of anastomosis, one of the more complicated and time consuming aspects of the procedure. Suturing always presents a challenge to an inexperienced laparoscopic or robotic surgeon, yet it is the most repetitive and least variable task making it perfect for simulation training. Once a surgeon becomes competent at suturing using the inanimate model, they should be able to focus on the more intricate steps of the procedure such as ureterolysis, stent placement, flap creation, and overall decision making.
TABLE 3. Performance of Procedure Experts, Robotic Experts, and Trainees Using GOALS, SSI, and GRS Assessment Tools
Time, mean SD), min GOALS domains Depth perception Bimanual dexterity Efficiency Tissue handling Autonomy Total GOALS score (maximum 35) Total SSI score (maximum 28) Total GRS score (maximum 30)
Procedure Experts (PE)
Robotic Experts (RE)
Trainees (T)
P PE vs RE
P PE vs T
P RE vs T
11.25 (1.8)
12.62 (1.60)
14.73 (1.03)
0.57
0.11
0.30
5 (0.25) 5 (0.28) 4.4 (0.29) 3.2 (0.46) 3.6 (0.50) 21.2 (1.47) 23.4 (1.7) 26 (1.75)
4.1 (0.16) 3.4 (0.18) 3.1 (0.19) 2.8 (0.30) 3.5 (0.32) 17.3 (0.94) 21 (1.09) 21.3 (1.13)
1 1 0.35 0.01 0.05 0.02 0.03 0.05
0.04 0.004 0.001 0.01 0.034 0.004 0.008 0.002
0.02 0.001 0.004 0.54 0.88 0.05 0.30 0.06
5 (0.28) 5 (0.3) 4.75 (0.33) 4.75 (0.52) 5 (0.56) 24 (1.6) 28 (1.9) 30 (1.95)
All data are presented as mean (SD).
* 2013 Lippincott Williams & Wilkins
www.fpmrs.net
349
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Tunitsky et al
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
TABLE 4. Performance Using GOALS, SSI and GRS Assessment Tools by Surgical Experience GOALS
GRS
SSI
Surgical Experience
Rho
P
Rho
P
Rho
P
Total number of open ureteroneocystostomies Total number of laparoscopic or robotic ureteroneocystostomies Annual laparoscopic procedures requiring suturing Annual robotic procedures requiring suturing
0.49 0.53 0.40 0.74
0.02 0.01 0.07 0.0001
0.49 0.54 0.27 0.63
0.02 0.01 0.23 0.002
0.46 0.55 0.39 0.71
0.04 0.01 0.08 0.0003
Rho, Spearman rho correlation coefficient.
Our model involves simple design and construction, using inexpensive components that are easily obtained at a hardware store. Whereas the simulation material used in our model is commercially available, one may use other materials such as Penrose or plastic tubing. The benefit of the hydrogel material was its close resemblance in texture and size to the live ureter and bladder as reflected in the responses from participants. Overall, the participants rated the training model as authentic to the live procedure and also useful for training, and several trainees commented that they would have found it useful to practice on the model before the live surgery, lending support for the model’s usefulness and face validity. Most of the participants estimated the price of the model to be between US $50 and US $200. The actual price of the model of US $150 is in the range acceptable to the participants (US $50 for the one-time setup and US $100 for the simulated material set, which can be used approximately 10 times). We were able to demonstrate evidence of the models’ construct validity using valid and reliable assessment scales. The GOALS assessment scale was developed specifically for laparoscopic training8 and has been used in studies describing robotic models.15 The GOALS scores for performance on our model correlated with surgeon’s expertise, as designated by the group assignment. The performance also correlated with surgical experience the specific procedure and with the surgeon’s annual robotic volume. We included a group of gynecologic expert robotic surgeons who did not have experience with ureteral reimplantation procedures to elucidate if our model was specific to the ureteral reimplantation or if it solely tested only skills for robotic suturing. Whereas the robotic experts’ total GOALS scores were significantly better than those of the trainees, there was no significant difference in the domains of, tissue handling and autonomy, which supports that the model was able to discriminate between surgeons experienced with the specific procedure and those experienced in robotic surgery. This lends support to the notion that while overall robotic experience correlates with better overall performance on the model, the knowledge of ureteral surgery is as important factor that distinguishes a robotic surgeon from an expert urology robotic surgeon. This is also why we did not detect a difference in performance using the modified GRS and SSI scales.9,10 The domains of GRS and SSI scales that we tested in study address skills that reflect familiarity with the specific procedure more than a general surgical skill level (GRS: respect for tissue, time and motion, instrument handling, flow of operation, use of assistance, and knowledge of specific procedure; SSI: maintenance of visibility, use of assistants, tissue handling, knot tying/ligation, procedure completion, time and motion, flow of operation and forward planning, and knowledge of specific procedure). Not surprisingly, the robotic experts who despite having superior surgical skills were not familiar
350
www.fpmrs.net
with ureteral procedures scored similarly to the trainees using these domains. When evaluating the performance by laparoscopic volume, not surprisingly, we did not see a significant correlation. Among the participants, the gynecology experts and the trainees had high annual laparoscopic volume but were least familiar with the procedure. Thus, the lack of knowledge or experience performing the specific procedure counteracted the annual laparoscopic experience. It is possible that if our cohort included expert laparoscopic urologists who were not expert robotic surgeons, we would have been able to comment on correlation between laparoscopic skills and the performance on the model. The overall time required to complete the task was inversely proportional to the experience of the participant. This difference, however, was not significant. The trainee time varied the most and did not correlate with the educational level. In fact, some of the more experienced fellows who have scored similarly to the experts on the performance scale took the longest time to complete the task. One possible explanation for this finding is that when a simulated procedure is performed by surgeons who are familiar with ureteral surgery, they are likely focused on achieving the best possible result, one that is most authentic to the live procedure. In contrast, those unfamiliar with the surgery, not having a frame of reference, may be more concerned with completing the task faster, disregarding the final outcome. In addition, we left the choice of suturing technique up to the participants. One of the procedure experts, 4 of 5 robotic experts, and 4 of the 12 trainees (all undergoing fellowship training) elected to complete the task using
FIGURE 3. Passing mark. * 2013 Lippincott Williams & Wilkins
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Female Pelvic Medicine & Reconstructive Surgery
&
Volume 19, Number 6, November/December 2013
interrupted sutures. This likely increased their time, irrespective of the result. Unfortunately, we did not evaluate the final success of the anastomosis by testing for leak owing to the limitations of the model; however, we suspect that the quality of anastomosis would not always correlate with lower times. In developing and evaluating our simulation model, we followed the framework that assesses use of an educational tool by addressing its validity, reliability, acceptability, educational impact, and cost-effectiveness.6,7 Appropriate face and construct validity testing was conducted. Whereas reliability was not directly addressed, validated and reliable scales were used to assess construct validity. Although the educational impact and cost-effectiveness were addressed as part of the questionnaire assessing face validity, these parameters would be better addressed once the model is implemented into the surgical training curriculum and correlation is made between the practice using the model and improvement in live performance. We acknowledge that an important limitation of our study was that the evaluation of performance was not blinded, as we did not have recording capabilities in our training robotic tower. However, the GOALS scale has previously demonstrated good inter-rater reliability when it was initially introduced by Vassiliou et al.8 Thus, whereas masking the operating surgeon to the evaluator is a limitation, we feel that our results would have been similar if masking was possible. Furthermore, the model does not simulate all of the important aspects of ureteral reimplantation such as ureterolysis, introduction of a stent, bladder mobilization, creation of a psoas hitch or a Boari flap. Future models should explore incorporating some of these tasks into the simulator. In conclusion, this novel training model for ureteral reimplantation is easy to construct, affordable, and has demonstrated evidence of face, content, and construct validity. It is a valuable education tool that would allow trainees to achieve competency with a complex step of an infrequently performed procedure before attempting this surgery on a live patient. REFERENCES 1. Kozinn SI, Canes D, Sorcini A, et al. Robotic versus open distal ureteral reconstruction and reimplantation for benign stricture disease. J Endourol 2012;26:147Y151. 2. Rassweiler JJ, Gozen AS, Erdogru T, et al. Ureteral reimplantation for management of ureteral strictures: a retrospective comparison
* 2013 Lippincott Williams & Wilkins
Simulation Model for Ureteral Anastomosis
of laparoscopic and open techniques. Eur Urol 2007;51:512Y522; discussion 522Y523. 3. Uberoi J, Harnisch B, Sethi AS, et al. Robot-assisted laparoscopic distal ureterectomy and ureteral reimplantation with psoas hitch. J Endourol 2007;21:368Y373; discussion 372Y373. 4. Gilmour DT, Das S, Flowerdew G. Rates of urinary tract injury from gynecologic surgery and the role of intraoperative cystoscopy. Obstet Gynecol 2006;107:1366Y1372. 5. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011;306:978Y988. 6. Jansen JJ, Scherpbier AJ, Metz JC, et al. Performance-based assessment in continuing medical education for general practitioners: construct validity. Med Educ 1996;30:339Y344. 7. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ 2005;39:309Y317. 8. Vassiliou MC, Feldman LS, Andrew CG, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005;190:107Y113. 9. Chen CC, Korn A, Klingele C, et al. Objective assessment of vaginal surgical skills. Am J Obstet Gynecol 2010;203:79.e1Y79.e8. 10. Reznick R, Regehr G, MacRae H, et al. Testing technical skill via an innovative ‘‘bench station’’ examination. Am J Surg 1997;173:226Y230. 11. Burrows PJ, Bingham L, Brailovsky CA. A modified contrasting groups method used for setting the passmark in a small scale standardised patient examination. Adv Health Sci Educ Theory Pract 1999;4:145Y154. 12. Clauser BE, Clyman SG. A contrasting-groups approach to standard setting for performance assessments of clinical skills. Acad Med 1994;69:S42YS44. 13. Fraser SA, Klassen DR, Feldman LS, et al. Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 2003;17:964Y967. 14. Gumbs AA, Hogle NJ, Fowler DL. Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills. J Am Coll Surg 2007;204:308Y313. 15. Hung AJ, Ng CK, Patil MB, et al. Validation of a novel robotic-assisted partial nephrectomy surgical training model. BJU Int 2012;110:870Y874.
www.fpmrs.net
351
Copyright © 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
