ORIGINAL REPORTS
Laparoscopic Skills Maintenance: A Randomized Trial of Virtual Reality and Box Trainer Simulators Montaha W. Khan, MBBS,* Diwei Lin, MBBS,* Nicholas Marlow, MA PH,† Meryl Altree, BNg,† Wendy Babidge, PhD,*,† John Field, PhD AStat,‡ Peter Hewett, MBBS,* and Guy Maddern, PhD*,† *
University of Adelaide, Department of Surgery, The Queen Elizabeth Hospital, Adelaide, Australia; Australian Safety and Efficacy Register of New Interventional Procedures—Surgical, Royal Australasian College of Surgeons, Adelaide, Australia; and ‡John Field Consulting Pty Ltd, Adelaide, Australia †
OBJECTIVE: A number of simulators have been developed to teach surgical trainees the basic skills required to effectively perform laparoscopic surgery; however, consideration needs to be given to how well the skills taught by these simulators are maintained over time. This study compared the maintenance of laparoscopic skills learned using box trainer and virtual reality simulators. DESIGN: Participants were randomly allocated to be trained and assessed using either the Society of American Gastrointestinal Endoscopic Surgeons Fundamentals of Laparoscopic Surgery (FLS) simulator or the Surgical Science virtual reality simulator. Once participants achieved a predetermined level of proficiency, they were assessed 1, 3, and 6 months later. At each assessment, participants were given 2 practice attempts and assessed on their third attempt. SETTING: The study was conducted through the Simulated
Surgical Skills Program that was held at the Royal Australasian College of Surgeons, Adelaide, Australia. RESULTS: Overall, 26 participants (13 per group) completed the training and all follow-up assessments. There were no significant differences between simulation-trained cohorts for age, gender, training level, and the number of surgeries previously performed, observed, or assisted. Scores for the FLS-trained participants did not significantly change over the follow-up period. Scores for LapSim-trained participants significantly deteriorated at the first 2 followup points (1 and 3 months) (p o 0.050), but returned to be near initial levels by the final follow-up (6 months).
Correspondence: Inquiries to Guy J. Maddern, PhD, FRACS, ASERNIP-S, 199 Ward Street, North Adelaide, South Australia 5006, Australia; fax: þ6188-219-0999; e-mail: [email protected]
CONCLUSIONS: This research showed that basic laparo-
scopic skills learned using the FLS simulator were maintained more consistently than those learned on the LapSim simulator. However, by the final follow-up, both simulatortrained cohorts had skill levels that were not significantly different to those at proficiency after the initial training C 2014 Association of Program period. ( J Surg 71:79-84. J Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: laparoscopy, education COMPETENCIES: Medical
Knowledge, Practice-Based Learning and Improvement, Systems-Based Practice
INTRODUCTION The use of laparoscopic surgery in Australia began during the late 1980s and has now become the technique of choice for numerous surgical procedures. Although laparoscopic surgery was initially the domain of general and gynecological surgeons, it has now gained prominence across all surgical specialties.1 The increase in popularity of laparoscopic surgery over traditional open surgery may be owing to its advocated benefits, including improved esthetics, lower infection rates, and faster healing times. However, a number of specific phenomena, such as decreased tactile sensation, working in a 3-dimensional setting with a 2-dimensional image, and the fulcrum effect of using instruments through a trocar, make this technique difficult to learn.2 A number of different laparoscopic simulators have been used to teach surgeons basic laparoscopic skills outside the operating room, in a risk-free environment, where mistakes can be easily identified and corrected. The use of simulators has been increasingly recognized as a powerful learning tool, and the skills acquired on either a
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2013.05.009
79
box trainer or a virtual reality (VR) simulator3 have been demonstrated to transfer to the operative setting.3-8 Furthermore, laparoscopic simulators can be used to train surgeons to a predetermined level of proficiency.9 The achievement of proficiency standards using simulators can provide hospitals and accreditation boards with a level of certainty that surgeons can perform to a known standard; however, achieving proficiency does not necessarily mean that this performance standard would be maintained. This research examined the hypothesis that the maintenance of basic laparoscopic skills, assessed in a simulated environment, is dependent on the type of simulator training. After an initial training period using either a box trainer or a VR simulation, follow-up assessments were collected over a 6-month period. The results provide an important insight into how laparoscopic skills are maintained, whether this is influenced by simulation type, and how this in turn affects the overall efficacy of simulator training curricula.
MATERIAL AND METHODS Study Design The research methodology involved random allocation to a simulator, proficiency training using the allocated simulator, and follow-up assessments. Simple randomization was performed at enrollment by participants selecting a sealed envelope that revealed their allocated simulator. The study was conducted through the Simulated Surgical Skills Program that was held at the Royal Australasian College of Surgeons, North Adelaide, South Australia between July 2009 and May 2010. Participants trained on their allocated simulator until they reached a predetermined level of proficiency. The study was approved by the Human Research Ethics Committees of the Royal Australasian College of Surgeons, University of Adelaide, Royal Adelaide Hospital and The Queen Elizabeth Hospital. Participants A total of 41 participants were enrolled from the following groups: final year undergraduate medical students, interns, registered medical officers, and surgical trainees. Each had responded to the advertisements inviting participation in the study that were circulated via hospital email or posted on e-Bulletin boards by project staff. Following registration, participants attended an introductory session where they were given an information sheet, pretrial questionnaire, consent form, and complaint sheet. Participants were provided with an overview of the project, were briefed on their roles, and if willing to participate, signed the consent form. Their availability was noted and training sessions were 80
scheduled accordingly. Simple randomization of participants to the simulators was performed by individually selecting a sealed envelope that revealed their allocated simulator at enrollment. Participants were involved in normal working activities during the period of study, but none of them partook in any other laparoscopic surgical simulation training.
Equipment and Training The Fundamentals of Laparoscopic Surgery (FLS) Simulator FLS simulator is a low-fidelity simulator developed by the Society of American Gastrointestinal Endoscopic Surgeons (United States). Each FLS simulator used NEC 19-in liquid-crystal display monitors. All FLS stations used congruent sets of laparoscopic equipment consisting of 2 Maryland graspers, 1 left-handed and 1 right-handed needle holder, 1 ratcheted grasper, 1 knot pusher, and 1 laparoscopic scissor. Consumable cutting gauze and Penrose drains were purchased from Society of American Gastrointestinal Endoscopic Surgeons. Syneture 2.0, metric 3 coated braided silk 75 cm sutures were cut down for the intracorporeal knot-tying task. Syneture 2.0, metric 3 coated braided silk 120 cm sutures were used for the extracorporeal knot-tying tasks; however, these were replaced toward the end of the project by Dynek 2.0, metric 3 coated braided silk 120 cm sutures owing to cost considerations. Ethicon PDS2 metric 3.5, 18-in violet monofilament endoloops were used for the ligating loop task. Training conducted on the FLS simulator consisted of 5 tasks: peg transfer, precision cutting, simple suture with an extracorporeal knot, simple suture with an intracorporeal knot, and placement of a ligating loop.
LapSim Simulator The LapSim VR simulators were developed by Surgical Science (Goteborg, Sweden). The Surgical Science LapSim version 2009 Basic Skills' module v3.0.2 was run on 3 custom-built personal computers. Two ran on Windows Vista, used an ACER 193XD monitor, and were connected to a Xitact IHP simulator. A third personal computer ran on Windows XP, with the image displayed on a 17-in Eizo monitor, and used an Immersion Surgical Workstations simulator. Training on the LapSim simulator consisted of 7 tasks: camera navigation, instrument navigation, coordination, grasping, cutting, clip applying, and suturing. Both groups of participants were mentored and assessed by research staff who were familiar with the skills required to reach proficiency on each simulator. Supervision was the same for both groups.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
Outcome Measures The pretrial questionnaire, completed by participants, was purpose designed and included basic demographic data, surgical experience, and also questions regarding participation in other activities that could have a potential influence on a participant's ability to learn and retain laparoscopic skills (e.g., playing a musical instrument or involvement in other fine motor activities). For participants in the FLS-trained group, each task was performed for 20 minutes, after which an assessment was undertaken. For 4 tasks (i.e., peg transfer, precision cutting, simple suture with an extracorporeal knot, and simple suture with an intracorporeal knot), proficiency was deemed to have been achieved when the participant was able to successfully complete the task in consecutive assessments. The fifth task, placement of a ligating loop, used testing intervals that were based on the number of attempts. Participants were given 2 practices and were then assessed on their third attempt; if they passed the third attempt, they were immediately reassessed. The same reassessment mechanism was used to determine proficiency in this task. The FLS task-assessment criteria were based on parameters reported by Ritter and Scott.10 The proficiency of LapSim-trained participants at all 7 tasks was assessed at every third attempt of a particular task using the inbuilt measuring tool in the LapSim simulator. Participants were deemed to have passed a task if they achieved a proficient score at 2 consecutive testing instances (e.g., 3 and 6, 6 and 9, and 9 and 12). The taskassessment criteria were based on parameters agreed upon by a panel of experienced local surgeons. Once proficiency had been achieved at the initial training period, both simulator cohorts were asked to return at 1, 3, and 6 months (December 2009, February 2010, and May 2010) for follow-up assessments. During the follow-up period, no participant had additional simulator exposure, but all had routine hospital experience according to their level. For these assessments, participants were given 2 practice attempts followed by the assessment. This method was followed for all tasks on both simulators. Each follow-up assessment was measured using the same criteria as the assessments during the initial training period. Although the 2 simulators chosen for use in this study are inherently different from each other for the specific tasks involved and how they measure performance, they are both designed to aid in the acquisition of basic laparoscopic surgical skills, hence the comparison of skills maintenance is considered valid. Statistical Analysis Demographic and experience data of the cohorts were compared using t tests or Fisher exact test for counts. Simulator scores were expressed as the mean score per task. Although the numbers generated when assessing the
FLS and LapSim are similar, the 2 are not directly comparable. Therefore, changes over time were examined statistically in 2 ways. First, scores for each testing instance (i.e., initially, 1, 3, and 6 months) were compared. This was done using analysis of variance techniques to account for differences in individual mean scores. Second, models were fitted to scores over time, to take the actual day of assessment into account. Model predictors were the mean score and a restricted cubic spline in days for each participant. To retain balance, both of these analyses used only those participants who completed all 3 follow-up assessments.
RESULTS Participants Starting in July 2009, a total of 41 participants commenced this research and reached proficiency on the designated simulator; however, 26 (13 FLS trained; 13 LapSim trained) completed all 3 follow-up assessments (63% of the initial cohort). The demographics and experience characteristics of both cohorts are reported in Table 1. No complaints were received about the administration of the project from any participants. Assessment Occasions Although assessments were scheduled at 1, 3, and 6 months after initial proficiency was achieved, participants were often unavailable at the designated time for varying reasons such as changes in surgical rotation. As a result, the assessment dates were spread around the scheduled times. The first round of follow-up assessments for the FLS cohort was between 23 and 114 days (mean ¼ 37), the second was between 85 and 170 days (mean ¼ 107), and the third was between 142 and 255 days (mean ¼ 193). For the LapSim cohort, the first assessment was between 28 and 58 days (mean ¼ 35), the second was between 72 and 112 days (mean ¼ 94), and the third was between 156 and 250 days (mean ¼ 198). There were no significant differences between mean times of FLS and LapSim assessments. Scores for the FLS-trained participants once proficiency had been achieved at the end of the training period and at the 3 follow-up points are shown in Table 2. As can be seen, high scores were achieved at the end of the training period and were maintained at the follow-up assessments, with no significant difference seen between assessment points. However, the variability of scores was significantly greater at followup assessments than at the end of the training period. Results for the VR LapSim-trained cohort are shown in Table 3. This cohort demonstrated a significant initial decline in mean score from the end of the training period to the first assessment (p o 0.001). This difference remained significantly different from baseline at the second assessment (p ¼ 0.011), but had lost significance by the
Journal of Surgical Education Volume 71/Number 1 January/February 2014
81
TABLE 1. Demographic and Experience Data of the 26 Participants With Complete Follow-Up According to Simulator Type Variables
FLS (n ¼ 13)
LapSim p (n ¼ 13) Values
26.5 (4.7) 27.1 (3.0) 0.697 Age (y)* Female sex 3 (23) 4 (31) 1.000 Right handed 12 (92) 10 (77) 0.593 Training level Student 6 (50) 4 (33) 0.496 Intern 0 1 (8) RMO 6 (50) 5 (42) Registrar 0 2 (17) Surgery performed 3 (23) 2 (15) 1.000 Number of surgeries performed 0 10 (77) 11 (92) 1.000 1 to 10 2 (15) 1 (8) 11 to 30 0 0 430 1 (8) 0 Surgery observed 13 (100) 13 (100) 1.000 Number of surgeries observed 0 0 0 0.371 1 to 10 2 (15) 1 (8) 11 to 30 9 (69) 6 (46) 430 2 (15) 6 (46) Surgeries assisted 11 (85) 11 (85) 1.000 Number of surgeries assisted 0 2 (15) 2 (15) 0.062 1 to 10 4 (31) 4 (31) 11 to 30 6 (46) 1 (8) 430 1 (8) 6 (46) Participant’s perceived level of competence at laparoscopic surgery† 1 9 (82) 8 (67) 0.784 2 1 (9) 3 (25) 3 1 (9) 1 (8) 4 0 0 5 0 0 2 (15) 3 (23) 1.000 Participation in activities involving fine motor skills‡ Music years 0 8 (72) 10 (77) 1.000 1 to 10 1 (9) 1 (8) 11 to 30 2 (18) 2 (15) Number of instruments played 0 8 (62) 10 (77) 0.551 1 4 (31) 1 (8) 2 1 (8) 1 (8) 3 0 0 4 0 1 (8) Gaming (h/wk) 0 2 (15) 5 (39) 0.691 ≤1 3 (23) 2 (15) 1.1 to 3 5 (39) 3 (23) 3.1 to 6 2 (15) 1 (8) 46 1 (8) 2 (15) Values in parentheses are percentages unless indicated otherwise. SD, standard deviation; RMO, resident medical officer. *Values are means (standard deviations). † Participants were required to self-assess their level of competency on a scale of 1 to 5 (1 ¼ novice to 5 ¼ expert). ‡ Fine motor skills were a predefined set of hobbies including sewing, knitting, crocheting, and model making. Percentages derived from available participant responses. 82
third assessment. The variation in participants’ scores followed a similar pattern; scores at the first and second follow-up assessments were significantly more variable than those at the end of the initial training period, but by the third assessment variability was no longer significantly different to that seen initially. Change Over Time The relationship between follow-up assessment scores and length of time was examined after attaining proficiency. Changes over time for each simulator are shown in Figure using local quadratic regression fits. Models for scores accounted for 65% of the FLS score variation and 50% of the LapSim score variation. For the FLS cohort, the spline term in days was not significant after attaining proficiency, indicating that the data are well described by the mean score alone with no change over time. This was not so for the LapSim scores, where the rapid initial decline in score followed by the gradual rise was significant as shown in Figure. These results are consistent with the previous analysis (Table 3).
DISCUSSION This study found that maintenance of basic laparoscopic skill levels differed according to the type of simulator training. Training with a low-fidelity FLS simulator resulted in relatively stable scores over the entire study period of 6 months, with little difference between initial and followup scores or their variability. In contrast, training with a VR LapSim stimulator resulted in more variable scores, with the mean scores at the first and second follow-up assessments being significantly less than those at proficiency (p o 0.001). This initial loss improved, so that by the third follow-up assessment, at approximately 6 months, the mean scores of LapSim-trained participants were not significantly different from those seen initially at proficiency. The initial decrease in skill levels of VR simulator–trained participants, which we found, is an accepted phenomenon that occurs when skills are not practiced over time11 and concurs with the findings of previous studies.12-14 Windsor and Zoha12 found that VR-trained participants were considerably worse at the second of the 2 training sessions conducted 1 month apart (statistical analyses were not reported). Sinha et al.13 reported that LapSim-trained participants had an initial decline in fine motor skill tasks after proficiency was first attained. However, they maintained most skills even after an extended period of time. Stefanidis et al.14 assessed the skills retention approximately 2 weeks and 7 months after having reached proficiency in a VR simulator or a video trainer and found that there was greater early performance decay for the VR simulator, but after the early performance decrement, there was no skill loss for both simulators.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
TABLE 2. Initial and Follow-Up Data for the 13 Participants in the FLS Training Cohort Who Completed All 3 Follow-Up Assessments Follow-Up Parameter
At Proficiency
Number of participants Days since proficiency achieved Mean Median Range Score Mean Median Range Standard deviation Coefficient of variation (%)
13 0 0 0 97.5 97.4 94.8 to 101.4 2.0 a 2.1
1
2
13
13
37 29 23 to 114
p Values
13
107 99 85 to 170
96.0 97.4 85.6 to 102.4 5.0 b 5.2
3
96.3 97.2 82.0 to 104.2 6.0 b 6.2
193 175 142 to 255 96.0 97.8 85.8 to 101.4 5.1 b 5.3
0.561 0.007*
*Values followed by the same letter are not significantly different at p ¼ 0.050.
Thus, although firm conclusions cannot be drawn because of the limited data, the results from our study suggest that a poorer retention of skills may be associated with VR simulation training, especially in the intermediate term, and that low-fidelity simulation may be preferable for the long-term retention of laparoscopic skills. The results of this research have important implications for surgical education. It demonstrates that participants who attained a predetermined level of proficiency during an initial training period may have a decrement of skills initially, but retain most of their skill set over an extended period of time. This suggests that continuous ongoing refresher training may not be required. However, to avoid the initial decline in proficiency seen soon after training, at least with VR simulation training, surgical educators may opt to run refresher courses shortly after the initial training period. Improved laparoscopic skills and retention of these skills would ultimately lead to improved patient safety. Although the current study assessed a larger group of participants compared to other published studies, there was still a large number of losses to follow-up. It is a
well-documented issue that the longer a study continues, the greater the rate of participant withdrawal or loss to follow-up. Although only 26 of the initial 41 participants (63%) who reached proficiency completed all 3 follow-up assessments, we believe this loss to follow-up had little negative effect on the results. The FLS-trained scores for any completed follow-up assessments for the “noncompleters” were not significantly different to the scores of those who completed all assessments. For the LapSim group, there were no significant differences between noncompleters and those who completed the study at proficiency or at the first assessment; however, the 8 noncompleters had a significantly lower mean score at the second assessment than those who completed all 3 assessments (mean scores ¼ 71.7 and 80.5, respectively; p ¼ 0.004). The reason for this is unclear; however, had all “noncompleters” data been included in the final analysis, their reduction in score would have further enhanced the drop-off in LapSim performance at the second assessment. Finally, it is important to acknowledge that the skills set tested were limited to simulator skills. In summary, basic laparoscopic skill maintenance was more
TABLE 3. Initial and Follow-Up Data for the 13 Participants in the LapSim Training Cohort Who Completed All 3 Follow-Up Assessments Follow-Up Parameter
At Proficiency
Number of participants Days since proficiency achieved Mean Median Range Score Mean Median Range Standard deviation Coefficient of variation (%)
13 0 0 0 87.0 a 86.0 82.1 to 93.3 3.3 a 3.7
1
2
3
13
13
13
35 32 28 to 58
94 94 72 to 112
198 195 156 to 250
75.9 b 73.8 56.2 to 89.4 10.7 b 14.0
80.5 bc 81.5 67.8 to 90.4 6.2 bc 7.8
84.8 ca 85.8 76.6 to 92.8 5.0 ca 5.9
p Values
o0.001* 0.008*
*Values followed by the same letter are not significantly different at p ¼ 0.050. Journal of Surgical Education Volume 71/Number 1 January/February 2014
83
room in blinded, randomised controlled trial. Stud Health Technol Inform. 2007;125:76-81. 4. Ahlberg G, Enochsson L, Gallagher AG, et al.
Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193:797-804. 5. Larsen CR, Søerensen JL, Grantcharov TP, et al. Effect
of virtual reality training on laparoscopic surgery: randomised controlled trial. B Med J. 2009;338:b1802. 6. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L,
Rosenberg J, Funch-Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91:146-150. 7. Dawe S, Windsor J, Cregan P et al. Surgical simulation FIGURE. Smoothed scores over time for FLS and LapSim participants after attaining proficiency at the initial training period. Shaded areas are the nonparametric bootstrap 95% confidence intervals. Rugs show times of follow-up testing for FLS (top) and LapSim (bottom) participants. Note that FLS and LapSim scores are not directly comparable.
consistently achieved after initial training using a FLS simulator than a VR LapSim simulator, though in the longer term skill levels were similar. Further studies are necessary to assess if the same pattern of skills retention is found when participants are assessed in a real operative setting after being trained on different modalities of simulators
for training: skills transfer to the operating room (update). ASERNIP-S Report 80. 2012 [November 2012]. Available at: http://www.surgeons.org/asernip-s. 8. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA,
Fayez R, Fried GM. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. Am J Surg. 2010;199: 115-120. 9. Seymour NE, Gallagher AG, Roman SA, et al. Virtual
reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236:458-463.
ACKNOWLEDGMENTS
10. Ritter EM, Scott DJ. Design of a proficiency-based
The funding for this research was provided by the Commonwealth Government of Australia, Department of Health and Ageing. The project was administered by the Royal Australasian College of Surgeons.
11. Arthur W Jr, Bennett W Jr, Stanush PL, McNelly TL.
skills training curriculum for the Fundamentals of Laparoscopic Surgery. Surg Innov. 2007;14:107-112. Factors that influence skill decay and retention: a quantitative review and analysis. Hum Perform. 1998;11:57-101. 12. Windsor JA, Zoha F. The laparoscopic performance of
REFERENCES 1. Johnson WR. Laparoscopic surgery: time for re-
evaluation. Med J Aust. 1996;165:355-356.
novice surgical trainees: testing for acquisition, loss, and reacquisition of psychomotor skills. Surg Endosc. 2005;19:1058-1063.
2. Grantcharov TP, Funch-Jensen P. Can everyone
13. Sinha P, Hogle NJ, Fowler DL. Do the laparoscopic
achieve proficiency with the laparoscopic technique? Learning curve patterns in technical skills acquisition Am J Surg. 2009;197:447-449.
skills of trainees deteriorate over time? Surg Endosc. 2008;22:2018-2025.
3. Cosman PH, High TJ, Shearer CJ, Merrett ND,
Biankin AV, Cartmill JA. Skills acquired on virtual reality laparoscopic simulators transfer into operating
84
14. Stefanidis D, Korndorffer JR, Sierra R, Touchard C,
Dunne B, Scott DJ. Skill retention following proficiency-based laparoscopic simulator training. Surgery. 2005;138:165-170.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
Laparoscopic Skills Maintenance: A Randomized Trial of Virtual Reality and Box Trainer Simulators Montaha W. Khan, MBBS,* Diwei Lin, MBBS,* Nicholas Marlow, MA PH,† Meryl Altree, BNg,† Wendy Babidge, PhD,*,† John Field, PhD AStat,‡ Peter Hewett, MBBS,* and Guy Maddern, PhD*,† *
University of Adelaide, Department of Surgery, The Queen Elizabeth Hospital, Adelaide, Australia; Australian Safety and Efficacy Register of New Interventional Procedures—Surgical, Royal Australasian College of Surgeons, Adelaide, Australia; and ‡John Field Consulting Pty Ltd, Adelaide, Australia †
OBJECTIVE: A number of simulators have been developed to teach surgical trainees the basic skills required to effectively perform laparoscopic surgery; however, consideration needs to be given to how well the skills taught by these simulators are maintained over time. This study compared the maintenance of laparoscopic skills learned using box trainer and virtual reality simulators. DESIGN: Participants were randomly allocated to be trained and assessed using either the Society of American Gastrointestinal Endoscopic Surgeons Fundamentals of Laparoscopic Surgery (FLS) simulator or the Surgical Science virtual reality simulator. Once participants achieved a predetermined level of proficiency, they were assessed 1, 3, and 6 months later. At each assessment, participants were given 2 practice attempts and assessed on their third attempt. SETTING: The study was conducted through the Simulated
Surgical Skills Program that was held at the Royal Australasian College of Surgeons, Adelaide, Australia. RESULTS: Overall, 26 participants (13 per group) completed the training and all follow-up assessments. There were no significant differences between simulation-trained cohorts for age, gender, training level, and the number of surgeries previously performed, observed, or assisted. Scores for the FLS-trained participants did not significantly change over the follow-up period. Scores for LapSim-trained participants significantly deteriorated at the first 2 followup points (1 and 3 months) (p o 0.050), but returned to be near initial levels by the final follow-up (6 months).
Correspondence: Inquiries to Guy J. Maddern, PhD, FRACS, ASERNIP-S, 199 Ward Street, North Adelaide, South Australia 5006, Australia; fax: þ6188-219-0999; e-mail: [email protected]
CONCLUSIONS: This research showed that basic laparo-
scopic skills learned using the FLS simulator were maintained more consistently than those learned on the LapSim simulator. However, by the final follow-up, both simulatortrained cohorts had skill levels that were not significantly different to those at proficiency after the initial training C 2014 Association of Program period. ( J Surg 71:79-84. J Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: laparoscopy, education COMPETENCIES: Medical
Knowledge, Practice-Based Learning and Improvement, Systems-Based Practice
INTRODUCTION The use of laparoscopic surgery in Australia began during the late 1980s and has now become the technique of choice for numerous surgical procedures. Although laparoscopic surgery was initially the domain of general and gynecological surgeons, it has now gained prominence across all surgical specialties.1 The increase in popularity of laparoscopic surgery over traditional open surgery may be owing to its advocated benefits, including improved esthetics, lower infection rates, and faster healing times. However, a number of specific phenomena, such as decreased tactile sensation, working in a 3-dimensional setting with a 2-dimensional image, and the fulcrum effect of using instruments through a trocar, make this technique difficult to learn.2 A number of different laparoscopic simulators have been used to teach surgeons basic laparoscopic skills outside the operating room, in a risk-free environment, where mistakes can be easily identified and corrected. The use of simulators has been increasingly recognized as a powerful learning tool, and the skills acquired on either a
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2013.05.009
79
box trainer or a virtual reality (VR) simulator3 have been demonstrated to transfer to the operative setting.3-8 Furthermore, laparoscopic simulators can be used to train surgeons to a predetermined level of proficiency.9 The achievement of proficiency standards using simulators can provide hospitals and accreditation boards with a level of certainty that surgeons can perform to a known standard; however, achieving proficiency does not necessarily mean that this performance standard would be maintained. This research examined the hypothesis that the maintenance of basic laparoscopic skills, assessed in a simulated environment, is dependent on the type of simulator training. After an initial training period using either a box trainer or a VR simulation, follow-up assessments were collected over a 6-month period. The results provide an important insight into how laparoscopic skills are maintained, whether this is influenced by simulation type, and how this in turn affects the overall efficacy of simulator training curricula.
MATERIAL AND METHODS Study Design The research methodology involved random allocation to a simulator, proficiency training using the allocated simulator, and follow-up assessments. Simple randomization was performed at enrollment by participants selecting a sealed envelope that revealed their allocated simulator. The study was conducted through the Simulated Surgical Skills Program that was held at the Royal Australasian College of Surgeons, North Adelaide, South Australia between July 2009 and May 2010. Participants trained on their allocated simulator until they reached a predetermined level of proficiency. The study was approved by the Human Research Ethics Committees of the Royal Australasian College of Surgeons, University of Adelaide, Royal Adelaide Hospital and The Queen Elizabeth Hospital. Participants A total of 41 participants were enrolled from the following groups: final year undergraduate medical students, interns, registered medical officers, and surgical trainees. Each had responded to the advertisements inviting participation in the study that were circulated via hospital email or posted on e-Bulletin boards by project staff. Following registration, participants attended an introductory session where they were given an information sheet, pretrial questionnaire, consent form, and complaint sheet. Participants were provided with an overview of the project, were briefed on their roles, and if willing to participate, signed the consent form. Their availability was noted and training sessions were 80
scheduled accordingly. Simple randomization of participants to the simulators was performed by individually selecting a sealed envelope that revealed their allocated simulator at enrollment. Participants were involved in normal working activities during the period of study, but none of them partook in any other laparoscopic surgical simulation training.
Equipment and Training The Fundamentals of Laparoscopic Surgery (FLS) Simulator FLS simulator is a low-fidelity simulator developed by the Society of American Gastrointestinal Endoscopic Surgeons (United States). Each FLS simulator used NEC 19-in liquid-crystal display monitors. All FLS stations used congruent sets of laparoscopic equipment consisting of 2 Maryland graspers, 1 left-handed and 1 right-handed needle holder, 1 ratcheted grasper, 1 knot pusher, and 1 laparoscopic scissor. Consumable cutting gauze and Penrose drains were purchased from Society of American Gastrointestinal Endoscopic Surgeons. Syneture 2.0, metric 3 coated braided silk 75 cm sutures were cut down for the intracorporeal knot-tying task. Syneture 2.0, metric 3 coated braided silk 120 cm sutures were used for the extracorporeal knot-tying tasks; however, these were replaced toward the end of the project by Dynek 2.0, metric 3 coated braided silk 120 cm sutures owing to cost considerations. Ethicon PDS2 metric 3.5, 18-in violet monofilament endoloops were used for the ligating loop task. Training conducted on the FLS simulator consisted of 5 tasks: peg transfer, precision cutting, simple suture with an extracorporeal knot, simple suture with an intracorporeal knot, and placement of a ligating loop.
LapSim Simulator The LapSim VR simulators were developed by Surgical Science (Goteborg, Sweden). The Surgical Science LapSim version 2009 Basic Skills' module v3.0.2 was run on 3 custom-built personal computers. Two ran on Windows Vista, used an ACER 193XD monitor, and were connected to a Xitact IHP simulator. A third personal computer ran on Windows XP, with the image displayed on a 17-in Eizo monitor, and used an Immersion Surgical Workstations simulator. Training on the LapSim simulator consisted of 7 tasks: camera navigation, instrument navigation, coordination, grasping, cutting, clip applying, and suturing. Both groups of participants were mentored and assessed by research staff who were familiar with the skills required to reach proficiency on each simulator. Supervision was the same for both groups.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
Outcome Measures The pretrial questionnaire, completed by participants, was purpose designed and included basic demographic data, surgical experience, and also questions regarding participation in other activities that could have a potential influence on a participant's ability to learn and retain laparoscopic skills (e.g., playing a musical instrument or involvement in other fine motor activities). For participants in the FLS-trained group, each task was performed for 20 minutes, after which an assessment was undertaken. For 4 tasks (i.e., peg transfer, precision cutting, simple suture with an extracorporeal knot, and simple suture with an intracorporeal knot), proficiency was deemed to have been achieved when the participant was able to successfully complete the task in consecutive assessments. The fifth task, placement of a ligating loop, used testing intervals that were based on the number of attempts. Participants were given 2 practices and were then assessed on their third attempt; if they passed the third attempt, they were immediately reassessed. The same reassessment mechanism was used to determine proficiency in this task. The FLS task-assessment criteria were based on parameters reported by Ritter and Scott.10 The proficiency of LapSim-trained participants at all 7 tasks was assessed at every third attempt of a particular task using the inbuilt measuring tool in the LapSim simulator. Participants were deemed to have passed a task if they achieved a proficient score at 2 consecutive testing instances (e.g., 3 and 6, 6 and 9, and 9 and 12). The taskassessment criteria were based on parameters agreed upon by a panel of experienced local surgeons. Once proficiency had been achieved at the initial training period, both simulator cohorts were asked to return at 1, 3, and 6 months (December 2009, February 2010, and May 2010) for follow-up assessments. During the follow-up period, no participant had additional simulator exposure, but all had routine hospital experience according to their level. For these assessments, participants were given 2 practice attempts followed by the assessment. This method was followed for all tasks on both simulators. Each follow-up assessment was measured using the same criteria as the assessments during the initial training period. Although the 2 simulators chosen for use in this study are inherently different from each other for the specific tasks involved and how they measure performance, they are both designed to aid in the acquisition of basic laparoscopic surgical skills, hence the comparison of skills maintenance is considered valid. Statistical Analysis Demographic and experience data of the cohorts were compared using t tests or Fisher exact test for counts. Simulator scores were expressed as the mean score per task. Although the numbers generated when assessing the
FLS and LapSim are similar, the 2 are not directly comparable. Therefore, changes over time were examined statistically in 2 ways. First, scores for each testing instance (i.e., initially, 1, 3, and 6 months) were compared. This was done using analysis of variance techniques to account for differences in individual mean scores. Second, models were fitted to scores over time, to take the actual day of assessment into account. Model predictors were the mean score and a restricted cubic spline in days for each participant. To retain balance, both of these analyses used only those participants who completed all 3 follow-up assessments.
RESULTS Participants Starting in July 2009, a total of 41 participants commenced this research and reached proficiency on the designated simulator; however, 26 (13 FLS trained; 13 LapSim trained) completed all 3 follow-up assessments (63% of the initial cohort). The demographics and experience characteristics of both cohorts are reported in Table 1. No complaints were received about the administration of the project from any participants. Assessment Occasions Although assessments were scheduled at 1, 3, and 6 months after initial proficiency was achieved, participants were often unavailable at the designated time for varying reasons such as changes in surgical rotation. As a result, the assessment dates were spread around the scheduled times. The first round of follow-up assessments for the FLS cohort was between 23 and 114 days (mean ¼ 37), the second was between 85 and 170 days (mean ¼ 107), and the third was between 142 and 255 days (mean ¼ 193). For the LapSim cohort, the first assessment was between 28 and 58 days (mean ¼ 35), the second was between 72 and 112 days (mean ¼ 94), and the third was between 156 and 250 days (mean ¼ 198). There were no significant differences between mean times of FLS and LapSim assessments. Scores for the FLS-trained participants once proficiency had been achieved at the end of the training period and at the 3 follow-up points are shown in Table 2. As can be seen, high scores were achieved at the end of the training period and were maintained at the follow-up assessments, with no significant difference seen between assessment points. However, the variability of scores was significantly greater at followup assessments than at the end of the training period. Results for the VR LapSim-trained cohort are shown in Table 3. This cohort demonstrated a significant initial decline in mean score from the end of the training period to the first assessment (p o 0.001). This difference remained significantly different from baseline at the second assessment (p ¼ 0.011), but had lost significance by the
Journal of Surgical Education Volume 71/Number 1 January/February 2014
81
TABLE 1. Demographic and Experience Data of the 26 Participants With Complete Follow-Up According to Simulator Type Variables
FLS (n ¼ 13)
LapSim p (n ¼ 13) Values
26.5 (4.7) 27.1 (3.0) 0.697 Age (y)* Female sex 3 (23) 4 (31) 1.000 Right handed 12 (92) 10 (77) 0.593 Training level Student 6 (50) 4 (33) 0.496 Intern 0 1 (8) RMO 6 (50) 5 (42) Registrar 0 2 (17) Surgery performed 3 (23) 2 (15) 1.000 Number of surgeries performed 0 10 (77) 11 (92) 1.000 1 to 10 2 (15) 1 (8) 11 to 30 0 0 430 1 (8) 0 Surgery observed 13 (100) 13 (100) 1.000 Number of surgeries observed 0 0 0 0.371 1 to 10 2 (15) 1 (8) 11 to 30 9 (69) 6 (46) 430 2 (15) 6 (46) Surgeries assisted 11 (85) 11 (85) 1.000 Number of surgeries assisted 0 2 (15) 2 (15) 0.062 1 to 10 4 (31) 4 (31) 11 to 30 6 (46) 1 (8) 430 1 (8) 6 (46) Participant’s perceived level of competence at laparoscopic surgery† 1 9 (82) 8 (67) 0.784 2 1 (9) 3 (25) 3 1 (9) 1 (8) 4 0 0 5 0 0 2 (15) 3 (23) 1.000 Participation in activities involving fine motor skills‡ Music years 0 8 (72) 10 (77) 1.000 1 to 10 1 (9) 1 (8) 11 to 30 2 (18) 2 (15) Number of instruments played 0 8 (62) 10 (77) 0.551 1 4 (31) 1 (8) 2 1 (8) 1 (8) 3 0 0 4 0 1 (8) Gaming (h/wk) 0 2 (15) 5 (39) 0.691 ≤1 3 (23) 2 (15) 1.1 to 3 5 (39) 3 (23) 3.1 to 6 2 (15) 1 (8) 46 1 (8) 2 (15) Values in parentheses are percentages unless indicated otherwise. SD, standard deviation; RMO, resident medical officer. *Values are means (standard deviations). † Participants were required to self-assess their level of competency on a scale of 1 to 5 (1 ¼ novice to 5 ¼ expert). ‡ Fine motor skills were a predefined set of hobbies including sewing, knitting, crocheting, and model making. Percentages derived from available participant responses. 82
third assessment. The variation in participants’ scores followed a similar pattern; scores at the first and second follow-up assessments were significantly more variable than those at the end of the initial training period, but by the third assessment variability was no longer significantly different to that seen initially. Change Over Time The relationship between follow-up assessment scores and length of time was examined after attaining proficiency. Changes over time for each simulator are shown in Figure using local quadratic regression fits. Models for scores accounted for 65% of the FLS score variation and 50% of the LapSim score variation. For the FLS cohort, the spline term in days was not significant after attaining proficiency, indicating that the data are well described by the mean score alone with no change over time. This was not so for the LapSim scores, where the rapid initial decline in score followed by the gradual rise was significant as shown in Figure. These results are consistent with the previous analysis (Table 3).
DISCUSSION This study found that maintenance of basic laparoscopic skill levels differed according to the type of simulator training. Training with a low-fidelity FLS simulator resulted in relatively stable scores over the entire study period of 6 months, with little difference between initial and followup scores or their variability. In contrast, training with a VR LapSim stimulator resulted in more variable scores, with the mean scores at the first and second follow-up assessments being significantly less than those at proficiency (p o 0.001). This initial loss improved, so that by the third follow-up assessment, at approximately 6 months, the mean scores of LapSim-trained participants were not significantly different from those seen initially at proficiency. The initial decrease in skill levels of VR simulator–trained participants, which we found, is an accepted phenomenon that occurs when skills are not practiced over time11 and concurs with the findings of previous studies.12-14 Windsor and Zoha12 found that VR-trained participants were considerably worse at the second of the 2 training sessions conducted 1 month apart (statistical analyses were not reported). Sinha et al.13 reported that LapSim-trained participants had an initial decline in fine motor skill tasks after proficiency was first attained. However, they maintained most skills even after an extended period of time. Stefanidis et al.14 assessed the skills retention approximately 2 weeks and 7 months after having reached proficiency in a VR simulator or a video trainer and found that there was greater early performance decay for the VR simulator, but after the early performance decrement, there was no skill loss for both simulators.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
TABLE 2. Initial and Follow-Up Data for the 13 Participants in the FLS Training Cohort Who Completed All 3 Follow-Up Assessments Follow-Up Parameter
At Proficiency
Number of participants Days since proficiency achieved Mean Median Range Score Mean Median Range Standard deviation Coefficient of variation (%)
13 0 0 0 97.5 97.4 94.8 to 101.4 2.0 a 2.1
1
2
13
13
37 29 23 to 114
p Values
13
107 99 85 to 170
96.0 97.4 85.6 to 102.4 5.0 b 5.2
3
96.3 97.2 82.0 to 104.2 6.0 b 6.2
193 175 142 to 255 96.0 97.8 85.8 to 101.4 5.1 b 5.3
0.561 0.007*
*Values followed by the same letter are not significantly different at p ¼ 0.050.
Thus, although firm conclusions cannot be drawn because of the limited data, the results from our study suggest that a poorer retention of skills may be associated with VR simulation training, especially in the intermediate term, and that low-fidelity simulation may be preferable for the long-term retention of laparoscopic skills. The results of this research have important implications for surgical education. It demonstrates that participants who attained a predetermined level of proficiency during an initial training period may have a decrement of skills initially, but retain most of their skill set over an extended period of time. This suggests that continuous ongoing refresher training may not be required. However, to avoid the initial decline in proficiency seen soon after training, at least with VR simulation training, surgical educators may opt to run refresher courses shortly after the initial training period. Improved laparoscopic skills and retention of these skills would ultimately lead to improved patient safety. Although the current study assessed a larger group of participants compared to other published studies, there was still a large number of losses to follow-up. It is a
well-documented issue that the longer a study continues, the greater the rate of participant withdrawal or loss to follow-up. Although only 26 of the initial 41 participants (63%) who reached proficiency completed all 3 follow-up assessments, we believe this loss to follow-up had little negative effect on the results. The FLS-trained scores for any completed follow-up assessments for the “noncompleters” were not significantly different to the scores of those who completed all assessments. For the LapSim group, there were no significant differences between noncompleters and those who completed the study at proficiency or at the first assessment; however, the 8 noncompleters had a significantly lower mean score at the second assessment than those who completed all 3 assessments (mean scores ¼ 71.7 and 80.5, respectively; p ¼ 0.004). The reason for this is unclear; however, had all “noncompleters” data been included in the final analysis, their reduction in score would have further enhanced the drop-off in LapSim performance at the second assessment. Finally, it is important to acknowledge that the skills set tested were limited to simulator skills. In summary, basic laparoscopic skill maintenance was more
TABLE 3. Initial and Follow-Up Data for the 13 Participants in the LapSim Training Cohort Who Completed All 3 Follow-Up Assessments Follow-Up Parameter
At Proficiency
Number of participants Days since proficiency achieved Mean Median Range Score Mean Median Range Standard deviation Coefficient of variation (%)
13 0 0 0 87.0 a 86.0 82.1 to 93.3 3.3 a 3.7
1
2
3
13
13
13
35 32 28 to 58
94 94 72 to 112
198 195 156 to 250
75.9 b 73.8 56.2 to 89.4 10.7 b 14.0
80.5 bc 81.5 67.8 to 90.4 6.2 bc 7.8
84.8 ca 85.8 76.6 to 92.8 5.0 ca 5.9
p Values
o0.001* 0.008*
*Values followed by the same letter are not significantly different at p ¼ 0.050. Journal of Surgical Education Volume 71/Number 1 January/February 2014
83
room in blinded, randomised controlled trial. Stud Health Technol Inform. 2007;125:76-81. 4. Ahlberg G, Enochsson L, Gallagher AG, et al.
Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193:797-804. 5. Larsen CR, Søerensen JL, Grantcharov TP, et al. Effect
of virtual reality training on laparoscopic surgery: randomised controlled trial. B Med J. 2009;338:b1802. 6. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L,
Rosenberg J, Funch-Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91:146-150. 7. Dawe S, Windsor J, Cregan P et al. Surgical simulation FIGURE. Smoothed scores over time for FLS and LapSim participants after attaining proficiency at the initial training period. Shaded areas are the nonparametric bootstrap 95% confidence intervals. Rugs show times of follow-up testing for FLS (top) and LapSim (bottom) participants. Note that FLS and LapSim scores are not directly comparable.
consistently achieved after initial training using a FLS simulator than a VR LapSim simulator, though in the longer term skill levels were similar. Further studies are necessary to assess if the same pattern of skills retention is found when participants are assessed in a real operative setting after being trained on different modalities of simulators
for training: skills transfer to the operating room (update). ASERNIP-S Report 80. 2012 [November 2012]. Available at: http://www.surgeons.org/asernip-s. 8. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA,
Fayez R, Fried GM. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. Am J Surg. 2010;199: 115-120. 9. Seymour NE, Gallagher AG, Roman SA, et al. Virtual
reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236:458-463.
ACKNOWLEDGMENTS
10. Ritter EM, Scott DJ. Design of a proficiency-based
The funding for this research was provided by the Commonwealth Government of Australia, Department of Health and Ageing. The project was administered by the Royal Australasian College of Surgeons.
11. Arthur W Jr, Bennett W Jr, Stanush PL, McNelly TL.
skills training curriculum for the Fundamentals of Laparoscopic Surgery. Surg Innov. 2007;14:107-112. Factors that influence skill decay and retention: a quantitative review and analysis. Hum Perform. 1998;11:57-101. 12. Windsor JA, Zoha F. The laparoscopic performance of
REFERENCES 1. Johnson WR. Laparoscopic surgery: time for re-
evaluation. Med J Aust. 1996;165:355-356.
novice surgical trainees: testing for acquisition, loss, and reacquisition of psychomotor skills. Surg Endosc. 2005;19:1058-1063.
2. Grantcharov TP, Funch-Jensen P. Can everyone
13. Sinha P, Hogle NJ, Fowler DL. Do the laparoscopic
achieve proficiency with the laparoscopic technique? Learning curve patterns in technical skills acquisition Am J Surg. 2009;197:447-449.
skills of trainees deteriorate over time? Surg Endosc. 2008;22:2018-2025.
3. Cosman PH, High TJ, Shearer CJ, Merrett ND,
Biankin AV, Cartmill JA. Skills acquired on virtual reality laparoscopic simulators transfer into operating
84
14. Stefanidis D, Korndorffer JR, Sierra R, Touchard C,
Dunne B, Scott DJ. Skill retention following proficiency-based laparoscopic simulator training. Surgery. 2005;138:165-170.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
