Minimally Invasive Therapy. 2014;23:190–197
ORIGINAL ARTICLE
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
The use of virtual reality simulation to determine potential for endoscopic surgery skill acquisition
TOMER MANN1, LISA GILLINDER2, AMIR SZOLD3 1
Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel, 2Princess Alexandra Hospitel, Brisbane, Queensland, Australia, and 3Assia Medical Group, Tel Aviv, Israel
Abstract Background: Efficient acquisition of endoscopic technique is essential for high-level care in surgical practice. In contrast to similar substantial risk industries, there is no standard instrument capable of detecting the potential of surgical residency candidates to develop such skills. Material and methods: We used the Simbionix “Lapmentor” Virtual reality simulator basic skills tasks 1, 5 and 6 to establish baseline performance of 17 subjects lacking surgical experience, then divided them into two groups. One group trained on the Lapmentor, a validated trainer. The second group trained on a video box trainer using 3 FLS tasks, which correlate with real OR performance. After completing the training program, each group was tested on its training modality and correlations were sought between performance in the screening tasks and final scores in both groups. Results: Time in Lapmentor task 1 showed significant correlations with total FLS scores (R 0.807 P 0.015), in addition to other benchmark parameters. With the Lapmentor group, time on task 5 demonstrated correlation with itself on the final scores (R 0.794 P 0.011). Conclusions: Time in the Lapmentor task 1 demonstrates correlations with FLS scores, which translate to better OR performance. The Lapmentor thus shows potential to be used as a screening test for surgical talent.
Key words: Minimally invasive surgery, simulation, surgical education, training
Introduction Assessment of surgical skills is a challenging undertaking. Its necessity is comparable with other high-risk industries, such as air travel. Yet, screening candidates for surgical training is currently seldom attempted in the selection process for surgical residencies, as opposed to the elaborate evaluations applied in similar industries (1). Quantification of markers of psychomotor competence will become increasingly important as endoscopic techniques solidify their position as the gold standard treatment for a great number of conditions. Demonstration of features such as hand-eye coordination, spatial orientation, depth perception and hand dexterity could be used to identify natural endoscopic potential, much like the psychomotor tests taken by cadets for pilot courses in an air force are applied to select the proper trainees.
As a result of workload initiatives, training programs are shortening the de-facto training curriculum to meet system demands. The need for early identification of individuals most likely to achieve proficiency in a short period of time and produce a better overall performance using the challenging endoscopic techniques renders psychomotor potential evaluation of future surgeons essential. Since such evaluation is difficult and subjective, it is not routinely attempted. A recent study demonstrated a lack of correlation between self assessment and surgical dexterity, as well as an absence of self selection of surgical candidates for residency programs, as candidates are unaware of their true potential (1). Furthermore, when tested, internal medicine interns actually scored higher than surgical residency candidates in most simulation tasks, emphasizing the implications of the lack of selection criteria for technical proficiency skills (1).
Correspondence: A. Szold, Assia Medical Group, P.O. Box 58048, Tel Aviv 61580, Israel; E-mail: [email protected] ISSN 1364-5706 print/ISSN 1365-2931 online 2014 Informa Healthcare DOI: 10.3109/13645706.2014.894529
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
VR simulation to determine potential for endoscopic skill acquisition A validated instrument capable of reliably and accurately measuring parameters that predict which individuals are capable of steeper learning curves and higher performance level is not currently in existence. Such a device could potentially change resident selection and ultimately produce surgeons with better technical proficiency. The Lapmentor (Simbionix, Lod, Israel) VR simulator is designed to shorten training times in the OR by moving the learning environment to the lab. It is validated to distinguish between proficiency levels (2,3) and aid in improving endoscopic technique (4–6). It does so by objectively measuring multiple parameters of performance, such as economy of motion, total path length, and number of hand movements. These are recorded allowing the user to track their improvement. The purpose of this study was to assess the use of the Lapmentor for determining surgical potential, by establishing whether its properties could be utilised to predict ‘natural talent’ for surgery. Since the emergence of the Lapmentor and other virtual reality simulators at the beginning of the last decade, many of its attributes have been researched, including face (7) and construct validity (8,9). It has been shown to improve performances in animal models (10) as well as real O.R performance (11). It was demonstrated that camera training on the simulator is equally effective, while more time efficient, than training on real live camera at the OR (12), that the skills acquired are transferable to unfamiliar tasks (13), and that its use as pre-surgical “warm up” improves surgical outcomes (14,15). This field has been researched extensively in recent years and our knowledge base on virtual reality simulators in general, and the Lapmentor in particular, has grown substantially. Having that in mind, it is important to acknowledge that simulator technology is still developing, and present day virtual reality devices fall short of their intended potential as trainers and testing instruments. Current day simulators still require much more research to establish their abilities and limitations. Thus far, they have failed to show superiority to the simple video box trainers in the instruction of surgical techniques, despite their much higher cost (16,17). Furthermore, evidence is still lacking as to the transferability of skills to the operating room, as well as to which performance parameters actually reflect true skill (16). The Lapmentor has many of its tasks to date demonstrating limited construct validity (18), and so its role in surgical training is not yet well established. So far, our ability to assess surgical potential of inexperienced candidates, and their chances of developing proficiency in the shortest amount of time possible remains modest, as few works have tackled this
191
topic head on. In an era where endoscopy takes center stage in the world of surgery, the ability to estimate natural technical tendencies and manual aptitude for surgical training seems increasingly valuable. This study attempts to explore whether the Lapmentor is capable of objectively assessing surgical talent.
Material and methods Lapmentor VR simulator The Lapmentor VR laparoscopic surgical simulator is a high fidelity virtual reality simulator that features a haptic feedback system, among other means, to create a realistic training experience. It has been validated to distinguish between levels of proficiency (construct validity, 2, 3), as well as for improving learning of basic laparoscopic skills (4) and real OR performances (5,6). The simulator includes curriculums in laparoscopic basic skills, procedural tasks and full surgical procedures. Subjects interact with the simulated tissues with two generic laparoscopic instruments through a haptic feedback device. Instrument types (grasper, scissors, clip applier, diathermy, harmonic scalpel, endostapler, and hook) are selected through an onscreen menu. Video trainer A traditional video box trainer is comprised of a plastic box, off the shelf laparoscopic camera, graspers and scissors, and materials needed for performance of the fundamentals of laparoscopic surgery (FLS) tasks selected: Peg transfer kit, gauze, and endoloops. Subjects perform manipulations on items inside a box, using real laparoscopic equipment, while watching their performances on a video screen. The FLS protocol encompasses five standardized tasks, complete with an evaluation criteria and training curriculum, which has been extensively validated for demonstrating improvement, shortening training curves and improving real OR performances (19–23). The three FLS tasks we used .
. .
Peg transfer- transferring six pegs from one side of a pin board to the other, and back again, using alternating hands. Pattern cutting- cutting a circle drawn on gauze, using laparoscopic scissors. Endoloop- applying loops on a designated device, as closely to the target line as possible
192
T. Mann et al. The parameters measured were:
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
Study subjects Our subject group consisted of 17 students (four female, 13 male) ages 24–31, who had no surgical skills or previous exposure to surgical simulation. Recruiting was done by personal communication and responders were excluded if they had any endoscopic surgical or simulator experience. Subjects were assessed by the Lapmentor to measure each subject’s naïve ‘baseline’ performance, and then put through a training program, followed by measurement of their final achievements. All subjects followed a standardised strict training program to eliminate bias from differences in practice time, number of repeats or personal motivation. Since subjects had no prior experience it was expected that the difference measured after completion training would be attributable to inherent talent. Baseline performances were assessed using three of the Lapmentor basic skill tasks (numbers 1, 5 and 6) which were shown to have high construct validity (2): .
.
.
Task 1, camera manipulation: Subjects need to keep the camera focused on a moving ball and zoom in on targets which appear between fixed items on the screen. Task 5, clip application: Subjects need to apply clips on specific areas on tubes, filling up a pool, thus preventing it from filling up. These tubes need to be first pulled by a grasper to achieve applicable position. Task 6, object retrieval: Subjects need to utilize graspers in order to retrieve balls from under items on screen and place them in baskets.
Tasks All subjects were initially evaluated on the Lapmentor VR simulator as follows: Each subject performed all three tasks on the first session of training. Before commencing each task, subjects were given a basic demonstration in groups of five with an opportunity to ask questions. The demonstrations lasted three minutes per task, following which each subject was allowed three attempts at each task before measurements of baseline performance were taken. The total practice time for all three attempts at a task was 10–15 minutes. No further assistance was provided to the subjects once practice was finished. At the end of the familiarisation with each task, all 17 subjects performed the task and their results were saved on the simulator. This was considered the baseline level. In this manner all three basic skills tasks chosen (tasks no. 1, 5, 6, based on their high face and construct validity (2)) were performed by all subjects.
.
.
.
Task no. 1 – total time, number of correct hits, maintaining the horizontal view while using the 0 camera (%), total path length of camera (cm), average speed of camera movement (cm/sec), total number of camera shots, accuracy rate - target hits (%), the time the horizontal view is maintained (±15 ) while using the 0 camera. Task no. 5- total time, number of clipped ducts, number of movements of left instrument, number of movements of right instrument, average speed of left instrument movement (cm/sec), average speed of right instrument movement (cm/sec), total path length of left instrument (cm), total path length of left instrument (cm), number of lost clips, total number of clipping attempts, economy of movement - right instrument (%),economy of movement - left instrument (%),economy of movement - clipper (%), economy of movement -grasper (%), Ideal path length of clipper (cm), ideal path length of grasper (cm), total path length of clipper (cm), total path length of grasper (cm), accuracy rate applied clips (%), relevant path length - right instrument (cm), relevant path length - left instrument (cm), relevant path length - clipper(cm), relevant path length - grasper (cm). Task no. 6 - total time, number of exposed green balls that are collected, number of lost balls which miss the basket, number of lost balls which miss the basket, number of movements of right instrument, average speed of left instrument movement (cm/ sec), average speed of right instrument movement (cm/sec), total path length of left instrument (cm), total path length of right instrument (cm), ideal path length of right instrument (cm), ideal path length of left instrument (cm), economy of movement - right instrument (%), economy of movement - left instrument (%), relevant path length right instrument (cm), relevant path length - right instrument (cm).
After the initial session, subjects were divided into two groups with equal distribution of baseline performances. This was achieved by ranking performances in the following parameters: Time, accuracy rate, % horizontal (task 1); time, % correctly applied clips, % economy of motion (average for both hands) (task 5); time, number of correctly placed balls, total number of movements, % economy of motion right hand (task 6). All relative rankings in these parameters were summated, with economy of motion given a double weight for its seemingly superior face validity. This produced a relative rank (1–17) of all subjects. ‘Randomised block design’ statistical analysis tool was applied to create equal groups. Groups were
193
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
VR simulation to determine potential for endoscopic skill acquisition then randomly selected to train either exclusively on the Lapmentor or on the video box trainer, and ultimately each group was tested on the modality on which it had trained. The Lapmentor group training comprised of two sessions, two weeks apart, allocating 15 minutes for each task (1, 5 and 6), therefore total practice time per session was 45 minutes. As many repetitions as possible during this time were allowed. Following the end of the second training session, results were recorded for all three tasks to establish final scores. For the video box trainer group, a training program was devised using three FLS tasks: Peg transfer, endoloop application and pattern cutting. Each subject watched a guidance video prior to beginning practicing each task. Like the training program in the Lapmentor group, training was comprised of two 45-minute sessions held two weeks apart, during which each FLS task was given 15 minutes training time for maximum repetitions. At the end of the second session, subjects performed each of the tasks once more while being measured for performance. The parameters for establishing the final score were:Time for all three tasks, accuracy in millimetres for the endoloop and pattern cutting tasks, and number of dropped pegs for the peg transfer task.
Performance evaluation For the video group, the final score was calculated by summating time and errors, a lower score indicating a better performance. Statistical analysis using Pearson test was applied to determine whether correlation existed between initial screening scores on the Lapmentor and final post training scores in the FLS tasks. For the Lapmentor group, performance analysis was based on parameters shown to have construct validity (2), those being: Time & accuracy rate for task 1, time & speed for task no. 5, and time, speed, number of movements (as registration of left instrument was impaired, we used no. of movements of right instrument, instead of total movements) and total path for task 6. Correlation between initial screening and end results was then sought using Wilcoxon signed ranks test. Results A statistically significant correlation was found between time in Lapmentor task no. 1 and endoloop time, combined time for the peg transfer and pattern cutting, total time in all tasks, and total score (total time plus accuracy in all three tasks) (Table I, Figure 1)
Table I. Pearson’s correlations between video group performances in the screening tasks on the Lapmentor (rows) and the FLS tasks (columns). As evident, statistically significant correlations exist with time in Lapmentor task 1. No other parameter showed to be statistically significant. FLS Tasks
Lapmentor Task 1- time
Task 1-accuracy
R
Endoloop_time
Total time
Total errors
Combined time for pattern cutting and PEG transfer
Total scores
0.798(*)
0.796(*)
0.625
0.731(*)
0.807(*)
P
0.018
0.018
0.098
0.039
0.015
R
–0.269
–0.564
–0.629
–0.601
–0.586
0.115
0.127
P
0.520
0.145
0.095
Task 5-time
R
0.223
0.241
–
P
0.596
0.565
–
–
–
Task 5- average speed
R
0.166
0.012
–
–
–
P
0.694
0.978
–
–
–
Task 6- time
R
–0.045
–0.058
–
–
–
P
0.916
0.892
–
–
–
Task 6- average speed
R
–0.009
–0.425
–
–
–
–
–
P
0.983
0.293
–
–
–
Task 6-total movements
R
0.055
0.346
–
–
–
P
0.896
0.401
–
–
–
Task 6-total path
R
0.310
0.507
–
–
–
P
0.455
0.199
–
–
–
194
T. Mann et al. 30
25
Total score
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
20
15 R = 0.8 10
5
0 00:00
00:30
01:00
01:30 02:00 02:30 03:00 Time in minutes for task 1
03:30
04:00
Figure 1. Correlation between time on Lapmentor screening task no.1 and final score on the FLS tasks.
As one subject had very inconsistent performances, his elimination resulted in even stronger correlations between the abovementioned parameters, and emergence of additional ones, those being pattern cutting time, pattern cutting accuracy, peg transfer time, total accuracy, combined time and accuracy in peg transfer and pattern cutting. Eliminating this subject also resulted in the emergence of other predicting parameters- accuracy in task no. 1 demonstrated correlations to total accuracy, combined time and accuracy for peg transfer and pattern cutting. Total path in task 6 showed a correlation with pattern cutting time. (Table II, Figure 2). In the Lapmentor group, by contrast, no correlations between initial and final performances was noted, with the exception of time in task 5 at the beginning, predicting time in task 5 at the final test. (r = 0.79, p = 0.11). As for the Lapmentor’s capabilities as a trainer, multiple parameters show statistically significant improvement of performances, using Wilcoxon signed rank test. These parameters are: Time & accuracy in task 1, time in task 5, and time, speed, total path and total movements in task 6.(respectively: r-2.54, p = 0.11, r-2.52, p = 0.12, r-2.66, p = 0.08, r-2.66, p = 0.08, r-2.54, p = 0.11, r-2.31, p = 0.21). This is also demonstrated in the FDR analysis in Table III. Discussion The purpose of this study was to assess whether the Simbionix Lapmentor could be used as a screening
tool for surgeons who will have superior OR success. Such a task would require a long follow-up, with repeat re-evaluations of OR benchmarks, as residents progress in their surgical curriculum until reaching expert level. As this is beyond the scope of this study, we used a surrogate benchmark- the FLS scores, rationale being that these scores are proven to correlate well with real OR performance (19–23). The advantage of this study design was that it allows for a quick evaluation of whether a simple screening for surgical potential is available today, an area little explored thus far. The main limitation was our small sample size of 17 subjects, rendering it difficult to achieve statistical significance. Such significance, if achieved despite of this disadvantage, could imply real potential for the Lapmentor to be used for such purpose. We took 17 subjects without any previous endoscopic experience, and screened their potential by testing them on the Lapmentor. We then divided them into two groups, to be trained on two separate well established training modalities (video trainer, and the Lapmentor VR trainer). This was done in order to asses wether good performances on the screening tasks predict superior performances on these two different modalities, which would strengthen the Lapmentor’s face validity and allow for comparison. We let the first group train on the Lapmenor, repeatedly performing three of its basic skills tasks, while the other group trained on the video box trainer, performing three FLS tasks.
195
VR simulation to determine potential for endoscopic skill acquisition
Table II. Pearson’s correlation between performance on the screening tasks and final video box scores, after omitting a very inconsistent subject. FLS tasks Pattern cutting time 0.895(**)
0.782(*)
Endoloop_time
Total time
Total errors
Combined time for pattern cutting and PEG transfer
0.793(*)
0.880(**)
0.766(*)
0.856(*)
Total scores 0.900(**)
Task 1-time
R P
0.007
Task 1-accuracy
R
–0.745
P
0.055
0.039
0.593
0.114
Task 5-time
R
0.471
0.441
0.188
0.349
P
0.286
0.322
0.686
0.443
–
–
–
Task 5- average speed
R
–0.155
–0.670
0.141
0.067
–
–
–
P
0.739
0.099
0.763
0.887
–
–
–
Task 6-time
R
0.096
–0.082
–0.087
0.009
–
–
–
P
0.837
0.861
0.853
0.984
–
–
Task 6-average speed
R
–0.561
0.003
–0.045
–0.389
–
–
P
0.190
0.995
0.924
0.388
–
–
–
Task 6- total movements
R
0.563
0.568
0.015
0.451
–
–
–
P
0.188
0.183
0.975
0.309
–
–
–
Task 6- total path
R
0.789(*)
0.727
0.276
0.677
–
–
–
P
0.035
0.064
0.549
0.095
–
–
–
0.038 –0.780(*)
0.034
0.009
–0.247
–0.650
0.045 –0.791(*) 0.034 –
0.014
0.006
–0.732
–0.680
0.062
0.093
–
–
– -
As shown, almost all FLS parameters now demonstrate correlations to time in task 1, and do so with greater strength and significance. The first two columns represent two extra parameters demonstrated to be significant, which do not appear on Table I.
The groups were then tested, each on the modality on which it had trained. Results were analysed in order to establish whether any parameters of the screening tests could predict final scores in either group. As each of the modalities (Lapmentor, video
30 25
Total score
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
Lapmentor
Pattern cutting errors
20 15 R = 0.9 10 5 0 00:00 00:30 01:00 01:30 02:00 02:30 03:00 03:30 04:00 Time in minutes for task 1
Figure 2. Correlation between time on Lapmentor screening task no.1 and final score on the FLS tasks. Subject 1 omitted, showing a steeper slope.
box trainer) has been proven to distinguish between proficiency levels, a high final score on either should reflect real superior skill. Time on task no. 1 was found to be a good predictor of total FLS score (R = 0.8, P = 0.015), as well as of a few other intermediate benchmarks. While analysing the data we noticed one subject had very inconsistent results. Upon discussion with subject it turned out that he came to the last session when final scores were measured after having worked a double shift and was very tired. We than recalculated the data eliminating this subject, this time, the correlation between time on task 1 and final score grew stronger (R = 0.9, P = 0.06), and all other intermediate benchmark became statistically significant, with the exception of peg transfer time. Further than that, accuracy in task 1, which showed only a positive trend in the results for the entire group, became significant for predicting total FLS accuracy and combined pattern cutting and peg transfer results after eliminating this subject. Another correlation which emerged in this setting was path in task 6 predicting combined time for pattern cutting time.
196
T. Mann et al.
Table III. FDR analysis of multiple variables, parameters which demonstrate improvement with training on the Lapmentor are shown on the rightmost column, indicating its efficacy as a trainer.
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
Number
Test
parameter
Mean
std Err
t Value
Pr > {t}
P FDR
1
6
Number of movements- right instrument
–70
11.964
–5.85
0.000
0.001
2
6
Total time
–59
11.295
–5.26
0.001
0.001
3
6
Economy of movement- right instrument (%)
4
5
Economy of movement- grasper (%)
9
1.8136
5.19
0.001
0.001
16
3.1071
5.08
0.001
0.001
5
5
Total time
–52
10.807
–4.85
0.001
0.002
6
1
Total time
–55
11.635
–4.77
0.001
0.002
7
5
Economy of movement- right instrument (%)
17
3.6061
4.64
0.002
0.002
8
1
Accuracy rate- target hits (%)
18
4.4308
4.14
0.003
0.004
9
5
Number of movements-right instrument
–49
11.99
–4.08
0.004
0.004
10
6
Total path length- right instrument (cm)
–199
49.288
–4.03
0.004
0.004
–129
34.368
–3.74
0.006
0.006
–26
7.089
–3.63
0.007
0.007
11
6
Relevant path length- right instrument (cm)
12
1
The time the horizontal view is maintained (±15)
13
5
Accuracy rate- applied clips (%)
14
6
Number of movements- left instrument
15
5
16
1
17
5
Economy of movement- clipper (%)
18
5
Relevant path length- right instrument (cm)
18
5.2743
3.45
0.009
0.009
–3.27
0.011
0.012
3.748
3.25
0.012
0.012
0.8498
–3.14
0.014
0.015
2.99
0.017
0.018
–2.92
0.019
0.020
–42
12.738
Economy of movement- left instrument (%)
12
Total number of camera shots
–3 13 –214
4.3815 73.232
19
5
Total path length- right instrument (cm)
–226
78.168
–2.89
0.020
0.021
20
5
Relevant path length- grasper(cm)
–201
72.374
–2.78
0.024
0.025
21
5
Total path length- grasper (cm)
–213
77.574
–2.75
0.025
0.026
22
1
Total path length- camera (cm)
–129
49.778
–2.6
0.032
0.033
23
5
Number of movements- left instrument
–38
14.96
–2.56
0.034
0.035
24
6
Total path length- right instrument (cm)
–118
49.34
–2.39
0.044
0.046
These changes are most likely the result of our modest sample size, and may go to support an even greater screening potential with larger cohorts. As for the Lapmentor ability to predict future performances on itself, with the exception of time in task no.5 at the screening test predicting final performances on itself (R = 0.79, P = 0.01) no other correlations were found, signifying the simulator is very poor for predicting performance on itself. The reason for that is, in our opinion, the fact that the simulator is an efficient training instrument. This has been demonstrated repeatedly in the past, and again in this study, as exemplified by the FDR analysis (Table III). Since the vast majority of parameters show significant improvement while training and the same three basic skill tasks were used for screening, training and final testing, initial differences between subjects grew smaller with training, as most subjects succeeded in achieving prefect or near perfect scores by the end of training (for example- % target hit in
task 1, no. of dropped balls in task 6), obscuring initial differences. This theory could be verified by using a different modality to test final results (FLS tasks, OR evaluations) or a more advanced task on the simulator such as the advanced laparoscopic skills task or a procedural task.
Conclusion Time (and possibly accuracy) in the Lapmentor’s task no.1 predicts an individual candidate’s chances of succeeding in FLS tasks, and thus may correspond with better OR performances at a later stage. These results suggest the possible benefit of integrating VR simulators such as the Lapmentor in the selection process of surgical residents. More work with larger samples is needed to further establish the Lapmentor’s potential for use as a screening tool, and possibly find other parameters
VR simulation to determine potential for endoscopic skill acquisition which can be used to more accurately predict psychomotor potential as part of the acceptance criteria to surgical residencies. Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
References 1. Panait L, Larios JM, Brenes RA, Fancher TT, Ajemian MS, Dudrick SJ, et al. Surgical skills assessment of applicants to general surgery residency. J Surg Res. 2011;170:189–94. 2. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg. 2009;96:1086– 93. 3. Zhang A, Hünerbein M, Dai Y, Beller S. Construct validity testing of a laparoscopic surgery simulator (LAP Mentor): evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc. 2008;22:1440–4. 4. Kim TH, Ha JM, Cho JW, You YC, Sung GT. Assessment of the Laparoscopic Training Validity of a Virtual Reality Simulator (LAP Mentor). Korean J Urol. 2009;50:989–95. 5. Lucas SM, Zeltser IS, Bensalah K, Tuncel A, Jenkins A, Pearle MS, et al. Training on a virtual reality laparoscopic simulator improves performance of an unfamiliar live laparoscopic procedures. J Urol. 2008;180:2305–6. 6. Sroka G, Feldman LS, Vassiliou MC. Simulator training to proficiency improves laparoscopic performance in the operating room – a randomized controlled trial. Am J Surg. 2010; 199:115–20. 7. Ayodiji ID, Schijven M, Kamimowicz J, Greve JW. Face validation of the simbionix LAP mentor virtual reality training module and its applicability in the surgical curriculum. Surg Endosc. 2007;21:1641–9. 8. Zhang A, Hünerbein M, Dai Y, Schlag PM, Beller S. Construct validity testing of a laparoscopic surgery simulator (LAP Mentor): evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc. 2008;22:1440–4. 9. Yamaguchi S, Konishi K, Yasunaga T, Yoshida D, Kinjo N, Kobayashi K, et al. Simulator construct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulator. Surg Endosc. 2007;21:2253–7. 10. Andreatta PB, Woodrum DT, Birkmeyer JD, Yellamanchilli RK, Doherty GM, Gauger PG, et al. Laparoscopic Skillss are improved with LAP Mentor training: results of a randomized, double-blinded study. Ann Surg. 2006;243:854–60.
197
11. Beyer L, Troyer JD, Mancini J, Bladou F, Berdah SV, Karsenty G. Impact of laparoscopy simulator training on the technical skills of future surgeons in the Operating room: a prospective study. Am J Surg. 2011;202:265–72. 12. Franzeck FM, Rosenthal R, Muller MK, Nocito A, Wittich F, Maurus C, et al. Prospective randomized controlled trial of simulator-based versus traditional in-surgery Laparoscopic camera navigation training. Surg Endosc. 2012;26:235–41. 13. Lucas SM, Zeltser IS, Bensalah K, Tuncel A, Jenkins A, Pearle MS, et al. Training on a virtual reality laparoscopic simulator improves performance of an unfamiliar live laparoscopic procedure. J Urol. 2008;180:2305–6. 14. Moldovanu R, Târcoveanu E, Dimofte G, Lupas¸cu C, Bradea C. Preopertive warm-up using a virtual reality simulator. JSLS. 2011;15:533–8. 15. Lee JY, Mucksavage P, Derbi DC, Osann KE, Winfield HN, Kahol K, et al. Laparoscopic warm-up exercises improve performance of senior-level trainees during laparoscopic renal surgery. J Endourol. 2012;26:545–50. 16. Fairhurst K, Strickland A, Maddern G. The Lapsim virtual reality simulator: promising but not yet proven. Surg Endosc. 2011;25:343–55. 17. Diesen DL, Erhunmwunsee L, Bennet KM, Ben-David K, Yurcisin B, Ceppa EP, et al. Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic training of novices. J Surg Endosc. 2011;68: 282–9. 18. Andreatta PB, Woodrum DT, Gauger PG, Minter RM. Lapmentor matrices possess limited construct validity. Simul Healthc. 2008;3:16–25. 19. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez R, Fried G. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic Performance in the operating room – a randomized controlled trial. Am J Surg. 2010;199:115–20. 20. Korndorffer JR, Dunne JB, Sierra R, Stefanidis D, Touchard CL, Scott DJ. Simulator training for laparoscopic suturing using performance goals translates to the OR. J Am Coll Surg. 2005;201:23–9. 21. Stefanidis D, Sierra R, Korndorffer JR, Dunne JB, Markley S, Touchard C, et al. Intensive CME course training on simulators results in proficiency for laparoscopic suturing. Am J Surg. 2006;191:23–7. 22. Keyser EJ, Derossis AM, Antoniuk M, Sigman HH, Fried GM. A simplified simulator for the training and evaluation of laparoscopic skills. Surg Endosc. 2000;14: 149–53. 23. Fried GM, Derossis AM, Bothwell J, Sigman HH. Comparison of laparoscopic performance in vivo with performance measured in laparoscpopic Simulator. Surg Endosc. 1999;13: 1077–81.
ORIGINAL ARTICLE
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
The use of virtual reality simulation to determine potential for endoscopic surgery skill acquisition
TOMER MANN1, LISA GILLINDER2, AMIR SZOLD3 1
Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel, 2Princess Alexandra Hospitel, Brisbane, Queensland, Australia, and 3Assia Medical Group, Tel Aviv, Israel
Abstract Background: Efficient acquisition of endoscopic technique is essential for high-level care in surgical practice. In contrast to similar substantial risk industries, there is no standard instrument capable of detecting the potential of surgical residency candidates to develop such skills. Material and methods: We used the Simbionix “Lapmentor” Virtual reality simulator basic skills tasks 1, 5 and 6 to establish baseline performance of 17 subjects lacking surgical experience, then divided them into two groups. One group trained on the Lapmentor, a validated trainer. The second group trained on a video box trainer using 3 FLS tasks, which correlate with real OR performance. After completing the training program, each group was tested on its training modality and correlations were sought between performance in the screening tasks and final scores in both groups. Results: Time in Lapmentor task 1 showed significant correlations with total FLS scores (R 0.807 P 0.015), in addition to other benchmark parameters. With the Lapmentor group, time on task 5 demonstrated correlation with itself on the final scores (R 0.794 P 0.011). Conclusions: Time in the Lapmentor task 1 demonstrates correlations with FLS scores, which translate to better OR performance. The Lapmentor thus shows potential to be used as a screening test for surgical talent.
Key words: Minimally invasive surgery, simulation, surgical education, training
Introduction Assessment of surgical skills is a challenging undertaking. Its necessity is comparable with other high-risk industries, such as air travel. Yet, screening candidates for surgical training is currently seldom attempted in the selection process for surgical residencies, as opposed to the elaborate evaluations applied in similar industries (1). Quantification of markers of psychomotor competence will become increasingly important as endoscopic techniques solidify their position as the gold standard treatment for a great number of conditions. Demonstration of features such as hand-eye coordination, spatial orientation, depth perception and hand dexterity could be used to identify natural endoscopic potential, much like the psychomotor tests taken by cadets for pilot courses in an air force are applied to select the proper trainees.
As a result of workload initiatives, training programs are shortening the de-facto training curriculum to meet system demands. The need for early identification of individuals most likely to achieve proficiency in a short period of time and produce a better overall performance using the challenging endoscopic techniques renders psychomotor potential evaluation of future surgeons essential. Since such evaluation is difficult and subjective, it is not routinely attempted. A recent study demonstrated a lack of correlation between self assessment and surgical dexterity, as well as an absence of self selection of surgical candidates for residency programs, as candidates are unaware of their true potential (1). Furthermore, when tested, internal medicine interns actually scored higher than surgical residency candidates in most simulation tasks, emphasizing the implications of the lack of selection criteria for technical proficiency skills (1).
Correspondence: A. Szold, Assia Medical Group, P.O. Box 58048, Tel Aviv 61580, Israel; E-mail: [email protected] ISSN 1364-5706 print/ISSN 1365-2931 online 2014 Informa Healthcare DOI: 10.3109/13645706.2014.894529
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
VR simulation to determine potential for endoscopic skill acquisition A validated instrument capable of reliably and accurately measuring parameters that predict which individuals are capable of steeper learning curves and higher performance level is not currently in existence. Such a device could potentially change resident selection and ultimately produce surgeons with better technical proficiency. The Lapmentor (Simbionix, Lod, Israel) VR simulator is designed to shorten training times in the OR by moving the learning environment to the lab. It is validated to distinguish between proficiency levels (2,3) and aid in improving endoscopic technique (4–6). It does so by objectively measuring multiple parameters of performance, such as economy of motion, total path length, and number of hand movements. These are recorded allowing the user to track their improvement. The purpose of this study was to assess the use of the Lapmentor for determining surgical potential, by establishing whether its properties could be utilised to predict ‘natural talent’ for surgery. Since the emergence of the Lapmentor and other virtual reality simulators at the beginning of the last decade, many of its attributes have been researched, including face (7) and construct validity (8,9). It has been shown to improve performances in animal models (10) as well as real O.R performance (11). It was demonstrated that camera training on the simulator is equally effective, while more time efficient, than training on real live camera at the OR (12), that the skills acquired are transferable to unfamiliar tasks (13), and that its use as pre-surgical “warm up” improves surgical outcomes (14,15). This field has been researched extensively in recent years and our knowledge base on virtual reality simulators in general, and the Lapmentor in particular, has grown substantially. Having that in mind, it is important to acknowledge that simulator technology is still developing, and present day virtual reality devices fall short of their intended potential as trainers and testing instruments. Current day simulators still require much more research to establish their abilities and limitations. Thus far, they have failed to show superiority to the simple video box trainers in the instruction of surgical techniques, despite their much higher cost (16,17). Furthermore, evidence is still lacking as to the transferability of skills to the operating room, as well as to which performance parameters actually reflect true skill (16). The Lapmentor has many of its tasks to date demonstrating limited construct validity (18), and so its role in surgical training is not yet well established. So far, our ability to assess surgical potential of inexperienced candidates, and their chances of developing proficiency in the shortest amount of time possible remains modest, as few works have tackled this
191
topic head on. In an era where endoscopy takes center stage in the world of surgery, the ability to estimate natural technical tendencies and manual aptitude for surgical training seems increasingly valuable. This study attempts to explore whether the Lapmentor is capable of objectively assessing surgical talent.
Material and methods Lapmentor VR simulator The Lapmentor VR laparoscopic surgical simulator is a high fidelity virtual reality simulator that features a haptic feedback system, among other means, to create a realistic training experience. It has been validated to distinguish between levels of proficiency (construct validity, 2, 3), as well as for improving learning of basic laparoscopic skills (4) and real OR performances (5,6). The simulator includes curriculums in laparoscopic basic skills, procedural tasks and full surgical procedures. Subjects interact with the simulated tissues with two generic laparoscopic instruments through a haptic feedback device. Instrument types (grasper, scissors, clip applier, diathermy, harmonic scalpel, endostapler, and hook) are selected through an onscreen menu. Video trainer A traditional video box trainer is comprised of a plastic box, off the shelf laparoscopic camera, graspers and scissors, and materials needed for performance of the fundamentals of laparoscopic surgery (FLS) tasks selected: Peg transfer kit, gauze, and endoloops. Subjects perform manipulations on items inside a box, using real laparoscopic equipment, while watching their performances on a video screen. The FLS protocol encompasses five standardized tasks, complete with an evaluation criteria and training curriculum, which has been extensively validated for demonstrating improvement, shortening training curves and improving real OR performances (19–23). The three FLS tasks we used .
. .
Peg transfer- transferring six pegs from one side of a pin board to the other, and back again, using alternating hands. Pattern cutting- cutting a circle drawn on gauze, using laparoscopic scissors. Endoloop- applying loops on a designated device, as closely to the target line as possible
192
T. Mann et al. The parameters measured were:
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
Study subjects Our subject group consisted of 17 students (four female, 13 male) ages 24–31, who had no surgical skills or previous exposure to surgical simulation. Recruiting was done by personal communication and responders were excluded if they had any endoscopic surgical or simulator experience. Subjects were assessed by the Lapmentor to measure each subject’s naïve ‘baseline’ performance, and then put through a training program, followed by measurement of their final achievements. All subjects followed a standardised strict training program to eliminate bias from differences in practice time, number of repeats or personal motivation. Since subjects had no prior experience it was expected that the difference measured after completion training would be attributable to inherent talent. Baseline performances were assessed using three of the Lapmentor basic skill tasks (numbers 1, 5 and 6) which were shown to have high construct validity (2): .
.
.
Task 1, camera manipulation: Subjects need to keep the camera focused on a moving ball and zoom in on targets which appear between fixed items on the screen. Task 5, clip application: Subjects need to apply clips on specific areas on tubes, filling up a pool, thus preventing it from filling up. These tubes need to be first pulled by a grasper to achieve applicable position. Task 6, object retrieval: Subjects need to utilize graspers in order to retrieve balls from under items on screen and place them in baskets.
Tasks All subjects were initially evaluated on the Lapmentor VR simulator as follows: Each subject performed all three tasks on the first session of training. Before commencing each task, subjects were given a basic demonstration in groups of five with an opportunity to ask questions. The demonstrations lasted three minutes per task, following which each subject was allowed three attempts at each task before measurements of baseline performance were taken. The total practice time for all three attempts at a task was 10–15 minutes. No further assistance was provided to the subjects once practice was finished. At the end of the familiarisation with each task, all 17 subjects performed the task and their results were saved on the simulator. This was considered the baseline level. In this manner all three basic skills tasks chosen (tasks no. 1, 5, 6, based on their high face and construct validity (2)) were performed by all subjects.
.
.
.
Task no. 1 – total time, number of correct hits, maintaining the horizontal view while using the 0 camera (%), total path length of camera (cm), average speed of camera movement (cm/sec), total number of camera shots, accuracy rate - target hits (%), the time the horizontal view is maintained (±15 ) while using the 0 camera. Task no. 5- total time, number of clipped ducts, number of movements of left instrument, number of movements of right instrument, average speed of left instrument movement (cm/sec), average speed of right instrument movement (cm/sec), total path length of left instrument (cm), total path length of left instrument (cm), number of lost clips, total number of clipping attempts, economy of movement - right instrument (%),economy of movement - left instrument (%),economy of movement - clipper (%), economy of movement -grasper (%), Ideal path length of clipper (cm), ideal path length of grasper (cm), total path length of clipper (cm), total path length of grasper (cm), accuracy rate applied clips (%), relevant path length - right instrument (cm), relevant path length - left instrument (cm), relevant path length - clipper(cm), relevant path length - grasper (cm). Task no. 6 - total time, number of exposed green balls that are collected, number of lost balls which miss the basket, number of lost balls which miss the basket, number of movements of right instrument, average speed of left instrument movement (cm/ sec), average speed of right instrument movement (cm/sec), total path length of left instrument (cm), total path length of right instrument (cm), ideal path length of right instrument (cm), ideal path length of left instrument (cm), economy of movement - right instrument (%), economy of movement - left instrument (%), relevant path length right instrument (cm), relevant path length - right instrument (cm).
After the initial session, subjects were divided into two groups with equal distribution of baseline performances. This was achieved by ranking performances in the following parameters: Time, accuracy rate, % horizontal (task 1); time, % correctly applied clips, % economy of motion (average for both hands) (task 5); time, number of correctly placed balls, total number of movements, % economy of motion right hand (task 6). All relative rankings in these parameters were summated, with economy of motion given a double weight for its seemingly superior face validity. This produced a relative rank (1–17) of all subjects. ‘Randomised block design’ statistical analysis tool was applied to create equal groups. Groups were
193
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
VR simulation to determine potential for endoscopic skill acquisition then randomly selected to train either exclusively on the Lapmentor or on the video box trainer, and ultimately each group was tested on the modality on which it had trained. The Lapmentor group training comprised of two sessions, two weeks apart, allocating 15 minutes for each task (1, 5 and 6), therefore total practice time per session was 45 minutes. As many repetitions as possible during this time were allowed. Following the end of the second training session, results were recorded for all three tasks to establish final scores. For the video box trainer group, a training program was devised using three FLS tasks: Peg transfer, endoloop application and pattern cutting. Each subject watched a guidance video prior to beginning practicing each task. Like the training program in the Lapmentor group, training was comprised of two 45-minute sessions held two weeks apart, during which each FLS task was given 15 minutes training time for maximum repetitions. At the end of the second session, subjects performed each of the tasks once more while being measured for performance. The parameters for establishing the final score were:Time for all three tasks, accuracy in millimetres for the endoloop and pattern cutting tasks, and number of dropped pegs for the peg transfer task.
Performance evaluation For the video group, the final score was calculated by summating time and errors, a lower score indicating a better performance. Statistical analysis using Pearson test was applied to determine whether correlation existed between initial screening scores on the Lapmentor and final post training scores in the FLS tasks. For the Lapmentor group, performance analysis was based on parameters shown to have construct validity (2), those being: Time & accuracy rate for task 1, time & speed for task no. 5, and time, speed, number of movements (as registration of left instrument was impaired, we used no. of movements of right instrument, instead of total movements) and total path for task 6. Correlation between initial screening and end results was then sought using Wilcoxon signed ranks test. Results A statistically significant correlation was found between time in Lapmentor task no. 1 and endoloop time, combined time for the peg transfer and pattern cutting, total time in all tasks, and total score (total time plus accuracy in all three tasks) (Table I, Figure 1)
Table I. Pearson’s correlations between video group performances in the screening tasks on the Lapmentor (rows) and the FLS tasks (columns). As evident, statistically significant correlations exist with time in Lapmentor task 1. No other parameter showed to be statistically significant. FLS Tasks
Lapmentor Task 1- time
Task 1-accuracy
R
Endoloop_time
Total time
Total errors
Combined time for pattern cutting and PEG transfer
Total scores
0.798(*)
0.796(*)
0.625
0.731(*)
0.807(*)
P
0.018
0.018
0.098
0.039
0.015
R
–0.269
–0.564
–0.629
–0.601
–0.586
0.115
0.127
P
0.520
0.145
0.095
Task 5-time
R
0.223
0.241
–
P
0.596
0.565
–
–
–
Task 5- average speed
R
0.166
0.012
–
–
–
P
0.694
0.978
–
–
–
Task 6- time
R
–0.045
–0.058
–
–
–
P
0.916
0.892
–
–
–
Task 6- average speed
R
–0.009
–0.425
–
–
–
–
–
P
0.983
0.293
–
–
–
Task 6-total movements
R
0.055
0.346
–
–
–
P
0.896
0.401
–
–
–
Task 6-total path
R
0.310
0.507
–
–
–
P
0.455
0.199
–
–
–
194
T. Mann et al. 30
25
Total score
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
20
15 R = 0.8 10
5
0 00:00
00:30
01:00
01:30 02:00 02:30 03:00 Time in minutes for task 1
03:30
04:00
Figure 1. Correlation between time on Lapmentor screening task no.1 and final score on the FLS tasks.
As one subject had very inconsistent performances, his elimination resulted in even stronger correlations between the abovementioned parameters, and emergence of additional ones, those being pattern cutting time, pattern cutting accuracy, peg transfer time, total accuracy, combined time and accuracy in peg transfer and pattern cutting. Eliminating this subject also resulted in the emergence of other predicting parameters- accuracy in task no. 1 demonstrated correlations to total accuracy, combined time and accuracy for peg transfer and pattern cutting. Total path in task 6 showed a correlation with pattern cutting time. (Table II, Figure 2). In the Lapmentor group, by contrast, no correlations between initial and final performances was noted, with the exception of time in task 5 at the beginning, predicting time in task 5 at the final test. (r = 0.79, p = 0.11). As for the Lapmentor’s capabilities as a trainer, multiple parameters show statistically significant improvement of performances, using Wilcoxon signed rank test. These parameters are: Time & accuracy in task 1, time in task 5, and time, speed, total path and total movements in task 6.(respectively: r-2.54, p = 0.11, r-2.52, p = 0.12, r-2.66, p = 0.08, r-2.66, p = 0.08, r-2.54, p = 0.11, r-2.31, p = 0.21). This is also demonstrated in the FDR analysis in Table III. Discussion The purpose of this study was to assess whether the Simbionix Lapmentor could be used as a screening
tool for surgeons who will have superior OR success. Such a task would require a long follow-up, with repeat re-evaluations of OR benchmarks, as residents progress in their surgical curriculum until reaching expert level. As this is beyond the scope of this study, we used a surrogate benchmark- the FLS scores, rationale being that these scores are proven to correlate well with real OR performance (19–23). The advantage of this study design was that it allows for a quick evaluation of whether a simple screening for surgical potential is available today, an area little explored thus far. The main limitation was our small sample size of 17 subjects, rendering it difficult to achieve statistical significance. Such significance, if achieved despite of this disadvantage, could imply real potential for the Lapmentor to be used for such purpose. We took 17 subjects without any previous endoscopic experience, and screened their potential by testing them on the Lapmentor. We then divided them into two groups, to be trained on two separate well established training modalities (video trainer, and the Lapmentor VR trainer). This was done in order to asses wether good performances on the screening tasks predict superior performances on these two different modalities, which would strengthen the Lapmentor’s face validity and allow for comparison. We let the first group train on the Lapmenor, repeatedly performing three of its basic skills tasks, while the other group trained on the video box trainer, performing three FLS tasks.
195
VR simulation to determine potential for endoscopic skill acquisition
Table II. Pearson’s correlation between performance on the screening tasks and final video box scores, after omitting a very inconsistent subject. FLS tasks Pattern cutting time 0.895(**)
0.782(*)
Endoloop_time
Total time
Total errors
Combined time for pattern cutting and PEG transfer
0.793(*)
0.880(**)
0.766(*)
0.856(*)
Total scores 0.900(**)
Task 1-time
R P
0.007
Task 1-accuracy
R
–0.745
P
0.055
0.039
0.593
0.114
Task 5-time
R
0.471
0.441
0.188
0.349
P
0.286
0.322
0.686
0.443
–
–
–
Task 5- average speed
R
–0.155
–0.670
0.141
0.067
–
–
–
P
0.739
0.099
0.763
0.887
–
–
–
Task 6-time
R
0.096
–0.082
–0.087
0.009
–
–
–
P
0.837
0.861
0.853
0.984
–
–
Task 6-average speed
R
–0.561
0.003
–0.045
–0.389
–
–
P
0.190
0.995
0.924
0.388
–
–
–
Task 6- total movements
R
0.563
0.568
0.015
0.451
–
–
–
P
0.188
0.183
0.975
0.309
–
–
–
Task 6- total path
R
0.789(*)
0.727
0.276
0.677
–
–
–
P
0.035
0.064
0.549
0.095
–
–
–
0.038 –0.780(*)
0.034
0.009
–0.247
–0.650
0.045 –0.791(*) 0.034 –
0.014
0.006
–0.732
–0.680
0.062
0.093
–
–
– -
As shown, almost all FLS parameters now demonstrate correlations to time in task 1, and do so with greater strength and significance. The first two columns represent two extra parameters demonstrated to be significant, which do not appear on Table I.
The groups were then tested, each on the modality on which it had trained. Results were analysed in order to establish whether any parameters of the screening tests could predict final scores in either group. As each of the modalities (Lapmentor, video
30 25
Total score
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
Lapmentor
Pattern cutting errors
20 15 R = 0.9 10 5 0 00:00 00:30 01:00 01:30 02:00 02:30 03:00 03:30 04:00 Time in minutes for task 1
Figure 2. Correlation between time on Lapmentor screening task no.1 and final score on the FLS tasks. Subject 1 omitted, showing a steeper slope.
box trainer) has been proven to distinguish between proficiency levels, a high final score on either should reflect real superior skill. Time on task no. 1 was found to be a good predictor of total FLS score (R = 0.8, P = 0.015), as well as of a few other intermediate benchmarks. While analysing the data we noticed one subject had very inconsistent results. Upon discussion with subject it turned out that he came to the last session when final scores were measured after having worked a double shift and was very tired. We than recalculated the data eliminating this subject, this time, the correlation between time on task 1 and final score grew stronger (R = 0.9, P = 0.06), and all other intermediate benchmark became statistically significant, with the exception of peg transfer time. Further than that, accuracy in task 1, which showed only a positive trend in the results for the entire group, became significant for predicting total FLS accuracy and combined pattern cutting and peg transfer results after eliminating this subject. Another correlation which emerged in this setting was path in task 6 predicting combined time for pattern cutting time.
196
T. Mann et al.
Table III. FDR analysis of multiple variables, parameters which demonstrate improvement with training on the Lapmentor are shown on the rightmost column, indicating its efficacy as a trainer.
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
Number
Test
parameter
Mean
std Err
t Value
Pr > {t}
P FDR
1
6
Number of movements- right instrument
–70
11.964
–5.85
0.000
0.001
2
6
Total time
–59
11.295
–5.26
0.001
0.001
3
6
Economy of movement- right instrument (%)
4
5
Economy of movement- grasper (%)
9
1.8136
5.19
0.001
0.001
16
3.1071
5.08
0.001
0.001
5
5
Total time
–52
10.807
–4.85
0.001
0.002
6
1
Total time
–55
11.635
–4.77
0.001
0.002
7
5
Economy of movement- right instrument (%)
17
3.6061
4.64
0.002
0.002
8
1
Accuracy rate- target hits (%)
18
4.4308
4.14
0.003
0.004
9
5
Number of movements-right instrument
–49
11.99
–4.08
0.004
0.004
10
6
Total path length- right instrument (cm)
–199
49.288
–4.03
0.004
0.004
–129
34.368
–3.74
0.006
0.006
–26
7.089
–3.63
0.007
0.007
11
6
Relevant path length- right instrument (cm)
12
1
The time the horizontal view is maintained (±15)
13
5
Accuracy rate- applied clips (%)
14
6
Number of movements- left instrument
15
5
16
1
17
5
Economy of movement- clipper (%)
18
5
Relevant path length- right instrument (cm)
18
5.2743
3.45
0.009
0.009
–3.27
0.011
0.012
3.748
3.25
0.012
0.012
0.8498
–3.14
0.014
0.015
2.99
0.017
0.018
–2.92
0.019
0.020
–42
12.738
Economy of movement- left instrument (%)
12
Total number of camera shots
–3 13 –214
4.3815 73.232
19
5
Total path length- right instrument (cm)
–226
78.168
–2.89
0.020
0.021
20
5
Relevant path length- grasper(cm)
–201
72.374
–2.78
0.024
0.025
21
5
Total path length- grasper (cm)
–213
77.574
–2.75
0.025
0.026
22
1
Total path length- camera (cm)
–129
49.778
–2.6
0.032
0.033
23
5
Number of movements- left instrument
–38
14.96
–2.56
0.034
0.035
24
6
Total path length- right instrument (cm)
–118
49.34
–2.39
0.044
0.046
These changes are most likely the result of our modest sample size, and may go to support an even greater screening potential with larger cohorts. As for the Lapmentor ability to predict future performances on itself, with the exception of time in task no.5 at the screening test predicting final performances on itself (R = 0.79, P = 0.01) no other correlations were found, signifying the simulator is very poor for predicting performance on itself. The reason for that is, in our opinion, the fact that the simulator is an efficient training instrument. This has been demonstrated repeatedly in the past, and again in this study, as exemplified by the FDR analysis (Table III). Since the vast majority of parameters show significant improvement while training and the same three basic skill tasks were used for screening, training and final testing, initial differences between subjects grew smaller with training, as most subjects succeeded in achieving prefect or near perfect scores by the end of training (for example- % target hit in
task 1, no. of dropped balls in task 6), obscuring initial differences. This theory could be verified by using a different modality to test final results (FLS tasks, OR evaluations) or a more advanced task on the simulator such as the advanced laparoscopic skills task or a procedural task.
Conclusion Time (and possibly accuracy) in the Lapmentor’s task no.1 predicts an individual candidate’s chances of succeeding in FLS tasks, and thus may correspond with better OR performances at a later stage. These results suggest the possible benefit of integrating VR simulators such as the Lapmentor in the selection process of surgical residents. More work with larger samples is needed to further establish the Lapmentor’s potential for use as a screening tool, and possibly find other parameters
VR simulation to determine potential for endoscopic skill acquisition which can be used to more accurately predict psychomotor potential as part of the acceptance criteria to surgical residencies. Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by University of Washington on 01/08/15 For personal use only.
References 1. Panait L, Larios JM, Brenes RA, Fancher TT, Ajemian MS, Dudrick SJ, et al. Surgical skills assessment of applicants to general surgery residency. J Surg Res. 2011;170:189–94. 2. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg. 2009;96:1086– 93. 3. Zhang A, Hünerbein M, Dai Y, Beller S. Construct validity testing of a laparoscopic surgery simulator (LAP Mentor): evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc. 2008;22:1440–4. 4. Kim TH, Ha JM, Cho JW, You YC, Sung GT. Assessment of the Laparoscopic Training Validity of a Virtual Reality Simulator (LAP Mentor). Korean J Urol. 2009;50:989–95. 5. Lucas SM, Zeltser IS, Bensalah K, Tuncel A, Jenkins A, Pearle MS, et al. Training on a virtual reality laparoscopic simulator improves performance of an unfamiliar live laparoscopic procedures. J Urol. 2008;180:2305–6. 6. Sroka G, Feldman LS, Vassiliou MC. Simulator training to proficiency improves laparoscopic performance in the operating room – a randomized controlled trial. Am J Surg. 2010; 199:115–20. 7. Ayodiji ID, Schijven M, Kamimowicz J, Greve JW. Face validation of the simbionix LAP mentor virtual reality training module and its applicability in the surgical curriculum. Surg Endosc. 2007;21:1641–9. 8. Zhang A, Hünerbein M, Dai Y, Schlag PM, Beller S. Construct validity testing of a laparoscopic surgery simulator (LAP Mentor): evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc. 2008;22:1440–4. 9. Yamaguchi S, Konishi K, Yasunaga T, Yoshida D, Kinjo N, Kobayashi K, et al. Simulator construct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulator. Surg Endosc. 2007;21:2253–7. 10. Andreatta PB, Woodrum DT, Birkmeyer JD, Yellamanchilli RK, Doherty GM, Gauger PG, et al. Laparoscopic Skillss are improved with LAP Mentor training: results of a randomized, double-blinded study. Ann Surg. 2006;243:854–60.
197
11. Beyer L, Troyer JD, Mancini J, Bladou F, Berdah SV, Karsenty G. Impact of laparoscopy simulator training on the technical skills of future surgeons in the Operating room: a prospective study. Am J Surg. 2011;202:265–72. 12. Franzeck FM, Rosenthal R, Muller MK, Nocito A, Wittich F, Maurus C, et al. Prospective randomized controlled trial of simulator-based versus traditional in-surgery Laparoscopic camera navigation training. Surg Endosc. 2012;26:235–41. 13. Lucas SM, Zeltser IS, Bensalah K, Tuncel A, Jenkins A, Pearle MS, et al. Training on a virtual reality laparoscopic simulator improves performance of an unfamiliar live laparoscopic procedure. J Urol. 2008;180:2305–6. 14. Moldovanu R, Târcoveanu E, Dimofte G, Lupas¸cu C, Bradea C. Preopertive warm-up using a virtual reality simulator. JSLS. 2011;15:533–8. 15. Lee JY, Mucksavage P, Derbi DC, Osann KE, Winfield HN, Kahol K, et al. Laparoscopic warm-up exercises improve performance of senior-level trainees during laparoscopic renal surgery. J Endourol. 2012;26:545–50. 16. Fairhurst K, Strickland A, Maddern G. The Lapsim virtual reality simulator: promising but not yet proven. Surg Endosc. 2011;25:343–55. 17. Diesen DL, Erhunmwunsee L, Bennet KM, Ben-David K, Yurcisin B, Ceppa EP, et al. Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic training of novices. J Surg Endosc. 2011;68: 282–9. 18. Andreatta PB, Woodrum DT, Gauger PG, Minter RM. Lapmentor matrices possess limited construct validity. Simul Healthc. 2008;3:16–25. 19. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez R, Fried G. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic Performance in the operating room – a randomized controlled trial. Am J Surg. 2010;199:115–20. 20. Korndorffer JR, Dunne JB, Sierra R, Stefanidis D, Touchard CL, Scott DJ. Simulator training for laparoscopic suturing using performance goals translates to the OR. J Am Coll Surg. 2005;201:23–9. 21. Stefanidis D, Sierra R, Korndorffer JR, Dunne JB, Markley S, Touchard C, et al. Intensive CME course training on simulators results in proficiency for laparoscopic suturing. Am J Surg. 2006;191:23–7. 22. Keyser EJ, Derossis AM, Antoniuk M, Sigman HH, Fried GM. A simplified simulator for the training and evaluation of laparoscopic skills. Surg Endosc. 2000;14: 149–53. 23. Fried GM, Derossis AM, Bothwell J, Sigman HH. Comparison of laparoscopic performance in vivo with performance measured in laparoscpopic Simulator. Surg Endosc. 1999;13: 1077–81.
