775 C OPYRIGHT Ó 2015

BY

T HE J OURNAL

OF

B ONE

AND J OINT

S URGERY, I NCORPORATED

Topics in Training Testing Basic Competency in Knee Arthroscopy Using a Virtual Reality Simulator Exploring Validity and Reliability Mads Emil Jacobsen, MS, Morten Jon Andersen, MD, Claus Ol Hansen, MD, and Lars Konge, MD, PhD Investigation performed at the Centre for Clinical Education (CEKU) at Rigshospitalet, Copenhagen, Denmark

Background: Diagnostic knee arthroscopy is a common procedure that orthopaedic residents are expected to learn early in their training. Arthroscopy requires a different skill set from traditional open surgery, and many orthopaedic residents feel less prepared for arthroscopic procedures. Virtual reality simulation training and testing provide an opportunity to ensure basic competency before proceeding to supervised procedures in patients. Methods: Twenty-six physicians (thirteen novices and thirteen experienced arthroscopic surgeons) were voluntarily recruited to perform a test consisting of five arthroscopic procedures on a knee arthroscopy simulator. Performance was evaluated by obtaining predefined metrics from the simulator for each procedure, and z-scores, describing suboptimal performance, were calculated from the metrics. The intercase reliability of the simulator metrics was explored by calculating an intraclass correlation coefficient. Finally, a pass-or-fail standard was set with use of the contrasting groups method, and the consequences of the pass-or-fail standard were explored. Results: One procedure was excluded from the final test because of a lack of validity. The total Z-scores for the four procedures included in the final test showed an intercase reliability of 0.87 (95% confidence interval, 0.78 to 0.93). The total mean z-score (and standard deviation) was 38.6 ± 27.3 points for the novices and 0.0 ± 9.1 points for the experienced surgeons (p < 0.0005). The pass-or-fail standard was set at a total z-score of 15.5 points, resulting in two of the novices passing the test and a single experienced surgeon failing the test. Conclusions: By combining four procedures on a virtual reality arthroscopy simulator, it was possible to create a valid, reliable, and feasible test of basic arthroscopic competency and to establish a credible pass-or-fail standard. Clinical Relevance: The simulation-based test and pass-or-fail standard could aid in assessing and ensuring basic competency of future orthopaedic residents before proceeding to supervised procedures in patients.

Peer Review: This article was reviewed by the Editor-in-Chief and one Deputy Editor, and it underwent blinded review by two or more outside experts. It was also reviewed by an expert in methodology and statistics. The Deputy Editor reviewed each revision of the article, and it underwent a final review by the Editor-in-Chief prior to publication. Final corrections and clarifications occurred during one or more exchanges between the author(s) and copyeditors.

Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. None of the authors, or their institution(s), have had any financial relationship, in the thirty-six months prior to submission of this work, with any entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. Also, no author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.

J Bone Joint Surg Am. 2015;97:775-81

d

http://dx.doi.org/10.2106/JBJS.N.00747

776 TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 97 -A N U M B E R 9 M AY 6, 2 015 d

d

d

Knee arthroscopy is one of the most common orthopaedic procedures1. Arthroscopy requires a different skill set from traditional open orthopaedic surgery, as the use of the arthroscope and entry through portals require good hand-eye coordination, the ability to triangulate, and technical dexterity. Madan and Pai2 found that orthopaedic residents feel less prepared for arthroscopic procedures than open surgery. A challenge for the arthroscopic education of future orthopaedic surgeons is the increased focus on production and cost-effectiveness in the operating theater and decreased working hours for residents. These challenges leave limited room for the traditional principle of master-apprentice education3-5. In addition, the main drawback of the traditional master-apprentice model of surgical training is the fact that inexperienced trainees have a higher risk of complications, such as damage to chondral surfaces, early in the learning curve6-8. Virtual reality simulators have been proposed as a tool by which residents can acquire the basic arthroscopic skills before performing supervised procedures on patients3,4,6. Reliable and valid tests with credible pass-or-fail standards are necessary to ensure the basic competency of trainees before they are allowed to proceed to supervised procedures on patients9. Furthermore, testing at the end of training has been shown to improve motivation and to enhance learning10. The aims of this study were to create a test, to explore the reliability of the test, and to collect evidence of validity. In addition, we wanted to establish a credible pass-or-fail standard and to explore the consequences of this standard. Our research questions were: (1) Are the metrics provided by the simulator able to discriminate between novices and experienced arthroscopic surgeons? (2) How many proce-

T E S T I N G B A S I C C O M P E T E N C Y I N K N E E A RT H R O S C O P Y U S I N G A V I RT UA L R E A L I T Y S I M U L AT O R

dures should be performed on the simulator to ensure adequate reliability? (3) Can a credible pass-or-fail standard for the test be established? Materials and Methods The simulator used in this study was the ARTHRO Mentor (Simbionix, Airport 11 City, Israel) . The simulator consists of a stand with a fiberglass model of a right knee, connected to a computer and two robot arms (PHANTOM Omni; Sensable, Wilmington, Massachusetts) (Fig. 1). The knee has an anteromedial portal and an anterolateral portal and can be manipulated to flexion and extension as well as varus and valgus positions. The two robot arms mimic the surgical tools, as well as generating active haptic feedback. The endoscopic picture, along with the surgical instruments, is virtually generated on a screen connected to the computer. The whole stand can be adjusted in height. The task to be completed is written in English at the bottom of the screen. To avoid bias by unequal language skills, which is construct-irrelevant variance, all tasks were translated into written Danish and the supervisor of the tests (M.E.J.) ensured that the translations followed the specific task. Based on pilot testing by an expert arthroscopic surgeon (C.O.H.), we developed a test consisting of five different knee arthroscopic procedures included in the simulator software. Before the test, all participants were allowed to familiarize themselves with the simulator by completing two tasks: one locating and focusing on targets using the scope inside a virtual operating theater, and one locating and probing randomly placed blue spheres within the knee anatomy, using the arthroscope and a probe. The standardized warm-up was halted either at the completion of the two tasks or after a maximum total of ten minutes. The time limit of ten minutes was chosen on the basis of pilot testing and by taking into account the total time consumption, which we wanted to keep as close to one hour as possible, ensuring the feasibility of the test. The first procedure of the test, procedure 1, was a diagnostic arthroscopy, using only the arthroscope through the anterolateral portal. With the scope, the participant had to locate ten blue spheres at defined anatomical landmarks inside the knee, corresponding to a diagnostic arthroscopy. Only one sphere would be visible at a time, and the path in which to locate it was fixed. When a sphere was located and in focus, it would turn green. After two

Fig. 1

The ARTHRO Mentor virtual reality knee arthroscopy simulator (left panel) and the virtually generated arthroscopic picture (right panel).

777 TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 97 -A N U M B E R 9 M AY 6, 2 015 d

d

d

T E S T I N G B A S I C C O M P E T E N C Y I N K N E E A RT H R O S C O P Y U S I N G A V I RT UA L R E A L I T Y S I M U L AT O R

TABLE I Demographic Characteristics Characteristics Age* (yr)

Novices (N = 13)

Experienced Surgeons (N = 13)

29 (25 to 40)

51 (36 to 58)

Sex† Female

4

1

Male

9

12

Time as an orthopaedic surgery resident‡ (mo) Time as an orthopaedic surgeon including residency‡ (mo) No. of supervised knee arthroscopies performed# No. of unsupervised knee arthroscopies performed#

11 ± 16

NA§

NA§

175 ± 108

1.8 (0 to 9)

NA§

0

1542 (250 to 5000)

*The values are given as the median, with the range in parentheses. †The values are given as the number of surgeons. ‡The values are given as the mean and the standard deviation. §NA = not available. #The values are given as the mean, with the range in parentheses.

seconds, the sphere would disappear, and the next sphere would appear. The procedure was completed when all ten spheres had been located and in focus for two seconds. The second procedure, procedure 2, was a probe examination of a medial bucket-handle lesion, with the scope in the anterolateral portal and the probe in the anteromedial portal. Different parts of the medial meniscus, as well as the medial femoral condyle and tibial plateau, had to be probed in a fixed order. When probing the correct landmark, it would turn red, and, after two seconds, a new landmark was to be probed. The third procedure, procedure 3, was a probe examination of a lateral partial discoid meniscus. Except for the scope being inserted in the anteromedial portal and the probe being inserted in the anterolateral portal, the procedure was identical to the second procedure. The fourth procedure, procedure 4, was a probe examination of the whole knee. Six blue spheres, as in procedure 1, had to be located and probed. As in procedure 1, only one of the blue spheres was visible at a time, and their location was fixed. The location of the spheres was the same as in procedure 1, but two spheres in the medial gutter and two spheres in the lateral gutter were omitted. The fifth procedure, procedure 5, was the only therapeutic procedure included: a resection of a horizontal tear in the medial meniscus. With the scope in the anterolateral portal and a grasper in the anteromedial portal, the participant had to resect as much of the meniscus as he or she thought appropriate. When the participant was satisfied with the resection, the procedure was terminated. Procedure 1 had three simulator metrics: time, camera distance, and camera roughness. In addition to the three simulator metrics that were registered in procedure 1, procedures 2 through 4 had two additional simulator metrics: probe distance and probe roughness. Simulator metrics in procedure 5 included time, camera distance, camera roughness, grasper distance, open grasper distance, grasper roughness, open grasper roughness, the amount of the resected tear, and the percentage of the remaining meniscus. All of the simulator metrics for the distance covered describe the total length in millimeters that the tip of the instrument has traveled inside the knee, as measured by the haptic devices and the simulator software. Simulator metrics for roughness describe the number of collisions with cartilage that would have caused damage, as defined by the simulator company and measured by the haptic devices and simulator software. All mentioned simulator metrics, apart from the amount of the resected tear, describe suboptimal performance; therefore, a low score indicates a good performance. Twenty-six physicians, separated into two groups, were included in the study to explore the reliability and validity of the test: thirteen novices, consisting of orthopaedic interns or residents who had never before performed unsupervised knee arthroscopies, were in the first group, and thirteen experienced orthopaedic surgeons, each working at one of two specialized arthroscopic centers and having performed a minimum of 200 knee arthroscopies, were in the second group. Two of the experienced surgeons had briefly tried a

different arthroscopic simulator during an orthopaedic conference, whereas none of the novices had had any prior simulator experience. Recruitment was done by invitation. Informed consent was given by all participants prior to participation and no ethical approval was necessary according to the National Ethics Committee. All tests were performed on the same simulator in the same setting. The principal researcher (M.E.J.) operated the computer and supervised all tests. During the standardized warm-up, the principal researcher was standing by to help with the handling of the simulator. During the test, each procedure was explained before it began, but no assistance was given during the procedures. All information given to participants was previously written to prevent bias. All data from the tests were extracted from the simulator and imported to SPSS Statistics (version 22; IBM, Armonk, New York) for statistical analysis. Differences in simulator metrics were considered significant at p < 0.05. Independent-samples t tests were used to compare the performance of the novice group with that of the experienced surgeons group. Each of the numerical simulator metrics in each procedure was tested for significant differences between the two groups. The Levene test for equality of variances was performed; where equal variances could be assumed (p ‡ 0.05), we applied the Student t test, and where equal variances could not be assumed (p < 0.05), we applied the Welch t test. Each simulator metric from procedures 1 through 4 was converted into a z-score calculated from the mean value and the standard deviation of the experienced surgeons group. By adding these z-scores, we were able to create one combined z-score for each procedure as well as a total z-score for the whole test. Z-scores for each simulator metric from procedures 1 through 4 were analyzed further with respect to intercase reliability by calculating an intraclass correlation coefficient. The total z-score distributions for the novices and the experienced surgeons were then plotted with use of the contrasting 12 groups method . The pass-or-fail standard was set at the intersection between the distributions of the two groups, and the consequences of the pass-or-fail standard were explored.

Source of Funding There was no external funding for the study. The study was conducted completely independent from the simulator company.

Results Table I shows the demographic characteristics of the twenty-six participants. Table II shows the performances of the novices and the experienced arthroscopic surgeons in procedures 1 through 4. In procedure 1, all simulator metrics showed significant differences between novices and experienced surgeons.

778 TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 97 -A N U M B E R 9 M AY 6, 2 015 d

d

d

T E S T I N G B A S I C C O M P E T E N C Y I N K N E E A RT H R O S C O P Y U S I N G A V I RT UA L R E A L I T Y S I M U L AT O R

TABLE II Procedure Performance for Novices and Experienced Arthroscopic Surgeons on a Virtual Reality Arthroscopy Simulator Procedure and Simulator Metrics

Novices*

Experienced Surgeons*

P Value

4637 ± 1508 26.4 ± 7.7

1876 ± 1610 14.3 ± 9.9

Testing basic competency in knee arthroscopy using a virtual reality simulator: exploring validity and reliability.

Diagnostic knee arthroscopy is a common procedure that orthopaedic residents are expected to learn early in their training. Arthroscopy requires a dif...
752KB Sizes 3 Downloads 11 Views