e177(1) C OPYRIGHT Ó 2013
BY
T HE J OURNAL
OF
B ONE
AND J OINT
S URGERY, I NCORPORATED
Topics in Training Assessing Competence of Orthopaedic Residents: The Reliability and Validity of an Objective Structured Clinical Examination After a Sports Medicine Rotation Tim Dwyer, MBBS, FRACS, FRCSC, John S. Theodoropoulos, MD, MSC, FRCSC, Jodi Herold, PhD, Patrick Henry, MD, FRCSC, David Wasserstein, MD, MSc, FRCSC, M. Lucas Murnaghan, MD, MEd, FRCSC, Veronica Wadey, MD, MA, FRCSC, Brian Hodges, MD, MEd, FRCPC, PhD, John Semple, MD, MSc, FRCSC, and Darrell Ogilvie-Harris, MD, MSc, FRCSC Investigation performed at Women’s College Hospital, University of Toronto, Toronto, Ontario, Canada
The traditional method of orthopaedic training in Canada utilizes a time-based system, combined with a formal exit examination. Increasing interest in competency-based curriculum has raised the issue of how to test in-training competence. Currently, the most commonly used method is the In-Training Evaluation Report (ITER), which is known to be relatively subjective in nature; the true establishment of competence would likely benefit from the addition of an objective assessment1,2. The definition of competence varies, but relates to an individual’s ability to perform in the workplace to the required standard3. In surgery, competence is required in many areas, including surgical skill and clinical decision-making. Currently, orthopaedics is lacking a reproducible, objective measurement of resident competence. Objective Structured Clinical Examinations (OSCEs) are an important aspect of certifying examinations in many countries. First developed in the late 1970s, an OSCE consists of multiple, timed stations at which each candidate is faced with a discrete patient or case-based clinical task and is evaluated in an objective and structured way; both the task and the assessment
are standardized4-6. An OSCE is especially suited to testing aspects of the role of medical expert4,7,8. To our knowledge, there is very little published literature on the validity and reliability of OSCEs in orthopaedics. Recently, Beran et al. used an OSCE to assess history-taking and physical examination skills of orthopaedic residents, which served to highlight deficiencies in resident knowledge9. The ability to formulate valid and reliable in-training OSCEs may become an essential tool in competency-based orthopaedic training and may be used to assess resident competence regularly throughout training, to identify residents with knowledge deficits compared with their peers, and to allow appropriate remediation. A compulsory module at our university is a continuous three-month sports medicine rotation, in which residents are expected to acquire the medical knowledge to deal with common sports medicine conditions. Most of the residents undertake two sports medicine rotations, one as a junior resident and one as a senior resident. The majority of exposure to sports medicine occurs during these rotations. We hypothesized that a sports medicine OSCE would demonstrate sufficient
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. One or more of the authors, or his or her institution, has had a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. No author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
J Bone Joint Surg Am. 2013;95:e177(1-9)
d
http://dx.doi.org/10.2106/JBJS.M.00148
e177(2) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
TABLE I Number of Residents in Each Postgraduate Year Postgraduate Year
No. of Residents (N = 43)
1
8
2
6
3
11
4
7
5
11
TABLE II Group Scores for Each of the Six OSCE Stations (N = 43) Station
Group Scores*
1. Meniscal tear
61.5 ± 11.1
2. Rotator cuff tear
49.9 ± 11.7
3. Cam lesion of the hip and a labral tear
48.7 ± 10.5
4. ACL rupture
53.7 ± 15.3
5. Posterior shoulder dislocation
67.2 ± 14.1
6. Ankle instability
61.2 ± 12.5
*The values are given as the mean and the standard deviation in percentage.
reliability and validity to be used for orthopaedic resident in-training assessment. Materials and Methods Examination Development An established methodology was followed to create the OSCE. Orthopaedic Knowledge Update: Sports Medicine 4 was designated as the defined body of 10 knowledge, and assigned as required reading . This establishment of a defined curriculum enabled a clear set of learning objectives to be created, with validation 6 by an expert panel . The majority of the chapters in that text were discussed at resident teaching sessions with a staff surgeon, held on average three times a week; all scenarios tested in the OSCE were covered in that text. A blueprint of clinical problems was then created, with a focus on the CanMEDS (the Royal College of Physicians and Surgeons of Canada’s physician framework for medical 11 education) role of medical expert . The blueprint utilized six domains of medical expert thought to be pertinent: history, examination, image interpretation
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
(investigations and clinical photos), clinical decision-making, consent, and surgical technique. An hour-long OSCE with six stations was created; the OSCE was deliberately kept short to maximize feasibility. Six stations were written; each station had a duration of ten minutes and was based on a common clinical scenario. Scenarios included a fifty-one-year-old man with a traumatic rotator cuff tear, an eighteen-year-old male soccer player with a bucket-handle meniscus tear, a twenty-one-year-old man with a cam lesion of the hip and a labral tear, a seventeen-year-old boy with a torn anterior cruciate ligament, a sixtyseven-year-old woman with a posterior shoulder dislocation, and a twentysix-year-old female volleyball player with lateral ligament instability of the ankle. Computer-based stations were created, with relevant clinical photographs, imaging, and intraoperative photographs displayed on a screen. At each station, the examiner asked a series of predetermined questions based on the clinical scenario and the images. Candidates’ verbal answers were scored with use of a binary checklist assessing performance (mentioned or not mentioned and correct or incorrect). The OSCE questions and binary checklists underwent formal content review at a series of focus groups of orthopaedic surgeons to establish content validity. The knowledge standard was set at the level of the Canadian orthopaedic certifying examination. Scoring was performed in combination with a single global rating of performance at each station, as global ratings may have a higher degree of reliability and may be more sensitive to levels of training than checklists 12-14 alone . To increase objectivity, examiners assigned the global rating independently of the checklist score, in which 1 indicated clear fail, 2 indicated borderline, 3 indicated pass, 4 indicated good pass, and 5 indicated excellent 15-17 pass . The OSCE was administered in trials with two residents of different postgraduate years. Examiners were also trained in the use and application of the checklist and global ratings. Of the six examiners, four (T.D., D.O-H., P.H., and J.S.T.) were both authors of this study and members of the focus group and were very familiar with each scenario and its checklist. With regard to the examiners who were not part of the focus group, the principal investigator (T.D.) spent time discussing each scenario, the checklist, and the global rating with these examiners in detail, addressing any queries or concerns. All examiners were instructed that a pass on the global rating should be equivalent to a pass at an orthopaedic certifying examination level, irrespective of the year of training.
Study Design The OSCE was offered to all orthopaedic residents, postgraduate year 1 (PGY1) through postgraduate year 5 (PGY5), at this institution. Participation was voluntary and written consent was obtained from each resident prior to participation. Examiners at each station were either orthopaedic surgeons or fellows; the same examiner marked each station for the duration of the study, conducted over a two-week period. It was not possible to blind all examiners to the year of training of the residents. However, examiners were asked to disregard the year of training when assessing performance.
TABLE III Individual Station Parameters Station 1. Meniscal tear
Cronbach Alpha If Item Deleted
Corrected Item-Total Correlation
0.92
0.57
2. Rotator cuff tear
0.88
0.83
3. Cam lesion of the hip and a labral tear
0.90
0.68
4. ACL rupture
0.88
0.83
5. Posterior shoulder dislocation
0.89
0.81
6. Ankle instability
0.88
0.82
e177(3) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
TABLE IV Inter-Item Reliability of Subscores and Effect of the Postgraduate Year on Performance Domains
Domain History Examination
No. of Items
Cronbach Alpha Coefficient
P Value for the ANOVA for Effect of Postgraduate Year
Post Hoc Tests*
4
0.49
0.993
No difference
12
0.77
0.001
PGY4 and PGY5 were greater than PGY1, PGY2, and PGY3
Image interpretation
24
0.86
0.000
PGY5 was greater than PGY1 and PGY2
Clinical decision-making
20
0.82
0.000
PGY5 was greater than PGY1, PGY2, and PGY3
Consent
5
0.50
0.019
PGY5 was greater than PGY1
Surgical technique
6
0.78
0.000
PGY5 was greater than PGY1, PGY2, and PGY3
*Significance was set at p < 0.05.
Information was collected regarding the number of sports medicine rotations that each resident had previously undertaken, as well as which residents had undertaken a sports medicine rotation in the last six months. During this period, a competency-based curriculum model had been used, with a set curriculum and exit examinations in the form of OSCE examinations; only some of the residents had been exposed to the new curriculum. After the examination, candidates were invited to provide feedback on the examination via a survey, using a 5-point Likert scale.
Statistical Analysis All data were de-identified, were entered into an Excel spreadsheet, and were analyzed with use of SPSS version 19 (Armonk, New York). The checklists, global ratings, postgraduate year of training, sports medicine experience, and questionnaire results were recorded for each resident. Reliability (the precision or reproducibility of the examination process) was calculated with use of the alpha coefficient of internal consistency (Cronbach alpha). Individual station reliability scores were calculated with use of the Cronbach alpha if Item Deleted and 17 corrected item total correlation coefficients . The Cronbach alpha if Item Deleted is calculated for the individual stations, by removing each station before recalcu-
lating the overall reliability. If removing any one station increases the Cronbach alpha, it implies that the station is performing poorly. Construct validity was based on the degree to which the OSCE discriminated between different postgraduate years and was assessed with use of one-way analysis of variance (ANOVA) in which the year of training was the independent variable and total test scores and domain-specific subscores were the dependent variables. The Scheffe test was used for post hoc analysis to understand the differences in scores between each possible pair of the years of training. The correlation between the checklist scores and the global ratings at the individual station level was assessed with use of the Pearson correlation and the Spearman rho. Concurrent validity was examined by correlating the resident’s performance on the OSCE with his or her most recent overall and sports component scores on the Orthopaedic In-Training Examination (OITE), an annual multiple-choice-question examination taken by all orthopaedic residents at our institution.
Ethical Considerations Approval was obtained from the institutional research ethics boards for this study.
Fig. 1
Figs. 1-A and 1-B Fig. 1-A Box plot of total checklist score compared with the postgraduate year (PGY) of training. Scores on total checklists were significantly higher for PGY5 residents compared with those for PGY1, PGY2, and PGY3 residents (p < 0.05); no significant difference was seen between PGY4 and PGY5 residents (Fig. 2). Fig. 1-B Box plot of global rating 1 to 5 compared with the postgraduate year of training. A significant difference (p < 0.0001) was seen between PGY1 and PGY5 residents. The degree symbol and the asterisk represent group outliers.
e177(4) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
Fig. 2
Figs. 2-A through 2-F Box plots of total checklist scores compared with the postgraduate year of training for each of the six domains. There was no significant difference in the domain of history (p = 0.993) (Fig. 2-A), but a significant difference was seen in the domains of examination (p = 0.001) (Fig. 2-B), image interpretation (p < 0.001) (Fig. 2-C), clinical decision-making (p < 0.001) (Fig. 2-D), consent (p = 0.019) (Fig. 2-E), and surgical technique (p < 0.001) (Fig. 2-F). The degree symbol and the asterisk represent group outliers.
e177(5) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
TABLE V Pearson Correlations Between OSCE Domains and OITE Scores for Thirty-five Participants*
OSCE Domain
OITE Total Score
History
20.075
OITE Sports Subscore 0.076
Examination
0.585†
0.457†
Image interpretation
0.744†
0.627†
Clinical decision-making
0.736†
0.629†
Consent
0.383‡
0.282
Surgical technique
0.728†
0.560†
*Eight PGY1 residents had not taken a written OITE at the time of this study, so data were unavailable for correlation.†The correlation is significant at p < 0.01 (two-tailed test).‡The correlation is significant at p < 0.05 (two-tailed test).
Source of Funding No funding was required for the study.
Results A total of forty-three residents of the sixty-four eligible residents participated in the sports medicine OSCE (Table I). Of the forty-three residents, five were enrolled in the competencybased curriculum program, and eleven had undertaken the sports medicine rotation within the last six months. The total OSCE test score was calculated as the average of the percentage correct of the six station-level scores. The mean group score (and standard deviation) on the total test was 57.0% ± 12.5%. The mean scores for the six stations are presented in Table II. The highest-scoring station was station 5 (posterior shoulder dislocation), and station 2 (rotator cuff tear) and station 3 (cam lesion of the hip and a labral tear) were the most difficult stations. The Cronbach alpha was 0.91 for the six stations, a result within acceptable limits for a high-stakes OSCE (generally accepted as 0.80 or higher)4. Individual station (item) parameters are detailed in Table III. The Cronbach alpha if Item Deleted decreased for every station except for station 1 (meniscal tear); however, the increase for station 1 was minimal. The corrected item total correlation coefficients for the six stations were all very high. The Spearman rho was 0.96 overall, ranging between 0.77 and 0.92 for each of the stations, with all stations demonstrating significant correlation between the checklist scores and the global rating (p < 0.01) on both the Pearson correlation and the non-parametric correlations (Spearman rho). The scores on the total test score were significantly higher for PGY5 residents than for PGY1, PGY2, and PGY3 residents (p < 0.05); no significant difference was seen between PGY4 and PGY5 residents (Fig. 1). The total checklist scores for PGY4 residents were significantly increased compared with those for PGY1 residents only; no significant difference was seen among PGY1, PGY2, and PGY3 residents. A similar pattern
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
of results was seen for the mean of the six station-level global ratings. Items corresponding to each of the six domains were averaged across stations, and the Cronbach alpha was calculated for each domain (Table IV). One-way ANOVA of the total subscores for each of the domains (with the exception of history) showed a significant difference depending on the postgraduate year of training of the resident. Post hoc analyses showed that the domains of clinical decision-making and surgical technique had the greatest discriminatory ability, with the checklist scores of the PGY5 residents significantly different from those of the PGY1, PGY2, and PGY3 residents (Fig. 2). No significant difference was seen in either total checklist scores or global ratings between competency-based curriculum and regular stream residents in the traditional time-based training program. However, a significant effect (p < 0.001) of the number of sports medicine rotations that each resident had undertaken (zero, one, two, or three) was seen on total test scores and in all domains with the exception of history (Fig. 3). Two-way analysis revealed that this significant increase in total checklist scores was independent of the year of training (p < 0.05). No significant improvement in total checklist scores or domain scores was seen with residents who had undertaken a sports medicine rotation within the last six months (relative to those with more remote experience), with the exception of history, although a trend was seen for improved total test scores. For the thirty-five subjects who had taken the written OITE, the total test scores were highly correlated with both OITE total test scores (Pearson correlation coefficient r = 0.72, p < 0.001) and OITE sports subscores (r = 0.61, p < 0.001). High positive correlations were also seen among OSCE domains and examination, image interpretation, clinical decision-making, and surgical technique (Table V). There was a 100% completion rate of the post-OSCE survey. Overall, 91% of the residents agreed or strongly agreed that the clinical scenarios accurately reflected scenarios that general orthopaedic surgeons might encounter in their practice, while 80% agreed or strongly agreed that an OSCE was the best method of testing their medical knowledge. With regard to usefulness, 100% of residents agreed or strongly agreed that this OSCE helped prepare them for their final examinations, with 98% of residents agreeing or strongly agreeing that it would be useful to take an in-training OSCE after every module or rotation. Discussion The result of this study, to our knowledge one of the first to examine the use of an OSCE in orthopaedics, has revealed excellent reliability. Furthermore, this OSCE has shown good validity, with the ability to discriminate among different years of training, both overall and within domains of knowledge. Importantly, it has been demonstrated that it is possible to achieve these goals with a one-hour OSCE, and without the use of standardized patients, making this a feasible assessment tool. We believe that by following the standard process
e177(6) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
Fig. 3
Figs. 3-A through 3-F Box plots of total checklist scores compared with the number of sports rotations for each of the six domains. Again, there was no significant difference in history (p = 0.281) (Fig. 3-A), but a significant difference was seen in the domains of examination (p = 0.005) (Fig. 3-B), image interpretation (p = 0.001) (Fig. 3-C), clinical decision-making (p = 0.001) (Fig. 3-D), consent (p = 0.01) (Fig. 3-E), and surgical technique (p = 0.001) (Fig. 3-F). The degree symbol represents group outliers.
e177(7) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
outlined in the Materials and Methods section, reliable and valid OSCEs can be developed for orthopaedic trainee evaluation, in all modules of orthopaedics. The overall Cronbach alpha of 0.91 in this OSCE was extremely high; as a rule, >0.6 is required for making acceptance decisions (i.e., selection to resident training), with ‡0.80 required for making pass or fail decisions in high-stakes examinations4. It is generally accepted that examinations with low numbers of stations are more likely to have lower reliabilities; an OSCE used in general surgery, examining multiple subspecialties, utilized thirty-eight stations4. This latter type of assessment model would not be feasible for regular in-training assessment of competence. It may be that testing only a narrow range of content (sports medicine) is responsible for the high reliability seen. Some information has been obtained regarding knowledge domains that improve over postgraduate training. In all domains except history, the PGY5 residents demonstrated significantly superior checklist scores. In this study, the PGY5 residents took the sports medicine OSCE just before taking their final certifying examinations; their superiority in the domains of surgical technique and clinical decision-making is, in part, expected. History was a poor discriminator; it may be that this is taught well early in the medical education process. However, the scores are not particularly high across all years; in fact, the ability to take a competent history may require increased attention in our current curriculum. The consenting process seemed poorly understood across all postgraduate years. These results provided us with valuable feedback; in future rotations, an increased emphasis will be placed on consent and postoperative complications. We found that increasing sports medicine experience correlated to an increase in scores. Increasing experience is closely related to the advanced year of training; however, further analysis suggested that this effect was independent of the year of training. Surprisingly, having undertaken a recent sports medicine rotation (run along a competency-based model) did not affect performance, although there was a trend to improved scores. However, this assessment was limited in our study because of the low number of residents (eleven) who had been exposed to the new curriculum. We also identified a correlation between the results of the sports medicine OSCE, with the OITE total test scores and the OITE sports subscores. This correlation may be explained by improved sports medicine knowledge enhancing performance in both the OSCE and the OITE or by the inherent test-taking abilities of some residents. Obviously, good results on each form of assessment signify sound medical knowledge and judgment and do not imply competence in surgical skill. At this time, conclusions regarding competence cannot be drawn, as some of the residents have yet to undertake a sports medicine rotation. These residents scored poorly on the OSCE, as expected. To address the issue of standard setting, evaluation of residents can only take place upon completion of a module. This research is currently under way at this institution, focusing on an absolute or criterion-referenced approach, in which the pass or fail point is set on the basis of what a student should
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
know6,18-20. More than thirty different criterion-referenced methods have been described; our research will focus on finding the most reliable and feasible method21. Furthermore, the use of global ratings for each of the domains and the use of fellows rather than staff surgeons will be studied. It is uncertain whether a PGY1 resident should have the same expected degree of competence as a PGY5 resident after a rotation. We feel that although it might not be reasonable to expect the same level of surgical skill from a junior resident, achieving competence in clinical judgment and decision-making should be possible. This is, of course, very debatable; it could be argued that increased experience in orthopaedics will improve performance in both medical knowledge and surgical skill. The results of this study seem to suggest that an increasing level of experience is correlated to improved performance (especially image interpretation, clinical decision-making, and surgical technique) more so than exposure to a recent sports medicine rotation, although this was not the primary goal of the research. Again, we hope to investigate this issue further by studying methods of standard setting, especially in relation to determining the competence of junior residents compared with that of senior residents. This sports medicine OSCE was designed to test medical knowledge and clinical judgment, not to assess the acquisition of surgical skills such as arthroscopy. Options for assessing surgical skills include Objective Structured Assessment of Technical Skills (OSATS), Structured Assessment of Clinical Evaluation Report (STACER), and components of the ITER. At our institution, after bimonthly teaching in a dry arthroscopy laboratory, residents are assessed with an OSATS (global ratings) while performing an anterior cruciate ligament (ACL) reconstruction and tying an arthroscopic knot on Sawbones models. Research is being conducted to determine if there is a correlation between the OSATS results and the sports medicine OSCE, with a goal to develop a comprehensive assessment model for residents at the end of rotations. We recognize that the definition of an OSCE is variable6. An OSCE is an organizational framework, consisting of multiple, timed stations at which each candidate is faced with a discrete clinical task and is evaluated in an objective, structured way4-6. Stations may or may not involve a standardized patient14. One of the major advantages of not using standardized patients is that it reduces expenses to a minimum, an important factor if an OSCE is to be used on a regular basis. If desired, the structure of the OSCE could be altered such that examiners acted as a standardized patient, answering history questions posed by candidates. In this study, we used a combination of checklists and global ratings. Regehr et al. demonstrated that global ratings completed without the use of a task-specific checklist were equally valid and reliable as global ratings used in association with a checklist, suggesting that the completion of a checklist does not influence the use of a global rating scale22. Furthermore, the global rating scales demonstrated greater interstation reliability, construct validity, and better concurrent validity than did the checklists. It is thought that relying on
e177(8) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
checklists alone results in possible over-objectification of performance and that global ratings can be used by expert raters to assess the less quantifiable aspects of performance23,24. For this reason, a combination was used. A potential issue in our methodology was that the postgraduate-year level of some residents will have been known to examiners, raising the potential for bias. Having residents rotate from station to station and examiner to examiner reduces this risk. Examiners were also asked to disregard the postgraduateyear level of the resident when assigning the global rating. Although using examiners from outside institutions could reduce this potential for bias, this would inevitably raise costs and would reduce feasibility. It is planned to extend the OSCE assessment to the arthroplasty module at our institution and to increase the use of orthopaedic fellows rather than staff as examiners; however, the potential for bias remains. Ultimately, by conducting a formal OSCE assessment at the conclusion of every module in training, we believe that this potential for bias will be minimized. This would be the ideal model for implementation on a national basis. Other limitations of this study included the relatively low number of candidates, especially compared with those in the published literature on the use of OSCEs in medical school examinations. However, as this is the largest training program in Canada, it would be difficult to increase numbers without using multiple centers. It may also be that there was some selection bias involved, as residents were asked to volunteer to take the OSCE; some poorly performing residents may not have participated, potentially skewing our results. Furthermore, this OSCE was run over a period of two weeks; ideally, the OSCE would be administered in a single setting to prevent resident awareness of examination cases. However, evidence in the literature seems to show that information sharing among students has little effect on overall mean scores, especially in the absence of access to the checklists25-27. Finally, this OSCE does not assess surgical skill, only medical knowledge and judgment.
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
In conclusion, it has been demonstrated that a sports medicine OSCE can have acceptable reliability and validity, sufficient to be used to assess the in-training competency of orthopaedic residents. Furthermore, this examination method is feasible, with minimal costs and high acceptability among candidates. n
Tim Dwyer, MBBS, FRACS, FRCSC John S. Theodoropoulos, MD, MSC, FRCSC Jodi Herold, PhD David Wasserstein, MD, MSc, FRCSC M. Lucas Murnaghan, MD, MEd, FRCSC John Semple, MD, MSc, FRCSC Darrell Ogilvie-Harris, MD, MSc, FRCSC Women’s College Hospital, Toronto Western and Sunnybrook Hospital, University of Toronto Orthopaedics Sports Medicine, 76 Grenville Street, Toronto, ON M5S 1B1, Canada. E-mail address for T. Dwyer: [email protected] Patrick Henry, MD, FRCSC Sunnybrook Health Sciences Centre 2075 Bayview Avenue, Toronto, ON M4N 3N5, Canada Veronica Wadey, MD, MA, FRCSC Sunnybrook Health Sciences Centre, 43 Wellesley Street, 6th Floor, Toronto, ON M4Y 1H1, Canada Brian Hodges, MD, MEd, FRCPC, PhD University Health Network, University of Toronto, Toronto General Hospital, 200 Elizabeth Street, Toronto, ON M5G 2C4, Canada
References 1. Jefferies A, Simmons B, Tabak D, McIlroy JH, Lee KS, Roukema H, Skidmore M. Using an objective structured clinical examination (OSCE) to assess multiple physician competencies in postgraduate training. Med Teach. 2007 Mar;29(2-3): 183-91. 2. Catton P, Tallett S, Rothman A. A guide to resident assessment for program directors. Ann R Coll Phys Surg Can. 1997;30:403-9. 3. Norman G. Examining the examination: Canadian versus US certification exam. Can Assoc Radiol J. 2000 Jun;51(3):208-9; author reply 211. 4. Cohen R, Reznick RK, Taylor BR, Provan J, Rothman A. Reliability and validity of the objective structured clinical examination in assessing surgical residents. Am J Surg. 1990 Sep;160(3):302-5. 5. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 1979 Jan;13(1):41-54. 6. Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004 Feb;38(2):199-203. 7. Schwartz RW, Witzke DB, Donnelly MB, Stratton T, Blue AV, Sloan DA. Assessing residents’ clinical performance: cumulative results of a four-year study with the Objective Structured Clinical Examination. Surgery. 1998 Aug;124(2): 307-12. 8. Hodges B, Regehr G, Hanson M, McNaughton N. Validation of an objective structured clinical examination in psychiatry. Acad Med. 1998 Aug;73(8):910-2.
9. Beran MC, Awan H, Rowley D, Samora JB, Griesser MJ, Bishop JY. Assessment of musculoskeletal physical examination skills and attitudes of orthopaedic residents. J Bone Joint Surg Am. 2012 Mar 21;94(6):e36. 10. Kibler WB, editor. Orthopaedic Knowledge Update: Sports Medicine 4. 4th ed. Rosemont, IL: American Academy of Orthopaedic Surgeons; 2009. 11. Frank JRE. The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better Care. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2005. 12. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ. 2003 Nov;37(11):1012-6. 13. Reznick RK, Regehr G, Yee G, Rothman A, Blackmore D, Dauphin´ee D; Medical Council of Canada. Process-rating forms versus task-specific checklists in an OSCE for medical licensure. Acad Med. 1998 Oct;73(10)(Suppl):S97-9. 14. Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999 Oct;74(10): 1129-34. 15. Hodges B, Regehr G, Hanson M, McNaughton N. An objective structured clinical examination for evaluating psychiatric clinical clerks. Acad Med. 1997 Aug;72(8):715-21. 16. Pell G, Roberts TE. Setting standards for student assessment. Int J Res Method Educ. 2006;29(1):91-103.
e177(9) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
17. Pell G, Fuller R, Homer M, Roberts T; International Association for Medical Education. How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49. Med Teach. 2010;32(10):802-11. 18. Schoonheim-Klein M, Muijtjens A, Habets L, Manogue M, van der Vleuten C, van der Velden U. Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. Eur J Dent Educ. 2009 Aug;13(3):162-71. 19. Turnbull JM. What is... normative versus criterion-referenced assessment. Med Teach. 1989;11(2):145-50. 20. Norcini JJ Jr. Standards and reliability in evaluation: when rules of thumb don’t apply. Acad Med. 1999 Oct;74(10):1088-90. 21. Cusimano MD. Standard setting in medical education. Acad Med. 1996 Oct;71(10)(Suppl):S112-20. 22. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998 Sep;73(9):993-7.
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
23. Norman GR, Van der Vleuten CP, De Graaff E. Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Med Educ. 1991 Mar;25(2): 119-26. 24. Van der Vleuten CP, Norman GR, De Graaff E. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991 Mar;25(2):110-8. 25. Champlain AF, Macmillan MK, Margolis MJ, Klass DJ, Lewis E, Ahearn S. Modeling the effects of a test security breach on a large-scale standardized patient examination with a sample of international medical graduates. Acad Med. 2000 Oct;75(10)(Suppl):S109-11. 26. Colliver JA, Travis TA, Robbs RS, Barnhart AJ, Shirar LE, Vu NV. Test security in standardized-patient examinations: analysis with scores on working diagnosis and final diagnosis. Acad Med. 1992 Oct;67(10)(Suppl):S7-9. 27. Swartz MH, Colliver JA, Cohen DS, Barrows HS. The effect of deliberate, excessive violations of test security on performance on a standardized-patient examination. Acad Med. 1993 Oct;68(10)(Suppl):S76-8.
BY
T HE J OURNAL
OF
B ONE
AND J OINT
S URGERY, I NCORPORATED
Topics in Training Assessing Competence of Orthopaedic Residents: The Reliability and Validity of an Objective Structured Clinical Examination After a Sports Medicine Rotation Tim Dwyer, MBBS, FRACS, FRCSC, John S. Theodoropoulos, MD, MSC, FRCSC, Jodi Herold, PhD, Patrick Henry, MD, FRCSC, David Wasserstein, MD, MSc, FRCSC, M. Lucas Murnaghan, MD, MEd, FRCSC, Veronica Wadey, MD, MA, FRCSC, Brian Hodges, MD, MEd, FRCPC, PhD, John Semple, MD, MSc, FRCSC, and Darrell Ogilvie-Harris, MD, MSc, FRCSC Investigation performed at Women’s College Hospital, University of Toronto, Toronto, Ontario, Canada
The traditional method of orthopaedic training in Canada utilizes a time-based system, combined with a formal exit examination. Increasing interest in competency-based curriculum has raised the issue of how to test in-training competence. Currently, the most commonly used method is the In-Training Evaluation Report (ITER), which is known to be relatively subjective in nature; the true establishment of competence would likely benefit from the addition of an objective assessment1,2. The definition of competence varies, but relates to an individual’s ability to perform in the workplace to the required standard3. In surgery, competence is required in many areas, including surgical skill and clinical decision-making. Currently, orthopaedics is lacking a reproducible, objective measurement of resident competence. Objective Structured Clinical Examinations (OSCEs) are an important aspect of certifying examinations in many countries. First developed in the late 1970s, an OSCE consists of multiple, timed stations at which each candidate is faced with a discrete patient or case-based clinical task and is evaluated in an objective and structured way; both the task and the assessment
are standardized4-6. An OSCE is especially suited to testing aspects of the role of medical expert4,7,8. To our knowledge, there is very little published literature on the validity and reliability of OSCEs in orthopaedics. Recently, Beran et al. used an OSCE to assess history-taking and physical examination skills of orthopaedic residents, which served to highlight deficiencies in resident knowledge9. The ability to formulate valid and reliable in-training OSCEs may become an essential tool in competency-based orthopaedic training and may be used to assess resident competence regularly throughout training, to identify residents with knowledge deficits compared with their peers, and to allow appropriate remediation. A compulsory module at our university is a continuous three-month sports medicine rotation, in which residents are expected to acquire the medical knowledge to deal with common sports medicine conditions. Most of the residents undertake two sports medicine rotations, one as a junior resident and one as a senior resident. The majority of exposure to sports medicine occurs during these rotations. We hypothesized that a sports medicine OSCE would demonstrate sufficient
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. One or more of the authors, or his or her institution, has had a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. No author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
J Bone Joint Surg Am. 2013;95:e177(1-9)
d
http://dx.doi.org/10.2106/JBJS.M.00148
e177(2) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
TABLE I Number of Residents in Each Postgraduate Year Postgraduate Year
No. of Residents (N = 43)
1
8
2
6
3
11
4
7
5
11
TABLE II Group Scores for Each of the Six OSCE Stations (N = 43) Station
Group Scores*
1. Meniscal tear
61.5 ± 11.1
2. Rotator cuff tear
49.9 ± 11.7
3. Cam lesion of the hip and a labral tear
48.7 ± 10.5
4. ACL rupture
53.7 ± 15.3
5. Posterior shoulder dislocation
67.2 ± 14.1
6. Ankle instability
61.2 ± 12.5
*The values are given as the mean and the standard deviation in percentage.
reliability and validity to be used for orthopaedic resident in-training assessment. Materials and Methods Examination Development An established methodology was followed to create the OSCE. Orthopaedic Knowledge Update: Sports Medicine 4 was designated as the defined body of 10 knowledge, and assigned as required reading . This establishment of a defined curriculum enabled a clear set of learning objectives to be created, with validation 6 by an expert panel . The majority of the chapters in that text were discussed at resident teaching sessions with a staff surgeon, held on average three times a week; all scenarios tested in the OSCE were covered in that text. A blueprint of clinical problems was then created, with a focus on the CanMEDS (the Royal College of Physicians and Surgeons of Canada’s physician framework for medical 11 education) role of medical expert . The blueprint utilized six domains of medical expert thought to be pertinent: history, examination, image interpretation
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
(investigations and clinical photos), clinical decision-making, consent, and surgical technique. An hour-long OSCE with six stations was created; the OSCE was deliberately kept short to maximize feasibility. Six stations were written; each station had a duration of ten minutes and was based on a common clinical scenario. Scenarios included a fifty-one-year-old man with a traumatic rotator cuff tear, an eighteen-year-old male soccer player with a bucket-handle meniscus tear, a twenty-one-year-old man with a cam lesion of the hip and a labral tear, a seventeen-year-old boy with a torn anterior cruciate ligament, a sixtyseven-year-old woman with a posterior shoulder dislocation, and a twentysix-year-old female volleyball player with lateral ligament instability of the ankle. Computer-based stations were created, with relevant clinical photographs, imaging, and intraoperative photographs displayed on a screen. At each station, the examiner asked a series of predetermined questions based on the clinical scenario and the images. Candidates’ verbal answers were scored with use of a binary checklist assessing performance (mentioned or not mentioned and correct or incorrect). The OSCE questions and binary checklists underwent formal content review at a series of focus groups of orthopaedic surgeons to establish content validity. The knowledge standard was set at the level of the Canadian orthopaedic certifying examination. Scoring was performed in combination with a single global rating of performance at each station, as global ratings may have a higher degree of reliability and may be more sensitive to levels of training than checklists 12-14 alone . To increase objectivity, examiners assigned the global rating independently of the checklist score, in which 1 indicated clear fail, 2 indicated borderline, 3 indicated pass, 4 indicated good pass, and 5 indicated excellent 15-17 pass . The OSCE was administered in trials with two residents of different postgraduate years. Examiners were also trained in the use and application of the checklist and global ratings. Of the six examiners, four (T.D., D.O-H., P.H., and J.S.T.) were both authors of this study and members of the focus group and were very familiar with each scenario and its checklist. With regard to the examiners who were not part of the focus group, the principal investigator (T.D.) spent time discussing each scenario, the checklist, and the global rating with these examiners in detail, addressing any queries or concerns. All examiners were instructed that a pass on the global rating should be equivalent to a pass at an orthopaedic certifying examination level, irrespective of the year of training.
Study Design The OSCE was offered to all orthopaedic residents, postgraduate year 1 (PGY1) through postgraduate year 5 (PGY5), at this institution. Participation was voluntary and written consent was obtained from each resident prior to participation. Examiners at each station were either orthopaedic surgeons or fellows; the same examiner marked each station for the duration of the study, conducted over a two-week period. It was not possible to blind all examiners to the year of training of the residents. However, examiners were asked to disregard the year of training when assessing performance.
TABLE III Individual Station Parameters Station 1. Meniscal tear
Cronbach Alpha If Item Deleted
Corrected Item-Total Correlation
0.92
0.57
2. Rotator cuff tear
0.88
0.83
3. Cam lesion of the hip and a labral tear
0.90
0.68
4. ACL rupture
0.88
0.83
5. Posterior shoulder dislocation
0.89
0.81
6. Ankle instability
0.88
0.82
e177(3) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
TABLE IV Inter-Item Reliability of Subscores and Effect of the Postgraduate Year on Performance Domains
Domain History Examination
No. of Items
Cronbach Alpha Coefficient
P Value for the ANOVA for Effect of Postgraduate Year
Post Hoc Tests*
4
0.49
0.993
No difference
12
0.77
0.001
PGY4 and PGY5 were greater than PGY1, PGY2, and PGY3
Image interpretation
24
0.86
0.000
PGY5 was greater than PGY1 and PGY2
Clinical decision-making
20
0.82
0.000
PGY5 was greater than PGY1, PGY2, and PGY3
Consent
5
0.50
0.019
PGY5 was greater than PGY1
Surgical technique
6
0.78
0.000
PGY5 was greater than PGY1, PGY2, and PGY3
*Significance was set at p < 0.05.
Information was collected regarding the number of sports medicine rotations that each resident had previously undertaken, as well as which residents had undertaken a sports medicine rotation in the last six months. During this period, a competency-based curriculum model had been used, with a set curriculum and exit examinations in the form of OSCE examinations; only some of the residents had been exposed to the new curriculum. After the examination, candidates were invited to provide feedback on the examination via a survey, using a 5-point Likert scale.
Statistical Analysis All data were de-identified, were entered into an Excel spreadsheet, and were analyzed with use of SPSS version 19 (Armonk, New York). The checklists, global ratings, postgraduate year of training, sports medicine experience, and questionnaire results were recorded for each resident. Reliability (the precision or reproducibility of the examination process) was calculated with use of the alpha coefficient of internal consistency (Cronbach alpha). Individual station reliability scores were calculated with use of the Cronbach alpha if Item Deleted and 17 corrected item total correlation coefficients . The Cronbach alpha if Item Deleted is calculated for the individual stations, by removing each station before recalcu-
lating the overall reliability. If removing any one station increases the Cronbach alpha, it implies that the station is performing poorly. Construct validity was based on the degree to which the OSCE discriminated between different postgraduate years and was assessed with use of one-way analysis of variance (ANOVA) in which the year of training was the independent variable and total test scores and domain-specific subscores were the dependent variables. The Scheffe test was used for post hoc analysis to understand the differences in scores between each possible pair of the years of training. The correlation between the checklist scores and the global ratings at the individual station level was assessed with use of the Pearson correlation and the Spearman rho. Concurrent validity was examined by correlating the resident’s performance on the OSCE with his or her most recent overall and sports component scores on the Orthopaedic In-Training Examination (OITE), an annual multiple-choice-question examination taken by all orthopaedic residents at our institution.
Ethical Considerations Approval was obtained from the institutional research ethics boards for this study.
Fig. 1
Figs. 1-A and 1-B Fig. 1-A Box plot of total checklist score compared with the postgraduate year (PGY) of training. Scores on total checklists were significantly higher for PGY5 residents compared with those for PGY1, PGY2, and PGY3 residents (p < 0.05); no significant difference was seen between PGY4 and PGY5 residents (Fig. 2). Fig. 1-B Box plot of global rating 1 to 5 compared with the postgraduate year of training. A significant difference (p < 0.0001) was seen between PGY1 and PGY5 residents. The degree symbol and the asterisk represent group outliers.
e177(4) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
Fig. 2
Figs. 2-A through 2-F Box plots of total checklist scores compared with the postgraduate year of training for each of the six domains. There was no significant difference in the domain of history (p = 0.993) (Fig. 2-A), but a significant difference was seen in the domains of examination (p = 0.001) (Fig. 2-B), image interpretation (p < 0.001) (Fig. 2-C), clinical decision-making (p < 0.001) (Fig. 2-D), consent (p = 0.019) (Fig. 2-E), and surgical technique (p < 0.001) (Fig. 2-F). The degree symbol and the asterisk represent group outliers.
e177(5) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
TABLE V Pearson Correlations Between OSCE Domains and OITE Scores for Thirty-five Participants*
OSCE Domain
OITE Total Score
History
20.075
OITE Sports Subscore 0.076
Examination
0.585†
0.457†
Image interpretation
0.744†
0.627†
Clinical decision-making
0.736†
0.629†
Consent
0.383‡
0.282
Surgical technique
0.728†
0.560†
*Eight PGY1 residents had not taken a written OITE at the time of this study, so data were unavailable for correlation.†The correlation is significant at p < 0.01 (two-tailed test).‡The correlation is significant at p < 0.05 (two-tailed test).
Source of Funding No funding was required for the study.
Results A total of forty-three residents of the sixty-four eligible residents participated in the sports medicine OSCE (Table I). Of the forty-three residents, five were enrolled in the competencybased curriculum program, and eleven had undertaken the sports medicine rotation within the last six months. The total OSCE test score was calculated as the average of the percentage correct of the six station-level scores. The mean group score (and standard deviation) on the total test was 57.0% ± 12.5%. The mean scores for the six stations are presented in Table II. The highest-scoring station was station 5 (posterior shoulder dislocation), and station 2 (rotator cuff tear) and station 3 (cam lesion of the hip and a labral tear) were the most difficult stations. The Cronbach alpha was 0.91 for the six stations, a result within acceptable limits for a high-stakes OSCE (generally accepted as 0.80 or higher)4. Individual station (item) parameters are detailed in Table III. The Cronbach alpha if Item Deleted decreased for every station except for station 1 (meniscal tear); however, the increase for station 1 was minimal. The corrected item total correlation coefficients for the six stations were all very high. The Spearman rho was 0.96 overall, ranging between 0.77 and 0.92 for each of the stations, with all stations demonstrating significant correlation between the checklist scores and the global rating (p < 0.01) on both the Pearson correlation and the non-parametric correlations (Spearman rho). The scores on the total test score were significantly higher for PGY5 residents than for PGY1, PGY2, and PGY3 residents (p < 0.05); no significant difference was seen between PGY4 and PGY5 residents (Fig. 1). The total checklist scores for PGY4 residents were significantly increased compared with those for PGY1 residents only; no significant difference was seen among PGY1, PGY2, and PGY3 residents. A similar pattern
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
of results was seen for the mean of the six station-level global ratings. Items corresponding to each of the six domains were averaged across stations, and the Cronbach alpha was calculated for each domain (Table IV). One-way ANOVA of the total subscores for each of the domains (with the exception of history) showed a significant difference depending on the postgraduate year of training of the resident. Post hoc analyses showed that the domains of clinical decision-making and surgical technique had the greatest discriminatory ability, with the checklist scores of the PGY5 residents significantly different from those of the PGY1, PGY2, and PGY3 residents (Fig. 2). No significant difference was seen in either total checklist scores or global ratings between competency-based curriculum and regular stream residents in the traditional time-based training program. However, a significant effect (p < 0.001) of the number of sports medicine rotations that each resident had undertaken (zero, one, two, or three) was seen on total test scores and in all domains with the exception of history (Fig. 3). Two-way analysis revealed that this significant increase in total checklist scores was independent of the year of training (p < 0.05). No significant improvement in total checklist scores or domain scores was seen with residents who had undertaken a sports medicine rotation within the last six months (relative to those with more remote experience), with the exception of history, although a trend was seen for improved total test scores. For the thirty-five subjects who had taken the written OITE, the total test scores were highly correlated with both OITE total test scores (Pearson correlation coefficient r = 0.72, p < 0.001) and OITE sports subscores (r = 0.61, p < 0.001). High positive correlations were also seen among OSCE domains and examination, image interpretation, clinical decision-making, and surgical technique (Table V). There was a 100% completion rate of the post-OSCE survey. Overall, 91% of the residents agreed or strongly agreed that the clinical scenarios accurately reflected scenarios that general orthopaedic surgeons might encounter in their practice, while 80% agreed or strongly agreed that an OSCE was the best method of testing their medical knowledge. With regard to usefulness, 100% of residents agreed or strongly agreed that this OSCE helped prepare them for their final examinations, with 98% of residents agreeing or strongly agreeing that it would be useful to take an in-training OSCE after every module or rotation. Discussion The result of this study, to our knowledge one of the first to examine the use of an OSCE in orthopaedics, has revealed excellent reliability. Furthermore, this OSCE has shown good validity, with the ability to discriminate among different years of training, both overall and within domains of knowledge. Importantly, it has been demonstrated that it is possible to achieve these goals with a one-hour OSCE, and without the use of standardized patients, making this a feasible assessment tool. We believe that by following the standard process
e177(6) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
Fig. 3
Figs. 3-A through 3-F Box plots of total checklist scores compared with the number of sports rotations for each of the six domains. Again, there was no significant difference in history (p = 0.281) (Fig. 3-A), but a significant difference was seen in the domains of examination (p = 0.005) (Fig. 3-B), image interpretation (p = 0.001) (Fig. 3-C), clinical decision-making (p = 0.001) (Fig. 3-D), consent (p = 0.01) (Fig. 3-E), and surgical technique (p = 0.001) (Fig. 3-F). The degree symbol represents group outliers.
e177(7) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
outlined in the Materials and Methods section, reliable and valid OSCEs can be developed for orthopaedic trainee evaluation, in all modules of orthopaedics. The overall Cronbach alpha of 0.91 in this OSCE was extremely high; as a rule, >0.6 is required for making acceptance decisions (i.e., selection to resident training), with ‡0.80 required for making pass or fail decisions in high-stakes examinations4. It is generally accepted that examinations with low numbers of stations are more likely to have lower reliabilities; an OSCE used in general surgery, examining multiple subspecialties, utilized thirty-eight stations4. This latter type of assessment model would not be feasible for regular in-training assessment of competence. It may be that testing only a narrow range of content (sports medicine) is responsible for the high reliability seen. Some information has been obtained regarding knowledge domains that improve over postgraduate training. In all domains except history, the PGY5 residents demonstrated significantly superior checklist scores. In this study, the PGY5 residents took the sports medicine OSCE just before taking their final certifying examinations; their superiority in the domains of surgical technique and clinical decision-making is, in part, expected. History was a poor discriminator; it may be that this is taught well early in the medical education process. However, the scores are not particularly high across all years; in fact, the ability to take a competent history may require increased attention in our current curriculum. The consenting process seemed poorly understood across all postgraduate years. These results provided us with valuable feedback; in future rotations, an increased emphasis will be placed on consent and postoperative complications. We found that increasing sports medicine experience correlated to an increase in scores. Increasing experience is closely related to the advanced year of training; however, further analysis suggested that this effect was independent of the year of training. Surprisingly, having undertaken a recent sports medicine rotation (run along a competency-based model) did not affect performance, although there was a trend to improved scores. However, this assessment was limited in our study because of the low number of residents (eleven) who had been exposed to the new curriculum. We also identified a correlation between the results of the sports medicine OSCE, with the OITE total test scores and the OITE sports subscores. This correlation may be explained by improved sports medicine knowledge enhancing performance in both the OSCE and the OITE or by the inherent test-taking abilities of some residents. Obviously, good results on each form of assessment signify sound medical knowledge and judgment and do not imply competence in surgical skill. At this time, conclusions regarding competence cannot be drawn, as some of the residents have yet to undertake a sports medicine rotation. These residents scored poorly on the OSCE, as expected. To address the issue of standard setting, evaluation of residents can only take place upon completion of a module. This research is currently under way at this institution, focusing on an absolute or criterion-referenced approach, in which the pass or fail point is set on the basis of what a student should
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
know6,18-20. More than thirty different criterion-referenced methods have been described; our research will focus on finding the most reliable and feasible method21. Furthermore, the use of global ratings for each of the domains and the use of fellows rather than staff surgeons will be studied. It is uncertain whether a PGY1 resident should have the same expected degree of competence as a PGY5 resident after a rotation. We feel that although it might not be reasonable to expect the same level of surgical skill from a junior resident, achieving competence in clinical judgment and decision-making should be possible. This is, of course, very debatable; it could be argued that increased experience in orthopaedics will improve performance in both medical knowledge and surgical skill. The results of this study seem to suggest that an increasing level of experience is correlated to improved performance (especially image interpretation, clinical decision-making, and surgical technique) more so than exposure to a recent sports medicine rotation, although this was not the primary goal of the research. Again, we hope to investigate this issue further by studying methods of standard setting, especially in relation to determining the competence of junior residents compared with that of senior residents. This sports medicine OSCE was designed to test medical knowledge and clinical judgment, not to assess the acquisition of surgical skills such as arthroscopy. Options for assessing surgical skills include Objective Structured Assessment of Technical Skills (OSATS), Structured Assessment of Clinical Evaluation Report (STACER), and components of the ITER. At our institution, after bimonthly teaching in a dry arthroscopy laboratory, residents are assessed with an OSATS (global ratings) while performing an anterior cruciate ligament (ACL) reconstruction and tying an arthroscopic knot on Sawbones models. Research is being conducted to determine if there is a correlation between the OSATS results and the sports medicine OSCE, with a goal to develop a comprehensive assessment model for residents at the end of rotations. We recognize that the definition of an OSCE is variable6. An OSCE is an organizational framework, consisting of multiple, timed stations at which each candidate is faced with a discrete clinical task and is evaluated in an objective, structured way4-6. Stations may or may not involve a standardized patient14. One of the major advantages of not using standardized patients is that it reduces expenses to a minimum, an important factor if an OSCE is to be used on a regular basis. If desired, the structure of the OSCE could be altered such that examiners acted as a standardized patient, answering history questions posed by candidates. In this study, we used a combination of checklists and global ratings. Regehr et al. demonstrated that global ratings completed without the use of a task-specific checklist were equally valid and reliable as global ratings used in association with a checklist, suggesting that the completion of a checklist does not influence the use of a global rating scale22. Furthermore, the global rating scales demonstrated greater interstation reliability, construct validity, and better concurrent validity than did the checklists. It is thought that relying on
e177(8) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
checklists alone results in possible over-objectification of performance and that global ratings can be used by expert raters to assess the less quantifiable aspects of performance23,24. For this reason, a combination was used. A potential issue in our methodology was that the postgraduate-year level of some residents will have been known to examiners, raising the potential for bias. Having residents rotate from station to station and examiner to examiner reduces this risk. Examiners were also asked to disregard the postgraduateyear level of the resident when assigning the global rating. Although using examiners from outside institutions could reduce this potential for bias, this would inevitably raise costs and would reduce feasibility. It is planned to extend the OSCE assessment to the arthroplasty module at our institution and to increase the use of orthopaedic fellows rather than staff as examiners; however, the potential for bias remains. Ultimately, by conducting a formal OSCE assessment at the conclusion of every module in training, we believe that this potential for bias will be minimized. This would be the ideal model for implementation on a national basis. Other limitations of this study included the relatively low number of candidates, especially compared with those in the published literature on the use of OSCEs in medical school examinations. However, as this is the largest training program in Canada, it would be difficult to increase numbers without using multiple centers. It may also be that there was some selection bias involved, as residents were asked to volunteer to take the OSCE; some poorly performing residents may not have participated, potentially skewing our results. Furthermore, this OSCE was run over a period of two weeks; ideally, the OSCE would be administered in a single setting to prevent resident awareness of examination cases. However, evidence in the literature seems to show that information sharing among students has little effect on overall mean scores, especially in the absence of access to the checklists25-27. Finally, this OSCE does not assess surgical skill, only medical knowledge and judgment.
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
In conclusion, it has been demonstrated that a sports medicine OSCE can have acceptable reliability and validity, sufficient to be used to assess the in-training competency of orthopaedic residents. Furthermore, this examination method is feasible, with minimal costs and high acceptability among candidates. n
Tim Dwyer, MBBS, FRACS, FRCSC John S. Theodoropoulos, MD, MSC, FRCSC Jodi Herold, PhD David Wasserstein, MD, MSc, FRCSC M. Lucas Murnaghan, MD, MEd, FRCSC John Semple, MD, MSc, FRCSC Darrell Ogilvie-Harris, MD, MSc, FRCSC Women’s College Hospital, Toronto Western and Sunnybrook Hospital, University of Toronto Orthopaedics Sports Medicine, 76 Grenville Street, Toronto, ON M5S 1B1, Canada. E-mail address for T. Dwyer: [email protected] Patrick Henry, MD, FRCSC Sunnybrook Health Sciences Centre 2075 Bayview Avenue, Toronto, ON M4N 3N5, Canada Veronica Wadey, MD, MA, FRCSC Sunnybrook Health Sciences Centre, 43 Wellesley Street, 6th Floor, Toronto, ON M4Y 1H1, Canada Brian Hodges, MD, MEd, FRCPC, PhD University Health Network, University of Toronto, Toronto General Hospital, 200 Elizabeth Street, Toronto, ON M5G 2C4, Canada
References 1. Jefferies A, Simmons B, Tabak D, McIlroy JH, Lee KS, Roukema H, Skidmore M. Using an objective structured clinical examination (OSCE) to assess multiple physician competencies in postgraduate training. Med Teach. 2007 Mar;29(2-3): 183-91. 2. Catton P, Tallett S, Rothman A. A guide to resident assessment for program directors. Ann R Coll Phys Surg Can. 1997;30:403-9. 3. Norman G. Examining the examination: Canadian versus US certification exam. Can Assoc Radiol J. 2000 Jun;51(3):208-9; author reply 211. 4. Cohen R, Reznick RK, Taylor BR, Provan J, Rothman A. Reliability and validity of the objective structured clinical examination in assessing surgical residents. Am J Surg. 1990 Sep;160(3):302-5. 5. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 1979 Jan;13(1):41-54. 6. Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004 Feb;38(2):199-203. 7. Schwartz RW, Witzke DB, Donnelly MB, Stratton T, Blue AV, Sloan DA. Assessing residents’ clinical performance: cumulative results of a four-year study with the Objective Structured Clinical Examination. Surgery. 1998 Aug;124(2): 307-12. 8. Hodges B, Regehr G, Hanson M, McNaughton N. Validation of an objective structured clinical examination in psychiatry. Acad Med. 1998 Aug;73(8):910-2.
9. Beran MC, Awan H, Rowley D, Samora JB, Griesser MJ, Bishop JY. Assessment of musculoskeletal physical examination skills and attitudes of orthopaedic residents. J Bone Joint Surg Am. 2012 Mar 21;94(6):e36. 10. Kibler WB, editor. Orthopaedic Knowledge Update: Sports Medicine 4. 4th ed. Rosemont, IL: American Academy of Orthopaedic Surgeons; 2009. 11. Frank JRE. The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better Care. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2005. 12. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ. 2003 Nov;37(11):1012-6. 13. Reznick RK, Regehr G, Yee G, Rothman A, Blackmore D, Dauphin´ee D; Medical Council of Canada. Process-rating forms versus task-specific checklists in an OSCE for medical licensure. Acad Med. 1998 Oct;73(10)(Suppl):S97-9. 14. Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999 Oct;74(10): 1129-34. 15. Hodges B, Regehr G, Hanson M, McNaughton N. An objective structured clinical examination for evaluating psychiatric clinical clerks. Acad Med. 1997 Aug;72(8):715-21. 16. Pell G, Roberts TE. Setting standards for student assessment. Int J Res Method Educ. 2006;29(1):91-103.
e177(9) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 95 -A N U M B E R 22 N O V E M B E R 20, 2 013 d
d
d
17. Pell G, Fuller R, Homer M, Roberts T; International Association for Medical Education. How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49. Med Teach. 2010;32(10):802-11. 18. Schoonheim-Klein M, Muijtjens A, Habets L, Manogue M, van der Vleuten C, van der Velden U. Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. Eur J Dent Educ. 2009 Aug;13(3):162-71. 19. Turnbull JM. What is... normative versus criterion-referenced assessment. Med Teach. 1989;11(2):145-50. 20. Norcini JJ Jr. Standards and reliability in evaluation: when rules of thumb don’t apply. Acad Med. 1999 Oct;74(10):1088-90. 21. Cusimano MD. Standard setting in medical education. Acad Med. 1996 Oct;71(10)(Suppl):S112-20. 22. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998 Sep;73(9):993-7.
U S E O F A N OSCE T O A S S E S S I N -T R A I N I N G C O M P E T E N C E I N O RT H O PA E D I C R E S I D E N T S
23. Norman GR, Van der Vleuten CP, De Graaff E. Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Med Educ. 1991 Mar;25(2): 119-26. 24. Van der Vleuten CP, Norman GR, De Graaff E. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991 Mar;25(2):110-8. 25. Champlain AF, Macmillan MK, Margolis MJ, Klass DJ, Lewis E, Ahearn S. Modeling the effects of a test security breach on a large-scale standardized patient examination with a sample of international medical graduates. Acad Med. 2000 Oct;75(10)(Suppl):S109-11. 26. Colliver JA, Travis TA, Robbs RS, Barnhart AJ, Shirar LE, Vu NV. Test security in standardized-patient examinations: analysis with scores on working diagnosis and final diagnosis. Acad Med. 1992 Oct;67(10)(Suppl):S7-9. 27. Swartz MH, Colliver JA, Cohen DS, Barrows HS. The effect of deliberate, excessive violations of test security on performance on a standardized-patient examination. Acad Med. 1993 Oct;68(10)(Suppl):S76-8.
