The Laryngoscope C 2017 The American Laryngological, V
Rhinological and Otological Society, Inc.
Systematic Review
Toolbox of Assessment Tools of Technical Skills in Otolaryngology–Head and Neck Surgery: A Systematic Review Mathilde Labbe; Meredith Young, PhD; Lily H. P. Nguyen, MD, MSc, FRCSC Objective: To support the development of programs of assessment of technical skills in the operating room (OR), we systematically reviewed the literature to identify assessment tools specific to otolaryngology–head and neck surgery (OTLHNS) core procedures and summarized their characteristics. Methods: We systematically searched Embase, MEDLINE, PubMed, and Cochrane to identify and report on assessment tools that can be used to assess residents’ technical surgical skills in the operating room for OTL-HNS core procedures. Results: Of the 736 unique titles retrieved, 16 articles met inclusion criteria, covering 11 different procedures (in otology, rhinology, laryngology, head and neck, and general otolaryngology). The tools were composed of a task-specific checklist and/or global rating scale and were developed in the OR, on human cadavers, or in a simulation setting. Conclusions: Our study reports on published tools for assessing technical skills for OTL-HNS residents during core procedures conducted in the OR. These assessment tools could facilitate the provision of timely feedback to trainees including specific goals for improvement. However, the paucity of publications suggests little agreement on how to best perform workbased direct-observation assessment for core surgical procedures in OTL-HNS. The sparsity of tools specific to OTL-HNS may become a barrier to a fluid transition to competency-based medical education. Key Words: Assessment, toolbox, systematic review, competency, medical education, technical skills. Laryngoscope, 00:000–000, 2017
INTRODUCTION Competency-based medical education (CBME) is in the midst of bringing about major changes to residency training1 and necessitates the use of defensible methods of assessment to appropriately judge residents’ competency. Judgments of competence are likely best made in the context of programmatic assessment,2 the careful combination of multiple types of assessment of trainee performance across time.3 Otolaryngology–head and neck surgery (OTL-HNS) residency programs are encouraged to move beyond written or oral examinations, and to emphasize the inclusion of work-based assessments of residents.4 Effective and frequent assessment directly in the workplace is a key component to the CBME implementation given the greater need for
From the Faculty of Medicine (M.L.), Center for Medical Education (M.Y., L.H.P.N.), Department of Medicine (M.Y.), and Department of Otolaryngology (L.H.P.N.), McGill University, Montreal, Quebec, Canada Additional Supporting Information may be found in the online version of this article. Editor’s Note: This Manuscript was accepted for publication on September 5, 2017. The authors have no funding, financial relationships, or conflicts of interest to disclose. Send correspondence to Lily H. P. Nguyen, Department of Otolaryngology–Head and Neck Surgery, Montreal Children’s Hospital, McGill University, 1001 Boulevard Decarie, A02.3015, H4A 3J1 Montreal, Quebec, Canada. E-mail: [email protected] DOI: 10.1002/lary.26943
Laryngoscope 00: Month 2017
coaching and formative assessment on a continual basis and the decoupling of development of competence from time in training.5 Thus, to support judgments of competence, provide feedback, and document the trajectory of a resident toward competence, it is critical to have assessment tools that are of high quality and designed for use in direct-observation workplace-based assessment contexts. Of the many roles expected of a competent physician,6,7 surgical technical skills are well represented in the learning competences expected of surgical residents.1 To provide feedback or to assess residence performance of OTL-HNS procedures, assessment tools for direct observation should be developed, appropriately utilized, and their use carefully monitored. Other surgical specialties have extensive literature reporting on several validated assessment tools of technical skills, as well as toolboxes to help surgical educators select the instruments best suited for their context.8–10 In a recent large systematic review of all surgical specialties that presented 106 assessment tools, only one of the core procedures for OTL-HNS was represented.8 To our knowledge, there is no work reporting on the available OTL-HNS assessment tools for evaluating residents’ technical surgical skills in the operating room (OR). Here, we report on a systematic review process that has resulted in a toolbox that could be used by surgical educators to inform the development of programs of assessment to support CBME in OTL-HNS. et al.: Toolbox for Assessment of Technical Skills Labbe
1
METHODS Search Strategy A systematic search of the scientific literature was designed with the support of a medical librarian to identify the literature reporting on tools for assessing technical skills in OTL-HNS. The search strategy was adapted and executed in MEDLINE, PubMed, Embase, and the Cochrane Database of Systematic Reviews (see Supporting Appendix I in the online version of this article). The search keywords included a combination of words related to technical skills assessment such as “measurement,” “evaluation,” “competency,” and “proficiency,” as well as specific names of OTLHNS–related procedures that were compiled by the authors. The complete set of search strategies can be obtained from the authors on request. The search included published articles from inception of the databases (as early as 1946) to 2016 in OTL-HNS.
Inclusion and Exclusion Criteria Articles in the English or French language, that presented a novel assessment tool focused on assessing the technical skills of a trainee, that could be used in the operating room, and were developed for resident assessment were included. Rater-based assessments, defined as instruments designed for completing during or following direct observation in the OR by a clinical supervisor,11,12 were identified. Tools developed to assess performance of a trainee on specific procedures were included, whether the tool was developed and validated in the OR, in a simulation setting, or on cadavers. Assessment tools that were solely based on quantitative metrics (such as time to completion of a task) and publications where the main goal of the study was not to directly report on a new assessment tool were excluded.
Study Selection Every title was reviewed by two independent authors for inclusion and exclusion following rater training (M.L. and J.L.). Discrepancies were discussed to reach consensus. The same process was subsequently performed for abstracts and full-text articles. Inter-rater reliability was not measured.
Data Extraction For all articles that met inclusion criteria, title, author(s), year of publication, journal, country where the study was conducted, setting (e.g., OR, simulation, cadaver), sample size, and level of training of the participants was extracted. Information on the tool itself was also extracted, including rating-scale format (e.g., global rating scale [GRS], task-specific checklist [TSC]), number of items, and type of scale (e.g., binary, Likert scale).
RESULTS Search Results Results were reported according to the Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA) guidelines13 (Fig. 1). Seven hundred thirty-six unique articles were identified. Following review of titles and abstracts, 68 full-text articles were reviewed. Sixteen were found to meet the complete list of inclusion criteria and were included in our review.
Description of the Assessment Tools Characteristics of the 16 assessment tools included in this review are detailed in Table I. Tools reflected a Laryngoscope 00: Month 2017
2
Fig. 1. Identification, screening, and study selection process according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting structure. OTL 5 otolaryngology. [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]
total of 11 different OTL-HNS procedures; four in otology, two in laryngology, two in rhinology, two in head and neck, and one in general otolaryngology. Eleven (68.8%) of the tools were composed of both a TSC and a GRS, three (18.6%) only used a GRS, and two (12.6%) only used a TSC. Five tools (31.3%) had a final component consisting of a global judgment on overall competency, and seven tools (43.8%) had a space for narrative feedback. Fourteen tools (87.5%) used a five-point Likert scale, and two (12.5%) used a binary scale (yes/no or complete/not complete).
Description of Development and Validation Context The country of origin of each instrument is detailed it Table I. All tools were designed to assess trainees at the resident level. Five tools (31.3%) reported comparing resident and faculty member performance during their validation process. The specific postgraduate training year of the participants is described in Table I. Eleven (68.8%) of the tools were developed and validated in the OR, four (25.0%) were validated on human cadavers, and one (6.2%) on a simulator.
DISCUSSION To our knowledge, we present the first systematic review of direct-observation workplace-based assessment tools for technical skills in OTL-HNS residency training. To support the development of an assessment program within competency-based medical education, we offer a toolbox of 16 instruments covering a total of 11 OTLHNS procedures that could be used to assess technical skills in the operating room. A task-specific checklist (TSC) component was present in the vast majority of our tools. This form of assessment focuses on directly observable actions and et al.: Toolbox for Assessment of Technical Skills Labbe
Laryngoscope 00: Month 2017
et al.: Toolbox for Assessment of Technical Skills Labbe
3
GRS 9
UK
Myringotomy and tube insertion37,38
GRS 12
USA TSC 10
5-point Likert scale
5-point Likert scale
5-point Likert scale 5-point Likert scale
Binary, 4-point Likert scale
Binary
Binary, 5-point Likert scale 5-point Likert scale
5-point Likert scale
5-point Likert scale 5-point Likert scale
5-point Likert scale
5-point Likert scale
5-point Likert scale
5-point Likert scale
5-point Likert scale
Evaluation
1–5
3, fellow, staff
1–5 1–6
1–5
2–5
2–5
2, 3 1 staff
Junior, senior, staff
2–5 2–6
SHO to SpR4
1–5 1 staff
1–5
2–5
Unspecified 1 staff
PGY Level
OR
OR
Human cadaver OR
Cadaver bones
Human cadaver
OR
OR
OR
OR OR
OR
Human cadaver OR
OR (humans or animals)
Simulator
Setting
Global Judgment
GRS 5 global rating scale; N/A 5 not available; OR 5 operating room; PGY 5 postgraduate year; SHO 5 senior house officer; SpR4 5 specialist registar 4; TSC 5 task-specific checklist.
Saudi Arabia
TSC 22, GRS 10 TSC 11, GRS 11
USA USA
Tonsillectomy43,44
Tracheotomy45
TSC 16, GRS 4
Canada
Mastoidectomy40–42
TSC 35
USA
Temporal bone dissection/
TSC 8, GRS 7
Saudi Arabia
Septoplasty39
TSC 30, GRS 10
TSC 18,GRS N/A TSC, 10, GRS 9
Canada
GRS 8
USA USA
TSC 10, GRS 7
Canada and Saudi Arabia
UK
TSC 21, GRS 10
USA
Hemi/ thyroidectomy35,36
TSC 12, GRS 10
USA
Direct laryngoscopy and rigid bronchoscopy32 Endoscopic sinus surgery17,33,34
TSC 8, GRS 7
No. of Items
Australia
Origin
Cochlear implant surgery31
Procedure
TABLE I. Summary of Published Technical Skills Assessment Tools in Otolaryngology–Head and Neck Surgery Procedures.
Narrative Feedback
thus provides a defensible mean to assess performance. TSCs are useful in providing timely feedback with specific improvement goals to trainees as well as defining areas of weaknesses that could require remedial activities.14,15 As they provide feedback that is specific to a technical task rather than overall competency, TSCs could be of most value when used in a formative setting, a low stake environment where the goal is to monitor the trainee’s learning. However, given the specificity of TSCs, their use necessitates careful consideration of what is included in a checklist for a given procedure, and the existence of a particular tool for each of the procedures of a surgical subspecialty. A GRS component was found in a large majority of our tools. An obvious advantage of those nonspecific scales is the acceptability for surgical educators, who would need to become familiar with only one general assessment.16 A GRS offers an interesting opportunity for assessing surgical competence rather than technical skills alone.15 When compared to a TSC, the disadvantage of a GRS may be the necessity for more extensive rater training.16,17 The implementation of any new assessment tool should be accompanied by rater training and quality monitoring practices. Lack of rater training may lead to inconsistent ratings of performance, and may leave the assessment scores difficult to interpret.17 A GRS has, however, been reported to be more reliable and better able to differentiate across levels of expertise.18 With proper rater training, a GRS could probably be best situated for high-stakes summative settings where the goal is to evaluate the trainee’s skills at the end of a rotation, and could provide a better assessment of overall surgical competency. Although some centers are working to develop the procedure-specific assessment tools similar to those compiled here, there is a clear trend in the literature outside of OTL-HNS toward a more centralized way of assessing technical skills of residents. The O-SCORE19,20 and the Zwisch model21,22 are examples of tools that offer what has been considered a more intuitive assessment approach, where the trainee is rated based on the involvement of the attending surgeon during the procedure. This concept of entrustability is directly linked to CBME goals.23 Tools like the O-SCORE or the Zwisch model could overcome known disadvantages of more traditional instruments, such as long delays to completion of ratings,24,25 and could palliate concerns that CBME increases the assessment role of a clinical educator to the detriment of the quality of the learning environment.26–28 Further research is needed to evaluate the feasibility of more generic tools such as the O-SCORE in the context of OTL-HNS procedures. Regardless of the structure of assessment tools identified in our review, we found that out of the 114 OTL-HNS procedures that are considered as core competencies to achieve during residency training,3 only 11 were covered by the assessment tools identified and summarized in this review. This is in contrast to other specialties such as general surgery, where certain procedures already have extensive literature published on assessment of residents’ technical skills.9,10 This paucity Laryngoscope 00: Month 2017
4
of literature may become a barrier to the successful implementation of CBME, and should raise concerns regarding the evidence base of assessment within OTLHNS.26 The main limitation of our study is that we did not specifically evaluate or incorporate the sources of validity evidence nor the specific purpose (i.e., formative or summative) of the tools reported here. Although several modern validity frameworks have been integrated into the medical education literature (notably Kane via Cook et al.29 and Messick via Downing30), no formal evaluation of validity evidence was included in this review. However, we reported on the context (e.g., OR, simulation) in which each tool was originally developed, which suggests the circumstances where the instrument is best suited for use. Upon implementation of those instruments in a residency curriculum, attention should always be paid to the original reported validity evidence, and we strongly encourage careful quality monitoring for any assessment tool used. Another limitation of this study lies in the fact that we excluded metric-based tools to focus on rater-based assessments. However, given the emphasis on direct observation and workplace-based assessment within CBME, we feel that limiting our review to rater-based assessments was appropriate.
CONCLUSION We created a toolbox of assessment tools reported for the rater-based assessment of resident’s technical skills for OTL-HNS operative procedures following a systematic review. These tools could be used to create a program of assessment to support formative and summative assessment within OTL-HNS residency training programs. More research is, however, required in OTL-HNS to expand available tools, expand evidence of validity supporting their use across a variety of contexts, and for consideration of adopting a global instrument to assess surgical competency.
Acknowledgments The authors thank Joshua Lubov, a medical student at McGill University, for his help with data collection.
BIBLIOGRAPHY 1. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach 2010;32:638–645. 2. Schuwirth L, van der Vleuten C, Durning SJ. What programmatic assessment in medical education can learn from healthcare. Perspect Med Educ 2017;6:211–215. 3. van der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. Twelve tips for programmatic assessment [published online November 20, 2014]. Med Teach doi: 10.3109/0142159X.2014. 973388. 4. Potts JR III. Assessment of competence: the Accreditation Council for Graduate Medical Education/Residency Review Committee perspective. Surg Clin North Am 2016;96:15–24. 5. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach 2010;32: 676–682. 6. Royal College of Physicians and Surgeons of Canada. The CanMEDS 2015 framework. Available at: http://canmeds.royalcollege.ca. Accessed May 2017. 7. Accreditation Council for Graduate Medical Education general competencies. Available at: http://www.acgme.org. Accessed June 2016. 8. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB. Observational tools for assessment of procedural skills: a systematic review. Am J Surg 2011;202:469–480.e466.
et al.: Toolbox for Assessment of Technical Skills Labbe
9. Ghaderi I, Manji F, Park YS, et al. Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 2015;261:251– 262. 10. Watanabe Y, Bilgic E, Lebedeva E, et al. A systematic review of performance assessment tools for laparoscopic cholecystectomy. Surg Endosc 2016;30:832–844. 11. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA 2009;302:1316–1326. 12. Epstein RM. Assessment in medical education. N Engl J Med 2007;356: 387–396. 13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 2009;62:e1–e34. 14. Archer JC. State of the science in health professional education: effective feedback. Med Educ 2010;44:101–108. 15. Norman GR, Van der Vleuten CP, De Graaff E. Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Med Educ 1991;25:119–126. 16. Ilgen JS, Ma IW, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ 2015;49:161–173. 17. Lin SY, Laeeq K, Ishii M, et al. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy 2009;23:354–359. 18. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ 2003;37:1012–1016. 19. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med 2012;87:1401–1407. 20. MacEwan MJ, Dudek NL, Wood TJ, Gofton WT. Continued Validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): use in the simulated environment. Teach Learn Med 2016;28:72–79. 21. DaRosa DA, Zwischenberger JB, Meyerson SL, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ 2013;70:24–30. 22. George BC, Teitelbaum EN, Meyerson SL, et al. Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ 2014;71:e90–e96. 23. Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med 2016;91:186–190. 24. Laeeq K, Francis HW, Varela DA, Malik MU, Cummings CW, Bhatti NI. The timely completion of objective assessment tools for evaluation of technical skills. Laryngoscope 2012;122:2418–2421. 25. Kim MJ, Williams RG, Boehler ML, Ketchum JK, Dunnington GL. Refining the evaluation of operating room performance. J Surg Educ 2009;66: 352–356. 26. Hawkins RE, Welcher CM, Holmboe ES, et al. Implementation of competency-based medical education: are we addressing the concerns and challenges? Med Educ 2015;49:1086–1102. 27. Leung WC. Competency based medical training: review. BMJ 2002;325: 693–696.
Laryngoscope 00: Month 2017
28. Malone K, Supri S. A critical time for medical education: the perils of competence-based reform of the curriculum. Adv Health Sci Educ Theory Pract 2012;17:241–246. 29. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ 2015;49:560–575. 30. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ 2003;37:830–837. 31. Piromchai P, Kasemsiri P, Wijewickrema S, Ioannou I, Kennedy G, O’Leary S. The construct validity and reliability of an assessment tool for competency in cochlear implant surgery. Biomed Res Int 2014;2014: 192741. 32. Ishman SL, Brown DJ, Boss EF, et al. Development and pilot testing of an operative competency assessment tool for pediatric direct laryngoscopy and rigid bronchoscopy. Laryngoscope 2010;120:2294–2300. 33. Marglani O, Alherabi A, Al-Andejani T, Javer A, Al-Zalabani A, Chalmers A. Development of a tool for Global Rating of Endoscopic Surgical Skills (GRESS) for assessment of otolaryngology residents. B-ENT 2012;8:191– 195. 34. Syme-Grant J, White PS, McAleer JP. Measuring competence in endoscopic sinus surgery. Surgery 2008;6:37–44. 35. Stack BC Jr, Siegel E, Bodenner D, Carr MM. A study of resident proficiency with thyroid surgery: creation of a thyroid-specific tool. Otolaryngol Head Neck Surg 2010;142:856–862. 36. Diaz Voss Varela DA, Malik MU, Thompson CB, Cummings CW, Bhatti NI, Tufano RP. Comprehensive assessment of thyroidectomy skills development: a pilot project. Laryngoscope 2012;122:103–109. 37. Bowles PF, Harries M, Young P, Das P, Saunders N, Fleming JC. A validation study on the use of intra-operative video recording as an objective assessment tool for core ENT surgery. Clin Otolaryngol 2014;39:102– 107. 38. Schwartz J, Costescu A, Mascarella MA, et al. Objective assessment of Myringotomy and tympanostomy tube insertion: a prospective singleblinded validation study. Laryngoscope 2016;126:2140–2146. 39. Obeid AA, Al-Qahtani KH, Ashraf M, Alghamdi FR, Marglani O, Alherabi A. Development and testing for an operative competency assessment tool for nasal septoplasty surgery. Am J Rhinol Allergy 2014;28:e163– e167. 40. Butler NN, Wiet GJ. Reliability of the Welling scale (WS1) for rating temporal bone dissection performance. Laryngoscope 2007;117:1803–1808. 41. Zirkle M, Taplin MA, Anthony R, Dubrowski A. Objective assessment of temporal bone drilling skills. Ann Otol Rhinol Laryngol 2007;116:793– 798. 42. Laeeq K, Bhatti NI, Carey JP, et al. Pilot testing of an assessment tool for competency in mastoidectomy. Laryngoscope 2009;119:2402–2410. 43. Ahmed A, Ishman SL, Laeeq K, Bhatti NI. Assessment of improvement of trainee surgical skills in the operating room for tonsillectomy. Laryngoscope 2013;123:1639–1644. 44. Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument to measure surgical performance at tonsillectomy. Laryngoscope 2005;115:2127–2137. 45. Al-Qahtani KH, Alkhalidi AM, Islam T. Tool for assessing surgical tracheostomy skills in otolaryngology residents. B-ENT 2015;11:275–280.
et al.: Toolbox for Assessment of Technical Skills Labbe
5
Rhinological and Otological Society, Inc.
Systematic Review
Toolbox of Assessment Tools of Technical Skills in Otolaryngology–Head and Neck Surgery: A Systematic Review Mathilde Labbe; Meredith Young, PhD; Lily H. P. Nguyen, MD, MSc, FRCSC Objective: To support the development of programs of assessment of technical skills in the operating room (OR), we systematically reviewed the literature to identify assessment tools specific to otolaryngology–head and neck surgery (OTLHNS) core procedures and summarized their characteristics. Methods: We systematically searched Embase, MEDLINE, PubMed, and Cochrane to identify and report on assessment tools that can be used to assess residents’ technical surgical skills in the operating room for OTL-HNS core procedures. Results: Of the 736 unique titles retrieved, 16 articles met inclusion criteria, covering 11 different procedures (in otology, rhinology, laryngology, head and neck, and general otolaryngology). The tools were composed of a task-specific checklist and/or global rating scale and were developed in the OR, on human cadavers, or in a simulation setting. Conclusions: Our study reports on published tools for assessing technical skills for OTL-HNS residents during core procedures conducted in the OR. These assessment tools could facilitate the provision of timely feedback to trainees including specific goals for improvement. However, the paucity of publications suggests little agreement on how to best perform workbased direct-observation assessment for core surgical procedures in OTL-HNS. The sparsity of tools specific to OTL-HNS may become a barrier to a fluid transition to competency-based medical education. Key Words: Assessment, toolbox, systematic review, competency, medical education, technical skills. Laryngoscope, 00:000–000, 2017
INTRODUCTION Competency-based medical education (CBME) is in the midst of bringing about major changes to residency training1 and necessitates the use of defensible methods of assessment to appropriately judge residents’ competency. Judgments of competence are likely best made in the context of programmatic assessment,2 the careful combination of multiple types of assessment of trainee performance across time.3 Otolaryngology–head and neck surgery (OTL-HNS) residency programs are encouraged to move beyond written or oral examinations, and to emphasize the inclusion of work-based assessments of residents.4 Effective and frequent assessment directly in the workplace is a key component to the CBME implementation given the greater need for
From the Faculty of Medicine (M.L.), Center for Medical Education (M.Y., L.H.P.N.), Department of Medicine (M.Y.), and Department of Otolaryngology (L.H.P.N.), McGill University, Montreal, Quebec, Canada Additional Supporting Information may be found in the online version of this article. Editor’s Note: This Manuscript was accepted for publication on September 5, 2017. The authors have no funding, financial relationships, or conflicts of interest to disclose. Send correspondence to Lily H. P. Nguyen, Department of Otolaryngology–Head and Neck Surgery, Montreal Children’s Hospital, McGill University, 1001 Boulevard Decarie, A02.3015, H4A 3J1 Montreal, Quebec, Canada. E-mail: [email protected] DOI: 10.1002/lary.26943
Laryngoscope 00: Month 2017
coaching and formative assessment on a continual basis and the decoupling of development of competence from time in training.5 Thus, to support judgments of competence, provide feedback, and document the trajectory of a resident toward competence, it is critical to have assessment tools that are of high quality and designed for use in direct-observation workplace-based assessment contexts. Of the many roles expected of a competent physician,6,7 surgical technical skills are well represented in the learning competences expected of surgical residents.1 To provide feedback or to assess residence performance of OTL-HNS procedures, assessment tools for direct observation should be developed, appropriately utilized, and their use carefully monitored. Other surgical specialties have extensive literature reporting on several validated assessment tools of technical skills, as well as toolboxes to help surgical educators select the instruments best suited for their context.8–10 In a recent large systematic review of all surgical specialties that presented 106 assessment tools, only one of the core procedures for OTL-HNS was represented.8 To our knowledge, there is no work reporting on the available OTL-HNS assessment tools for evaluating residents’ technical surgical skills in the operating room (OR). Here, we report on a systematic review process that has resulted in a toolbox that could be used by surgical educators to inform the development of programs of assessment to support CBME in OTL-HNS. et al.: Toolbox for Assessment of Technical Skills Labbe
1
METHODS Search Strategy A systematic search of the scientific literature was designed with the support of a medical librarian to identify the literature reporting on tools for assessing technical skills in OTL-HNS. The search strategy was adapted and executed in MEDLINE, PubMed, Embase, and the Cochrane Database of Systematic Reviews (see Supporting Appendix I in the online version of this article). The search keywords included a combination of words related to technical skills assessment such as “measurement,” “evaluation,” “competency,” and “proficiency,” as well as specific names of OTLHNS–related procedures that were compiled by the authors. The complete set of search strategies can be obtained from the authors on request. The search included published articles from inception of the databases (as early as 1946) to 2016 in OTL-HNS.
Inclusion and Exclusion Criteria Articles in the English or French language, that presented a novel assessment tool focused on assessing the technical skills of a trainee, that could be used in the operating room, and were developed for resident assessment were included. Rater-based assessments, defined as instruments designed for completing during or following direct observation in the OR by a clinical supervisor,11,12 were identified. Tools developed to assess performance of a trainee on specific procedures were included, whether the tool was developed and validated in the OR, in a simulation setting, or on cadavers. Assessment tools that were solely based on quantitative metrics (such as time to completion of a task) and publications where the main goal of the study was not to directly report on a new assessment tool were excluded.
Study Selection Every title was reviewed by two independent authors for inclusion and exclusion following rater training (M.L. and J.L.). Discrepancies were discussed to reach consensus. The same process was subsequently performed for abstracts and full-text articles. Inter-rater reliability was not measured.
Data Extraction For all articles that met inclusion criteria, title, author(s), year of publication, journal, country where the study was conducted, setting (e.g., OR, simulation, cadaver), sample size, and level of training of the participants was extracted. Information on the tool itself was also extracted, including rating-scale format (e.g., global rating scale [GRS], task-specific checklist [TSC]), number of items, and type of scale (e.g., binary, Likert scale).
RESULTS Search Results Results were reported according to the Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA) guidelines13 (Fig. 1). Seven hundred thirty-six unique articles were identified. Following review of titles and abstracts, 68 full-text articles were reviewed. Sixteen were found to meet the complete list of inclusion criteria and were included in our review.
Description of the Assessment Tools Characteristics of the 16 assessment tools included in this review are detailed in Table I. Tools reflected a Laryngoscope 00: Month 2017
2
Fig. 1. Identification, screening, and study selection process according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting structure. OTL 5 otolaryngology. [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]
total of 11 different OTL-HNS procedures; four in otology, two in laryngology, two in rhinology, two in head and neck, and one in general otolaryngology. Eleven (68.8%) of the tools were composed of both a TSC and a GRS, three (18.6%) only used a GRS, and two (12.6%) only used a TSC. Five tools (31.3%) had a final component consisting of a global judgment on overall competency, and seven tools (43.8%) had a space for narrative feedback. Fourteen tools (87.5%) used a five-point Likert scale, and two (12.5%) used a binary scale (yes/no or complete/not complete).
Description of Development and Validation Context The country of origin of each instrument is detailed it Table I. All tools were designed to assess trainees at the resident level. Five tools (31.3%) reported comparing resident and faculty member performance during their validation process. The specific postgraduate training year of the participants is described in Table I. Eleven (68.8%) of the tools were developed and validated in the OR, four (25.0%) were validated on human cadavers, and one (6.2%) on a simulator.
DISCUSSION To our knowledge, we present the first systematic review of direct-observation workplace-based assessment tools for technical skills in OTL-HNS residency training. To support the development of an assessment program within competency-based medical education, we offer a toolbox of 16 instruments covering a total of 11 OTLHNS procedures that could be used to assess technical skills in the operating room. A task-specific checklist (TSC) component was present in the vast majority of our tools. This form of assessment focuses on directly observable actions and et al.: Toolbox for Assessment of Technical Skills Labbe
Laryngoscope 00: Month 2017
et al.: Toolbox for Assessment of Technical Skills Labbe
3
GRS 9
UK
Myringotomy and tube insertion37,38
GRS 12
USA TSC 10
5-point Likert scale
5-point Likert scale
5-point Likert scale 5-point Likert scale
Binary, 4-point Likert scale
Binary
Binary, 5-point Likert scale 5-point Likert scale
5-point Likert scale
5-point Likert scale 5-point Likert scale
5-point Likert scale
5-point Likert scale
5-point Likert scale
5-point Likert scale
5-point Likert scale
Evaluation
1–5
3, fellow, staff
1–5 1–6
1–5
2–5
2–5
2, 3 1 staff
Junior, senior, staff
2–5 2–6
SHO to SpR4
1–5 1 staff
1–5
2–5
Unspecified 1 staff
PGY Level
OR
OR
Human cadaver OR
Cadaver bones
Human cadaver
OR
OR
OR
OR OR
OR
Human cadaver OR
OR (humans or animals)
Simulator
Setting
Global Judgment
GRS 5 global rating scale; N/A 5 not available; OR 5 operating room; PGY 5 postgraduate year; SHO 5 senior house officer; SpR4 5 specialist registar 4; TSC 5 task-specific checklist.
Saudi Arabia
TSC 22, GRS 10 TSC 11, GRS 11
USA USA
Tonsillectomy43,44
Tracheotomy45
TSC 16, GRS 4
Canada
Mastoidectomy40–42
TSC 35
USA
Temporal bone dissection/
TSC 8, GRS 7
Saudi Arabia
Septoplasty39
TSC 30, GRS 10
TSC 18,GRS N/A TSC, 10, GRS 9
Canada
GRS 8
USA USA
TSC 10, GRS 7
Canada and Saudi Arabia
UK
TSC 21, GRS 10
USA
Hemi/ thyroidectomy35,36
TSC 12, GRS 10
USA
Direct laryngoscopy and rigid bronchoscopy32 Endoscopic sinus surgery17,33,34
TSC 8, GRS 7
No. of Items
Australia
Origin
Cochlear implant surgery31
Procedure
TABLE I. Summary of Published Technical Skills Assessment Tools in Otolaryngology–Head and Neck Surgery Procedures.
Narrative Feedback
thus provides a defensible mean to assess performance. TSCs are useful in providing timely feedback with specific improvement goals to trainees as well as defining areas of weaknesses that could require remedial activities.14,15 As they provide feedback that is specific to a technical task rather than overall competency, TSCs could be of most value when used in a formative setting, a low stake environment where the goal is to monitor the trainee’s learning. However, given the specificity of TSCs, their use necessitates careful consideration of what is included in a checklist for a given procedure, and the existence of a particular tool for each of the procedures of a surgical subspecialty. A GRS component was found in a large majority of our tools. An obvious advantage of those nonspecific scales is the acceptability for surgical educators, who would need to become familiar with only one general assessment.16 A GRS offers an interesting opportunity for assessing surgical competence rather than technical skills alone.15 When compared to a TSC, the disadvantage of a GRS may be the necessity for more extensive rater training.16,17 The implementation of any new assessment tool should be accompanied by rater training and quality monitoring practices. Lack of rater training may lead to inconsistent ratings of performance, and may leave the assessment scores difficult to interpret.17 A GRS has, however, been reported to be more reliable and better able to differentiate across levels of expertise.18 With proper rater training, a GRS could probably be best situated for high-stakes summative settings where the goal is to evaluate the trainee’s skills at the end of a rotation, and could provide a better assessment of overall surgical competency. Although some centers are working to develop the procedure-specific assessment tools similar to those compiled here, there is a clear trend in the literature outside of OTL-HNS toward a more centralized way of assessing technical skills of residents. The O-SCORE19,20 and the Zwisch model21,22 are examples of tools that offer what has been considered a more intuitive assessment approach, where the trainee is rated based on the involvement of the attending surgeon during the procedure. This concept of entrustability is directly linked to CBME goals.23 Tools like the O-SCORE or the Zwisch model could overcome known disadvantages of more traditional instruments, such as long delays to completion of ratings,24,25 and could palliate concerns that CBME increases the assessment role of a clinical educator to the detriment of the quality of the learning environment.26–28 Further research is needed to evaluate the feasibility of more generic tools such as the O-SCORE in the context of OTL-HNS procedures. Regardless of the structure of assessment tools identified in our review, we found that out of the 114 OTL-HNS procedures that are considered as core competencies to achieve during residency training,3 only 11 were covered by the assessment tools identified and summarized in this review. This is in contrast to other specialties such as general surgery, where certain procedures already have extensive literature published on assessment of residents’ technical skills.9,10 This paucity Laryngoscope 00: Month 2017
4
of literature may become a barrier to the successful implementation of CBME, and should raise concerns regarding the evidence base of assessment within OTLHNS.26 The main limitation of our study is that we did not specifically evaluate or incorporate the sources of validity evidence nor the specific purpose (i.e., formative or summative) of the tools reported here. Although several modern validity frameworks have been integrated into the medical education literature (notably Kane via Cook et al.29 and Messick via Downing30), no formal evaluation of validity evidence was included in this review. However, we reported on the context (e.g., OR, simulation) in which each tool was originally developed, which suggests the circumstances where the instrument is best suited for use. Upon implementation of those instruments in a residency curriculum, attention should always be paid to the original reported validity evidence, and we strongly encourage careful quality monitoring for any assessment tool used. Another limitation of this study lies in the fact that we excluded metric-based tools to focus on rater-based assessments. However, given the emphasis on direct observation and workplace-based assessment within CBME, we feel that limiting our review to rater-based assessments was appropriate.
CONCLUSION We created a toolbox of assessment tools reported for the rater-based assessment of resident’s technical skills for OTL-HNS operative procedures following a systematic review. These tools could be used to create a program of assessment to support formative and summative assessment within OTL-HNS residency training programs. More research is, however, required in OTL-HNS to expand available tools, expand evidence of validity supporting their use across a variety of contexts, and for consideration of adopting a global instrument to assess surgical competency.
Acknowledgments The authors thank Joshua Lubov, a medical student at McGill University, for his help with data collection.
BIBLIOGRAPHY 1. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach 2010;32:638–645. 2. Schuwirth L, van der Vleuten C, Durning SJ. What programmatic assessment in medical education can learn from healthcare. Perspect Med Educ 2017;6:211–215. 3. van der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. Twelve tips for programmatic assessment [published online November 20, 2014]. Med Teach doi: 10.3109/0142159X.2014. 973388. 4. Potts JR III. Assessment of competence: the Accreditation Council for Graduate Medical Education/Residency Review Committee perspective. Surg Clin North Am 2016;96:15–24. 5. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach 2010;32: 676–682. 6. Royal College of Physicians and Surgeons of Canada. The CanMEDS 2015 framework. Available at: http://canmeds.royalcollege.ca. Accessed May 2017. 7. Accreditation Council for Graduate Medical Education general competencies. Available at: http://www.acgme.org. Accessed June 2016. 8. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB. Observational tools for assessment of procedural skills: a systematic review. Am J Surg 2011;202:469–480.e466.
et al.: Toolbox for Assessment of Technical Skills Labbe
9. Ghaderi I, Manji F, Park YS, et al. Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 2015;261:251– 262. 10. Watanabe Y, Bilgic E, Lebedeva E, et al. A systematic review of performance assessment tools for laparoscopic cholecystectomy. Surg Endosc 2016;30:832–844. 11. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA 2009;302:1316–1326. 12. Epstein RM. Assessment in medical education. N Engl J Med 2007;356: 387–396. 13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 2009;62:e1–e34. 14. Archer JC. State of the science in health professional education: effective feedback. Med Educ 2010;44:101–108. 15. Norman GR, Van der Vleuten CP, De Graaff E. Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Med Educ 1991;25:119–126. 16. Ilgen JS, Ma IW, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ 2015;49:161–173. 17. Lin SY, Laeeq K, Ishii M, et al. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy 2009;23:354–359. 18. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ 2003;37:1012–1016. 19. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med 2012;87:1401–1407. 20. MacEwan MJ, Dudek NL, Wood TJ, Gofton WT. Continued Validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): use in the simulated environment. Teach Learn Med 2016;28:72–79. 21. DaRosa DA, Zwischenberger JB, Meyerson SL, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ 2013;70:24–30. 22. George BC, Teitelbaum EN, Meyerson SL, et al. Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ 2014;71:e90–e96. 23. Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med 2016;91:186–190. 24. Laeeq K, Francis HW, Varela DA, Malik MU, Cummings CW, Bhatti NI. The timely completion of objective assessment tools for evaluation of technical skills. Laryngoscope 2012;122:2418–2421. 25. Kim MJ, Williams RG, Boehler ML, Ketchum JK, Dunnington GL. Refining the evaluation of operating room performance. J Surg Educ 2009;66: 352–356. 26. Hawkins RE, Welcher CM, Holmboe ES, et al. Implementation of competency-based medical education: are we addressing the concerns and challenges? Med Educ 2015;49:1086–1102. 27. Leung WC. Competency based medical training: review. BMJ 2002;325: 693–696.
Laryngoscope 00: Month 2017
28. Malone K, Supri S. A critical time for medical education: the perils of competence-based reform of the curriculum. Adv Health Sci Educ Theory Pract 2012;17:241–246. 29. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ 2015;49:560–575. 30. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ 2003;37:830–837. 31. Piromchai P, Kasemsiri P, Wijewickrema S, Ioannou I, Kennedy G, O’Leary S. The construct validity and reliability of an assessment tool for competency in cochlear implant surgery. Biomed Res Int 2014;2014: 192741. 32. Ishman SL, Brown DJ, Boss EF, et al. Development and pilot testing of an operative competency assessment tool for pediatric direct laryngoscopy and rigid bronchoscopy. Laryngoscope 2010;120:2294–2300. 33. Marglani O, Alherabi A, Al-Andejani T, Javer A, Al-Zalabani A, Chalmers A. Development of a tool for Global Rating of Endoscopic Surgical Skills (GRESS) for assessment of otolaryngology residents. B-ENT 2012;8:191– 195. 34. Syme-Grant J, White PS, McAleer JP. Measuring competence in endoscopic sinus surgery. Surgery 2008;6:37–44. 35. Stack BC Jr, Siegel E, Bodenner D, Carr MM. A study of resident proficiency with thyroid surgery: creation of a thyroid-specific tool. Otolaryngol Head Neck Surg 2010;142:856–862. 36. Diaz Voss Varela DA, Malik MU, Thompson CB, Cummings CW, Bhatti NI, Tufano RP. Comprehensive assessment of thyroidectomy skills development: a pilot project. Laryngoscope 2012;122:103–109. 37. Bowles PF, Harries M, Young P, Das P, Saunders N, Fleming JC. A validation study on the use of intra-operative video recording as an objective assessment tool for core ENT surgery. Clin Otolaryngol 2014;39:102– 107. 38. Schwartz J, Costescu A, Mascarella MA, et al. Objective assessment of Myringotomy and tympanostomy tube insertion: a prospective singleblinded validation study. Laryngoscope 2016;126:2140–2146. 39. Obeid AA, Al-Qahtani KH, Ashraf M, Alghamdi FR, Marglani O, Alherabi A. Development and testing for an operative competency assessment tool for nasal septoplasty surgery. Am J Rhinol Allergy 2014;28:e163– e167. 40. Butler NN, Wiet GJ. Reliability of the Welling scale (WS1) for rating temporal bone dissection performance. Laryngoscope 2007;117:1803–1808. 41. Zirkle M, Taplin MA, Anthony R, Dubrowski A. Objective assessment of temporal bone drilling skills. Ann Otol Rhinol Laryngol 2007;116:793– 798. 42. Laeeq K, Bhatti NI, Carey JP, et al. Pilot testing of an assessment tool for competency in mastoidectomy. Laryngoscope 2009;119:2402–2410. 43. Ahmed A, Ishman SL, Laeeq K, Bhatti NI. Assessment of improvement of trainee surgical skills in the operating room for tonsillectomy. Laryngoscope 2013;123:1639–1644. 44. Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument to measure surgical performance at tonsillectomy. Laryngoscope 2005;115:2127–2137. 45. Al-Qahtani KH, Alkhalidi AM, Islam T. Tool for assessing surgical tracheostomy skills in otolaryngology residents. B-ENT 2015;11:275–280.
et al.: Toolbox for Assessment of Technical Skills Labbe
5
