The Laryngoscope C 2014 The American Laryngological, V
Rhinological and Otological Society, Inc.
Early Detection of Factual Knowledge Deficiency and Remediation in Otolaryngology Residency Education Michael P. Platt, MD, MS; Elizabeth M. Davis, MD; Kenneth Grundfast, MD; Gregory Grillone, MD Objectives/Hypothesis: Within otolaryngology residency training, the annual Otolaryngology Training Examination (OTE) is the primary method used to assess, quantify, and compare the factual knowledge acquired by each resident. The objective of this study was to develop a more frequent method for tracking of factual knowledge to prevent educational delay. Study Design: Retrospective analysis of educational scores. Methods: For each didactic lecture within a single otolaryngology residency training program, multiple choice questions were provided before and after each lecture. Questions were based on lecture objectives that were derived from the American Board of Otolaryngology curriculum. Scores were tracked over the course of 1 academic year and compared to the scores of residents on the OTE administered in that year to determine correlation with a validated measure of factual knowledge. The effect of remedial measures on improvement in OTE scores was determined. Results: Over the course of 1 academic year, there were 328 questions presented to 12 residents before and after 32 lectures in the didactic program. Ten residents completed an average of 244 questions. Overall OTE scores demonstrated a significant and very strong correlation to lecture question scores (Pearson r 5 0.86, P 5.002). Remedial measures for residents during the previous 5 years who had inadequate OTE scores were successful in improving scores (P 5.002). Conclusions: A structured didactic program that uses review questions to assess knowledge can be used to track acquisition of factual knowledge. Early identification of residents with deficiencies facilitates the development of individualized learning plans that result in successful remediation. Key Words: Resident education, graduate medical education, otolaryngology education, in-service examination. Level of Evidence: NA Laryngoscope, 124:E309–E311, 2014
INTRODUCTION Medical knowledge is one of the six core domains identified by the Accreditation Council on Graduate Medical Education (ACGME) as a competency that must be acquired by residents during their years in postgraduate medical education. At the conclusion of 5 years of otolaryngology residency, the extent to which the resident has acquired competency in the six ACGME domains is judged by the residency program director (RPD), and those residents who have demonstrated competency are recommended by the RPD to be examined by the American Board of Otolaryngology (ABOto), and as such, are eligible for certification by the ABOto. Monitoring of resident medical knowledge throughout the years of residency training is particularly challenging given the paucity of objective measures of learning in
From the Department of Otolaryngology–Head and Neck Surgery, Boston University School of Medicine, Boston, Massachusetts, U.S.A Editor’s Note: This Manuscript was accepted for publication January 7, 2014. The authors have no funding, financial relationships, or conflicts of interest to disclose. Send correspondence to Gregory Grillone, MD, Boston University Medical Center, FGH Building, 820 Harrison Ave, 4th Floor, Boston, MA 02118. E-mail: [email protected] DOI: 10.1002/lary.24589
Laryngoscope 124: August 2014
residency. In fact, there are few validated standard objective measures of factual knowledge that can be used during the years of residency training in otolaryngology to assist residents in learning their gaps in knowledge. The Otolaryngology Training Examination (OTE) can be used effectively to establish a baseline of specialty knowledge at the beginning of residency training, and then monitor changes on a yearly basis over the course of the residency training. The 2012 examination consisted of 300 multiple choice questions that evaluated recall of factual knowledge, interpretation skills, and problem-solving abilities, of which 192 items were used for scoring purposes.1 The OTE questions were classified by subtest areas of allergy, fundamentals, head and neck, laryngology, otology, pediatrics, plastic and reconstructive, rhinology, and sleep. The content of these subject areas are outlined in the ABOto syllabus, which encompasses the required collection of information that is needed for completion of residency training.1 Formal teaching of otolaryngology knowledge is a requirement of the ACGME. There is currently no described method for tracking of educational knowledge in more frequent intervals than the yearly OTE. Residents whose factual knowledge is deficient cannot be identified or tracked more frequently than once each year. Early identification of residents with educational Platt et al.: Early Detection of Knowledge Deficiency
E309
delay is an important prerequisite for successful implementation of remediation plans.2,3 This study evaluates a new method for frequent monitoring of resident knowledge so that early remediation can be instituted for residents who are deficient in factual knowledge.
MATERIALS AND METHODS Approval of the Boston University Medical Center Institutional Review Board was obtained for this study. A didactic program was developed for an otolaryngology residency program that included clearly specified learning objectives and multiple choice questions for each lecture. Lecture objectives based on the ABOto curriculum were created by the residency faculty and individual lecturers. Review questions that were based on the learning objectives were written for each lecture by the individual lecturer or another otolaryngology faculty member. For each lecture, half of the questions were given before and half after each lecture (pre- and post-lecture, total 4–10 questions per lecture). Question responses for each resident were tracked via electronic response system or paper over the course of 1 academic year. The results of pre- and post-lecture questions were compared to scores of each resident on the OTE given toward the end of the didactic program in that year. Statistical analysis was performed with linear dependence analysis using Pearson product-moment correlation, with significance set at P
Rhinological and Otological Society, Inc.
Early Detection of Factual Knowledge Deficiency and Remediation in Otolaryngology Residency Education Michael P. Platt, MD, MS; Elizabeth M. Davis, MD; Kenneth Grundfast, MD; Gregory Grillone, MD Objectives/Hypothesis: Within otolaryngology residency training, the annual Otolaryngology Training Examination (OTE) is the primary method used to assess, quantify, and compare the factual knowledge acquired by each resident. The objective of this study was to develop a more frequent method for tracking of factual knowledge to prevent educational delay. Study Design: Retrospective analysis of educational scores. Methods: For each didactic lecture within a single otolaryngology residency training program, multiple choice questions were provided before and after each lecture. Questions were based on lecture objectives that were derived from the American Board of Otolaryngology curriculum. Scores were tracked over the course of 1 academic year and compared to the scores of residents on the OTE administered in that year to determine correlation with a validated measure of factual knowledge. The effect of remedial measures on improvement in OTE scores was determined. Results: Over the course of 1 academic year, there were 328 questions presented to 12 residents before and after 32 lectures in the didactic program. Ten residents completed an average of 244 questions. Overall OTE scores demonstrated a significant and very strong correlation to lecture question scores (Pearson r 5 0.86, P 5.002). Remedial measures for residents during the previous 5 years who had inadequate OTE scores were successful in improving scores (P 5.002). Conclusions: A structured didactic program that uses review questions to assess knowledge can be used to track acquisition of factual knowledge. Early identification of residents with deficiencies facilitates the development of individualized learning plans that result in successful remediation. Key Words: Resident education, graduate medical education, otolaryngology education, in-service examination. Level of Evidence: NA Laryngoscope, 124:E309–E311, 2014
INTRODUCTION Medical knowledge is one of the six core domains identified by the Accreditation Council on Graduate Medical Education (ACGME) as a competency that must be acquired by residents during their years in postgraduate medical education. At the conclusion of 5 years of otolaryngology residency, the extent to which the resident has acquired competency in the six ACGME domains is judged by the residency program director (RPD), and those residents who have demonstrated competency are recommended by the RPD to be examined by the American Board of Otolaryngology (ABOto), and as such, are eligible for certification by the ABOto. Monitoring of resident medical knowledge throughout the years of residency training is particularly challenging given the paucity of objective measures of learning in
From the Department of Otolaryngology–Head and Neck Surgery, Boston University School of Medicine, Boston, Massachusetts, U.S.A Editor’s Note: This Manuscript was accepted for publication January 7, 2014. The authors have no funding, financial relationships, or conflicts of interest to disclose. Send correspondence to Gregory Grillone, MD, Boston University Medical Center, FGH Building, 820 Harrison Ave, 4th Floor, Boston, MA 02118. E-mail: [email protected] DOI: 10.1002/lary.24589
Laryngoscope 124: August 2014
residency. In fact, there are few validated standard objective measures of factual knowledge that can be used during the years of residency training in otolaryngology to assist residents in learning their gaps in knowledge. The Otolaryngology Training Examination (OTE) can be used effectively to establish a baseline of specialty knowledge at the beginning of residency training, and then monitor changes on a yearly basis over the course of the residency training. The 2012 examination consisted of 300 multiple choice questions that evaluated recall of factual knowledge, interpretation skills, and problem-solving abilities, of which 192 items were used for scoring purposes.1 The OTE questions were classified by subtest areas of allergy, fundamentals, head and neck, laryngology, otology, pediatrics, plastic and reconstructive, rhinology, and sleep. The content of these subject areas are outlined in the ABOto syllabus, which encompasses the required collection of information that is needed for completion of residency training.1 Formal teaching of otolaryngology knowledge is a requirement of the ACGME. There is currently no described method for tracking of educational knowledge in more frequent intervals than the yearly OTE. Residents whose factual knowledge is deficient cannot be identified or tracked more frequently than once each year. Early identification of residents with educational Platt et al.: Early Detection of Knowledge Deficiency
E309
delay is an important prerequisite for successful implementation of remediation plans.2,3 This study evaluates a new method for frequent monitoring of resident knowledge so that early remediation can be instituted for residents who are deficient in factual knowledge.
MATERIALS AND METHODS Approval of the Boston University Medical Center Institutional Review Board was obtained for this study. A didactic program was developed for an otolaryngology residency program that included clearly specified learning objectives and multiple choice questions for each lecture. Lecture objectives based on the ABOto curriculum were created by the residency faculty and individual lecturers. Review questions that were based on the learning objectives were written for each lecture by the individual lecturer or another otolaryngology faculty member. For each lecture, half of the questions were given before and half after each lecture (pre- and post-lecture, total 4–10 questions per lecture). Question responses for each resident were tracked via electronic response system or paper over the course of 1 academic year. The results of pre- and post-lecture questions were compared to scores of each resident on the OTE given toward the end of the didactic program in that year. Statistical analysis was performed with linear dependence analysis using Pearson product-moment correlation, with significance set at P
