2014 APDS SPRING MEETING
Fostering and Assessing Professionalism and Communication Skills in Neurosurgical Education Ricardo B.V. Fontes, PhD,* Nathan R. Selden, PhD,† and Richard W. Byrne, MD* Department of Neurosurgery, Rush University Medical Center, Chicago, Illinois; and †Department of Neurological Surgery, Oregon Health & Science University, Portland, Oregon *
INTRODUCTION: Incorporation of the 6 ACGME core
competencies into surgical training has proven a considerable challenge particularly for the two primarily behavioral competencies, professionalism and interpersonal and communication skills. We report on experience with two specific interventions to foster the teaching and continuous evaluation of these competencies for neurosurgery residents. MATERIAL AND METHODS: In 2010, the Society of
Neurological Surgeons (SNS) organized the first comprehensive Neurosurgery Boot Camp courses, held at six locations throughout the US and designed to assess and teach not only psychomotor skills but also components of all six Accreditation Council for Graduate Medical Education (ACGME) core competencies. These courses are comprised of various educational methodologies, including online material, faculty lectures, clinical scenario and group discussions, manual skills stations, and pre- and post-course assessments. Resident progress in each of the 6 ACGME competencies is now tracked using the neurosurgical Milestones, developed by the ACGME in collaboration with the SNS. In addition, the Milestones drafting group for neurosurgery has formulated a milestone-compatible evaluation system to directly populate Milestone reports. These evaluations utilize formative, summative, and 360-degree evaluations that are considered by a faculty core competency committee in finalizing milestones levels for each resident.
RESULTS: Initial attendance at the 2010 Boot Camp course
was 94% of the incoming resident class and in subsequent years, 100%. Pre- and post-course surveys demonstrated a R.W.B. would like to disclose consultant work for Integra and Stryker, which are not relevant to the topic of this paper. N.R.S. is the chair of the ACGME Neurological Surgery Milestones Group and the chair of the Committee on Resident Education of the Society of Neurological Surgeons. R.B.V.F. has nothing to disclose. Correspondence: Inquiries to Ricardo B.V. Fontes, MD, PhD, Department of Neurosurgery, Rush University Medical Center, 1725 W Harrison, Suite 855, Chicago, IL 60612; fax: (312) 942-2176; E-mail: [email protected], [email protected]
significant and sustained increase in knowledge. The value of these courses has been recognized by the ACGME, which requires Boot Camp or equivalent participation prior to acting with indirect supervision during clinical activities. Neurosurgery was one of 7 early Milestone adopter specialties, beginning use in July, 2013. Early milestone data will establish benchmarks prior to utilization for “high stake” decisions such as promotion, graduation, and termination. CONCLUSIONS: The full impact of the neurosurgical
Boot Camps and Milestones on residency education remains to be measured, although published data from the first years of the Boot Camp Courses demonstrate broad acceptance and early effectiveness. A complementary junior resident course has now been introduced for rising secondyear residents. The Milestones compatible evaluation system now provides for multi-source formative and summative evaluation of neurosurgical residents within the new ACGME reporting rubric. Combined with consensus milestone assignments, this system provides new specificity and objectivity to resident evaluations. The correlation of milestone level assignments with other measurements of educational outcome awaits further study. ( J Surg 71:e83C 2014 Association of Program Directors in Surgery. e89. J Published by Elsevier Inc. All rights reserved.) KEY WORDS: internship and residency, assessment, edu-
cational, graduate medical education, neurosurgery COMPETENCIES: Professionalism, Interpersonal and Com-
munication Skills, Practice-Based Learning and Improvement
INTRODUCTION Professionalism may be defined as “the skill, good judgment, and polite behavior that is expected from a person who is trained to do a job well.”1 It is derived from the Latin word profiteor, to profess, in the connotation of making a formal commitment as in a monastic oath.2 Therefore, simply
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2014.06.016
e83
engaging in a profession does not make one a “professional.” Many authors have tried to specify the additional characteristics of professionalism, such as the possession of special skills and knowledge, the exercise of autonomous thought and judgment, and a responsibility to individuals and society through commitment to a set of principles.2 In addition, the Accreditation Council for Graduate Medical Education (ACGME) has noted the existence of certain moral principles underlying professionalism: excellence, humanism, accountability, and altruism.3 All the “old professions,” including medicine, the clergy, teaching, and law, adopt definitions of professionalism that go beyond simply engaging in a trade.2 For physicians and surgeons, the process of becoming a professional involves the transition from a medical student to an experienced clinician. Until the 1970s, the teaching of professionalism in medicine involved exposure to “role models,” such as relatively distant senior faculty, more accessible junior faculty, residents, and even peer medical students using examples and parables and demonstrating a set of unwritten rules and attitudes, with little explanation provided and no reflection required on the part of the trainee.4,5 Today, most medical schools and residency programs provide an explicit professionalism curriculum, including course and lecture topics such as ethics, humanism, and the doctor-patient relationship. However, the ideal process for professionalism training is likely to encompass elements of both approaches. Although formal courses provide organized and foundational knowledge, their effectiveness is uncertain, whereas the “hidden curriculum” of mentorship, taught during dayto-day activities in the hospital, clinic, and operating room, is insufficiently comprehensive and objective.3 These combined training strategies require 3 features to be successful: (1) setting well-defined professionalism goals within the residency program, (2) the use of continuous, multimodal assessment, and (3) the freedom to employ teaching strategies appropriate to the defining characteristics of each clinical specialty, particularly in the case of demanding and challenging surgical disciplines. We report some early experience with 2 specific interventions developed to foster the teaching and continuous evaluation of professionalism and interpersonal communication skills within neurosurgery programs in the United States: the Society of Neurological Surgeons’ (SNS) “boot camp” courses (postgraduate year [PGY]-1 and -2) and the ACGME Neurological Surgery Milestones and their associated assessment tools.
BACKGROUND In 2002, the ACGME and the residency review committee for each subspecialty mandated that residency education incorporate the 6 ACGME core competencies into the training curriculum: (1) patient care, (2) medical e84
knowledge, (3) practice-based learning and improvement, (4) interpersonal and communication skills (ICS), (5) professionalism, and (6) systems-based practice.6 Of these requirements, 4 have been tenets of residency training since William Halsted conceived the modern surgical training in the 1880s.7 However, the formal teaching of the 2 primarily behavioral competencies, “ICS” and “professionalism,” posed a complex and novel challenge to neurosurgical educators, because of the perceived abstract nature of these competencies and the lack of previous formal training in these competencies by neurosurgical educators themselves. Surgeons, including neurosurgeons, are mostly conservative regarding educational hierarchies and pedagogy.8-11 Some surgical educators argued that behavioral competencies are adequately taught using the apprenticeship model. Nevertheless, in response to the new regulatory imperative, residency programs slowly developed and implemented specific curricula for professionalism and ICS education, likely improving patient care delivery, safety, and satisfaction.12,13 Within this context, neurosurgical educators developed the SNS boot camp courses.
SNS BOOT CAMP COURSE INITIATIVE The SNS is the oldest neurosurgical professional organization in the world and comprises program directors, department chairs, and other educational leaders within the specialty of neurological surgery in the United States.14 Following the introduction of ACGME core competencies in 2009, as described earlier, the ACGME incorporated the first year of postgraduate training (PGY-1 or “intern”) formally into all US neurosurgery programs, thus replacing the so-called preliminary year in general surgery.15 Concerns about the heterogeneity of incoming residents in medical knowledge, psychomotor skill, and proficiency in professionalism, communications, and leadership led the Committee on Resident Education of the SNS to establish foundational courses for entering and for junior neurosurgical residents to teach and assess this material. Subsequently, an SNS steering committee initially evaluated a small number of existing “pilot courses” that focused on psychomotor skills related to basic neurosurgical procedures, with some variable exposure to professionalism and ICS concepts. The aim was to expand these courses, develop a systematic curriculum vetted by program directors nationally (through the SNS), and replicate them at different regional locations throughout the country, to reach all trainees in US residencies. In 2010, the first national courses were held at 6 regional locations with overwhelmingly positive trainee and faculty reviews.16 These courses were held over 3 weekends (2 simultaneous courses per weekend). The course structure included 9 didactic lectures, 10 procedural skill stations, and 6 emergency craniotomy skill stations (Table 1). All course
Journal of Surgical Education Volume 71/Number 6 November/December 2014
TABLE 1. Didactic Content of the SNS Boot Camp for PGY-1 Residents Standardized Lectures Professionalism, supervision, and pearls* Neurological and neurotrauma assessment Emergency cranial radiology assessment Emergency spinal radiology assessment ICP management Unstable neurosurgical patient: case scenarios* Emergency evaluation and management of hydrocephalus patients Making the incision: scalp blood supply Airway management and intubation
Bedside Neurosurgical Procedures and Equipment Stations VP shunt tap and valve programming† Lumbar puncture and drain† ICP monitor† External ventricular drain† Cranial fixation and cervical traction† Central and arterial lines†
Craniotomy Skills Stations Drilling and bone dissection† Cranial flaps† Dural closure† Flap fixation† Cranioplasty† Skin closure
Clinical decision simulator†
ICP, intracranial pressure; VP, ventriculoperitoneal. *Behavioral competency component. † Stations where residents are assessed with checklists as described in the text.
material was approved by the steering committee and included considerable emphasis on all the ACGME competencies, including professionalism and ICS, in addition to psychomotor skills. All course material was made available online for preparatory learning and review at http://www. societyns.org/BootCamp/BootCampCourses.asp, and all residents completed precourse and postcourse evaluations including the assessment of representative didactic knowledge of taught and nontaught (control) material. From inception, each course has begun with a lecture on professionalism. All courses were presented free of charge to the residents, including transportation, housing, and meals.15,17 The faculty included residency directors and other senior educators, who were not paid or provided other honoraria. In 2010, 94% of incoming US residents attended 1 of 6 regional courses, with 100% attending since 2011. Postcourse evaluations of both faculty and residents demonstrated that 100% of attendees considered that didactic objectives were achieved and recommended continuing this program. Performance assessment has been an additional priority for the courses. Besides precourse and postcourse surveys that have demonstrated significant improvement in knowledge basis (p o 0.001), the residents are also graded on their technical skill on the procedural stations.15 Currently, procedural proficiency is assessed using a set of “checklists” including key points of each procedure that must be satisfied before the attendee can receive a Pass grade. The residents are offered the opportunity to repeat each failed technical station; repeated failures are communicated to their respective program directors so that they can be addressed before the resident progresses to perform these procedures in the clinical setting. Behavioral aspects during group exercises are currently assessed with a Pass/Fail system based on the examiner’s assessment but similar checklists are under development for these exercises. Knowledge retention is finally assessed 6 months after the course through a survey sent to residents. The response rates have averaged 88% of attendees and demonstrated satisfactory retention when compared with the
initial testing.18 The value of these courses has been recognized by the ACGME and its residency review committee for Neurological Surgery, which now requires participation in the SNS PGY-1 boot camp course or an equivalent structured program before indirect supervision for patient care and bedside procedures in the live clinical environment.19 Based on the very good results obtained with the PGY-1 boot camp courses, the SNS has expanded this program to include a second set of boot camp courses attended by upcoming PGY-2 junior residents. The first of these took place in May 2013. Three courses are now held annually, each over a 3-day period. At the PGY-2 level, additional emphasis is placed on professionalism, communications, and leadership. Of the 23 PGY-2 lectures and small-group activities, 13 directly involve the behavioral competencies (Table 2). The topics include “Disclosure of Medical Errors and Risk Management,” “Leadership and Professionalism,” and “Breaking bad news.” Scenarios with behavioral and decision-making components, such as “operating room crisis” and “intensive care unit crisis,” are also included. These lectures are also available online for preparatory learning and review.
MULTIMODAL AND CONTINUOUS EVALUATION—ACGME MILESTONES AND OTHER INSTRUMENTS Clear and objective evaluation of medical trainees is not easy to accomplish. Although difficult to implement, the use of effective evaluation systems can promote improved patient care and has been adopted in some form by virtually every surgical training program in the United States.6 There is also evidence that poor evaluations during training are an independent predictor of disciplinary action by state medical boards.20 In 2009, the ACGME started a multiyear process to restructure residency training to include reporting of educational outcomes in the 6 core
Journal of Surgical Education Volume 71/Number 6 November/December 2014
e85
TABLE 2. Didactic Content of the SNS Boot Camp for PGY-2 Residents
Standardized Lectures
Mentored Small-Group Exercises and Behavioral Scenario Simulations
Cerebral vascular anatomy
Informed consent*†
Disclosure of medical errors and risk management†
Breaking bad news: poor prognosis and complication/medical error*†
Handoffs† ICU Simulator Leadership and professionalism† OR crisis† Cranial fixation Postoperative and chronic pain management
Leadership discussion*† DVT/PE Surgical infections Handoffs† Pain management
Neuronavigation Quality improvement† Safety and surgical checklists† Surgical anatomy of the spine Informed consent and exercises† ICU crisis and scenarios† Breaking bad news† Ventricular anatomy
Model-Based Clinical Simulations (SimMan, Haptic Simulator, and Cadavers) ICU crisis: SAH, hydrocephalus, spine fracture, seizure, and traumatic brain injury* OR crisis: sinus bleeding, air embolism, loss of fixation, brain swelling, and hemorrhage* Tumor resection Endoscopy Endovascular Stereotactic frame placement Navigation registration Basic cranial approaches: suboccipital, pterional, and frontoorbital Basic spinal approaches: laminectomy and pedicle screw placement
DVT, deep venous thrombosis; ICU, intensive care unit; OR, operating room; PE, pulmonary embolism; SAH, subarachnoid hemorrhage. *Stations where residents are assessed with checklists as described in the text. † Behavioral competency taught and assessed.
competencies, the milestones. The milestones are intended to track the progress of individual residents toward documented levels of skill and knowledge for safe independent practice and also to evaluate the effectiveness of the educational environment in each residency program. The result of this effort is the Next Accreditation System (NAS), which was implemented in a pilot phase in July 2013.21 In other words, NAS has abandoned the “biopsy” model (in which program accreditation evaluations were performed every 5 years) and replaced it with continuous assessment using annual and biannual reporting of milestones tracking, resident and faculty survey data, and other educational outcomes. The ACGME milestones are subspecialty-specific tools for the tracking and reporting of resident developmental progression across the breadth of training in each of the 6 ACGME competencies. The Neurological Surgery milestones were developed by an ACGME-appointed working group, with oversight by an SNS Milestones Advisory Group.16 Each milestone tool uses developmental descriptors instead of a numerical ranking system to gauge resident performance in each of the 6 ACGME competencies and also to assess progress toward independent practice over time.21 Before milestone implementation, the ACGME working group and the SNS held 3 national 4-hour workshops attended by 280 program leaders (program directors and coordinators), complete with mock resident evaluations and faculty evaluation sessions. e86
Using the NAS, resident evaluations occur every 6 months and are finalized through consideration of all evaluation and resident portfolio data by a Clinical Competency Committee comprising program clinical faculty, often with a resident representative. Evaluation results and final milestone level assignments are privately disclosed to each resident by the program director or another faculty mentor, so each resident can track their individual progress and respond, with assistance where necessary, to deficiencies. Milestone reporting also offers the possibility of using systematic nationwide benchmarks, with validation of individual data by a Clinical Competency Committee to make promotion and separation decisions for residents failing to progress, where necessary. Neurological surgery, as one of the early adopters of this evaluation model, implemented the use of the milestones in July 2013, with the initial milestone reports provided to the ACGME in December 2013.16 The neurosurgical milestones comprise 24 “milestone sets,” including 2 each covering the 4 general competencies (systems-based practice, professionalism, ICS, and practicebased learning) and 16 covering medical knowledge and patient care for each of 8 subspecialty areas (Fig.).22 Proficiency in each milestone is measured with descriptors and graded from level 1 (expected performance of an incoming resident) through level 4 (performance targeted for residency completion). Level 5 milestones are suited to the performance of an advanced resident who might have
Journal of Surgical Education Volume 71/Number 6 November/December 2014
FIGURE. From 24 neurosurgical milestone sets, 2 examples are shown: one from a technical competency (patient care, A) and another from a general competency (professionalism, B). Progression is assessed from left to right, and completion of all items within a level is required for consideration of the next level. (Copyright by the Accreditation Council for Graduate Medical Education and the American Board of Neurological Surgery.)
subspecialty interest in a particular area or goals appropriate to fellowship training or the practice of a recent residency graduate. Although the “graduation target” is level 4 proficiency, such proficiency in all areas is not currently a formal “graduation requirement,” which remains the purview of the Residency Program Director. ACGME review committees,
including neurological surgery, have stated that analysis of milestone data to ensure quality and accuracy is necessary before using the milestones for high-stake decisions.23 The ACGME Neurological Surgery Milestones Group has also created new milestones-compatible evaluations, which use the specific developmental descriptors of the milestones
Journal of Surgical Education Volume 71/Number 6 November/December 2014
e87
themselves so that their results may directly determine and populate milestones reports, following review and consensus validation by the CCC. The “3601” evaluations, which incorporate assessments by raters from a variety of professional roles (self, peers, nurses, and faculty) are particularly useful in this context.24 Although isolated self-assessment by physicians has limited predictive value against externally validated measure of performance, “3601” evaluations are generally more accurate, especially for residents in the lowest-performing ranges, who tend to overestimate their performance, particularly for the general competencies, including professionalism and communication skills.25 Conversely, high-performing residents tend to underestimate their performance across general and technical competencies.24 Other types of resident evaluation tools may be used to compile data relevant to individual milestone level determination and reporting, including objective structured assessments of technical skills and varied forms of self-assessment evaluations already in place in many institutions (Selden NR, personal communication, 2014). The objectivity of milestone developmental level reporting ultimately depends on compiling assessments from multiple clinical observers and validating the overall interpretation of these results via consensus building in the CCC. The extent of data collected and analyzed in this process poses a new and unfunded regulatory mandate on residency training programs. To reduce the associated administrative burden, the SNS, working with US national neurosurgical membership societies, plans to automate the collection and initial analysis of milestones-compatible evaluation form data. Finally, it is preferable that these and other educational innovations in neurosurgical training and assessment are carefully coordinated. For example, the milestone developmental reporting tools require mastery of a series of technical skills and professional characteristics by “level 1” residents that have therefore been included in the PGY-1 introductory boot camp course. Similarly, the SNS continues to design specific curricular elements necessary to achieve higher-level milestones in other focused areas that are more efficiently taught using regional or national teaching, simulation, or assessment methodologies and equipment.
REFERENCES
CONCLUSION
10. Idema S, Buis DR, Idema AJS, Vandertop WP.
Surgical residency training has changed significantly over the last decade. Innovations such as a structured program for professionalism and ICS teaching and multimodal evaluations have been instituted at virtually every neurological surgery program in the Unites States. Although there is no definitive method for teaching and assessing professionalism, specialty-specific goals require direct specialty involvement and development of relevant curricula. It is thus preferable that surgeons lead, rather than be led, in this process.
e88
1. Merriam-Webster.
Professionalism. Available at: 〈http://www.merriam-webster.com/dictionary/profes sionalism〉; 2014 Accessed 30.03.14.
2. Apuzzo MLJ. Facets of professionalism. World Neurosurg.
2013;79(1):1. 2012.11.050.
doi:http://dx.doi.org/10.1016/j.wneu.
3. Stern DT, Papadakis M. The developing physician—
becoming a professional. N Engl J Med. 2006;355 (17):1794-1799. doi:http://dx.doi.org/10.1056/NEJ Mra054783. 4. Stern DT, ed. Measuring Medical Professionalism.
New York: Oxford University Press; 2006. 5. Bosk CL. Forgive and Remember: Managing Medical
Failure. 2nd ed. Chicago: University of Chicago Press; 2003. 6. Hochberg MS, Berman RS, Kalet AL, et al. The
professionalism curriculum as a cultural change agent in surgical residency education. Am J Surg. 2012;203 (1):14-20. doi:http://dx.doi.org/10.1016/j.amjsurg. 2011.05.007. 7. Imber G. Genius on the Edge: The Bizarre Double
Life of Dr. William Stewart Halsted. New York, NY: Kaplan Pub.; 2010. 8. Cohen-Gadol AA, Piepgras DG, Krishnamurthy S,
Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery. 2005;56(2):398-403; [discussion 398-403]. 9. Hoh BL, Neal DW, Kleinhenz DT, Hoh DJ, Mocco J,
Barker FG 2nd. Higher complications and no improvement in mortality in the ACGME resident duty-hour restriction era: an analysis of more than 107,000 neurosurgical trauma patients in the Nationwide Inpatient Sample database. Neurosurgery. 2012; 70(6):1369-1381 [discussion 1381-1382]. http://dx. doi.org/10.1227/NEU.0b013e3182486a75. Higher complications and no improvement in mortality in the ACGME resident duty-hour restriction era: an analysis of more than 107000 neurosurgical trauma patients in the nationwide inpatient sample database. Neurosurgery. 2013;72(1):E142-E143. doi:http://dx. doi.org/10.1227/NEU.0b013e318276064f. 11. Schaller K. Neurosurgical training under European
law. Acta Neurochir (Wien). 2013;155(3):547. doi: http://dx.doi.org/10.1007/s00701-012-1579-7.
Journal of Surgical Education Volume 71/Number 6 November/December 2014
12. Hochberg MS, Kalet A, Zabar S, Kachur E, Gillespie C,
Berman RS. Can professionalism be taught? Encouraging evidence Am J Surg. 2010;199(1):86-93. doi:http: //dx.doi.org/10.1016/j.amjsurg.2009.10.002. 13. Hochberg MS, Seib CD, Berman RS, Kalet AL, Zabar
SR, Pachter HL. Perspective: malpractice in an academic medical center: a frequently overlooked aspect of professionalism education. Acad Med J Assoc Am Med Coll. 2011;86(3):365-368. 10.1097/ACM. 0b013e3182086d72. 14. The Society of Neurological Surgeons—About the
SNS. Available at: 〈http://www.societyns.org/about_ sns.html〉; 2010 Accessed 30.03.14.
15. Selden NR, Origitano TC, Burchiel KJ, et al. A
national fundamentals curriculum for neurosurgery PGY1 residents: the 2010 Society of Neurological Surgeons Boot Camp Courses. Neurosurgery. 2012;70 (4):971-981. doi:http://dx.doi.org/10.1227/NEU.0b013 e31823d7a45. 16. Swing SR, Beeson MS, Carraccio C, et al. Educational
milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ. 2013; 5(1):98-106. doi:http://dx.doi.org/10.4300/JGME05-01-33.
of postgraduate year 1 training. J Neurosurg. 2013;119 (3):796-802. doi:http://dx.doi.org/10.3171/2013.3. JNS122114. 19. Review Committee for Neurological Surgery ACGME.
Frequently asked questions: Neurological Surgery. Available at: 〈http://www.acgme.org/acgmeweb/Por tals/0/PDFs/FAQ/160_neurological_surgery_ FAQs_07012013.pdf〉; 2013 Accessed 30.03.14. 20. Papadakis MA, Teherani A, Banach MA, et al. Disciplinary
action by medical boards and prior behavior in medical school. N Engl J Med. 2005;353(25):2673-2682. doi: http://dx.doi.org/10.1056/NEJMsa052596. 21. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next
GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051-1056. doi:http: //dx.doi.org/10.1056/NEJMsr1200117. 22. Selden NR, Abosch A, Byrne RW, et al. Neurological
surgery milestones. J Grad Med Educ. 2013;5(1 suppl 1): 24-35. doi:http://dx.doi.org/10.4300/JGME-05-01s1-04. 23. The
Neurological Surgery Milestone Project. Available at: 〈http://acgme.org/acgmeweb/Portals/0/ PDFs/Milestones/NeurologicalSurgeryMilestones.pdf〉; 2013 Accessed 31.03.14.
17. Selden NR, Barbaro N, Origitano TC, Burchiel KJ.
24. Lipsett PA, Harris I, Downing S. Resident self-other
Fundamental skills for entering neurosurgery residents: report of a Pacific region “boot camp” pilot course, 2009. Neurosurgery. 2011;68(3):759-764. [discussion 764]. http://dx.doi.org/10.1227/NEU.0b013e318207 7969.
assessor agreement: influence of assessor, competency, and performance level. Arch Surg. 2011;146(8):901-906. doi:http://dx.doi.org/10.1001/archsurg.2011.172.
18. Selden NR, Anderson VC, McCartney S, Origitano
TC, Burchiel KJ, Barbaro NM. Society of Neurological Surgeons boot camp courses: knowledge retention and relevance of hands-on learning after 6 months
25. Davis DA, Mazmanian PE, Fordis M, Van Harrison
R, Thorpe KE, Perrier L. Accuracy of physician selfassessment compared with observed measures of competence: a systematic review. J Am Med Assoc. 2006; 296(9):1094-1102, doi:http://dx.doi.org/10.1001/ jama.296.9.1094.
Journal of Surgical Education Volume 71/Number 6 November/December 2014
e89
Fostering and Assessing Professionalism and Communication Skills in Neurosurgical Education Ricardo B.V. Fontes, PhD,* Nathan R. Selden, PhD,† and Richard W. Byrne, MD* Department of Neurosurgery, Rush University Medical Center, Chicago, Illinois; and †Department of Neurological Surgery, Oregon Health & Science University, Portland, Oregon *
INTRODUCTION: Incorporation of the 6 ACGME core
competencies into surgical training has proven a considerable challenge particularly for the two primarily behavioral competencies, professionalism and interpersonal and communication skills. We report on experience with two specific interventions to foster the teaching and continuous evaluation of these competencies for neurosurgery residents. MATERIAL AND METHODS: In 2010, the Society of
Neurological Surgeons (SNS) organized the first comprehensive Neurosurgery Boot Camp courses, held at six locations throughout the US and designed to assess and teach not only psychomotor skills but also components of all six Accreditation Council for Graduate Medical Education (ACGME) core competencies. These courses are comprised of various educational methodologies, including online material, faculty lectures, clinical scenario and group discussions, manual skills stations, and pre- and post-course assessments. Resident progress in each of the 6 ACGME competencies is now tracked using the neurosurgical Milestones, developed by the ACGME in collaboration with the SNS. In addition, the Milestones drafting group for neurosurgery has formulated a milestone-compatible evaluation system to directly populate Milestone reports. These evaluations utilize formative, summative, and 360-degree evaluations that are considered by a faculty core competency committee in finalizing milestones levels for each resident.
RESULTS: Initial attendance at the 2010 Boot Camp course
was 94% of the incoming resident class and in subsequent years, 100%. Pre- and post-course surveys demonstrated a R.W.B. would like to disclose consultant work for Integra and Stryker, which are not relevant to the topic of this paper. N.R.S. is the chair of the ACGME Neurological Surgery Milestones Group and the chair of the Committee on Resident Education of the Society of Neurological Surgeons. R.B.V.F. has nothing to disclose. Correspondence: Inquiries to Ricardo B.V. Fontes, MD, PhD, Department of Neurosurgery, Rush University Medical Center, 1725 W Harrison, Suite 855, Chicago, IL 60612; fax: (312) 942-2176; E-mail: [email protected], [email protected]
significant and sustained increase in knowledge. The value of these courses has been recognized by the ACGME, which requires Boot Camp or equivalent participation prior to acting with indirect supervision during clinical activities. Neurosurgery was one of 7 early Milestone adopter specialties, beginning use in July, 2013. Early milestone data will establish benchmarks prior to utilization for “high stake” decisions such as promotion, graduation, and termination. CONCLUSIONS: The full impact of the neurosurgical
Boot Camps and Milestones on residency education remains to be measured, although published data from the first years of the Boot Camp Courses demonstrate broad acceptance and early effectiveness. A complementary junior resident course has now been introduced for rising secondyear residents. The Milestones compatible evaluation system now provides for multi-source formative and summative evaluation of neurosurgical residents within the new ACGME reporting rubric. Combined with consensus milestone assignments, this system provides new specificity and objectivity to resident evaluations. The correlation of milestone level assignments with other measurements of educational outcome awaits further study. ( J Surg 71:e83C 2014 Association of Program Directors in Surgery. e89. J Published by Elsevier Inc. All rights reserved.) KEY WORDS: internship and residency, assessment, edu-
cational, graduate medical education, neurosurgery COMPETENCIES: Professionalism, Interpersonal and Com-
munication Skills, Practice-Based Learning and Improvement
INTRODUCTION Professionalism may be defined as “the skill, good judgment, and polite behavior that is expected from a person who is trained to do a job well.”1 It is derived from the Latin word profiteor, to profess, in the connotation of making a formal commitment as in a monastic oath.2 Therefore, simply
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2014.06.016
e83
engaging in a profession does not make one a “professional.” Many authors have tried to specify the additional characteristics of professionalism, such as the possession of special skills and knowledge, the exercise of autonomous thought and judgment, and a responsibility to individuals and society through commitment to a set of principles.2 In addition, the Accreditation Council for Graduate Medical Education (ACGME) has noted the existence of certain moral principles underlying professionalism: excellence, humanism, accountability, and altruism.3 All the “old professions,” including medicine, the clergy, teaching, and law, adopt definitions of professionalism that go beyond simply engaging in a trade.2 For physicians and surgeons, the process of becoming a professional involves the transition from a medical student to an experienced clinician. Until the 1970s, the teaching of professionalism in medicine involved exposure to “role models,” such as relatively distant senior faculty, more accessible junior faculty, residents, and even peer medical students using examples and parables and demonstrating a set of unwritten rules and attitudes, with little explanation provided and no reflection required on the part of the trainee.4,5 Today, most medical schools and residency programs provide an explicit professionalism curriculum, including course and lecture topics such as ethics, humanism, and the doctor-patient relationship. However, the ideal process for professionalism training is likely to encompass elements of both approaches. Although formal courses provide organized and foundational knowledge, their effectiveness is uncertain, whereas the “hidden curriculum” of mentorship, taught during dayto-day activities in the hospital, clinic, and operating room, is insufficiently comprehensive and objective.3 These combined training strategies require 3 features to be successful: (1) setting well-defined professionalism goals within the residency program, (2) the use of continuous, multimodal assessment, and (3) the freedom to employ teaching strategies appropriate to the defining characteristics of each clinical specialty, particularly in the case of demanding and challenging surgical disciplines. We report some early experience with 2 specific interventions developed to foster the teaching and continuous evaluation of professionalism and interpersonal communication skills within neurosurgery programs in the United States: the Society of Neurological Surgeons’ (SNS) “boot camp” courses (postgraduate year [PGY]-1 and -2) and the ACGME Neurological Surgery Milestones and their associated assessment tools.
BACKGROUND In 2002, the ACGME and the residency review committee for each subspecialty mandated that residency education incorporate the 6 ACGME core competencies into the training curriculum: (1) patient care, (2) medical e84
knowledge, (3) practice-based learning and improvement, (4) interpersonal and communication skills (ICS), (5) professionalism, and (6) systems-based practice.6 Of these requirements, 4 have been tenets of residency training since William Halsted conceived the modern surgical training in the 1880s.7 However, the formal teaching of the 2 primarily behavioral competencies, “ICS” and “professionalism,” posed a complex and novel challenge to neurosurgical educators, because of the perceived abstract nature of these competencies and the lack of previous formal training in these competencies by neurosurgical educators themselves. Surgeons, including neurosurgeons, are mostly conservative regarding educational hierarchies and pedagogy.8-11 Some surgical educators argued that behavioral competencies are adequately taught using the apprenticeship model. Nevertheless, in response to the new regulatory imperative, residency programs slowly developed and implemented specific curricula for professionalism and ICS education, likely improving patient care delivery, safety, and satisfaction.12,13 Within this context, neurosurgical educators developed the SNS boot camp courses.
SNS BOOT CAMP COURSE INITIATIVE The SNS is the oldest neurosurgical professional organization in the world and comprises program directors, department chairs, and other educational leaders within the specialty of neurological surgery in the United States.14 Following the introduction of ACGME core competencies in 2009, as described earlier, the ACGME incorporated the first year of postgraduate training (PGY-1 or “intern”) formally into all US neurosurgery programs, thus replacing the so-called preliminary year in general surgery.15 Concerns about the heterogeneity of incoming residents in medical knowledge, psychomotor skill, and proficiency in professionalism, communications, and leadership led the Committee on Resident Education of the SNS to establish foundational courses for entering and for junior neurosurgical residents to teach and assess this material. Subsequently, an SNS steering committee initially evaluated a small number of existing “pilot courses” that focused on psychomotor skills related to basic neurosurgical procedures, with some variable exposure to professionalism and ICS concepts. The aim was to expand these courses, develop a systematic curriculum vetted by program directors nationally (through the SNS), and replicate them at different regional locations throughout the country, to reach all trainees in US residencies. In 2010, the first national courses were held at 6 regional locations with overwhelmingly positive trainee and faculty reviews.16 These courses were held over 3 weekends (2 simultaneous courses per weekend). The course structure included 9 didactic lectures, 10 procedural skill stations, and 6 emergency craniotomy skill stations (Table 1). All course
Journal of Surgical Education Volume 71/Number 6 November/December 2014
TABLE 1. Didactic Content of the SNS Boot Camp for PGY-1 Residents Standardized Lectures Professionalism, supervision, and pearls* Neurological and neurotrauma assessment Emergency cranial radiology assessment Emergency spinal radiology assessment ICP management Unstable neurosurgical patient: case scenarios* Emergency evaluation and management of hydrocephalus patients Making the incision: scalp blood supply Airway management and intubation
Bedside Neurosurgical Procedures and Equipment Stations VP shunt tap and valve programming† Lumbar puncture and drain† ICP monitor† External ventricular drain† Cranial fixation and cervical traction† Central and arterial lines†
Craniotomy Skills Stations Drilling and bone dissection† Cranial flaps† Dural closure† Flap fixation† Cranioplasty† Skin closure
Clinical decision simulator†
ICP, intracranial pressure; VP, ventriculoperitoneal. *Behavioral competency component. † Stations where residents are assessed with checklists as described in the text.
material was approved by the steering committee and included considerable emphasis on all the ACGME competencies, including professionalism and ICS, in addition to psychomotor skills. All course material was made available online for preparatory learning and review at http://www. societyns.org/BootCamp/BootCampCourses.asp, and all residents completed precourse and postcourse evaluations including the assessment of representative didactic knowledge of taught and nontaught (control) material. From inception, each course has begun with a lecture on professionalism. All courses were presented free of charge to the residents, including transportation, housing, and meals.15,17 The faculty included residency directors and other senior educators, who were not paid or provided other honoraria. In 2010, 94% of incoming US residents attended 1 of 6 regional courses, with 100% attending since 2011. Postcourse evaluations of both faculty and residents demonstrated that 100% of attendees considered that didactic objectives were achieved and recommended continuing this program. Performance assessment has been an additional priority for the courses. Besides precourse and postcourse surveys that have demonstrated significant improvement in knowledge basis (p o 0.001), the residents are also graded on their technical skill on the procedural stations.15 Currently, procedural proficiency is assessed using a set of “checklists” including key points of each procedure that must be satisfied before the attendee can receive a Pass grade. The residents are offered the opportunity to repeat each failed technical station; repeated failures are communicated to their respective program directors so that they can be addressed before the resident progresses to perform these procedures in the clinical setting. Behavioral aspects during group exercises are currently assessed with a Pass/Fail system based on the examiner’s assessment but similar checklists are under development for these exercises. Knowledge retention is finally assessed 6 months after the course through a survey sent to residents. The response rates have averaged 88% of attendees and demonstrated satisfactory retention when compared with the
initial testing.18 The value of these courses has been recognized by the ACGME and its residency review committee for Neurological Surgery, which now requires participation in the SNS PGY-1 boot camp course or an equivalent structured program before indirect supervision for patient care and bedside procedures in the live clinical environment.19 Based on the very good results obtained with the PGY-1 boot camp courses, the SNS has expanded this program to include a second set of boot camp courses attended by upcoming PGY-2 junior residents. The first of these took place in May 2013. Three courses are now held annually, each over a 3-day period. At the PGY-2 level, additional emphasis is placed on professionalism, communications, and leadership. Of the 23 PGY-2 lectures and small-group activities, 13 directly involve the behavioral competencies (Table 2). The topics include “Disclosure of Medical Errors and Risk Management,” “Leadership and Professionalism,” and “Breaking bad news.” Scenarios with behavioral and decision-making components, such as “operating room crisis” and “intensive care unit crisis,” are also included. These lectures are also available online for preparatory learning and review.
MULTIMODAL AND CONTINUOUS EVALUATION—ACGME MILESTONES AND OTHER INSTRUMENTS Clear and objective evaluation of medical trainees is not easy to accomplish. Although difficult to implement, the use of effective evaluation systems can promote improved patient care and has been adopted in some form by virtually every surgical training program in the United States.6 There is also evidence that poor evaluations during training are an independent predictor of disciplinary action by state medical boards.20 In 2009, the ACGME started a multiyear process to restructure residency training to include reporting of educational outcomes in the 6 core
Journal of Surgical Education Volume 71/Number 6 November/December 2014
e85
TABLE 2. Didactic Content of the SNS Boot Camp for PGY-2 Residents
Standardized Lectures
Mentored Small-Group Exercises and Behavioral Scenario Simulations
Cerebral vascular anatomy
Informed consent*†
Disclosure of medical errors and risk management†
Breaking bad news: poor prognosis and complication/medical error*†
Handoffs† ICU Simulator Leadership and professionalism† OR crisis† Cranial fixation Postoperative and chronic pain management
Leadership discussion*† DVT/PE Surgical infections Handoffs† Pain management
Neuronavigation Quality improvement† Safety and surgical checklists† Surgical anatomy of the spine Informed consent and exercises† ICU crisis and scenarios† Breaking bad news† Ventricular anatomy
Model-Based Clinical Simulations (SimMan, Haptic Simulator, and Cadavers) ICU crisis: SAH, hydrocephalus, spine fracture, seizure, and traumatic brain injury* OR crisis: sinus bleeding, air embolism, loss of fixation, brain swelling, and hemorrhage* Tumor resection Endoscopy Endovascular Stereotactic frame placement Navigation registration Basic cranial approaches: suboccipital, pterional, and frontoorbital Basic spinal approaches: laminectomy and pedicle screw placement
DVT, deep venous thrombosis; ICU, intensive care unit; OR, operating room; PE, pulmonary embolism; SAH, subarachnoid hemorrhage. *Stations where residents are assessed with checklists as described in the text. † Behavioral competency taught and assessed.
competencies, the milestones. The milestones are intended to track the progress of individual residents toward documented levels of skill and knowledge for safe independent practice and also to evaluate the effectiveness of the educational environment in each residency program. The result of this effort is the Next Accreditation System (NAS), which was implemented in a pilot phase in July 2013.21 In other words, NAS has abandoned the “biopsy” model (in which program accreditation evaluations were performed every 5 years) and replaced it with continuous assessment using annual and biannual reporting of milestones tracking, resident and faculty survey data, and other educational outcomes. The ACGME milestones are subspecialty-specific tools for the tracking and reporting of resident developmental progression across the breadth of training in each of the 6 ACGME competencies. The Neurological Surgery milestones were developed by an ACGME-appointed working group, with oversight by an SNS Milestones Advisory Group.16 Each milestone tool uses developmental descriptors instead of a numerical ranking system to gauge resident performance in each of the 6 ACGME competencies and also to assess progress toward independent practice over time.21 Before milestone implementation, the ACGME working group and the SNS held 3 national 4-hour workshops attended by 280 program leaders (program directors and coordinators), complete with mock resident evaluations and faculty evaluation sessions. e86
Using the NAS, resident evaluations occur every 6 months and are finalized through consideration of all evaluation and resident portfolio data by a Clinical Competency Committee comprising program clinical faculty, often with a resident representative. Evaluation results and final milestone level assignments are privately disclosed to each resident by the program director or another faculty mentor, so each resident can track their individual progress and respond, with assistance where necessary, to deficiencies. Milestone reporting also offers the possibility of using systematic nationwide benchmarks, with validation of individual data by a Clinical Competency Committee to make promotion and separation decisions for residents failing to progress, where necessary. Neurological surgery, as one of the early adopters of this evaluation model, implemented the use of the milestones in July 2013, with the initial milestone reports provided to the ACGME in December 2013.16 The neurosurgical milestones comprise 24 “milestone sets,” including 2 each covering the 4 general competencies (systems-based practice, professionalism, ICS, and practicebased learning) and 16 covering medical knowledge and patient care for each of 8 subspecialty areas (Fig.).22 Proficiency in each milestone is measured with descriptors and graded from level 1 (expected performance of an incoming resident) through level 4 (performance targeted for residency completion). Level 5 milestones are suited to the performance of an advanced resident who might have
Journal of Surgical Education Volume 71/Number 6 November/December 2014
FIGURE. From 24 neurosurgical milestone sets, 2 examples are shown: one from a technical competency (patient care, A) and another from a general competency (professionalism, B). Progression is assessed from left to right, and completion of all items within a level is required for consideration of the next level. (Copyright by the Accreditation Council for Graduate Medical Education and the American Board of Neurological Surgery.)
subspecialty interest in a particular area or goals appropriate to fellowship training or the practice of a recent residency graduate. Although the “graduation target” is level 4 proficiency, such proficiency in all areas is not currently a formal “graduation requirement,” which remains the purview of the Residency Program Director. ACGME review committees,
including neurological surgery, have stated that analysis of milestone data to ensure quality and accuracy is necessary before using the milestones for high-stake decisions.23 The ACGME Neurological Surgery Milestones Group has also created new milestones-compatible evaluations, which use the specific developmental descriptors of the milestones
Journal of Surgical Education Volume 71/Number 6 November/December 2014
e87
themselves so that their results may directly determine and populate milestones reports, following review and consensus validation by the CCC. The “3601” evaluations, which incorporate assessments by raters from a variety of professional roles (self, peers, nurses, and faculty) are particularly useful in this context.24 Although isolated self-assessment by physicians has limited predictive value against externally validated measure of performance, “3601” evaluations are generally more accurate, especially for residents in the lowest-performing ranges, who tend to overestimate their performance, particularly for the general competencies, including professionalism and communication skills.25 Conversely, high-performing residents tend to underestimate their performance across general and technical competencies.24 Other types of resident evaluation tools may be used to compile data relevant to individual milestone level determination and reporting, including objective structured assessments of technical skills and varied forms of self-assessment evaluations already in place in many institutions (Selden NR, personal communication, 2014). The objectivity of milestone developmental level reporting ultimately depends on compiling assessments from multiple clinical observers and validating the overall interpretation of these results via consensus building in the CCC. The extent of data collected and analyzed in this process poses a new and unfunded regulatory mandate on residency training programs. To reduce the associated administrative burden, the SNS, working with US national neurosurgical membership societies, plans to automate the collection and initial analysis of milestones-compatible evaluation form data. Finally, it is preferable that these and other educational innovations in neurosurgical training and assessment are carefully coordinated. For example, the milestone developmental reporting tools require mastery of a series of technical skills and professional characteristics by “level 1” residents that have therefore been included in the PGY-1 introductory boot camp course. Similarly, the SNS continues to design specific curricular elements necessary to achieve higher-level milestones in other focused areas that are more efficiently taught using regional or national teaching, simulation, or assessment methodologies and equipment.
REFERENCES
CONCLUSION
10. Idema S, Buis DR, Idema AJS, Vandertop WP.
Surgical residency training has changed significantly over the last decade. Innovations such as a structured program for professionalism and ICS teaching and multimodal evaluations have been instituted at virtually every neurological surgery program in the Unites States. Although there is no definitive method for teaching and assessing professionalism, specialty-specific goals require direct specialty involvement and development of relevant curricula. It is thus preferable that surgeons lead, rather than be led, in this process.
e88
1. Merriam-Webster.
Professionalism. Available at: 〈http://www.merriam-webster.com/dictionary/profes sionalism〉; 2014 Accessed 30.03.14.
2. Apuzzo MLJ. Facets of professionalism. World Neurosurg.
2013;79(1):1. 2012.11.050.
doi:http://dx.doi.org/10.1016/j.wneu.
3. Stern DT, Papadakis M. The developing physician—
becoming a professional. N Engl J Med. 2006;355 (17):1794-1799. doi:http://dx.doi.org/10.1056/NEJ Mra054783. 4. Stern DT, ed. Measuring Medical Professionalism.
New York: Oxford University Press; 2006. 5. Bosk CL. Forgive and Remember: Managing Medical
Failure. 2nd ed. Chicago: University of Chicago Press; 2003. 6. Hochberg MS, Berman RS, Kalet AL, et al. The
professionalism curriculum as a cultural change agent in surgical residency education. Am J Surg. 2012;203 (1):14-20. doi:http://dx.doi.org/10.1016/j.amjsurg. 2011.05.007. 7. Imber G. Genius on the Edge: The Bizarre Double
Life of Dr. William Stewart Halsted. New York, NY: Kaplan Pub.; 2010. 8. Cohen-Gadol AA, Piepgras DG, Krishnamurthy S,
Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery. 2005;56(2):398-403; [discussion 398-403]. 9. Hoh BL, Neal DW, Kleinhenz DT, Hoh DJ, Mocco J,
Barker FG 2nd. Higher complications and no improvement in mortality in the ACGME resident duty-hour restriction era: an analysis of more than 107,000 neurosurgical trauma patients in the Nationwide Inpatient Sample database. Neurosurgery. 2012; 70(6):1369-1381 [discussion 1381-1382]. http://dx. doi.org/10.1227/NEU.0b013e3182486a75. Higher complications and no improvement in mortality in the ACGME resident duty-hour restriction era: an analysis of more than 107000 neurosurgical trauma patients in the nationwide inpatient sample database. Neurosurgery. 2013;72(1):E142-E143. doi:http://dx. doi.org/10.1227/NEU.0b013e318276064f. 11. Schaller K. Neurosurgical training under European
law. Acta Neurochir (Wien). 2013;155(3):547. doi: http://dx.doi.org/10.1007/s00701-012-1579-7.
Journal of Surgical Education Volume 71/Number 6 November/December 2014
12. Hochberg MS, Kalet A, Zabar S, Kachur E, Gillespie C,
Berman RS. Can professionalism be taught? Encouraging evidence Am J Surg. 2010;199(1):86-93. doi:http: //dx.doi.org/10.1016/j.amjsurg.2009.10.002. 13. Hochberg MS, Seib CD, Berman RS, Kalet AL, Zabar
SR, Pachter HL. Perspective: malpractice in an academic medical center: a frequently overlooked aspect of professionalism education. Acad Med J Assoc Am Med Coll. 2011;86(3):365-368. 10.1097/ACM. 0b013e3182086d72. 14. The Society of Neurological Surgeons—About the
SNS. Available at: 〈http://www.societyns.org/about_ sns.html〉; 2010 Accessed 30.03.14.
15. Selden NR, Origitano TC, Burchiel KJ, et al. A
national fundamentals curriculum for neurosurgery PGY1 residents: the 2010 Society of Neurological Surgeons Boot Camp Courses. Neurosurgery. 2012;70 (4):971-981. doi:http://dx.doi.org/10.1227/NEU.0b013 e31823d7a45. 16. Swing SR, Beeson MS, Carraccio C, et al. Educational
milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ. 2013; 5(1):98-106. doi:http://dx.doi.org/10.4300/JGME05-01-33.
of postgraduate year 1 training. J Neurosurg. 2013;119 (3):796-802. doi:http://dx.doi.org/10.3171/2013.3. JNS122114. 19. Review Committee for Neurological Surgery ACGME.
Frequently asked questions: Neurological Surgery. Available at: 〈http://www.acgme.org/acgmeweb/Por tals/0/PDFs/FAQ/160_neurological_surgery_ FAQs_07012013.pdf〉; 2013 Accessed 30.03.14. 20. Papadakis MA, Teherani A, Banach MA, et al. Disciplinary
action by medical boards and prior behavior in medical school. N Engl J Med. 2005;353(25):2673-2682. doi: http://dx.doi.org/10.1056/NEJMsa052596. 21. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next
GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051-1056. doi:http: //dx.doi.org/10.1056/NEJMsr1200117. 22. Selden NR, Abosch A, Byrne RW, et al. Neurological
surgery milestones. J Grad Med Educ. 2013;5(1 suppl 1): 24-35. doi:http://dx.doi.org/10.4300/JGME-05-01s1-04. 23. The
Neurological Surgery Milestone Project. Available at: 〈http://acgme.org/acgmeweb/Portals/0/ PDFs/Milestones/NeurologicalSurgeryMilestones.pdf〉; 2013 Accessed 31.03.14.
17. Selden NR, Barbaro N, Origitano TC, Burchiel KJ.
24. Lipsett PA, Harris I, Downing S. Resident self-other
Fundamental skills for entering neurosurgery residents: report of a Pacific region “boot camp” pilot course, 2009. Neurosurgery. 2011;68(3):759-764. [discussion 764]. http://dx.doi.org/10.1227/NEU.0b013e318207 7969.
assessor agreement: influence of assessor, competency, and performance level. Arch Surg. 2011;146(8):901-906. doi:http://dx.doi.org/10.1001/archsurg.2011.172.
18. Selden NR, Anderson VC, McCartney S, Origitano
TC, Burchiel KJ, Barbaro NM. Society of Neurological Surgeons boot camp courses: knowledge retention and relevance of hands-on learning after 6 months
25. Davis DA, Mazmanian PE, Fordis M, Van Harrison
R, Thorpe KE, Perrier L. Accuracy of physician selfassessment compared with observed measures of competence: a systematic review. J Am Med Assoc. 2006; 296(9):1094-1102, doi:http://dx.doi.org/10.1001/ jama.296.9.1094.
Journal of Surgical Education Volume 71/Number 6 November/December 2014
e89
