ORIGINAL REPORTS
S/He Who Can, Does and Teaches. S/He Who Cannot, Doesn’t John L. Falcone, MD, Peter F. Ferson, MD, and Giselle G. Hamad, MD Department of Surgery, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania BACKGROUND: The saying, “[h]e who can, does. He who cannot, teaches.” suggests that those who have the skills to perform do so, whereas those who do not have those skills become teachers. We hypothesize that this saying as it relates to general surgery residents is not true.
CONCLUSIONS: There is some evidence that superior
METHODS: This was a retrospective study of general
KEY WORDS: Fundamentals of Laparoscopic Surgery,
surgery chief residents from 2009 to 2013. Technical ability was assessed with the Fundamentals of Laparoscopic Surgery examination performance. Teaching ability was assessed with medical student evaluations on a 9-point Likert scale as well as with receipt of teaching awards: The Arnold P. Gold Teaching Award, the surgical teaching award given by each graduating class of the medical school, and resident induction into Alpha Omega Alpha. MannWhitney U tests were performed between resident groups based on teaching award status and Fundamentals of Laparoscopic Surgery examination outcomes, using an α ¼ 0.05.
general surgery, medical students, residents, teaching, technical skills
RESULTS: For 32 chief residents (7 female), the median score on the Manual Skills Section was 531 (interquartile range [IQR] [478-563]). On the Cognitive Section, the cohort of residents who won each award outperformed the residents who did not win that specific award (all p o 0.05). On the Manual Skills Section, residents who received any teaching award/the Arnold P. Gold Teaching Award (n ¼ 7) performed similar to residents who did not receive this award (n ¼ 25) (584 [IQR {491-620}] vs. 527 [IQR {482-537}]) (p ¼ 0.09). Residents who received the surgical teaching award from the medical school (n ¼ 3) performed similar to residents who did not receive this award (n ¼ 29) (608 [IQR {474-637}] vs. 527 [IQR {482-555}]) (p ¼ 0.23). Eligible residents who were inducted into Alpha Omega Alpha (n ¼ 4) outperformed residents who were not inducted (n ¼ 18) (596 [IQR {564-637}] vs. 527 [IQR {446-551}]) (p ¼ 0.01).
Correspondence: Inquiries to John L. Falcone, MD, MS, University of Pittsburgh Medical Center, Presbyterian University Hospital, F-675, 200 Lothrop Street, Pittsburgh, PA 5213; fax: þ412 647 4889; e-mail: [email protected]
96
resident teachers have greater content knowledge and a higher C 2014 degree of laparoscopic skills. ( J Surg 71:96-101. J Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
COMPETENCIES: Medical Knowledge, Interpersonal and
Communication Skills, Practice-Based Improvement, Professionalism
Learning
and
INTRODUCTION There is a well-known saying written by Bernard Shaw, “[h]e who can, does. He who cannot, teaches.”1 This saying suggests the pejorative notion that those who have the skills to perform do so, whereas those who do not have those skills become teachers.2 It is clear that technical ability is an important part of overall surgical competence.3,4 Given the emphasis placed on laparoscopic surgical skill proficiency by the Society of American Gastrointestinal and Endoscopic Surgeons and the American Board of Surgery, the Fundamentals of Laparoscopic Surgery (FLS) curriculum has been a standardized method to test cognitive and psychomotor competence.5 Since 2009, surgical residents have been required to pass the FLS examination to be eligible to take the American Board of Surgery Qualifying Examination.6,7 There is construct validity and discriminant validity to the FLS curriculum, as experienced individuals outperform intermediate individuals; individuals with intermediate experience, in turn, outperform novices.8 Importantly, FLS skills are clinically transferrable to the operating room setting.9 It is also clear that surgical residents play an important role in medical student education.10-14 Importantly, resident teaching contributes significantly toward a medical student’s acquisition of knowledge.11 As a corollary, student satisfaction with the surgical clerkship is closely related to resident
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2013.06.003
METHODS This was a retrospective study from 2009 to 2013 of the relationship of technical ability and teaching ability for chief general surgery residents. The study was approved by the Institutional Review Board. General surgery resident technical ability was assessed with the proctored FLS examination results. Raw chief resident performance on the Cognitive Section and the proctored Manual Skills Section of the FLS examination were obtained. A Pearson correlation coefficient was found between these 2 variables. General surgery resident teaching ability was assessed in 2 ways. First, archived electronic medical student assessments of residents on a 9-point Likert scale were obtained and described. This scale was a single global assessment of the resident using the following characterization scheme: Unsatisfactory (1, 2, and 3), Satisfactory (4, 5, and 6), and Superior (7, 8, and 9). Second, residents were described by teaching awards and honors received during residency training. The Arnold P. Gold “Little Apple” Teaching Award, the surgical teaching award given by each graduating class of the medical school (Charles C. Moore, M.D. Teaching Award), and induction into Alpha Omega Alpha (AOA) during residency training were used. Resident trainees can be inducted into AOA by the local chapter.16 Study subjects who were inducted into AOA before residency training were excluded regarding this honor for comparison purposes, as they were not eligible to be inducted a second time. The relationship between content mastery, technical ability, and teaching ability was evaluated in 2 ways. First, simple linear regression was performed using the FLS scores as the independent variable and medical student evaluation as the dependent variable. Second, Mann-Whitney U tests were
performed between resident groups based on teaching award status and FLS examination outcomes. To compare teaching outcomes, a Mann-Whitney U test was also performed between resident groups based on teaching award status and medical student Likert scale–based assessments. Subgroup analyses by gender were also performed. Nonparametric statistical tests were performed with Stata 11.1 statistical software (StataCorp, College Station, TX), using an α ¼ 0.05.
RESULTS From 2009 to 2013, there were 32 chief graduating residents at our institution. Data for FLS examination outcomes and teaching awards were available for all 32/32 (100%) chief residents. There were 23/32 (71.9%) chief residents with electronic medical student evaluations. There were 7/32 (21.9%) females and 25/32 (78.1%) males (p o 0.001). The median residency graduating class was 6 residents (interquartile range [IQR] [6-7 residents]). With regard to FLS examination performance, the median score on the Cognitive Section was 580 (IQR [550-620]). The median score on the Manual Skills Section was 531 (IQR [478-563]). A scatter plot of resident performance is shown in Figure 1. The Pearson correlation coefficient was 0.25. The first-attempt pass rate on the FLS examination was 96.9% (31/32). There were no differences in Cognitive Section performance, Manual Skills Section performance, or first-attempt outcomes on the FLS examination by gender (all p 4 0.05). The median number of medical student evaluations per resident was 23 (IQR [16-31]). The median Likert scale– based assessment per resident was 7.96 (IQR [7.47-8.41]). There were 7/32 (21.9%) chief residents who received the Arnold P. Gold “Little Apple” Teaching Award. There were 3/32 (9.4%) residents who received the Charles C. Moore, M.D. Teaching Award. There were 10/32 (31.3%) residents inducted into AOA before residency training. There 800 y = 0.3403x + 321.71 R² = 0.0617
FLS Manual Skills Score
teaching behaviors.12 The importance to foster and train residents as teachers has been recognized in the literature.12,13 Teaching by surgical residents benefits both medical students and surgical residents alike.14 A phenomenon associated with excellence in teaching is that surgical residents who have won teaching awards are more likely to pass both the American Board of Surgery Qualifying Examination and Certifying Examination on the first attempt compared with residents who have not won teaching awards.15 In procedural specialties, it has been suggested that pioneers are both technically excellent and the most able teachers.2 However, this idea has not been studied. The primary aim of this study is to evaluate the relationship between technical skill and teaching ability. Our hypothesis is that surgical residents who are more technically proficient are also more capable medical student teachers. We want to show that he who can, does and teaches, whereas he who cannot, does not do either in direct contrast to the quote penned by Bernard Shaw.
700 600 500 400 300 200 200
300
400 500 600 FLS Cognitive Score
700
800
FLS: Fundamentals of Laparoscopic Surgery
FIGURE 1. Performance on the Cognitive and Manual Skills Sections of the FLS examination for 32 general surgery chief residents.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
97
were 4/22 (18.2%) eligible residents inducted into AOA. There were 7/32 (21.9%) chief residents who received any teaching awards during residency training. There were 2 residents who received all 3 honors. There were no differences in medical student evaluations or teaching award recognition by gender (all p 4 0.05). The simple linear regression analyses are shown in Figure 2. The Pearson correlation coefficient between FLS Cognitive Section score and medical student evaluation was -0.05. Simple linear regression with FLS Cognitive Section score as the independent variable and medical student evaluation as the dependent variable showed that the slope of the least-squares regression line was not different than zero (p ¼ 0.81). The Pearson correlation coefficient between FLS Manual Skills Section score and medical student evaluation was 0.14. Simple linear regression with FLS Manual Skills Section score as the independent variable and medical student evaluation as the dependent variable showed that the slope of the least-squares regression line was greater than zero, but not significantly different than zero (0.0011) (95% confidence interval [−0.0025 to 0.0047]) (p ¼ 0.53). On the FLS Cognitive Section, a Mann-Whitney U test showed that chief residents who received any teaching award/the Arnold P. Gold “Little Apple” Teaching Award (n ¼ 7) outperformed chief residents who did not receive this award (n ¼ 25) (620 [IQR {580-650}] vs. 560 [IQR {540-600}]) (p ¼ 0.04). Chief residents who received the Charles C. Moore, M.D. Teaching Award (n ¼ 3) outperformed chief residents who did not receive this award (n ¼ 29) (680 [IQR {650-680}] vs. 560 [IQR {540-600}]) (p ¼ 0.01). Chief residents who were inducted into AOA (n ¼ 4) outperformed the eligible chief residents who did not receive this award (n ¼ 18) (650 [IQR {610-680}] vs. 570 [IQR {540-600}]) (p ¼ 0.03).
On the FLS Manual Skills Section, a Mann-Whitney U test showed that chief residents who received any teaching award/the Arnold P. Gold “Little Apple” Teaching Award (n ¼ 7) performed the same as chief residents who did not receive this award (n ¼ 25) (584 [IQR {491-620}] vs. 527 [IQR {482-537}]) (p ¼ 0.09). Chief residents who received the Charles C. Moore, M.D. Teaching Award (n ¼ 3) performed the same as chief residents who did not receive this award (n ¼ 29) (608 [IQR {474-637}] vs. 527 [IQR {482-555}]) (p ¼ 0.23). The eligible chief residents who were inducted into AOA (n ¼ 4) outperformed the eligible chief residents who did not receive this award (n ¼ 18) (596 [IQR {564-637}] vs. 527 [IQR {446-551}]) (p ¼ 0.01). Comparing Likert scale–based medical student evaluations and teaching awards, a Mann-Whitney U test showed that chief residents who received any teaching award/the Arnold P. Gold “Little Apple” Teaching Award (n ¼ 5) had similar evaluations than chief residents who did not receive this award (n ¼ 18) (8.04 [IQR {7.57-8.57}] vs. 7.81 [IQR {7.27-8.17}]) (p ¼ 0.20). Chief residents who received the Charles C. Moore, M.D. Teaching Award (n ¼ 2) had similar evaluations than chief residents who did not receive this award (n ¼ 21) (8.31 [IQR {8.17-8.44}] vs. 7.78 [IQR {7.39-8.23}]) (p ¼ 0.17). Chief residents inducted into AOA (n ¼ 3) had similar evaluations than chief residents who did not receive this award (n ¼ 13) (8.28 [IQR {8.048.57}] vs. 7.78 [IQR {7.38-8.32}]) (p ¼ 0.25).
DISCUSSION In this study, we evaluated the relationship between content knowledge, technical skill, and teaching ability, hypothesizing that surgical residents who are more technically proficient are also more capable medical student teachers. First, we found
9 Medical Student Evaluation Score
Medical Student Evaluation Score
9 8 7 6 5
y = -0.0006x + 8.2464 R² = 0.0027
4 3 2 1 200
500 300 400 600 700 FLS Cognitive Section Score
800
8 7 y = 0.0011x + 7.2695 R² = 0.0195
6 5 4 3 2 1 200
300 400 500 600 700 800 FLS Manual Skills Section Score
FLS: Fundamentals of Laparoscopic Surgery
FIGURE 2. Simple linear regression of FLS Cognitive Section Score (A) and Manual Skills Section Score (B) as related to medical student evaluation of residents on a 9-point Likert scale for 32 general surgery chief residents. 98
Journal of Surgical Education Volume 71/Number 1 January/February 2014
that residents who have received any of the teaching awards have a higher degree of content knowledge as evidenced by higher FLS Cognitive Section scores that reached a level of statistical significance. This is consistent with the wellestablished notion that effective teaching is associated with higher degrees of content knowledge.17,18 We have also found that eligible resident inductees into AOA also had significantly higher FLS Manual Skills scores. This finding was consistent with the study hypothesis. Although not all statistically significant, there were clear trends that chief residents who have won any teaching award performed better on the FLS Manual Skills portion of the examination. Further supporting our study hypothesis is that the only chief resident who failed the Manual Skills portion of the FLS examination did not receive any teaching awards. Additionally, the chief resident with the highest overall Manual Skills Section score on the FLS examination received all of the possible teaching awards. All of these findings are consistent with our study hypothesis, and contradictory to the essence of the saying that “[h]e who can, does. He who cannot, teaches.” The findings of the simple linear regression analyses, however, do not support the study hypothesis. Overall, FLS Cognitive scores and FLS Manual Skills scores as a predictor of Likert scale–based medical student evaluation are poor. The Pearson correlation coefficients are weak. Interestingly, the slope of the linear regression line using Cognitive scores as the independent variable is less than zero; in the setting of very poor correlation coefficient with no significant slope, there is essentially no meaningful correlation between these 2 variables. The slope of the linear regression line using Manual Skills scores is greater than zero. This is consistent with the study hypothesis; however, it is not statistically different than zero. The findings in the linear regression analyses would suggest that the medical student Likert scale–based evaluations do not accurately measure teaching ability. These evaluations are global evaluations and not specifically focused on teaching ability. On a 9-point scale, the vast majority of the evaluations were in the highest quartile of the measurement tool range, limiting the variability in these evaluations. In addition, there were no statistically significant higher evaluations of chief residents who received teaching awards and the chief residents who did not receive teaching awards. There does not seem to be discriminant validity of a global resident assessment. We conclude from the linear regression analyses that the global assessment of residents is not influenced by the nature of resident content knowledge or technical ability. The medical students are likely evaluating other variables such as personality or the nature of feedback received. Likert scale– based evaluation of perceived teaching depends on feedback. In a study of knot tying by medical student, student performance was directly related to feedback. Students who received technical feedback perform better than students who received general compliments. However, the
students in the feedback group did not rate the teachers as well as the students in the general compliments group.21 As a result of this study, we believe that the nature of medical student evaluations of residents (and faculty members) should be critically evaluated and a reliable and valid instrument should be instituted. We believe that resident teaching skills should be fostered, in agreement with prior studies.13,14 We also found that there was a small correlation coefficient between the knowledge and psychomotor domains of the FLS examination. This is the first known description of correlation of the Cognitive Section and the Manual Skills Section of the FLS examination. It has been suggested that a high degree of correlation is not necessarily optimal; a very high correlation coefficient between examination modalities may represent redundancy in assessment, whereas a more limited correlation supports the use of both modalities as part of an integrated assessment strategy.19,20 This finding supports the validity of the elements that comprise the FLS examination. There are a few major limitations to this study. The most important was the lack of validation in any of the teaching evaluation tools used in the study. First, the use of our institutional 9-point Likert scale has not been validated and is entirely subjective in nature. We are unsure of what domains are measured, by this Likert scale. To compliment this subjective evaluation, the more objective teaching award status was used. However, although winning a teaching award is binomial as an objective variable, the criteria and rating in the process of receiving these awards are also entirely subjective. Although the teaching awards used in this study have face validity, the content validity and all other validation measures remain unknown. Further study in this domain of surgical education is necessary and is important in the domain for recognition of superior teaching ability. The second major limitation involves the lower study power in this smaller study. Although the entire cohort of chief general surgery resident outcomes on the FLS examination was evaluated from our institution, the low power greatly increases the chance that a type II error occurred. With larger numbers, some of the near-significant trends may have become statistically significant. We plan to continue to follow FLS examination performance and the evaluations of residents by medical students. However, even with the low study power, there were clear trends and differences found. The Manual Skills Section of the FLS examination may not be the optimal way of evaluating technical ability, as the interest and anticipated laparoscopic surgery practice is varied amongst chief general surgery residents. This study is also a single-institution experience, and therefore not generalizable. Further research in this domain on the national level could help validate some of the study findings and conclusions. Finally, there were 9 chief residents who did not have any archived electronic medical
Journal of Surgical Education Volume 71/Number 1 January/February 2014
99
student evaluations. These 9 chief residents graduated in 2009 and 2010, during the relative infancy of electronic documentation of medical student assessments. The omission of these data could bias the results of the study. Conversely, there are a few strengths to this study. The use of scoring on the proctored and standardized FLS examination gives face and content validity to the technical ability being measured. The FLS examination has been rigorously studied, and it has discriminant validity such that the 5 tasks are a valid measure of general surgery technical skills that are transferrable to the operation room.8,9 The standardized outcomes are determined by trained individuals. In addition, multiple measures of teaching ability are stronger than using either outcome individually. Moreover, this study suggests that there are no gender differences in the cognitive, psychomotor, or teaching domains. The study design is straightforward and attempts to answer a very unique and important question in surgery. This study is the first known study that compares technical and teaching skills to test the saying that “[h]e who can, does. He who cannot, teaches.” Overall, we found evidence that the chief general surgery residents who have won teaching awards have greater content knowledge than chief residents who have not won teaching awards. Residents who have been inducted into AOA have superior technical ability than residents not inducted into AOA. There are statistical trends that residents who have won any teaching awards have a higher degree of laparoscopic skills as shown by performance on the FLS examination. We also conclude that a global evaluation of resident performance may not be useful, and there are significant questions surrounding the validity of all teaching awards. Ultimately, we feel that there is evidence that the saying from Bernard Shaw should be changed to “s/ he who can, does and teaches. S/he who cannot, doesn’t.” However, the original quotation cannot totally be disproved in the setting of nonvalidated measures and the higher potential for type II error
3. Baldwin PJ, Paisley AM, Brown SP. Consultant
surgeons’ opinion of the skills required of basic surgical trainees. Br J Surg. 1999;86(8):1078-1082. 4. Datta V, Chang A, Mackay S, Darzi A. The relation-
ship between motion analysis and surgical technical assessments. Am J Surg. 2002;184(1):70-73. 5. Fundamentals of Laparoscopic Surgery. Society of
American Gastrointestinal and Endoscopic Surgeons. Available at: 〈http://www.flsprogram.org/〉; 2013 Accessed 14.02.13. 6. Guidelines for Institutions Granting Privileges Utiliz-
ing Laparoscopic and/or Thoracoscopic Techniques. Society of American Gastrointestinal and Endoscopic Surgeons (SAGES). Available at: 〈http://www.sages. org/publication/id/14/〉; 2013 Accessed 14.02.13. 7. ABS to Require ACLS, ATLS and FLS for General
Surgery Certification, American Board of Surgery. Available at: 〈http://www.absurgery.org/default.jsp? news_newreqs〉; 2013 Accessed 14.02.13. 8. Ritter EM, Kindelan TW, Michael C, Pimentel EA,
Bowyer MW. Concurrent validity of augmented reality metrics applied to the fundamentals of laparoscopic surgery (FLS). Surg Endosc. 2007;21(8):1441-1445. 9. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez
R, Fried GM. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room—A randomized controlled trial. Am J Surg. 2010;199(1):115-120. 10. Morris M, Caskey R, Mitchell M, Sawaya D. Surgical
skills training restructured for the 21st century. J Surg Res. 2012;177(1):33-36. 11. Pelletier M, Belliveau P. Role of surgical residents in
undergraduate surgical education. Can J Surg. 1999;42 (6):451-456. 12. Xu G, Wolfson P, Robeson M, Rodgers JF, Veloski JJ,
ACKNOWLEDGMENT The authors would like to sincerely thank Kathy Haupt and Maggie Mrozinski for help with data acquisition for this study.
Brigham TP. Students' satisfaction and perceptions of attending physicians' and residents' teaching role. Am J Surg. 1998;176(1):46-48. 13. Sheets KJ, Hankin FM, Schwenk TL. Preparing
surgery house officers for their teaching role. Am J Surg. 1991;161(4):443-449. 14. Jamshidi R. Formalizing teaching responsibilities for
junior surgical housestaff encourages educator development. J Surg Educ. 2008;65(6):514-517.
REFERENCES 1. Shaw Bernard. Man and Superman. The Revolution-
ist’s Handbook and Pocket Companion. Cambridge, MA: The University Press; 1903. 2. Safian RD. He who can, does; he who does, teaches.
Catheter Cardiovasc Interv. 2011;78(2):187-188. 100
15. Shellito JL, Osland JS, Helmer SD, Chang FC.
American Board of Surgery examinations: can we identify surgery residency applicants and residents who will pass the examinations on the first attempt? Am J Surg. 2010;199(2):216-222.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
16. How Members are Chosen. Alpha Omega Alpha
Honor Medical Society. Available at: 〈http://www. alphaomegaalpha.org/how.html〉; 2013 Accessed 20.04.13.
17. Sutkin G, Wagner E, Harris I, Schiffer R. What makes
a good clinical teacher in medicine? A review of the literature Acad Med. 2008;83(5):452-466. 18. Srinivasan M, Li ST, Meyers FJ, et al. “Teaching as a
Competency”: competencies for medical educators. Acad Med. 2011;86(10):1211-1220. 19. Nunnink L, Venkatesh B, Krishnan A, Vidhani K,
Udy A. A prospective comparison between written
examination and either simulation-based or oral viva examination of intensive care trainees' procedural skills. Anaesth Intensive Care. 2010;38(5):876-882. 20. Falcone JL, Hamad GG. The state of performance on
the American Board of Surgery Qualifying Examination and Certifying Examination and the effect of residency program size on program pass rates. Surgery. 2012;151(5):639-642. 21. Boehler ML, Rogers DA, Schwind CJ, et al. An
investigation of medical student reactions to feedback: a randomised controlled trial. Med Educ. 2006;40 (8):746-749.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
101
S/He Who Can, Does and Teaches. S/He Who Cannot, Doesn’t John L. Falcone, MD, Peter F. Ferson, MD, and Giselle G. Hamad, MD Department of Surgery, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania BACKGROUND: The saying, “[h]e who can, does. He who cannot, teaches.” suggests that those who have the skills to perform do so, whereas those who do not have those skills become teachers. We hypothesize that this saying as it relates to general surgery residents is not true.
CONCLUSIONS: There is some evidence that superior
METHODS: This was a retrospective study of general
KEY WORDS: Fundamentals of Laparoscopic Surgery,
surgery chief residents from 2009 to 2013. Technical ability was assessed with the Fundamentals of Laparoscopic Surgery examination performance. Teaching ability was assessed with medical student evaluations on a 9-point Likert scale as well as with receipt of teaching awards: The Arnold P. Gold Teaching Award, the surgical teaching award given by each graduating class of the medical school, and resident induction into Alpha Omega Alpha. MannWhitney U tests were performed between resident groups based on teaching award status and Fundamentals of Laparoscopic Surgery examination outcomes, using an α ¼ 0.05.
general surgery, medical students, residents, teaching, technical skills
RESULTS: For 32 chief residents (7 female), the median score on the Manual Skills Section was 531 (interquartile range [IQR] [478-563]). On the Cognitive Section, the cohort of residents who won each award outperformed the residents who did not win that specific award (all p o 0.05). On the Manual Skills Section, residents who received any teaching award/the Arnold P. Gold Teaching Award (n ¼ 7) performed similar to residents who did not receive this award (n ¼ 25) (584 [IQR {491-620}] vs. 527 [IQR {482-537}]) (p ¼ 0.09). Residents who received the surgical teaching award from the medical school (n ¼ 3) performed similar to residents who did not receive this award (n ¼ 29) (608 [IQR {474-637}] vs. 527 [IQR {482-555}]) (p ¼ 0.23). Eligible residents who were inducted into Alpha Omega Alpha (n ¼ 4) outperformed residents who were not inducted (n ¼ 18) (596 [IQR {564-637}] vs. 527 [IQR {446-551}]) (p ¼ 0.01).
Correspondence: Inquiries to John L. Falcone, MD, MS, University of Pittsburgh Medical Center, Presbyterian University Hospital, F-675, 200 Lothrop Street, Pittsburgh, PA 5213; fax: þ412 647 4889; e-mail: [email protected]
96
resident teachers have greater content knowledge and a higher C 2014 degree of laparoscopic skills. ( J Surg 71:96-101. J Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
COMPETENCIES: Medical Knowledge, Interpersonal and
Communication Skills, Practice-Based Improvement, Professionalism
Learning
and
INTRODUCTION There is a well-known saying written by Bernard Shaw, “[h]e who can, does. He who cannot, teaches.”1 This saying suggests the pejorative notion that those who have the skills to perform do so, whereas those who do not have those skills become teachers.2 It is clear that technical ability is an important part of overall surgical competence.3,4 Given the emphasis placed on laparoscopic surgical skill proficiency by the Society of American Gastrointestinal and Endoscopic Surgeons and the American Board of Surgery, the Fundamentals of Laparoscopic Surgery (FLS) curriculum has been a standardized method to test cognitive and psychomotor competence.5 Since 2009, surgical residents have been required to pass the FLS examination to be eligible to take the American Board of Surgery Qualifying Examination.6,7 There is construct validity and discriminant validity to the FLS curriculum, as experienced individuals outperform intermediate individuals; individuals with intermediate experience, in turn, outperform novices.8 Importantly, FLS skills are clinically transferrable to the operating room setting.9 It is also clear that surgical residents play an important role in medical student education.10-14 Importantly, resident teaching contributes significantly toward a medical student’s acquisition of knowledge.11 As a corollary, student satisfaction with the surgical clerkship is closely related to resident
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2013.06.003
METHODS This was a retrospective study from 2009 to 2013 of the relationship of technical ability and teaching ability for chief general surgery residents. The study was approved by the Institutional Review Board. General surgery resident technical ability was assessed with the proctored FLS examination results. Raw chief resident performance on the Cognitive Section and the proctored Manual Skills Section of the FLS examination were obtained. A Pearson correlation coefficient was found between these 2 variables. General surgery resident teaching ability was assessed in 2 ways. First, archived electronic medical student assessments of residents on a 9-point Likert scale were obtained and described. This scale was a single global assessment of the resident using the following characterization scheme: Unsatisfactory (1, 2, and 3), Satisfactory (4, 5, and 6), and Superior (7, 8, and 9). Second, residents were described by teaching awards and honors received during residency training. The Arnold P. Gold “Little Apple” Teaching Award, the surgical teaching award given by each graduating class of the medical school (Charles C. Moore, M.D. Teaching Award), and induction into Alpha Omega Alpha (AOA) during residency training were used. Resident trainees can be inducted into AOA by the local chapter.16 Study subjects who were inducted into AOA before residency training were excluded regarding this honor for comparison purposes, as they were not eligible to be inducted a second time. The relationship between content mastery, technical ability, and teaching ability was evaluated in 2 ways. First, simple linear regression was performed using the FLS scores as the independent variable and medical student evaluation as the dependent variable. Second, Mann-Whitney U tests were
performed between resident groups based on teaching award status and FLS examination outcomes. To compare teaching outcomes, a Mann-Whitney U test was also performed between resident groups based on teaching award status and medical student Likert scale–based assessments. Subgroup analyses by gender were also performed. Nonparametric statistical tests were performed with Stata 11.1 statistical software (StataCorp, College Station, TX), using an α ¼ 0.05.
RESULTS From 2009 to 2013, there were 32 chief graduating residents at our institution. Data for FLS examination outcomes and teaching awards were available for all 32/32 (100%) chief residents. There were 23/32 (71.9%) chief residents with electronic medical student evaluations. There were 7/32 (21.9%) females and 25/32 (78.1%) males (p o 0.001). The median residency graduating class was 6 residents (interquartile range [IQR] [6-7 residents]). With regard to FLS examination performance, the median score on the Cognitive Section was 580 (IQR [550-620]). The median score on the Manual Skills Section was 531 (IQR [478-563]). A scatter plot of resident performance is shown in Figure 1. The Pearson correlation coefficient was 0.25. The first-attempt pass rate on the FLS examination was 96.9% (31/32). There were no differences in Cognitive Section performance, Manual Skills Section performance, or first-attempt outcomes on the FLS examination by gender (all p 4 0.05). The median number of medical student evaluations per resident was 23 (IQR [16-31]). The median Likert scale– based assessment per resident was 7.96 (IQR [7.47-8.41]). There were 7/32 (21.9%) chief residents who received the Arnold P. Gold “Little Apple” Teaching Award. There were 3/32 (9.4%) residents who received the Charles C. Moore, M.D. Teaching Award. There were 10/32 (31.3%) residents inducted into AOA before residency training. There 800 y = 0.3403x + 321.71 R² = 0.0617
FLS Manual Skills Score
teaching behaviors.12 The importance to foster and train residents as teachers has been recognized in the literature.12,13 Teaching by surgical residents benefits both medical students and surgical residents alike.14 A phenomenon associated with excellence in teaching is that surgical residents who have won teaching awards are more likely to pass both the American Board of Surgery Qualifying Examination and Certifying Examination on the first attempt compared with residents who have not won teaching awards.15 In procedural specialties, it has been suggested that pioneers are both technically excellent and the most able teachers.2 However, this idea has not been studied. The primary aim of this study is to evaluate the relationship between technical skill and teaching ability. Our hypothesis is that surgical residents who are more technically proficient are also more capable medical student teachers. We want to show that he who can, does and teaches, whereas he who cannot, does not do either in direct contrast to the quote penned by Bernard Shaw.
700 600 500 400 300 200 200
300
400 500 600 FLS Cognitive Score
700
800
FLS: Fundamentals of Laparoscopic Surgery
FIGURE 1. Performance on the Cognitive and Manual Skills Sections of the FLS examination for 32 general surgery chief residents.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
97
were 4/22 (18.2%) eligible residents inducted into AOA. There were 7/32 (21.9%) chief residents who received any teaching awards during residency training. There were 2 residents who received all 3 honors. There were no differences in medical student evaluations or teaching award recognition by gender (all p 4 0.05). The simple linear regression analyses are shown in Figure 2. The Pearson correlation coefficient between FLS Cognitive Section score and medical student evaluation was -0.05. Simple linear regression with FLS Cognitive Section score as the independent variable and medical student evaluation as the dependent variable showed that the slope of the least-squares regression line was not different than zero (p ¼ 0.81). The Pearson correlation coefficient between FLS Manual Skills Section score and medical student evaluation was 0.14. Simple linear regression with FLS Manual Skills Section score as the independent variable and medical student evaluation as the dependent variable showed that the slope of the least-squares regression line was greater than zero, but not significantly different than zero (0.0011) (95% confidence interval [−0.0025 to 0.0047]) (p ¼ 0.53). On the FLS Cognitive Section, a Mann-Whitney U test showed that chief residents who received any teaching award/the Arnold P. Gold “Little Apple” Teaching Award (n ¼ 7) outperformed chief residents who did not receive this award (n ¼ 25) (620 [IQR {580-650}] vs. 560 [IQR {540-600}]) (p ¼ 0.04). Chief residents who received the Charles C. Moore, M.D. Teaching Award (n ¼ 3) outperformed chief residents who did not receive this award (n ¼ 29) (680 [IQR {650-680}] vs. 560 [IQR {540-600}]) (p ¼ 0.01). Chief residents who were inducted into AOA (n ¼ 4) outperformed the eligible chief residents who did not receive this award (n ¼ 18) (650 [IQR {610-680}] vs. 570 [IQR {540-600}]) (p ¼ 0.03).
On the FLS Manual Skills Section, a Mann-Whitney U test showed that chief residents who received any teaching award/the Arnold P. Gold “Little Apple” Teaching Award (n ¼ 7) performed the same as chief residents who did not receive this award (n ¼ 25) (584 [IQR {491-620}] vs. 527 [IQR {482-537}]) (p ¼ 0.09). Chief residents who received the Charles C. Moore, M.D. Teaching Award (n ¼ 3) performed the same as chief residents who did not receive this award (n ¼ 29) (608 [IQR {474-637}] vs. 527 [IQR {482-555}]) (p ¼ 0.23). The eligible chief residents who were inducted into AOA (n ¼ 4) outperformed the eligible chief residents who did not receive this award (n ¼ 18) (596 [IQR {564-637}] vs. 527 [IQR {446-551}]) (p ¼ 0.01). Comparing Likert scale–based medical student evaluations and teaching awards, a Mann-Whitney U test showed that chief residents who received any teaching award/the Arnold P. Gold “Little Apple” Teaching Award (n ¼ 5) had similar evaluations than chief residents who did not receive this award (n ¼ 18) (8.04 [IQR {7.57-8.57}] vs. 7.81 [IQR {7.27-8.17}]) (p ¼ 0.20). Chief residents who received the Charles C. Moore, M.D. Teaching Award (n ¼ 2) had similar evaluations than chief residents who did not receive this award (n ¼ 21) (8.31 [IQR {8.17-8.44}] vs. 7.78 [IQR {7.39-8.23}]) (p ¼ 0.17). Chief residents inducted into AOA (n ¼ 3) had similar evaluations than chief residents who did not receive this award (n ¼ 13) (8.28 [IQR {8.048.57}] vs. 7.78 [IQR {7.38-8.32}]) (p ¼ 0.25).
DISCUSSION In this study, we evaluated the relationship between content knowledge, technical skill, and teaching ability, hypothesizing that surgical residents who are more technically proficient are also more capable medical student teachers. First, we found
9 Medical Student Evaluation Score
Medical Student Evaluation Score
9 8 7 6 5
y = -0.0006x + 8.2464 R² = 0.0027
4 3 2 1 200
500 300 400 600 700 FLS Cognitive Section Score
800
8 7 y = 0.0011x + 7.2695 R² = 0.0195
6 5 4 3 2 1 200
300 400 500 600 700 800 FLS Manual Skills Section Score
FLS: Fundamentals of Laparoscopic Surgery
FIGURE 2. Simple linear regression of FLS Cognitive Section Score (A) and Manual Skills Section Score (B) as related to medical student evaluation of residents on a 9-point Likert scale for 32 general surgery chief residents. 98
Journal of Surgical Education Volume 71/Number 1 January/February 2014
that residents who have received any of the teaching awards have a higher degree of content knowledge as evidenced by higher FLS Cognitive Section scores that reached a level of statistical significance. This is consistent with the wellestablished notion that effective teaching is associated with higher degrees of content knowledge.17,18 We have also found that eligible resident inductees into AOA also had significantly higher FLS Manual Skills scores. This finding was consistent with the study hypothesis. Although not all statistically significant, there were clear trends that chief residents who have won any teaching award performed better on the FLS Manual Skills portion of the examination. Further supporting our study hypothesis is that the only chief resident who failed the Manual Skills portion of the FLS examination did not receive any teaching awards. Additionally, the chief resident with the highest overall Manual Skills Section score on the FLS examination received all of the possible teaching awards. All of these findings are consistent with our study hypothesis, and contradictory to the essence of the saying that “[h]e who can, does. He who cannot, teaches.” The findings of the simple linear regression analyses, however, do not support the study hypothesis. Overall, FLS Cognitive scores and FLS Manual Skills scores as a predictor of Likert scale–based medical student evaluation are poor. The Pearson correlation coefficients are weak. Interestingly, the slope of the linear regression line using Cognitive scores as the independent variable is less than zero; in the setting of very poor correlation coefficient with no significant slope, there is essentially no meaningful correlation between these 2 variables. The slope of the linear regression line using Manual Skills scores is greater than zero. This is consistent with the study hypothesis; however, it is not statistically different than zero. The findings in the linear regression analyses would suggest that the medical student Likert scale–based evaluations do not accurately measure teaching ability. These evaluations are global evaluations and not specifically focused on teaching ability. On a 9-point scale, the vast majority of the evaluations were in the highest quartile of the measurement tool range, limiting the variability in these evaluations. In addition, there were no statistically significant higher evaluations of chief residents who received teaching awards and the chief residents who did not receive teaching awards. There does not seem to be discriminant validity of a global resident assessment. We conclude from the linear regression analyses that the global assessment of residents is not influenced by the nature of resident content knowledge or technical ability. The medical students are likely evaluating other variables such as personality or the nature of feedback received. Likert scale– based evaluation of perceived teaching depends on feedback. In a study of knot tying by medical student, student performance was directly related to feedback. Students who received technical feedback perform better than students who received general compliments. However, the
students in the feedback group did not rate the teachers as well as the students in the general compliments group.21 As a result of this study, we believe that the nature of medical student evaluations of residents (and faculty members) should be critically evaluated and a reliable and valid instrument should be instituted. We believe that resident teaching skills should be fostered, in agreement with prior studies.13,14 We also found that there was a small correlation coefficient between the knowledge and psychomotor domains of the FLS examination. This is the first known description of correlation of the Cognitive Section and the Manual Skills Section of the FLS examination. It has been suggested that a high degree of correlation is not necessarily optimal; a very high correlation coefficient between examination modalities may represent redundancy in assessment, whereas a more limited correlation supports the use of both modalities as part of an integrated assessment strategy.19,20 This finding supports the validity of the elements that comprise the FLS examination. There are a few major limitations to this study. The most important was the lack of validation in any of the teaching evaluation tools used in the study. First, the use of our institutional 9-point Likert scale has not been validated and is entirely subjective in nature. We are unsure of what domains are measured, by this Likert scale. To compliment this subjective evaluation, the more objective teaching award status was used. However, although winning a teaching award is binomial as an objective variable, the criteria and rating in the process of receiving these awards are also entirely subjective. Although the teaching awards used in this study have face validity, the content validity and all other validation measures remain unknown. Further study in this domain of surgical education is necessary and is important in the domain for recognition of superior teaching ability. The second major limitation involves the lower study power in this smaller study. Although the entire cohort of chief general surgery resident outcomes on the FLS examination was evaluated from our institution, the low power greatly increases the chance that a type II error occurred. With larger numbers, some of the near-significant trends may have become statistically significant. We plan to continue to follow FLS examination performance and the evaluations of residents by medical students. However, even with the low study power, there were clear trends and differences found. The Manual Skills Section of the FLS examination may not be the optimal way of evaluating technical ability, as the interest and anticipated laparoscopic surgery practice is varied amongst chief general surgery residents. This study is also a single-institution experience, and therefore not generalizable. Further research in this domain on the national level could help validate some of the study findings and conclusions. Finally, there were 9 chief residents who did not have any archived electronic medical
Journal of Surgical Education Volume 71/Number 1 January/February 2014
99
student evaluations. These 9 chief residents graduated in 2009 and 2010, during the relative infancy of electronic documentation of medical student assessments. The omission of these data could bias the results of the study. Conversely, there are a few strengths to this study. The use of scoring on the proctored and standardized FLS examination gives face and content validity to the technical ability being measured. The FLS examination has been rigorously studied, and it has discriminant validity such that the 5 tasks are a valid measure of general surgery technical skills that are transferrable to the operation room.8,9 The standardized outcomes are determined by trained individuals. In addition, multiple measures of teaching ability are stronger than using either outcome individually. Moreover, this study suggests that there are no gender differences in the cognitive, psychomotor, or teaching domains. The study design is straightforward and attempts to answer a very unique and important question in surgery. This study is the first known study that compares technical and teaching skills to test the saying that “[h]e who can, does. He who cannot, teaches.” Overall, we found evidence that the chief general surgery residents who have won teaching awards have greater content knowledge than chief residents who have not won teaching awards. Residents who have been inducted into AOA have superior technical ability than residents not inducted into AOA. There are statistical trends that residents who have won any teaching awards have a higher degree of laparoscopic skills as shown by performance on the FLS examination. We also conclude that a global evaluation of resident performance may not be useful, and there are significant questions surrounding the validity of all teaching awards. Ultimately, we feel that there is evidence that the saying from Bernard Shaw should be changed to “s/ he who can, does and teaches. S/he who cannot, doesn’t.” However, the original quotation cannot totally be disproved in the setting of nonvalidated measures and the higher potential for type II error
3. Baldwin PJ, Paisley AM, Brown SP. Consultant
surgeons’ opinion of the skills required of basic surgical trainees. Br J Surg. 1999;86(8):1078-1082. 4. Datta V, Chang A, Mackay S, Darzi A. The relation-
ship between motion analysis and surgical technical assessments. Am J Surg. 2002;184(1):70-73. 5. Fundamentals of Laparoscopic Surgery. Society of
American Gastrointestinal and Endoscopic Surgeons. Available at: 〈http://www.flsprogram.org/〉; 2013 Accessed 14.02.13. 6. Guidelines for Institutions Granting Privileges Utiliz-
ing Laparoscopic and/or Thoracoscopic Techniques. Society of American Gastrointestinal and Endoscopic Surgeons (SAGES). Available at: 〈http://www.sages. org/publication/id/14/〉; 2013 Accessed 14.02.13. 7. ABS to Require ACLS, ATLS and FLS for General
Surgery Certification, American Board of Surgery. Available at: 〈http://www.absurgery.org/default.jsp? news_newreqs〉; 2013 Accessed 14.02.13. 8. Ritter EM, Kindelan TW, Michael C, Pimentel EA,
Bowyer MW. Concurrent validity of augmented reality metrics applied to the fundamentals of laparoscopic surgery (FLS). Surg Endosc. 2007;21(8):1441-1445. 9. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez
R, Fried GM. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room—A randomized controlled trial. Am J Surg. 2010;199(1):115-120. 10. Morris M, Caskey R, Mitchell M, Sawaya D. Surgical
skills training restructured for the 21st century. J Surg Res. 2012;177(1):33-36. 11. Pelletier M, Belliveau P. Role of surgical residents in
undergraduate surgical education. Can J Surg. 1999;42 (6):451-456. 12. Xu G, Wolfson P, Robeson M, Rodgers JF, Veloski JJ,
ACKNOWLEDGMENT The authors would like to sincerely thank Kathy Haupt and Maggie Mrozinski for help with data acquisition for this study.
Brigham TP. Students' satisfaction and perceptions of attending physicians' and residents' teaching role. Am J Surg. 1998;176(1):46-48. 13. Sheets KJ, Hankin FM, Schwenk TL. Preparing
surgery house officers for their teaching role. Am J Surg. 1991;161(4):443-449. 14. Jamshidi R. Formalizing teaching responsibilities for
junior surgical housestaff encourages educator development. J Surg Educ. 2008;65(6):514-517.
REFERENCES 1. Shaw Bernard. Man and Superman. The Revolution-
ist’s Handbook and Pocket Companion. Cambridge, MA: The University Press; 1903. 2. Safian RD. He who can, does; he who does, teaches.
Catheter Cardiovasc Interv. 2011;78(2):187-188. 100
15. Shellito JL, Osland JS, Helmer SD, Chang FC.
American Board of Surgery examinations: can we identify surgery residency applicants and residents who will pass the examinations on the first attempt? Am J Surg. 2010;199(2):216-222.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
16. How Members are Chosen. Alpha Omega Alpha
Honor Medical Society. Available at: 〈http://www. alphaomegaalpha.org/how.html〉; 2013 Accessed 20.04.13.
17. Sutkin G, Wagner E, Harris I, Schiffer R. What makes
a good clinical teacher in medicine? A review of the literature Acad Med. 2008;83(5):452-466. 18. Srinivasan M, Li ST, Meyers FJ, et al. “Teaching as a
Competency”: competencies for medical educators. Acad Med. 2011;86(10):1211-1220. 19. Nunnink L, Venkatesh B, Krishnan A, Vidhani K,
Udy A. A prospective comparison between written
examination and either simulation-based or oral viva examination of intensive care trainees' procedural skills. Anaesth Intensive Care. 2010;38(5):876-882. 20. Falcone JL, Hamad GG. The state of performance on
the American Board of Surgery Qualifying Examination and Certifying Examination and the effect of residency program size on program pass rates. Surgery. 2012;151(5):639-642. 21. Boehler ML, Rogers DA, Schwind CJ, et al. An
investigation of medical student reactions to feedback: a randomised controlled trial. Med Educ. 2006;40 (8):746-749.
Journal of Surgical Education Volume 71/Number 1 January/February 2014
101
