The American Journal of Surgery (2015) 210, 710-714
Association of Women Surgeons
Interdisciplinary cognitive task analysis: a strategy to develop a comprehensive endoscopic retrograde cholangiopancreatography protocol for use in fellowship training Erin Canopy, M.D.a, Matt Evans, M.D.a, Margaret Boehler, R.N., M.S.a, Nicole Roberts, Ph.D.b, Hilary Sanfey, M.B.B.Ch.a, John Mellinger, M.D.a,* a
Department of Surgery, Southern Illinois University School of Medicine, 701 N First St., PO Box 19638, Springfield, IL 62794, USA; bMedical Education Department, Sophie Davis School of Biomedical Education, New York, NY, USA
KEYWORDS: Cognitive task analysis; Endoscopic retrograde cholangiopan creatography; Fellowship training
Abstract BACKGROUND: Endoscopic retrograde cholangiopancreatography is a challenging procedure performed by surgeons and gastroenterologists. We employed cognitive task analysis to identify steps and decision points for this procedure. METHODS: Standardized interviews were conducted with expert gastroenterologists (7) and surgeons (4) from 4 institutions. A procedural step and cognitive decision point protocol was created from audio-taped transcriptions and was refined by 5 additional surgeons. RESULTS: Conceptual elements, sequential actions, and decision points were iterated for 5 tasks: patient preparation, duodenal intubation, selective cannulation, imaging interpretation with related therapeutic intervention, and complication management. A total of 180 steps were identified. Gastroenterologists identified 34 steps not identified by surgeons, and surgeons identified 20 steps not identified by gastroenterologists. CONCLUSION: The findings suggest that for complex procedures performed by diverse practitioners, more experts may help delineate distinctive emphases differentiated by training background and type of practice. Ó 2015 Elsevier Inc. All rights reserved.
In the current era of medicine, there are many procedures employed by more than one specialty that, despite similar if not identical practice goals, may be
The authors declare no conflicts of interest. * Corresponding author. Tel.: 11-217-545-7240; fax: 11-217-545-4401. E-mail address: [email protected]
Manuscript received February 17, 2015; revised manuscript May 8, 2015 0002-9610/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.amjsurg.2015.05.006
pursued through different training pathways and assessed by varying evaluations of competence. Endoscopic retrograde cholangiopancreatography is one example of a technically challenging procedure performed by both surgeons and gastroenterologists. Recently, the evaluation of competency in performing endoscopic retrograde cholangiopancreatography has become a point of debate because of differences in specialty-specific training requirements.
E. Canopy et al.
Task deconstruction for advanced endoscopy
A number of gastroenterologic-oriented publications have suggested that a minimum of 180 to 200 procedures are required for trainees to achieve a minimum acceptable standard of selective duct cannulation of 80% or more.1–3 Other gastroenterologist-based studies have suggested that a minimum of 350 cases are required before a trainee can consistently achieve deep cannulation of a native papilla.4 On the surgical side, Vitale et al5 found that prior endoscopy experience significantly reduced the training required to reach the desired cannulation rate of 85%. Fellows without previous experience achieved the goal cannulation rate in an average of 9.6 months with 148 endoscopic retrograde cholangiopancreatographies, whereas fellows with prior endoscopy experience achieved the rate in an average of 6.1 months with 85 such procedures.5 We previously employed cognitive task analysis to explore the approach that these 2 specialties use to train fellows and found that both providers indeed think differently in their approach to deconstructing the steps to perform an endoscopic retrograde cholangiopancreatography.6 Specifically, gastroenterologists seem to use a more theory based approach to task deconstruction, whereas surgeons adopt a more practical approach.6 The length of training required in each field is also different. Fellowship training may be as short as one additional year of training after a surgical residency5 and up to 3 to 4 years of additional training for gastroenterology fellowships following internal medicine residency.7 Accordingly, while both groups have cited research supporting their training standards,1,8,9 there is a need to further explain the differences in rationale and training paradigms to standardize training for this complex skill irrespective of specialty background. Studies have shown that experts learn to function automatically rather than actively thinking through each step; therefore, they may unintentionally leave out up to 70% of information while teaching a complex task. Furthermore, they may articulate their decision-making process less than 15% of the time while teaching.10 There has been an interest recently in using cognitive task analysis to enhance the teaching and assessment of intraoperative judgment, decision making, and error recognition with endoscopic and surgical procedures.10–13 Cognitive task analysis is a system of deconstructing automated skills to create a checklist of critical decisionmaking steps and options to avoid error at each step, facilitating the teaching of clinical judgment and decision making.12,13 Prior work has demonstrated that conducting cognitive task analysis evaluations requires interviews of 3 to 5 subject experts to describe the actions and decision steps for a specific task.12,13 These interviews are then analyzed and broken down into a series of distinct, concrete steps from otherwise automated responses.12,13 To date, this method has not been utilized for a more complex endoscopic procedure, such as endoscopic retrograde cholangiopancreatography. We evaluated the use of this technique in interdisciplinary fashion as a means of developing a comprehensive outline of procedural steps and decision points in
711 performing, and ultimately teaching a trainee to perform, endoscopic retrograde cholangiopancreatography.
Methods Institutional Review Board approval was obtained from the Springfield Committee on Research into Human Subjects. Cognitive task analysis interviews were conducted with 7 expert gastroenterologists and 4 expert surgeons. We defined ‘‘experts’’ as physicians who regularly perform endoscopic retrograde cholangiopancreatography in their clinical practice, are responsible for the training of fellows in advanced endoscopy, and/or have published on the topic of this procedure. All experts were known to the authors. The experts were from 4 institutions: 3 were tertiary academic centers and 1 was a community hospital. A nurse educator who is trained in the use of cognitive task analysis conducted each interview. All interviews were structured and conducted in a standardized manner, audio recorded, and then transcribed. In each interview, the expert was asked to explain the procedure in stepwise fashion, as if they were instructing a trainee. The nurse educator was allowed to ask additional questions if a response was unclear. The nurse educator developed individual procedural protocols from the transcriptions of the interviews. A 3stage approach was then adopted to define and refine the final gold standard protocol. In the first step, the individual protocols were returned to each, original expert to review for accuracy. Once accuracy was confirmed, the individual protocols were compiled into a common protocol. In the second stage, these same 11 physicians representing both gastroenterology and surgery training backgrounds reviewed the common protocol to create a gold standard protocol. The gastroenterologists chose not to participate further in the third stage of the study. In this third and final stage, the 4 surgeon participants were asked to invite their endoscopic retrograde cholangiopancreatography trained fellows who had graduated and were established in practice to further refine the protocol. These 8 endoscopic retrograde cholangiopancreatography fellowship-trained surgeons, none of whom were included in the original study, were asked to review the gold standard protocol and indicate whether a step was critical to perform or not. Critical steps were retained and noncritical steps were discarded, based on majority opinion.
Results Analysis of the resulting integrated protocol identified key components of the procedure including indications and contraindications, equipment issues, fundamental background concepts, and 5 foundational task areas: patient preparation, duodenal intubation, selective cannulation, imaging interpretation with related therapeutic intervention, and complication management (Table 1). Conceptual elements as well as sequential actions and
The American Journal of Surgery, Vol 210, No 4, October 2015
decision points were iterated for each of these 5 tasks. Experts from both groups of specialists identified 180 total steps. Gastroenterologists identified 34 steps that were not identified by surgeons, and surgeons identified 20 steps that were not identified by gastroenterologists. These 11 study participants representative of both training backgrounds were asked to further refine the protocol so derived. They did not add additional steps and, in fact, identified 56 steps (9 steps identified by both groups, 34 from gastroenterologists, and 13 from surgeons) that they did not feel were critical to include and were excluded from the final protocol. A total of 5 of the 8 endoscopic retrograde cholangiopancreatography fellowship-trained surgeons responded to the follow-up survey resulting in a 62.5% response rate. All but one of the 5 was more than 5 years post-training. This group was asked to indicate which steps were critical to know for training from the expert-derived gold standard protocol as outlined above. The initial protocol developed from the cognitive task analysis interviews consisted of 280 concepts and steps. After the survey (Stage 3), the protocol was refined to a total of 214 concepts and steps with no additional concepts or steps added by survey respondents (Table 1). The final gold standard protocol consisted of elements provided by both specialties. Forty-six percent of the protocol steps were agreed upon by both specialties, with 34% from gastroenterologist interviews only and an additional 20% from surgeon interviews only (Table 2).
Comments Technical skill development in medicine has been undergoing significant reflection and refinement in the past 10 to 15 years. Included in this development is an increasing focus on proficiency-based training and preclinical skill leveraging through simulation. Deconstructive efforts to stratify skills into definable, teachable, and testable educational elements have become an important part of such trends. It is well known that experts struggle to deconstruct complex tasks into teachable steps. Therefore the use of cognitive task analysis to define and thus explain these critical steps could facilitate training. Cognitive task analysis in particular has proved to be a useful strategy in these endeavors, including endoscopic procedures.10–13 Earlier studies with cognitive task analysis have explored procedures perhaps less complex than endoscopic retrograde cholangiopancreatography, thus raising the question of whether modifications in the use of this methodology might require modification for a more complex procedure practiced by individuals with distinct training paradigms in current gastrointestinal practice. Prior studies have suggested that in order for a cognitive task analysis to be accurate, 3 to 5 experts need to be interviewed.13 In this multidisciplinary study, we employed an initial group of 11 expert surgeons and gastroenterologists, and an additional 5 expert surgeons to refine the final product. By using a multidisciplinary approach, this article illustrates the difference in expert opinion between 2
Final gold standard protocol as defined pre- and postsurvey
General concepts Indications Contraindications Absolute Relative Task list Task 1: Prepare the patient Concepts Steps Task 2: Intubate the duodenum Concepts Steps Task 3: Access the desired duct Concepts Steps Task 4: Interpret, diagnose, treat Concepts Steps Task 5: Manage complications Concepts Steps Total
3 3 5
3 3 5
2 38 280
2 32 214
Postsurvey, only 6 of the 7 protocol steps on general concepts were deemed to be critical for the gold standard protocol.
E. Canopy et al.
Task deconstruction for advanced endoscopy
Table 2 Percentage of total protocol steps in each key task area in the final protocol identified by gastroenterologists alone, surgeons alone, or by both disciplines Prepare patient Intubate duodenum Access desired duct Interpret, diagnose, treat Manage complications Average frequencies
25% 45% 37% 25% 38% 34%
30% 25% 15% 14% 16% 20%
45% 30% 48% 61% 46% 46%
specialties in directing key steps of a complex procedure, which may have been otherwise missed by a single specialty study and arguably strengthens the credibility of our final gold standard protocol. The findings suggest that for complex procedures performed by individuals of varying training backgrounds, a larger number of experts may be useful in identifying distinctive areas of emphasis differentiated by training background and type of practice in which the technique is utilized. For example, Table 2 demonstrates that there was less than 50% concordance between the 2 specialties for most of the key tasks. In addition, with more complex procedures and associated cognitive task analysis products, a larger number of expert reviews may be useful in refining the product than what has been previously recommended. These findings may be of significance in planning interdisciplinary training paradigms of the future in complex therapeutic endoscopy and gastrointestinal surgery. The study has several important limitations. The surgeons who participated were trained by only 2 different surgeons (both of whom were also participants), while the gastroenterologists came from less inbred training backgrounds. Conceivably, this difference in training backgrounds may have contributed to a more nuanced pattern of responses from the gastroenterology providers and, in turn, a higher than expected rate of revision of the protocol when further refined through the survey mechanism by the surgical cohort. This difference may further explain some of the ongoing iterative changes demonstrated more evidently in the secondary refinements of the protocol than what would be predicted by prior cognitive task analysis studies. One may also argue that technical competence with a procedure such as endoscopic retrograde cholangiopancreatography may be described adequately without detailing all the nuances a given expert may recognize, depending on their practice setting. Regardless, different training backgrounds may influence the type of disease process, severity, and frequency, particularly in more complex procedures such as endoscopic retrograde cholangiopancreatography (eg, stone disease for surgeons vs primary sclerosing cholangitis in a tertiary gastroenterologic practice). In these more specialized practices, a process of iteration and constant comparison may prove useful to further elaborate on the protocol steps included in the multidisciplinary gold standard protocol that this study produced. Finally, during the final
refinement of our gold standard protocol, the gastroenterology participants declined further involvement, leaving only surgery-trained participants to review and further refine the final protocol. This may have further biased our findings and the final end product may accordingly benefit from further evaluation by gastroenterologists for balance and consistency’s sake.
Conclusion Cognitive task analysis is a useful tool for deconstructing a complex task such as endoscopic retrograde cholangiopancreatography for teaching purposes. With more complex and multidisciplinary procedures such as endoscopic retrograde cholangiopancreatography, it may be important to include more experts and more iterative refinements of the cognitive task analysis product before reaching a final consensus result. Procedural complexity, training background, and practice-based nuances that vary among specialties may all impact this process. Cognitive task analysis holds promise as a tool for exploring objective consensus in developing comprehensive training paradigms in settings where procedures are employed by practitioners of diverse training background. Cognitive task analysis may also prove a more objective source for categorizing and focusing training in such settings than can be achieved by simple calculations of procedural volume.
Acknowledgments The authors wish to thank Aman Ali, MD, Southern Illinois University; Jim Fullerton, MD, Springfield Clinic; M. Chris Huang, MD, Southern Illinois University; Jeffrey Ponsky, MD, The Cleveland Clinic; and Gary Vitale, MD, University of Louisville for their assistance with this study.
References 1. Chutkan RK, Ahmad AS, Cohen J, et al. ERCP core curriculum. Gastrointest Endosc 2006;63:361–76. 2. Ang TL, Cheng J, Khor JLC, et al. Guideline on training and credentialing in endoscopic retrograde cholangiopancreatography. Singapore Med J 2011;52:654–7.
714 3. Jowell PS, Baillie J, Branch MS, et al. Quantitative assessment of procedural competence. A prospective study of training in endoscopic retrograde cholangiopancreatography. Ann Intern Med 1996;125:983–9. 4. Verma D, Gostout CJ, Petersen BT, et al. Establishing a true assessment of endoscopic competence in ERCP during training and beyond: a single-operator learning curve for deep biliary cannulation in patients with native papillary anatomy. Gastrointest Endosc 2007;65:394–400. 5. Vitale GC, Zavaleta CM, Vitale DS, et al. Training surgeons in endoscopic retrograde cholangiopancreatography. Surg Endosc 2006;20:149–52. 6. Boehler M, Roberts N, Sanfey H, et al. Do surgeons and gastroenterologists describe endoscopic retrograde cholangiopancreatography differently? A qualitative study. J Surg Ed(submitted); 2015. 7. Rosenthal LS. Is a fourth year of training necessary to become competent in EUS and ERCP? Notes from the 2008 class of advanced endoscopy fellows. Gastrointest Endosc 2008;68:1150–2. 8. Guidelines for Training Surgeons in Endoscopic Retrograde Cholangiopancreatography. Available at: http://www.sages.org/publications/guidelines/
The American Journal of Surgery, Vol 210, No 4, October 2015
guidelines-for-training-in-diagnostic-and-therapeutic-endoscopicretrograde-cholangiopancreatography-ercp/. Accessed February 16, 2015. ASGE Training Committee. Principles of training in GI endoscopy. Gastrointest Endosc 2012;75:231–5. Sullivan ME, Yates KA, Inaba K, et al. The use of cognitive task analysis to reveal the instructional limitations of experts in the teaching of procedural skills. Acad Med 2014;89:811–6. Sullivan ME, Brown CVR, Peyre SE, et al. The use of cognitive task analysis to improve the learning of percutaneous tracheostomy placement. Am J Surg 2007;193:96–9. Sullivan ME, Ortega A, Wasserberg N, et al. Assessing the teaching of procedural skills: can cognitive task analysis add to our traditional teaching methods? Am J Surg 2008;195:20–3. Sullivan ME, Yates KA, Baker CJ, et al. Cognitive-task-analysis and its role in teaching technical skills. In: Tsuda ST, Scott DJ, Jones DB, editors. Textbook of Simulation, Skills and Team Training. 1st ed. Woodbury, CT: Cine´-Med, Inc; 2012.