Use of the Isabel Decision Support System to Improve Diagnostic Accuracy of Pediatric Nurse Practitioner and Family Nurse Practitioner Students Rita Marie John, DNP, EdD, CPNP1, Elizabeth Hall, FNP, DNP1, Suzanne Bakken, DNSc1,2 1 School of Nursing and 2Department of Biomedical Informatics, Columbia New York, New York, U.S.A. Abstract Patient safety is a priority for healthcare today. Despite a large proportion of malpractice claims the result of diagnostic error, the use of diagnostic decision support to improve diagnostic accuracy has not been widely used among healthcare professionals. Moreover, while the use of diagnostic decision support has been studied in attending physicians, residents, medical students and advanced practice nurses, the use of decision support among Advanced Practice Nurse (APN) students has not been studied. The authors have implemented the Isabel diagnostic decision support system into the curriculum and are evaluating its impact. The goals of the evaluation study are to describe the diagnostic accuracy and self-reported confidence levels of Pediatric Nurse Practitioner (PNP) and Family Nurse Practitioner (FNP) students over the course of their programs, to examine changes in diagnostic accuracy and self-reported confidence levels over the study period, and to evaluate differences between FNP and PNP students in diagnostic accuracy and self-reported confidence levels for pediatric cases. This paper summarizes establishment of the academic/industry collaboration, case generation, integration of Isabel into the curriculum, and evaluation design. Introduction The risk of death from medical errors has been estimated at 44,000 to 98,000.1 Preventing injury to patients as a result of their interaction with the health care system is a goal of patient safety with several improvement over the past ten years.2 More recently, patient safety experts have targeted diagnostic error as an area for improvement.3 Diagnostic error is defined as missing, delaying, or failure to diagnosis a condition that is subsequently diagnosed by a definitive test or finding. 3 Diagnostic errors may be missed by the clinician who fails to realize that the diagnostic mistake.3,4 The focus on improved diagnosis is a newer area in the patient safety area5 despite the fact that malpractice claims as a result of diagnostic errors is the one of the most frequent kind of claim, accounting for 31.87% of malpractice claims.6 The estimate of diagnostic error rate is approximately 15%.7 Diagnostic decision support systems (DDSS) have been around for more than three decades and systems such as Iliad, DxPlain, and Internist were developed using artificial intelligence approaches from computer science. Isabel, a more recently developed web-based DDSS, takes advantage of the immense volume of diagnostic information available across electronic sources to create a checklist of potential diagnoses related to terms entered by the user. The use of DDSS has been studied in attending physicians, residents, medical students and advanced practice nurses (APN) 8,9,10,11,12, but not in APN students. As part of the larger Wireless Informatics for Safe and Evidence-based (WISE) APN project – whose goal is to reduce health disparities in underserved population through preparing APN students, faculty, and preceptors to use informatics approaches for improving patient safety and enhancing evidencebased practice in a culturally-competent manner – the authors have implemented the Isabel DDSS into the curriculum and are evaluating its impact. The goals of the evaluation study are to describe the diagnostic accuracy and self-reported confidence levels of Pediatric Nurse Practitioner (PNP) and Family Nurse Practitioner (FNP) students over the course of their programs, to examine changes in diagnostic accuracy and self-reported confidence levels over the study period, and to evaluate differences between FNP and PNP students in diagnostic accuracy and self-reported confidence levels for pediatric cases. This paper summarizes the establishment of the academic/industry collaboration, case generation, integration of Isabel into the curriculum, and evaluation design. Academic/Industry Collaboration Isabel has been in use in selective parts of the BS/MS entry-to-practice program and PNP program for more than 5 years, but no formal evaluation has been conducted previously. In addition, Isabel had been used in practice by the PNP program director (John). Isabel leadership approached the School of Nursing to discuss opportunities for collaboration and introduced the Isabel IDEAS program. The IDEAS program is designed to use with an online case and allows the faculty to write in the diagnostic decision-making involved in the case. The student can read the denouement of the case after the diagnoses, diagnostics, and treatment plan are written. Isabel IDEAS tracks the
student’s use of the system, the amount of time spent on the system, and the use of system diagnostic aids including books, Pubmed, and Google. A previous study using Isabel IDEAS with 20 medical students evaluating four cases pre-Isabel DDSS and post-Isabel DDSS revealed a significant improvement in the diagnostic accuracy (p < 0.05).14 In order to avoid a potential conflict of interest in the evaluation study and its associated publications, it was decided that the only role that the Isabel DSSS team would play would be to provide access to the system through activating student accounts and providing program support. No financial support was received by the researchers. The cases generated by the faculty will become a part of the IDEAS system after the study is over. Case Generation A plan was developed to generate cases in a systematic fashion using a standardized the DNP resident pediatric comprehensive evaluation guidelines. 15 To mimic real world patient scenarios, where there is information presented by patients that is not significant in developing the differential, cases include material that is not salient to the case. Cases were generated from clinical practice of either previous students or faculty members. Each case contained key symptoms and signs for that particular diagnosis to help students learn about common clinical presentations. The case generation is time consuming for both the faculty writing the cases and the reviewers. The denouement of the cases requires care using multiple resources to make sure that the explanations meet the needs of APN students. The cases and the diagnostic denouement were reviewed by two PNPs with greater than 25 years of experience rather than by physicians as these are NP cases and contain many nursing interventions within the plan. The diagnoses for the cases were divided by body system to evenly distribute the types of cases. After the cases were written, the key history points were put into the ISABEL IDEAS system to see if the correct diagnosis was generated by the system. Table 1 shows the system, the actual diagnosis, the semester it was introduced, and the level of difficulty of the cases determined by the case developers and the reviewers. Table 1. System, diagnosis, level of difficulty, and semester System
Final Diagnosis
Level of Difficulty
Semester
Dermatology
Tinea Capitis Lichen Striatus Retropharyngeal Abscess Conjunctivitis Epstein Barr Virus mononucleosis Cervical Adenitis Hypertrophic Cardiomyopathy Long QT syndrome with Torsades de pointe Foreign Body Cough due to GERD Leukemia ABO setup/erythema toxicum Appendicitis Intussusception Urinary Tract Infection with pyelonephritis Pelvic inflammatory disease with Tubo-ovarian abscess Legg Calve Perthes Slipped Capital Femoral Epiphysis Malaria Typhoid fever Absence Seizure
Easy Moderate Moderate Easy Easy
Spring 2011 Fall 2011 Spring 2011 Fall 2011 Fall 2011
Easy Moderate
Fall 2011 Spring 2011
Hard Hard Moderate Hard Moderate
Fall 2011 Fall 2011 Fall 2011 Spring 2011 Spring 2011
Moderate Moderate Moderate
Fall 2011 Fall 2011 Spring 2011
Moderate
Spring 2011
Easy
Spring 2011
Hard Hard Easy
Spring 2011 Spring 2011 Fall 2011
HEENT Lymph
Cardiac
Pulmonary Hematology
Gastroenterology Urology Gynecology
Orthopedics
Infectious Disease Neurological
Migraine Headache
Moderate
Fall 2011
In most of the cases, the students are given some supporting diagnostics to help in generating the differential. For example, in the female with the hypertrophic cardiomyopathy, an electrocardiogram is included as part of the initial case. Similarly, in the patient with the Tinea Capitis, a picture is available in the system for the student to evaluate; in the patient with a limp, a hip X-ray is included that shows aseptic necrosis of the femoral head; and in the patient with the retropharyngeal abscess, a lateral neck X-ray includes the presence a retropharyngeal abscess. The cases had enough diagnostic results to help students generate the correct differential diagnosis. The students were expected to be able to generate additional diagnostics that they would consider ordering based the case. To facilitate learning beyond a relatively simple comparison of student results with the correct results, extensive information about each differential diagnosis was provided and the diagnostic reasoning beyond why this particularly diagnosis was correct or incorrect was elucidated. Information about the pros and cons of diagnostic tests as well as costs involved was discussed in the diagnostics section. Finally, each case’s treatment was outlined: diagnostics tests, therapeutic interventions, counseling and education, health care maintenance, referrals, and followup care. The substantial faculty effort spent in the development and review of cases and integration into Isabel IDEAS enables the student to review the decisions that she made, review the appropriate actions, and evaluate the outcome of her decision-making in a non-threatening environment. Integration into the Curriculum Review of cases is integrated into the PNP and FNP curricula. Prior to using Isabel IDEAS, the students read the case, identify the key history points, develop an initial differential diagnosis list, diagnostic work up, and treatment plan and rate their level of confidence (0-100) for each aspect. Then the students enter the key history points that they previously identified into Isabel IDEAS. After a list of possible diagnoses is generated by Isabel, the student can use any of the resources of the system to further consider the differential diagnoses generated by the system. Subsequently, generate a final list of differential diagnoses, diagnostic workup, and treatment plan along with associated levels of confidence (0-100). The figure below shows the diagnoses generated by the system once the key history points are entered.
In order to ensure that students spend an adequate amount of time on each case to promote learning, case evaluations are graded. The students do not lose credit for a wrong but reasonable diagnosis. Rather, a rubric was developed to evaluate the results of the case reviews: 1) time spent in the system, 2) appropriateness of the key history points, 3) consistency of the differential diagnosis list with the key history points, 4) consistency of diagnostics with the differential diagnosis list, 5) completeness of the diagnostic work up, 6) appropriateness of the treatment plan, 7) completeness of treatment plan. The grade for review of cases in Isabel IDEAS is weighted the same in both PNP and FNP programs. The following figure shows the denouement of the case that enables the researcher to evaluate the student’s entry.
Evaluation The evaluation design is a repeated measures (2 points in time) comparative (PNP vs. FNP) design. The research questions are: 1) What is the diagnostic accuracy of PNP and FNP students before and after use of Isabel IDEAS? 2) What are the self-reported confidence levels in diagnostic accuracy of PNP and FNP students before and after use of Isabel IDEAS? 3) Are there changes in diagnostic accuracy and self-reported confidence levels over time? 4) Are there differences between PNP and FNP students in diagnostic accuracy and self-reported confidence levels for pediatric cases? The sample comprises 37 PNP and 40 FNP students. All are female except one FNP student. All have less than 5 years of RN experience, and more than 90% are between the ages of 20-29 years. No students reported continued use of Isabel since their initial exposure in the BS/MS informatics course. The main outcome measures are the accuracy of the diagnosis and participants' self rated level of confidence in the diagnosis before and after the use of Isabel. The PNP and the FNP students will be evaluated as a group by two-way analysis of variance with diagnostic accuracy and diagnostic confidence as dependent variables. For comparisons between the specialties, the between factor is specialty (PNP vs. FNP) and the within factor is time (semester). Interaction effects will also be evaluated. Conclusion The collaboration between Isabel DDSS and the Columbia University School of Nursing has enabled APN students to gain experience with DDSS to develop differential diagnoses for a variety of pediatric cases. The generation of the cases by faculty requires clinical expertise and attention to detail to make sure cases are complete and the denouement meets the needs of relatively inexperienced students. Acknowledgments The authors thank Isabel for access and support for Isabel DDSS and IDEAS. The integration of Isabel into the curriculum and its evaluation are supported by Wireless Informatics for Safe and Evidence-based APN Care (Health Services Resources Administration D11 HP07346, Suzanne Bakken, Principal Investigator. References 1.
2.
Brennan TA, Leape LL, Laird NM, Herbert L, Localio AR, Lawthers AG, et al Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Student I. N Engl J Med. 1991;324(60):370-6. Wachter R. Patient safety at ten: unmistakable progress, troubling gaps. Health Affairs. 2010; 29(1), 1-9.
3.
Newman-Toker DE, Pronovost PJ. Diagnostic errors: the next frontier for patient safety. JAMA. 2009;301(10):1060-2. 4. Berner E, & Graber M. Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008;121(5A):S2-S23. 5. Berner E. Diagnostic error in medicine: introduction. Adv in Health Sci Educ. 2009;14:1-5. 6. Carroll AE, Buddenbaum JL. Malpractice claims involving pediatricians; epidemiology and etiology. Pediatrics. 2007;120;10 7. Elstein AS. Clinical reasoning in medicine. In: Higgs JJM, ed. Clinical reasoning in the Health Professions 8. Graber ML, Mathew A. Performance of a Web-based clinical diagnosis support sytem for internist. J Gen Inern Med;23(suppl 1):37-40. 9. Graber M, Tompkins D. & Holland J. Resources medical students use to derive a differential diagnosis. Med Teach. 2010;31:522-27. 10. Amy LS., Borowitz SM, Brown P, Medelsohn MJ, Lyman JA. Impact of a web-based diagnosis reminder system on errors of diagnosis. AMAI 2006 Symposium Proceedings. 843 11. Medow MA., Arkes HR, Shaffer VA. Are residents’ decisions influenced more by a decision aid or specialist’s opinion? A randomized controlled trial. J Gen Intern Med. 2010;25(4):316-20. 12. Weber S. A qualitative analysis of how advanced practice nurses use clinical decision support systems. J of Am Acad Nurs, 2007;19:652-667. 13. 14. Elstein AS: Thinking about diagnostic thinking: a 30-year perspective, Adv Health Sci Educ Theory Pract . 2009;14(Suppl 1):7-18. 15. Carlson J., Abel M, Bridges D, Tomkowiak The impact of a diagnostic reminder system on student clinical reasoning during simulated case studies. Sim Healthc.2011 Feb;6(1):11-7. 16. John RM, Honig, J. DNP approach to providing Pediatric and adolescent care. In: J Smolowitz, J Honig, C Reinisch, Writing DNP Clinical case Narratives. New York: Springer Books. 2010. 17. Kawamot K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of the trials to identify features critical to success. BMJ;330:765-73.
.
student’s use of the system, the amount of time spent on the system, and the use of system diagnostic aids including books, Pubmed, and Google. A previous study using Isabel IDEAS with 20 medical students evaluating four cases pre-Isabel DDSS and post-Isabel DDSS revealed a significant improvement in the diagnostic accuracy (p < 0.05).14 In order to avoid a potential conflict of interest in the evaluation study and its associated publications, it was decided that the only role that the Isabel DSSS team would play would be to provide access to the system through activating student accounts and providing program support. No financial support was received by the researchers. The cases generated by the faculty will become a part of the IDEAS system after the study is over. Case Generation A plan was developed to generate cases in a systematic fashion using a standardized the DNP resident pediatric comprehensive evaluation guidelines. 15 To mimic real world patient scenarios, where there is information presented by patients that is not significant in developing the differential, cases include material that is not salient to the case. Cases were generated from clinical practice of either previous students or faculty members. Each case contained key symptoms and signs for that particular diagnosis to help students learn about common clinical presentations. The case generation is time consuming for both the faculty writing the cases and the reviewers. The denouement of the cases requires care using multiple resources to make sure that the explanations meet the needs of APN students. The cases and the diagnostic denouement were reviewed by two PNPs with greater than 25 years of experience rather than by physicians as these are NP cases and contain many nursing interventions within the plan. The diagnoses for the cases were divided by body system to evenly distribute the types of cases. After the cases were written, the key history points were put into the ISABEL IDEAS system to see if the correct diagnosis was generated by the system. Table 1 shows the system, the actual diagnosis, the semester it was introduced, and the level of difficulty of the cases determined by the case developers and the reviewers. Table 1. System, diagnosis, level of difficulty, and semester System
Final Diagnosis
Level of Difficulty
Semester
Dermatology
Tinea Capitis Lichen Striatus Retropharyngeal Abscess Conjunctivitis Epstein Barr Virus mononucleosis Cervical Adenitis Hypertrophic Cardiomyopathy Long QT syndrome with Torsades de pointe Foreign Body Cough due to GERD Leukemia ABO setup/erythema toxicum Appendicitis Intussusception Urinary Tract Infection with pyelonephritis Pelvic inflammatory disease with Tubo-ovarian abscess Legg Calve Perthes Slipped Capital Femoral Epiphysis Malaria Typhoid fever Absence Seizure
Easy Moderate Moderate Easy Easy
Spring 2011 Fall 2011 Spring 2011 Fall 2011 Fall 2011
Easy Moderate
Fall 2011 Spring 2011
Hard Hard Moderate Hard Moderate
Fall 2011 Fall 2011 Fall 2011 Spring 2011 Spring 2011
Moderate Moderate Moderate
Fall 2011 Fall 2011 Spring 2011
Moderate
Spring 2011
Easy
Spring 2011
Hard Hard Easy
Spring 2011 Spring 2011 Fall 2011
HEENT Lymph
Cardiac
Pulmonary Hematology
Gastroenterology Urology Gynecology
Orthopedics
Infectious Disease Neurological
Migraine Headache
Moderate
Fall 2011
In most of the cases, the students are given some supporting diagnostics to help in generating the differential. For example, in the female with the hypertrophic cardiomyopathy, an electrocardiogram is included as part of the initial case. Similarly, in the patient with the Tinea Capitis, a picture is available in the system for the student to evaluate; in the patient with a limp, a hip X-ray is included that shows aseptic necrosis of the femoral head; and in the patient with the retropharyngeal abscess, a lateral neck X-ray includes the presence a retropharyngeal abscess. The cases had enough diagnostic results to help students generate the correct differential diagnosis. The students were expected to be able to generate additional diagnostics that they would consider ordering based the case. To facilitate learning beyond a relatively simple comparison of student results with the correct results, extensive information about each differential diagnosis was provided and the diagnostic reasoning beyond why this particularly diagnosis was correct or incorrect was elucidated. Information about the pros and cons of diagnostic tests as well as costs involved was discussed in the diagnostics section. Finally, each case’s treatment was outlined: diagnostics tests, therapeutic interventions, counseling and education, health care maintenance, referrals, and followup care. The substantial faculty effort spent in the development and review of cases and integration into Isabel IDEAS enables the student to review the decisions that she made, review the appropriate actions, and evaluate the outcome of her decision-making in a non-threatening environment. Integration into the Curriculum Review of cases is integrated into the PNP and FNP curricula. Prior to using Isabel IDEAS, the students read the case, identify the key history points, develop an initial differential diagnosis list, diagnostic work up, and treatment plan and rate their level of confidence (0-100) for each aspect. Then the students enter the key history points that they previously identified into Isabel IDEAS. After a list of possible diagnoses is generated by Isabel, the student can use any of the resources of the system to further consider the differential diagnoses generated by the system. Subsequently, generate a final list of differential diagnoses, diagnostic workup, and treatment plan along with associated levels of confidence (0-100). The figure below shows the diagnoses generated by the system once the key history points are entered.
In order to ensure that students spend an adequate amount of time on each case to promote learning, case evaluations are graded. The students do not lose credit for a wrong but reasonable diagnosis. Rather, a rubric was developed to evaluate the results of the case reviews: 1) time spent in the system, 2) appropriateness of the key history points, 3) consistency of the differential diagnosis list with the key history points, 4) consistency of diagnostics with the differential diagnosis list, 5) completeness of the diagnostic work up, 6) appropriateness of the treatment plan, 7) completeness of treatment plan. The grade for review of cases in Isabel IDEAS is weighted the same in both PNP and FNP programs. The following figure shows the denouement of the case that enables the researcher to evaluate the student’s entry.
Evaluation The evaluation design is a repeated measures (2 points in time) comparative (PNP vs. FNP) design. The research questions are: 1) What is the diagnostic accuracy of PNP and FNP students before and after use of Isabel IDEAS? 2) What are the self-reported confidence levels in diagnostic accuracy of PNP and FNP students before and after use of Isabel IDEAS? 3) Are there changes in diagnostic accuracy and self-reported confidence levels over time? 4) Are there differences between PNP and FNP students in diagnostic accuracy and self-reported confidence levels for pediatric cases? The sample comprises 37 PNP and 40 FNP students. All are female except one FNP student. All have less than 5 years of RN experience, and more than 90% are between the ages of 20-29 years. No students reported continued use of Isabel since their initial exposure in the BS/MS informatics course. The main outcome measures are the accuracy of the diagnosis and participants' self rated level of confidence in the diagnosis before and after the use of Isabel. The PNP and the FNP students will be evaluated as a group by two-way analysis of variance with diagnostic accuracy and diagnostic confidence as dependent variables. For comparisons between the specialties, the between factor is specialty (PNP vs. FNP) and the within factor is time (semester). Interaction effects will also be evaluated. Conclusion The collaboration between Isabel DDSS and the Columbia University School of Nursing has enabled APN students to gain experience with DDSS to develop differential diagnoses for a variety of pediatric cases. The generation of the cases by faculty requires clinical expertise and attention to detail to make sure cases are complete and the denouement meets the needs of relatively inexperienced students. Acknowledgments The authors thank Isabel for access and support for Isabel DDSS and IDEAS. The integration of Isabel into the curriculum and its evaluation are supported by Wireless Informatics for Safe and Evidence-based APN Care (Health Services Resources Administration D11 HP07346, Suzanne Bakken, Principal Investigator. References 1.
2.
Brennan TA, Leape LL, Laird NM, Herbert L, Localio AR, Lawthers AG, et al Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Student I. N Engl J Med. 1991;324(60):370-6. Wachter R. Patient safety at ten: unmistakable progress, troubling gaps. Health Affairs. 2010; 29(1), 1-9.
3.
Newman-Toker DE, Pronovost PJ. Diagnostic errors: the next frontier for patient safety. JAMA. 2009;301(10):1060-2. 4. Berner E, & Graber M. Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008;121(5A):S2-S23. 5. Berner E. Diagnostic error in medicine: introduction. Adv in Health Sci Educ. 2009;14:1-5. 6. Carroll AE, Buddenbaum JL. Malpractice claims involving pediatricians; epidemiology and etiology. Pediatrics. 2007;120;10 7. Elstein AS. Clinical reasoning in medicine. In: Higgs JJM, ed. Clinical reasoning in the Health Professions 8. Graber ML, Mathew A. Performance of a Web-based clinical diagnosis support sytem for internist. J Gen Inern Med;23(suppl 1):37-40. 9. Graber M, Tompkins D. & Holland J. Resources medical students use to derive a differential diagnosis. Med Teach. 2010;31:522-27. 10. Amy LS., Borowitz SM, Brown P, Medelsohn MJ, Lyman JA. Impact of a web-based diagnosis reminder system on errors of diagnosis. AMAI 2006 Symposium Proceedings. 843 11. Medow MA., Arkes HR, Shaffer VA. Are residents’ decisions influenced more by a decision aid or specialist’s opinion? A randomized controlled trial. J Gen Intern Med. 2010;25(4):316-20. 12. Weber S. A qualitative analysis of how advanced practice nurses use clinical decision support systems. J of Am Acad Nurs, 2007;19:652-667. 13. 14. Elstein AS: Thinking about diagnostic thinking: a 30-year perspective, Adv Health Sci Educ Theory Pract . 2009;14(Suppl 1):7-18. 15. Carlson J., Abel M, Bridges D, Tomkowiak The impact of a diagnostic reminder system on student clinical reasoning during simulated case studies. Sim Healthc.2011 Feb;6(1):11-7. 16. John RM, Honig, J. DNP approach to providing Pediatric and adolescent care. In: J Smolowitz, J Honig, C Reinisch, Writing DNP Clinical case Narratives. New York: Springer Books. 2010. 17. Kawamot K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of the trials to identify features critical to success. BMJ;330:765-73.
.
