An Evaluation of a Natural Language Processing Tool for Identifying and Encoding Allergy Information in Emergency Department Clinical Notes Foster R. Goss DO, MMSc1, Joseph M. Plasek, MS2, Jason J. Lau, BS2, Diane L. Seger, RPh3, Frank Y. Chang, MSE4, Li Zhou, MD, PhD2,4,5 1 Tufts Medical Center, Department of Emergency Medicine and Clinical Decision Making, Boston, MA; 2Division of General Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA; 3Clinical & Quality Analysis, Partners HealthCare System, Boston, MA; 4Clinical Informatics, Partners eCare, Partners HealthCare System, Boston, MA; 5Harvard Medical School, Boston, MA Abstract Emergency department (ED) visits due to allergic reactions are common. Allergy information is often recorded in free-text provider notes; however, this domain has not yet been widely studied by the natural language processing (NLP) community. We developed an allergy module built on the MTERMS NLP system to identify and encode food, drug, and environmental allergies and allergic reactions. The module included updates to our lexicon using standard terminologies, and novel disambiguation algorithms. We developed an annotation schema and annotated 400 ED notes that served as a gold standard for comparison to MTERMS output. MTERMS achieved an F-measure of 87.6% for the detection of allergen names and no known allergies, 90% for identifying true reactions in each allergy statement where true allergens were also identified, and 69% for linking reactions to their allergen. These preliminary results demonstrate the feasibility using NLP to extract and encode allergy information from clinical notes. Introduction Allergic reactions result in over 1 million visits per year to emergency departments (ED)1. Foods and antibiotics are among the most common causes of these visits with an estimated 525,600 and 112,116 visits annually2, 3. Adverse drug events (ADEs) due to a patient receiving a medication to which they were known to be allergic is present in both the inpatient4 and outpatient5 settings. Careful documentation and encoding of patient allergies is critical to patient safety and ensuring drug allergy checking and clinical decision alerts are triggered. Natural language processing (NLP) has shown promise in this domain6, yet few have adapted its use to detecting allergies from clinical notes, which often contains allergy information both within and outside the allergy section of the chart. This paper presents our early experience and preliminary findings in developing an allergy module for a general NLP system, named Medical Text Extraction, Reasoning, and Mapping System (MTERMS), to extract and encode allergy information from clinical text. We assessed the system performance of the MTERMS in processing free-text ED notes for allergen names and associated allergic reactions. Background Emergency physicians are often at the forefront of treating and managing acute allergic reactions. Their clinical notes typically convey the allergen, the patient’s reaction and their response to treatment. While allergy information is usually recorded in the allergy section, it may not be included in the allergy section at the time of presentation or it may be located elsewhere (e.g., medical decision making or ED course). Additionally, the physician’s narrative may include important details reflecting their certainty the patient is having an allergic reaction and the treatment given. As such, pertinent allergy information may be buried in clinical text, raising a concern for patient safety, as this information is not interoperable with computerized drug-allergy alerting or drug-drug interaction checking. Most NLP applications to date have focused on other specific domains including clinical problems7, medications8, vaccination reactions9, suicide ideation10, smoking status11, or ADE detection12, etc. While allergy (a type of adverse event) is an important domain, it has not yet been widely investigated. In cTAKES13, a NLP system by Savona et al, allergies to a given medication are handled by setting the
580
negation attribute of that medication to “is negated”. In MedLEE, terms such as “allergy” or “toxic” are captured and assigned to the semantic type “reaction”14. Melton et al15 used MedLEE16, 17 to identify 45 adverse event types (e.g., loss or impairment of bodily functions) using discharge summaries. A set of computerized queries implemented the inclusion and exclusion logic for each event and identified the adverse events against the NLP output. Melton’s approach achieved an overall sensitivity of 28%, specificity of 98.5%, and PPV of 45%. Most ADE detection algorithms use simple keyword search methods that search notes for relevant trigger words14. In general, these studies have reported low sensitivity (69%18, 23%19) and PPV (7%20, 12%21, 52%18, and 41%19). One study14 searched keywords (e.g., “error,” “mistake” or “iatrogenic”) to detect medical errors in discharge summaries, residents’ transfer of service notes, and outpatient visit notes with PPVs ranging from 3.4% to 24.4%, depending on the keywords used. Apart from performance issues, one major limitation of a keyword searching method is that it only considers a limited set of terms, hence most of these tools are unable to automatically extract and encode important clinical information (such as medications and symptoms). This lack of detailed clinical information as a part of the structured output limits its usage in further analysis or other research purposes. Several studies6, 22 using NLP techniques have been conducted on analyzing allergy repositories. Skentzos et al22 developed NLP software to identify cholesterol lowering statin drug side effects documented in narrative provider notes and achieved a recall of 86.5% and precision of 91.9%. The Skentzos algorithm was further utilized to conduct a retrospective cohort study to determine the factors associated with documentation of statin side effects in a structured allergy repository23. Recently, a study by Epstein et al6 evaluated the use of NLP to encode free-text food and drug allergens and mapped identified allergens to RxNorm. Epstein’s study focused only on allergens within the patients allergy list and did not include reactions and environmental allergens. To date, no comprehensive NLP systems have been evaluated for processing allergy information in clinical notes with dynamic mapping to standard terminologies. Automated encoding of allergy information is challenging as it requires the integration of multiple standard terminologies24. Initial standard terminology recommendations for encoding allergy have been put forth by the health information standards panel (HISTP). These include, SNOMED CT25 for allergy/adverse reactions, RxNorm26 for medications, National Drug File Reference Terminology (NDR-RT)27 for drug classes and the Unique Ingredient Identifier28 for food and substance allergens29, 30. The National Council for Prescription Drug Programs (NCPDP)31 has proposed allergy value sets for encoding allergy that includes RxNorm as the source terminology. The Centers for Disease Control (CDC) has released a value set for encoding allergies that includes food, drug and environmental allergens, each mapped to SNOMED CT32. The coverage of five terminologies (RxNorm, SNOMED CT, UNII, NDF-RT and MedDRA) for encoding common allergies was recently evaluated by Goss et al who found SNOMED CT and RxNorm can satisfy most criteria for encoding common allergens with sufficient content coverage24. We have developed a generic NLP application, called Medical Text Extraction, Reasoning and Mapping System (MTERMS)33, which is initially used for encoding medication information for medication reconciliation and for mapping between medication terminologies33, 34. MTERMS processes free-text entries into a structured XML output using a pipeline approach that includes a Pre-Processor, Semantic Tagger, Terminology Mapper, Context Analyzer, and Parser. The Terminology Mapper of MTERMS maps free-text medical concepts to multiple standard terminologies (e.g., SNOMED CT, RxNorm, NDF-RT, etc.) instead of a single specific terminology and where necessary, establishes dynamic mapping between these terminologies. In this study, we extended MTERMS by adding an allergy module for processing free-text allergy information.
Methods and System Design System Design The NLP allergy module is built on the MTERMS platform, and consists of a new lexicon for allergy concepts, an updated context analyzer consisting of new disambiguation algorithms, and updates to the XML output schema (Figure 1). MTERMS provides dynamic mapping of concepts across each terminology in its lexicon.
581
Figure 1: MTERMS system architecture for allergy module.
Lexicon Development Our lexicon uses multiple recommended standard terminologies for encoding allergy concepts24, 30 as well as local terminologies. For drug allergens, we use RxNorm and the same method to create the medication concept database described by Zhou et al34. We compiled a list of medications typically used to treat allergic reactions (e.g., Benadryl, Solumedrol) from allergy experts. Drug classes are inherited from NDFRT (e.g., Ace Inhibitors), using those present in RxNorm and from ICD-10-CM “Allergy Status” terms (e.g., Antibiotics)35. For food and environmental allergen names, we use an available allergy value set from the CDC (Public Health Information Network Vocabulary Access and Distribution System (PHIN VADS)) as well as a subset of terms collected from the sub-tree (subClassOf) “allergen class” of SNOMED CT. Contrast media concepts were compiled from RxNorm and allergy experts. We use the 2014 versions of RxNorm, SNOMED CT, UNII, and the 2011 allergy value set from the CDC (Public Health Information Network Vocabulary Access and Distribution System (PHIN VADS)). To facilitate the integration of updates released from terminology sources, separate SQL tables were allocated to lexical variants or common misspellings. For reaction names, we use a subset of terms compiled from the most frequently observed allergy reactions within Partners Enterprise-wide Allergy Repository (PEAR), a longitudinal allergy database shared within the Partners provider/hospital network23. These reactions were then mapped to SNOMED CT. Search and Disambiguation Algorithms Drug, food, and environmental agents mentioned in the clinical text can have different meanings. A drug may be a treatment for a condition or environmental agent may be the setting a patient was surrounded by. Since allergens and associated reactions can be found anywhere in a clinical note, our algorithms conduct disambiguation by considering contextual information. For example, when inside of the allergies section, our algorithms search and encode all matches to our lexicon for allergens and reactions due to the context of the section. Outside of the allergy section, our algorithms identify allergens and reactions using a set of rules (regular expressions) that search for the presence of indicators (e.g., allergic, allergy, caused, “started after”, “likely due to”). If the pattern is recognized as being one indicating an allergy, then we add the annotations for the allergen and associated reactions. An example of a pattern outside of the allergies section is shown in Figure 2. Medications used as common treatments for allergic reactions and their symptoms (e.g., hydrocortisone, Pepcid) often show up in the same sentence as the drug allergen, and thus need to be disambiguated. We
582
use a set of rules based on indicators (e.g., resolved, treated, relieved) taking into account position of indicator with treatment drug, relative to the reaction and allergen. These rules are applicable both within and outside the Allergies section of the clinical note. Our current disambiguation algorithms are limited to the sentence level. We also compiled a subset of reaction terms or conditions that indicate an allergy (e.g., anaphylaxis, hives, Stevens-Johnson Syndrome, urticarial rash) to aid in disambiguation of allergic reactions from common reasons to visit the ED that are not necessarily immune-mediated (e.g., rash, itch). When any of these allergic reactions or conditions are found in the text, MTERMS will assert an allergy even if an allergen was not mentioned. Allergy Module Structured XML Output Design The allergy observation module was adapted from the HL7 allergy and intolerance working group model36 and used to develop an XML schema for representing some of the key elements within an allergy observation (e.g., type of allergy, allergen, reaction). A simplified example of our XML representation from the NLP tool is shown in Figure 3. Figure 2: XML representation of an allergy to Augmentin (the free-text input was mentioned in the Assessment section of an ED note) Free-text within Assessment section: “My impression is this is an allergic urticarial rash possibly from Augmentin” NLP output:
580
negation attribute of that medication to “is negated”. In MedLEE, terms such as “allergy” or “toxic” are captured and assigned to the semantic type “reaction”14. Melton et al15 used MedLEE16, 17 to identify 45 adverse event types (e.g., loss or impairment of bodily functions) using discharge summaries. A set of computerized queries implemented the inclusion and exclusion logic for each event and identified the adverse events against the NLP output. Melton’s approach achieved an overall sensitivity of 28%, specificity of 98.5%, and PPV of 45%. Most ADE detection algorithms use simple keyword search methods that search notes for relevant trigger words14. In general, these studies have reported low sensitivity (69%18, 23%19) and PPV (7%20, 12%21, 52%18, and 41%19). One study14 searched keywords (e.g., “error,” “mistake” or “iatrogenic”) to detect medical errors in discharge summaries, residents’ transfer of service notes, and outpatient visit notes with PPVs ranging from 3.4% to 24.4%, depending on the keywords used. Apart from performance issues, one major limitation of a keyword searching method is that it only considers a limited set of terms, hence most of these tools are unable to automatically extract and encode important clinical information (such as medications and symptoms). This lack of detailed clinical information as a part of the structured output limits its usage in further analysis or other research purposes. Several studies6, 22 using NLP techniques have been conducted on analyzing allergy repositories. Skentzos et al22 developed NLP software to identify cholesterol lowering statin drug side effects documented in narrative provider notes and achieved a recall of 86.5% and precision of 91.9%. The Skentzos algorithm was further utilized to conduct a retrospective cohort study to determine the factors associated with documentation of statin side effects in a structured allergy repository23. Recently, a study by Epstein et al6 evaluated the use of NLP to encode free-text food and drug allergens and mapped identified allergens to RxNorm. Epstein’s study focused only on allergens within the patients allergy list and did not include reactions and environmental allergens. To date, no comprehensive NLP systems have been evaluated for processing allergy information in clinical notes with dynamic mapping to standard terminologies. Automated encoding of allergy information is challenging as it requires the integration of multiple standard terminologies24. Initial standard terminology recommendations for encoding allergy have been put forth by the health information standards panel (HISTP). These include, SNOMED CT25 for allergy/adverse reactions, RxNorm26 for medications, National Drug File Reference Terminology (NDR-RT)27 for drug classes and the Unique Ingredient Identifier28 for food and substance allergens29, 30. The National Council for Prescription Drug Programs (NCPDP)31 has proposed allergy value sets for encoding allergy that includes RxNorm as the source terminology. The Centers for Disease Control (CDC) has released a value set for encoding allergies that includes food, drug and environmental allergens, each mapped to SNOMED CT32. The coverage of five terminologies (RxNorm, SNOMED CT, UNII, NDF-RT and MedDRA) for encoding common allergies was recently evaluated by Goss et al who found SNOMED CT and RxNorm can satisfy most criteria for encoding common allergens with sufficient content coverage24. We have developed a generic NLP application, called Medical Text Extraction, Reasoning and Mapping System (MTERMS)33, which is initially used for encoding medication information for medication reconciliation and for mapping between medication terminologies33, 34. MTERMS processes free-text entries into a structured XML output using a pipeline approach that includes a Pre-Processor, Semantic Tagger, Terminology Mapper, Context Analyzer, and Parser. The Terminology Mapper of MTERMS maps free-text medical concepts to multiple standard terminologies (e.g., SNOMED CT, RxNorm, NDF-RT, etc.) instead of a single specific terminology and where necessary, establishes dynamic mapping between these terminologies. In this study, we extended MTERMS by adding an allergy module for processing free-text allergy information.
Methods and System Design System Design The NLP allergy module is built on the MTERMS platform, and consists of a new lexicon for allergy concepts, an updated context analyzer consisting of new disambiguation algorithms, and updates to the XML output schema (Figure 1). MTERMS provides dynamic mapping of concepts across each terminology in its lexicon.
581
Figure 1: MTERMS system architecture for allergy module.
Lexicon Development Our lexicon uses multiple recommended standard terminologies for encoding allergy concepts24, 30 as well as local terminologies. For drug allergens, we use RxNorm and the same method to create the medication concept database described by Zhou et al34. We compiled a list of medications typically used to treat allergic reactions (e.g., Benadryl, Solumedrol) from allergy experts. Drug classes are inherited from NDFRT (e.g., Ace Inhibitors), using those present in RxNorm and from ICD-10-CM “Allergy Status” terms (e.g., Antibiotics)35. For food and environmental allergen names, we use an available allergy value set from the CDC (Public Health Information Network Vocabulary Access and Distribution System (PHIN VADS)) as well as a subset of terms collected from the sub-tree (subClassOf) “allergen class” of SNOMED CT. Contrast media concepts were compiled from RxNorm and allergy experts. We use the 2014 versions of RxNorm, SNOMED CT, UNII, and the 2011 allergy value set from the CDC (Public Health Information Network Vocabulary Access and Distribution System (PHIN VADS)). To facilitate the integration of updates released from terminology sources, separate SQL tables were allocated to lexical variants or common misspellings. For reaction names, we use a subset of terms compiled from the most frequently observed allergy reactions within Partners Enterprise-wide Allergy Repository (PEAR), a longitudinal allergy database shared within the Partners provider/hospital network23. These reactions were then mapped to SNOMED CT. Search and Disambiguation Algorithms Drug, food, and environmental agents mentioned in the clinical text can have different meanings. A drug may be a treatment for a condition or environmental agent may be the setting a patient was surrounded by. Since allergens and associated reactions can be found anywhere in a clinical note, our algorithms conduct disambiguation by considering contextual information. For example, when inside of the allergies section, our algorithms search and encode all matches to our lexicon for allergens and reactions due to the context of the section. Outside of the allergy section, our algorithms identify allergens and reactions using a set of rules (regular expressions) that search for the presence of indicators (e.g., allergic, allergy, caused, “started after”, “likely due to”). If the pattern is recognized as being one indicating an allergy, then we add the annotations for the allergen and associated reactions. An example of a pattern outside of the allergies section is shown in Figure 2. Medications used as common treatments for allergic reactions and their symptoms (e.g., hydrocortisone, Pepcid) often show up in the same sentence as the drug allergen, and thus need to be disambiguated. We
582
use a set of rules based on indicators (e.g., resolved, treated, relieved) taking into account position of indicator with treatment drug, relative to the reaction and allergen. These rules are applicable both within and outside the Allergies section of the clinical note. Our current disambiguation algorithms are limited to the sentence level. We also compiled a subset of reaction terms or conditions that indicate an allergy (e.g., anaphylaxis, hives, Stevens-Johnson Syndrome, urticarial rash) to aid in disambiguation of allergic reactions from common reasons to visit the ED that are not necessarily immune-mediated (e.g., rash, itch). When any of these allergic reactions or conditions are found in the text, MTERMS will assert an allergy even if an allergen was not mentioned. Allergy Module Structured XML Output Design The allergy observation module was adapted from the HL7 allergy and intolerance working group model36 and used to develop an XML schema for representing some of the key elements within an allergy observation (e.g., type of allergy, allergen, reaction). A simplified example of our XML representation from the NLP tool is shown in Figure 3. Figure 2: XML representation of an allergy to Augmentin (the free-text input was mentioned in the Assessment section of an ED note) Free-text within Assessment section: “My impression is this is an allergic urticarial rash possibly from Augmentin” NLP output:
