Epilepsy & Behavior 29 (2013) 578–580
Contents lists available at ScienceDirect
Epilepsy & Behavior journal homepage: www.elsevier.com/locate/yebeh
Brief Communication
Validating a natural language processing tool to exclude psychogenic nonepileptic seizures in electronic medical record-based epilepsy research H. Hamid a,b,⁎, S.J. Fodeh a,b, A.G. Lizama a,b, R. Czlapinski b, M.J. Pugh c, W.C. LaFrance Jr. d, C.A. Brandt a,b a
Connecticut VA Healthcare System, USA Yale University, USA c University of Texas Health Science Center at San Antonio, San Antonio VA Hospital, USA d Brown University, Providence VA Hospital, USA b
a r t i c l e
i n f o
Article history: Received 18 July 2013 Revised 15 September 2013 Accepted 16 September 2013 Available online 14 October 2013 Keywords: Psychogenic nonepileptic seizures Epidemiology Electronic health record Natural language processing
a b s t r a c t Rationale: As electronic health record (EHR) systems become more available, they will serve as an important resource for collecting epidemiologic data in epilepsy research. However, since clinicians do not have a systematic method for coding psychogenic nonepileptic seizures (PNES), patients with PNES are often misclassified as having epilepsy, leading to sampling error. This study validates a natural language processing (NLP) tool that uses linguistic information to help identify patients with PNES. Methods: Using the VA national clinical database, 2200 notes of Iraq and Afghanistan veterans who completed video electroencephalograph (VEEG) monitoring were reviewed manually, and the veterans were identified as having documented PNES or not. Reviewers identified PNES-related vocabulary to inform a NLP tool called Yale cTakes Extension (YTEX). Using NLP techniques, YTEX annotates syntactic constructs, named entities, and their negation context in the EHR. These annotations are passed to a classifier to detect patients without PNES. The classifier was evaluated by calculating positive predictive values (PPVs), sensitivity, and F-score. Results: Of the 742 Iraq and Afghanistan veterans who received a diagnosis of epilepsy or seizure disorder by VEEG, 44 had documented events on VEEG: 22 veterans (3.0%) had definite PNES only, 20 (2.7%) had probable PNES, and 2 (0.3%) had both PNES and epilepsy documented. The remaining 698 veterans did not have events captured during the VEEG admission and/or did not have a definitive diagnosis. Our classifier achieved a PPV of 93%, a sensitivity of 99%, and a F-score of 96%. Conclusion: Our study demonstrates that the YTEX NLP tool and classifier is highly accurate in excluding PNES, diagnosed with VEEG, in EHR systems. The tool may be very valuable in preventing false positive identification of patients with epilepsy in EHR-based epidemiologic research. © 2013 Published by Elsevier Inc.
1. Introduction As academic programs develop comprehensive electronic health record (EHR) systems, opportunities to conduct large scale, cost-effective epidemiologic studies grow. Case identification relies on the International Classification of Disease—9th Edition (ICD-9) to identify cases of epilepsy. However, since practitioners code for psychogenic nonepileptic seizure (PNES) in a nonsystematic manner using a variety of codes, EHRbased epilepsy studies may suffer from significant misclassification errors, compromising epidemiologic studies with sample bias. In a single hospital sample, up to 29% of veterans admitted for video electroencephalogram (VEEG) monitoring are diagnosed with PNES [1]. We
⁎ Corresponding author at: 950 Campbell Avenue West Haven, CT 06516, USA. Fax: +1 203 937 3464. E-mail address: [email protected] (H. Hamid). 1525-5050/$ – see front matter © 2013 Published by Elsevier Inc. http://dx.doi.org/10.1016/j.yebeh.2013.09.025
hypothesize that a substantial number of those identified with epilepsy using algorithms validated for use in electronic databases will be misclassified. This study aims to exclude patients with PNES using natural language processing (NLP) tools to extract clinical notes from an EHR. Natural language processing (NLP) tools allow for large volume EHRs and chart reviews that may be both time-saving and costeffective. Recently, NLP tools have been developed to identify patients with heart conditions [2], those with bone fractures [3], those who experience falls [4], and those who undergo treatment for posttraumatic stress disorder [5]. 2. Methods The Veterans Administration Central Institutional Review Board approved this research protocol. The study data were accessed using the VA Informatics and Computing Infrastructure (VINCI), which is a secure
H. Hamid et al. / Epilepsy & Behavior 29 (2013) 578–580
computing environment available for VA researchers to access national clinical data. Operation Enduring Freedom/Operation Iraqi Freedom/ Operation New Dawn (OIF/OEF/OND) veterans who had completed video-electroencephalogram (VEEG) monitoring (CPT codes 95956, 95950, 95951, 95953, 95057, 95827) from 2000 to 2010 were included in this analysis. All clinical progress notes with “neurology” in the note title one month before or after the date of VEEG were retrieved for chart review. Two reviewers were trained (by HH) to perform chart reviews, and then 100 charts with known epilepsy and PNES diagnostic status were reviewed by each trained reviewer to assess interrater agreement with the medical record diagnosis of PNES. The kappa statistic was equivalent to 0.80 among both reviewers. The reviewers then manually classified each note into the following: “Definite PNES” when an event was captured by VEEG and described in the clinical note as PNES or when the note described that the patient had a history of PNES; “Probable PNES” when the event captured was not typical of the events the patient had at home and the VEEG showed no epileptiform activity; “Definite Epilepsy and PNES” when two events were captured and one was clearly epileptic and the other nonepileptic by VEEG; “Indefinite/Unknown” when no event was captured; and “Epilepsy Only” when only epileptic seizures were captured, based on documentation within the clinical notes. Of the 2200 notes reviewed, 1199 were excluded because they had no relevant information regarding evaluation of seizures. Of the 1101 notes that were retained, 100 notes with definite PNES were compared to 1001 nonPNES charts. Reviewers coded PNES-related vocabulary, including “psychogenic non-epileptic”, “psychogenic seizures”, “non-epileptic”, “pseudoseizures”, “NES”, and “PNES” in order to help develop the NLP rules and classifier. The reviewers also identified the location of PNES-related vocabulary by section of the note, including “History of Present Illness/Subjective”, “Past Medical History”, “Impression”, “Assessment”, and “Plan”. We applied the Yale cTakes Extension (YTEX) and a machinelearning classifier to identify charts of patients that had no PNES diagnoses [6]. Yale cTakes Extension is a natural language processing tool which uses a modular pipeline of unstructured information management application (UIMA) annotators to annotate syntactic constructs [7], named entities, and their negation context in clinical text. Specifically, it conducts sentence detection, tokenizing, part-of-speech tagging and stemming, and uses an algorithm based on the simple regular expression algorithm (NegEx) for negation detection. It also includes a DictionaryLookup module that performs named entity recognition by matching spans of text to entries from the Unified Medical Language System (UMLS) Metathesaurus [8]. The UMLS Metathesaurus is comprised of over 100 source vocabularies and assigns each vocabulary with a concept unique identifier (CUI). We added additional entries to the YTEX dictionary that define PNES-specific concepts from terms identified by chart review. The output of YTEX is the annotations from the pipeline. Yale cTakes Extension output is exported as a “bag of concepts” in which each chart or clinical note is represented by a collection of concepts. We examined each document to account for the existence of PNES vocabularies. We used the negated concepts as produced by YTEX to collect evidences of the absence of PNES in a clinical note, which we labeled as non-PNES. We built a classification model based on a naïve Bayes classifier to distinguish non-PNES charts. A 10-fold cross validation was applied to validate the classifier where data are divided into 10 equally sized partitions, and then the classifier is trained 10 times, with validation each time on one-fold and tested on the remaining 9-fold. The average performance of the classifier is reported. We evaluated the classifier by calculating positive predictive values (PPVs), sensitivity, and F-score, which is a measure of a test's accuracy that combines precision and recall. The PPV (or precision rate) is the proportion of positive test results that are true positives, and sensitivity (or recall) measures the proportion of true positives (people having epilepsy and not PNES) who are correctly identified.
579
3. Results Of the 2802 OIF/OEF/OND veterans who received a diagnosis of epilepsy or seizure disorder (ICD-9 345.X), 742 veterans had VEEGdocumented events; 22 veterans (3.0%) had definite PNES only, 20 (2.3%) had probable PNES, 5 (0.7%) had definite epilepsy, 2 (0.3%) had both PNES and epilepsy documented, and the remainder did not have their event definitively characterized. Our classifier, developed to distinguish in a clinical note people with probable or definite PNES, achieved a PPV of 93% and a sensitivity of 99%. The non-PNES charts were identified as having an F-score of 96%.
4. Discussion Our study demonstrates that NLP is highly accurate in excluding patients diagnosed with PNES from EHR databases. The classifier may significantly reduce false positive diagnoses of epilepsy by excluding identifying notes of patients with PNES. For example, in this study, 7.7% who carried the diagnosis of epilepsy or seizure disorder actually have PNES. As many as 80,000 veterans are estimated to carry the diagnosis of epilepsy or seizure disorder; therefore, EHR-based research may misclassify as many as 6160 veterans with PNES nationally. While this study used the VA national EHR database given that all VA neurologists who conduct VEEG also have academic affiliations, the classifier may be applicable in academic hospital samples. We chose to sample OIF/OEF/OND veterans because of the suspected high rates of PNES in this population as well as our technical experience working with their EHR. The majority of OIF/OEF/OND veterans are younger and likely would not have been eligible to serve in the military if they had carried a diagnosis of epilepsy during their teenage years. Therefore, the low rate of epilepsy in this study is not surprising. While this study aimed to validate a NLP tool to exclude PNES, not to characterize veterans with PNES, we suspect that the low frequency of seizures and low average length of VEEG monitoring (which had an average of 4.3 days in veterans' health-care epilepsy centers nationally [9]) led to the high rate of indefinite VEEG studies. The primary limitation of this tool is that it requires VEEG monitoring to exclude patients with PNES. In order to maximize the sensitivity of the NLP tool (and minimize false negative classification of PNES), we chose this stringent criterion because VEEG is the gold standard for distinction between epileptic seizures and PNES. There is a potential selection bias in only including patients admitted for VEEG monitoring because they are more likely to have poorly controlled seizures. People may suffer from PNES but have very infrequent seizures and, therefore, may not be admitted for VEEG or have VEEG that fails to capture an event. Consequently, we included patients who were classified under “Probable PNES” and carried a history of PNES to maximize the sensitivity of the classifier. In conclusion, this study validates a NLP tool used to exclude patients with PNES, who have completed VEEG monitoring, in an EHR. The classifier tool can be critical in reducing false positive diagnoses of epilepsy and seizure disorders with patients who actually have PNES. Furthermore, as epilepsy centers, such as the VA Epilepsy Centers of Excellence, continue to develop, diagnostic accuracy and documentation of PNES are expected to improve. This classifier is valuable in conducting future EHR-based epidemiologic research in epilepsy and seizure disorders, outlined as a priority by the Institute of Medicine in 2012 [10].
Statements of ethics There are no conflicts of interest among any of the coauthors. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.
580
H. Hamid et al. / Epilepsy & Behavior 29 (2013) 578–580
References [1] Salinsky M, Spencer D, Boudreau E, Ferguson F. Psychogenic nonepileptic seizures in US veterans. Neurology 2011;77(10):945–50. [2] Garvin JH, DuVall SL, South BR, Bray BE, Bolton D, Heavirland J, et al. Automated extraction of ejection fraction for quality measurement using regular expressions in Unstructured Information Management Architecture (UIMA) for heart failure. J Am Med Inform Assoc 2012;19(5):859–66. [3] Womack JA, Scotch M, Gibert C, Chapman W, Yin M, Justice AC, et al. A comparison of two approaches to text processing: facilitating chart reviews of radiology reports in electronic medical records. Perspect Health Inf Manag 2010;7:1a. [4] McCart JA, Berndt DJ, Jarman J, Finch DK, Luther SL. Finding falls in ambulatory care clinical documents using statistical text mining. J Am Med Inform Assoc 2012.
[5] Shiner B, D'Avolio LW, Nguyen TM, Zayed MH, Watts BV, Fiore L. Automated classification of psychotherapy note text: implications for quality assessment in PTSD care. J Eval Clin Pract 2012;18(3):698–701. [6] Garla V, Lo Re III V, Dorey-Stein Z, Kidwai F, Scotch M, Womack J, et al. The Yale cTAKES extensions for document classification: architecture and application. J Am Med Inform Assoc 2011;18(5):614–20. [7] Unstructured Information Management Applications. http://uima.apache.org/ documentation.html (last accessed 10/03/13). [8] UMLS® Reference Manual [Internet]. Bethesda (MD): National Library of Medicine (US); 2009 Sep-. Preface. Available from: http://www.ncbi.nlm.nih.gov/books/NBK9681/. [9] Department of Veterans Affairs. Epilepsy Centers of Excellence 2012 annual report; 2012 [Washington D.C.]. [10] England MJ, Liverman CT, Schultz AM, Strawbridge LM. Epilepsy across the spectrum: promoting health and understanding. editors. Epilepsy Behav. 2012;25(2):266–76. http://dx.doi.org/10.1016/j.yebeh.2012.06.016 [Epub 2012 Oct. 5].
Contents lists available at ScienceDirect
Epilepsy & Behavior journal homepage: www.elsevier.com/locate/yebeh
Brief Communication
Validating a natural language processing tool to exclude psychogenic nonepileptic seizures in electronic medical record-based epilepsy research H. Hamid a,b,⁎, S.J. Fodeh a,b, A.G. Lizama a,b, R. Czlapinski b, M.J. Pugh c, W.C. LaFrance Jr. d, C.A. Brandt a,b a
Connecticut VA Healthcare System, USA Yale University, USA c University of Texas Health Science Center at San Antonio, San Antonio VA Hospital, USA d Brown University, Providence VA Hospital, USA b
a r t i c l e
i n f o
Article history: Received 18 July 2013 Revised 15 September 2013 Accepted 16 September 2013 Available online 14 October 2013 Keywords: Psychogenic nonepileptic seizures Epidemiology Electronic health record Natural language processing
a b s t r a c t Rationale: As electronic health record (EHR) systems become more available, they will serve as an important resource for collecting epidemiologic data in epilepsy research. However, since clinicians do not have a systematic method for coding psychogenic nonepileptic seizures (PNES), patients with PNES are often misclassified as having epilepsy, leading to sampling error. This study validates a natural language processing (NLP) tool that uses linguistic information to help identify patients with PNES. Methods: Using the VA national clinical database, 2200 notes of Iraq and Afghanistan veterans who completed video electroencephalograph (VEEG) monitoring were reviewed manually, and the veterans were identified as having documented PNES or not. Reviewers identified PNES-related vocabulary to inform a NLP tool called Yale cTakes Extension (YTEX). Using NLP techniques, YTEX annotates syntactic constructs, named entities, and their negation context in the EHR. These annotations are passed to a classifier to detect patients without PNES. The classifier was evaluated by calculating positive predictive values (PPVs), sensitivity, and F-score. Results: Of the 742 Iraq and Afghanistan veterans who received a diagnosis of epilepsy or seizure disorder by VEEG, 44 had documented events on VEEG: 22 veterans (3.0%) had definite PNES only, 20 (2.7%) had probable PNES, and 2 (0.3%) had both PNES and epilepsy documented. The remaining 698 veterans did not have events captured during the VEEG admission and/or did not have a definitive diagnosis. Our classifier achieved a PPV of 93%, a sensitivity of 99%, and a F-score of 96%. Conclusion: Our study demonstrates that the YTEX NLP tool and classifier is highly accurate in excluding PNES, diagnosed with VEEG, in EHR systems. The tool may be very valuable in preventing false positive identification of patients with epilepsy in EHR-based epidemiologic research. © 2013 Published by Elsevier Inc.
1. Introduction As academic programs develop comprehensive electronic health record (EHR) systems, opportunities to conduct large scale, cost-effective epidemiologic studies grow. Case identification relies on the International Classification of Disease—9th Edition (ICD-9) to identify cases of epilepsy. However, since practitioners code for psychogenic nonepileptic seizure (PNES) in a nonsystematic manner using a variety of codes, EHRbased epilepsy studies may suffer from significant misclassification errors, compromising epidemiologic studies with sample bias. In a single hospital sample, up to 29% of veterans admitted for video electroencephalogram (VEEG) monitoring are diagnosed with PNES [1]. We
⁎ Corresponding author at: 950 Campbell Avenue West Haven, CT 06516, USA. Fax: +1 203 937 3464. E-mail address: [email protected] (H. Hamid). 1525-5050/$ – see front matter © 2013 Published by Elsevier Inc. http://dx.doi.org/10.1016/j.yebeh.2013.09.025
hypothesize that a substantial number of those identified with epilepsy using algorithms validated for use in electronic databases will be misclassified. This study aims to exclude patients with PNES using natural language processing (NLP) tools to extract clinical notes from an EHR. Natural language processing (NLP) tools allow for large volume EHRs and chart reviews that may be both time-saving and costeffective. Recently, NLP tools have been developed to identify patients with heart conditions [2], those with bone fractures [3], those who experience falls [4], and those who undergo treatment for posttraumatic stress disorder [5]. 2. Methods The Veterans Administration Central Institutional Review Board approved this research protocol. The study data were accessed using the VA Informatics and Computing Infrastructure (VINCI), which is a secure
H. Hamid et al. / Epilepsy & Behavior 29 (2013) 578–580
computing environment available for VA researchers to access national clinical data. Operation Enduring Freedom/Operation Iraqi Freedom/ Operation New Dawn (OIF/OEF/OND) veterans who had completed video-electroencephalogram (VEEG) monitoring (CPT codes 95956, 95950, 95951, 95953, 95057, 95827) from 2000 to 2010 were included in this analysis. All clinical progress notes with “neurology” in the note title one month before or after the date of VEEG were retrieved for chart review. Two reviewers were trained (by HH) to perform chart reviews, and then 100 charts with known epilepsy and PNES diagnostic status were reviewed by each trained reviewer to assess interrater agreement with the medical record diagnosis of PNES. The kappa statistic was equivalent to 0.80 among both reviewers. The reviewers then manually classified each note into the following: “Definite PNES” when an event was captured by VEEG and described in the clinical note as PNES or when the note described that the patient had a history of PNES; “Probable PNES” when the event captured was not typical of the events the patient had at home and the VEEG showed no epileptiform activity; “Definite Epilepsy and PNES” when two events were captured and one was clearly epileptic and the other nonepileptic by VEEG; “Indefinite/Unknown” when no event was captured; and “Epilepsy Only” when only epileptic seizures were captured, based on documentation within the clinical notes. Of the 2200 notes reviewed, 1199 were excluded because they had no relevant information regarding evaluation of seizures. Of the 1101 notes that were retained, 100 notes with definite PNES were compared to 1001 nonPNES charts. Reviewers coded PNES-related vocabulary, including “psychogenic non-epileptic”, “psychogenic seizures”, “non-epileptic”, “pseudoseizures”, “NES”, and “PNES” in order to help develop the NLP rules and classifier. The reviewers also identified the location of PNES-related vocabulary by section of the note, including “History of Present Illness/Subjective”, “Past Medical History”, “Impression”, “Assessment”, and “Plan”. We applied the Yale cTakes Extension (YTEX) and a machinelearning classifier to identify charts of patients that had no PNES diagnoses [6]. Yale cTakes Extension is a natural language processing tool which uses a modular pipeline of unstructured information management application (UIMA) annotators to annotate syntactic constructs [7], named entities, and their negation context in clinical text. Specifically, it conducts sentence detection, tokenizing, part-of-speech tagging and stemming, and uses an algorithm based on the simple regular expression algorithm (NegEx) for negation detection. It also includes a DictionaryLookup module that performs named entity recognition by matching spans of text to entries from the Unified Medical Language System (UMLS) Metathesaurus [8]. The UMLS Metathesaurus is comprised of over 100 source vocabularies and assigns each vocabulary with a concept unique identifier (CUI). We added additional entries to the YTEX dictionary that define PNES-specific concepts from terms identified by chart review. The output of YTEX is the annotations from the pipeline. Yale cTakes Extension output is exported as a “bag of concepts” in which each chart or clinical note is represented by a collection of concepts. We examined each document to account for the existence of PNES vocabularies. We used the negated concepts as produced by YTEX to collect evidences of the absence of PNES in a clinical note, which we labeled as non-PNES. We built a classification model based on a naïve Bayes classifier to distinguish non-PNES charts. A 10-fold cross validation was applied to validate the classifier where data are divided into 10 equally sized partitions, and then the classifier is trained 10 times, with validation each time on one-fold and tested on the remaining 9-fold. The average performance of the classifier is reported. We evaluated the classifier by calculating positive predictive values (PPVs), sensitivity, and F-score, which is a measure of a test's accuracy that combines precision and recall. The PPV (or precision rate) is the proportion of positive test results that are true positives, and sensitivity (or recall) measures the proportion of true positives (people having epilepsy and not PNES) who are correctly identified.
579
3. Results Of the 2802 OIF/OEF/OND veterans who received a diagnosis of epilepsy or seizure disorder (ICD-9 345.X), 742 veterans had VEEGdocumented events; 22 veterans (3.0%) had definite PNES only, 20 (2.3%) had probable PNES, 5 (0.7%) had definite epilepsy, 2 (0.3%) had both PNES and epilepsy documented, and the remainder did not have their event definitively characterized. Our classifier, developed to distinguish in a clinical note people with probable or definite PNES, achieved a PPV of 93% and a sensitivity of 99%. The non-PNES charts were identified as having an F-score of 96%.
4. Discussion Our study demonstrates that NLP is highly accurate in excluding patients diagnosed with PNES from EHR databases. The classifier may significantly reduce false positive diagnoses of epilepsy by excluding identifying notes of patients with PNES. For example, in this study, 7.7% who carried the diagnosis of epilepsy or seizure disorder actually have PNES. As many as 80,000 veterans are estimated to carry the diagnosis of epilepsy or seizure disorder; therefore, EHR-based research may misclassify as many as 6160 veterans with PNES nationally. While this study used the VA national EHR database given that all VA neurologists who conduct VEEG also have academic affiliations, the classifier may be applicable in academic hospital samples. We chose to sample OIF/OEF/OND veterans because of the suspected high rates of PNES in this population as well as our technical experience working with their EHR. The majority of OIF/OEF/OND veterans are younger and likely would not have been eligible to serve in the military if they had carried a diagnosis of epilepsy during their teenage years. Therefore, the low rate of epilepsy in this study is not surprising. While this study aimed to validate a NLP tool to exclude PNES, not to characterize veterans with PNES, we suspect that the low frequency of seizures and low average length of VEEG monitoring (which had an average of 4.3 days in veterans' health-care epilepsy centers nationally [9]) led to the high rate of indefinite VEEG studies. The primary limitation of this tool is that it requires VEEG monitoring to exclude patients with PNES. In order to maximize the sensitivity of the NLP tool (and minimize false negative classification of PNES), we chose this stringent criterion because VEEG is the gold standard for distinction between epileptic seizures and PNES. There is a potential selection bias in only including patients admitted for VEEG monitoring because they are more likely to have poorly controlled seizures. People may suffer from PNES but have very infrequent seizures and, therefore, may not be admitted for VEEG or have VEEG that fails to capture an event. Consequently, we included patients who were classified under “Probable PNES” and carried a history of PNES to maximize the sensitivity of the classifier. In conclusion, this study validates a NLP tool used to exclude patients with PNES, who have completed VEEG monitoring, in an EHR. The classifier tool can be critical in reducing false positive diagnoses of epilepsy and seizure disorders with patients who actually have PNES. Furthermore, as epilepsy centers, such as the VA Epilepsy Centers of Excellence, continue to develop, diagnostic accuracy and documentation of PNES are expected to improve. This classifier is valuable in conducting future EHR-based epidemiologic research in epilepsy and seizure disorders, outlined as a priority by the Institute of Medicine in 2012 [10].
Statements of ethics There are no conflicts of interest among any of the coauthors. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.
580
H. Hamid et al. / Epilepsy & Behavior 29 (2013) 578–580
References [1] Salinsky M, Spencer D, Boudreau E, Ferguson F. Psychogenic nonepileptic seizures in US veterans. Neurology 2011;77(10):945–50. [2] Garvin JH, DuVall SL, South BR, Bray BE, Bolton D, Heavirland J, et al. Automated extraction of ejection fraction for quality measurement using regular expressions in Unstructured Information Management Architecture (UIMA) for heart failure. J Am Med Inform Assoc 2012;19(5):859–66. [3] Womack JA, Scotch M, Gibert C, Chapman W, Yin M, Justice AC, et al. A comparison of two approaches to text processing: facilitating chart reviews of radiology reports in electronic medical records. Perspect Health Inf Manag 2010;7:1a. [4] McCart JA, Berndt DJ, Jarman J, Finch DK, Luther SL. Finding falls in ambulatory care clinical documents using statistical text mining. J Am Med Inform Assoc 2012.
[5] Shiner B, D'Avolio LW, Nguyen TM, Zayed MH, Watts BV, Fiore L. Automated classification of psychotherapy note text: implications for quality assessment in PTSD care. J Eval Clin Pract 2012;18(3):698–701. [6] Garla V, Lo Re III V, Dorey-Stein Z, Kidwai F, Scotch M, Womack J, et al. The Yale cTAKES extensions for document classification: architecture and application. J Am Med Inform Assoc 2011;18(5):614–20. [7] Unstructured Information Management Applications. http://uima.apache.org/ documentation.html (last accessed 10/03/13). [8] UMLS® Reference Manual [Internet]. Bethesda (MD): National Library of Medicine (US); 2009 Sep-. Preface. Available from: http://www.ncbi.nlm.nih.gov/books/NBK9681/. [9] Department of Veterans Affairs. Epilepsy Centers of Excellence 2012 annual report; 2012 [Washington D.C.]. [10] England MJ, Liverman CT, Schultz AM, Strawbridge LM. Epilepsy across the spectrum: promoting health and understanding. editors. Epilepsy Behav. 2012;25(2):266–76. http://dx.doi.org/10.1016/j.yebeh.2012.06.016 [Epub 2012 Oct. 5].
