Electronic health record systems: risks and benefits.

21 e1

February 2014 Volume 21 Issue e1

Volume 21 Issue e1 Pages e1–e179 A SCHOLARLY JOURNAL OF INFORMATICS IN HEALTH AND BIOMEDICINE

Editor-in-Chief Professor Lucila Ohno-Machado The impact of electronic health records on people with diabetes in three different emergency departments S M Speedie, Y-T Park, J Du, N Theera-Ampornpunt, B A Bershow, R A Gensinger Jr, D T Routhe, D P Connelly

JAMIA

Development and use of active clinical decision support for preemptive pharmacogenomics G C Bell, K R Crews, M R Wilkinson, C E Haidar, J K Hicks, D K Baker, N M Kornegay, W Yang, S J Cross, S C Howard, R R Freimuth, W E Evans, U Broeckel, M V Relling, J M Hoffman A novel clinician interface to improve clinician access to up-to-date genetic results A R Wilcox, P M Neri, L A Volk, L P Newmark, E H Clark, L J Babb, M Varugheese, S J Aronson, H L Rehm, D W Bates A taste of individualized medicine: physicians’ reactions to automated genetic interpretations H Lærum, S Bremer, S Bergan, T Grünfeld

February 2014

www.jamia.org

amiajnl_21_E1_Cover.indd 1

15/01/14 6:13 PM

Editor-in-Chief Lucila Ohno-Machado, MD, PhD

Editorial Board

San Diego, CA

2011–2013

Editorial Assistant Marissa Wolff

Suzanne Bakken, RN, DNSc New York, New York Douglas Bell, MD, PhD Los Angeles, California Olivier Bodenreider, MD, PhD Bethesda, Maryland R Scott Evans, MS, PhD Salt Lake City, Utah George Hripcsak, MD, MS New York, New York Michael Kahn, MD, PhD Denver, Colorado Leslie Lenert, MD, MS Salt Lake City, Utah Henry Lowe, MD Palo Alto, California Kenneth Mandl, MD, MPH Boston, Massachusetts Philip Payne, PhD Columbus, Ohio Alan Rector, PhD Manchester, United Kingdom Neil Sarkar, PhD, MLS Burlington, Vermont Michael Shabot, MD Houston, Texas Steven Shea, MD, MS New York, New York Catherine Staes, PhD Salt Lake City, Utah Jaap Suermondt, PhD Palo Alto, California Martin Were, MD, PhD Indianapolis, Indiana

Apex, North Carolina

2012–2014

Associate Editors Patricia Flatley Brennan, PhD, RN Madison, Wisconsin

Atul Butte, MD, PhD Palo Alto, California

Enrico Coiera, PhD Sydney, Australia

Charles Friedman, PhD Washington, DC

Betsy Humphreys, MLS Bethesda, Maryland

Kevin Johnson, MD, MS Nashville, Tennessee

Harold Lehmann, MD, PhD Baltimore, Maryland

Prakash Nadkarni, MD Danville, Pennsylvania

Assistant Editor Michael F Chiang, MD, MA Portland, Oregon

Guidelines for Author and Reviewers Full instructions are available online at http://jamia.bmj.com/ifora. Articles must be submitted electronically via http:// submit-jamia.bmj.com. Authors retain copyright but are required to grant JAMIA an exclusive licence to publish (http:// jamia.bmj.com/ifora/ licence.dtl). Disclaimer: No responsibility is assumed by the Publisher or by the American Medical Informatics Association for any injury and/or damage to persons or property as a result of any actual or alleged libelous statements, infringement of intellectual property or privacy rights, or products liability, whether resulting from negligence or otherwise, or from any use or operation of any ideas, instructions, procedures, products, or methods contained in this issue of the Journal of the American Medical Informatics Association. The publication of an advertisement herein does not constitute on the part of the Publisher or the American Medical Informatics Association a guarantee or endorsement of the quality or value of the advertised products or services described therein or of any of the representations or the claims made by the advertisers with respect to such products or services.

Michael J Ackerman, PhD Besthesda, Maryland James G Anderson, PhD West Lafayette, Indiana Dominik Aronsky, MD, PhD Zurich, Switzerland Joan Ash, PhD, MLS, MBA Portland, Oregon Michael J Becich, MD, PhD Pittsburgh, Pennsylvania Ricardo Bellazzi, PhD Pavia, Italy Christopher Chute, MD, DrPH Rochester, Minnesota Carol Friedman, PhD New York, New York

Mark E Frisse, MD, MBA Nashville, Tennessee John Holmes, PhD Philadelphia, Pennsylvania Ju Han Kim, MD, PhD Seoul, South Korea Michio Kimura, MD, PhD Shizuoka, Japan Yu-Chuan (Jack) Li, MD, PhD Taipei, Taiwan Christian Lovis, MD, MPH Geneva, Switzerland Victor Maojo, PhD Madrid, Spain Stuart Speedie, PhD Minneapolis, Minnesota Peter Tarczy-Hornoch, MD Seattle, Washington Jeremy C Wyatt, PhD Coventry, UK

2013–2015 Julia Adler-Milstein, PhD Ann Arbor, Michigan Russ B Altman, MD, PhD Palo Alto, California Eta S Berner, EdD Birmingham, Alabama Cynthia A Brandt, MD, MPH New Haven, Connecticut Gary D Byrd, PhD Buffalo, New York Dina Demner-Fushman, PhD Bethesda, Maryland Stephan Dreiseitl, PhD Hagenberg, Austria Stephen Johnson, PhD New York, New York Gilad J Kuperman, MD, PhD New York, New York Yves Lussier, MD Chicago, Illinois Brad Malin, PhD Nashville, Tennessee K Ann McKibbon, MLS, PhD Ontario, Canada Thomas H Payne, MD Seattle, Washington Eric Poon, MD, MPH Boston, Massachusetts Rachel Richesson, PhD, MPH Tampa, Florida W John Wilbur, MD, PhD Bethesda, Maryland Hua Xu, PhD Houston, Texas

Student Editorial Board 2012–2013 Robert J Carroll, MS Nashville, Tennessee Jeffrey Duncan, MS Salt Lake City, Utah Zach Landis-Lewis, MS, MLIS Pittburgh, Pennsylvania Mei-Sing Ong, PhD Sydney, Australia Hojjat Salmasian, MD, MPH New York, New York

2013-2014 Ravi Atreya Nashville, Tennessee Robin Chisholm Indianapolis, Indiana Paul DeMuro Portland, Oregon Jason Jacobs Salt Lake City, Utah Xiaoqian Jiang San Diego, California Thai Le Seattle, Washington Maxim Topaz

Philadelphia, Pennsylvania

Contact Details Editorial Office T: +919 267 6831 E: [email protected] Permissions http://group.bmj.com/group/rights-licensing/permissions Supplement Enquiries T: +44 (0)20 7383 6088 E: [email protected] Subscriptions (except USA) Subscription Manager, BMJ Journals, BMJ Publishing Group, PO Box 299, London WC1H 9TD, UK T: +44 (0)20 7111 1105 E: [email protected] http://jamia.bmj.com/site/help/index.xhtml Subscriptions (USA) PP&F, PO Box 361, Birmingham, AL 35201-0361, USA T: +1 800 348 6473 (toll free in the USA) E: [email protected] Display Advertising Sales T: +44 (0)20 7383 6783 E: [email protected] http://group.bmj.com/group/advertising

Subscription Information The Journal of the American Medical Informatics Association is published bimonthly; subscribers have access to all supplements

Institutional Rates 2014 Print and online US$702; £360; €486 (small) US$847; £434; €586 (medium) US$935; £479; €647 (large) Online only US$603; £309; €418 (small) US$747; £383; €518 (medium) US$843; £432; €584 (large)

Personal Rates 2014 Print (include online access at no additional cost) US$353; £181; €245 Online only US$328; £168; €227 Personal print or online only and institutional print subscriptions may be purchased online at http://group.bmj.com/group/ subs-sales/subscriptions (payment by Visa/Mastercard only).

Display & Online Advertising Sales (USA) T: +1 201 767 4170 E: [email protected] Author Reprints Reprints Administrator T: +44 (0)150 251 5161 E: [email protected] Commercial Reprints (except USA & Canada) Nadia Gurney-Randall T: +44 (0)20 8445 5825 M: 07866 262344 E: [email protected] Commercial Reprints (USA & Canada) Marsha Fogler T: +1 800 482 1450 (toll free in the USA) T: +1 856 489 4446 (outside the USA) E: [email protected]

Residents of some EC countries must pay VAT; for details, call us or visit www.bmj.com/subscriptions/vatandpaymentinfo.dt

amiajnl_21_E1_EB.indd 1

09/01/14 5:06 PM

Contents

Volume 21 Issue e1 | JAMIA February 2014

JAMIA is AMIA’s premier peer-reviewed journal for biomedical and health informatics. Covering the full spectrum of activities in the field, JAMIA includes informatics articles in the areas of clinical care, clinical research, translational science, implementation science, imaging, education, consumer health, public health, and policy. JAMIA’s articles describe innovative informatics research and systems that help to advance biomedical science and to promote health. Case reports, perspectives and reviews also help readers stay connected with the most important informatics developments in implementation, policy and education.

Editor-in-Chief

Highlights

Lucila Ohno-Machado

e1

Associate Editors

Patricia Flatley Brennan Atul Butte Enrico Coiera Charles Friedman Betsy Humphreys Kevin Johnson Harold Lehmann Prakash Nadkarni

Electronic health record systems: risks and benefits L Ohno-Machado

Perspective e2

Workshop on using natural language processing applications for enhancing clinical decision making: an executive summary V M Pai, M Rodgers, R Conroy, J Luo, R Zhou, B Seto

Assistant Editor

Michael F Chiang

Reviews Editorial Assistant

Marissa Wolff Editorial Office

T: +919-267-6831 F: +919-287-2768 E: [email protected] ISSN: 1067-5027 (print) ISSN: 1527-974X (online) Impact factor 3.571 Disclaimer: No responsibility is assumed by the Publisher or by the American Medical Informatics Association for any injury and/or damage to persons or property as a result of any actual or alleged libelous statements, infringement of intellectual property or privacy rights, or products liability, whether resulting from negligence or otherwise, or from any use or operation of any ideas, instructions, procedures, products, or methods contained in this issue of the Journal of the American Medical Informatics Association.The publication of an advertisement herein does not constitute on the part of the Publisher or the American Medical Informatics Association a guarantee or endorsement of the quality or value of the advertised products or services described therein or of any of the representations or the claims made by the advertisers with respect to such products or services. Copyright: © 2014 by the American Medical Informatics Association (print ISSN 1067-5027) is the official journal of the American Medical Informatics Association, which holds copyright and ownership of all materials published in the journal unless otherwise stated, and is published bi-monthly by the BMJ Group, BMA House, Tavistock Square, London WC1H 9JR, UK, typeset by Techset and printed in the USA on acid free paper. Periodicals postage paid at Hanover, PA.

e6

Views of healthcare professionals to linkage of routinely collected healthcare data: a systematic literature review

e55 A highly scalable, interoperable clinical decision support service H S Goldberg, M D Paterno, B H Rocha, M Schaeffer, A Wright, J L Erickson, B Middleton

e63 Classification of medication incidents associated with information technology K-C Cheung, W van der Veen, M L Bouvy, M Wensing, P M L A van den Bemt, P A G M de Smet

e71 The impact of electronic health records on people with diabetes in three different emergency departments S M Speedie, Y-T Park, J Du, N Theera-Ampornpunt, B A Bershow, R A Gensinger Jr, D T Routhe, D P Connelly

Y M Hopf, C Bond, J Francis, J Haughney, P J Helms

e11 Literature review of SNOMED CT use D Lee, N de Keizer, F Lau, R Cornet

e20 An integrative review of information systems and terminologies used in local health departments J Olsen, M J Baisch

Research and applications e28 Exploring the sociotechnical intersection of patient safety and electronic health record implementation

e78 A typology of electronic health record workarounds in small-to-medium size primary care practices A Friedman, J C Crosson, J Howard, E C Clark, M Pellerano, B-T Karsh, B Crabtree, C R Jaén, D J Cohen

e84 Joint segmentation and named entity recognition using dual decomposition in Chinese discharge summaries Y Xu, Y Wang, T Liu, J Liu, Y Fan, Y Qian, J Tsujii, E I Chang

D W Meeks, A Takian, D F Sittig, H Singh, N Barber

e35 Appropriateness of commercially available and partially customized medication dosing alerts among pediatric patients

MORE CONTENTS 䉴

J S Stultz, M C Nahata This article has been chosen by the Editor to be of special interest or importance and is freely available online.

e43 Evaluating the accuracy of electronic pediatric drug dosing rules E S Kirkendall, S A Spooner, J R Logan OPEN ACCESS

POSTMASTER: Send address changes to JAMIA, Mercury International Ltd, 365 Blair Road, Avenel, NJ 07001, USA.

Receive regular table of contents by email. Register using this QR code.

amiajnl_21_E1_TOC.indd 1

This article has been made freely available online under the BMJ Journals Open Access scheme. See http://adc.bmj.com/site/ about/guidelines.xhtml#open

e50 Open source electronic health records and chronic disease management J C Goldwater, N J Kwon, A Nathanson, A E Muckle, A Brown, K Cornejo

This journal is a member of and subscribes to the principles of the Committee on Publication Ethics www.publicationethics.org.uk

15/01/14 6:50 PM

Contents

e93 Development and use of active clinical decision support for preemptive pharmacogenomics G C Bell, K R Crews, M R Wilkinson, C E Haidar, J K Hicks, D K Baker, N M Kornegay, W Yang, S J Cross, S C Howard, R R Freimuth, W E Evans, U Broeckel, M V Relling, J M Hoffman

e100 Electronic medical records and physician stress in primary care: results from the MEMO Study S Babbott, L B Manwell, R Brown, E Montague, E Williams, M Schwartz, E Hess, M Linzer

e107 Comparison of two kinds of interface, based on guided navigation or usability principles, for improving the adoption of computerized decision support systems: application to the prescription of antibiotics R Tsopra, J-P Jais, A Venot, C Duclos

e117 A novel clinician interface to improve clinician access to up-to-date genetic results A R Wilcox, P M Neri, L A Volk, L P Newmark, E H Clark, L J Babb, M Varugheese, S J Aronson, H L Rehm, D W Bates

e122 Automating annotation of information-giving for analysis of clinical conversation

Volume 21 Issue e1 | JAMIA February 2014

e147 You and me and the computer makes three: variations in exam room use of the electronic health record J J Saleem, M E Flanagan, A L Russ, C K McMullen, L Elli, S A Russell, K J Bennett, M S Matthias, S U Rehman, M D Schwartz, R M Frankel

e152 Electronic health record use and preventive counseling for US children and adolescents C M Rand, A Blumkin, P G Szilagyi

e157 Engaging patients in medication reconciliation via a patient portal following hospital discharge L Heyworth, A M Paquin, J Clark, V Kamenker, M Stewart, T Martin, S R Simon

e163 Automated identification of patients with a diagnosis of binge eating disorder from narrative electronic health records B K Bellows, J LaFleur, A W C Kamauu, T Ginter, T B Forbush, S Agbor, D Supina, P Hodgkins, S L DuVall

e169 The effect of word familiarity on actual and perceived text difficulty G Leroy, D Kauchak

E Mayfield, M B Laws, I B Wilson, C P Rosé

e129 Applying operations research to optimize a novel population management system for cancer screening A H Zai, S Kim, A Kamis, K Hung, J G Ronquillo, H C Chueh, S J Atlas

e136 Supervised embedding of textual predictors with applications in clinical diagnostics for pediatric cardiology T E Perry, H Zha, K Zhou, P Frias, D Zeng, M Braunstein

Case report e173 Implementing health information exchange for public health reporting: a comparison of decision and risk management of three regional health information organizations in New York state A B Phillips, R V Wilson, R Kaushal, J A Merrill with the HITEC investigators

PostScript e178 Preserving an integrated view of informatics E V Bernstam, J D Tenenbaum, G J Kuperman

Brief communication e143 A taste of individualized medicine: physicians’ reactions to automated genetic interpretations H Lærum, S Bremer, S Bergan, T Grünfeld

amiajnl_21_E1_TOC.indd 2

15/01/14 6:50 PM

Highlights

Electronic health record systems: risks and benefits doi:10.1136/amiajnl-2014-002635 This special issue of JAMIA presents several applications of electronic health record (EHR) systems for clinical decision making. The integration of genetic information into EHRs has been receiving a lot of attention in the past few years. Wilcox et al (see page e117) describe the implementation of a user interface in an EHR that resulted in timely dissemination of gene variant information to clinicians. Lærum et al (see page e143) describe results of a qualitative study showing physicians’ positive reactions to automated genetic information. Bell et al (see page e93) report on the development, implementation and evaluation of a clinical decision support system (CDSS) for pharmacogenomics, showing high adherence to prescriptions guided by their system. The use of the EHR itself may result in improved process outcomes. Rand et al (see page e152) describe how its presence resulted in an increased number of counselling topics being delivered in paediatric settings. Speedie et al (see page e71) show a tendency towards reduced utilisation and better outcomes for diabetes patients treated in ED settings equipped with EHRs. Zai et al (see page e129) show how operations research can be used to help simulate scenarios for cancer screening to help resource allocation. However, there is also room for improvement. For example, Stultz and Nahata (see page e35) conclude that a CDSS for drug dosing among paediatric patients displays

J Am Med Inform Assoc February 2014 Vol 21 No e1

Lucila Ohno-Machado, Editor-in-chief too many inappropriate alerts, a finding also reported by Kirkendall et al (see page e43). In addition to accuracy, it is important to ensure that clinicians effectively utilise EHRs and CDSS. Saleem et al (see page e147) report on perceived barriers and variations in EHR use among clinicians. Patients also exhibit variation in their use of a portal designed for medication reconciliation, as shown by Heyworth et al (see page e157). Tsopra et al (see page e107) demonstrate how the utilisation of CDSS by physicians increases when a user interface based on their decision processes and usability principles is employed. Cheung et al (see page e63) report on the number and nature of medication incidents that can be attributed to poorly designed information system interfaces. Meeks et al (see page e28) describe how sociotechnical models can help stakeholders understand patient safety risks that can be associated with information technology. Goldwater et al (see page e50) highlight how a flexible open source EHR system was widely adopted in some community health centres. However, EHR utilisation may also be associated with higher clinician stress, as documented by Babbott et al (see page e100). Friedman et al (see page e78) provide a categorisation EHR workarounds employed by primary care practitioners. EHR utilisation and interoperability are key requirements of health information exchange (HIE) systems. Hopf et al (see

page e6) review the perspectives of healthcare professionals on linkage of healthcare data. Goldberg et al (see page e55) report on a CDSS designed to be scalable and interoperable. Interoperability is also critically important for public health, as described in a case report by Phillips et al (see page e173) and in a review by Olsen and Baisch (see page e20). Finally, structuring data in healthcare documents relies heavily on mapping concepts in the EHR to standards. Lee et al (see page e11) review the literature on SNOMED CT, and several authors report on the usefulness of natural language processing and related techniques in structuring data from clinical documents, including Pai et al (see page e2), Xu et al (see page e84), Mayfield et al (see page e122), Perry et al (see page e136), Bellows et al (see page e163), and Leroy and Kauchak (see page e169). To conclude this journal issue, correspondence by Bernstam et al (see page e178) discusses the pros and cons of promoting separate informatics sub-specialty conferences. Having this discussion is itself a sign that our discipline has reached adequate critical mass and has grown well beyond a small group of generalists. We need to move forward with an eye towards a future in which informatics permeates every aspect of healthcare and biomedical research. I hope this issue of JAMIA motivates our authors and readers to reflect on the exciting road that lies ahead.

e1

Perspective

Workshop on using natural language processing applications for enhancing clinical decision making: an executive summary Vinay M Pai,1 Mary Rodgers,1,2 Richard Conroy,1 James Luo,1 Ruixia Zhou,1 Belinda Seto1 1

National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Bethesda, Maryland, USA 2 School of Medicine, University of Maryland, Baltimore, Maryland, USA Correspondence to Dr Vinay Pai, National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, 6707 Democracy Blvd., Ste 200, Bethesda, MD 20892, USA; [email protected] Received 8 April 2013 Revised 13 June 2013 Accepted 24 July 2013 Published Online First 6 August 2013

ABSTRACT In April 2012, the National Institutes of Health organized a two-day workshop entitled ‘Natural Language Processing: State of the Art, Future Directions and Applications for Enhancing Clinical Decision-Making’ (NLP-CDS). This report is a summary of the discussions during the second day of the workshop. Collectively, the workshop presenters and participants emphasized the need for unstructured clinical notes to be included in the decision making workflow and the need for individualized longitudinal data tracking. The workshop also discussed the need to: (1) combine evidence-based literature and patient records with machine-learning and prediction models; (2) provide trusted and reproducible clinical advice; (3) prioritize evidence and test results; and (4) engage healthcare professionals, caregivers, and patients. The overall consensus of the NLP-CDS workshop was that there are promising opportunities for NLP and CDS to deliver cognitive support for healthcare professionals, caregivers, and patients.

INTRODUCTION

To cite: Pai VM, Rodgers M, Conroy R, et al. J Am Med Inform Assoc 2014;21:e2–e5. e2

With the explosion of new biomedical data, knowledge, and guidelines, clinical information has far exceeded human cognitive capacity. Clinical decision support (CDS) systems have great potential to make this information accessible and readily comprehensible to humans. CDS systems are computer-based software systems designed to help health professionals, patients, and care providers make informed clinical decisions, to provide rapid access to evidencebased guidance, and to suggest when additional information is needed or alternative hypotheses need to be considered. Natural language processing (NLP), with its purpose of enabling computers to derive meaning from natural language, has the potential to greatly enhance the function of the CDS systems. In April 2012, the National Institutes of Health organized a two-day workshop entitled ‘Natural Language Processing: State of the Art, Future Directions and Applications for Enhancing Clinical Decision-Making’.1 This workshop was designed to assess the current state of the art, challenges, and opportunities of NLP and CDS. The discussion of challenges and opportunities involved a wide range of stakeholders, including clinicians, academicians, and representatives from federal agencies, health insurance organizations, and industry. The viewpoints from the first day of the meeting, on the state-of-the-art and future directions for NLP, are being published elsewhere.2

During the second day of the workshop, the participants discussed how NLP and CDS could be harnessed to: ▸ incorporate legacy and unstructured clinical notes, ▸ develop longitudinal models for interpreting patient’s progression in health and wellness, ▸ enhance medical reasoning ▸ provide trusted and reproducible clinical advice, ▸ prioritize evidence and test results, and ▸ engage healthcare professionals, patients, and caregivers to promote effective communication and coordination of care.

STRUCTURED VERSUS UNSTRUCTURED DATA A major theme in the workshop was the need to address barriers to incorporating machine-readable unstructured notes into the CDS process. For the most part, traditional medical records have been unstructured notes that include medical history, detailed profiles of patients, medical examinations, pertinent interactions, and the clinician’s thought process. Roth et al3 indicated that qualitative measures (ie, disease-specific history, family history, patient education, and social history) affecting improvement in quality of care are difficult to capture in a structured note framework. The limitations of CDS systems as rule-based solutions that act on constrained ontologies may be addressed with the integration of NLP in CDS. NLP is instrumental in using free-text information to drive automated decision support, representing clinical knowledge interventions in standardized formats, and leveraging unstructured narrative.4 The workshop attendees agreed that it would be preferable to have the data structured after it is captured in the electronic health records (EHRs) rather than having it structured as part of the capturing process. One way to address this conundrum is by using NLP to extract context and meaning from the narrative text content of EHR with clinical information extraction (CIE) tools. Speakers noted the following issues with structured note entry: (1) there is a lack of uniformity in clinical definitions and procedures; (2) not all variables are captured to provide the best course of action under complex guidelines; (3) integration of a constrained system into the workflow of busy clinicians seeing complex patient cases is non-trivial; and (4) it is hard to train people to use encoded text systems consistently. Participants noted that the development and refinement of these tools in the clinical

Pai VM, et al. J Am Med Inform Assoc 2014;21:e2–e5. doi:10.1136/amiajnl-2013-001896

Perspective environment has been restricted due to the limited availability of training datasets because of patient confidentiality and privacy concerns and variability in text quality. In order for CIE tools to progress towards applications in the clinic, there is a need to develop trust to promote data sharing and to use human experts to verify extracted information.

PERSONALIZED LONGITUDINAL HEALTHCARE Since current EHRs are typically document-based and episodebased, they do not readily capture elements of the patient’s data that transcend care transitions or tease apart longitudinal chronic-care management of multiple co-morbidities. Elderly patients, in particular, tend to see a large number of different providers, who may not all be at the same institution, making it difficult to access complete medical records. Decision rules are primarily cross-sectional and not longitudinal because clinical evidence guidelines are most commonly developed based on episodic datasets or simple changes without taking into account complex personal histories. Radiology was presented as an example where decision support based on longitudinal analysis could be enhanced. While radiology reports are text-based, they usually have a constrained vocabulary and a limited number of concepts for each imaging modality. An NLP-CDS system would need to: (a) determine whether prior tests were positive or negative, (b) find previous recommendations, (c) identify any unexpected or unresolved findings in the past, and (d) consider the value of a new test, particularly one involving ionizing radiation. Ideally, an NLP-CDS system should be capable of detecting any unexpected findings in a summary report based on contextual analysis. Additionally, such a system should be able to automatically generate protocol guidance based on current findings and medical history and flag details for further attention.

ENHANCING MEDICAL REASONING Rule-based and statistical systems are two different techniques that have been developed to prioritize evidence and enhance medical reasoning using NLP. Participants noted that both approaches can trade-off sensitivity and specificity, an important characteristic to be able to accommodate a variety of applications. For example, specificity is a priority when triggering automated, patient-specific notices because of potential alert fatigue, while sensitivity is more important for identifying alternative diagnoses. Participants discussed the rule-based languages, including the Arden syntax5 promoted by HL7,6 and its application in ambulatory care. For example, in such care settings, drug–drug interactions can be handled for patients allergic to certain drugs, alternative medications can be recommended, abnormal lab results can be flagged, and providers can be alerted to opportunities for immunizations and preventive services. Rule-based systems have been developed using clinical practice guidelines formulated from clinical trials. However, since the clinical trials typically do not enroll patients with comorbidities, these rulebased systems fail to appropriately prioritize evidence and test results for patients with polypharmacy and/or comorbidity. Probabilistic analyses based on statistical models and machine learning approaches were discussed, including the recently demonstrated IBM DeepQA ‘Watson’ system.7 8 According to Duda and Shortliffe, a knowledge-based system can be described as an artificial intelligence (AI) program whose performance depends more on the explicit presence of a large body of knowledge than on the presence of ingenious computational procedures.9 Participants recognized that considerable research and implementation work

has been done in the development of computational approaches and knowledge-based systems utilizing NLP for supporting clinical decision-making.10–27 One of the current approaches that was discussed was the Lexicon-Mediated Entropy Reduction (LEXIMER) system,28 which extracts recommendations from a database of millions of clinical reports based on whether the reports are positive or negative. Thus, beside expert advice or literature, data mining could be used to determine percentage of positive studies or to evaluate previous recommendations. The DeepQA system used both structured and unstructured text to create a large body of knowledge on which statistical methods could be used. While this approach is not expected to comprehend very complex medical guidelines, there are plans to provide it with shallow semantics and reasoning tools to interpret a large number of evidence-based medical guidelines. Participants recognized that for the DeepQA system to be successful, it would need to be enhanced with multimodal analytics, provided by a framework like the Unstructured Information Management Architecture (UIMA).29 Three concerns were noted: (1) eventually machine learning saturates, (2) errors can propagate downstream, and (3) a good generalized knowledge base is difficult to generate for use by different AI systems. Participants observed that small doses of knowledge could inform and optimize statistical processes in ways that would be challenging for any amount of computation.

EVALUATING CLINICAL DECISION SUPPORT ENGINES Development of measures for evaluating CDS engines within the same clinical context is hindered by the lack of access to a standardized corpus of data. The challenge for researchers is the need to tackle legal, privacy, and institutional review board concerns for enabling access to the colossal amount of data currently available. The participants indicated that a federated database of anonymized medical data would be useful to enable the evaluation of CDS engines. Regarding the metrics for evaluating these engines, participants considered two approaches: (1) whether CDS systems generate advice that follows evidence-based guidelines reproducibly; or (2) whether the outcome is as expected. The latter can be difficult to analyze because it is idiosyncratic based on a particular patient and a particular situation. Even for a trial comparing CDS on a cohort of patients, the time required for determining whether different outcomes are reached can be so long that it may be difficult to draw adequate conclusions about the relative efficacy of a CDS system. While the ultimate correct outcome for decision support is the outcome and not the process change, the complexity in analyzing outcomes means that the optimal approach for evaluating CDS engines may be to determine how well the advice they provide follows evidencebased guidelines.

PRIORITIZING EVIDENCE AND TEST RESULTS There is a need for a framework, which could be ontological,9 that permits guidelines with specifications for recommendations and actions as well as algorithms for suggesting the temporal order of interventions. This would enable the CDS system to go beyond identifying very simple health problems and manage complex clinical scenarios that unfold in a complicated temporal sequence. However, this type of framework requires a lot of information which is not accessible through the coded data but is primarily in the narrative text. In addition to polypharmacy and comorbidities, the information needed includes currently not documented information such as patient preferences, provider preferences, and social support. The validity of secondary use of


e3

Perspective large clinical datasets in retrospective studies should be considered carefully in view of the potential for ‘missing’ coded data and the multi-faceted nature of human disease. Further work needs to be done to incorporate the non-coded data that exists in the EHRs, not in the context of clinical trials but of a compendium of medical treatment of similar patients, in order to enable decision-making for complex patients with polypharmacy and/or comorbidities.

Provenance and peer review Not commissioned; externally peer reviewed. Data sharing statement The PowerPoint presentations as well as the transcribed notes for the workshop can be found at: http://www.nibib.nih.gov/NewsEvents/ MeetingsEvents/MeetingSummaries/LP2012.

REFERENCES 1

STAKEHOLDER ENGAGEMENT There are multiple stakeholders in CDSsystems. These systems assist in making decisions that generally distinguish between three zones: do-not-treat, collect more knowledge through testing, and treating the patient. The creation of a knowledge base, or information repository, is essential for any CDS to be successful in this clinical process. However the speakers noted that populating the knowledge base with the appropriate set of structured data is essential to a strong statistically-based NLP system. Most EHRs do not include medical knowledge-base in one package. Medical knowledge-base is usually added as a component either by the users or implementers of EHR systems or by the medical knowledge-base vendors. There is a need for uniting the teams that are creating the medical knowledge and the clinical teams that are using that knowledge to support patient care. Another challenge for any good knowledge base is that it should be flexible enough to be compatible with different EHR systems and a variety of CDS engines. One of the concerns raised during the workshop was the need to expand the medical knowledge base to include information for and about patients with low locus of control, with limited education or limited English proficiency (LEP), or with low interest in maintaining personal health. In other words, how could the medical knowledge base generalize to incorporate information about patients who may or may not be within the healthcare system? There is also a need to incorporate patient preferences and utilities within the CDS system. CDS systems need to be informed about patients’ interest in their genetic testing, and reasoning about their diseases and treatements.

SUMMARY

2

3 4 5 6

7 8 9 10 11 12 13 14 15 16

Participants agreed that there were many challenges to be addressed, including importing hand-written notes, capturing oral dictation, and seamless implementations in clinical and non-clinical environments. However, participants believed that NLP and CDS were promising technologies for enabling delivery of cognitive support to healthcare professionals, patients, and caregivers by providing easily understandable synthesis and summary of the ever-expanding medical evidence and knowledge-base. Participants expressed optimism that NLP-enhanced CDS systems could become a ubiquitous tool in providing improved personalized healthcare.

17

18

19 20 21

Acknowledgements We would like to thank the organizing committee of the NLP-CDS workshop for setting up an excellent discussion forum. Besides the authors, the organizing committee included: Dr Blackford Middleton, Dr Olga Brazhnik, Dr Elaine Collier, Dr Milton Corn, Dr Dina Fushman, Dr Mike Huerta, Dr Thomas Rindflesch, Dr Steven Hirschfeld, Dr George Reddmond, Dr Abdul Shaik, Dr Jennie Larkin, Dr Rongling Li, Dr Peter Lyster, and Dr James DeLeo.

22

Contributors VMP extracted the notes from the transcribed pages of the workshop and drafted the early version of the manuscript with MR. All authors contributed to editing the drafts of the manuscript and approved the final paper.

23 24

Funding National Institutes of Health. Competing interests None. e4

25

NIH Workshop on Natural Language Processing: State of the Art, Future Directions and Applications for Enhancing Clinical Decision-Making, Bethesda, MD. http:// www.nibib.nih.gov/NewsEvents/MeetingsEvents/MeetingSummaries/LP2012 (accessed 12 Jun 2013). Friedman C, Rindflesch TC, Corn M. Natural language processing: state of the art and prospects for significant progress, a workshop sponsored by the national library of medicine. J Biomed Inform 2013. [Epub ahead of print 25 Jun 2013]. doi: 10.1016/j.jbi.2013.06.004 Roth CP, Lim YW, Pevnick JM, et al. The challenge of measuring quality of care from the electronic health record. Am J Med Qual 2009;24:385–94. Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform 2009;42:760–72. Hripcsak G, Clayton PD, Pryor TA, et al. The arden syntax for medical logic modules. Proceedings of the Annual Symposium on Computer Applications in Medical Care, 1990:200. Jenders RA, Sujansky W, Broverman CA, et al. Towards improved knowledge sharing: assessment of the HL7 Reference Information Model to support medical logic module queries. Proc AMIA Annu Fall Symp 1997:308–12. Ferrucci D. Build Watson: an overview of DeepQA for the Jeopardy! challenge. Proceedings of the 19th international conference on Parallel architectures and compilation techniques; Vienna, Austria: ACM, 2010:1–2. Ferrucci D, Brown E, Chu-Carroll J, et al. Building watson: an overview of the DeepQA Project. Ai Mag 2010;31:59–79. Tu SW, Musen MA. Modeling data and knowledge in the EON guideline architecture. Stud Health Technol Inform 2001:280–84. Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics 1983:207–15. Shortliffe EH. Computer programs to support clinical decision making. JAMA 1987;258:61–6. Hersh WR, Greenes RA. SAPHIRE—an information retrieval system featuring concept matching, automatic indexing, probabilistic retrieval, and hierarchical relationships. Comput Biomed Res 1990;23:410–25. Kahn CE. Artificial intelligence in radiology: decision support systems. Radiographics 1994;14:849–61. Sim I, Gorman P, Greenes RA, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001;8:527–34. Bashyam V, Taira RK. A study of lexical behavior of sentences in chest radiology reports. AMIA Annual Symposium Proceedings; American Medical Informatics Association, 2005:891. Bashyam V, Taira RK. Indexing anatomical phrases in neuro-radiology reports to the UMLS 2005AA. AMIA Annual Symposium Proceedings; American Medical Informatics Association, 2005:26. Huang Y, Lowe HJ, Klein D, et al. Improved identification of noun phrases in clinical radiology reports using a high-performance statistical natural language parser augmented with the UMLS specialist lexicon. J Am Med Inform Assoc 2005;12:275–85. Mendonça EA, Haas J, Shagina L, et al. Extracting information on pneumonia in infants using natural language processing of radiology reports. J Biomed Inform 2005;38:314–21. Dang PA, Kalra MK, Blake MA, et al. Natural language processing using online analytic processing for assessing recommendations in radiology reports. J Am Coll Radiol 2008;5:197. Dang PA, Kalra MK, Blake MA, et al. Use of Radcube for extraction of finding trends in a large radiology practice. J Digit Imaging 2009;22:629–40. Dang PA, Kalra MK, Schultz TJ, et al. Informatics in radiology render: an online searchable radiology study repository. Radiographics 2009;29:1233–46. Cheng LT, Zheng J, Savova GK, et al. Discerning tumor status from unstructured MRI reports—completeness of information in existing reports and utility of automated natural language processing. JDigit Imaging 2010; 23:119–32. Do BH, Wu A, Biswal S, et al. Informatics in radiology: RADTF: a semantic search– enabled, natural language processor–generated radiology teaching file. Radiographics 2010;30:2039–48. Rubin D, Wang D, Chambers DA, et al. Natural language processing for lines and devices in portable chest x-rays. AMIA Annual Symposium Proceedings; American Medical Informatics Association, 2010:692. Greenes RA. Clinical decision support: the road ahead. Academic Press, 2011.


Perspective 26 27

Yetisgen-Yildiz M, Gunn ML, Xia F, et al. Automatic identification of critical follow-up recommendation sentences in radiology reports. AMIA Annual Symposium Proceedings; American Medical Informatics Association, 2011:1593. Wang S, Summers RM. Machine learning and radiology. Med Image Anal 2012;16:933–51.

28 29


Dreyer KJ, Kalra MK, Maher MM, et al. Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: Validation study1. Radiology 2005;234:323–29. Ferrucci D, Lally A. UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat Lang Eng 2004;10:327–48.

e5

Review

Views of healthcare professionals to linkage of routinely collected healthcare data: a systematic literature review Y M Hopf,1 C Bond,1 J Francis,2 J Haughney,1 P J Helms3 ▸ Additional material is published online only. To view please visit the journal online (http://dx.doi.org/10.1136/ amiajnl-2012-001575). 1

Centre of Academic Primary Care, University of Aberdeen, Aberdeen, UK 2 School of Health Sciences, City University London, London, UK 3 Department of Child Health, University of Aberdeen, Royal Aberdeen Children’s Hospital, Aberdeen, UK Correspondence to Dr Yvonne Marina Hopf, Centre of Academic Primary Care, University of Aberdeen, Polwarth Building, Foresterhill, Aberdeen AB25 2ZD, UK; [email protected] Received 14 December 2012 Revised 1 May 2013 Accepted 4 May 2013 Published Online First 28 May 2013

ABSTRACT Objective To review the literature on the views of healthcare professionals to the linkage of healthcare data and to identify any potential barriers and/or facilitators to participation in a data linkage system. Methods Published papers describing the views of healthcare professionals (HCPs) to data sharing and linkage were identified by searches of Medline, EMBASE, SCOPUS, CINAHL, and PsychINFO. The searches were limited to papers published in the English language from 2001 to 2011. Results A total of 2917 titles were screened. From these, 18 papers describing the views of HCPs about data linkage or data sharing of routinely collected healthcare data at an individual patient level were included. Views were generally positive, and potential benefits were reported. Facilitators included having trust in the system including data governance, reliability, and feedback. Some negative views, identified as barriers were also expressed including costs, data governance, technical issues, and privacy concerns. Effects on the physician–patient relationship, and workload were also identified as deterrent. Discussion From the published literature included in this review, the views of HCPs were in general positive towards data sharing for public health purposes. The identification of barriers to contributing to a data linkage system allows these to be addressed in a planned data linkage project for pharmacovigilance. The main barriers identified were concerns about costs, governance and interference with the prescriber–patient relationship. These would have to be addressed if healthcare professionals are to support a data linkage system to improve patient safety. BACKGROUND AND SIGNIFICANCE

To cite: Hopf YM, Bond C, Francis J, et al. J Am Med Inform Assoc 2014;21: e6–e10. e6

Pharmacovigilance is the process by which adverse effects of drugs are detected, assessed, understood, and prevented.1 However, current systems are generally considered to be suboptimal2 3 and new approaches are required. Existing systems of pharmacovigilance vary and may be unique to individual countries, but often use elements of signal generation or dedicated follow-up studies. The term ‘signal generation’ refers to strategies designed to identify potential causal relationships between drug exposure and an adverse event. The Yellow Card Scheme (YCS) in the UK relies on voluntary reporting of adverse drug reactions (ADRs) by healthcare professionals (HCPs) and, more recently, by patients.4 In adults, all suspected adverse reactions to newly licensed drugs (currently bearing an inverse black triangle in the British National Formulary) should be

reported.3 5 For children, any suspected adverse drug reaction, independent of licensing status or severity of event, should be reported.6 Although the YCS is well established, its recognized limitations include the voluntary nature of reporting, the lack of a denominator (the total number of exposed individuals), the duration of therapy, and the variable quality of the data received.7 Although pre-licensing clinical studies aim to identify possible ADRs in addition to their clinical effectiveness, the limited number of patients in these studies reduces the chance of identifying uncommon reactions. At licensing it is not uncommon for less than 2000 patients to have been exposed to the given drug, whereas as many as 30 000 exposed individuals may be required to identify an ADR with an incidence of 1:10 000.3 Establishing sufficiently large exposure cohorts can be challenging, particularly for orphan drugs or rare conditions.8 This is compounded, particularly in children, by the frequent use of off-label or unlicensed drugs which are not subject to rigorous post-marketing surveillance.9 Linkage at individual patient level of routinely acquired health data between primary and secondary care could be important10; rates of off-label or unlicensed prescribing are higher in secondary (hospital) care,10 11 but current systems cannot link reactions reported to the general practitioner (GP) with the original hospital prescription. Linkage of routine healthcare datasets by unique patient identifiers could provide an alternative or complementary approach to the identification of ADRs. It would permit following exposed individuals in real time and provide a denominator. Routine data linkage would also enable creation of exposure cohorts in order to monitor long-term outcomes and enable a more efficient screening for side-effects or ADRs due to an ever increasing data pool.12 The use of routine healthcare data in the identification of potential adverse reactions to medicines through signal generation as well as the investigations of associations between an adverse reaction and medicines has been described previously.8 13 Whole population single databases compiled for administrative purposes such as the Hospital Episodes Statistics (HES) in England have demonstrated their potential in monitoring disease trends and important health outcomes,14 and in Sweden the use of linked national datasets for whole population epidemiology, or what is judged to be clinically relevant research, has been made possible without individual patient consent by the Swedish Health Act and European Union directive 95/46/EC.15 Although data linkage is mostly seen as advantageous, in particular for pharmacovigilance,8 10 16 17 some concerns have been voiced regarding

Hopf YM, et al. J Am Med Inform Assoc 2014;21:e6–e10. doi:10.1136/amiajnl-2012-001575

Review confidentiality and data protection for patient identifiable data18–20 as well as practical issues about incomplete or missing data in routinely collected datasets.12 21 22 A Wellcome report published in 2009 in the UK recognized the potential of the use of electronic records for data linkage and research but also highlighted that clinical data can rarely be anonymous, in the full sense of the word.23 To the best of our knowledge there is no current system which uses routine linkage of healthcare data for the purpose of identifying and monitoring ADRs. The CHIMES (Child Medical Records for Safer Medicines) program in Scotland is a research project which is developing a new system for drug monitoring and surveillance based on a linkage of routinely collected healthcare data from primary and secondary care, and prescription data. The focus of this initiative will be on children as it is known that ADRs in this population are particularly under-reported. Whilst the use of anonymised data from linked routinely collected primary and secondary care healthcare datasets is generally supported as long as the ethical, legal, and practical issues are taken into account,20 21 24 little is known about the views of HCPs who are the main providers of and in some cases the designated ‘guardians’ of such data. This information is important because if concerns are not identified and addressed, then HCPs would not engage in this process, reducing its efficiency. The aim of the work reported here was to describe the views of HCPs to data sharing or data linkage of clinical data for research purposes.

OBJECTIVE The objective of this literature review was to address the following questions: (1) What are the perceived barriers and facilitators to the linkage of routinely acquired healthcare data from the perspective of HCPs? (2) Would data linkage of routinely acquired healthcare data for clinical and research purposes be acceptable to HCPs?

METHODS A systematic approach was used to review the current literature (as shown in figure 1).

Bibliographic databases and search strategy Medline (Ovid Medline In-process and other Non-Indexed Citations and Medline (R) 1984), EMBASE (Embase Classic and Embase 1947), SCOPUS, CINAHL, and PsychINFO were searched. Search terms were adjusted to match individual database criteria. Each search comprised three broad domains: (i) medical records and data linkage, (ii) different types of HCPs, and (iii) views and opinions. The full search strategy is available on request.

Study inclusion criteria Peer-reviewed, empirical papers, and conference abstracts covering primary and secondary research were eligible for inclusion. Qualitative and quantitative studies were included. The search was restricted to papers published in English from 2001 to 2011. HCPs eligible for inclusion were medical doctors, nurses, or pharmacists. Papers were included if they reported on views of HCPs on data sharing (ie, the shared use of information about an individual patient across settings), or data linkage (ie, the secondary use of aggregated, merged data across settings) of healthcare data, including clinical, administrative, and prescribing information, for example, from primary to secondary care.

Study selection Titles and abstracts were reviewed by YH for eligibility. Full articles were retrieved for assessment or further clarification, for example if no abstract was available.

Data collection process A data abstraction form (available on request) was used to record standardized information from each paper as follows: authors, citation, design, aims and objectives of the study, methods, setting and participants (number if provided), the type of data linked, the purpose of data linkage/sharing, and a summary of the key findings on barriers and facilitators.

Quality assurance A random sample of papers was discussed at fortnightly research team meetings to confirm inclusion or exclusion decisions. A further random selection of papers was reviewed in duplicate by the researcher (YH) with several weeks between assessments. Papers for which the initial reviewer (YH) was unsure about inclusion or exclusion (n=26) were discussed with a second member of the research team (CB). In 23 cases the second reviewer confirmed the initial decision of YH (exclusion for n=20 papers and inclusion for n=3). For the remaining three papers a decision was made after discussion (n=2 included, n=1 excluded).

RESULTS Screening and identification of papers The search identified 2917 unique titles for screening. Selection of abstracts and papers is detailed in figure 1. A total of 188 abstracts were reviewed. For seven papers no abstract was available and the full paper was reviewed. One hundred and fifty-six papers were eligible for full review. Two papers were unavailable, that is, could not be retrieved during the review time, leaving 154 full-text articles. Authors were contacted in six cases to ask for further information/clarification (two authors answered and supplied further information). Papers/abstracts were excluded due to the following reasons: not about data sharing/linking across settings (n=47), no views of HCPs to data sharing/linkage (n=35), not describing empirical research (n=28), and other reasons (n=26).

Study characteristics An overview of the characteristics of the 18 included studies is presented in online supplementary table S1. The majority were conducted in the USA,26–34 followed by Canada.33 35–39 Two studies were conducted in each of the UK38 40 and the Netherlands,41 42 and one study in Finland.43 Many studies used surveys to explore views,28 30 32 34 36 41 43 six studies used qualitative methods such as interviews and focus groups,27 28 35 39 41 42 and the remaining five used a mixture of both and were classified as mixed methods research.26 31 33 37 40 Several sampling methods were used including purposive,30 31 33 convenience,35 39 random,32 40 41 and a priori.28 Although nine papers did not explicitly state their sampling method, it appeared from the results in four cases that a purposive/convenience sample was used29 34 37 42 and in one that an a-priori sample was employed.36 Survey response rates ranged from 37%38 to 77.1%,32 with a median response of 58.5%.


e7

Review

Figure 1 PRISMA flowchart of literature review (based on Moher et al25).

Participant characteristics The majority of participants were medical doctors working in either primary or secondary care.26–28 30–36 40–43 Other participants had a background in nursing,26 30 public health,38 39 or pharmacy.35 40 Some studies only acknowledged the use of ‘stakeholders’ who were drawn from relevant organizations, and another did not specify the background of their participants other than stating that they were healthcare providers.29

Data shared/linked The extent of proposed data sharing/linkage differed widely between studies. The majority sought to identify views to data sharing for research,27 29–32 34 36 41 43 two on pilots of data sharing26 35 and the remainder on hypothetical scenarios.28 33 37–40 42 The most common linkages were between laboratory and radiology records,26–28 30 32 34 36 followed by patient records from either primary or secondary care.27 31 34 37 41 Other studies described shared medication data which included data from computerized physician order entry systems, discharge summaries, prescriptions, and medication lists.27 28 30–32 35 37 40 42 Three studies did not specify the clinical data to be shared.29 38 43

Type and purpose of data sharing/linkage One study addressed views to sharing data with patients,26 and there was also sharing of data at an aggregate level, that is, anonymised data, with public health agencies.29 33 38 39 The majority of papers described data sharing (hypothetical and real) at the individual patient level, usually between pharmacies, and primary and secondary care, and in the USA between pharmacies and insurance companies.27–32 34–37 40–43

Views on data sharing/linkage Although the majority of views expressed about data sharing were positive, there were some negative views.28–32 34 35 38–40 43 Studies e8

categorized as ‘undecided’ in online supplementary table S1 reported both positive and negative views which appeared to be influenced by prior experience, as those with experience of linked data were generally in favor and those without were generally negative.27 33 41 A full range of views about data sharing were not identified in two studies as their purpose was to identify potential barriers to data sharing.36 37 The use of patient data for public health purposes was described by four papers: Rudin et al29 described the views of clinicians about sharing their data with public health departments, AbdelMalik et al38 discussed the need for patientidentifiable data for public health and the restrictions imposed by current legislation, and El Emam et al33 and Heidebrecht et al39 discussed the use of data for assessing immunization coverage. These studies showed that HCPs were positive about data sharing for a public health purpose. Only one study found that the view towards the secondary use of patient data would depend on the degree of identifiability.33

Barriers to data sharing/linkage The key findings of each study are summarized in online supplementary table S1. A frequently mentioned barrier to data sharing related to start-up and maintenance costs, including remuneration for participating providers.26 27 31 32 34 38 Concerns about data governance were also common, including data security, legal restrictions, and data quality.27–29 31–33 38 40 Technical problems such as lack of interoperability between IT systems were also identified,29 31 37 40 42 although Paré et al37 reported that these were less of an issue for data sharing per se. Privacy issues were cited in four studies.31 32 38 42 Consent was seen as necessary, although this was deemed impractical to obtain for large anonymised whole population studies in one study,33 and as a potential barrier by a minority (19%) of participants in another.34


Review As a group, physicians often suggested possible interference with their patient–physician relationships,33 threats to their professional autonomy, and a potential increase in the use of such data for litigation.42 Participants also reported a potential for an increased workload associated with uploading, verifying, and updating data.26 28 34 39 43 Lack of awareness of Health Information Exchange, that is, the use of ‘electronic movement of health-related information among organizations according to nationally (US) recognized standards31’ was identified as a barrier to data sharing,28 38 as was being a ‘non-user’ of existing infrastructures, such as electronic prescribing systems.41 The lack of a shared vision31 or commitment from management37 and competition between healthcare providers27 were also cited as obstacles.

Facilitators to data sharing/linkage Several studies listed possible improvements in patient care and safety29–31 34 35 40 as facilitators for data sharing. Rudin et al29 identified trust in the system as both a barrier and a facilitator as concerns of physicians about the sharing of clinical data appeared to be less in those who used linked IT based patient information systems. El Emam et al33 identified several governance features, such as comprehensive data sharing agreements, the use of de-identified data, and mandatory reporting, for example public health purposes that would reassure HCPs and facilitate support for research with linked data. The involvement of the relevant HCPs in the development of data sharing procedures would appear to facilitate data sharing,28 as would perceived ownership of any given project.36 Being a current user of a data sharing system,41 having a preference to view health records electronically,34 involvement in quality reporting initiatives,31 or perceived improvements in patient care associated with data sharing all acted as facilitators.29 30 34 40 HCPs perceived data sharing as beneficial, and with the potential of reducing healthcare costs28 32 by saving clinician time in accessing relevant patient data32 40 and providing timely access to comprehensive whole population trends and longitudinal data.39 Fontaine et al31 found that using a precursor system, that is, introducing a ‘light’ version of the planned system before full engagement, helped to dispel concerns of HCPs. Other facilitators included the clinical usefulness of the system,27 36 a well designed and easy to use interface,28 34 reliable system performance,30 43 and the ability to give and receive feedback.36

DISCUSSION Summary of evidence Data sharing was generally supported, and particularly so if HCPs had prior experience of its application as explicitly described in one study which compared the differences in views between those HCPs with and without experience in data sharing.41 Although no study identified a solely negative stance towards the sharing or linking of clinical data, several barriers were identified. Set-up costs for the required hardware and internet links along with subsequent system maintenance were perceived to be a problem,26 27 31 34 40 41 despite the potential for a reduction in healthcare costs overall. Potential improvements in patient care and safety were seen as facilitators of data sharing in contrast to lack of any perceived usefulness and patient benefits acting as deterrents for participation. Hence, system utility and performance would be keys for a successful data sharing project as they can act as both barriers and facilitators. The results from this review indicate that HCPs

are unlikely to support any data linkage system that is complicated, time-consuming, or costly. On the other hand, if benefits could be demonstrated, for example by providing easy access to comprehensive and longitudinal data, and particularly if the data were able to support strategic goals, this would work as a facilitator for data sharing. The direct involvement of HCP ‘champions’ willing to drive the project, in particular from doctors, nurses, and pharmacists, was identified to be crucial for success, as were willingness to co-operate, involvement, and psychological ownership which led to more enthusiasm.

Limitations This review summarizes a heterogeneous set of studies from different countries, with different methodologies and different data sharing or linkage schemes. Thus, only limited generalization and interpretation of the data is possible. Although the quality of included papers was not formally assessed using standard scales, when any uncertainties about the exact nature of the study were found, authors were contacted to clarify information about study methods, such as number of interviews conducted, nature of data sharing, and clarification of participants’ characteristics. Duplicate data extraction was not performed. However, a random sample of both included and excluded papers (n=32) was discussed. In addition, to facilitate confidence in reliability, a selection of papers was assessed twice by the same researcher, several weeks apart, and outcomes in terms of paper inclusion or data extracted were identical. Data sharing was associated with different terms, particularly in the USA, including electronic medical records, electronic health records, health information exchange, and computer information systems, often without explanation for readers unfamiliar with the described setting. One of the terms used to describe data sharing was ‘computerized physician/provider order entry’ (CPOE), a term that was excluded as closer examination of the papers using this term showed that CPOE described solely the sharing of information about a patient within a single setting, generally between the wards and pharmacy of a single hospital. The search strategy was not amended in order to include new terms identified during the review, such as ‘electronic health record’ (EHR), as they were already being successfully identified by the original search strategy. However, this might have led to a failure to identify all possible relevant papers. It was also not always clear whether electronic health records (EHRs) were shared within a single setting, such as a single practice or hospital or across settings, that is, record accessibility in primary and secondary care. This level of detail was central as the aim of the current review was to identify views of HCPs on data sharing and linkage across health sectors to inform the development of the CHIMES program for developing a more efficient system for pharmacovigilance. Although authors were contacted if information within their published paper was inadequate for an inclusion/exclusion decision, not every author provided the requested information, which led to five papers being excluded. Other problems encountered were the inclusion of views from mixed populations of HCPs, managers, and the lay public and in which answers provided could not be attributed by population group. This limited the validity of the results, as reported views might have been voiced by a healthcare manager or system administrator without a clinical background.


e9

Review CONCLUSION

15

Most studies described the sharing of data at an individual patient level and hence the observed views about barriers and facilitators to data sharing concerned the sharing of patient identifiable data. Identified barriers included costs, governance issues, and a perceived interference with the prescriber/patient relationship. Facilitators to data sharing were direct involvement of relevant HCPs in system design and the accessibility, perceived usefulness, and potential perceived benefits of the system. Benefits included easy access to complete and comprehensive patient data and the potential for improving quality of care and patient safety. In general, the views of HCPs were positive towards data sharing and linkage but the identified barriers will have to be addressed in future data linkage projects to facilitate support of healthcare professionals. Contributors PJH was the chief investigator of the overall program, conceived the research and led writing of the proposal for funding. CB and JH were co-investigators and led the writing of the work package which included this literature review. YMH was responsible for the draft of the literature review protocol, the daily study conduct and co-ordination, acquisition of data, analysis, producing tables and figures and interpretation of data. YMH drafted/co-led writing of the paper and incorporating feedback from co-authors on successive drafts. CB and JF contributed to the literature review protocol design and subsequent analysis, and co-led the writing of the paper. All authors commented on the initial drafts of the paper and revision of successive drafts. The final version of the manuscript was approved by all authors. Funding This work was supported by the Chief Scientist Office (Child Medical Records for Safer Medicines (CHIMES) Applied Research Program, grant number ARPG/07/4).

16 17 18 19 20 21 22

23 24 25 26 27

Competing interests: None.

28

Provenance and peer review Not commissioned; externally peer reviewed.

29 30

REFERENCES 1 2

3 4 5 6

7 8 9 10 11 12 13 14

e10

World Health Organization. The importance of pharmacovigilance. Geneva: World Health Organization, 2002. Avery AJ, Anderson C, Bond CM, et al. Evaluation of patient reporting of adverse drug reactions to the UK ‘Yellow Card Scheme’: literature review, descriptive and qualitative analyses, and questionnaire surveys. Health Technol Assess 2011;15:1–234. BMA Board of Science. Reporting adverse drug reactions—a guide for healthcare professionals. London: British Medical Association, 2006. Blenkinsopp A, Wilkie P, Wang M, et al. Patient reporting of suspected adverse drug reactions: a review of published literature and international experience. Br J Clin Pharmacol 2007;63:148–56. Joint Formulary Committee 2012. British National Formulary. 63rd edn. London: British Medical Association and Royal Pharmaceutical Society of Great Britain, 2012. Paediatric Formulary Committee 2011. British National Formulary for Children 2011–2012. London: British Medical Association, the Royal Pharmaceutical Society of Great Britain, the Royal College of Paediatrics and Child Health, and the Neonatal and Paediatric Pharmacists Group, 2011. Clarkson A, Choonara I. Surveillance for fatal suspected adverse drug reactions in the UK. Arch Dis Child 2002;87:462–6. Luo X, Cappelleri JC, Frush K. A systematic review on the application of pharmaco-epidemiology in assessing prescription drug-related adverse events in pediatrics. Curr Med Res Opin 2007;23:1015–24. Neonatal and Paediatric Pharmacists Group. Introduction to paediatric pharmaceutical care. 1st edn. Glasgow: NHS Education for Scotland, 2005. Neubert A, Sturkenboom M, Murray ML, et al. Databases for paediatric medicine research in Europe-assessment and critical appraisal. Pharmacoepidemiol Drug Saf 2008;17:1155–67. Choonara I, Conroy S. Unlicensed and off-label drug use in children: implications for safety. Drug Saf 2002;25:1–5. Glover GR. A comprehensive clinical database for mental health care in England. Soc Psychiatry Psychiatr Epidemiol 2000;35:523. Naina Mohamed I, Helms PJ, Simpson CR, et al. Using primary care prescribing databases for pharmacovigilance. Br J Clin Pharmacol 2011;71:244–9. Aylin P, Bottle A, Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. Br Med J 2007;334:1044.

31 32 33

34 35 36

37 38 39 40

41 42 43

Wettermark B, Hammar N, Fored Cm, et al. The new Swedish Prescribed Drug Register—Opportunities for pharmacoepidemiological research and experience from the first six months. Pharmacoepidemiol Drug Saf 2007;16:726–35. Crombie IK, Brown SV, Hamley JG. Postmarketing drug surveillance by record linkage in Tayside. J Epidemiol Community Health 1984;38:226–31. Helms PJ, Ekins Daukes S, Taylor MW, et al. Utility of routinely acquired primary care data for paediatric disease epidemiology and pharmacoepidemiology. Br J Clin Pharmacol 2005;59:684–90. Evans JMM, McNaughton D, Donnan PT, et al. Pharmacoepidemiological research at the Medicines Monitoring Unit, Scotland: data protection and confidentiality. Pharmacoepidemiol Drug Saf 2001;10:669–73. Black N. Secondary use of personal data for health and health services research: why identifiable data are essential. J Health Serv Res Policy 2003;8 (suppl_1):36–40. Lowrance W. Learning from experience: privacy and the secondary use of data in health research. J Health Serv Res Policy 2003;8(suppl 1):2–7. Oostenbrink R, Moons KGM, Bleeker SE, et al. Diagnostic research on routine care data: prospects and problems. J Clin Epidemiol 2003;56:501–6. Jansen ACM, van Aalst-Cohen ES, Hutten BA, et al. Guidelines were developed for data collection from medical records for use in retrospective analyses. J Clin Epidemiol 2005;58:269–74. Wellcome Trust. Towards Consensus for Best Practice: Use of patient records from general practice for research. London: Wellcome Trust, 2009. Evans J, MacDonald TM. Record-linkage for pharmacovigilance in Scotland. Br J Clin Pharmacol 1999;47:105–10. Moher D, Liberati A, Tezlaff J, et al. and for the PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J 2009;339:b2535. Earnest MA, Ross SE, Wittevrongel L, et al. Use of a patient-accessible electronic medical record in a practice for congestive heart failure: patient and physician experiences. J Am Med Inform Assoc 2004;11:410–17. Grossman JM, Bodenheimer TS, McKenzie K. Hospital-physician portals: the role of competition in driving clinical data exchange. Health Aff (Millwood) 2006;25:1629–36. Shapiro JS, Kannry J, Kushniruk AW, et al. Emergency physicians’ perceptions of health information exchange. J Am Med Inform Assoc 2007;14:700–5. Rudin RS, Simon SR, Volk LA, et al. Understanding the decisions and values of stakeholders in health information exchanges: experiences from Massachusetts. Am J Public Health 2009;99:950–5. Chisolm DJ, Purnell TS, Cohen DM, et al. Clinician perceptions of an electronic medical record during the first year of implementation in emergency services. Pediatr Emerg Care 2010;26:107–10. Fontaine P, Zink T, Boyle RG, et al. Health information exchange: participation by Minnesota Primary Care Practices. Arch Intern Med 2010;170:622–9. Wright A, Soran C, Jenter CA, et al. Physician attitudes toward health information exchange: results of a statewide survey. J Am Med Inform Assoc 2010;17:66–70. El Emam K, Mercer J, Moreau K, et al. Physician privacy concerns when disclosing patient data for public health purposes during a pandemic influenza outbreak. BMC Public Health 2011;11:454. Patel V, Abramson EL, Edwards A, et al. Physicians’ potential use and preferences related to health information exchange. Int J Med Inf 2011;80:171–80. Sellors C, Sellors J, Levine M, et al. Computer networking to enhance pharmacist-physician communication: a pilot demonstration project in community settings. Can Pharm J 2004;137:26–30. Paré G, Sicotte C, Jacques H. The effects of creating psychological ownership on physicians’ acceptance of clinical information systems. J Am Med Inform Assoc 2006;13:197–205. Paré G, Sicotte C, Jaana M, et al. Prioritizing the risk factors influencing the success of clinical information system projects: a Delphi study in Canada. Methods Inf Med 2008;47:251–9. AbdelMalik P, Boulos MNK, Jones R. The perceived impact of location privacy: a web-based survey of public health perspectives and requirements in the UK and Canada. BMC Public Health 2008;8:156. Heidebrecht CL, Foisy J, Pereira JA, et al. Perceptions of immunization information systems for collecting pandemic H1N1 immunization data within Canada’s public health community: a qualitative study. BMC Public Health 2010;10:523. Porteous T, Bond C, Robertson R, et al. Electronic transfer of prescription-related information: comparing views of patients, general practitioners, and pharmacists. Br J Gen Pract 2003;53:204–9. Boonstra A, Boddy D, Fischbacher M. The limited acceptance of an electronic prescription system by general practitioners: reasons and practical implications. New Tech Work Employ 2004;19:128–44. Boonstra A, Boddy D, Bell S. Stakeholder management in IOS projects: analysis of an attempt to implement an electronic patient file. Eur J Inform Syst 2008;17:100–11. Jaatinen P, Aarnio P, Asikainen P. The foundations for a regional information system based on a reference register. J Inform Technol Healthcare 2006;4:154–64.


Review

Literature review of SNOMED CT use Dennis Lee,1 Nicolette de Keizer,2 Francis Lau,1 Ronald Cornet2,3 ▸ Additional material is published online only. To view these files please visit the journal online (http://dx.doi. org/10.1136/amiajnl-2013001636). 1

School of Health Information Science, University of Victoria, Victoria, British Columbia, Canada 2 Department of Medical Informatics, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands 3 Department of Biomedical Engineering, Linköping University, Linköping, Sweden Correspondence to Dennis Lee, School of Health Information Science, University of Victoria, PO Box 3050 STN CSC, Victoria, BC, Canada V8W 3P5; [email protected] Received 9 January 2013 Revised 10 June 2013 Accepted 19 June 2013 Published Online First 4 July 2013

ABSTRACT Objective The aim of this paper is to report on the use of the systematised nomenclature of medicine clinical terms (SNOMED CT) by providing an overview of published papers. Methods Published papers on SNOMED CT between 2001 and 2012 were identified using PubMed and Embase databases using the keywords ‘systematised nomenclature of medicine’ and ‘SNOMED CT’. For each paper the following characteristics were retrieved: SNOMED CT focus category (ie, indeterminate, theoretical, pre-development/design, implementation and evaluation/commodity), usage category (eg, prospective content coverage, used to classify or code in a study), medical domain and country. Results Our search strategy identified 488 papers. A comparison between the papers published between 2001–6 and 2007–12 showed an increase in every SNOMED CT focus category. The number of papers classified as ‘theoretical’ increased from 46 to 78, ‘pre-development/design’ increased from 61 to 173 and ‘implementation’ increased from 10 to 34. Papers classified as ‘evaluation/commodity’ only started to appear from 2010. Conclusions The majority of studies focused on ‘theoretical’ and ‘pre-development/design’. This is still encouraging as SNOMED CT is being harmonized with other standardized terminologies and is being evaluated to determine the content coverage of local terms, which is usually one of the first steps towards adoption. Most implementations are not published in the scientific literature, requiring a look beyond the scientific literature to gain insights into SNOMED CT implementations.

INTRODUCTION

To cite: Lee D, de Keizer N, Lau F, et al. J Am Med Inform Assoc 2014;21: e11–e19.

The use of free text and local terms in electronic medical records is widespread and is a source of poor data quality and a barrier to semantic interoperability, data mining, secondary use of data and computerized clinical decision support.1 The systematised nomenclature of medicine clinical terms (SNOMED CT) is an international clinical reference terminology that has the potential to improve data quality and patient safety, and facilitate semantic interoperability by capturing clinical data in a standardized, unambiguous and granular manner. January 2013 marked the 11th year since SNOMED CT was first released. Since January 2002, 22 new versions, released semi-annually, have been circulated. The International Health Terminology Standards Development Organisation (IHTSDO) was established 6 years ago to coordinate the maintenance and promotion of SNOMED CT as a clinical reference terminology, and 19 countries have designated SNOMED CT as the preferred clinical reference terminology for use in electronic medical records.

Lee D, et al. J Am Med Inform Assoc 2014;21:e11–e19. doi:10.1136/amiajnl-2013-001636

In this study, our objective was to investigate the use of SNOMED CT by providing an overview of published studies. Whereas the 40-year SNOMED literature review by Cornet et al,2 in 2008 focused on papers published between 1966 and 2006 using any version of SNOMED, this study focused only on SNOMED CT papers published between 2001 and 2012.

METHODS Identifying papers Searches using PubMed (http://www.ncbi.nlm.nih. gov/pubmed) and Embase (http://www.embase.com) were performed using the terms ‘SNOMED’ and ‘systematised nomenclature of medicine’ between 2001 and 2012. Although SNOMED CT was first released in 2002, we presumed there were papers that discussed the upcoming release of SNOMED CT published in 2001. Only papers that were written in English or had an English abstract were included in this study. The search strategy is available in supplementary appendix A (available online only).

Classification criteria We used a set of classification criteria similar to that used in the 40-year review,2 with the addition of one new criterion, the SNOMED CT focus category. A summary of the classification criteria is available in table 1.

SNOMED CT focus category

We identified five SNOMED CT focus categories: indeterminate, theoretical, pre-development/design, implementation, and evaluation/commodity. ‘Indeterminate’ refers to SNOMED CT being used as an example of a terminology system without any further detail on its use or implementation, is referenced in a letter by a reader, editor or author, or is included in a survey or review. ‘Theoretical’ refers to SNOMED CT being discussed as a terminology system but not used in conjunction with a clinical project/study. There are likely to be no outcomes but rather descriptive work on the development of SNOMED CT or envisioned outcomes. The next three focus categories address the application of SNOMED CT. ‘Pre-development/design’ refers to SNOMED CT being assessed to determine if it fulfills requirements and whether it is feasible to be used in a full-scale implementation as a terminology standard. ‘Implementation’ refers to SNOMED CT being used in a study, pilot project or operational setting. ‘Evaluation/commodity’ refers to SNOMED CT being evaluated to determine the effects of the implementation and demonstrate its value (eg, how it can enhance the quality of care) or is used in an operational setting where e11

Review example, ‘to prove merit in terms of costs’ and ‘to prove merit in terms of quality of care’ were merged into ‘prove merit’. Each of the 15 usage categories was linked to one and only one of the five SNOMED CT focus categories (see table 2). As a paper could span multiple usage categories, we used the most prominent usage category in classifying the paper. For example, a paper3 that described the comparison of a problem list with SNOMED CT or the annotation of clinical narratives with SNOMED CT to determine the content coverage was classified as ‘retrospective content coverage’. If the concepts identified were used in a study (research or non-operational setting), for example, to calculate the prevalence of a disease, that paper4 was classified as ‘used to classify or code in a study’. If the setting was an operational setting in which the concepts identified were stored in actual patient records and used for patient care, that paper5 was classified as ‘implementation of SNOMED CT’.

Table 1 Criteria used to classify SNOMED CT-related papers No

Criteria

Definition

1.

SNOMED CT focus category

2.

Usage category

3. 4.

Medical domain Country

Refers to the focus of the paper (ie, indeterminate, theoretical, pre-development/design, implementation, evaluation/commodity) Refers to how SNOMED CT is primarily used. Each usage category belongs exclusively to one focus category. Refer to table 2 for the list of usage categories and their definitions Refers to the medical domain of the paper Refers to the country in which the study took place, if available or the country of the first author. If the study spanned multiple countries, the paper was classified as ‘multiple’

SNOMED CT, systematised nomenclature of medicine clinical terms.

the focus has moved from capturing data to using the data captured in routine patient care.

Usage category The usage category refers to the primary purpose for using SNOMED CT. The 14 usage categories from the 40-year review2 were re-examined and several categories were created, renamed and merged. Categories were created and renamed to reflect new ways in which SNOMED CT was being used and to clarify the categories. The main reason for merging the categories was due to low frequency counts. In the 40-year review,2 there were five categories in which one paper was assigned to a category. For

Classifying method A web-based application was developed that cataloged the abstracts and papers, and enabled the co-authors to classify the papers independently. Functions were also available for the authors to compare their results with each other, add comments and review the results of papers from the 40-year review. The abstracts were used to classify a paper and the full paper was referred to if details needed to classify the paper were not evident in the abstract. To ensure interrater reliability, 10 papers were selected and classified individually by the co-authors. The results were compared and discussed until a consensus was reached on the differences and definitions on classification categories were refined.

Table 2 List of usage categories and definition, and corresponding focus category SNOMED CT focus category

No

Usage category

Status

Definition

1.

Other

New

2.

As an example

Same

3.

Same

7.

Illustrate terminology systems theory Description of SNOMED CT and other standards Terminology auditing Compare to or map to other terminology systems Translation

8.

Prospective content coverage

Same

9.

Prospective interrater agreement

New

10.

Planned standard for electronic health records Design considerations

Same

Includes letters submitted to journals and reports on the results of surveys, literature reviews and systematic reviews References SNOMED CT briefly as a standard terminology or that it is used in a study with few additional details Describes terminology systems theory such as frameworks for describing terminologies and potential benefits of using standardized terminologies Describes SNOMED CT and other terminologies including technical aspects (eg, hierarchy) and non-technical aspects (eg, potential benefits and challenges) Reports on auditing methods that have been applied to SNOMED CT to detect errors SNOMED CT is compared to other standardized terminology systems mainly in terms of content coverage Describes the needs for translating SNOMED CT into other languages or the progress and results of translation studies SNOMED CT is compared to non-standardized terminology systems such as local interface terminologies for content coverage Similar to prospective content coverage, but the focus is on comparing the results of between two or more coders SNOMED CT is planned for use in an EHR but the focus is on the overall EHR infrastructure and not on SNOMED CT Describes implementation considerations such as the use of search algorithms and version management SNOMED CT is used only for a study and not in a routine setting

4. 5. 6.

11. 12. 13. 14. 15.

Used to classify or code in a study Implementation of SNOMED CT Prove merit Retrieve or analyse patient data

New Renamed Same New

Same Same Same Merged Same

SNOMED CT is implemented in a pilot or operational setting Studies that demonstrate the benefits of using SNOMED CT in operational settings SNOMED CT has been in used in routine patient care and the focus has moved from capturing data with SNOMED CT to using the data captured

Indeterminate

Theoretical

Pre-development/design

Implementation

Evaluation/commodity

Status refers to the comparison with the usage categories in the 40-year review and indicates whether the usage category is new, is the same, was renamed or was merged. EHR, electronic health record; SNOMED CT, systematised nomenclature of medicine clinical terms.

e12


Review The authors then worked in pairs to classify an additional 30 papers to ensure there was an agreement on how the criteria were to be assigned to a paper. Additional discussions took place to resolve any ambiguity, and when all differences in classification were reconciled, the first author proceeded to classify the rest of the papers. Twenty-five papers were flagged by the first author when the usage category was uncertain. These papers were reviewed by the other authors and discussions took place to reconcile the classification.

RESULTS The searches on PubMed (n=537) and Embase (n=594) resulted in 702 unique papers (see figure 1). Two hundred and fourteen (30%) papers were excluded because the version of SNOMED was not clinical terms (n=127, 18%), the paper made no mention of SNOMED CT (n=55, 8%), an English abstract was not available for a foreign language paper (n=21, 3%), and an abstract or full paper could not be located (n=9, 1%). In all, 488 unique papers were reviewed. The list and classification of the 488 papers are available in supplementary appendix B (available online only), while a summary of the papers classified as ‘pre-development/design’, ‘implementation’ and ‘evaluation/commodity’ is available in supplementary appendix C (available online only).

SNOMED CT focus The results of the classification of papers by SNOMED CT focus category and by year are shown in figure 2. The number of papers classified as ‘theoretical’ has remained relatively the same at between 11 and 15 papers over the past 8 years. A comparison of the papers published from 2001 to 2006, and papers published from 2007 to 2012 showed an increase in every

SNOMED CT focus category. The number of papers classified as ‘theoretical’ increased from 46 to 78, ‘pre-development/ design’ increased from 61 to 173, and ‘implementation’ increased from 10 to 34. Papers classified as ‘evaluation/commodity’ only started to appear in 2010.

Usage category The results by usage category are shown in figure 1. A further breakdown of the usage categories by subcategories is shown in table 3. In this section we describe the most common usage category for each SNOMED CT focus category except for ‘indeterminate’.

Theoretical: compare to or map to other terminology systems (n=74) SNOMED CT was compared to or mapped to at least 40 standardized terminologies. The exact number is unknown as not all papers listed all the terminologies used, and therefore we are uncertain of the number of unique terminologies compared. The most common terminologies SNOMED CT was compared or mapped to were the International Classification of Diseases, both the 9th and 10th revisions (n=15), International Classification of Nursing Practice (n=6) and the Medical Dictionary for Regulatory Activities (n=5). SNOMED CT was also compared to the unified medical language system (UMLS) metathesaurus directly (n=6) and indirectly (n=12). The direct comparisons occurred when a terminology system was mapped to SNOMED CT and other terminology systems including the UMLS metathesaurus. The indirect comparisons occurred when the UMLS metathesaurus was primarily used to look up mappings to other terminologies. While ‘compare to or map to other terminology systems’ was the most common usage

Figure 1 Overview of scoring of papers. Lee D, et al. J Am Med Inform Assoc 2014;21:e11–e19. doi:10.1136/amiajnl-2013-001636

e13

Review Figure 2 Number of papers by maturity level and year.

category in this focus category, the new usage category ‘terminology audit’ included 24 papers, 20 of which were published in the past 6 years.

Pre-development/design: prospective content coverage (n=59) SNOMED CT was used in 59 studies to determine the degree to which SNOMED CT could provide content coverage for local Table 3

terms. The content coverage included comparing SNOMED CT against larger enterprise interface terminologies and data dictionaries such as the Vanderbilt EHR interface terminology6 and Mayo mastersheet index,1 as well as to smaller sets of terms in the domains of problem lists and diagnosis (n=7), care planning and guidelines (n=6) and nursing (n=4). Content coverage was usually assessed using exact matches, partial matches, no

Number of papers by subcategories

No

Usage category and subcategory

Number

1. 2. 3.

As an example—no subcategories Other—letters to editor (n=3), reply from authors (n=2), literature reviews (n=5), surveys (n=4) Illustrate terminology systems theory—terminology theory and ontological principles (n=14), semantic similarity (n=8), frameworks and models for categorizing terminology systems (n=6), need for mapping (n=5) Description of SNOMED CT and other standards—general description of SNOMED CT (n=35), development process and milestones of SNOMED CT (n=7), changes, improvements and advancement of SNOMED CT (n=7), use of definitions and qualifiers (n=5), use of relationship groups (n=4), use of description logic (n=3), potential benefits of SNOMED CT (n=3) Terminology auditing—abstraction network (n=8), ontological principles (n=4), lexical/linguistic (n=5), combination of methods (n=2), other methods with frequency of one each (n=8) Compare to or map to other terminology systems—39 other standardized terminology systems, most common were the International Classification of Diseases, 9th and 10th Revisions, (n=17) and International Classification for Nursing Practice (n=6). SNOMED CT was also compared to the UMLS directly (n=6) and indirectly through the UMLS metathesaurus (n=12) Translation—languages included French (n=5), Swedish (n=1) and Chinese (n=1) Prospective content coverage—interface terminologies, data dictionaries and medical corpora (n=7), chief complaints/problem lists (n=6), care planning and guidelines (n=6), newborn disorders (n=3), drugs (n=3), nursing (n=4), cardiovascular disorders (n=2), complex chronic conditions (n=2), ophthalmology (n=2), reason for visit/chief complaint for emergency department (n=2), pathology diagnoses (n=2), allergies (n=2) and others with frequency of one (n=21) Prospective inter-rater agreement—number of reviews were two (n=1), three (n=6) and 10 (n=1) Standard for electronic health records—electronic health records frameworks/infrastructure and integration with information models (n=24), binding to clinical models, templates or archetypes (n=14) Design considerations—search and retrieval algorithms (n=18), general implementation challenges (n=8), process and challenges related to the development of subsets (n=8), version control, management and migration (n=5), the role and use of interface terminologies in conjunction with SNOMED CT to facilitate data capture (n=3), encoding methodologies or comparison of coding techniques (n=3) Used to classify or code in a study—identifying and extracting mainly from free text narratives and reports, general medical conditions (n=6), cancer characteristics (n=4), emergency room (n=2), pneumonia and influenza cases (n=3), medications and drug concerns (n=2), intensive care (n=1), pathology (n=1) and negation (n=1). Implementation of SNOMED CT—terminology servers and services to support data entry (n=10), use of data entry templates (n=10), use of search boxes and auto-complete (n=3), use of natural language processing (n=1) Prove merit—no subcategories Retrieve or analyse patient data—use of SNOMED CT synonyms against free text (n=2), indexed free text with SNOMED CT concepts using natural language processing and queried indexed concepts (n=4), unclear if synonyms or concepts were used (n=1), subject matter experts encoded queries (n=1)

61 17 33

4.

5. 6.

7. 8.

9. 10. 11.

12.

13. 14. 15.

64

27 74

7 59

8 40 46

20

24 0 8

SNOMED CT, systematised nomenclature of medicine clinical terms; UMLS, unified medical language system.

e14


Review matches, and matches using post-coordination. Exact or complete matches were as high as 90% in areas such as the representation of disorders of newborn infants7 and as low as 19% in areas such as aesthetic ophthalmic plastic surgery.8 Post-coordination was required in over 40% of domains such as cardiovascular diseases, computed tomography procedures, and clinical phenotype data.

Implementation: implementation of SNOMED CT (n=24) This usage category can be further divided into the development of SNOMED CT terminology servers and services to support data entry (n=10) and the implementation of SNOMED CT in clinical settings in both pilot projects and operational settings (n=14). The terminology servers and services included visual exploration of terminologies and specialized search algorithms to navigate the hierarchy and retrieve relevant concepts for data entry (n=6), search for publications using SNOMED CT concepts (n=1), search for healthcare providers using consumer terms mapped to SNOMED CT and clinician expertise (n=1). Two other papers listed the features of their own terminology servers (n=1) and that of vendors (n=1). The user interfaces in which SNOMED CT was implemented can be further classified into three categories. First, items in checklists, questionnaires and data entry templates were mapped to SNOMED CT. In those cases, the options in the forms were fixed and did not require users to search for SNOMED CT descriptions directly (n=8). Local terms were presented to users in the form of pick lists and radio buttons while the data were recorded in the background with SNOMED CT. Domains included cancer,9–12 pressure ulcer wounds,13 radiology,14 obesity,15 and family planning.16 Second, search boxes and autocomplete fields were used to display results based on user input (n=5). SNOMED CT subsets were developed based on historical patient records so as to constrain the concepts used in the results rather than search against the entire SNOMED CT content. Domains included drugs,17 veterinary,18 intensive care,19 ambulatory care20 and general patient records.21 Third, natural language processing algorithms were used to locate potentially relevant SNOMED CT concepts from clinical narratives (n=1). Clinicians were shown the candidate concepts for review before the concepts were indexed to the patient record.5

Evaluation/commodity: retrieve and analyze patient records (n=8) Two papers used SNOMED CT to identify synonyms for neuromuscular blockade22 and Clostridium difficile infections23 as keywords for searching against clinical narratives. Four papers used natural language processing to index clinical narratives with SNOMED CT concepts followed by a query against those concepts. The queries were for cancer,24 infectious symptoms,25 and diabetes mellitus, cardiovascular diseases, asthma and congestive obstructive pulmonary disease,26 and 54 diseases such as esophageal reflux and HIV.27 In addition to just querying for the index concepts, the index concepts’ children in the SNOMED CT hierarchies were included in search queries although the value of querying for children concepts was not reported. One paper used SNOMED CT to identify occurrences of melanoma,28 but it was unclear whether synonym or concept matching of melanoma was used. In one paper, subject matter experts encoded 10 queries (eg, patients who had acute myocardial infarction and were on aspirin), which were then executed against a SNOMED CT-encoded patient database. Searches using SNOMED CT concepts were also shown to have better precision than keyword searches.27

Medical domain The papers spanned 36 medical domains and specialties. Problem list/diagnoses, nursing, drugs and pathology were the most common medical domains. The medical domains and specialties that occurred in at least 10 papers are shown in figure 3. Nursing primarily consisted of studies looking at the coverage of local nursing terms as well as standardized nursing terminologies such as International Classification for Nursing Practice.

Country The papers were from 22 countries with over half the papers coming from the USA (n=238, 53%) (see table 4 for the full list of countries). SNOMED CT-related papers originated from 10 of the 19 countries that are members of the IHTSDO while affiliates and non-member countries of the IHTSDO accounted for the other 13. The number of countries that have published SNOMED CT-related papers has steadily grown over the years, with the

Figure 3 Number of publications found for each medical domain.


e15

Review Table 4 Countries that belong to the IHTSDO or have published SNOMED CT-related papers in the scientific literature Country Argentina Australia Austria Belgium Brazil Canada China Cyprus Czech Republic Denmark Estonia France Germany Hungary Iceland Israel Italy

Joined IHTSDO?

2007

2007

First publish

No of papers

Country

Joined IHTSDO?

2007 2005 2011 2011 2009 2009 2009

4 26 2 1 1 15 3

2007 2011 2007 2011 2008 2009 2010

2010 2006

2 7

2002 2005 2008 2011

26 19 1 2

Lithuania Malta New Zealand Poland Singapore Slovak Republic Slovenia South Korea Spain Sweden Switzerland The Netherlands UK USA Multiple/unknown Total

2012

1

2009 2007 2010

2011 2012

2009 2007 2007 2007 2007

First publish

No of papers

2011

2

2008 2008 2006 2008 2005 2001 2001

10 14 12 2 21 34 255 28 488

IHTSDO, International Health Terminology Standards Development Organisation; SNOMED CT, systematised nomenclature of medicine clinical terms.

biggest increases coming in 2007–8 (see figure 4). Over the past 5 years, papers were coming from 14 to 16 countries per year.

proportion of studies by focus category over the past 6 years, with the exception of ‘evaluation/commodity’, has remained roughly the same.

DISCUSSION

Theoretical

In this study, we searched for SNOMED CT-related papers in PubMed and Embase and classified the papers by SNOMED CT focus category, usage category, medical domain and country. Over the past 6 years there has been an increase in the number of SNOMED CT-related studies centering on implementation and evaluation. Thirty-seven of the 44 papers classified as ‘implementation’ were published over the past 6 years, and all eight papers classified as ‘evaluation/commodity’ were published within the past 3 years. Nevertheless, the majority of the papers were classified as ‘pre-development/design’, which means SNOMED CT was mainly used in non-operational settings. The

While the number of papers classified as ‘theoretical’ has been steady over the past 8 years and range between 11 and 15 papers each year, one usage category within this focus category has seen a steady increase. ‘Terminology audit’, in which auditing methods such as the abstraction network and ontological principles have been developed and used to check SNOMED CT for consistency, has been steadily increasing since 2005. As SNOMED CT undergoes significant changes with each new release version29 30 we expect that these auditing methodologies will play a larger role in ensuring that SNOMED CT is consistent.

Figure 4 Number of papers per year by new countries, number of countries, cumulative countries and total papers.

e16


Review Pre-development/design The use of free text is one of the barriers to computerized clinical decision support and data re-use. However, fragmented and large numbers of standardized terminologies with partial and overlapping domain coverage is also a barrier.31 The large number of studies involved in comparing and mapping SNOMED CT to other standardized terminologies is encouraging as individuals and organizations are recognizing the need for harmonization. For example, nursing terminologies were one of the most frequently used terminologies that were compared to or mapped to SNOMED CT. Gaps in concept and synonym coverage identified in those studies can help to improve the completeness of nursing terms in SNOMED CT.32 33 After the usage categories of ‘description of SNOMED CT’ and ‘compare to or map to other terminology systems’, the third highest usage category was ‘prospective coverage’. In this category, SNOMED CT was evaluated to determine the content coverage of local terms. The high number of studies in this area is also encouraging because determining the content coverage was usually one of the first steps in the implementation studies identified in this study. The use of post-coordination in content coverage studies also indicates that while SNOMED CT may not include every pre-coordinated concept to represent a local term, it is possible to create semantically equivalent terms. As the crafting of post-coordinated expressions is more complex than just using pre-coordinated concepts, potential implementers will require additional training.

Implementation The number of studies classified as ‘implementation’ has more that tripled from 10 during the first 6 years when SNOMED CT was released to 34 over the past 6 years. Although SNOMED CT is reportedly used in over 50 countries and the number of studies classified as ‘implementation’ has been steadily increasing, there are still few papers that describe how SNOMED CT is being used in operational settings. Excluding the development of terminology servers and services, which are important and provide generic search and browsing capabilities, we encountered 14 studies of SNOMED CT in operational clinical settings and pilot projects. The sophistication of SNOMED CT implementations for data capture varied widely. Data entry ranged from mapping terms in data entry forms, templates and checklists to SNOMED CT in the background when users were only shown terms they were previously using, to the development of an interface terminology in which users were exposed to over thousands of descriptions and used auto-complete functionality to retrieve relevant terms, to the automatic indexing of clinical narratives using natural language processing techniques.

Evaluation/commodity We were only able to identify studies in the ‘retrieve and analyze patient data’ usage category. Data retrieval functionality ranged from very rudimentary use, such as the use of synonyms to search clinical narratives, to complex queries, such as the use of subsumption and querying against post-coordinated expressions. Unfortunately, the value of using subsumption queries was not reported. Success factors for implementing SNOMED CT included the development and use of tools that enabled SNOMED CT to be searched effectively and efficiently,34 usability and ease of use of clinical applications,19 the constraining of relevant concepts to create subsets in applicable domains,19 the incorporating of terms familiar to clinicians, and collaboration among clinical

users and technical developers.20 Challenges included the management of subsets and extensions,19 the development of intuitive interfaces and ensuring the relevancy of search results.20 Benefits, both realized and anticipated, included improved quality of documentation,16 improved efficiency and consistency of encoding,5 improved patient safety,17 reduced time and costs for transcribing, post-coding and quality management,5 16 35 ability to conduct biosurveillance monitoring,36 ability to audit patient records,26 support patient case queries,5 support integration with clinical practice guidelines,17 enable international benchmarking,35 and facilitate decision support systems.13 21 We did not encounter any studies that described the value of SNOMED CT in terms of improved outcomes. The three systems that developed decision support capabilities for detecting adverse drug events,21 managing wounds13 and obesity15 did not report on patient outcomes. While improved data standardization and the potential for conducting data analysis and reporting were frequently cited as benefits, these benefits have not been quantified and we have not found any studies that demonstrate the value of SNOMED CT from a clinical perspective in an operational setting (as opposed to a study). We suggest three reasons. First, a large proportion of the studies have been on prospective coverage, therefore organizations are still in the process of gauging the feasibility of adopting SNOMED CT. Second, organizations that have implemented SNOMED CT have been focusing on data capture and therefore have not reached the stage of using the captured data. In a separate survey we conducted, we found that most organizations that have implemented SNOMED CT have been focused on the implementation and have not had the time or resources to conduct full-scale evaluations.37 Third, we compared the papers in this study with two implementation inventories and found only five of the 23 implementations included in either or both of the IHTSDO implementation special group implementation webinars (http://www.ihtsdo.org/ events/conference-presentations/conference-archive/ implementation-experience) and Canada Health Infoway’s SNOMED CT in use website (https://sc.infoway-inforoute.ca/ standards-collaborative/snomed-ctr/snomed-ct-in-use) have been published in the scientific literature. It is unclear why 49 papers were retrieved when the search term ‘SNOMED’ or ‘systematized nomenclature of medicine’ was used but neither the abstract nor paper made any reference to SNOMED. For example, ‘bioinformatics and biological reality’38 was retrieved via PubMed and Embase but neither the medical subject headings (MeSH) terms, abstract nor paper contained any references to SNOMED. In another example, ‘in defense of the desiderata’39 included ‘systematised nomenclature of medicine*’ as one of the MeSH terms but the paper did not mention SNOMED. On the other hand, there are known SNOMED CT papers that are cataloged within PubMed and Embase that were not retrieved using those keywords. For example, the literature review, ‘A review of auditing methods applied to the content of controlled biomedical terminologies’, by Zhu et al,40 which cataloged the types of auditing methods applied to SNOMED CT (and other terminologies) was not retrieved using the keywords. To check the completeness of our search results, we compared the search results for papers published by the Journal of American Medical Informatics Association (JAMIA) using JAMIA’s website and PubMed. The results are available in supplementary appendix D (available online only). Refer to the appendix for the search strategy and full results. PubMed produced 27 results while JAMIA produced 24 results when searching in the title and abstract, and 167 results when searching the full text. A comparison of the 27 and


e17

Review 24 papers by PubMed and JAMIA showed that 23 papers overlapped. The one paper that was not retrieved by PubMed was a letter response from the authors.41 It should be noted that the letter was retrieved using Embase. The 143 difference between the search in the title and abstract versus the full text was usually the result of SNOMED CT being briefly mentioned as an example of a terminology system or the title in one of the references. Therefore, while it is possible that our search strategy missed some papers, it is unlikely to have missed substantial numbers.

11

Limitations

12

We only reviewed papers cataloged in PubMed and Embase and only included papers that were published in English or had an English abstract. Our review of two inventories of SNOMED CT use and the papers included in our study showed that the majority of implementations are not published in the scientific literature or are not captured in PubMed or Embase. Therefore, a limitation of this study is that it includes a publication bias. A second limitation is that the majority of the papers were reviewed only by the first author. To ensure consensus in the classification of the papers, 40 (9%) papers were reviewed by at least two authors to ensure a high level of agreement on how to assign the usage categories. In addition, 25 (6%) papers that the first author flagged were reviewed by a second author.

7 8 9 10

13

14 15 16

17 18

CONCLUSION Our literature review of 488 SNOMED CT-related papers showed that the majority of studies focused on theoretical and pre-development/design. This is still encouraging as work is being done to harmonize SNOMED CT with other standardized terminologies, and SNOMED CT is being evaluated to determine the content coverage of local terms, which is usually one of the first steps towards adopting SNOMED CT. The number of implementation studies has increased steadily although not many are in operational settings. We found that most implementations are not published in the scientific literature; therefore, a look beyond the scientific literature is needed to gain insights into SNOMED CT implementations. Contributors All authors reviewed and classified papers. DL drafted the initial manuscript, which was edited by the other authors. All authors participated in reviewing the comments by the associate editor and reviewers and contributed to addressing the concerns raised. The final version was approved by all authors.

19 20 21

22

23 24 25

Competing interests None. Provenance and peer review Not commissioned; externally peer reviewed.

26

REFERENCES 1 2 3

4 5 6

e18

Elkin PL, Trusko BE, Koppel R, et al. Secondary use of clinical data. Stud Health Technol Inform 2010;155:14–29. http://www.ncbi.nlm.nih.gov/pubmed/20543306. Cornet R, de Keizer N. Forty years of SNOMED: a literature review. BMC Med Inform Decis Mak 2008;8 (Suppl. 1):S2. http://www.ncbi.nlm.nih.gov/pubmed/ 19007439. Richesson RL, Andrews JE, Krischer JP. Use of SNOMED CT to represent clinical research data: a semantic characterization of data items on case report forms in vasculitis research. J Am Med Inform Assoc 2006;13:536–46. http://www.ncbi.nlm. nih.gov/pubmed/16799121. Long W. Extracting diagnoses from discharge summaries. AMIA Annu Symp Proc 2005:470–4. http://www.ncbi.nlm.nih.gov/pubmed/16779084. Ryan A, Patrick J, Herkes R. Introduction of enhancement technologies into the intensive care service, Royal Prince Alfred Hospital, Sydney. HIM J 2008;37:40–5. http://www.ncbi.nlm.nih.gov/pubmed/18245864. Wade G, Rosenbloom ST. Experiences mapping a legacy interface terminology to SNOMED CT. BMC Med Inform Decis Mak 2008;8 (Suppl. 1):S3. http://www.ncbi. nlm.nih.gov/pubmed/19007440.

27 28 29

30 31 32

James AG, Spackman KA. Representation of disorders of the newborn infant by SNOMED CT. Stud Health Technol Inform 2008;136:833–8. http://www.ncbi.nlm. nih.gov/pubmed/18487835. Lee S, Tsirbas A, Goldberg RA, et al. Standardized terminology for aesthetic ophthalmic plastic surgery. Ophthal Plast Reconstr Surg 2006;22:371–4. http:// www.ncbi.nlm.nih.gov/pubmed/16985422. Van Berkum MM. SNOMED CT encoded cancer protocols. AMIA Annu Symp Proc 2003:1039. http://www.ncbi.nlm.nih.gov/pubmed/14728542. Sherman S, Shats O, Fleissner E, et al. Multicenter breast cancer collaborative registry. Cancer Inform 2011;10:217–26. http://www.ncbi.nlm.nih.gov/pubmed/ 21918596. Sherman S, Shats O, Ketcham MA, et al. PCCR: pancreatic cancer collaborative registry. Cancer Inform 2011;10:83–91. http://www.ncbi.nlm.nih.gov/pubmed/ 21552494. Lusky K. Pilot points way to speedier cancer surveillance. CAP Today 2005;19:5–6, 8. http://www.ncbi.nlm.nih.gov/pubmed/15787106. Kim HY, Park HA. Development and evaluation of data entry templates based on the entity-attribute-value model for clinical decision support of pressure ulcer wound management. Int J Med Inform 2012;81:485–92. http://www.ncbi.nlm.nih.gov/ pubmed/22079242. Robinson TJ, DuVall SL, Wiggins RH III. Creation and storage of standards-based pre-scanning patient questionnaires in PACS as DICOM objects. J Digit Imaging 2011;24:823–7. http://www.ncbi.nlm.nih.gov/pubmed/20976611. Lee NJ, Bakken S. Development of a prototype personal digital assistant-decision support system for the management of adult obesity. Int J Med Inform 2007;76 (Suppl. 2):S281–92. http://www.ncbi.nlm.nih.gov/pubmed/17606400. Zetterberg C, Ahlzén K, Ericsson E, et al. An example of a multi-professional process-oriented structured documentation bound to SNOMED CT. Stud Health Technol Inform 2012;180:1215–17. http://www.ncbi.nlm.nih.gov/pubmed/22874405. Farfán Sedano FJ, Terrón Cuadrado M, García Rebolledo EM, et al. Implementation of SNOMED CT to the medicines database of a general hospital. Stud Health Technol Inform 2009;148:123–30. http://www.ncbi.nlm.nih.gov/pubmed/19745242. Zaninelli M, Campagnoli A, Reyes M, et al. The O3-Vet project: integration of a standard nomenclature of clinical terms in a veterinary electronic medical record for veterinary hospitals. Comput Methods Programs Biomed 2012;108:760–72. http:// www.ncbi.nlm.nih.gov/pubmed/22595264. Bakhshi-Raiez F, de Keizer NF, Cornet R, et al. A usability evaluation of a SNOMED CT based compositional interface terminology for intensive care. Int J Med Inform 2012;81:351–62. http://www.ncbi.nlm.nih.gov/pubmed/22030036. Liu J, Lane K, Lo E, et al. Addressing SNOMED CT implementation challenges through multi-disciplinary collaboration. Stud Health Technol Inform 2010;160:981–5. http://www.ncbi.nlm.nih.gov/pubmed/20841830. Cao F, Sun X, Wang X, et al. Ontology-based knowledge management for personalized adverse drug events detection. Stud Health Technol Inform 2011;169:699–703. http://www.ncbi.nlm.nih.gov/pubmed/21893837. Arnot-Smith J, Smith AF. Patient safety incidents involving neuromuscular blockade: analysis of the UK National Reporting and Learning System data from 2006 to 2008. Anaesthesia 2010;65:1106–13. http://www.ncbi.nlm.nih.gov/pubmed/ 20840604. Benoit SR, McDonald LC, English R, et al. Automated surveillance of Clostridium difficile infections using BioSense. Infect Control Hosp Epidemiol 2011;32:26–33. http://www.ncbi.nlm.nih.gov/pubmed/21128815. Nguyen A, Moore J, Zuccon G, et al. Classification of pathology reports for cancer registry notifications. Stud Health Technol Inform 2012;178:150–6. http://www. ncbi.nlm.nih.gov/pubmed/22797034. Matheny ME, Fitzhenry F, Speroff T, et al. Detection of infectious symptoms from VA emergency department and primary care clinical documentation. Int J Med Inform 2012;81:143–56. http://www.ncbi.nlm.nih.gov/pubmed/22244191. Liaw ST, Chen HY, Maneze D, et al. Health reform: is routinely collected electronic information fit for purpose? Emerg Med Australas 2012;24:57–63. http://www.ncbi. nlm.nih.gov/pubmed/22313561. Koopman B, Bruza P, Sitbon L, et al. Towards semantic search and inference in electronic medical records: an approach using concept-based information retrieval. Australas Med J 2012;5:482–8. http://www.ncbi.nlm.nih.gov/pubmed/23115582. Hussain F, Muller F, Husain E. Under-reporting of invasive malignant melanomas in North East of Scotland. Br J Dermatol 2010;163:67. Spackman KA. Rates of change in a large clinical terminology: three years experience with SNOMED Clinical Terms. AMIA Annu Symp Proc 2005:714–18. http://www.ncbi.nlm.nih.gov/pubmed/16779133. Wade G, Rosenbloom ST. The impact of SNOMED CT revisions on a mapped interface terminology: terminology development and implementation issues. J Biomed Inform 2009;42:490–3. http://www.ncbi.nlm.nih.gov/pubmed/19285570. Ingenerf J, Reiner J, Seik B. Standardized terminological services enabling semantic interoperability between distributed and heterogeneous systems. Int J Med Inform 2001;64:223–40. http://www.ncbi.nlm.nih.gov/pubmed/11734388. Hardiker NR, Casey A, Coenen A, et al. Mutual enhancement of diverse terminologies. AMIA Annu Symp Proc 2006:319–23. http://www.ncbi.nlm.nih.gov/ pubmed/17238355.


Review 33 34

35 36

Park HA, Lundberg CB, Coenen A, et al. Evaluation of the content coverage of SNOMED-CT to represent ICNP version 1 catalogues. Stud Health Technol Inform 2009;146:303–7. http://www.ncbi.nlm.nih.gov/pubmed/9592854. Richesson R, Young K, Guillette H, et al. Standard terminology on demand: facilitating distributed and real-time use of SNOMED CT during the clinical research process. AMIA Annu Symp Proc 2006:1076. http://www.ncbi.nlm.nih.gov/pubmed/ 17238695. Tvede I, Bredegaard K, Andersen JS. Quality improvements based on detailed and precise terminology. Stud Health Technol Inform 2010;155:71–7. http://www.ncbi. nlm.nih.gov/pubmed/20543312. Elkin PL, Froehling D, Wahner-Roedler D, et al. NLP-based identification of pneumonia cases from free-text radiological reports. AMIA Annu Symp Proc 2008:172–6. http://www.ncbi.nlm.nih.gov/pubmed/18998791.

37 38 39 40

41


Lee D, Cornet R, Lau F, et al. A survey of SNOMED CT implementations. J Biomed Inform 2013; 46:87–96. Johansson I. Bioinformatics and biological reality. J Biomed Inform 2006;39:274–87. http://www.ncbi.nlm.nih.gov/pubmed/16198638. Cimino JJ. In defense of the desiderata. J Biomed Inform 2006;39:299–306. http:// www.ncbi.nlm.nih.gov/pubmed/16386470. Zhu X, Fan JW, Baorto DM, et al. A review of auditing methods applied to the content of controlled biomedical terminologies. J Biomed Inform 2009;42:413–25. http://www.ncbi.nlm.nih.gov/pubmed/19285571. Wilcke JR, Green JM, Spackman KA, et al. Concerning SNOMED-CT content for public health case reports. J Am Med Inform Assoc. 2010;17:613; author reply 613–4. http://www.ncbi.nlm.nih.gov/pubmed/20842802.

e19

Review

An integrative review of information systems and terminologies used in local health departments Jeanette Olsen,1,2 Mary Jo Baisch1 ▸ Additional material is published online only. To view this file please visit the journal online (http://dx.doi.org/10. 1136/amiajnl-2013-001714). 1

Department of Nursing, University of Wisconsin Milwaukee, Milwaukee, Wisconsin, USA 2 Department of Allied Health, Wisconsin Indianhead Technical College, Rice Lake, Wisconsin, USA Correspondence to Jeanette Olsen, Department of Allied Health, Wisconsin Indianhead Technical College, 1900 College Drive, Rice Lake, WI 54829, USA; [email protected] Received 15 February 2013 Revised 31 July 2013 Accepted 23 August 2013 Published Online First 13 September 2013

ABSTRACT Objective The purpose of this integrative review based on the published literature was to identify information systems currently being used by local health departments and to determine the extent to which standard terminology was used to communicate data, interventions, and outcomes to improve public health informatics at the local health department (LHD) level and better inform research, policy, and programs. Materials and methods Whittemore and Knafl’s integrative review methodology was used. Data were obtained through key word searches of three publication databases and reference lists of retrieved articles and consulting with experts to identify landmark works. The final sample included 45 articles analyzed and synthesized using the matrix method. Results The results indicated a wide array of information systems were used by LHDs and supported diverse functions aligned with five categories: administration; surveillance; health records; registries; and consumer resources. Detail regarding specific programs being used, location or extent of use, or effectiveness was lacking. The synthesis indicated evidence of growing interest in health information exchange groups, yet few studies described use of data standards or standard terminology in LHDs. Discussion Research to address these gaps is needed to provide current, meaningful data that inform public health informatics research, policy, and initiatives at and across the LHD level. Conclusions Coordination at a state or national level is recommended to collect information efficiently about LHD information systems that will inform improvements while minimizing duplication of efforts and financial burden. Until this happens, efforts to strengthen LHD information systems and policies may be significantly challenged. BACKGROUND AND SIGNIFICANCE

To cite: Olsen J, Baisch MJ. J Am Med Inform Assoc 2014;21:e20–e27. e20

Health promotion, prevention, and surveillance of priority health conditions, with epidemiology as a fundamental component, are major areas of focus in public health practice1 necessitating robust information collection, analysis, interpretation, and communication methods.2 Over the past decade, advances in technology have universally changed the way data are located, analyzed and communicated. However, healthcare has trailed other industries in the use of information systems for professional support,3 and public health information systems have lacked the level of coordination necessary to improve understanding of the health of the population and to evaluate the outcomes of public health investments and initiatives.4 While there are numerous reports with recommendations for public health

information system development, implementation, and coordination, little is known about the current systems in use by local health departments. This integrative review explores relevant literature to identify information systems and standard terminologies currently being used by local health departments. This information may serve as a foundation for planning and coordination of public health informatics (PHI) at the local health department (LHD) level and better inform research, policy, and programs. LHDs are agencies within each state that serve small jurisdictions, such as township, city, county, or multicounty areas.5 Through the use of information technology, LHDs can improve efficiency and effectiveness in the delivery of care, yet the use of separate information systems and insufficient coordination between local and state public health entities is common.6 An understanding of the systems currently in use is necessary to identify needs across systems and to initiate improvements.7 Increasingly, the need for public health data in support of public health services and systems research is emphasized.8 Information technology is recognized for its ability to improve public health effectiveness through the collection, examination, and dissemination of data.2 7 9 Whereas informatics is considered to be the science of information,10 PHI is defined as ‘a systematic application of information and computer science and technology to public health practice, research, and learning’11 (p. 67). The benefits of PHI are actualized in multiple ways. For example, the timely collection and exchange of data through partnerships at local, state, and federal levels can support efforts to improve the health of the public.12 The connection of data and information from numerous sources can inform public health research.13 In addition, the use of informatics tools to implement standards-based interventions can transform public health practice.3 For maximum effectiveness, however, PHI systems should share a common set of population-level indicators4 supported through standardized languages or taxonomies to enable effective data sharing and analysis.14 The benefits of PHI are particularly relevant for LHDs, because well-designed information systems have the potential to provide access to current data for use in the planning and evaluation of health improvement efforts.7 However, LHD efforts to assess program outcomes are challenged by a lack of sophistication in both data collection systems and the information technology infrastructure.15 This is exemplified by a number of issues. Standardized organization and nomenclature is lacking;2 established methodologies for information storage and

Olsen J, et al. J Am Med Inform Assoc 2014;21:e20–e27. doi:10.1136/amiajnl-2013-001714

Review sharing are absent;16 population health data are not easily assessible in the USA because they exist in various formats across multiple agencies and web sites;17 and national standards for safeguarding public health agency data have not been established.18

OBJECTIVE Obtaining necessary data and information is challenging and requires effective and efficient use of financial and technological resources. However, these resources may not be readily available due to the rapid pace of informatics’ advancements in an environment of dwindling funds and new legislative regulations.19 Although there are numerous calls for improved PHI infrastructure and standards, there are few reports in the literature concerning the use of current information systems at the local public health level. The purpose of this integrative review of literature was to identify software programs and information systems currently being used by LHDs and to determine the extent to which standard terminology was used to communicate data, interventions, and outcomes. The goal is to inform future research, policy, and programs to improve PHI at the LHD level.

MATERIALS AND METHODS Whittemore and Knafl’s20 five-stage review methodology was used to guide this integrative review. The process involved the following five steps: articulation of the research problem; execution of a well-defined literature search; evaluation of the literature for quality of data; analysis of the data; and statement of conclusions. Studies using different research designs were included with the aim of presenting diverse perspectives and expanding knowledge.20 A systematic search of existing professional literature on PHI was conducted with the assistance of a research librarian in February 2012. The PubMed computerized database was explored using the MeSH term ‘public health informatics’ and the subheadings of: ‘classification’; ‘instrumentation’; ‘methods’; ‘organization and administration’; ‘standards’; ‘statistics and numerical data’; and ‘trends’. Limits were set for ‘English language’ and ‘humans’. In addition, the Academic Search Premier and CINAHL computerized databases were searched using the key words: ‘PHI and nomenclature’; ‘PHI and semantics’; and ‘public health and medical health records, computerized’. Ancestry searching of references of retrieved articles was conducted for additional relevant studies. A second step to the search process was conducted to identify relevant non-published data and additional articles published since February 2012. To improve the external validity, five content experts in PHI were consulted for a list of important websites and reports that should be included in the review. This included artifacts from websites such as the Public Health Data Standards Consortium, the American Medical Informatics Association (AMIA), the PHI Institute, RTI International, RAND International, the National Opinion Research Center (NORC), Mathematica, and Healthcare Information and Management Systems (HIMSS). In alignment with the purpose of the study and integrative review methodology, published reports and research studies of diverse designs were included if they described information systems or standard terminology used in LHDs. All article titles and abstracts generated from the search (N=645) were examined for relevance to the study purpose and inclusion criteria. Those that did not meet the criteria specified above were screened out (n=397). The full article was reviewed if the study met inclusion criteria, alignment with inclusion criteria was unclear, or the abstract was unavailable (n=248). Among articles reviewed in full, four exclusion criteria were used: studies or

articles that proposed new systems or frameworks or made recommendations for infrastructure improvements (n=101); described systems or nomenclatures outside the USA (n=17); described studies that piloted new software statistical methods (n=19); or did not identify information systems or standard terminologies used in LHDs (n=66) (see figure 1). The final sample for this review included 45 articles and artifacts. Data were analyzed using the matrix method21 according to study aim, design, sample, data sources and analytic strategy, findings, and critique (see supplementary appendix I, available online only). Descriptions of information systems and standardized terminology in use were extracted and summarized. Findings were then synthesized through comparison, interpretation, and elucidation of categories. In addition, each report was evaluated for methodological quality using a summed scoring system based on four criteria (see table 1).

RESULTS This review of professional literature sought both to identify software programs and information systems used by LHDs and to determine the extent to which standard terminology was used to communicate data, interventions, and outcomes. The findings from each of these areas will be discussed separately in the sections below.

Public health information systems Thirty-four of the 45 records reviewed included software programs and information systems used in LHDs. Most striking was the large number of different programs in use (see box 1). For example, in a survey of LHDs (n=344), Magruder et al9 reported the use of more than 500 different software programs. In addition, a survey of LHDs in Oregon revealed that at least 27 different systems were considered to be ‘working well’.6 The information systems reported in the literature were used to support a variety of public health functions. Five categories aligned with these functions emerged from the data during the review process: administration; surveillance; health records; registries; and consumer resources. The findings will be presented accordingly.

Administration Use of e-mail and Microsoft Office software were reported for administrative functions and documentation.6 9 Magruder et al9 reported that Microsoft Office programs such as Word, Excel, PowerPoint, and Access were among the most frequently used programs in LHDs. Microsoft Access databases were also used in western Oregon LHDs.22 E-Chronicle was used in Minnesota to capture tobacco cessation activities.23 Given the amount of administrative responsibilities of LHDs, the lack of information about systems being used to support this function is significant.

Surveillance Among the responsibilities of public health agencies, disease surveillance is one of the most important.5 Electronic surveillance systems provide a means of facilitating disease outbreak data collection, automated analysis, and information dissemination.5 Information technology in the area of surveillance was a common theme in the literature (n=25). The National Association of County and City Health Officials (NACCHO)7 found that 52% of LHDs used an electronic surveillance system for this purpose. Software or information system titles were not reported. Use of the Health Alert Network (HAN) was reported in five articles.6 9 24–26 McDaniel et al25 noted that it was used across the country to alert schools, emergency responders, and healthcare


e21

Review

Figure 1 Flow of information through the integrative review. LHD, local health departments; PHSSR, Public Health Services and Systems Research. agencies of disease outbreaks, natural disasters, and environmental threats. HAN was also identified as one of the programs most frequently used by LHDs9 and one of several programs working well among Oregon LHDs.6 The Electronic Surveillance System for Early Notification of Community-based Epidemics (ESSENCE) was another system reported. It was used at both state and local levels in eight states and Washington, DC for a variety of surveillance activities including rabies, influenza, and gastrointestinal outbreaks, as well as general monitoring of hospital and emergency department data.22 27 In addition, it was used for disease surveillance at over 250 military health clinics.28 Surveillance using disease mapping systems was also reported. McDaniel et al25 noted that one department used geographic information systems (GIS) to identify people at high risk of West Nile virus disease. Williams et al29 also reported on the use of GIS in large LHDs to map risk factors and disease distribution. Brownstein et al30 described the use of HealthMap, a free online resource for real-time surveillance and monitoring of disease outbreaks, noting that LHDs were among its many users. BioSense, a Centers for Disease Control and Prevention program that tracks health problems and notifies health officials, was another surveillance system reported, although locations of e22

users were not specified.31 32 Similarly, the National Electronic Disease Surveillance System (NEDSS) was used by many state and local agencies.25 32–34 A variety of other systems was reported, such as EpiX,35 Public Health Issue Management36 and the Communicable Disease Database37 in Washington, Communicable Disease Reporting and Syndromic Surveillance in New Jersey,38 and EpiCom in Florida.39 Multiple statespecific or home-grown systems were also described.22 24 40–43 The articles in this review indicated that several information systems are being used to support surveillance efforts in LHDs. More information is needed, however, regarding how extensively each of these systems is being used throughout the nation, how well they are working, and what other systems are in use.

Electronic health record and practice management systems Health records provide a means of documenting client data and public health services. This was another information system category found in the literature. NACCHO7 reported that among LHDs that provide primary care, 77% use electronic practice management systems. In this report, the names of the practice management systems were not specified. NACCHO7 also reported that 55% of LHDs used either a partial or complete


Review Table 1 Methodological quality ratings of included studies* Report

Type of study†

Sampling method

Data collection method detail provided

Analysis‡

Quality rating score

Magruder et al9 Foldy56 Landis et al46 McDaniel et al25 Goedert31 Brownstein et al30 Smith et al44 Nangle et al32 Savory et al28 Feuchtbaum et al52 OOHPR6 NACCHO7 Monsen et al64 Lewis et al27 Heisey-Grove et al61 Shapiro et al57 Monsen et al65 Monsen et al66 NORC22 McHugh et al51 Williams et al29 Ringle et al54 Kauerauf33 Octania-Pole38 Ising41 Le42 Health IT News59 Pare50 CDPH62 Banger et al45 Wine et al49 State of Michigan26 State of Indiana43 OHIP60 State of NC24 HIMSS58 Hersh34 Guthrie35 CIR48 PHIN39 Mackiewski and Taft23 Smith53 Lawson et al40 Pina et al36 Turner et al37

4: 4: 1: 1: 1: 1: 5: 1: 4: 4: 5: 5: 4: 1: 6: 1: 6: 6: 3: 3: 3: 4: 1: 1: 1: 1: 1: 1: 2: 1: 1: 2: 2: 2: 2: 3: 1: 1: 1: 1: 5: 1: 1: 3: 3:

3: 2: 0: 0: 0: 0: 3: 0: 1: 3: 3: 3: 2: 0: 3: 0: 1: 2: 2: 0: 1: 3: 0: 0: 0: 0: 0: 0: 0: 0: 0: 0: 0: 0: 0: 2: 0: 0: 0: 0: 3: 0: 0: 2: 2:

1: methods and tools 1: methods and tools 0: not explained or NA 0: not explained or NA 0: not explained or NA 0: not explained or NA 1: methods and tools 0: not explained or NA 1: methods and tools 1: methods and tools 1: methods and tools 1: methods and tools 1: methods and tools 0: not explained or NA 1: methods and tools 0: not explained or NA 1: methods and tools 1: methods and tools 1: methods and tools 1: methods and tools 1: methods and tools 1: methods and tools 0: not explained or NA 0: not explained or NA 0: not explained or NA 0: not explained or NA 0: not explained or NA 0: not explained or NA 1: methods and tools 1: methods and tools 0: not explained or NA 0: not explained or NA 0: not explained or NA 0: not explained or NA 0: not explained or NA 1: methods and tools 0: not explained or NA 0: not explained or NA 0: not explained or NA 0: not explained or NA 1: methods and tools 1: methods and tools 0: not explained or NA 1: methods and tools 1: methods and tools

2: 2: 1: 1: 2: 2: 2: 2: 2: 2: 2: 2: 3: 1: 2: 2: 3: 3: 1: 1: 1: 2: 2: 2: 2: 1: 2: 2: 2: 1: 2: 1: 1: 1: 1: 1: 2: 1: 1: 2: 2: 1: 1: 1: 1:

10 9 2 2 3 3 11 3 8 10 11 11 10 2 12 3 11 12 7 5 6 10 3 3 3 2 3 3 5 3 3 3 3 3 3 7 3 2 2. 3 11 3 2 7 7

quantitative quantitative best practice report best practice report best practice report best practice report mixed best practice report quantitative quantitative mixed mixed quantitative best practice report experimental best practice report experimental experimental qualitative qualitative qualitative quantitative best practice report best practice report best practice report best practice report best practice report best practice report government report best practice report best practice report government report government report government report government report qualitative design best practice report best practice report best practice report best practice report mixed best practice report best practice report qualitative design qualitative design

random or 100% purposive or CM not explained or NA not explained or NA not explained or NA not explained or NA random or 100% not explained or NA convenience random or 100% random or 100% random or 100% purposive or CM not explained or NA random or 100% not explained or NA convenience purposive or CM purposive or CM not explained or NA convenience random or 100% not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA not explained or NA purposive or CM not explained or NA not explained or NA not explained or NA not explained or NA random or 100% not explained or NA not explained or NA purposive or CM purposive or CM

descriptive descriptive narrative narrative descriptive descriptive descriptive descriptive descriptive descriptive descriptive descriptive inferential narrative descriptive descriptive inferential inferential narrative narrative narrative descriptive descriptive descriptive descriptive narrative descriptive descriptive descriptive narrative descriptive narrative narrative narrative narrative narrative descriptive narrative narrative descriptive descriptive narrative narrative narrative narrative

*For reports and articles covering multiple topics, the quality ratings pertain to only to methods and results regarding information system and standard terminology use in LHD. †Type of study: 1=best practice report; 2=government report; 3=qualitative design; 4=quantitative descriptive design; 5=mixed with both qualitative and quantitative descriptive designs; 6=quantitative experimental design. ‡Analysis (highest level reported): narrative; 2=descriptive statistics; 3=inferential statistics. CDPH, California Department of Public Health; CIR, California Immunization Registry; CM, case matching; LHD, local health departments; NA, not applicable; NACCHO, National Association of County and City Health Officials; OHIP, Ohio Health Information Partnership; OOHPR, Office for Oregon Health Policy and Research; PHIN, Public Health Information Network.

electronic health record. Other authors reported that some state health departments had implemented public health electronic personal health record systems.22–24 32 44 45 In Oregon, approximately 87% of LHDs that provide direct primary and preventive care used either an electronic health record and/or practice management system with the majority using clinical management software available through Ahlers and Associates.6 In Wisconsin, the Secure Public Health Electronic Record

Environment (SPHERE) was developed with Federal Maternal and Child Health (Title V) Program grant funding specifically for monitoring, reporting, and documenting maternal, child, and family health data.46 Similar to the area of surveillance, the reviewed literature indicated that information systems are being used by LHDs for client health record keeping. However, there is a lack of information regarding specific systems being used, who is using


e23

Review

Box 1 Software programs and systems Program Microsoft Office programs (including Access, Word, Excel, Powerpoint, Outlook)6 9 22 Arcview 9 Health Alert Network (HAN)6 9 22 24 26 WebbStarr9 Epi Info9 Kansas Integrated Public Health System (KIPHS)9 Virginia Information System Integrated Online Network (VISION)9 Human Services Information System (HSIS)9 HOST programs9 QS programs9 Women, Infants, and Children (WIC)9 23 Healthspace9 Electronic Surveillance System for Early Notification of Community-based Epidemics (ESSENCE)22 27 28 Massachusetts Virtual Epidemiological Network (MAVEN)61 National Electronic Disease Surveillance System (NEDSS)25 32–34 BioSense31 32 Remote Outbreak Detection and Surveillance (RODS)32 Early Hearing Detection and Intervention (EHDI)32 Child Health Advanced Records Management system (CHARM)32 Geographic Information System (GIS)25 29 HealthMap30 Secure Public Health Electronic Record Environment (SPHERE)46 Ahlers6 CareWare6 ELR lab reporting6 Family Net Alert6 IRIS6 ORCHIDS6 22 TWIST6 22 Raintree FP6 Medicaid Management Information System6 ORPHEUS6 22 OVERS for vital records6 Phoenix for food safety6 SWS Online for drinking water6 Webrad for lab results6 Citrix6 Alert6 e-Sentinel6 Family Net6 E-Chronicle23 Public Health Issue Management System37 CD-Database36 EpiCom39 EpiX35 NC DETECT24 41 CDRSS38 HIV/AIDS Reporting System22 Merlin22 MDSS22 MSSS22 CHAMPS22 23 CareFacts23 e24

PH-Doc23 PRISM22 23 MAXIS23 Client Care Management System23 Social Security Information System23 Win Clinic Assessment Software Application23 Medical Fiscal Intermediary Shared System23 MN-ITS23 EMR-lite45 NC Health Information System24 Health Management Systems EMR22 ADAP Database22 CAREWare22 SpecimenGate22 NextGen HER22 Netsmart Insight22 EPIC HER22 Home-grown systems23 40 CD, communicable disease; HER, health electronic record. them, and how well they are working to meet documentation, billing, and outcome assessment needs.

Registries Information specific to individuals who have certain diseases or health conditions can be collected, stored, and utilized in the format of a registry.47 Another public health information system category identified in the literature was electronic ‘registries’ with immunization registries being the most frequently reported.6 7 22–25 32 48–51 For example, NACCHO7 stated that 65% of LHDs used a web-based database to store and access some or all immunization data. In Oregon, 88% (n=28) of LHDs use electronic immunization registries.6 Other authors noted that immunization databases were commonly used in public health without specifying where they were being used.25 32 Additional registries that were reported included those for death, child health, cancer, chronic disease, and newborn screening.32 52 53 These descriptions did not include information about system effectiveness or specify geographical areas in which they were used.

Consumer resources

The final category, defined as ‘consumer resources’, included systems that provided a means for LHDs to provide information to the public. Several programs were reported. McDaniel et al25 noted that one county public health department, as part of their GIS initiative, notified citizens at risk of West Nile virus and recommended prevention strategies. Ringle et al54 investigated how well LHDs communicated H1N1 information on their websites and found that 34% (n=52) did so within 24 h of the public health emergency declaration, with more than half linking to the Centers for Disease Control and Prevention website. An additional consumer resource reported was Web 2.0 technologies. According to NACCHO,7 these technologies were used by 53% of LHDs and included Facebook (47%), Twitter (37%), You Tube (16%), My Space (11%), and blogs (11%). LHDs serving more populated areas reported using more Web 2.0 technologies than those serving smaller populations.7

Use of standard terminologies in public health information systems Standard terminologies are systems of approved words or phrases within a field or profession.55 Use of standard Olsen J, et al. J Am Med Inform Assoc 2014;21:e20–e27. doi:10.1136/amiajnl-2013-001714

Review terminology by LHDs for communicating data, interventions, and outcomes was more difficult to ascertain from the literature. Although several articles addressed the topic (33%), few provided details regarding precisely what was being communicated with standard terminology, how, and by whom. Two categories were identified from the reviewed literature: health information exchange and specific data and terminology standards.

Health information exchange Growing interest in the creation of health information exchange groups to facilitate electronic information sharing across organizations within communities or regions, linking LHDs with hospitals and primary care practices, was evident in the literature. NACCHO7 reported that 30% of LHDs have a health information exchange group operating in their area; however, no details about the programs were reported. Similarly, in Oregon, the ability of LHDs to share information varied based on context. Only 19% (n=6) of LHDs were able to exchange data pertaining to disease surveillance electronically, but up to 88% (n=28) could do so with immunization registries.6 Foldy56 reported that 21 organizations in Wisconsin had health exchange projects in either the planning or operating phases. Shapiro et al57 described the success of several health departments in New York regarding efforts to support biosurveillance activities. HealthBridge was described as a network being used in the greater Cincinatti–northern Kentucky tri-state area.58 59 LHDs were reportedly part of HIE initiatives in North Carolina,24 Ohio,60 Utah,45 Florida,22 and Michigan.22 Finally, Nangle et al32 described the use of a health information exchange group by a LHD in Colorado for public health alert communication. Notably, details regarding use of standard terminology to facilitate information exchange including application of specific types of standards were not described.

Specific data and terminology standards To facilitate electronic data exchange, Health Level Seven (HL7) was one standard recommended in the literature. Heisey-Grove et al61 stated that HL7 messaging format was used in a project involving LHDs to improve the Massachusetts hepatitis C surveillance system. In addition, Nangle et al32 asserted the importance of using national data standards such as HL7 for exchange of clinical health information, and cited Utah’s clinical laboratory results program as an exemplar. Ohio,60 Indiana,43 and California62 also reported using HL7 for data exchange. Use of the Omaha System, a standardized taxonomy for client care documentation,63 was also reported in the literature (n=3). Monsen et al64 used the Omaha System to study low-income, high-risk maternal child health clients receiving services from county health departments. The authors reported an improvement of health problems and asserted that informatics tools and data supported description of client health problems and intervention effectiveness. In another study, the Omaha System was used to study family home visits provided by public health nurses.65 The system was used to classify client risk, and the authors reported that it facilitated their examination of the relationship between home visiting interventions and outcomes. In a final study using the Omaha System, Monsen et al66 reported mothers with and without intellectual disabilities showed improvement in all health problem areas following family home visits by public health nurses, suggesting that the use of standardized clinical data may be beneficial for describing problems, services, and outcomes in public health.

Synthesis This review of literature indicated that a large number of information systems were used by LHDs, yet there was a lack of detail regarding specific programs being used, location or extent of use, or their effectiveness. Notably, different systems were reported to support diverse LHD functions. This would imply that most LHDs were using multiple systems, potentially contributing to financial and staffing burdens. This review also showed growing interest in health information exchange groups. However, few studies described the use of data standards or standard terminology in LHDs. In addition, only 42% (n=19) of the reports were conducted as research studies, and only 33% (n=15) used purposive or random sampling methodology. In summary, significant gaps in the area of information system and standard terminology use at the LHD level persist. Unless they are addressed, efforts to strengthen LHD informatics programs and policies may be significantly challenged.

DISCUSSION The purpose of this review of literature was to identify information systems and programs used by LHDs and to determine the extent to which data and terminology standards were used to communicate data, interventions, and outcomes. The results revealed that a large number of different information systems was used, aligning with five categories: administration; surveillance; health records; registries; and consumer resources. Despite a lack of detail regarding specific systems or applications, this literature review indicated an increasing interest by LHD and public health organizations in programs to facilitate health information exchange.7 55 A variety of information systems is necessary to meet the functional needs of LHDs. However, the large number of programs reported in the literature raises concerns. In 2005, the PHI Institute and National Association of State Chief Informatics Officers asked the question, ‘Why develop multiple, similar systems when our problem and information needs are similar?’ They recommended an ‘enterprise view of health information’ in which data about populations are shared among partners. This review indicates that there has been little change since that time. The use of multiple, diverse systems is a barrier to efforts to catalog programs or document which departments are using each of them. LHDs may have difficulty determining which systems to use to meet their needs best while ensuring compatibility or connectivity with other state and regional organizations. This may result in a lack of coordination that duplicates efforts and increases the financial burdens on LHDs. Efficiency in data entry, storage, and analysis are distinct benefits of information technology; however, capturing the full value of informatics in public health requires rapid ability to exchange data with stakeholders. This is particularly important with the development of health information exchanges in which local healthcare and public health services can be linked with federal activities such as the HITECH and the Affordable Care Acts and state Medicaid initiatives. Although numerous articles citing the benefits of standard terminology and proposing specific frameworks were found in the literature,14 67–69 there were few publications specifically describing the use of standard terminology in public health at the local level and there was little evidence that this was occurring. Quite possibly this can be partly explained by lingering confusion regarding the meaning of informatics concepts. For example, NACCHO7 reported that even among LHD staff, many people do not know the meaning of the term ‘PHI’. Clearly, LHD staff must have a thorough understanding of the subject matter before they


e25

Review will be able to implement and fully utilize standardized frameworks effectively. Therefore, education and training in this area may be beneficial. The dynamic nature and rapid advances within information technology underscore an urgent need for research to examine current practices regarding the use of information systems and standard terminology among LHDs. In addition, significant gaps must be addressed to facilitate the research and evaluation needed for population health improvement.70 This study revealed that a large number of different information systems was being used by LHDs, yet little has been documented regarding specific systems, extent and location of use, or system effectiveness. In addition, literature about environmental, dental, and other important service areas was scarce. Furthermore, few articles specifically described the use of standard terminology or provided evidence that they were used regularly in LHDs. It is critical that these gaps be addressed to provide current, meaningful data that inform PHI research and improvement initiatives at the LHD level. The public health data exchange hierarchy emphasizes local communication with reporting to regional centers, followed by state and then national agencies. Therefore, it would be beneficial for further LHD informatics enquiry and initiatives to be coordinated with state and national leadership. This would support the efficient collection of relevant information regarding current informatics practices, challenges, and priorities in LHDs that could be used to direct program and policy development. Initial enquiry at the state level may also enhance regional system coordination, thus diminishing duplication of efforts and financial burden. Public health practice and research may benefit from greater attention to each of the following areas: administrative processes; health registries; electronic health and practice management systems; consumer resources; and surveillance systems.

Contributors Both authors contributed to the conception and design, acquisition of data, and analysis and interpretation of data for this review. The article was drafted by the first author and was critically revised for important intellectual content by the second author with both giving final approval of the version to be published. Competing interests None. Provenance and peer review Not commissioned; externally peer reviewed.

REFERENCES 1 2

3 4 5 6

7 8

9 10 11 12 13

Limitations This study is limited by the possibility that articles pertinent to the review may have been missed due to variances in key terms and concepts. Articles may also have been missed due to rapid changes in the field. These limitations were addressed by following a clearly articulated search method, consulting with a research librarian, and reviewing findings with multiple experts in the field.

14 15 16 17

CONCLUSION Information systems and terminology standards have significant potential to support efforts to improve the health of the public. Future advancement in PHI will necessitate a systems approach and an infrastructure grounded in interdependence and integration.69 Collaborating entities will need to use the same standards and compatible software.15 Funding will be necessary to accomplish the integration of current non-standardized systems and to maintain them in the ever changing healthcare environment.15 Results of this review indicated that a large number of information systems were being used by LHDs. However, there was a lack of data specifically reporting which systems were used, where or how extensively each was used, or how well they were working at the LHD level. In addition, literature on standard terminology use was minimal and details regarding its application to facilitate health information exchange were not described. These gaps in the literature must be addressed to provide current, meaningful data that inform PHI research and improvement initiatives at the LHD level. Coordination at a state or national level is recommended to collect information efficiently that will inform improvements to LHD information systems, minimizing duplication of efforts and financial burden. e26

18 19 20 21 22

23 24 25 26 27

Baisch MJ. Community health: an evolutionary concept analysis. J Adv Nurs 2009;65:2464–76. Richards J. Importance of public health informatics: a survey of public health schools and graduate programs in the United States. Student Health Technol Inform 2007;129:1410–13. Hanrahan LP. eHealth: the foundation for health system transformation. Wis Med J 2007;106:112–15. Institute of Medicine. For the public’s health: The role of measurement in action and accountability. Report. Washington, DC: The National Academies Press, 2011. Institute of Medicine. The future of the Public’s Health in the 21st century. Report. Washington, DC: The National Academies Press, 2003. Office for Oregon Health Policy and Research. Public health and information exchange: a survey of Oregon’s local health departments. [cited 30 January 2012]. http://www.oregon.gov/OHA/OHPR.HITOC/Documents/Public_Health/OR_LHD_srvy_ Rpt.pdf?ga=t. National Association of County & City Health Officials. The status of local health department informatics. Washington, DC: National Association of County and City Health Officials, 2010. Consortium for Setting the Public Health Services and Systems Research Agenda. A national research agenda for public health services and systems. Am J Prev Med 2012;42(5 Suppl. 1):S72–S8. Magruder C, Burke M, Hann NE, et al. Using information technology to improve the public health system. J Public Health Manag Pract 2005;11:123–30. Informatics. Dictionary.com. 2012 [cited 10 September 2012]. http://dictionary. reference.com/browse/informatics. Yasnoff WA, O’Carroll PW, Koo D, et al. Public health informatics: improving and transforming public health in the information age. J Public Health Manag Pract 2000;6:67–75. Kraft M. Informatics promotes public health. Chart 2008;105:3–4. Holve E, Papa K, Arnold S, et al. A needs assessment for data and methods in public health systems research. Academy Health 2010. http://www.academyhealth. org/files/interestgroups/phsr/FinalPhsrNAjan2010.pdf Merrill J, Keeling J, Gebbie K. Toward standardized, comparable public health systems data: a taxonomic description of essential public health work. Health Serv Res 2009;44:1818–41. Jacobson PD, Neumann PJ. A Framework to measure the value of public health services. Health Serv Res 2009;44:1880–96. Kun L, Rap P, Merrell R, et al. Improving the health care and public health critical infrastructure. IEEE Eng Med Biol Mag 2008;27:21–5. Friedman DJ, Parrish RG II. The population health record: concepts, definition, design, and implementation. J Am Med Inform Assoc 2010;17:359–66. Lee LM, Gostin LO. Ethical collection, storage, and use of public health data: a proposal for a national privacy protection. JAMA 2009;302:82–4. Massoudi B, Goodman K, Gotham I, et al. An informatics agenda for public health: summarized recommendations from the 2011 AMIA PHI Conference. J Am Med Inform Assoc 2012;19:688–95. Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs 2005;52:546–53. Garrard J. Health sciences literature review made easy: the matrix method. 2nd edn. Sudbury, MA: Jones and Bartlett, 2007. NORC at the University of Chicago. Assessing the status and prospects of state and local health department information technology infrastructure. 2013 [cited 23 May 2013]. http://aspe.hhs.gov/sp/reports/2013/PublicHealthInformatics/hitech_rpt.cfm. Mackiewski S, Taft L. Public health information systems in Minnesota: the uses and hopes among local public health staff. 2006 [cited 31 May 2013]. http://www. health.state.mn.us/e-health/mnphin/locpubsyst.pdf. State of North Carolina. North Carolina Statewide HIE strategic plan. 2010 [cited 24 May 2013]. http://himss.files.cms-plus.com/HIMSSorg/Content/files/Code%20469_ NCHIE_90HT0021_StrategicPlan_rv_083110.pdf. McDaniel AM, Schutte DL, Keller LO. Consumer health informatics: from genomics to population health. Nurs Outlook 2008;56:216–23. State of Michigan. State of Michigan MiHIN share services operational plan. 2010 [cited 24 May 2013]. http://himss.files.cms-plus.com/HIMSSorg/Content/files/Code% 20458_MiHIN_Shared_Services_OS_Plan_4–30-10_320163_7%20Combined.pdf. Lewis SH, Holtry RS, Loschen WA, et al. The collaborative experience of creating the National Capital Region Defense Surveillance Network. J Public Health Manag Pract 2011;17:248–54.


Review 28 29 30

31 32

33 34 35 36 37 38 39

40 41 42

43 44

45

46 47

Savory DJ, Cox KL, Emch M, et al. Enhancing spatial detection accuracy for syndromic surveillance with street level incidence data. Int J Health Geogr 2010;9:1–11. Williams M, Dubowitz T, Jacobson D, et al. Mapping the gaps: ideas for using GIS to enhance local health department priority setting and program planning. Santa Monica, CA: RAND Corporation, 2011. Brownstein JS, Freifield CC, Reis BY, et al. Surveillance sans frontieres: internet-based emerging infectious disease intelligence and the HealthMap project. PLoS Med 2008;5:e151. Goedert J. A new approach to public health data. Health Data Manag 2008;16:16. Nangle B, Xu W, Sundwall DN. Mission-driven priorities: public health in health information exchange. AMIA Annu Symp Proc 2009;2009:468–72. http://www. ncbi.nlm.nih.gov/pmc/articles/PMC2815484/ Kauerauf J. I-NEDSS Davies Award application. 2007 [cited 23 May 2013]. http:// himss.files.cms-plus.com/HIMSSorg/Content/files/davies/2007/ph/ INEDSSfinalDaviesAward.pdf. Hersh J. Pennsylvania’s National Electronic Disease Reporting system Davies Award application. 2004 [cited 25 May 2013]. http://himss.files.cms-plus.com/HIMSSorg/ Content/files/davies2004_publichealth_Pennsylvania.pdf. Guthrie B. Emerging infectious diseases: challanges with worldwide surveillance systems. HIMSS Annual Conference and Exhibition. New Orleans, LA, 2007. Pina J, Turner A, Kwan-Gett T, et al. Task analysis in action: The role of information systems in communicable disease reporting. AMIAAnnual Symposium proceedings/ AMIA Symposium; 2009 May 30, 2013; 2009: 531–5. Turner A, Ramey J, Lee ES. Connecting public health IT systems with enacted work: report of an ethnographic study. AMIA Annual Symposium proceedings/AMIA Symposium; 6 November 2008; Washington, DC; 2008:737–41. Octania-Pole S. New Jersey’s CDRSS Davies Award application. 2008 [cited 31 May 2013]. http://himss.files.cms-plus.com/HIMSSorg/Content/files/davies/2008/ph/New_ Jersey_CDRSS_Davies_2008.pdf. Public Health Information Network. Progress in public health informatics. 2009 [cited 31 May 2013]. http://www.cdc.gov/phin/spotlight/pdf/Progress_in_PH_ Informatics_20090224.pdf. Lawson BM, Fitzhugh EC, Hall SP, et al. Multifaceted syndromic surveillance in a public health department using the early aberration reporting system. J Public Health Manag Pract 2005;11:274–81. Ising A. North Carolina Bioterrorism and Emerging Infection Prevention System Davies Award application. 2005 [cited 23 May 2013]. http://himss.files.cms-plus. com/HIMSSorg/Content/files/2005/NorthCarolinaBEIPS.pdf. Le L. New York State environmental and public health tracking network data exchange systems Davies Award application. 2006 [cited 23 May 2013]. http:// himss.files.cms-plus.com/HIMSSorg/Content/files/davies/2006/Davies2006PH%20_ NYSEPHTN.pdf. State of Indiana. Strategic operational plan for health information exchange in the state of Indiana. 2010 [cited 24 May 2013]. http://himss.files.cms-plus.com/ HIMSSorg/content/files/Code%20450_IndianaSHIECAP-OSPlan.pdf. Smith V, Gifford K, Kramer S, et al. State e-health activities in 2007: Findings from a state survey. [Report] 2008 [cited 5 April 2012]; 1104. http://www. commonwealthfund.org/usr_doc/1104_Smith_state_e-hlt_activities_2007_findings_ st.pdf. Banger AK, Dullabh P, Eichner J, et al. Lessons learned from AHRQ’s state and regional demonstrations in health information technology. Rockville, MD: Agency for Healthcare Research and Quality, 2010. Landis MJ, Kratz S, Spaans-Esten L, et al. SPHERE: Tracking public health improvement with electronic records. Wis Med J 2007;106:116–19. National Committee on Vital and Health Statistics. Frequently asked questions about medical and public health registries. [cited 10 September 2012]; Department of Health and Human Services Website. http://www.ncvhs.hhs.gov/ 9701138b.htm.

48 49 50

51 52 53

54 55 56 57 58

59 60

61 62

63 64 65 66 67 68 69 70


California Immunization Registry. California immunization registry: Connected and Protected. 2013 [cited 31 May 2013]. http://www.cairweb.org/index.php? option=com_content&task=view&id=28&Itemid=. Wine M, Clark J. Health IT in Government: Transforming Health Care and Empowering Citizens. 2006 [cited 24 May 2013]; http://himss.files.cms-plus.com/ HIMSSorg/Content/files/healthit.pdf. Pare N. Utah Statewide Immunization Information System Davies Award Application. 2004 [cited 24 May 2013]. http://himss.files.cms-plus.com/HINSSorg/Content/files/ davies2004_publichealth_utah.pdf. McHugh M, Martinez R, Kilman R, et al. Local Health Departments Lead Efforts Against Vaccine-preventable Diseases; 2000, Report No.: PP00–24. Feuchtbaum L, Dowray S, Lorey F. The context and approach for the California newborn screening short- and long-term follow-up data system: preliminary findings. Genet Med 2000;12(Suppl. 12):S242–50. Smith B. Building Bridges: Reducing Chronic Disease in California. [cited 31 May 2013]. http://www.cdph.ca.gov/data/informatics/policy/Documents/ CmnGrndCDPHMeetApr. Ringle JS, Trentacost E, Lurie N. Tisk communication in the EArly stages of the H1N1 alert: how effective are state and local public health departments. CA: RAND Corporation, 2009. Creek J. A standard terminology for occupational therapy. Br J Occup Ther 2006;69:202–8. Foldy S. Inventory of electronic health information exchange in Wisconsin, 2006. Wis Med J 2006;106:120–5. Shapiro JS, Mostashari F, Hripcsak G, et al. Using health information exchange to improve public health. Am J Public Health 2011;101:616–23. HIMSS Health Care Information Exchange Open Source Task Force. Open Source Software Being Used for Health Information Exchange. 2010 [cited 24 May 2013]. http://himss.files.cms-plus.com/HIMSSorg/Content/files/ 2010-1020OpenSourceTFWhitePaper.pdf. Health IT News. Health Bridge: Transforming Healthcare Through HIE. 2009 [cited 23 May 2013]. http://www.himss.org/News/NewsDetail.aspx?ItemNumber=5326. Ohio Health Information Partnership. HIE State Plan: Strategic and Operational Plans for a Statewide Health Information Exchange. 2011 [cited 24 May 2013]. http:// himss.files.cms-plus.com/HIMSSorg/Content/files/Code%20471_OHIP_HIT% 20Strategic%20Plan.pdf. Heisey-Grove DM, Church DR, Haney GA, et al. Enhancing surveillance for Hepatitis C through public health informatics. Public Health Rep 2011;126:113–18. California Department of Public Health. Meaningful Use and Public Health. 2011 [cited 23 May 2013]. http://himss.files.cms-plus.com/HIMSSorg/Content/files/Code% 20441-%20CA%20Dept%20of%20Public%20Heatlh_ MeaningfulUseAndPublicHealth20110316.pdf. Omaha System. The Omaha System: Solving the Clnical Data-information Puzzle. 2011 [cited 5 April 2012]. http://www.omahasystem.org/. Monsen KA, Fulkerson JA, Lytton AB, et al. Comparing maternal child health problems and outcomes across public health nursing agencies. Matern Child Health J 2010;14:412–21. Monsen KA, Radosevich DM, Kerr MJ, et al. Public health nurses tailor interventions for families at risk. Public Health Nurs 2011;28:119–28. Monsen KA, Sanders A, Yu F, et al. Family home visiting outcomes for mothers with and without intellectual disabilities. J Intellect Disabil Res 2011;55:484–99. Mirhaji P. Public health surveillance meets translational informatics: a desiderata. J Assoc Lab Automation 2009;14:157–70. Srinivasan A, Kunapareddy N, Mirhaji P, et al. Semantic web representation of LOINC: an ontological perspective. AMIA Annu Symp Proc 2006;2006:1107. Mirhaji P, Byrne S, Kunapareddy N, et al. Semantic approach for text understanding of chief complaints data. AMIA Annu Symp Proc 2006;2006:1033. Monsen KA, Bekemeier BP, Newhouse R, Scutchfield FD. Development of a public health nursing data infrastructure. Public Health Nurs 2012;29:343–51.

e27

Research and applications

Exploring the sociotechnical intersection of patient safety and electronic health record implementation Derek W Meeks,1 Amirhossein Takian,2 Dean F Sittig,3 Hardeep Singh,4 Nick Barber5 1

Baylor College of Medicine, Department of Family and Community Medicine, VA HSR&D Center of Excellence, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA 2 Division of Health Studies, School of Health Sciences and Social Care, Brunel University London, Uxbridge, UK 3 University of Texas School of Biomedical Informatics and UTMemorial Hermann Center for Healthcare Quality and Safety, Houston, Texas, USA 4 Houston VA HSR&D Center of Excellence, Michael E. DeBakey Veterans Affairs Medical Center, Baylor College of Medicine, Department of Medicine, Section of Health Services Research, Houston, Texas, USA 5 Department of Practice and Policy, The UCL School of Pharmacy, London, UK Correspondence to Dr Derek W Meeks, Houston VA HSR&D Center of Excellence (152), 2002 Holcombe Boulevard, Houston, TX 77030, USA; [email protected] Received 25 February 2013 Revised 28 August 2013 Accepted 2 September 2013 Published Online First 19 September 2013

ABSTRACT Objective The intersection of electronic health records (EHR) and patient safety is complex. To examine the applicability of two previously developed conceptual models comprehensively to understand safety implications of EHR implementation in the English National Health Service (NHS). Methods We conducted a secondary analysis of interview data from a 30-month longitudinal, prospective, case study-based evaluation of EHR implementation in 12 NHS hospitals. We used a framework analysis approach to apply conceptual models developed by Sittig and Singh to understand better EHR implementation and use: an eight-dimension sociotechnical model and a three-phase patient safety model (safe technology, safe use of technology, and use of technology to improve safety). Results The intersection of patient safety and EHR implementation and use was characterized by risks involving technology (hardware and software, clinical content, and human–computer interfaces), the interaction of technology with non-technological factors, and improper or unsafe use of technology. Our data support that patient safety improvement activities as well as patient safety hazards change as an organization evolves from concerns about safe EHR functionality, ensuring safe and appropriate EHR use, to using the EHR itself to provide ongoing surveillance and monitoring of patient safety. Discussion We demonstrate the face validity of two models for understanding the sociotechnical aspects of safe EHR implementation and the complex interactions of technology within a healthcare system evolving from paper to integrated EHR. Conclusions Using sociotechnical models, including those presented in this paper, may be beneficial to help stakeholders understand, synthesize, and anticipate risks at the intersection of patient safety and health information technology. BACKGROUND

To cite: Meeks DW, Takian A, Sittig DF, et al. J Am Med Inform Assoc 2014;21:e28–e34. e28

The USA federal government, through stimulus spending and the Affordable Care Act, is encouraging widespread implementation of health information technology (HIT) to improve healthcare quality and patient safety.1 These efforts are founded on expectations of increased coordination of care, improved follow-up, and increased efficiency throughout the continuum of care.2 However, research suggests that technology may lead to new uncertainties and risks for patient safety through disrupting established work patterns, creating new risks in practice, and encouraging workarounds.3–10 In particular, the increasing adoption of electronic health records (EHR) has

revealed potential safety implications related to EHR design, implementation, and use.11–15 These risks are not related solely to the technological features of the EHR but may involve EHR users and their workflows, aspects of the organizations in which they function, and the rules and regulations that govern or oversee their activities. Furthermore, patient safety risks associated with EHR may vary along the EHR adoption and implementation timeline. Given the complexity and multifaceted nature of EHR-related safety risks, a comprehensive model is needed to understand and anticipate these risks in a sociotechnical context. Sittig and Singh16 17 developed an eightdimensional sociotechnical model to study the safety and effectiveness of HIT at all levels of design, development, implementation, use, and evaluation. Four earlier sociotechnical models informed the development of the eight-dimensional model: the model of Henriksen et al,18 the framework for analyzing risk and safety of Vincent et al,19 the systems engineering initiative of patient safety of Carayon et al,20 and the interactive sociotechnical analysis of Harrison et al.21 The model’s dimensions represent interdependent domains of an EHR-enabled healthcare system: hardware and software; clinical content; human–computer interface; people; workflow and communication; internal organization policies, procedures, and culture; external rules, regulations, and pressures; system measurement and monitoring (figure 1).16 17 For example, failure to follow up a critical laboratory result could be attributable to a software error that prevented transmission of the laboratory result to the correct provider (hardware and software), faulty display of information in the provider’s EHR window (human–computer interface), or inadequate coordination of roles within the clinical care team (workflow and communication).22 Efforts to improve EHR-related patient safety rely on identification of underlying risks as well as an appreciation of contributing areas of vulnerability (eg, people, organization policies and procedures, or system measurement).23 The sociotechnical intersection of patient safety and EHR is complex. First, this intersection conceptualizes the healthcare system as an evolving, complex adaptive system in which safety risks often emerge from users’ interactions with the EHR that lead to new clinical workflow processes. These new workflow processes involve different environmental (eg, human interaction with physical devices and their workspace),24 cultural (eg, role changes of clinicians in the EHR-enabled workflow),25 or even sociopolitical (eg, clinical power structure) factors.26 Second, these safety risks are multifactorial and rarely involve a single contributing factor.

Meeks DW, et al. J Am Med Inform Assoc 2014;21:e28–e34. doi:10.1136/amiajnl-2013-001762


Figure 1 Diagram illustrating the interaction between the eight-dimension sociotechnical and three-phase electronic health record (EHR) safety models. The goal is for organizations to move from a paper-based medical record system ‘up the escalator’ to become an EHR-enabled healthcare system. Within each phase of the three-phase model, all eight dimensions of the sociotechnical model come into play. HIT, health information technology.

Third, improving patient safety within an EHR-enabled healthcare system requires a journey in which the sociotechnical infrastructure and functionalities evolve over time. The sociotechnical model does not itself convey how it fits into the continuum of HIT safety that includes safe transition from paper to fully integrated EHR. Therefore, to understand the intersection of EHR and patient safety, Sittig and Singh27 further proposed a three-phase model to account for the variation in the stages of implementation, levels of complexity, and related patient safety concerns within an EHR-enabled healthcare system. The first phase is concerned with safety events that are unique and specific to technology (ie, unsafe technology), which often emerge early in the process of implementation. The second phase addresses unsafe or inappropriate use of technology as well as unsafe changes in the overall workflow that emerge due to technology use. The third phase addresses use of technology proactively to identify and monitor potential safety concerns before harm occurs to the patient. While the boundaries between the phases may not always be distinct, the threephase model could be useful for goal setting and identification of threats to patient safety.27 In light of emerging and often novel risks associated with EHR, comprehensive models such as those described above are needed to assess the variety of safety threats and near misses. Such efforts will advance the understanding of EHR-related safety events to allow for the planning of safer systems and processes. Previously, we conducted a longitudinal, sociotechnical evaluation of the implementation and adoption of EHR in English National Health Service (NHS) hospitals.28 29 As part of that study, we conducted interviews that yielded a large volume of open-ended comments, some of which reflected concerns about patient safety. That study demonstrated the importance of considering the sociotechnical context of EHR implementation, although the UK investigators did not apply a

formal framework to assess patient safety until now.30 Our aim was to explore and illustrate the application of the eightdimensional sociotechnical and three-phase EHR safety models to organize and interpret EHR-related patient safety concerns elicited during evaluation. Rather than conduct hypothesis testing, our goal was to highlight the ‘real-world’ usefulness of practical sociotechnical approaches to ensuring safe and effective EHR implementation and future use.

MATERIALS AND METHODS Setting and design In 2002, the UK Department of Health decided to implement three centrally procured national EHR applications, both made to order and commercially available, in the English NHS hospitals. Implementation was to be supported by a small number of centrally contracted local service providers, each responsible for delivering standard software systems to local hospitals, ensuring system integration, interoperability, and national connectivity within a geographical region. This was part of an overall US $19.6 (£12.7) billion strategic initiative to transform the NHS’s HIT infrastructure into an integrated set of electronic systems connected to national databases and a messaging service (the ‘NHS spine’).30 The data presented here were extracted from a 30-month (September 2008 to March 2011) prospective, longitudinal, and real-time case study-based evaluation during EHR implementation and adoption in 12 hospitals (nine acute and three mental health).31 The original research proposal was approved as a service evaluation by a NHS ethics committee.

Data collection The methods of data collection have been described elsewhere.28–30 Interviews were conducted at all stages of EHR implementation and adoption from initial awareness and planning to sustained use. In order to explore the implementation


e29

Research and applications processes across hospitals, interviewers sought to determine the organizational activities undertaken and their consequences for professional roles, workflows, and clinical practices. Participating hospitals were purposefully selected according to their projected implementation timelines and included a range of hospital types (ie, teaching, non-teaching, acute care, and mental health) to allow comparisons. The original investigators conducted semistructured interviews with a broad range of stakeholders: managers, implementation team members, information technology (IT) staff, junior and senior physicians, nurses, allied health professionals, administrative staff, external implementation-related stakeholders, and software developers. The six interviewers did not explicitly ask interviewees questions regarding patient safety. Interviews were audio-recorded and transcribed verbatim. Data were anonymized by redacting information that identified the individual participant or site.

in the data; however, they are presented within the domain judged to be most involved with the safety concern. Some dimensions of the sociotechnical model are better represented than others in the dataset, as demonstrated by the mappings of phases and dimensions in table 2. Similarly, most data were mapped to phases one and two of the three-phase model. Table 3 provides a high-level summary of the safety concerns present in the data. This table reveals that certain dimensions have heterogeneity while others have more homogeneous concerns expressed. For instance, in hardware and software concerns regarding EHR availability were prominent in phase one; data sharing and system–system interface issues were also seen. Conversely, in clinical content, most concerns were regarding phase two, in which users experienced difficulties (perceived or actual) with order entry through the EHR. We present the data according to the three-phase model to illustrate safety risks that emerged as most relevant to each phase of implementation.

Data analysis

Phase one

One author (AT) asked the original UK investigators to review transcripts for content related to patient safety. Out of 480 interviews conducted in the evaluation, AT confirmed 49 interviews in which patient safety content was present. The data were then analyzed using a framework analysis approach, a qualitative research method that has pre-set aims but accommodates new themes from the data.32 Framework analysis has five stages: familiarization; thematic analysis; indexing (coding); charting; and mapping and interpretation. We began by reviewing and summarizing relevant quotes regarding EHR-related patient safety concerns. Using the eight dimensions of the sociotechnical model as the framework, three reviewers (DWM, DFS, and HS) indexed the data. While acknowledging the interrelatedness of the models, for clarity we coded the dimension and phase most directly implicated in the safety concern. The data were then arranged according to the three-phase model (charting). This analysis was performed iteratively until consensus was obtained among the reviewers. Interrater reliability was not assessed as the aim of the study was to explore themes of patient safety and EHR implementation (mapping and interpretation), not rigorous classification with the two models. ATLAS.ti 6 by ATLAS.ti Scientific Software Development (http://www. atlasti.com) was used for data management.

RESULTS The interviewees’ roles in EHR implementation and the number of hospital represented are shown in table 1. The sociotechnical domains were not mutually exclusive, but were seen to interact

Table 1

Interviewee role and hospital representation

Interviewee role

No of interviewees

No of hospitals represented

Senior manager EHR implementation/IT team Healthcare practitioners Clinical managers Administrators Strategic health authorities Local IT service providers EHR software developers Total

7 9 16 6 3 3 2 3 49

6 6 6 5 5 N/A N/A N/A N/A

EHR, electronic health record; IT, information technology.

e30

In accordance with the model, phase one EHR safety concerns were unique and specific to technology. Within the framework of the sociotechnical model, specific comments were frequently mapped to the domains of hardware and software, clinical content, and human–computer interface. An example of a phase one safety concern regarding hardware and software was the acknowledgment of an insufficient data center and back-up procedures. ‘The danger with [hospitals] doing their own thing is that instead of having a proper data centre meeting certain standards you get it sort of in a shed out the back sort of thing and it’s not 24/7, it’s not resilient, it doesn’t have a fail over site that it can go to, it doesn’t have a fail over within, guaranteed two hours service level and it’s up to what they can negotiate with the supplier, so cost effectively it’s not as cost effective and from a resilience and safety point of view it’s not as good. I think the safety is probably one of the key things that doing it centrally and nationally is a lot more secure.’ IT Manager, Site H Sociotechnical model: hardware and software

A recurring safety concern, also related to hardware and software, was implementation of an EHR without necessary software features to support a clinical workflow that demanded those features. ‘If you think someone’s at risk of suicide and you kind of tick the box there and put some text in, you expect that will bounce

Table 2 Types of safety concerns categorized by sociotechnical dimensions and phases of EHR implementation and use

Hardware and software Clinical content Human–computer interface People Workflow and communication Internal organization policies, procedures, and culture External rules, regulations, and pressures System measurement and monitoring

Phase 1

Phase 2

Phase 3

11 3 4 1 1 3

2 7 4 4 6 0

0 0 0 0 0 0

2 0

0 0

0 1

EHR, electronic health record.


Research and applications Table 3

Summaries of interview data demonstrating safety concerns by phase and dimension

Sociotechnical dimension

Phase of use

Summary of safety concern

Hardware and software

Phase one

▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸ ▸

Phase two Clinical content

Phase one Phase two

Human–computer interface

Phase one Phase two

People

Phase one Phase two

Workflow and communication

Phase one Phase two

Internal organizational policies, procedures, and culture

Phase one

External rules, regulations, and pressures

Phase one

System measurement and monitoring

Phase three

Problems with EHR availability (login or network access) (n=4) Lack of basic EHR functionality (n=4) Problems related to data maintenance, sharing, or security (n=3) Problems with accessing appropriate clinical information Problem with system–system interfaces Undeveloped or non-standardized clinical content in the EHR (n=3) Parallel use of paper and EHR Problems or difficulties with use of order entry (n=6) User interface too burdensome or error prone for data entry (n=4) User interface does not support clinical workflow (n=3) Risk of copy and paste functionality Data security concerns Users sharing EHR access (n=3) Poor training leads to improper use Errors related to appointment scheduling applications EHR not integrated into clinical workflow EHR causes delays in work (n=3) Laboratory result routing unreliable (n=2) Multiple medical record numbers per patient increase risk of wrong selection Data confidentiality risks Local IT budget must support ongoing IT infrastructure requirements National IT budgeting important for safe EHR use after implementation Complexity of software and business models of vendors may affect future use Challenges and benefits of EHR-based quality reporting

EHR, electronic health record; IT, information technology.

through to the care plan module so they could then put a response to it and it stops things getting lost and what have you. It doesn’t do anything like that. When you identify needs it doesn’t bounce it through to the care planning functionality so that it’s already there so that you know what you’ve got to address, and if you forget to transfer the fact that this person is at risk of stabbing someone, then the system doesn’t offer any safeguards to drag it through.’ Healthcare provider, Site G Sociotechnical model: hardware and software

In contrast to the absence of a feature, some users identified a design or implementation they perceived to be error prone. For instance, users described EHR hardware and software issues or human–computer interface problems that contributed to patient safety concerns. ‘We’ve had a couple of instances in Radiology where we’ve not been able to cancel requests and patients have been scanned twice, so they’ve had a double exposure of radiation.’ Director, Site E Sociotechnical model: hardware and software ‘…[It’s] terribly easy to make a mistake, because you can bring up several Maria Smiths and if you are not careful and you don’t look at the date of birth, because they are just a list and they are right on top of each other, you could pick the wrong one.’

The prevailing theme from the data was the risk introduced when EHR was placed within a clinical context that did not facilitate safe use. For instance, a phase two concern was the improper integration of computers into clinical encounters in which EHR use cannot occur simultaneously with delivery of care (ie, in procedural or sterile areas). Another example was the barrier associated with the requirement to sign into the EHR, which resulted in password sharing and generic password use. ‘…you go to your colleague and you say, log me in and then you use other people’s cards. They had to have this generic access in A&E (emergency department) because actually this was a crazy situation. It broke all the rules for information and governance and data protection.’ Manager, Site E Sociotechnical model: people

Certain EHR features, such as copy and paste, were recognized as safety risks due to inappropriate use. In the example below, pathology specimens were mislabeled and the EHR was understood, in this instance, to increase risk of patient harm. ‘The ability to copy and paste in fields is dangerous. Incorrect details are being pasted into incorrect patient fields (i.e., prostate as specimen details in female patient request or missed miscarriage in clinical details for male patient).’ Healthcare provider, Site D

Receptionist, Site E Sociotechnical model: human–computer interface

Phase two In this phase patient safety is compromised through unsafe use of technology or unsafe changes in workflow. The most common dimensions in this phase were workflow and communication, people, human–computer interface, and clinical content.

Sociotechnical model: human–computer interface

Some workflow and communication problems were specific to certain practice areas for which use of the EHR, as implemented, was thought to be particularly ill suited. For instance, EHR users in the mental health hospitals felt the effort needed to document in the EHR was not only potentially unsafe, but impeded the ability to see patients in a timely manner.


e31

Research and applications ‘The psychiatric assessments are quite lengthy and there are quite a lot of notes that go with it. Doctors are not going to be able to do it while they are with the patient, because of issues like risk. … So it’s going to increase the time spent and you are then delayed seeing the next patient which is I think the big anxiety.’ Doctor, Site M Sociotechnical model: workflow and communication

Finally, as clinical workflow and communication was noted to become error prone when the medical record was in transition from paper to electronic form, clinical content also arose as an area of potential risk. ‘We have to print out now anyway and put into the paper notes because not everyone is on [software X]… But I can also see the fact that when everyone is on it you won’t have to do it.’ Healthcare provider, Site H Sociotechnical model: clinical content

Phase three This phase addresses EHR use to monitor and identify safety concerns before patients are harmed. This ultimate use of technology was reflected in only one interview. The participant noted the difficulty in reporting quality measures before EHR implementation and the potential advantages of an EHR-enabled healthcare system. ‘If everybody is using the same system, they have the same functionality available to them. There is only a limited amount of ways that you can record information from reporting and performance indicator and assessment sort of point of view. We often have difficulty meeting certain targets, because we don’t have a way of reporting it. It’s a real struggle. But, at least if everybody has the same struggle then you are comparable to everybody else and there aren’t these gaps. You are more easily able to make a comparison across organizations. I think that’s an advantage.’ Manager, Site M Sociotechnical model: system measurement and monitoring

DISCUSSION IT and EHR could potentially have large quality and safety benefits. However, there is increasing acknowledgement that the use of EHR could introduce unintended risks, and simultaneous efforts are needed to establish safe EHR design and implementation.14 As with other patient safety issues, a piecemeal, reactive approach to identifying and correcting EHR-related safety issues is unlikely to be efficient or effective. Systematic analysis of EHR-related safety concerns must be performed within a context that accounts for the evolving sociotechnical infrastructure and functionality that defines the journey to a safe EHR-enabled healthcare system. In this analysis from the evaluation of the NHS’s implementation of EHR, we attempted to demonstrate the ‘real-world’ usefulness of analyzing spontaneously reported safety concerns through two operational models related to HIT: an eight-dimension sociotechnical model and a three-phase EHR safety model. A sociotechnical approach may allow developers, IT managers, administrators, clinicians, and others to understand risks in the development, implementation, and use of EHR and HIT while accounting for complex interactions of technology within the healthcare system. Further application of these models may be helpful as government bodies e32

make HIT safety a greater priority within clinical environments.33 The three-phase model was useful to understand the context of safety risks given that our sites were still early in their EHR implementation journey, and therefore both phases one and two were sufficiently represented. Unfortunately, we were unable to identify many activities within phase three of the model. Furthermore, the eight-dimension model was found to have face validity to understand and classify EHR-related safety concerns within the technical, social, or clinical context in which they occur. Applications of such models could be useful to inform or prioritize implementation efforts. For example, we found, as anticipated, that phase one safety concerns arose most commonly in the hardware and software domains of the sociotechnical model. Therefore, organizations should ensure that proper hardware requirements are in place before EHR implementation (eg, adequate number of workstations, appropriate data center). Phase two concerns were frequently mapped to clinical content and workflow and communication. Phase two priorities could therefore involve understanding and changing the clinical workflow or the EHR configuration to facilitate safe care. Organizational and leadership factors are commonly recognized as important for success,34 but we suggest that understanding the local culture, workflow, and potential impact on productivity is equally necessary.31 35 Our combined model also suggests that as an organization evolves, both patient safety improvement activities and patient safety hazards also evolve from concerns about safe functionality and ensuring safe and appropriate use, to using the EHR itself to provide ongoing surveillance and monitoring of patient safety. Further exploration of this evolution could inform sociotechnical approaches to improving safety in future large-scale EHR implementations. The strengths of this qualitative analysis include the large scale of the EHR implementation and evaluation involving simultaneous interviews. Other qualitative investigations have analyzed EHR implementations, but primarily focused on barriers to implementation, system-wide challenges, or overall benefits and concerns rather than patient safety.35–39 Our high-level approach differs from that of other classification systems, notably that of Magrabi and colleagues,40 41 which includes both technical and human elements.42 For instance, the human elements it encompasses are generally related to the direct use of the computer, and to actions closely linked in time to the error at hand. By contrast, the model used in this paper encompasses a broader range of sociotechnical factors (eg, workflow and organizational factors) that are more temporally dissociated. Each approach might have its own advantages and limitations depending on what type of data is available for analysis, the depth and breadth of available data, and the rationale of why the analysis was undertaken. We also build on previous work demonstrating the use of sociotechnical models. For instance, in our previous work, we found this sociotechnical model applicable in specific clinical contexts (eg, test results and referral communication),43–48 but until this analysis, a formal model to study patient safety issues with EHR implementation was lacking (including within the previous body of work done by the UK investigators). Our sociotechnical model was adapted by the Institute of Medicine in their report on HIT safety albeit without the detailed technology dimensions that we believe are essential to appreciate the nuances involved with EHR use.14 To our knowledge, there are few if any practical models that are specific to HIT that provide guidance in this area. The combination of the sociotechnical model with the three-phase


Research and applications model allows us to view EHR safety from a systems engineering perspective. Through this lens, interaction of the two models is considered from four fundamental perspectives of complex systems: scale (quantitative size); function (the reason for existence); structure (the interconnection of system elements); and temporality (scales of time).49 In our combined model (figure 1), the phases differ in their ‘sociotechnical’ scale, function, structure, and temporality. Within each phase, the eightdimensional sociotechnical model can be used to understand unique safety issues. For instance, a phase one software problem may encompass a single function such as inappropriate matching of blood products due to a software coding or content error. While in phase three, errors in blood typing would be identified in real time through an organization-wide monitoring program that alerts clinicians whenever the blood type of a patient has ‘changed’. In other words, in phase one, we view the sociotechnical scale of the problem to be much more isolated and contained, while in the latter phases, the scale increases significantly: including users and the physical environment in phase two and, potentially, the entire organization in phase three. Another example is the different skills and roles of people involved in phase one who are responsible for configuring the hardware (eg, moving database servers to a physically secure location) and software (eg, setting up encryption keys on the periodic back-up systems) to ensure patient confidentiality. While in phase three, people ensuring patient safety would probably include informaticians developing surveillance and monitoring capabilities to identify potential breaches of patient confidentiality or health information management and human resource professionals to investigate these potential breaches and enforce policies to protect health information.50 51 The limitations of this study include the interview protocol’s lack of specificity to patient safety issues and the inability to assess impact on patient safety. The interviewers broadly focused on EHR implementation and did not intentionally seek detailed responses about patient safety. While safety concerns arose in several interviews, the interviews did not necessarily elicit the full range of potential EHR-related safety concerns. Although the concerns of those involved during implementation appeared appropriate, no additional effort was made to validate these concerns. As this was a secondary analysis of previously collected data, interview data regarding safety potentially could have been overlooked during the initial review by the original UK investigators because the data collection did not anticipate this use. The case study design may have reduced the generalizability of the findings, but despite different EHR software, cultures, and methods of healthcare delivery, we believe the usefulness of our analysis is the potential ability of the two models to identify EHR-related safety concerns and priorities to address them.

beneficial to help stakeholders understand, synthesize, and anticipate risks within the continuum of HIT safety that includes safe transition from paper to integrated EHR. Acknowledgements The authors are very grateful to the NHS hospitals that participated in their evaluation and to all individuals who kindly gave their time. They would like to thank their colleagues on the NHS CRS evaluation team, led by Professor Aziz Sheikh. They wish to thank the independent project steering committee, which was chaired by Professor David Bates, and also Michael W. Smith, PhD, for his contribution to the systems analysis approach. Contributors DWM, AT, DFS, HS and NB participated in the conception and design of this study. AT and NB conducted data collection and primary analysis to identify patient safety relevant data from the original UK study. DWM, DFS and HS participated in the data analysis and interpretation of results. DWM wrote the initial draft. All authors performed critical review of drafts and approved the submitted version. Funding This paper is independent research commissioned by the NHS Connecting for Health Evaluation Programme (005 08/S0709/97) led by Professor Richard Lilford. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health. HS is supported by the VA National Center of Patient Safety, Agency for Health Care Research and Quality, and in part by the Houston VA HSR&D Center of Excellence (HFP90-020). DWM is supported by the Baylor College of Medicine Department of Family and Community Medicine post-doctoral fellowship program and the Ruth L. Kirschstein national research service award (T32HP10031). These sources had no role in the preparation, review, or approval of the manuscript. Competing interests None. Ethics approval The NHS Connecting for Health Evaluation Programme received approval from a NHS ethics committee. Provenance and peer review Not commissioned; externally peer reviewed. Data sharing statement The authors of the original study may be contacted for the dataset.

REFERENCES 1 2 3 4 5 6 7 8 9 10

CONCLUSION Examining the intersection of HIT and patient safety with practical conceptual models can advance the EHR-enabled healthcare system towards the goal of improving patient safety. ‘Safe technology’ and ‘safe use of technology’ are necessary for efforts to improve and monitor patient safety; for example, phase three of the EHR-enabled healthcare system. We demonstrated how the combined use of two models has face validity to facilitate understanding of the sociotechnical aspects of safe EHR implementation and the complex interactions of technology within the evolving healthcare system. Our sociotechnical approach, along with other existing frameworks, may be

11

12 13 14 15

Blumenthal D. Stimulating the adoption of health information technology. N Engl J Med 2009;360:1477–9. Schiff GD, Bates DW. Can electronic clinical documentation help prevent diagnostic errors? N Engl J Med 2010;362:1066–9. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004;11:104–12. Balka E, Doyle-Waters M, Lecznarowicz D, et al. Technology, governance and patient safety: systems issues in technology and patient safety. Int J Med Inform 2007;76(Suppl. 1):S35–47. Bates W, Cohen M, Leape L, et al. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc 2001;8:299–308. Coleman RW. Translation and interpretation: the hidden processes and problems revealed by computerized physician order entry systems. J Crit Care 2004;19:279–82. Hundt AS, Adams JA, Schmid JA, et al. Conducting an efficient proactive risk assessment prior to CPOE implementation in an intensive care unit. Int J Med Inform 2013;82:25–38. Koppel R. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197–203. Patterson ES, Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inform Assoc 2002;9:540–53. Pirnejad H, Niazkhani Z, van der SH, et al. Impact of a computerized physician order entry system on nurse–physician collaboration in the medication process. Int J Med Inform 2008;77:735–44. Singh H, Mani S, Espadas D, et al. Prescription errors and outcomes related to inconsistent information transmitted through computerized order entry: a prospective study. Arch Intern Med 2009;169:982–9. Myers RB, Jones SL, Sittig DF. Review of reported clinical information system adverse events in US Food and Drug Administration databases. Appl Clin Inform 2011;2:63–74. Weiner JP, Kfuri T, Chan K, et al. “e-Iatrogenesis”: the most critical unintended consequence of CPOE and other HIT. J Am Med Inform Assoc 2007;14:387–8. IOM (Institute of Medicine). Health IT and patient safety: building safer systems for safer care. Washington, DC: The National Academies Press, 2012. Harrington L, Kennerly D, Johnson C. Safety issues related to the electronic medical record (EMR): synthesis of the literature from the last decade, 2000–2009. J Healthc Manag 2011;56:31–43.


e33

Research and applications 16 17

18 19 20 21 22 23

24 25 26

27 28 29

30 31 32 33

e34

Sittig DF, Singh H. Defining health information technology-related errors: new developments since to err is human. Arch Intern Med 2011;171:1281–4. Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010;19 (Suppl. 3):i68–74. Henriksen K, Kaye R, Morisseau D. Industrial ergonomic factors in the radiation oncology therapy environment. Advances in Industrial Ergonomics and Safety V. Taylor and Francis, 1993:325. Vincent C, Taylor-Adams S, Stanhope N. Framework for analysing risk and safety in clinical medicine. BMJ 1998;316:1154–7. Carayon P, Schoofs Hundt A, Karsh BT, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006;15(Suppl. 1):i50–8. Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care—an interactive sociotechnical analysis. J Am Med Inform Assoc 2007;14:542–9. Singh H, Wilson L, Petersen LA, et al. Improving follow-up of abnormal cancer screens using electronic health records: trust but verify test result communication. BMC Med Inform Decis Mak 2009;9:49. Singh H, Thomas EJ, Sittig DF, et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? Am J Med 2010;123:238–44. Koppel R, Wetterneck T, Telles JL, et al. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc 2008;15:408–23. Campbell EM, Guappone KP, Sittig DF, et al. Computerized provider order entry adoption: implications for clinical workflow. J Gen Intern Med 2009;24:21–6. Ash JS, Sittig DF, Campbell E, et al. An unintended consequence of CPOE implementation: shifts in power, control, and autonomy. American Medical Informatics Association, 2006:11. Sittig DF, Singh H. Electronic health records and national patient-safety goals. N Engl J Med 2012;367:1854–60. Robertson A, Cresswell K, Takian A, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ 2010;341:c4564. Sheikh A, Cornford T, Barber N, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in “early adopter” hospitals. BMJ 2011;343:d6054. Takian A, Petrakaki D, Cornford T, et al. Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems. BMC Health Serv Res 2012;12:105. Takian A, Sheikh A, Barber N. We are bitter, but we are better off: case study of the implementation of an electronic health record system into a mental health hospital in England. BMC Health Serv Res 2012;12:484. Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ 2000;320:114–16. Health Information Technology Safety Action & Surveillance Plan. The Office of the National Coordinator of Health Information Technology 2013 July 2 [cited 2013 Jul 11]. http://www.healthit.gov/sites/default/files/safety_plan_master.pdf

34 35 36 37 38 39 40 41 42 43 44 45

46 47 48 49 50 51

Takian A. Envisioning electronic health record systems as change management: the experience of an English hospital joining the National Programme for Information Technology. Stud Health Technol Inform 2012;180:901–5. Scott JT, Rundall TG, Vogt TM, et al. Kaiser Permanente’s experience of implementing an electronic medical record: a qualitative study. BMJ 2005;331:1313–16. Greiver M, Barnsley J, Glazier RH, et al. Implementation of electronic medical records: theory-informed qualitative study. Can Fam Physician 2011;57:e390–7. Zwaanswijk M, Verheij RA, Wiesman FJ, et al. Benefits and problems of electronic information exchange as perceived by health care professionals: an interview study. BMC Health Serv Res 2011;11:256. Yoon-Flannery K, Zandieh SO, Kuperman GJ, et al. A qualitative analysis of an electronic health record (EHR) implementation in an academic ambulatory setting. Inform Prim Care 2008;16:277–84. Spetz J, Burgess JF, Phibbs CS. What determines successful implementation of inpatient information technology systems? Am J Manag Care 2012;18:157–62. Magrabi F, Ong MS, Runciman W, et al. An analysis of computer-related patient safety incidents to inform the development of a classification. J Am Med Inform Assoc 2010;17:663–70. Magrabi F, Ong MS, Runciman W, et al. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012;19:45–53. Sparnon E, Marella WM. The role of the electronic health record in patient safety events. Harrisburg, PA: Pennsylvania Patient Safety Authority, 2012. Singh H, Esquivel A, Sittig DF, et al. Follow-up actions on electronic referral communication in a multispecialty outpatient setting. J Gen Intern Med 2011;26:64–9. Singh H, Spitzmueller C, Petersen NJ, et al. Primary care practitioners’ views on test result management in EHR-enabled health systems: a national survey. J Am Med Inform Assoc 2013;20:727–35. Hysong SJ, Sawhney MK, Wilson L, et al. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis Mak 2011;11:22. Singh H, Thomas EJ, Mani S, et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med 2009;169:1578–86. Singh H, Spitzmueller C, Petersen NJ, et al. Information overload and missed test results in electronic health record based settings. JAMA Intern Med 2013;173:702–4. Sittig DF, Singh H. Improving test result follow-up through electronic health records requires more than just an alert. J Gen Intern Med 2012;27:1235–7. De Weck OL, Roos D, Magee CL. Engineering systems: meeting human needs in a complex technological world. MIT Press, 2011. Boxwala AA, Kim J, Grillo JM, et al. Using statistical and machine learning to help institutions detect suspicious access to electronic health records. J Am Med Inform Assoc 2011;18:498–505. Powell C. Akron General fires employees for patient privacy violations in hospital shooting case. Akron Beacon J 2012.



Appropriateness of commercially available and partially customized medication dosing alerts among pediatric patients Jeremy S Stultz,1 Milap C Nahata2,3 1

Nationwide Children's Hospital, Department of Pharmacy, Ohio State University College of Pharmacy, Columbus, Ohio, USA 2 Institute of Therapeutic Innovations and Outcomes, College of Medicine, Columbus, Ohio, USA 3 College of Pharmacy, Ohio State University, Columbus, Ohio, USA Correspondence to Dr Milap C Nahata, Institute of Therapeutic Innovations and Outcomes, College of Pharmacy, Ohio State University, Parks Hall, 500 W. 12th Avenue, Columbus, OH 43210, USA; [email protected] Received 15 February 2013 Revised 31 May 2013 Accepted 7 June 2013 Published Online First 27 June 2013

ABSTRACT Objectives To evaluate dosing alert appropriateness, categorize orders with alerts, and compare the appropriateness of alerts due to customized and noncustomized dose ranges at a pediatric hospital. Methods This was a retrospective analysis of medication orders causing dosing alerts. Orders for outpatient prescriptions, patients ≥18 years of age, and research protocols were excluded. Patient medical records were reviewed and ordered doses compared with a widely used pediatric reference (Lexi-Comp) and institutional recommendations. The alerted orders were categorized and the occurrence of appropriate alerts was compared. Results There were 47 181 inpatient orders during the studied period; 1935 orders caused 3774 dosing alerts for 369 medications in 573 patients (median age 6.1 years). All alerted orders had an alert overridden by the prescriber. The majority (86.2%) of alerted orders inappropriately caused alerts; 58.0% were justifiable doses and 28.2% were within Lexi-Comp. However, 13.8% of alerted orders appropriately caused alerts; 8.0% were incorrect doses and 5.8% had no dosing recommendations available. Appropriately alerted orders occurred in 19.7% of alerted orders due to customized ranges compared to 12.8% due to non-customized ranges ( p=0.002). Preterm and term neonates, infants, and children (2–5 years) had higher proportions of inappropriate alerts compared to appropriate alerts (all p

Clinical benefits of electronic health record use: national findings.

Using Electronic Health Record Systems in Diabetes Care: Emerging Practices.

Benefits negotiation: three Swedish hospitals pursuit of potential electronic health record benefits.

Public trust in health information sharing: implications for biobanking and electronic health record systems.

Recent perspectives of electronic medical record systems.

Copy, paste, and cloned notes in electronic health records: prevalence, benefits, risks, and best practice recommendations.

"Big data" and the electronic health record.

Big data and the electronic health record.

Effects of Shared Electronic Health Record Systems on Drug-Drug Interaction and Duplication Warning Detection.

Impact of electronic health record systems on information integrity: quality and safety implications.

Electronic cigarettes: health risks and workplace policy.

Electronic cigarettes: the potential risks outweigh the benefits.

Health-oriented electronic oral health record: development and evaluation.

Quantitative ethnographic study of physician workflow and interactions with electronic health record systems.

Electronic cigarettes: the potential benefits outweigh the risks.

Parameterizing time in electronic health record studies.

The evolution of the electronic health record.

Disassociation for electronic health record privacy.

Patient-initiated electronic health record amendment requests.

Leveraging electronic health record systems to create and provide electronic cancer survivorship care plans: a pilot study.

Assessing electronic health record systems in emergency departments: Using a decision analytic Bayesian model.

A Standards-Based Architecture Proposal for Integrating Patient mHealth Apps to Electronic Health Record Systems.

Building bridges across electronic health record systems through inferred phenotypic topics.

Validation of a nurses' views on electronic medical record systems (EMR) questionnaire in Turkish health system.