Journal of the American Medical Association Advance Access Payne TH, et al,Informatics J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Reviewpublished March 30, 2015

Recommendations to Improve the Usability of Drug-Drug Interaction Clinical Decision Support Alerts

RECEIVED 17 October 2014 REVISED 7 January 2015 ACCEPTED 8 February 2015

Thomas H. Payne1, Lisa E. Hines2, Raymond C. Chan3, Seth Hartman4, Joan Kapusnik-Uner5, Alissa L. Russ6, Bruce W. Chaffee7, Christian Hartman8, Victoria Tamis9, Brian Galbreth10, Peter A. Glassman11, Shobha Phansalkar12, Heleen van der Sijs13, Sheila M. Gephart14, Gordon Mann15, Howard R. Strasberg12, Amy J. Grizzle16, Mary Brown2, Gilad J. Kuperman17, Chris Steiner18, Amanda Sullins19, Hugh Ryan19, Michael A. Wittie20, Daniel C. Malone2

ABSTRACT ....................................................................................................................................................

.................................................................................................................................................... Key words: clinical decision support; alerts; drug interactions; usability; CPOE (up to 5)

INTRODUCTION

preparing, dispensing, and administering medications, and monitoring patients for adverse drug events. The most common form of DDI decision support is interruptive alerts, although other types of CDS tools are also used. It is widely recognized that clinicians are generally unsatisfied with the lack of patient specificity and inappropriate context of DDI alerts.8–10 These alerts are occurring with escalating frequency with the increased use of EHRs.11,12 Factors contributing to excessive DDI alerts may include inconsistent evaluation and classification of interactions, lack of specificity in alerting logic, and perceived risk of legal liability.13,14 Other drug safety alerts, such as drug-allergy, drug-disease, and duplicate therapy alerts are also problematic and contribute to clinician dissatisfaction.

Clinical decision support (CDS) has the potential to make patient care more efficient, safe, and effective.1 CDS encompasses a variety of tools presented in an electronic health record (EHR) based on relevant patient and care process information in order to improve clinical decision making. Though drug-drug interactions (DDIs) are known to cause harm,2,3 there is little information on how best to use CDS tools to improve patient outcomes.4–7 End-users of DDI decision support are clinicians (i.e., healthcare professionals such as physicians, pharmacists, and nurses) who encounter alerts and other forms of DDI decision support in the process of prescribing, reviewing, verifying,

Correspondence to Daniel C. Malone, University of Arizona College of Pharmacy, 1295 N Martin Ave, PO Box 210202, Tucson, AZ 85721-0202, USA; [email protected]; Tel: (520) 626-3532; Fax: (520) 656-7355 C The Author 2015. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. V For Permissions, please email: [email protected] For numbered affiliations see end of article.

1

REVIEW

Objective: To establish preferred strategies for presenting drug-drug interaction (DDI) clinical decision support alerts. Materials and Methods: A DDI Clinical Decision Support Conference Series included a workgroup consisting of 24 clinical, usability, and informatics experts representing academia, health information technology (IT) vendors, healthcare organizations, and the Office of the National Coordinator for Health IT. Workgroup members met via web-based meetings 12 times from January 2013 to February 2014, and two in-person meetings to reach consensus on recommendations to improve decision support for DDIs. We addressed three key questions: (1) what, how, where, and when do we display DDI decision support? (2) should presentation of DDI decision support vary by clinicians? and (3) how should effectiveness of DDI decision support be measured? Results: Our recommendations include the consistent use of terminology, visual cues, minimal text, formatting, content, and reporting standards to facilitate usability. All clinicians involved in the medication use process should be able to view DDI alerts and actions by other clinicians. Override rates are common but may not be a good measure of effectiveness. Discussion: Seven core elements should be included with DDI decision support. DDI information should be presented to all clinicians. Finally, in their current form, override rates have limited capability to evaluate alert effectiveness. Conclusion: DDI clinical decision support alerts need major improvements. We provide recommendations for healthcare organizations and IT vendors to improve the clinician interface of DDI alerts, with the aim of reducing alert fatigue and improving patient safety.

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

meeting (November 2014), and the American Society of HealthSystem Pharmacy Midyear Clinical Meeting (December 2014).

These drug safety alerts are so common that it becomes difficult to distinguish important from less important ones, causing clinically relevant alerts to be ignored.15–17 Reports of override rates as high as 90% raise concern that DDI decision support needs fundamental revision.18 There is also wide variation across healthcare organizations and health information technology (IT) vendors in how DDI alerts are presented to clinicians, with no clear recommendations derived from studies or best practices to provide guidance. Thus, addressing the design of DDI decision support is a timely and important topic. Although the focus of this work is specific to DDI decision support, we recognize that other drug safety alerts, such as drugallergy alerts, pose similar decision support design challenges. We assembled a group of experts and conducted a series of meeting over 13 months to develop specific recommendations to improve the quality of DDI decision support. This paper describes recommendations from the Usability Workgroup for preferred DDI alerting strategies within CDS systems. These principles are intended to convey drug information effectively while reducing clinicians’ cognitive effort in order to improve medication safety.

RESULTS (Please refer to Supplemantary Appendix A for a glossary of terms used in this paper) Key Question 1: What, how, where, and when do we display DDI decision support? Research from human factors engineering indicates that visibility, color, and prioritization influence a clinician’s response to safety warnings.19 While this literature provides substantial information on warning design for industrial and other safety hazards, there is little research on designing drug-related warnings.20 Importantly, there is currently little consistency in how alert information is presented to clinicians. Variability in presentation may confuse and cognitively overload clinicians who routinely work with multiple systems. We recommend greater uniformity and consistency in DDI alert presentation across systems. Specific preferred methods are described below, with additional details and examples in Supplementary Appendix B. What Information to Include in DDI Alerts?. Four basic alert components are recommended by the safety literature: (1) a signal word indicating the seriousness of the DDI; (2) hazard information (e.g., clearly denoting the potentially harmful drug combination); (3) instructions or actions on how to reduce risk of injury; and (4) specific clinical consequences that may ensue if the hazard is not averted.19,21 In addition, recent research has identified that integrating contextual information and patient-specific data into algorithm logic, along with the presentation of evidence may improve DDI alerting.22–25 We recommend that seven components be integrated into alerting systems for DDI warnings: (1) drugs involved; (2) seriousness; (3) clinical consequences; (4) mechanism of the interaction; (5) contextual information/modifying factors; (6) recommended action(s); and (7) evidence.

MATERIALS AND METHODS

REVIEW

The workgroup consisted of 24 individuals (listed as coauthors), representing clinical, informatics, and computer interface design experts with diverse backgrounds including academia, health IT (EHR and drug knowledgebase) vendors, healthcare organizations, practicing clinicians, and the Office of the National Coordinator for Health IT. Two face-to-face meetings were conducted, the first meeting occurred with nearly all workgroup members attending (May 2013, Rockville, Maryland, USA) and the second with a subset of those (September 2013, Phoenix, Arizona, USA), with the option to join via teleconference. Workgroup members also met via webbased conference calls 12 times for at least one hour in duration from January 2013 to February 2014 to reach consensus. One workgroup member (T.H.P.) facilitated the meetings. We addressed three key questions to improve usability and increase consistency of DDI decision support systems:

Drugs Involved The interacting drug pair(s) should be clearly identified.26 In one study that included DDI alerts, 21 of 30 prescribers had difficulty determining why alerts were triggered.22 To clearly identify interacting drug pairs, we recommend using the medication name as ordered as well as generic ingredient names,24 which is especially important for combination products.

1. What, how, where, and when do we display DDI decision support? 2. Should presentation of DDI decision support vary by clinicians? 3. How should effectiveness of DDI decision support be measured?

Seriousness Category We recommend the use of consistent terms and definitions to indicate the potential seriousness of the DDI.13,27–29 Consistent terminology is important to aid clinician interpretation across all CDS systems. Currently, there is little to no research to determine what words or phrases would be most appropriate.

We achieved consensus through an iterative process of drafting recommendations, collecting verbal or written comments from workgroup members, and revising documents until no additional substantive comments were provided. We shared these recommendations with stakeholders (e.g., clinicians, EHR and knowledgebase vendors, professional and quality organizations, government agencies) through an open website and by presentations at national meetings, including a Human Factors and Ergonomics Society conference (March 2014), a premeeting symposia at the American Medical Informatics Association

Clinical Consequences (and Frequency) The potential adverse clinical outcome(s) for the patient taking interacting drugs should be clearly described (e.g., hyperkalemia, QT prolongation, reduced HIV drug efficacy), thereby allowing the clinician to utilize relevant patient

2

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

information and clinical judgment to balance the risks and benefits.25 When available, the frequency or incidence of adverse drug events (ADEs) associated with a specific DDI, as well as predisposing risk factors for the adverse outcomes, may be useful for estimating the risk for an individual patient.

within and across different EHR systems. Familiarity with the placement of components on the alert interface may facilitate quick, accurate recognition, and action by clinicians. Consistent Use of Color and Visual Cues We recommend consistent use of color and symbols to aid alert recognition.23,34 Alerts for the most serious (e.g., contraindicated) DDIs should be easily distinguished from those that are less serious. A useful analogy is the consistent size, shape, and color of specific types of traffic road signs, which allow drivers to readily distinguish a stop sign from other types of road signs (e.g., yield sign). Meanings of color and symbols should be clear to all clinicians.

Mechanism of Interaction The alert should describe the mechanism of the DDI, when known, allowing the clinician to gain understanding of the problem and to identify potential therapeutic alternatives. For example, knowing that the mechanism of the interaction involves inhibition of a particular metabolic pathway can assist clinicians in choosing therapeutic alternatives (e.g., selecting an alternative in the same pharmacologic or therapeutic class that uses a different metabolic pathway for elimination).

Consistent Terminology and Brevity We recommend the use of consistently phrased alerts and terms throughout and across DDI decision support systems. Consistent terminology promotes semantic clarity, and improves speed of processing visual cues.23 Information should be presented in concise language, with minimal text. Also, by minimizing the amount of text initially presented, a larger font size can be used to enhance readability.19 Minimize Impact on Workflow We recommend reserving interruptive alerts (i.e., those requiring action by clinicians before proceeding) for the most serious DDIs. Rather than alerting (interrupting) the healthcare professional about an action already taken, the system could proactively guide decision making to relatively safer alternative options, thereby eliminating the need for an interruptive alert. This preemptive strategy could decrease the “noise” while increasing both safety and efficiency. Generally, if a less likely or less serious DDI is detected, the healthcare professional should be able to receive the relevant information through noninterruptive CDS such as a link or Infobutton or retrospective surveillance reports intended for ongoing patient monitoring. During surveillance, if the presence of a DDI clinical consequence is detected, an interruptive alert or report may be presented to the healthcare professional responsible for that patient’s ongoing care. Caution is needed to avoid indiscriminate suppression of interruptive alerts.35 We do not recommend that institutions eliminate or completely turn off clinically relevant alerts; instead, alerts could be diverted from interruptive to noninterruptive form with on-demand access.

Recommended Action When presenting a DDI alert to a clinician it is important to provide guidance on strategies to mitigate potential harm.27,31,32 There are many possible courses of action, such as dose modification, order cancellation, ordering an alternate medication, or monitoring/surveillance actions.33 For many DDIs, several actions may be appropriate, so CDS systems should be able to present a list of suggestions that could be locally customizable to take into account formulary and other organizational factors. Evidence The evidence underlying DDIs can vary widely, but alerts should provide some information related to the strength and source of evidence (e.g., case report, clinical trial).22,24 FloorSchreudering et al.27 recommend presenting a set of symbols, letters, and/or numbers to communicate the evidence with clear explanations of the grading system. As part of the conference series, another group of experts developed recommendations for systematic evaluation of DDI evidence.

Where to Display DDI Alert Components (Salience Hierarchy). Clinicians’ ability to understand and respond to DDI alerts is influenced by information presented in the dialog box (the first screen presented to the clinician).22,26,36 There is an inherent trade-off between providing too many details that risk overwhelming the clinician, and too few details that fail to communicate the potential seriousness or the actions to be taken. Thus, the most critical information should be presented on the top-level screen of the alert, with linked information accessible on-demand about background and secondary considerations. For example, the DDI mechanism and supporting evidence may

How to Present DDI Alerts. Because clinicians often work in multiple settings, we recommend alert presentation consistency

3

REVIEW

Contextual Information/Modifying Factors A fundamental tenet of CDS design is that the right information should be presented to the right clinician(s) at the right time.30 In the case of DDI alerts, this includes any predisposing factor information (e.g., co-morbidity, lab results) to add relevance and to facilitate alert processing. Hansten and Horn suggest that adjudication of DDI alerts should be based on an assessment of patient-specific factors such as age, predisposing diseases, pharmacogenomic phenotype, and the specific drug regimen(s), such as dose, route, duration of therapy, sequence of initiating co-therapy, and timing of co-administration.31 This information should be included in the alerting logic or presented within the alert display. Evidence on factors that predispose patients to DDIs is lacking. Research using large datasets, such as PCORNet and the Food and Drug Administration’s sentinel network, may provide important insights on those patients at greatest risk of harm.

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

for a predetermined amount of time (“snooze” function) or send the alert (defer or forward) to a different clinician. Additionally, patient portals could help mitigate or detect early ADEs by reminding patients of laboratory tests needed for appropriate monitoring or to be watchful of important DDI-related symptoms. However, appropriate audit trails and workflows must be in place to ensure clinician accountability for these types of options to be viable. New methods of notification must be formally evaluated to ensure patient safety.35

not be necessary to view on the primary alert interface as long as the information is easily accessible via links.37 The workgroup considered the issue of developing a prototype dialog box but after much discussion, reached the conclusion that this was beyond the scope of the project because any prototype should be rigorously tested. As recognized by warnings literature, some recommendations may compete with one another and require careful balancing.19 For example, it will likely be challenging to present sufficient detail for decision-making, while at the same time keeping the information concise. Techniques such as iterative alert prototyping and usability testing are needed to help identify the appropriate balance for DDI alerts. We recommend further research with formal testing to determine the essential components for primary dialog box and the secondary components. Prototypes should include all seven alert components we recommend with the objective of unambiguous and intuitive design. Others provide examples of several of the principles we recommend.20,23,38

REVIEW

Key Question 2: Should presentation of DDI decision support vary by clinicians? A safe medication use process is built on interdisciplinary interactions and cooperation to assure that patients receive the best possible care. DDI decision support applies to all clinicians on the care team: prescriber, pharmacist, nurse, patient, and others (Figure 1). For nonprescribing clinicians, DDI alerts may be deployed as a second check to help ensure that patients receiving interacting drug pairs are being monitored or assessed. Patient care and safety are best achieved when all members of the team have knowledge of what other members are doing.40 As such, we advocate a team approach to managing DDIs (Figure 1). We recommend that general alert content be consistent among various types of clinicians. What may differ, however, is how the information is presented to various professionals. The message may be changed based on the context or functions, recognizing that professionals in different settings have different roles, responsibilities, and privileges. For example, prescriber recommendations may focus on ordering specific monitoring parameters, while pharmacists may be notified to ensure monitoring orders were placed and results reviewed. Another important question we considered is how an alert display should change if an individual clinician has been exposed many times to an alert yet there is no detectable behavior change? Similarly, should specialists with unique training or roles be allowed to “turn off” alerts for DDIs that they routinely manage (e.g., warfarin clinic pharmacists receiving warfarin interaction alerts when the patient is currently being monitored)? We are not aware of evidence that demonstrates that it is safe to eliminate DDI alerts for specialists. However, establishing more selective institutional DDI alerting practices overall may relieve much of the alert burden. EHR system architecture should allow institutions to easily make these changes based on clinician characteristics. Patients play an important role managing risks associated with DDIs, and need to be engaged in monitoring for signs of toxicity or loss of efficacy.41,42 Although evidence-based best practices for printed patient information are recommended,43,44 the logistics and manner in how patients should be informed is beyond our Workgroup’s scope of work.

When to Present DDI Alerts. We recommend that DDI alert information be displayed at the point of decision making. Unfortunately, many systems allow clinicians to proceed through the entire medication ordering process before the clinician receives an alert. This process is inefficient. Instead, CDS should be presented earlier in decision making and ordering workflow, for example, by guiding prescribers towards lower risk drugs during earlier stages of medication selection and thereby avoiding the need for alerts.39 How to Resolve DDI Alerts. Alert resolution should be facilitated via on-screen operations. So, when a DDI alert is generated, clinicians should be able to select from a list of actionable choices. These may include: discontinue one or more active medications, cancel the order, modify the dose, provide patient education, and order labs. There should be as few steps (e.g., keystrokes, mouse clicks, scrolling, window changes) as possible to resolve the potential alert. Override Reasons Override reasons based on Bayesian framework (see Supplementary Appendix C) should be used to help system designers identify false positive and false negative alerts. When a clinician believes an alert is not relevant for a particular patient, there should be an option to reject the information and provide rationale/comments for feedback. Actions and feedback should be stored for quality improvement efforts and made available for other clinicians to view, such as when this alert is displayed again for the same patient (see Key Question 3). Override reasons such as “will monitor” are not granular enough to describe patient monitoring or surveillance plans and should be avoided. Instead, if monitoring is recommended, then it should be actionable from the DDI alert interface.

Key Question 3: How should effectiveness of DDI decision support be measured? The value of CDS may be measured by the achievement of outcomes relative to the interaction cost (e.g., cognitive burden,

Other Actionable Choices We considered more sophisticated options for alert presentation and resolution. For example, clinicians could delay the alert

4

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

Figure 1: schematic of interprofessional decision support to facilitate patient safety (hospital setting).

full evaluation. Therefore, override rates provide a crude estimate of alert adherence. In the near term, override rates should be used to identify alerts that require a detailed evaluation process, including the incorporation of clinician feedback. This detailed evaluation should include the presence of modifying factors (e.g., lab values, co-morbidities), actions taken as a result of the alert (e.g., monitoring ordered), and a consensus on perceived value by clinicians (which is difficult to record with the alert) (see Supplementary Appendix C for an explanation of how Bayesian methods can measure alert effectiveness). There are a myriad of opportunities for optimizing alerts and increasing their value. While institutions should perform extensive review before implementing alert changes, such parsimonious alerting may be an appropriate strategy to ease alert fatigue. Standardizing DDI Alert Data Collection and Analysis. In the long term, we propose that a professional group or trusted agency be established to standardize collection and analysis of DDI decision support/alert data. We recommend the creation of a central repository for submitting de-identified DDI alert data that could be used for continuous quality improvement to optimize the value of individual alerts. Aggregating de-identified alert data with override rates, patient predisposing factors, and actions taken could maximize the power of data and establish CDS feedback loops to knowledgebase and EHR vendors and healthcare organizations. Ensuring that an alert does not fire when it should not fire (high specificity) is equally important as ensuring it does fire when it should fire (high sensitivity) (Supplementary Appendix C). The components of a centralized DDI alert data repository should be formatted to include true positives, true negatives, false

5

REVIEW

time), and thus encompasses efficiency in using resources.45 Effectiveness of CDS can be defined as a product of both measured value as well as perceived value, since components of value include, but are not limited to, the following: clinical outcomes, process efficiency measures, clinician satisfaction, heuristics, evidence, usability, and cost of ADEs. Examples of measured value for the DDI between warfarin and amiodarone include increased rates of appropriate international normalized ratio (INR) ordering, warfarin dose adjustments, patient counseling, and anticoagulation clinic follow up. The perceived value of alerts likely differs by clinician type and experience. For example, practicing cardiologists may perceive little value, while medical residents may perceive high value if it reminds them to order an INR. Using solely alert override rates to determine effectiveness of alerts may not account for these value-added actions, unless the actions were discreetly captured with the alert (Supplementary Appendix C). Conversely, alerts that do not provide value (measured or perceived) should be suppressed with precision (i.e., increase specificity) without jeopardizing sensitivity. Generally speaking, as alert effectiveness increases, alert value increases, and thus alert fatigue may decrease. Alert logic that does not consider mitigating factors associated with a low prevalence of adverse outcomes (e.g., single-doses of precipitant drugs) may produce a low positive predictive value,18 and a corresponding lower perceived value. Although important, override rates alone cannot be used to assess the effectiveness of alerts.18,46 A primary reason is that the thought process of clinicians and subsequent actions are not fully captured by current systems. For example, it is unclear if the override is due to disregard of the alert, careful consideration of the risks and benefits, or time constraints preventing a

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

Table 1: Areas for Future Research to Improve DDI Decision Support Area of Research

Description/Example

Alert Components

•

Identify essential information components to maximize DDI alert comprehension

Terminology

•

Most appropriate words, phrases, and definitions for DDI alert

•

Most appropriate terms and definitions to indicate the potential seriousness Identify the information that is essential to display on the primary alert interface vs. accessible via an embedded link Best methods to facilitate links to supplemental information or other screens to aid decision making

Information Placement

• •

Contextual Alerting

• •

Identification of predisposing risk factors (e.g., large-scale observational research) Optimal use and display of contextual data (e.g., patient age, lab results) to improve DDI alerts53–56 How patients might be involved in DDI ADE surveillance

REVIEW

Patient-focused CDS and ADE surveillance

•

Noninterruptive CDS

•

How alternative interface designs for CDS tools can be deployed to reduce ADEs associated with DDIs, and under what circumstances they are superior or inferior to interruptive alerts

Decision-making

•

Elucidating the cognitive strategies that clinicians use to make decisions about DDIs

Filtering/suppressing DDI alerts

•

Safety and effectiveness of alert filtering and modifications (e.g., dynamic diversion from interruptive to non-interruptive)

Prototype testing

•

Evaluation of prototypes incorporating these recommendations to determine what works best, including different settings clinician-types and clinical contexts

ADE ¼ adverse drug event; CDS ¼ clinical decision support; DDI ¼ drug-drug interaction

Improving the usability of DDI decision support is essential because patient safety is compromised when clinicians perceive DDI alerts as unimportant because of poor presentation or lack of relevance. Clinicians attend to issues with higher perceived value when simultaneously presented with numerous decisions and requests.47–49 We need to balance the cognitive resources, attention, and time devoted to DDI alerts with that devoted to patient interaction and other important decisions (e.g., considering diagnoses, ordering tests).50 There are many issues beyond the scope of this Workgroup that need to be addressed in order to achieve the desired improvements in DDI decision support associated with our recommendations. Our recommendations do not address inappropriate alerting related to inaccurate/incomplete medication lists, inactive/expired medication orders, or irrelevant dosage forms/routes of administration.24 For example, an alert may still trigger for both an oral and topical version of a drug, when a DDI is only relevant with the oral formulation.51,52 We assume some of these issues can be rectified by greater specification of alerting logic. While we do not comment further on these types of issues, our work is predicated on the assumption that all stakeholders wish to minimize noncredible alerts. Indeed, we believe that designers and clinicians must work collaboratively to assure that alerts are clinically relevant,

positives, and false negatives, which could be normalized using crowd-sourced feedback obtained from a large group of end users. We anticipate this capability will eventually lead to greater collaboration among institutions, perhaps even at the international level. Focusing on value and clinical relevance of DDI decision support will allow us to collectively measure outcomes not easily accessible today.

DISCUSSION In today’s CDS environment, clinically relevant DDI alerts are obfuscated by excessive alerts with suboptimal usability and clinician dissatisfaction. Our goal was to review, evaluate, and recommend principles to improve the usability of DDI decision support alerts. This paper summarizes our work and is intended to assist in streamlining and standardizing DDI alerting processes to reduce alert fatigue and patient harm. Our recommendations focus on consistent use of terminology, symbols/ icons, color, minimal text, formatting, content, and reporting standards to facilitate usability. Specific preferred methods for alert design are described in further detail with examples in Supplementary Appendix B. Many of our recommendations may apply more broadly to other drug safety alerts and decision support, such as drug-allergy and duplicate therapy alerts.

6

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

that the alert logic maximizes accuracy and credibility, and that other work on improving accuracy of medication lists continues. We recognize there will be barriers to the changes we are recommending. Given the proprietary nature of commercial CDS software, garnering support of vendors to relinquish control of individual design features it is not a minor hurdle. Implementing our recommendations is dependent on the willingness of vendors and institutions to adopt new alert algorithms and design standards (e.g., allowing laboratory monitoring directly from the DDI alert screen). Similarly, there may be objections to requiring submission of standard de-identified data because it involves increased time and labor, as well as perceived risk of being compared to other software systems. In the case of DDI alerts, safety concerns should be given higher priority than vendor incentives, given the potential to better understand the risk for, and ways to reduce, patient harm. Although our workgroup members represented many of the key stakeholders interested in DDI decision support alert usability, it was not feasible to elicit input from all possible perspectives. For example, patients, an important component of the medication use process, were not explicitly represented. While our recommendations will improve consistency and readability of alerts, it will do little to reduce alert fatigue in the absence of major changes to the knowledgebase content and alert logic. The success of these recommendations is directly tied to limiting interruptive DDI alerts (i.e., alerting for only the most serious DDIs requiring immediate clinician’s attention). Furthermore, the funding and development of a national alert data repository is not a small undertaking, and additional recommendations related to evaluating the effectiveness of DDI alerting systems will be difficult without the availability of population data. We emphasize that additional studies are needed to measure the impact of these suggested DDI alert recommendations and the next generation of DDI decision support. Additional research will be critical to evaluate the most effective DDI alert content and design. Evaluating outcomes (actual occurrence or avoidance of ADEs), and not just DDI override rates, is of high importance. Additional research is essential to determine the most effective methods to reduce alert fatigue and ultimately improve patient safety (Table 1). These recommendations focus on DDI alerts but may serve as a template for other types of drug safety alerts. For example, clinicians frequently encounter multiple medication safety alerts, often side-by-side on the same screen. The design considerations for DDI and drugallergy alerts are nearly identical. Therefore, most of the recommendations given here already apply to other medication-related alerts as well. Improving CDS systems is an iterative process, as knowledge about best practices for DDI alert design is dynamic and will evolve as systems become more sophisticated.

may help improve decision making and prevent alert fatigue. All clinicians should be able to view alerts and responses to alerts in order to optimize clinical care. Alert effectiveness needs further evaluation and should not solely rely on override rates in their current form. Our recommendations provide healthcare organizations and IT vendors with strategies to improve prescribing decisions, and thus improve patient safety.

FUNDING This project was funded by the Agency for Healthcare Research and Quality Grant No. 1R13HS021826-01 (Malone DC-PI). Additional support was provided by Cerner, Elsevier Clinical Solutions, Epocrates athenahealth, Inc, FDB (First Databank, Inc.), Truven Health Analytics, and Wolters Kluwer. Dr. Russ was supported in part by VA Health Services Research and Development grant #PPO 09-298 (PI: Russ) and a VA HSR&D Research Career Development Award (CDA 11-214). Dr. Gephart was supported in part by the Robert Wood Johnson Foundation Nurse Faculty Scholars Program (PI: Gephart, ID 72114) and the Agency for Healthcare Research and Quality (PI: Gephart, 1K08HS022908-01A1).

COMPETING INTERESTS AND CONFLICT OF INTEREST STATEMENT

CONTRIBUTORS Thomas H. Payne, Lisa E. Hines, Raymond C. Chan, Seth Hartman, Joan Kapusnik-Uner, Alissa L. Russ, Bruce W. Chaffee, Christian Hartman, Victoria Tamis, Brian Galbreth, Peter A. Glassman, Shobha Phansalkar, Heleen van der Sijs, Sheila M. Gephart, Gordon Mann, Howard R. Strasberg, Amy J. Grizzle, Mary Brown, Gilad J. Kuperman, Chris Steiner, Amanda Sullins, Hugh Ryan, Michael A. Wittie, and Daniel C. Malone made substantial contributions to the conception, analysis, and interpretation of the work, all have contributed to reviewing and revising it critically for intellectual content, all have approved the final version for publication, and all are in agreement to be accountable for all aspects of the work with respect to accuracy and integrity.

CONCLUSION

ACKNOWLEDGEMENTS

We identified several areas for improving the usability of DDI alerting systems. More consistency is recommended in how alerts are presented across vendors. Consistent presentation

This project was funded by the Agency for Healthcare Research and Quality Grant No. 1R13HS021826-01 (Malone DC-PI).

7

REVIEW

Daniel C. Malone reports grants from Agency for Healthcare Research and Quality and unrestricted support for this project from Cerner, Elsevier Clinical Solutions, Epocrates (athenahealth, Inc), FDB, Truven Health Analytics, and Wolters Kluwer. Joan Kapusnik-Uner is an employee of FDB. Shobha Phansalkar, Howard R. Strasberg, and Christian Hartman are employees of Wolters Kluwer Health – Clinical Solutions, Minneapolis, MN. Gordon Mann is an employee of Epic. Chris Steiner is an employee of Gold Standard Drug Databases/ Elsevier. Amanda Sullins and Hugh Ryan are employees of the Cerner Corporation. The other authors have no competing interests or conflicts of interest.

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

interaction alerts. J Am Pharm Assoc (2003). 2006;46(2): 148–153. 11. Hsiao CJ, Hing E. Use and characteristics of electronic health record systems among office-based physician practices: United States, 2001-2013. NCHS Data Brief. 2014;143:1–8 12. Marcotte L, Seidman J, Trudel K, et al. Achieving meaningful use of health information technology: a guide for physicians to the EHR incentive programs. Arch Intern Med. 2012;172(9):731–736. 13. Smithburger PL, Buckley MS, Bejian S, Burenheide K, Kane-Gill SL. A critical evaluation of clinical decision support for the detection of drug-drug interactions. Expert Opin Drug Saf. 2011;10(6):871–882. 14. Ridgeley MS, Greenberg MD. Too many alerts, too much liability: sorting through the malpractice implications of drugdrug interaction clinical decision support. St. Louis UJ Health L Policy. 2012;5(2):257–296. 15. Carspecken CW, Sharek PJ, Longhurst C, Pageler NM. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics. 2013;131(6):e1970–e1973. 16. van der Sijs H. Drug Safety Alerting in Computerized Physician Order Entry Unraveling and Counteracting Alert Fatigue. The Netherlands Erasmus Universiteit Rotterdam; 2006. 17. Grizzle AJ, Mahmood MH, Ko Y, et al. Reasons provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care. 2007;10:573–578. 18. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(2):138–147. 19. Wogalter MS. Handbook of Warnings. Mahwah, NJ: Lawrence Erlbaum Associates; 2006. 20. Russ AL, Zillich AJ, Melton BL, et al. Applying human factors principles to alert design increases efficiency and reduces prescribing errors in a scenario-based simulation. J Am Med Inform Assoc. 2014;e2:e287–e296. 21. Phansalkar S, Edworthy J, Hellier E, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc. 2010;17(5):493–501. 22. Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. Prescribers’ interactions with medication alerts at the point of prescribing: a multi-method, in situ investigation of the human-computer interaction. Int J Med Inform. Ireland: Elsevier Ireland Ltd., 2012;4:232–243. 23. Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform. 2013;82(6):492–503. 24. van Roon EN, Flikweert S, le Comte M, et al. Clinical relevance of drug-drug interactions: a structured assessment procedure. Drug Saf. 2005;28(12):1131–1139. 25. Ashworth M. Re: GPs’ views on computerized drug interaction alerts. J Clin Pharm Ther. 2002;27(5):311–312.

Views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs, Department of Health and Human Services, the U.S. government, or the Robert Wood Johnson Foundation. Additional support was provided by Cerner, Elsevier Clinical Solutions, Epocrates athenahealth, Inc, FDB (First Databank, Inc.), Truven Health Analytics, and Wolters Kluwer. Use of funds was at the sole discretion of the University of Arizona and according to Department of Health and Human Services requirements. The authors would like to thank Loretta Peters for her editorial assistance in preparing this manuscript.

SUPPLEMENTARY MATERIAL Supplementary material is available online at http://jamia. oxfordjournals.org/.

REFERENCES

REVIEW

1. Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc. 2001;8(4):299–308. 2. Hines LE, Murphy JE. Potentially harmful drug-drug interactions in the elderly: a review. Am J Geriatr Pharmacother. 2011;9(6):364–377. 3. Magro L, Moretti U, Leone R. Epidemiology and characteristics of adverse drug reactions caused by drug-drug interactions. Expert Opin Drug Saf. 2012;11(1):83–94. 4. Ojeleye O, Avery A, Gupta V, Boyd M. The evidence for the effectiveness of safety alerts in electronic patient medication record systems at the point of pharmacy order entry: a systematic review. BMC Med Inform Decis Mak. 2013;13:69. 5. Carling CL, Kirkehei I, Dalsbø TK, Paulsen E. Risks to patient safety associated with implementation of electronic applications for medication management in ambulatory care–a systematic review. BMC Med Inform Decis Mak. 2013;13: 133. 6. van Doormaal JE, van den Bemt PM, Zaal RJ, et al. The influence that electronic prescribing has on medication errors and preventable adverse drug events: an interrupted timeseries study. J Am Med Inform Assoc. 2009;16(6): 816–825. 7. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc. 2009;16(4):531–538. 8. Weingart SN, Simchowitz B, Shiman L, et al. Clinicians’ assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med. 2009;169(17):1627–1632. 9. Ko Y, Abarca J, Malone DC, et al. Practitioners’ views on computerized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 2007;14(1):56–64. 10. Abarca J, Malone DC, Skrepnek GH, et al. Community pharmacy managers’ perception of computerized drug-drug

8

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

9

REVIEW

39. Coleman JJ, van der Sijs H, Haefeli WE, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak. 2013;13:111. 40. Medical Teamwork and Patient Safety: The Evidence-based Relation. July 2005. Agency for Healthcare Research and Quality, Rockville, MD. Secondary Medical Teamwork and Patient Safety: The Evidence-based Relation. July 2005. Agency for Healthcare Research and Quality, Rockville, MD. http://archive.ahrq.gov/research/findings/final-reports/ medteam/index.html. 41. Mutebi A, Warholak TL, Hines LE, Plummer R, Malone DC. Assessing patients’ information needs regarding drug-drug interactions. J Am Pharm Assoc (2003). 2013;53(1):39–45. 42. Ryan R, Santesso N, Lowe D, et al. Interventions to improve safe and effective medicines use by consumers: an overview of systematic reviews. Cochrane Database Syst Rev. 2014(4):1–72, Art. No.: CD007768. 43. Woloshin S, Schwartz LM. Getting to better prescription drug information. J Gen Intern Med. 2012;27(12): 1582–1584. 44. Fischhoff B, Brewer NT, Downs JS. Communicating Risks and Benefits: An Evidence Based User’s Guide. Government Printing Office; 2012. 45. Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477–2481. 46. McCoy AB, Waitman LR, Lewis JB, et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc. 2012;19(3): 346–352. 47. Chaiken, S, Ledgerwood, A. A theory of heuristic and systematic information processing. In: Van Lange P, Kruglanski A, Higgins E, eds. The Handbook of Theories of Social Psychology. Thousand Oaks, CA: Sage; 2012:246–266. 48. Maheswaran D, Chaiken S. Promoting systematic processing in low-motivation settings: effect of incongruent information on processing and judgment. J Pers Soc Psychol. 1991;61(1):13–25. 49. Trumbo CW. Heuristic-systematic information processing and risk judgment. Risk Anal. 1999;19(3):391–400. 50. Yarnall KS, Pollak KI, Østbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health. 2003;93(4):635–641. 51. Saverno KR, Hines LE, Warholak TL, et al. Ability of pharmacy clinical decision-support software to alert users about clinically important drug-drug interactions. J Am Med Inform Assoc. 2011;18(1):32–37. 52. Horn JR, Gumpper KF, Hardy JC, McDonnell PJ, Phansalkar S, Reilly C. Clinical decision support for drug-drug interactions: improvement needed. Am J Health Syst Pharm. 2013;70(10):905–909. 53. Riedmann D, Jung M, Hackl WO, Stuhlinger W, van der Sijs H, Ammenwerth E. Development of a context model to

26. Zachariah M, Phansalkar S, Seidling HM, et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems–I-MeDeSA. J Am Med Inform Assoc. 2011;18 (Suppl 1):i62–i72. 27. Floor-Schreudering A, Geerts AF, Aronson JK, Bouvy ML, Ferner RE, De Smet PA. Checklist for standardized reporting of drug-drug interaction management guidelines. Eur J Clin Pharmacol. 2014;70(3):313–318. 28. Hines LE, Malone DC, Murphy JE. Recommendations for generating, evaluating, and implementing drug-drug interaction evidence. Pharmacotherapy. 2012;32(4):304–313. 29. Russ AL, Saleem JJ, McManus MS, Zillich AJ, Doebbeling BN. Computerized medication alerts and prescriber mental models: observing routine patient care. In: Proceedings of the Human Factors and Ergonomics Society, 53rd Annual Meeting; 2009:655–659. 30. Osheroff JA. Healthcare Information and Management Systems Society. Improving Outcomes with Clinical Decision Support: an Implementer’s Guide. 2nd ed. Chicago, IL: HIMSS; 2012. 31. Hansten PD, Horn JR, Hazlet TK. ORCA: OpeRational ClassificAtion of drug interactions. J Am Pharm Assoc. 2001;41(2):161–165. 32. Murphy JE, Malone DC, Olson BM, Grizzle AJ, Armstrong EP, Skrepnek GH. Development of computerized alerts with management strategies for 25 serious drugdrug interactions. Am J Health Syst Pharm. 2009;66(1): 38–44. 33. Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc. 2009;2009:548–552. 34. Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform. 2012;45(6):1202–1216. 35. Russ, AL, Saleem, JJ, Mcmanus, MS, Frankel, RM, ZIllich, AJ. The workflow of computerized medication ordering in primary care is not prescriptive. In: Proceedings of the Human Factors and Ergonomics Society, 54th Annual Meeting; 2010. 36. Seidling HM, Phansalkar S, Seger DL, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc. 2011;18(4):479–484. 37. Kuilboer MM, van der Lei J, Bohnen AM, van Bemmel JH. The availability of unavailable information. Proc AMIA Annu Fall Symp. 1997:749–753. 38. Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc. 2014;21(e2):e332–e340.

Payne TH, et al, J Am Med Inform Assoc 2015;0:1–10. doi:10.1093/jamia/ocv011, Review

55. Duke JD, Bolchini D. A successful model and visual design for creating context-aware drug-drug interaction alerts. AMIA Annu Symp Proc. 2011;2011:339–348. 56. Seidling HM, Klein U, Schaier M, et al. What, if all alerts were specific - Estimating the potential impact on drug interaction alert burden. Int J Med Inform. 2014;83(4): 285–291.

prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak. 2011;11:35. http://www.biomedcentral.com/1472-6947/11/35 54. Jung M, Riedmann D, Hackl WO, et al. Physicians’ perceptions on the usefulness of contextual information for prioritizing and presenting alerts in Computerized Physician Order Entry systems. BMC Med Inform Decis Mak. 2012; 12:111.

AUTHOR AFFILIATIONS .................................................................................................................................................... 1

Department of Medicine, University of Washington, Seattle, WA, USA

11

2

Department of Pharmacy Practice & Science, The University of Arizona College of Pharmacy, Tucson, AZ, USA

12

3

Information Technology, Sentara Healthcare, Virginia Beach, VA, USA

13

4

Department of Pharmacy Services, Oregon Health & Science University, Portland, OR, USA

14

5

15

Internal Medicine, Department of Veterans Affairs (VA), Greater Los Angeles Healthcare System, Los Angeles, CA, USA Medical Informatics, Wolters Kluwer Health – Clinical Solutions, Newton, MA, USA/San Diego, CA, USA Department of Hospital Pharmacy, Erasmus Medical Center, Rotterdam, Netherlands College of Nursing, The University of Arizona, Tucson, AZ, USA

Clinical Editorial, FDB (First Databank, Inc.), San Francisco, CA, USA

Pharmacy, Epic, Verona, WI, USA

REVIEW

16

Center for Health Outcomes & PharmacoEconomic Research, The University of Arizona College of Pharmacy, Tucson, AZ, USA

6

Center for Health Information and Communication, Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service, Richard L. Roudebush VA Medical Center; Regenstrief Institute, Inc. Indianapolis, IN USA; and Purdue University College of Pharmacy, West Lafayette, IN USA

17

Interoperability Informatics, New York-Presbyterian Hospital, New York, NY, USA

18

Editorial Systems, Gold Standard Drug Databases/Elsevier, Tampa, FL, USA

7

Department of Pharmacy, The University of Michigan Health System, Ann Arbor, MI, USA

19 8

Information Technology and Services, Cerner Corporation, Kansas City, MO, USA

9

20

Clinical Solutions, Pharmacy OneSource, Boston, MA, USA

Office of Clinical Quality and Safety, Office of the National Coordinator for Health Information Technology, Department of Health and Human Services, Washington, DC, USA

PeaceHealth St. John Medical Center, Longview WA, USA

10

At the time of this study was with Pharmacy Services, PeaceHealth Southwest Medical Center, Vancouver, WA, USA

10