pharmacoepidemiology and drug safety 2014; 23: 601–608 Published online 21 October 2013 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.3531
ORIGINAL REPORT
An automated causality assessment algorithm to detect drug-induced liver injury in electronic medical record data† T. Craig Cheetham1*, Janet Lee1, Christine M. Hunt2,3, Fang Niu1, Steph Reisinger4, Rich Murray4, Greg Powell2 and Julie Papay2 1
Kaiser Permanente Southern California, Pharmacy Analytical Services, CA, USA GlaxoSmithKline, NC, USA 3 Duke University, NC, USA 4 United BioSource Corporation, PA, USA 2
ABSTRACT Purpose The aim of this study was to develop an automated causality assessment algorithm to identify drug-induced liver injury. Methods The Roussel Uclaf Causality Assessment Method (RUCAM) is an algorithm for determining the causal association between a drug and liver injury. In collaboration with hepatology experts, definitions were developed for the RUCAM criteria to operationalize an electronic RUCAM (eRUCAM). The eRUCAM was tested in a population of patients taking 14 drugs with a characteristic phenotype for liver injury. Quality assurance for programming specifications involved comparisons between scores generated by the eRUCAM, for probable and highly probable cases, and expert manual RUCAM (n = 20). Concordance between eRUCAM and manual RUCAM subscores and total score was tested using the Wilcoxon signed rank test. Results Causality scores were the same for 6 of 20 patients (30%) by manual and eRUCAM algorithms. Analysis of subscores revealed ≥80% concordance between manual and eRUCAM for five of the seven criteria. In general, the total scores tended to be higher for the eRUCAM compared with the manual RUCAM. Programming issues were identified for criterion 5 ‘non-drug causes of liver injury’ where significant differences existed between manual and eRUCAM scoring (p = 0.001). For criterion 5, identical scores occurred in 9 of 20 patients (45%), and manual review identified additional codes, timing criteria, and laboratory results for improving subsequent eRUCAM revisions. Conclusion The eRUCAM had generally good concordance with manual RUCAM scoring. These preliminary findings suggest that the eRUCAM algorithm is feasible and could have application in clinical practice and drug safety surveillance. Copyright © 2013 John Wiley & Sons, Ltd. key words—liver injury; drug-induced; causality assessment; algorithm; drug safety; risk assessment; automated; pharmacoepidemiology Received 31 May 2013; Revised 13 September 2013; Accepted 19 September 2013
INTRODUCTION Drug-induced liver injury is the most frequent cause of acute liver failure in the USA, resulting in 600 liver transplants and over 100 deaths annually.1,2 More than 300 drugs have been associated with drug-induced liver injury, with manifestations varying from asymptomatic liver enzyme elevations to fulminant liver failure.3 The *Correspondence to: T. C. Cheetham, Kaiser Permanente Southern California Pharmacy Analytical Services 12254 Bellflower Blvd, Downey, CA 90242, USA. Email: [email protected] † Preliminary results were presented at the 28th International Conference on Pharmacoepidemiology & Therapeutic Risk Management (ICPE) 25 August 2012, Barcelona, Spain.
Copyright © 2013 John Wiley & Sons, Ltd.
most common drugs causing severe drug-induced liver injury include acetaminophen, antimicrobials, anti-epileptics, psychotropics, and antimetabolites.4 Despite measures to identify liver safety issues during drug development, most severe drug-induced liver injury is identified post-marketing. Thus, the burden falls on health care providers or post-marketing surveillance to identify cases of drug-induced liver injury. In practice, primary care physicians and hospitalists frequently misdiagnose drug-induced liver injury, resulting in diagnostic delays, prolonged exposure to the drug, and potential complications for patients.5 One reason is that drug-induced liver injury is generally a diagnosis of exclusion requiring other causes of liver injury
602
t. c. cheetham et al.
to be ruled out before a drug cause is considered likely. In 1993, international experts developed the Roussel Uclaf Causality Assessment Method (RUCAM) for identifying drug-induced liver injury.6 This simple causality scoring system accurately identifies drug-induced liver injury and generally correlates well with expert review.7,8 Some studies have reported low correlation between RUCAM and expert assessment and large inter-observer and intra-observer variability when using RUCAM.9,10 However, the RUCAM was recently endorsed by experts for causality assessment of drug-induced liver injury because of its sound evidence base.11 To identify drug-induced liver injury in the postmarketing setting, new systems and processes need to be developed. These systems need to be robust to allow identification using health care data or for use within an electronic medical record (EMR). A recent study established the feasibility of using computer algorithms to identify possible drug-induced liver injury in an EMR.12 The methodology has been further developed in this study to include a causality assessment component based on RUCAM criteria. We report here on the development of an electronic RUCAM (eRUCAM) causality assessment algorithm for drug-induced liver injury and its pilot testing in a population of patients receiving medications associated with hepatotoxicity. METHODS eRUCAM algorithm methodology The eRUCAM algorithm was developed using data formatted in conformance with the Observational Medical Outcomes Partnership (OMOP) common data model.13 We selected this publically available data model because the OMOP common data model was developed to support research involving the standardized detection of adverse drug effects, and it contains all of the data fields required by eRUCAM. The eRUCAM incorporates a modular design with each of seven modules focusing on one of the RUCAM criteria (Table 1 has a listing of the seven criteria). Subscores are generated for each criterion and summed to yield a total eRUCAM score, which provides the likelihood of drug-induced liver injury. Total scores are grouped as follows: highly probable (>8 points), probable (6–8 points), possible (3–5 points), unlikely (1–2 points), or excluded (≤0 points). To operationalize the definitions for eRUCAM and maintain consistency with current thinking on causality assessment for drug-induced liver injury, we engaged in discussions with an expert panel of hepatologists. The Copyright © 2013 John Wiley & Sons, Ltd.
Table 1. RUCAM criteria for assessing causality in drug-induced liver injury Criteria 1 Time between drug start and liver enzyme elevations 2 Time to resolution of liver enzyme elevations once the drug is stopped 3 Risk factors for drug-induced liver injury 4 Concomitant medication usage and timing 5 Non-drug causes of liver injury 6 Known information about the hepatotoxic potential of the drug 7 Liver enzyme elevations with drug rechallenge/readministration
Possible scores 0 to +2 2 to +3 0 to +2 3 to 0 3 to +2 0 to +2 2 to +3
programming steps are described here with additional detail in the appendix. Initial screening. The first step to the eRUCAM algorithm is the identification of drug exposures and screening for elevated liver enzymes (alanine aminotransferase [ALT] and alkaline phosphatase [ALKP]) temporally associated with drug initiation. The second step identifies the liver injury type (hepatocellular, cholestatic, or mixed) based on liver enzyme results drawn within 24 hours of each other. The type of liver injury was determined as follows:11
•
Hepatocellular: ALT ≥5 times upper limits of normal (5× ULN) and R ≥ 5 • Cholestatic: ALKP ≥ 2× ULN and R ≤ 2 • Mixed: ALT ≥ 5x ULN, ALKP ≥ ULN and 2 < R < 5 where R = [ALT value/ALT ULN]/[ALKP value/ALKP ULN] Muscle injury and alcoholic hepatitis are associated with ALT and aspartate aminotransferase (AST) elevations.14,15 To minimize inclusion of muscle injury, patients are not scored if their creatine phosphokinase or lactate dehydrogenase is greater than 5× ULN at the time of ALT or AST elevation; to exclude alcoholic hepatitis, the AST was required to be lower than the ALT at the initial scoring. Criterion 1: time between the drug start and liver enzyme elevations. Drug-induced liver injury generally occurs in the first 90 days following drug initiation or 30 days post-cessation.3,11 Liver enzyme elevations within this window of time receive more points. The drug start date is defined as the prescription dispensed date. Liver enzyme elevations must occur after the drug start date, and at least one of the liver enzyme elevations during follow-up needed to meet the ALT or ALKP threshold Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
603
drug-induced liver injury automated assessment
for defining the type of injury. The date of the first liver enzyme elevation meeting the threshold criteria defined the start of the liver injury for timing purposes.
Criterion 2: time to resolution of liver enzyme elevation once the drug is stopped. Criterion 2 examines the decline in serum liver enzyme levels over time following drug discontinuation.16 The percentage reduction is calculated on the basis of the difference between peak ALT (or ALKP for cholestatic and mixed injury) and the ULN. If the drug stop date is unknown, it is defined as the date of the first elevated liver enzyme test. Patients who receive a subsequent prescription for the same drug, by definition, have not stopped therapy and are scored accordingly.
Criterion 3: risk factors for drug-induced liver injury. Risk factors for drug-induced liver injury include age (≥55), pregnancy, and alcohol use.6 Age is determined on the date liver enzyme elevations are first noted. Coexistent pregnancy is defined on the basis of International Classification of Diseases, 9th edition (ICD-9) diagnosis and Current Procedural Terminology procedure codes, and alcohol use is based on ICD-9 codes for alcohol abuse, and laboratory results indicating an elevated mean corpuscular volume (The codes for alcohol abuse and pregnancy are provided in the appendix).
Criterion 4: concomitant medication usage and timing. Criterion 4 assesses the usage and timing of concomitant medications that are associated with liver injury. Points are deducted if the concomitant medication(s) and liver enzyme elevations meet the temporal association as defined in Criterion 1. For Criterion 4 scoring several steps were undertaken to develop a list of concomitant medications. In the first step, a list of medications known to be associated with drug-induced liver injury was generated from literature and expert review.3,17 Only those drugs with a high empirical Bayes geometric mean (EBGM ≥ 2.0) disproportionality score for drug-induced liver injury reporting in the FDA adverse event reporting system data were considered.18 Second, a listing of the 100 most commonly prescribed drugs in the USA was created. The final list of drugs for criterion 4 consisted of the intersection Copyright © 2013 John Wiley & Sons, Ltd.
of the two lists; drugs commonly prescribed in the USA with an elevated disproportionality score EBGM ≥ 2.0 for liver injury reporting. Other concomitant exposures such as toxic mushrooms were identified by ICD-9 codes. Criteria 5: non-drug causes of liver injury. Criterion 5 identifies non-drug causes of liver injury, which are separated into primary and secondary causes using diagnosis codes. The probability of drug-induced liver injury is lower in the presence of primary or secondary causes so points are deducted. The timing of non-drug causes and the start of liver injury is also important for scoring purposes so timeframes are specified. Table 2 lists the primary and secondary causes of hepatotoxicity as implemented in eRUCAM.
Criterion 6: known information about the hepatotoxic potential of the drug. Criterion 6 adds points based on information about the suspect drug and its association with drug-induced liver injury from US prescribing information or published case reports. No good source was found for US prescribing information so the Spanish drug-induced liver injury registry was used to specify drugs with published data indicating the potential for liver injury.3,17
Criterion 7: liver enzyme elevations with drug rechallenge/re-administration. Criterion 7 examines the presence or absence of liver enzyme elevations following drug re-administration (rechallenge) after the initial liver injury. Recurrent liver injury following Table 2.
Non-drug causes of liver injury*
Primary causes †
Acute viral hepatitis A, B, C, or E † Acute alcoholic hepatitis Biliary bbstruction* Recent hypotension* Severe passive liver congestion‡ Secondary causes
Non-alcoholic steatohepatitis‡ Hepatic cirrhosis‡ Hemochromatosis‡ Autoimmune hepatitis‡ Viral infections (cytomegalovirus, Epstein Barr or herpes simplex)* Based on ICD9 codes unless otherwise stated; † Identified within 30 days prior to and 90 days after the liver injury date. † Identified within 7 days prior to and 7 days after the liver injury date. ‡ Identified within 10 years prior to and 90 days after the liver injury date. *
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
604
t. c. cheetham et al.
rechallenge is highly suggestive of drug-induced liver injury so points are added to the eRUCAM score. Scoring is based on the relative increase in liver enzymes and whether the same drug or drug combination as in the initial liver injury occurred.
membership plus prescription benefit immediately prior to starting the drug. In addition, patients were required to have filled at least one prescription for any drug during the prior 12 months. Pilot chart review
Study setting The eRUCAM algorithm was tested on EMR data from Kaiser Permanente Southern California (KPSC). KPSC is a nonprofit, group-model managed care organization providing integrated healthcare services to approximately 3.4 million members. The KPSC membership closely mirrors the overall Southern California population.19 The EMR system documents all healthcare encounters. Detailed data are available for research purposes and includes patient demographics, prescription dispensing history, diagnosis and procedure codes associated with healthcare encounters, and laboratory test results. The IRB waived the requirement for informed consent. Study population For the initial eRUCAM pilot testing, patients who filled one or more prescriptions for drugs known to cause liver injury between 1 January 2003 and 31 August 2011 were identified. These study drugs included 13 drugs frequently associated with drug-induced liver injury in the USA and aripiprazole, which has recently been associated with serious hepatic disorders (Table 3).20,21 Patients needed to be 18 years or older at the time the drug was started and have 12 months of continuous
Table 3. Unadjudicated rates of probable or highly probable drug-induced liver injury cases by drug
Drug Isoniazid Phenytoin Levofloxacin Duloxetine Valproate Ciprofloxacin Amoxicillin/clavulanate Interferon-beta Lamotrigine Trimethoprim/ sulfamethoxazole Diclofenac Aripiprazole Nitrofurantoin Terbinafine
Number of cases
Number of exposures (% Total)
71 168 25 22 52 1050 237 5 17 435
16 745 (0.5%) 39 848 (1.2%) 17 781 (0.5%) 22 687 (0.7%) 58 351 (1.8%) 1 294 575 (39.0%) 302 095 (9.1%) 7 073 (0.2%) 30 226 (0.9%) 840 720 (25.3%)
42.4 42.2 14.1 9.7 8.9 8.1 7.8 7.1 5.6 5.2
29 10 172 18
67 563 (2.0%) 26 980 (0.8%) 531 354 (16.0%) 65 837 (2.0%)
4.3 3.7 3.2 2.7
Copyright © 2013 John Wiley & Sons, Ltd.
Rate per 10 000 patient exposures
Twenty cases of possible or probable drug-induced liver injury (based on eRUCAM scoring) were randomly selected for chart abstraction and adjudication to identify areas for algorithm improvement and to perform quality assurance. The objective was to beta-test the eRUCAM algorithm. The 20 abstracted and de-identified charts were reviewed by a board certified hepatologist (CMH) to generate a manual RUCAM score, which served as the gold standard. To minimize the capture of liver injury unrelated to the study drugs, a relatively healthy population was selected. Excluded from chart review were the following: (1) patients with a diagnosis of liver disease or comorbidity that could cause liver enzyme elevations in the 12 months prior to the index date; (2) patients with elevated liver enzyme tests in the 12 months prior to the index date; and (3) patients with concomitant medications [other than the 14 study drugs] potentially associated with liver injury. Statistical analysis Descriptive characteristics of the total exposed population as well as patient characteristics for the potential drug-induced liver injury population are reported. Unadjudicated rates of probable and highly probable liver injury events by eRUCAM were calculated per 10 000 exposures. For the 20 adjudicated cases, comparisons were made between the ‘gold standard’ manual RUCAM and eRUCAM scores using the Wilcoxon signed rank test. The concordance between the manual RUCAM and eRUCAM was calculated as the proportion of cases for which the two methods yielded identical causality groupings. RESULTS Between 1 January 2003 and 31 August 2011, a total of 1 239 071 patients received 3 321 835 study drug exposures. Four antibiotics (ciprofloxacin, trimethoprim/sulfamethoxazole, nitrofurantoin, and amoxicillin/clavulanate) accounted for 89.4% of the exposures. Across all study drug exposures, the eRUCAM algorithm identified 14 925 potential druginduced liver injury events (in 11 109 patients). The average age of the 11 109 patients was 59.7 years and 54.5% were women. The racial distribution was as Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
drug-induced liver injury automated assessment
follows: 44.5% White, 23.2% Hispanic, 10.2% Black, 5.9% Asian, and 16.2% other/unknown. The most common type of liver injury was cholestatic 65.0%, followed by hepatocellular 29.4%, and mixed 5.6%. In assessing causality between drug exposure and drug-induced liver injury (n = 14 925), the eRUCAM categorized 15.5% as probable or highly probable, 59.6% as possible, and 24.9% as unlikely related to the drug. The overall rate of probable or highly probable events was 6.9 cases per 10 000 exposures. Table 3 lists the unadjudicated rates of probable and highly probable events (n = 2311) per 10 000 exposures by drug; the highest rates occurred with isoniazid and phenytoin. The frequencies of possible scores under each of the criteria are reported in Table 4. The eRUCAM results for criteria 1 to 4 and 7 spanned the entire range of possible scores, and there were no obvious floor or ceiling effects. Scores for criterion 6 were fixed at 2 because the 14 drugs are known to cause liver injury. Only two of six possible Criterion 5 scores were generated by the eRUCAM with the vast majority of scores being zero, suggesting that the eRUCAM algorithm, as it was initially programmed, was not able to discern other non-drug causes of liver injury where negative points are awarded or in cases where other non-drug causes are ruled out and positive points are awarded. The pilot chart review included the following study drugs: isoniazid (n = 7), ciprofloxacin (n = 6), trimethoprim/sulfomethoxazole (n = 4), and one each for diclofenac, nitrofurantoin, and phenytoin. Overall there was 30% (6/20) concordance for causality between the manual RUCAM and the eRUCAM with concordance occurring for two probable/highly probable cases and four possible cases (Figure 1).
605
Overall the total scores tended to be higher for the eRUCAM compared with the manual RUCAM. Comparisons were made between the eRUCAM and manual RUCAM subscores for each of the seven criteria (Table 5). With five of the seven criteria, the RUCAM scores generated by manual review and the eRUCAM were identical at 80% or more of the time. The 100% concordance for criterion 6 reflects the selection of drugs used for this analysis, which are all known to cause liver injury. With criterion 5, only 9 of 20 (45%) cases received identical scores by both methods. Differences between the manual and electronic methods were found in the following situations: (1) patients with isolated elevations in ALKP where the eRUCAM did not account for bone fractures or bone metastases; (2) diagnoses of biliary obstruction as a cause for elevated liver enzymes falling outside the pre-specified time windows, which were therefore not counted by eRUCAM; or (3) patients with severe multi-organ failure, or sepsis, and hypotension that were not captured by eRUCAM. Another programming issue involving criterion 4 (concomitant medications) was revealed on chart review. Initially, a shortened list of 100 commonly used drugs with liver injury potential was selected for inclusion. However, several drugs associated with liver injury were not included on the initial list (rifampin, isotretinoin, erythromycin, and carbamazepine) and therefore not scored correctly by eRUCAM. DISCUSSION In this study, an automated eRUCAM causality assessment algorithm, for identifying drug-induced liver injury, was developed and tested in EMR data. Our results suggest that it is feasible to create an
Table 4. Frequency distribution of the eRUCAM scoring for the seven criteria
Copyright © 2013 John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
606
t. c. cheetham et al.
Figure 1. eRUCAM score versus manual RUCAM score for the 20 pilot cases*
Table 5. Comparison of differences between manual RUCAM and eRUCAM by criterion (n = 20) eRUCAM – manual RUCAM*
Frequency
Percent
p-value†
Criteria 1: time between the drug start and liver enzyme elevations 1 1 5 0.75 0 17 85 1 1 5 2 1 5 Criteria 2: time to resolution of liver enzyme elevations once the drug is stopped 2 1 0 4
1 1 17 1
5 5 85 5
1
10 80 10
1
75 25
0.06
45 15 40
0.001
Criteria 3: risk factors for drug-induced liver injury 1 0 1
2 16 2
Criteria 4: concomitant medication usage and timing 0 2
15 5
Criteria 5: non-drug causes of liver injury 0 2 3
9 3 8
Criteria 6: known information about the hepatotoxic potential of the suspect drug 0
20
100
.
Criteria 7: liver enzyme elevations with drug rechallenge/re-administration 0 2
19 1
95 5
1
*Number was generated by subtracting eRUCAM from the manual RUCAM (a score of zero indicates exact matching score). Wilcoxon signed rank test.
†
automated causality assessment algorithm with reasonable concordance between manual RUCAM and eRUCAM scoring. The exception was criterion 5 (non-drug causes of liver injury) where scores were Copyright © 2013 John Wiley & Sons, Ltd.
identical 45% of the time, and only two of the six possible scores were generated by eRUCAM. Chart abstractions revealed a number of programming issues not considered in the initial eRUCAM development. With criterion 5, these issues included the following: (1) reliance exclusively on ICD-9 codes for defining other causes without consideration of laboratory test results [specifically Hepatitis and viral disease testing]; (2) an incomplete listing of potential ICD-9 codes; and (3) the inability to obtain information from text fields. Additional issues were identified with Criterion 4, which only considered the most commonly prescribed drugs with elevated EBGM scores for inclusion, and as a result, some important drugs were missing from the concomitant drug list. Lastly, Criterion 3 included an elevated mean corpuscular volume (MCV) as an indicator for alcohol abuse but many other diseases associated with elevated MCV were found on chart review rendering this definition unworkable. Use of the internationally endorsed and validated RUCAM, with input from an expert panel of hepatologists to operationalize definitions and provide programming recommendations for the criteria was a key strength for developing the eRUCAM. RUCAM is the “gold standard” to objectively and accurately assess causality for drug-induced liver injury and is endorsed by many hepatologists, researchers, and regulatory authorities.22 Another strength of our approach was conducting preliminary comparisons between the eRUCAM and manual RUCAM. These comparisons revealed concordance for most criteria, verifying algorithm programming, and identifying areas for future improvement. An additional strength was eRUCAM’s uses of the publically available OMOP common data model, which enables others to test and further develop the algorithm. Several limitations need to be considered when interpreting these results. First, the eRUCAM has only Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
drug-induced liver injury automated assessment
been tested at one site (KPSC) and further testing and validation is needed. The goal is to make eRUCAM publically available through OMOP allowing others to test and develop the algorithm. Second, for the pilot chart abstraction, only possible and probable/highly probable charts were selected. This was performed to examine the coding decisions, but a broader range of charts needs to be reviewed in future efforts. Another limitation of the eRUCAM methodology is that it is restricted to information that is easily obtained from EMR data (diagnosis and procedure codes, prescriptions, and laboratories). There is information in the text fields (progress notes, consults, and procedure findings) of the EMR that can supplement diagnosis codes such as information on alcohol use, pregnancy, and alternative causes of liver injury that are available to an expert clinician during chart review. The use of natural language processing methodologies to mine these text fields for important key words and phrases is another area of investigation that could improve the capabilities of the eRUCAM causality assessment tool.23 Another potential area for future work includes refining the seven RUCAM criteria and tailoring the algorithm parameters with a determination of appropriate weighting for each criteria based on statistical evaluations. Our preliminary findings suggest that the eRUCAM causality assessment tool shows promise in identifying clinically important drug-induced liver injury events. With additional revisions and testing, the eRUCAM could provide the appropriate balance of sensitivity and specificity to aid in identifying liver injury in clinical and drug safety surveillance settings. In clinical settings, the eRUCAM could potentially be linked to an EMR and provide alerts to providers regarding possible drug-induced liver injuries, prompting further investigation into causes of elevated liver enzymes. For drug safety, the eRUCAM could be implemented into prospective surveillance systems to aid in the identification of potential drug-induced liver injury events. For example, the FDA Sentinel Initiative could employ an algorithm like the eRUCAM to monitor safety and identify signals for further evaluation.24 In conclusion, the eRUCAM causality assessment tool can potentially be used to detect drug-induced liver injury in the clinic and in drug safety surveillance. Further refinement and expanded use of the tool could enhance clinical practice and safety surveillance. CONFLICT OF INTEREST Julie I. Papay and Gregory Powell are full-time employees of GlaxoSmithKline (GSK). Christine M Copyright © 2013 John Wiley & Sons, Ltd.
607
Hunt was a full-time employee of GSK at the time of this work. The other authors reported no conflicts of interest. KEY POINTS • Our preliminary results suggest that it is possible to develop an automated causality assessment algorithm for drug-induced liver injury. • With additional testing and refinement, an automated causality assessment algorithm could be used in clinical settings and in drug safety surveillance.
ETHICS STATEMENT The IRB waived the requirement for informed consent. ACKNOWLEDGEMENTS The authors wish to thank the GlaxoSmithKline Hepatotoxicity Board for their review and advice in the development of the eRUCAM algorithm, Robert Azadian for his assistance in managing the project, and Patrick Ryan PhD (OMOP) for his assistance in the development of the proposal. This study was co-funded by KPSC-Pharmacy Analytical Services and United BioSource Corporation (UBC). UBC developed the computer programs in collaboration with GSK and KPSC investigators. REFERENCES 1. Ostapowicz G, Fontana RJ, Schiodt FV, et al. Results of a prospective study of acute liver failure at 17 tertiary care centers in the United States. Ann Intern Med 2002; 137: 947–954. 2. Bleibel W, Kim S, D’Silva K, et al. Drug-induced liver injury: review article. Dig Dis Sci 2007; 52: 2463–2471. 3. Suzuki A, Andrade RJ, Bjornsson E, et al. Drugs associated with hepatotoxicity and their reporting frequency of liver adverse events in VigiBase: unified list based on international collaborative work. Drug Saf 2010; 33: 503–522. 4. Reuben A, Koch DG, Lee WM. Drug-induced acute liver failure: results of a U. S. multicenter, prospective study. Hepatology 2010; 52: 2065–2076. 5. Aithal GP, Rawlins MD, Day CP. Accuracy of hepatic adverse drug reaction reporting in one English health region. BMJ 1999; 319: 1541. 6. Danan G, Benichou C. Causality assessment of adverse reactions to drugs--I. A novel method based on the conclusions of international consensus meetings: application to drug-induced liver injuries. J Clin Epidemiol 1993; 46: 1323–1330. 7. Andrade RJ, Camargo Raquel, Lucena MI, et al. Causality assessment in druginduced hepatotoxicity. Expert Opin Drug Saf 2004; 3: 329–344. 8. Miljkovic MM, Dobric S, Dragojevic-Simic V. Consistency between causality assessments obtained with two scales and their agreement with clinical judgments in hepatotoxicity. Pharmacoepidemiol Drug Saf 2011; 20: 272–285. 9. Rockey DC, Seeff LB, Rochon J, et al. Causality assessment in drug-induced liver injury using a structured expert opinion process: comparison to the Roussel-Uclaf causality assessment method. Hepatol. 2010; 51: 2117–2126. 10. Rochon J, Protiva P, Seef LB, et al. Reliability of the Roussel Uclaf causality assessment method for assessing causality in drug-induced liver injury. Hepatol. 2008; 48: 1175–1183. 11. Aithal GP, Watkins PB, Andrade RJ, et al. Case definition and phenotype standardization in drug-induced liver injury. Clin Pharmacol Ther 2011; 89: 806–815.
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
608
t. c. cheetham et al.
12. Shin J, Hunt CM, Suzuki A, et al. Characterizing phenotypes and outcomes of drug-associated liver injury using electronic medical record data. Pharmacoepi Drug Safety. 2013; 22: 190–198. 13. Overhage JM, Ryan PB, Reich CG, et al. Validation of a common data model for active safety surveillance research. J Am Med Inform Assoc 2012; 19: 54–60. 14. Nathwani RA, Pais S, Reynolds TB, et al. Serum alanine aminotransferase in skeletal muscle diseases. Hepatology 2005; 41: 380–382. 15. Green RM, Flamm S. AGA technical review on the evaluation of liver chemistry tests. Gastroenterology 2002; 123: 1367–1384. 16. Watkins PB, Seligman PJ, Pears JS, Avigan MI, Senior JR. Using controlled clinical trials to learn more about acute drug-induced liver injury. Hepatology 2008; 48: 1680–1689. 17. Spanish DILI Registry. http://www.spanishdili.uma.es/index.php?option=com_ content&view=article&id=92&Itemid=83&lang=en [13 December 2012] 18. DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. The Am Statistician. 1999; 53: 177–201. 19. Koebnick C, Langer-Gould R, Gould M, et al. Sociodemographic characteristics of members of a large, integrated health care system: comparison with US census bureau data. Perm J. 2012; 16: 37–41. 20. Chalasani N, Fontana RJ, Bonkovsky HL, et al. Causes, clinical features, and outcomes from a prospective study of drug-induced liver injury in the United States. Gastroenterology 2008; 135: 1924–1934, 1934 e1-4.
Copyright © 2013 John Wiley & Sons, Ltd.
21. FDA. New Molecular Entity Postmarketing Safety Evaluation Pilot Program Final Report. Volume 2011. http://www.fda.gov/Drugs/DrugSafety/ PostmarketDrugSafetyInformationforPatientsandProviders/ucm185252. htm#Results [December 2012] 22. Garcia-Cortez M, Stephens C, Lucena MI, et al. Causality assessment methods in drug-induced liver injury: strengths and weaknesses. J Hepatology 2011; 55: 683–691. http://jamia.bmj.com/content/early/2013/07/08/amiajnl-2013-001930. abstract. [10 September 2013] 23. Overby CL, Pathak J, Gottesman O, et al. A collaborative approach to developing an electronic health record phenotyping algorithm for drug-induced liver injury. 24. Platt R, Carnahan RM, Brown JS, et al. The U.S. Food and Drug Administration’s Mini-Sentinel program: status and direction. Pharmacoepi Drug Safety 2012; 21(S1): 1–8.
SUPPORTING INFORMATION Additional supporting information may be found in the online version of this article at the publisher’s web site.
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
ORIGINAL REPORT
An automated causality assessment algorithm to detect drug-induced liver injury in electronic medical record data† T. Craig Cheetham1*, Janet Lee1, Christine M. Hunt2,3, Fang Niu1, Steph Reisinger4, Rich Murray4, Greg Powell2 and Julie Papay2 1
Kaiser Permanente Southern California, Pharmacy Analytical Services, CA, USA GlaxoSmithKline, NC, USA 3 Duke University, NC, USA 4 United BioSource Corporation, PA, USA 2
ABSTRACT Purpose The aim of this study was to develop an automated causality assessment algorithm to identify drug-induced liver injury. Methods The Roussel Uclaf Causality Assessment Method (RUCAM) is an algorithm for determining the causal association between a drug and liver injury. In collaboration with hepatology experts, definitions were developed for the RUCAM criteria to operationalize an electronic RUCAM (eRUCAM). The eRUCAM was tested in a population of patients taking 14 drugs with a characteristic phenotype for liver injury. Quality assurance for programming specifications involved comparisons between scores generated by the eRUCAM, for probable and highly probable cases, and expert manual RUCAM (n = 20). Concordance between eRUCAM and manual RUCAM subscores and total score was tested using the Wilcoxon signed rank test. Results Causality scores were the same for 6 of 20 patients (30%) by manual and eRUCAM algorithms. Analysis of subscores revealed ≥80% concordance between manual and eRUCAM for five of the seven criteria. In general, the total scores tended to be higher for the eRUCAM compared with the manual RUCAM. Programming issues were identified for criterion 5 ‘non-drug causes of liver injury’ where significant differences existed between manual and eRUCAM scoring (p = 0.001). For criterion 5, identical scores occurred in 9 of 20 patients (45%), and manual review identified additional codes, timing criteria, and laboratory results for improving subsequent eRUCAM revisions. Conclusion The eRUCAM had generally good concordance with manual RUCAM scoring. These preliminary findings suggest that the eRUCAM algorithm is feasible and could have application in clinical practice and drug safety surveillance. Copyright © 2013 John Wiley & Sons, Ltd. key words—liver injury; drug-induced; causality assessment; algorithm; drug safety; risk assessment; automated; pharmacoepidemiology Received 31 May 2013; Revised 13 September 2013; Accepted 19 September 2013
INTRODUCTION Drug-induced liver injury is the most frequent cause of acute liver failure in the USA, resulting in 600 liver transplants and over 100 deaths annually.1,2 More than 300 drugs have been associated with drug-induced liver injury, with manifestations varying from asymptomatic liver enzyme elevations to fulminant liver failure.3 The *Correspondence to: T. C. Cheetham, Kaiser Permanente Southern California Pharmacy Analytical Services 12254 Bellflower Blvd, Downey, CA 90242, USA. Email: [email protected] † Preliminary results were presented at the 28th International Conference on Pharmacoepidemiology & Therapeutic Risk Management (ICPE) 25 August 2012, Barcelona, Spain.
Copyright © 2013 John Wiley & Sons, Ltd.
most common drugs causing severe drug-induced liver injury include acetaminophen, antimicrobials, anti-epileptics, psychotropics, and antimetabolites.4 Despite measures to identify liver safety issues during drug development, most severe drug-induced liver injury is identified post-marketing. Thus, the burden falls on health care providers or post-marketing surveillance to identify cases of drug-induced liver injury. In practice, primary care physicians and hospitalists frequently misdiagnose drug-induced liver injury, resulting in diagnostic delays, prolonged exposure to the drug, and potential complications for patients.5 One reason is that drug-induced liver injury is generally a diagnosis of exclusion requiring other causes of liver injury
602
t. c. cheetham et al.
to be ruled out before a drug cause is considered likely. In 1993, international experts developed the Roussel Uclaf Causality Assessment Method (RUCAM) for identifying drug-induced liver injury.6 This simple causality scoring system accurately identifies drug-induced liver injury and generally correlates well with expert review.7,8 Some studies have reported low correlation between RUCAM and expert assessment and large inter-observer and intra-observer variability when using RUCAM.9,10 However, the RUCAM was recently endorsed by experts for causality assessment of drug-induced liver injury because of its sound evidence base.11 To identify drug-induced liver injury in the postmarketing setting, new systems and processes need to be developed. These systems need to be robust to allow identification using health care data or for use within an electronic medical record (EMR). A recent study established the feasibility of using computer algorithms to identify possible drug-induced liver injury in an EMR.12 The methodology has been further developed in this study to include a causality assessment component based on RUCAM criteria. We report here on the development of an electronic RUCAM (eRUCAM) causality assessment algorithm for drug-induced liver injury and its pilot testing in a population of patients receiving medications associated with hepatotoxicity. METHODS eRUCAM algorithm methodology The eRUCAM algorithm was developed using data formatted in conformance with the Observational Medical Outcomes Partnership (OMOP) common data model.13 We selected this publically available data model because the OMOP common data model was developed to support research involving the standardized detection of adverse drug effects, and it contains all of the data fields required by eRUCAM. The eRUCAM incorporates a modular design with each of seven modules focusing on one of the RUCAM criteria (Table 1 has a listing of the seven criteria). Subscores are generated for each criterion and summed to yield a total eRUCAM score, which provides the likelihood of drug-induced liver injury. Total scores are grouped as follows: highly probable (>8 points), probable (6–8 points), possible (3–5 points), unlikely (1–2 points), or excluded (≤0 points). To operationalize the definitions for eRUCAM and maintain consistency with current thinking on causality assessment for drug-induced liver injury, we engaged in discussions with an expert panel of hepatologists. The Copyright © 2013 John Wiley & Sons, Ltd.
Table 1. RUCAM criteria for assessing causality in drug-induced liver injury Criteria 1 Time between drug start and liver enzyme elevations 2 Time to resolution of liver enzyme elevations once the drug is stopped 3 Risk factors for drug-induced liver injury 4 Concomitant medication usage and timing 5 Non-drug causes of liver injury 6 Known information about the hepatotoxic potential of the drug 7 Liver enzyme elevations with drug rechallenge/readministration
Possible scores 0 to +2 2 to +3 0 to +2 3 to 0 3 to +2 0 to +2 2 to +3
programming steps are described here with additional detail in the appendix. Initial screening. The first step to the eRUCAM algorithm is the identification of drug exposures and screening for elevated liver enzymes (alanine aminotransferase [ALT] and alkaline phosphatase [ALKP]) temporally associated with drug initiation. The second step identifies the liver injury type (hepatocellular, cholestatic, or mixed) based on liver enzyme results drawn within 24 hours of each other. The type of liver injury was determined as follows:11
•
Hepatocellular: ALT ≥5 times upper limits of normal (5× ULN) and R ≥ 5 • Cholestatic: ALKP ≥ 2× ULN and R ≤ 2 • Mixed: ALT ≥ 5x ULN, ALKP ≥ ULN and 2 < R < 5 where R = [ALT value/ALT ULN]/[ALKP value/ALKP ULN] Muscle injury and alcoholic hepatitis are associated with ALT and aspartate aminotransferase (AST) elevations.14,15 To minimize inclusion of muscle injury, patients are not scored if their creatine phosphokinase or lactate dehydrogenase is greater than 5× ULN at the time of ALT or AST elevation; to exclude alcoholic hepatitis, the AST was required to be lower than the ALT at the initial scoring. Criterion 1: time between the drug start and liver enzyme elevations. Drug-induced liver injury generally occurs in the first 90 days following drug initiation or 30 days post-cessation.3,11 Liver enzyme elevations within this window of time receive more points. The drug start date is defined as the prescription dispensed date. Liver enzyme elevations must occur after the drug start date, and at least one of the liver enzyme elevations during follow-up needed to meet the ALT or ALKP threshold Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
603
drug-induced liver injury automated assessment
for defining the type of injury. The date of the first liver enzyme elevation meeting the threshold criteria defined the start of the liver injury for timing purposes.
Criterion 2: time to resolution of liver enzyme elevation once the drug is stopped. Criterion 2 examines the decline in serum liver enzyme levels over time following drug discontinuation.16 The percentage reduction is calculated on the basis of the difference between peak ALT (or ALKP for cholestatic and mixed injury) and the ULN. If the drug stop date is unknown, it is defined as the date of the first elevated liver enzyme test. Patients who receive a subsequent prescription for the same drug, by definition, have not stopped therapy and are scored accordingly.
Criterion 3: risk factors for drug-induced liver injury. Risk factors for drug-induced liver injury include age (≥55), pregnancy, and alcohol use.6 Age is determined on the date liver enzyme elevations are first noted. Coexistent pregnancy is defined on the basis of International Classification of Diseases, 9th edition (ICD-9) diagnosis and Current Procedural Terminology procedure codes, and alcohol use is based on ICD-9 codes for alcohol abuse, and laboratory results indicating an elevated mean corpuscular volume (The codes for alcohol abuse and pregnancy are provided in the appendix).
Criterion 4: concomitant medication usage and timing. Criterion 4 assesses the usage and timing of concomitant medications that are associated with liver injury. Points are deducted if the concomitant medication(s) and liver enzyme elevations meet the temporal association as defined in Criterion 1. For Criterion 4 scoring several steps were undertaken to develop a list of concomitant medications. In the first step, a list of medications known to be associated with drug-induced liver injury was generated from literature and expert review.3,17 Only those drugs with a high empirical Bayes geometric mean (EBGM ≥ 2.0) disproportionality score for drug-induced liver injury reporting in the FDA adverse event reporting system data were considered.18 Second, a listing of the 100 most commonly prescribed drugs in the USA was created. The final list of drugs for criterion 4 consisted of the intersection Copyright © 2013 John Wiley & Sons, Ltd.
of the two lists; drugs commonly prescribed in the USA with an elevated disproportionality score EBGM ≥ 2.0 for liver injury reporting. Other concomitant exposures such as toxic mushrooms were identified by ICD-9 codes. Criteria 5: non-drug causes of liver injury. Criterion 5 identifies non-drug causes of liver injury, which are separated into primary and secondary causes using diagnosis codes. The probability of drug-induced liver injury is lower in the presence of primary or secondary causes so points are deducted. The timing of non-drug causes and the start of liver injury is also important for scoring purposes so timeframes are specified. Table 2 lists the primary and secondary causes of hepatotoxicity as implemented in eRUCAM.
Criterion 6: known information about the hepatotoxic potential of the drug. Criterion 6 adds points based on information about the suspect drug and its association with drug-induced liver injury from US prescribing information or published case reports. No good source was found for US prescribing information so the Spanish drug-induced liver injury registry was used to specify drugs with published data indicating the potential for liver injury.3,17
Criterion 7: liver enzyme elevations with drug rechallenge/re-administration. Criterion 7 examines the presence or absence of liver enzyme elevations following drug re-administration (rechallenge) after the initial liver injury. Recurrent liver injury following Table 2.
Non-drug causes of liver injury*
Primary causes †
Acute viral hepatitis A, B, C, or E † Acute alcoholic hepatitis Biliary bbstruction* Recent hypotension* Severe passive liver congestion‡ Secondary causes
Non-alcoholic steatohepatitis‡ Hepatic cirrhosis‡ Hemochromatosis‡ Autoimmune hepatitis‡ Viral infections (cytomegalovirus, Epstein Barr or herpes simplex)* Based on ICD9 codes unless otherwise stated; † Identified within 30 days prior to and 90 days after the liver injury date. † Identified within 7 days prior to and 7 days after the liver injury date. ‡ Identified within 10 years prior to and 90 days after the liver injury date. *
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
604
t. c. cheetham et al.
rechallenge is highly suggestive of drug-induced liver injury so points are added to the eRUCAM score. Scoring is based on the relative increase in liver enzymes and whether the same drug or drug combination as in the initial liver injury occurred.
membership plus prescription benefit immediately prior to starting the drug. In addition, patients were required to have filled at least one prescription for any drug during the prior 12 months. Pilot chart review
Study setting The eRUCAM algorithm was tested on EMR data from Kaiser Permanente Southern California (KPSC). KPSC is a nonprofit, group-model managed care organization providing integrated healthcare services to approximately 3.4 million members. The KPSC membership closely mirrors the overall Southern California population.19 The EMR system documents all healthcare encounters. Detailed data are available for research purposes and includes patient demographics, prescription dispensing history, diagnosis and procedure codes associated with healthcare encounters, and laboratory test results. The IRB waived the requirement for informed consent. Study population For the initial eRUCAM pilot testing, patients who filled one or more prescriptions for drugs known to cause liver injury between 1 January 2003 and 31 August 2011 were identified. These study drugs included 13 drugs frequently associated with drug-induced liver injury in the USA and aripiprazole, which has recently been associated with serious hepatic disorders (Table 3).20,21 Patients needed to be 18 years or older at the time the drug was started and have 12 months of continuous
Table 3. Unadjudicated rates of probable or highly probable drug-induced liver injury cases by drug
Drug Isoniazid Phenytoin Levofloxacin Duloxetine Valproate Ciprofloxacin Amoxicillin/clavulanate Interferon-beta Lamotrigine Trimethoprim/ sulfamethoxazole Diclofenac Aripiprazole Nitrofurantoin Terbinafine
Number of cases
Number of exposures (% Total)
71 168 25 22 52 1050 237 5 17 435
16 745 (0.5%) 39 848 (1.2%) 17 781 (0.5%) 22 687 (0.7%) 58 351 (1.8%) 1 294 575 (39.0%) 302 095 (9.1%) 7 073 (0.2%) 30 226 (0.9%) 840 720 (25.3%)
42.4 42.2 14.1 9.7 8.9 8.1 7.8 7.1 5.6 5.2
29 10 172 18
67 563 (2.0%) 26 980 (0.8%) 531 354 (16.0%) 65 837 (2.0%)
4.3 3.7 3.2 2.7
Copyright © 2013 John Wiley & Sons, Ltd.
Rate per 10 000 patient exposures
Twenty cases of possible or probable drug-induced liver injury (based on eRUCAM scoring) were randomly selected for chart abstraction and adjudication to identify areas for algorithm improvement and to perform quality assurance. The objective was to beta-test the eRUCAM algorithm. The 20 abstracted and de-identified charts were reviewed by a board certified hepatologist (CMH) to generate a manual RUCAM score, which served as the gold standard. To minimize the capture of liver injury unrelated to the study drugs, a relatively healthy population was selected. Excluded from chart review were the following: (1) patients with a diagnosis of liver disease or comorbidity that could cause liver enzyme elevations in the 12 months prior to the index date; (2) patients with elevated liver enzyme tests in the 12 months prior to the index date; and (3) patients with concomitant medications [other than the 14 study drugs] potentially associated with liver injury. Statistical analysis Descriptive characteristics of the total exposed population as well as patient characteristics for the potential drug-induced liver injury population are reported. Unadjudicated rates of probable and highly probable liver injury events by eRUCAM were calculated per 10 000 exposures. For the 20 adjudicated cases, comparisons were made between the ‘gold standard’ manual RUCAM and eRUCAM scores using the Wilcoxon signed rank test. The concordance between the manual RUCAM and eRUCAM was calculated as the proportion of cases for which the two methods yielded identical causality groupings. RESULTS Between 1 January 2003 and 31 August 2011, a total of 1 239 071 patients received 3 321 835 study drug exposures. Four antibiotics (ciprofloxacin, trimethoprim/sulfamethoxazole, nitrofurantoin, and amoxicillin/clavulanate) accounted for 89.4% of the exposures. Across all study drug exposures, the eRUCAM algorithm identified 14 925 potential druginduced liver injury events (in 11 109 patients). The average age of the 11 109 patients was 59.7 years and 54.5% were women. The racial distribution was as Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
drug-induced liver injury automated assessment
follows: 44.5% White, 23.2% Hispanic, 10.2% Black, 5.9% Asian, and 16.2% other/unknown. The most common type of liver injury was cholestatic 65.0%, followed by hepatocellular 29.4%, and mixed 5.6%. In assessing causality between drug exposure and drug-induced liver injury (n = 14 925), the eRUCAM categorized 15.5% as probable or highly probable, 59.6% as possible, and 24.9% as unlikely related to the drug. The overall rate of probable or highly probable events was 6.9 cases per 10 000 exposures. Table 3 lists the unadjudicated rates of probable and highly probable events (n = 2311) per 10 000 exposures by drug; the highest rates occurred with isoniazid and phenytoin. The frequencies of possible scores under each of the criteria are reported in Table 4. The eRUCAM results for criteria 1 to 4 and 7 spanned the entire range of possible scores, and there were no obvious floor or ceiling effects. Scores for criterion 6 were fixed at 2 because the 14 drugs are known to cause liver injury. Only two of six possible Criterion 5 scores were generated by the eRUCAM with the vast majority of scores being zero, suggesting that the eRUCAM algorithm, as it was initially programmed, was not able to discern other non-drug causes of liver injury where negative points are awarded or in cases where other non-drug causes are ruled out and positive points are awarded. The pilot chart review included the following study drugs: isoniazid (n = 7), ciprofloxacin (n = 6), trimethoprim/sulfomethoxazole (n = 4), and one each for diclofenac, nitrofurantoin, and phenytoin. Overall there was 30% (6/20) concordance for causality between the manual RUCAM and the eRUCAM with concordance occurring for two probable/highly probable cases and four possible cases (Figure 1).
605
Overall the total scores tended to be higher for the eRUCAM compared with the manual RUCAM. Comparisons were made between the eRUCAM and manual RUCAM subscores for each of the seven criteria (Table 5). With five of the seven criteria, the RUCAM scores generated by manual review and the eRUCAM were identical at 80% or more of the time. The 100% concordance for criterion 6 reflects the selection of drugs used for this analysis, which are all known to cause liver injury. With criterion 5, only 9 of 20 (45%) cases received identical scores by both methods. Differences between the manual and electronic methods were found in the following situations: (1) patients with isolated elevations in ALKP where the eRUCAM did not account for bone fractures or bone metastases; (2) diagnoses of biliary obstruction as a cause for elevated liver enzymes falling outside the pre-specified time windows, which were therefore not counted by eRUCAM; or (3) patients with severe multi-organ failure, or sepsis, and hypotension that were not captured by eRUCAM. Another programming issue involving criterion 4 (concomitant medications) was revealed on chart review. Initially, a shortened list of 100 commonly used drugs with liver injury potential was selected for inclusion. However, several drugs associated with liver injury were not included on the initial list (rifampin, isotretinoin, erythromycin, and carbamazepine) and therefore not scored correctly by eRUCAM. DISCUSSION In this study, an automated eRUCAM causality assessment algorithm, for identifying drug-induced liver injury, was developed and tested in EMR data. Our results suggest that it is feasible to create an
Table 4. Frequency distribution of the eRUCAM scoring for the seven criteria
Copyright © 2013 John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
606
t. c. cheetham et al.
Figure 1. eRUCAM score versus manual RUCAM score for the 20 pilot cases*
Table 5. Comparison of differences between manual RUCAM and eRUCAM by criterion (n = 20) eRUCAM – manual RUCAM*
Frequency
Percent
p-value†
Criteria 1: time between the drug start and liver enzyme elevations 1 1 5 0.75 0 17 85 1 1 5 2 1 5 Criteria 2: time to resolution of liver enzyme elevations once the drug is stopped 2 1 0 4
1 1 17 1
5 5 85 5
1
10 80 10
1
75 25
0.06
45 15 40
0.001
Criteria 3: risk factors for drug-induced liver injury 1 0 1
2 16 2
Criteria 4: concomitant medication usage and timing 0 2
15 5
Criteria 5: non-drug causes of liver injury 0 2 3
9 3 8
Criteria 6: known information about the hepatotoxic potential of the suspect drug 0
20
100
.
Criteria 7: liver enzyme elevations with drug rechallenge/re-administration 0 2
19 1
95 5
1
*Number was generated by subtracting eRUCAM from the manual RUCAM (a score of zero indicates exact matching score). Wilcoxon signed rank test.
†
automated causality assessment algorithm with reasonable concordance between manual RUCAM and eRUCAM scoring. The exception was criterion 5 (non-drug causes of liver injury) where scores were Copyright © 2013 John Wiley & Sons, Ltd.
identical 45% of the time, and only two of the six possible scores were generated by eRUCAM. Chart abstractions revealed a number of programming issues not considered in the initial eRUCAM development. With criterion 5, these issues included the following: (1) reliance exclusively on ICD-9 codes for defining other causes without consideration of laboratory test results [specifically Hepatitis and viral disease testing]; (2) an incomplete listing of potential ICD-9 codes; and (3) the inability to obtain information from text fields. Additional issues were identified with Criterion 4, which only considered the most commonly prescribed drugs with elevated EBGM scores for inclusion, and as a result, some important drugs were missing from the concomitant drug list. Lastly, Criterion 3 included an elevated mean corpuscular volume (MCV) as an indicator for alcohol abuse but many other diseases associated with elevated MCV were found on chart review rendering this definition unworkable. Use of the internationally endorsed and validated RUCAM, with input from an expert panel of hepatologists to operationalize definitions and provide programming recommendations for the criteria was a key strength for developing the eRUCAM. RUCAM is the “gold standard” to objectively and accurately assess causality for drug-induced liver injury and is endorsed by many hepatologists, researchers, and regulatory authorities.22 Another strength of our approach was conducting preliminary comparisons between the eRUCAM and manual RUCAM. These comparisons revealed concordance for most criteria, verifying algorithm programming, and identifying areas for future improvement. An additional strength was eRUCAM’s uses of the publically available OMOP common data model, which enables others to test and further develop the algorithm. Several limitations need to be considered when interpreting these results. First, the eRUCAM has only Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
drug-induced liver injury automated assessment
been tested at one site (KPSC) and further testing and validation is needed. The goal is to make eRUCAM publically available through OMOP allowing others to test and develop the algorithm. Second, for the pilot chart abstraction, only possible and probable/highly probable charts were selected. This was performed to examine the coding decisions, but a broader range of charts needs to be reviewed in future efforts. Another limitation of the eRUCAM methodology is that it is restricted to information that is easily obtained from EMR data (diagnosis and procedure codes, prescriptions, and laboratories). There is information in the text fields (progress notes, consults, and procedure findings) of the EMR that can supplement diagnosis codes such as information on alcohol use, pregnancy, and alternative causes of liver injury that are available to an expert clinician during chart review. The use of natural language processing methodologies to mine these text fields for important key words and phrases is another area of investigation that could improve the capabilities of the eRUCAM causality assessment tool.23 Another potential area for future work includes refining the seven RUCAM criteria and tailoring the algorithm parameters with a determination of appropriate weighting for each criteria based on statistical evaluations. Our preliminary findings suggest that the eRUCAM causality assessment tool shows promise in identifying clinically important drug-induced liver injury events. With additional revisions and testing, the eRUCAM could provide the appropriate balance of sensitivity and specificity to aid in identifying liver injury in clinical and drug safety surveillance settings. In clinical settings, the eRUCAM could potentially be linked to an EMR and provide alerts to providers regarding possible drug-induced liver injuries, prompting further investigation into causes of elevated liver enzymes. For drug safety, the eRUCAM could be implemented into prospective surveillance systems to aid in the identification of potential drug-induced liver injury events. For example, the FDA Sentinel Initiative could employ an algorithm like the eRUCAM to monitor safety and identify signals for further evaluation.24 In conclusion, the eRUCAM causality assessment tool can potentially be used to detect drug-induced liver injury in the clinic and in drug safety surveillance. Further refinement and expanded use of the tool could enhance clinical practice and safety surveillance. CONFLICT OF INTEREST Julie I. Papay and Gregory Powell are full-time employees of GlaxoSmithKline (GSK). Christine M Copyright © 2013 John Wiley & Sons, Ltd.
607
Hunt was a full-time employee of GSK at the time of this work. The other authors reported no conflicts of interest. KEY POINTS • Our preliminary results suggest that it is possible to develop an automated causality assessment algorithm for drug-induced liver injury. • With additional testing and refinement, an automated causality assessment algorithm could be used in clinical settings and in drug safety surveillance.
ETHICS STATEMENT The IRB waived the requirement for informed consent. ACKNOWLEDGEMENTS The authors wish to thank the GlaxoSmithKline Hepatotoxicity Board for their review and advice in the development of the eRUCAM algorithm, Robert Azadian for his assistance in managing the project, and Patrick Ryan PhD (OMOP) for his assistance in the development of the proposal. This study was co-funded by KPSC-Pharmacy Analytical Services and United BioSource Corporation (UBC). UBC developed the computer programs in collaboration with GSK and KPSC investigators. REFERENCES 1. Ostapowicz G, Fontana RJ, Schiodt FV, et al. Results of a prospective study of acute liver failure at 17 tertiary care centers in the United States. Ann Intern Med 2002; 137: 947–954. 2. Bleibel W, Kim S, D’Silva K, et al. Drug-induced liver injury: review article. Dig Dis Sci 2007; 52: 2463–2471. 3. Suzuki A, Andrade RJ, Bjornsson E, et al. Drugs associated with hepatotoxicity and their reporting frequency of liver adverse events in VigiBase: unified list based on international collaborative work. Drug Saf 2010; 33: 503–522. 4. Reuben A, Koch DG, Lee WM. Drug-induced acute liver failure: results of a U. S. multicenter, prospective study. Hepatology 2010; 52: 2065–2076. 5. Aithal GP, Rawlins MD, Day CP. Accuracy of hepatic adverse drug reaction reporting in one English health region. BMJ 1999; 319: 1541. 6. Danan G, Benichou C. Causality assessment of adverse reactions to drugs--I. A novel method based on the conclusions of international consensus meetings: application to drug-induced liver injuries. J Clin Epidemiol 1993; 46: 1323–1330. 7. Andrade RJ, Camargo Raquel, Lucena MI, et al. Causality assessment in druginduced hepatotoxicity. Expert Opin Drug Saf 2004; 3: 329–344. 8. Miljkovic MM, Dobric S, Dragojevic-Simic V. Consistency between causality assessments obtained with two scales and their agreement with clinical judgments in hepatotoxicity. Pharmacoepidemiol Drug Saf 2011; 20: 272–285. 9. Rockey DC, Seeff LB, Rochon J, et al. Causality assessment in drug-induced liver injury using a structured expert opinion process: comparison to the Roussel-Uclaf causality assessment method. Hepatol. 2010; 51: 2117–2126. 10. Rochon J, Protiva P, Seef LB, et al. Reliability of the Roussel Uclaf causality assessment method for assessing causality in drug-induced liver injury. Hepatol. 2008; 48: 1175–1183. 11. Aithal GP, Watkins PB, Andrade RJ, et al. Case definition and phenotype standardization in drug-induced liver injury. Clin Pharmacol Ther 2011; 89: 806–815.
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
608
t. c. cheetham et al.
12. Shin J, Hunt CM, Suzuki A, et al. Characterizing phenotypes and outcomes of drug-associated liver injury using electronic medical record data. Pharmacoepi Drug Safety. 2013; 22: 190–198. 13. Overhage JM, Ryan PB, Reich CG, et al. Validation of a common data model for active safety surveillance research. J Am Med Inform Assoc 2012; 19: 54–60. 14. Nathwani RA, Pais S, Reynolds TB, et al. Serum alanine aminotransferase in skeletal muscle diseases. Hepatology 2005; 41: 380–382. 15. Green RM, Flamm S. AGA technical review on the evaluation of liver chemistry tests. Gastroenterology 2002; 123: 1367–1384. 16. Watkins PB, Seligman PJ, Pears JS, Avigan MI, Senior JR. Using controlled clinical trials to learn more about acute drug-induced liver injury. Hepatology 2008; 48: 1680–1689. 17. Spanish DILI Registry. http://www.spanishdili.uma.es/index.php?option=com_ content&view=article&id=92&Itemid=83&lang=en [13 December 2012] 18. DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. The Am Statistician. 1999; 53: 177–201. 19. Koebnick C, Langer-Gould R, Gould M, et al. Sociodemographic characteristics of members of a large, integrated health care system: comparison with US census bureau data. Perm J. 2012; 16: 37–41. 20. Chalasani N, Fontana RJ, Bonkovsky HL, et al. Causes, clinical features, and outcomes from a prospective study of drug-induced liver injury in the United States. Gastroenterology 2008; 135: 1924–1934, 1934 e1-4.
Copyright © 2013 John Wiley & Sons, Ltd.
21. FDA. New Molecular Entity Postmarketing Safety Evaluation Pilot Program Final Report. Volume 2011. http://www.fda.gov/Drugs/DrugSafety/ PostmarketDrugSafetyInformationforPatientsandProviders/ucm185252. htm#Results [December 2012] 22. Garcia-Cortez M, Stephens C, Lucena MI, et al. Causality assessment methods in drug-induced liver injury: strengths and weaknesses. J Hepatology 2011; 55: 683–691. http://jamia.bmj.com/content/early/2013/07/08/amiajnl-2013-001930. abstract. [10 September 2013] 23. Overby CL, Pathak J, Gottesman O, et al. A collaborative approach to developing an electronic health record phenotyping algorithm for drug-induced liver injury. 24. Platt R, Carnahan RM, Brown JS, et al. The U.S. Food and Drug Administration’s Mini-Sentinel program: status and direction. Pharmacoepi Drug Safety 2012; 21(S1): 1–8.
SUPPORTING INFORMATION Additional supporting information may be found in the online version of this article at the publisher’s web site.
Pharmacoepidemiology and Drug Safety, 2014; 23: 601–608 DOI: 10.1002/pds
