541291
research-article2014
AJMXXX10.1177/1062860614541291American Journal of Medical QualityBrandt et al
Article American Journal of Medical Quality 1–7 © 2014 by the American College of Medical Quality Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614541291 ajmq.sagepub.com
Identifying Severe Sepsis via Electronic Surveillance
Bristol N. Brandt, BS1, Amanda B. Gartner, MSN2, Michael Moncure, MD1, Chad M. Cannon, MD1, Elizabeth Carlton, MSN2, Carol Cleek, MSN2, Chris Wittkopp, RHIT2, and Steven Q. Simpson, MD1
Abstract An electronic sepsis surveillance system (ESSV) was developed to identify severe sepsis and determine its time of onset. ESSV sensitivity and specificity were evaluated during an 11-day prospective pilot and a 30-day retrospective trial. ESSV diagnostic alerts were compared with care team diagnoses and with administrative records, using expert adjudication as the standard for comparison. ESSV was 100% sensitive for detecting severe sepsis but only 62.0% specific. During the pilot, the software identified 477 patients, compared with 18 by adjudication. In the 30-day trial, adjudication identified 164 severe sepsis patients, whereas ESSV detected 996. ESSV was more sensitive but less specific than care team or administrative data. ESSV-identified time of severe sepsis onset was a median of 0.00 hours later than adjudication (interquartile range = 0.05). The system can be a useful tool when implemented appropriately but lacks specificity, largely because of its reliance on discreet data fields. Keywords sepsis, quality improvement, electronic surveillance, electronic medical record Severe sepsis is a major public health problem in the United States. It is the 11th leading cause of death and costs the health care system almost $17 billion annually.1-5 The condition is characterized by infection, with associated system-wide inflammation and involves at least one organ dysfunction.2,3,6,7 Various studies demonstrate that the incidence of severe sepsis is increasing by approximately 8.5% annually, with the highest incidence among patients older than 65 years of age.1-3,8 Because so many people of the baby boom generation are currently reaching or exceeding this age mark, the incidence likely will climb at an accelerating rate in the next decade.1,4,9 Most current estimates of the incidence and cost of severe sepsis are based on administrative data (discharge diagnosis codes).2,3,10,11 Both incidence and cost are likely higher than the reported numbers because providers do not always recognize or document the condition appropriately.8,12,13 Alternative tools to aid providers in diagnosing severe sepsis, such as manual screening forms or automated screening performed through the electronic medical record (EMR), could help identify and properly treat these patients. Mortality estimates for severe sepsis range from 28.6% to 51.0%; however, timely intervention with antibiotics and early goal-directed therapy increases survival.1,6,10-14 Therefore, it is imperative that providers
identify patients with severe sepsis early in the clinical course. Because early detection and treatment of patients with this condition decreases mortality, the economic burden also could be lessened by early diagnosis.15,16 The increasing incidence of severe sepsis, along with the fact that hospital care teams frequently fail to diagnose it, has prompted the development of automated surveillance mechanisms.12,13 These software-based tools use data from the EMR, including vital signs and common laboratory results, to identify potentially septic patients so that timely care can be provided according to existing guidelines. The goal of this study was to determine if an electronic sepsis surveillance system (ESSV) could identify severe sepsis and identify its time of onset in patients at the study hospital. The findings of the ESSV were compared with diagnoses by hospital care teams and with post hoc adjudication of cases to determine its 1
University of Kansas School of Medicine, Kansas City, KS The University of Kansas Hospital, Kansas City, KS
2
Corresponding Author: Steven Q. Simpson, MD, Division of Pulmonary and Critical Care Medicine, University of Kansas, 3901 Rainbow Blvd, Mail Stop #3007, Kansas City, KS 66160. Email: [email protected]
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
2
American Journal of Medical Quality
sensitivity and specificity and to assess its potential for use in the clinical setting.
Methods Development The development of the ESSV began in August of 2009. The algorithms were developed by an iterative process involving a design team that comprised an intensivist, an emergency medicine physician, a trauma surgeon, critical care nursing personnel, software engineers, and personnel from the hospital’s organizational improvement and quality departments. Diagnostic algorithms were based on the standard criteria, as proposed by Bone et al.6 Multiple development sessions were held until the algorithm was deemed satisfactory. The software was designed to apply only to patients 14 years of age or older. The program was not designed to directly contact clinical care teams with alerts. Instead, it was designed to provide real-time alerts to clinical sepsis surveillance personnel who would only notify the care team if they confirmed a high probability of severe sepsis via chart review or other mechanisms. Clinical personnel on the design team informed the software engineers of the most clinically relevant location in the EMR (Epic, Verona, WI) from which to extract specific data points for the ESSV, thereby allowing for subsequent cross-checking between care teams, surveillance personnel, and the ESSV. The final algorithm is shown in Figure 1. The clinical personnel and software engineers worked together to overcome various barriers. For example, the implementation process demonstrated that serum creatinine levels were very often increased for reasons other than severe sepsis, including chronic renal disease. The algorithm was modified to accommodate that practical finding by eliminating those patients with a diagnosis of end-stage renal disease or the presence of erythropoietin therapy.
Patient Groups A real-time prospective pilot of the ESSV ran for 11 days in 2012 (January 31, and February 1-3, 6-8, 13, and 15-17) at The University of Kansas Hospital, a 606-bed tertiary academic medical center. Software ran continuously during this period, but clinical surveillance personnel examined the results for approximately 10 hours per day on each of the dates listed. Surveillance personnel evaluated each patient identified by the ESSV during the pilot to determine if severe sepsis was present, according to the standard criteria.6 The surveillance personnel consisted of a critical care nurse and an intensivist who reviewed the patient’s EMR and, when necessary, contacted the patient’s bedside nurse or physician to discuss sepsis signs or care.
Surveillance personnel classified patients as not having severe sepsis if, on investigation, one of the following was true: (1) the patient had no signs of infection or systemic inflammatory response syndrome (SIRS), (2) the organ dysfunction was nonacute, (3) the patient was already in the intensive care unit (ICU) or receiving palliative care on detection, (4) it was beyond the window for early goaldirected therapy at the time of review by surveillance personnel, or (5) the patient was flagged for later reassessment. The surveillance personnel used the same criteria as the ESSV to determine the presence of severe sepsis on review but with minor additions. PaCO2 < 32 mm Hg was considered equivalent to a respiratory rate >20 per minute for the identification of tachypnea/hyperventilation. Aspartate aminotransferase or alanine aminotransferase >90 U/L were considered evidence of hepatic dysfunction. D-dimer greater than the upper limit of normal in the study laboratory (230 ng/mL) also was considered evidence of abnormal thrombostasis. The hospital’s discharge database was used to identify all patients discharged from the hospital in February of 2012 and to identify the subgroup of patients discharged during this period with a diagnosis code for sepsis, severe sepsis, or septic shock (International Classification of Diseases, Ninth Revision [ICD-9] codes of 995.91, 995.92, and 785.52, respectively). For patients with these ICD-9 codes (ie, for patients diagnosed with sepsis during their hospitalization), the ESSV software was used to evaluate EMR data and to retrospectively identify the presence of severe sepsis and determine its time of onset. Additionally, the ESSV was applied retrospectively to the EMR data of all other patients discharged from the hospital during the month of February in order to detect severe sepsis in patients who were not diagnosed during their hospitalization.
Expert Adjudication Expert adjudication of patient medical records was used as the standard of comparison. Patient charts from each group (pilot, retrospective with diagnosis code, and retrospective without diagnosis code but discharged during the study period) were adjudicated retrospectively using a protocol detailing the specific order of search, the location of specific data points, and the method for extraction and interpretation of the data from individual records. Two investigators performed the adjudications (BNB and SQS) using the standard definitions to determine the presence of severe sepsis or septic shock.6 The adjudication protocol criteria were agreed on before accessing any patient records. Two examples of adjudication rules include evaluating a patient’s record prior to comparison with the ESSV result and requiring that individual SIRS criteria must be present within 2 hours of one another for
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
3
Brandt et al
Is pt Age > 14?
NO
Stop
YES
NO
Stop
Sepsis Prequalifica on (PQ) met? YES
Check for SIRS Criteria Two or more of the following must be present (VS SIRS signs in a 2 hr period; WBC/Bandemia within 12 hrs) • Oral or Core Temp ≥ 38.0°C • Oral or Core Temp ≤ 36.0°C • Respiratory Rate > 20 • Heart rate > 90 BPM • WBC ≥ 12,000μL or Bandemia > 10% • WBC ≤ 4,000μL
Dx of Infec on?
NO
Dx of OD?
YES
YES
PQ Met
PQ Met Check for one of the following:
• Acutely altered mental status – Lethargic, confused, agitated, or anxious • Systolic Blood Pressure (SBP) < 90, Mean Arterial Pressure (MAP) < 65mmHg, or SBP decrease . 40mmHg from baseline • PaO2/FiO2 ra on< 300 or SpO2 < 90% • Crea nine > 2.0 mg/dl or Crea nine increase > 0.5 mg/dl from lowest value during current encounter • Urine output < 0.5 mL/kg/hour for > 2 hours • Plasma Total bilirubin > 2mg/dl • Platelet count < 100,000 • Lactate > 2.0 mMol/L • INR > 1.5 or PTT > 60 seconds YES
Criteria Met? YES
NO
PQ Met
YES
Is the only sign of organ dysfunc on elevated crea nine? YES
Trigger – Pa ent Lands on “Poten al Sepsis” List Stop
YES
Dx of ESRD? NO
Stop
YES erythropoie n
Therapy? NO
Figure 1. The algorithm details the order of search and information obtained by the software to determine if an alert should be generated. Abbreviations: Dx, diagnosis; ESRD, end-stage renal disease; INR, international normalized ratio; OD, organ dysfunction; PTT, partial thromboplastin time; SIRS, systemic inflammatory response syndrome; WBC, white blood count.
the syndrome to be considered present. The protocol was used to keep adjudication consistent from patient to patient. If the medical record was ambiguous about the presence of infection, both adjudicators reviewed the chart until a diagnosis or its absence was agreed on. Both adjudicators also reviewed charts in which the diagnosis agreed with the ESSV but where one adjudicator found discrepancy with the ESSV with regarding time of onset of infection, severe sepsis, or septic shock. All patients who were identified prospectively during the ESSV pilot and who had severe sepsis on review by
the surveillance personnel were adjudicated. All patients discharged in February 2012 with a diagnosis code for sepsis, severe sepsis, or septic shock also were adjudicated in order to determine if severe sepsis was indeed present, the time of its onset, and if adjudication corroborated the care team’s characterization of severe sepsis or septic shock. To further evaluate the validity of the ESSV, the research team adjudicated charts of patients who were identified by the ESSV as having an infection, SIRS, and 2 organ dysfunctions during the entire month of February 2012, but who were not diagnosed with severe sepsis by
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
4
American Journal of Medical Quality Table 2. 2 × 2 Contingency Tables.
Table 1. Patients Excluded From Pilot Evaluation. Characteristics No signs of infection No SIRS Organ dysfunction nonacute On palliative care Already in ICU Beyond window for EGDT Flagged for reassessment
n
Total (%)
297 15 177 4 66 19 58
45.3 2.3 27.0 0.6 10.1 2.9 8.9
Abbreviations: EGDT, early goal-directed therapy; ICU, intensive care unit; SIRS, systemic inflammatory response syndrome.
their care teams and were not coded for severe sepsis on discharge. A random number generator was used to generate a statistical sample (20%) of the latter patients for adjudication. Only patients with 2 or more organ dysfunctions were considered for review in order to eliminate patients whose only organ dysfunction resulted from the presence of infection in that organ. As a quality improvement project, this work was exempted from formal approval by the institution’s institutional review board.
Results Electronic Surveillance Pilot Study During the pilot study, the ESSV identified 655 potential sepsis episodes involving 477 patients. The ESSV surveillance personnel excluded 458 patients on initial evaluation at the time of the pilot. Table 1 shows the reasons for exclusion by surveillance personnel. The review process required 1.9 ± 1.6 minutes per episode (mean ± standard deviation). The surveillance personnel notified the care teams about 19 patients whom the surveillance personnel believed had severe sepsis during the pilot period. On expert adjudication, 18 of these patients were confirmed to have severe sepsis during their hospitalization, such that adjudicators agreed with surveillance personnel in 94.7% of cases. Of the 19 patients, 15 had a diagnosis code for severe sepsis, indicating that the hospital care team diagnosed severe sepsis during the hospitalization; for 4 patients, the care team did not adjust diagnosis according to the recommendation of the surveillance personnel. Of the 4 patients who were undiagnosed by the care teams during their hospitalization, 3 had severe sepsis on adjudication. Of the 19 patients who were diagnosed by surveillance personnel, 16 were discharged in February and therefore had ESSV onset times that could be compared with adjudication times. Of the 16 adjudicated severe sepsis onset times, 9 agreed exactly with the ESSV. The average
Severe Sepsis, + Administrative data vs adjudication Administrative data, + 92 Administrative data, − 72 Total 164 ESSV vs adjudication ESSV, + 164 ESSV, − 0 Total 164 Care team vs adjudication Care team, + 101 Care team, − 63 Total 164
Severe Sepsis, −
Total
12 2176 2188
104 2248 2352
832 1356 2188
996 1356 2352
8 2180 2188
109 2243 2352
Abbreviation: ESSV, electronic sepsis surveillance system.
Table 3. Sensitivities and Specificities.
Specificity
Geometric Mean of Sensitivity and Specificity
61.6% 51.6%
99.6% 99.5%
0.783 0.717
100.0%
62.0%
0.787
Sensitivity Care team Administrative data ESSV
Abbreviation: ESSV, electronic sepsis surveillance system.
difference between the onset time according to the ESSV and that found by retrospective adjudication was −2.16 hours (ie, the ESSV determined severe sepsis to be present 2.16 hours before adjudicators believed it to be). The median time difference was 0.0 hours, with an interquartile range of 0.24 hours.
Severe Sepsis From Administrative Data The records of all patients discharged in the month of February 2012 with diagnosis codes for sepsis (64 patients), severe sepsis (104 patients), or septic shock (30 patients) were adjudicated. Severe sepsis was present on adjudication in 124 patients, 42 of them having septic shock. Of the 104 patients with a diagnosis code for severe sepsis, 92 were adjudicated to have severe sepsis (Table 2). The administrative data had a positive predictive value of 88.5% and a negative predictive value of 96.8%. Table 3 shows the sensitivity and specificity of the administrative data. Among the patients with severe sepsis, adjudication and the ESSV agreed on the exact time of severe sepsis onset for 58 of 124 onset times (46.8%). Table 4 shows a
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
5
Brandt et al Table 4. ESSV Versus Adjudicated Time of Severe Sepsis Onset.
Mean difference Median Interquartile range
Original
Excluding False Positives With No Infection
−25.71 hours 0.0 hours 3.30 hours
0.45 hours 0.0 hours 0.05 hours
Abbreviation: ESSV, electronic sepsis surveillance system.
comparison of the time of onset of severe sepsis as determined by the ESSV and by adjudication. Analysis of the large mean difference showed that a subset of the false positives identified by the ESSV had evidence of SIRS but no evidence of infection at the time of the ESSV alert. In all, 20 such patients were identified, with a mean onset difference among these 20 of −161.7 hours, substantially skewing the time of onset statistic. Table 4 shows the results after removal of this group of false positives.
Total Number of Severe Sepsis Patients When the ESSV was applied retrospectively to the electronic records of all patients discharged during the study period, it identified 996 patients (Table 2). All 124 patients from the administrative data group also were identified in this group. Of the remaining 872 patients, 110 had SIRS, an infection diagnosis, and 2 or more organ dysfunctions, suggesting the presence of severe sepsis. Adjudication of 22 of these 110 patient records (20%) revealed that 8 patients (36.4%) had severe sepsis. Extrapolation to the entire cohort suggests that 40 of the 110 patients identified by the ESSV had severe sepsis. When these 40 patients are added to the 124 from the administrative data, the estimated total number of patients with severe sepsis during the month of study was 164 (Table 2). The ESSV, therefore, had a positive predictive value of 16.5% and a negative predictive value of 100%. Table 3 shows the sensitivity and specificity of the ESSV for the diagnosis of severe sepsis.
Severe Sepsis Identified by Care Team Of the 164 patients adjudicated to have severe sepsis during the month of February, 101 (61.6%) were correctly diagnosed by their care teams during hospitalization, as evidenced by at least one progress note. The care teams also falsely diagnosed 8 patients who did not have organ dysfunction (Table 2). The administrative data demonstrated that 6 of the 101 correctly diagnosed patients (5.9%) were not coded for severe sepsis, in spite of the diagnosis being recorded in the EMR during
hospitalization. The care team had a positive predictive value of 92.7% and a negative predictive value of 97.2%. Table 3 shows the sensitivity and specificity of the care teams for diagnosing severe sepsis.
Sensitivity and Specificity Using the combined total of 164 patients, 124 from the adjudication and 40 from extrapolation, as the standard, sensitivity, specificity, and the geometric mean of the 2 were determined for the care team, the administrative data, and the ESSV software. The results of these calculations are shown in Table 3.
Discussion The advent of EMRs and of software with the capability of evaluating massive amounts of data in real time allows for the possibility of highly sensitive electronic decision support. The study results show that the ESSV is able to identify patients with severe sepsis prospectively and retrospectively using both clinical data in the EMR and administrative data. Because of its use of data in the EMR, the software is able to identify patients that care teams, and therefore discharge coders, do not identify during or after hospitalization. Although the care team and the administrative data appear to be highly specific for diagnosing and detecting severe sepsis, it is likely that this finding is affected by the relatively low number of patients with severe sepsis compared with the large number of non–sepsis-related hospital discharges. The predictive value of a positive diagnosis by the care team was 92.7%. Both the care team and the administrative data lack sensitivity. The administrative data showed an even lower level of sensitivity compared with the care team, given that multiple patients were diagnosed with severe sepsis by their care teams but were not coded for severe sepsis on discharge. A more sensitive detection method, such as the ESSV, could be useful for identifying these currently undiagnosed patients. The ESSV proved to be 100% sensitive in identifying patients with severe sepsis but at the expense of low specificity. The predictive value of a positive ESSV screen was only 16.5%. Clearly, the ESSV in its current iteration adds little diagnostic support. In fact, the Bayesian probability of severe sepsis, given a positive ESSV alert, is only 7% based on the study data. There were 2 principal reasons for the lack of specificity. The ESSV was rarely able to determine the presence or absence of infection in patients with SIRS and organ dysfunction. The ESSV, as currently written, also struggled to separate acute from chronic organ dysfunction. Both of these weaknesses in the software stem from its inability to extract data from
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
6
American Journal of Medical Quality
progress notes or other reports (ie, free text) and from its complete reliance on discrete data fields. If care teams name an infection or describe acute organ dysfunction in their notes but do not make an entry in the EMR problem list, which contains discrete ICD-9 data fields, the ESSV is unable to use the information. If either the ESSV or the EMR was capable of natural language processing to detect the presence of infection, then ESSV identification of severe sepsis would be more specific and its determination of severe sepsis onset time would be more accurate. Additionally, natural language processing would allow the detection of findings such as chest infiltrates in radiology reports. The inability to use free text appears to be a common feature of the current generation of sepsis identification software.12-14 The study hospital’s EMR has the capability of identifying SIRS and organ dysfunction in much the same way as the ESSV but likewise lacks the ability to convert free text to usable data. One might posit that the addition of microbiology results to the ESSV algorithm would improve its specificity. However, severe sepsis is a time-sensitive diagnosis, and it is important that the diagnosis is made quickly. The criteria for diagnosing severe sepsis specifically state “known or suspected infection.” The purpose of that bit of ambiguity is that microbiology results typically are not available for at least 24 hours after infection or sepsis is considered a diagnostic option. Therefore, save the occasional rapid Streptococcus test, the addition of microbiology results cannot help the algorithm to be more timely and would be of little clinical use. The research team’s previous analyses of best practice alerts in the EMR demonstrated that they were commonly ignored (S. Q. Simpson et al, unpublished data, March 2007-December 2011), which pushed the team to develop a different surveillance system. However, the ESSV’s low level of specificity suggests a mechanism whereby bedside nurses could become fatigued by such alerts. Given the current status of software comparable to the ESSV and its reliance on discrete data fields, a buffer between the software’s output and the bedside clinical care team appears to be necessary. In the study hospital, the research team chose to implement surveillance personnel tasked with evaluating the ESSV’s alerts to remove false positive alerts before notifying the care teams. These surveillance personnel were used during the pilot study to prevent alert fatigue by the bedside clinicians. Adjudication agreed with the surveillance personnel 95% of the time compared with 61.6% agreement with the care team. This illustrates the usefulness and importance of the surveillance personnel and shows that the ESSV, when used with these intermediate personnel between the software and the bedside clinicians, can be a valuable tool for identifying patients with severe sepsis who otherwise would go unidentified and undiagnosed. However, the
care team appears to sometimes ignore surveillance personnel recommendations, as seen in 4 instances during the pilot when providers did not diagnosis their patients with severe sepsis even after notification by the surveillance personnel. Because this study investigates only one setting and one electronic system, it is difficult to determine how generalizable the results may be. However, this ESSV is similar to others that have been reported and to sepsis tools that are developed and marketed by EMR vendors, in that it relies on data from discrete data fields in the EMR.17 Previous studies using surveillance software comparable to the ESSV demonstrated results similar to those of the present study. Sawyer et al13 piloted a similar sepsis alert system for non-ICU patients.1 A real-time alert was sent directly to a ward’s charge nurse, who further assessed the patient and notified the treating or on-call physician of the alert. The alerts increased interventions but did not significantly influence hospital length of stay or mortality. The tool had a low positive predictive value of 19.5% along with a negative predictive value of 95.8%.5,18 These findings are similar to those of the present study with the ESSV, which had a positive predictive value of 16.5% and a negative predictive value of 100%. Hooper et al12 also implemented an electronic monitoring system for detecting sepsis but included only medical ICU patients with modified SIRS (one of the SIRS criteria must be an elevated temperature or white blood cell count).6,7 The alert resulted in a text page to the primary team physician contact. Unlike in the previous study, clinical practices were not altered by the alert system. This may be the result of the clinical personnel’s familiarity with sepsis and the continuous patient monitoring found in an ICU. The alert system appeared to have a higher sensitivity, specificity, positive predictive value, and negative predictive value (99%, 82%, 41%, and 97%, respectively) than seen with some other alert systems, but this may have been a direct reflection of the high prevalence of sepsis in the ICU. Before deploying resources throughout the hospital, the research team wished to determine the accuracy of the study institution’s surveillance system. The purpose of this study was to determine the sensitivity and specificity of the software program, with the goal of developing an appropriate model for delivering the ESSV alerts. The research team suspected that the ESSV would be overly sensitive and wanted to better understand how to deal with this weakness. Currently, the ESSV technology is not ready to send alerts directly to bedside clinicians because of the high number of false positives. These findings led the team to implement the software by alerting surveillance personnel rather than bedside care providers, so that false positives could be winnowed and the alerts received at the bedside would have clinical significance.
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
7
Brandt et al A next generation of sepsis detection software should have natural language processing capability for scanning progress notes and other reports. Such a capability would enable the software to detect potentially septic patients sooner and with greater specificity than the currently reported generation of software. A next-generation software with natural language processing also would likely be able to better differentiate between acute and chronic conditions and to ignore organ dysfunctions when an infection is present in that organ. Such a system would have the potential to sufficiently eliminate false positives, such that bedside teams could be routinely notified without engendering alert fatigue. This would allow for surveillance personnel to be deployed in implementing sepsis guidelines rather than fine-tuning the ESSV diagnosis. In summary, this study tested the ability of a software system that extracts data from the EMR to identify patients with severe sepsis throughout the hospital. It was found that the system, which cannot specifically identify the presence of infection or of chronic organ dysfunction, is highly sensitive but sufficiently lacking in specificity to be useful as a tool for warning bedside personnel, although the interjection of surveillance personnel substantially increases the specificity of reports to bedside providers. The research team suggests that the next generation of software should make use of some form of natural language processing to identify these confounding factors and increase the specificity of the computerized system. Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The authors received no financial support for the research, authorship, and/or publication of this article.
References 1. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303-1310. 2. Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med. 2007;35:1244-1250.
3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348:1546-1554. 4. Cohn D, Paul Taylor. Baby boomers approach age 65 glumly. http://www.pewsocialtrends.org/2010/12/20/babyboomers-approach-65-glumly/. Accessed June 4, 2014. 5. Murphy SL, Xu J, Kochanek KD. Deaths: final data for 2010. http://www.cdc.gov/nchs/data/nvsr/nvsr61/ nvsr61_04.pdf. Accessed June 11, 2013. 6. Bone R, Balk R, Cerra F, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101:1644-1655. 7. Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2003;31:1250-1256. 8. Poeze M, Ramsay G, Gerlach H, Rubulotta F, Levy M. An international sepsis survey: a study of doctors’ knowledge and perception about sepsis. Crit Care. 2004;8:R409-R413. 9. Sasse KC, Nauenberg E, Long A, Anton B, Tucker HJ, Hu T-W. Long-term survival after intensive care unit admission with sepsis. Crit Care Med. 1995;23:1040-1047. 10. Kumar A, Roberts D, Wood KE, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med. 2006;34:1589-1596. 11. Rivers E, Nguyen B, Havstad S, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345:1368-1377. 12. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit. Crit Care Med. 2012;40:2096-2101. 13. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39:469-473. 14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57:500-504. 15. Jones AE, Troyer JL, Kline JA. Cost-effectiveness of an emergency department-based early sepsis resuscitation protocol. Crit Care Med. 2011;39:1306-1312. 16. Shorr AF, Micek ST, Jackson WL, Kollef MH. Economic implications of an evidence-based sepsis protocol: can we improve outcomes and lower costs? Crit Care Med. 2007;35:1257-1262. 17. Cerner Corporation. St. John Sepsis agent. https://store. cerner.com/items/1527. Accessed January 6, 2014. 18. Thiel SW, Asghar MF, Micek ST, Reichley RM, Doherty JA, Kollef MH. Hospital-wide impact of a standardized order set for the management of bacteremic severe sepsis. Crit Care Med. 2009;37:819-824.
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
research-article2014
AJMXXX10.1177/1062860614541291American Journal of Medical QualityBrandt et al
Article American Journal of Medical Quality 1–7 © 2014 by the American College of Medical Quality Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614541291 ajmq.sagepub.com
Identifying Severe Sepsis via Electronic Surveillance
Bristol N. Brandt, BS1, Amanda B. Gartner, MSN2, Michael Moncure, MD1, Chad M. Cannon, MD1, Elizabeth Carlton, MSN2, Carol Cleek, MSN2, Chris Wittkopp, RHIT2, and Steven Q. Simpson, MD1
Abstract An electronic sepsis surveillance system (ESSV) was developed to identify severe sepsis and determine its time of onset. ESSV sensitivity and specificity were evaluated during an 11-day prospective pilot and a 30-day retrospective trial. ESSV diagnostic alerts were compared with care team diagnoses and with administrative records, using expert adjudication as the standard for comparison. ESSV was 100% sensitive for detecting severe sepsis but only 62.0% specific. During the pilot, the software identified 477 patients, compared with 18 by adjudication. In the 30-day trial, adjudication identified 164 severe sepsis patients, whereas ESSV detected 996. ESSV was more sensitive but less specific than care team or administrative data. ESSV-identified time of severe sepsis onset was a median of 0.00 hours later than adjudication (interquartile range = 0.05). The system can be a useful tool when implemented appropriately but lacks specificity, largely because of its reliance on discreet data fields. Keywords sepsis, quality improvement, electronic surveillance, electronic medical record Severe sepsis is a major public health problem in the United States. It is the 11th leading cause of death and costs the health care system almost $17 billion annually.1-5 The condition is characterized by infection, with associated system-wide inflammation and involves at least one organ dysfunction.2,3,6,7 Various studies demonstrate that the incidence of severe sepsis is increasing by approximately 8.5% annually, with the highest incidence among patients older than 65 years of age.1-3,8 Because so many people of the baby boom generation are currently reaching or exceeding this age mark, the incidence likely will climb at an accelerating rate in the next decade.1,4,9 Most current estimates of the incidence and cost of severe sepsis are based on administrative data (discharge diagnosis codes).2,3,10,11 Both incidence and cost are likely higher than the reported numbers because providers do not always recognize or document the condition appropriately.8,12,13 Alternative tools to aid providers in diagnosing severe sepsis, such as manual screening forms or automated screening performed through the electronic medical record (EMR), could help identify and properly treat these patients. Mortality estimates for severe sepsis range from 28.6% to 51.0%; however, timely intervention with antibiotics and early goal-directed therapy increases survival.1,6,10-14 Therefore, it is imperative that providers
identify patients with severe sepsis early in the clinical course. Because early detection and treatment of patients with this condition decreases mortality, the economic burden also could be lessened by early diagnosis.15,16 The increasing incidence of severe sepsis, along with the fact that hospital care teams frequently fail to diagnose it, has prompted the development of automated surveillance mechanisms.12,13 These software-based tools use data from the EMR, including vital signs and common laboratory results, to identify potentially septic patients so that timely care can be provided according to existing guidelines. The goal of this study was to determine if an electronic sepsis surveillance system (ESSV) could identify severe sepsis and identify its time of onset in patients at the study hospital. The findings of the ESSV were compared with diagnoses by hospital care teams and with post hoc adjudication of cases to determine its 1
University of Kansas School of Medicine, Kansas City, KS The University of Kansas Hospital, Kansas City, KS
2
Corresponding Author: Steven Q. Simpson, MD, Division of Pulmonary and Critical Care Medicine, University of Kansas, 3901 Rainbow Blvd, Mail Stop #3007, Kansas City, KS 66160. Email: [email protected]
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
2
American Journal of Medical Quality
sensitivity and specificity and to assess its potential for use in the clinical setting.
Methods Development The development of the ESSV began in August of 2009. The algorithms were developed by an iterative process involving a design team that comprised an intensivist, an emergency medicine physician, a trauma surgeon, critical care nursing personnel, software engineers, and personnel from the hospital’s organizational improvement and quality departments. Diagnostic algorithms were based on the standard criteria, as proposed by Bone et al.6 Multiple development sessions were held until the algorithm was deemed satisfactory. The software was designed to apply only to patients 14 years of age or older. The program was not designed to directly contact clinical care teams with alerts. Instead, it was designed to provide real-time alerts to clinical sepsis surveillance personnel who would only notify the care team if they confirmed a high probability of severe sepsis via chart review or other mechanisms. Clinical personnel on the design team informed the software engineers of the most clinically relevant location in the EMR (Epic, Verona, WI) from which to extract specific data points for the ESSV, thereby allowing for subsequent cross-checking between care teams, surveillance personnel, and the ESSV. The final algorithm is shown in Figure 1. The clinical personnel and software engineers worked together to overcome various barriers. For example, the implementation process demonstrated that serum creatinine levels were very often increased for reasons other than severe sepsis, including chronic renal disease. The algorithm was modified to accommodate that practical finding by eliminating those patients with a diagnosis of end-stage renal disease or the presence of erythropoietin therapy.
Patient Groups A real-time prospective pilot of the ESSV ran for 11 days in 2012 (January 31, and February 1-3, 6-8, 13, and 15-17) at The University of Kansas Hospital, a 606-bed tertiary academic medical center. Software ran continuously during this period, but clinical surveillance personnel examined the results for approximately 10 hours per day on each of the dates listed. Surveillance personnel evaluated each patient identified by the ESSV during the pilot to determine if severe sepsis was present, according to the standard criteria.6 The surveillance personnel consisted of a critical care nurse and an intensivist who reviewed the patient’s EMR and, when necessary, contacted the patient’s bedside nurse or physician to discuss sepsis signs or care.
Surveillance personnel classified patients as not having severe sepsis if, on investigation, one of the following was true: (1) the patient had no signs of infection or systemic inflammatory response syndrome (SIRS), (2) the organ dysfunction was nonacute, (3) the patient was already in the intensive care unit (ICU) or receiving palliative care on detection, (4) it was beyond the window for early goaldirected therapy at the time of review by surveillance personnel, or (5) the patient was flagged for later reassessment. The surveillance personnel used the same criteria as the ESSV to determine the presence of severe sepsis on review but with minor additions. PaCO2 < 32 mm Hg was considered equivalent to a respiratory rate >20 per minute for the identification of tachypnea/hyperventilation. Aspartate aminotransferase or alanine aminotransferase >90 U/L were considered evidence of hepatic dysfunction. D-dimer greater than the upper limit of normal in the study laboratory (230 ng/mL) also was considered evidence of abnormal thrombostasis. The hospital’s discharge database was used to identify all patients discharged from the hospital in February of 2012 and to identify the subgroup of patients discharged during this period with a diagnosis code for sepsis, severe sepsis, or septic shock (International Classification of Diseases, Ninth Revision [ICD-9] codes of 995.91, 995.92, and 785.52, respectively). For patients with these ICD-9 codes (ie, for patients diagnosed with sepsis during their hospitalization), the ESSV software was used to evaluate EMR data and to retrospectively identify the presence of severe sepsis and determine its time of onset. Additionally, the ESSV was applied retrospectively to the EMR data of all other patients discharged from the hospital during the month of February in order to detect severe sepsis in patients who were not diagnosed during their hospitalization.
Expert Adjudication Expert adjudication of patient medical records was used as the standard of comparison. Patient charts from each group (pilot, retrospective with diagnosis code, and retrospective without diagnosis code but discharged during the study period) were adjudicated retrospectively using a protocol detailing the specific order of search, the location of specific data points, and the method for extraction and interpretation of the data from individual records. Two investigators performed the adjudications (BNB and SQS) using the standard definitions to determine the presence of severe sepsis or septic shock.6 The adjudication protocol criteria were agreed on before accessing any patient records. Two examples of adjudication rules include evaluating a patient’s record prior to comparison with the ESSV result and requiring that individual SIRS criteria must be present within 2 hours of one another for
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
3
Brandt et al
Is pt Age > 14?
NO
Stop
YES
NO
Stop
Sepsis Prequalifica on (PQ) met? YES
Check for SIRS Criteria Two or more of the following must be present (VS SIRS signs in a 2 hr period; WBC/Bandemia within 12 hrs) • Oral or Core Temp ≥ 38.0°C • Oral or Core Temp ≤ 36.0°C • Respiratory Rate > 20 • Heart rate > 90 BPM • WBC ≥ 12,000μL or Bandemia > 10% • WBC ≤ 4,000μL
Dx of Infec on?
NO
Dx of OD?
YES
YES
PQ Met
PQ Met Check for one of the following:
• Acutely altered mental status – Lethargic, confused, agitated, or anxious • Systolic Blood Pressure (SBP) < 90, Mean Arterial Pressure (MAP) < 65mmHg, or SBP decrease . 40mmHg from baseline • PaO2/FiO2 ra on< 300 or SpO2 < 90% • Crea nine > 2.0 mg/dl or Crea nine increase > 0.5 mg/dl from lowest value during current encounter • Urine output < 0.5 mL/kg/hour for > 2 hours • Plasma Total bilirubin > 2mg/dl • Platelet count < 100,000 • Lactate > 2.0 mMol/L • INR > 1.5 or PTT > 60 seconds YES
Criteria Met? YES
NO
PQ Met
YES
Is the only sign of organ dysfunc on elevated crea nine? YES
Trigger – Pa ent Lands on “Poten al Sepsis” List Stop
YES
Dx of ESRD? NO
Stop
YES erythropoie n
Therapy? NO
Figure 1. The algorithm details the order of search and information obtained by the software to determine if an alert should be generated. Abbreviations: Dx, diagnosis; ESRD, end-stage renal disease; INR, international normalized ratio; OD, organ dysfunction; PTT, partial thromboplastin time; SIRS, systemic inflammatory response syndrome; WBC, white blood count.
the syndrome to be considered present. The protocol was used to keep adjudication consistent from patient to patient. If the medical record was ambiguous about the presence of infection, both adjudicators reviewed the chart until a diagnosis or its absence was agreed on. Both adjudicators also reviewed charts in which the diagnosis agreed with the ESSV but where one adjudicator found discrepancy with the ESSV with regarding time of onset of infection, severe sepsis, or septic shock. All patients who were identified prospectively during the ESSV pilot and who had severe sepsis on review by
the surveillance personnel were adjudicated. All patients discharged in February 2012 with a diagnosis code for sepsis, severe sepsis, or septic shock also were adjudicated in order to determine if severe sepsis was indeed present, the time of its onset, and if adjudication corroborated the care team’s characterization of severe sepsis or septic shock. To further evaluate the validity of the ESSV, the research team adjudicated charts of patients who were identified by the ESSV as having an infection, SIRS, and 2 organ dysfunctions during the entire month of February 2012, but who were not diagnosed with severe sepsis by
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
4
American Journal of Medical Quality Table 2. 2 × 2 Contingency Tables.
Table 1. Patients Excluded From Pilot Evaluation. Characteristics No signs of infection No SIRS Organ dysfunction nonacute On palliative care Already in ICU Beyond window for EGDT Flagged for reassessment
n
Total (%)
297 15 177 4 66 19 58
45.3 2.3 27.0 0.6 10.1 2.9 8.9
Abbreviations: EGDT, early goal-directed therapy; ICU, intensive care unit; SIRS, systemic inflammatory response syndrome.
their care teams and were not coded for severe sepsis on discharge. A random number generator was used to generate a statistical sample (20%) of the latter patients for adjudication. Only patients with 2 or more organ dysfunctions were considered for review in order to eliminate patients whose only organ dysfunction resulted from the presence of infection in that organ. As a quality improvement project, this work was exempted from formal approval by the institution’s institutional review board.
Results Electronic Surveillance Pilot Study During the pilot study, the ESSV identified 655 potential sepsis episodes involving 477 patients. The ESSV surveillance personnel excluded 458 patients on initial evaluation at the time of the pilot. Table 1 shows the reasons for exclusion by surveillance personnel. The review process required 1.9 ± 1.6 minutes per episode (mean ± standard deviation). The surveillance personnel notified the care teams about 19 patients whom the surveillance personnel believed had severe sepsis during the pilot period. On expert adjudication, 18 of these patients were confirmed to have severe sepsis during their hospitalization, such that adjudicators agreed with surveillance personnel in 94.7% of cases. Of the 19 patients, 15 had a diagnosis code for severe sepsis, indicating that the hospital care team diagnosed severe sepsis during the hospitalization; for 4 patients, the care team did not adjust diagnosis according to the recommendation of the surveillance personnel. Of the 4 patients who were undiagnosed by the care teams during their hospitalization, 3 had severe sepsis on adjudication. Of the 19 patients who were diagnosed by surveillance personnel, 16 were discharged in February and therefore had ESSV onset times that could be compared with adjudication times. Of the 16 adjudicated severe sepsis onset times, 9 agreed exactly with the ESSV. The average
Severe Sepsis, + Administrative data vs adjudication Administrative data, + 92 Administrative data, − 72 Total 164 ESSV vs adjudication ESSV, + 164 ESSV, − 0 Total 164 Care team vs adjudication Care team, + 101 Care team, − 63 Total 164
Severe Sepsis, −
Total
12 2176 2188
104 2248 2352
832 1356 2188
996 1356 2352
8 2180 2188
109 2243 2352
Abbreviation: ESSV, electronic sepsis surveillance system.
Table 3. Sensitivities and Specificities.
Specificity
Geometric Mean of Sensitivity and Specificity
61.6% 51.6%
99.6% 99.5%
0.783 0.717
100.0%
62.0%
0.787
Sensitivity Care team Administrative data ESSV
Abbreviation: ESSV, electronic sepsis surveillance system.
difference between the onset time according to the ESSV and that found by retrospective adjudication was −2.16 hours (ie, the ESSV determined severe sepsis to be present 2.16 hours before adjudicators believed it to be). The median time difference was 0.0 hours, with an interquartile range of 0.24 hours.
Severe Sepsis From Administrative Data The records of all patients discharged in the month of February 2012 with diagnosis codes for sepsis (64 patients), severe sepsis (104 patients), or septic shock (30 patients) were adjudicated. Severe sepsis was present on adjudication in 124 patients, 42 of them having septic shock. Of the 104 patients with a diagnosis code for severe sepsis, 92 were adjudicated to have severe sepsis (Table 2). The administrative data had a positive predictive value of 88.5% and a negative predictive value of 96.8%. Table 3 shows the sensitivity and specificity of the administrative data. Among the patients with severe sepsis, adjudication and the ESSV agreed on the exact time of severe sepsis onset for 58 of 124 onset times (46.8%). Table 4 shows a
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
5
Brandt et al Table 4. ESSV Versus Adjudicated Time of Severe Sepsis Onset.
Mean difference Median Interquartile range
Original
Excluding False Positives With No Infection
−25.71 hours 0.0 hours 3.30 hours
0.45 hours 0.0 hours 0.05 hours
Abbreviation: ESSV, electronic sepsis surveillance system.
comparison of the time of onset of severe sepsis as determined by the ESSV and by adjudication. Analysis of the large mean difference showed that a subset of the false positives identified by the ESSV had evidence of SIRS but no evidence of infection at the time of the ESSV alert. In all, 20 such patients were identified, with a mean onset difference among these 20 of −161.7 hours, substantially skewing the time of onset statistic. Table 4 shows the results after removal of this group of false positives.
Total Number of Severe Sepsis Patients When the ESSV was applied retrospectively to the electronic records of all patients discharged during the study period, it identified 996 patients (Table 2). All 124 patients from the administrative data group also were identified in this group. Of the remaining 872 patients, 110 had SIRS, an infection diagnosis, and 2 or more organ dysfunctions, suggesting the presence of severe sepsis. Adjudication of 22 of these 110 patient records (20%) revealed that 8 patients (36.4%) had severe sepsis. Extrapolation to the entire cohort suggests that 40 of the 110 patients identified by the ESSV had severe sepsis. When these 40 patients are added to the 124 from the administrative data, the estimated total number of patients with severe sepsis during the month of study was 164 (Table 2). The ESSV, therefore, had a positive predictive value of 16.5% and a negative predictive value of 100%. Table 3 shows the sensitivity and specificity of the ESSV for the diagnosis of severe sepsis.
Severe Sepsis Identified by Care Team Of the 164 patients adjudicated to have severe sepsis during the month of February, 101 (61.6%) were correctly diagnosed by their care teams during hospitalization, as evidenced by at least one progress note. The care teams also falsely diagnosed 8 patients who did not have organ dysfunction (Table 2). The administrative data demonstrated that 6 of the 101 correctly diagnosed patients (5.9%) were not coded for severe sepsis, in spite of the diagnosis being recorded in the EMR during
hospitalization. The care team had a positive predictive value of 92.7% and a negative predictive value of 97.2%. Table 3 shows the sensitivity and specificity of the care teams for diagnosing severe sepsis.
Sensitivity and Specificity Using the combined total of 164 patients, 124 from the adjudication and 40 from extrapolation, as the standard, sensitivity, specificity, and the geometric mean of the 2 were determined for the care team, the administrative data, and the ESSV software. The results of these calculations are shown in Table 3.
Discussion The advent of EMRs and of software with the capability of evaluating massive amounts of data in real time allows for the possibility of highly sensitive electronic decision support. The study results show that the ESSV is able to identify patients with severe sepsis prospectively and retrospectively using both clinical data in the EMR and administrative data. Because of its use of data in the EMR, the software is able to identify patients that care teams, and therefore discharge coders, do not identify during or after hospitalization. Although the care team and the administrative data appear to be highly specific for diagnosing and detecting severe sepsis, it is likely that this finding is affected by the relatively low number of patients with severe sepsis compared with the large number of non–sepsis-related hospital discharges. The predictive value of a positive diagnosis by the care team was 92.7%. Both the care team and the administrative data lack sensitivity. The administrative data showed an even lower level of sensitivity compared with the care team, given that multiple patients were diagnosed with severe sepsis by their care teams but were not coded for severe sepsis on discharge. A more sensitive detection method, such as the ESSV, could be useful for identifying these currently undiagnosed patients. The ESSV proved to be 100% sensitive in identifying patients with severe sepsis but at the expense of low specificity. The predictive value of a positive ESSV screen was only 16.5%. Clearly, the ESSV in its current iteration adds little diagnostic support. In fact, the Bayesian probability of severe sepsis, given a positive ESSV alert, is only 7% based on the study data. There were 2 principal reasons for the lack of specificity. The ESSV was rarely able to determine the presence or absence of infection in patients with SIRS and organ dysfunction. The ESSV, as currently written, also struggled to separate acute from chronic organ dysfunction. Both of these weaknesses in the software stem from its inability to extract data from
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
6
American Journal of Medical Quality
progress notes or other reports (ie, free text) and from its complete reliance on discrete data fields. If care teams name an infection or describe acute organ dysfunction in their notes but do not make an entry in the EMR problem list, which contains discrete ICD-9 data fields, the ESSV is unable to use the information. If either the ESSV or the EMR was capable of natural language processing to detect the presence of infection, then ESSV identification of severe sepsis would be more specific and its determination of severe sepsis onset time would be more accurate. Additionally, natural language processing would allow the detection of findings such as chest infiltrates in radiology reports. The inability to use free text appears to be a common feature of the current generation of sepsis identification software.12-14 The study hospital’s EMR has the capability of identifying SIRS and organ dysfunction in much the same way as the ESSV but likewise lacks the ability to convert free text to usable data. One might posit that the addition of microbiology results to the ESSV algorithm would improve its specificity. However, severe sepsis is a time-sensitive diagnosis, and it is important that the diagnosis is made quickly. The criteria for diagnosing severe sepsis specifically state “known or suspected infection.” The purpose of that bit of ambiguity is that microbiology results typically are not available for at least 24 hours after infection or sepsis is considered a diagnostic option. Therefore, save the occasional rapid Streptococcus test, the addition of microbiology results cannot help the algorithm to be more timely and would be of little clinical use. The research team’s previous analyses of best practice alerts in the EMR demonstrated that they were commonly ignored (S. Q. Simpson et al, unpublished data, March 2007-December 2011), which pushed the team to develop a different surveillance system. However, the ESSV’s low level of specificity suggests a mechanism whereby bedside nurses could become fatigued by such alerts. Given the current status of software comparable to the ESSV and its reliance on discrete data fields, a buffer between the software’s output and the bedside clinical care team appears to be necessary. In the study hospital, the research team chose to implement surveillance personnel tasked with evaluating the ESSV’s alerts to remove false positive alerts before notifying the care teams. These surveillance personnel were used during the pilot study to prevent alert fatigue by the bedside clinicians. Adjudication agreed with the surveillance personnel 95% of the time compared with 61.6% agreement with the care team. This illustrates the usefulness and importance of the surveillance personnel and shows that the ESSV, when used with these intermediate personnel between the software and the bedside clinicians, can be a valuable tool for identifying patients with severe sepsis who otherwise would go unidentified and undiagnosed. However, the
care team appears to sometimes ignore surveillance personnel recommendations, as seen in 4 instances during the pilot when providers did not diagnosis their patients with severe sepsis even after notification by the surveillance personnel. Because this study investigates only one setting and one electronic system, it is difficult to determine how generalizable the results may be. However, this ESSV is similar to others that have been reported and to sepsis tools that are developed and marketed by EMR vendors, in that it relies on data from discrete data fields in the EMR.17 Previous studies using surveillance software comparable to the ESSV demonstrated results similar to those of the present study. Sawyer et al13 piloted a similar sepsis alert system for non-ICU patients.1 A real-time alert was sent directly to a ward’s charge nurse, who further assessed the patient and notified the treating or on-call physician of the alert. The alerts increased interventions but did not significantly influence hospital length of stay or mortality. The tool had a low positive predictive value of 19.5% along with a negative predictive value of 95.8%.5,18 These findings are similar to those of the present study with the ESSV, which had a positive predictive value of 16.5% and a negative predictive value of 100%. Hooper et al12 also implemented an electronic monitoring system for detecting sepsis but included only medical ICU patients with modified SIRS (one of the SIRS criteria must be an elevated temperature or white blood cell count).6,7 The alert resulted in a text page to the primary team physician contact. Unlike in the previous study, clinical practices were not altered by the alert system. This may be the result of the clinical personnel’s familiarity with sepsis and the continuous patient monitoring found in an ICU. The alert system appeared to have a higher sensitivity, specificity, positive predictive value, and negative predictive value (99%, 82%, 41%, and 97%, respectively) than seen with some other alert systems, but this may have been a direct reflection of the high prevalence of sepsis in the ICU. Before deploying resources throughout the hospital, the research team wished to determine the accuracy of the study institution’s surveillance system. The purpose of this study was to determine the sensitivity and specificity of the software program, with the goal of developing an appropriate model for delivering the ESSV alerts. The research team suspected that the ESSV would be overly sensitive and wanted to better understand how to deal with this weakness. Currently, the ESSV technology is not ready to send alerts directly to bedside clinicians because of the high number of false positives. These findings led the team to implement the software by alerting surveillance personnel rather than bedside care providers, so that false positives could be winnowed and the alerts received at the bedside would have clinical significance.
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
7
Brandt et al A next generation of sepsis detection software should have natural language processing capability for scanning progress notes and other reports. Such a capability would enable the software to detect potentially septic patients sooner and with greater specificity than the currently reported generation of software. A next-generation software with natural language processing also would likely be able to better differentiate between acute and chronic conditions and to ignore organ dysfunctions when an infection is present in that organ. Such a system would have the potential to sufficiently eliminate false positives, such that bedside teams could be routinely notified without engendering alert fatigue. This would allow for surveillance personnel to be deployed in implementing sepsis guidelines rather than fine-tuning the ESSV diagnosis. In summary, this study tested the ability of a software system that extracts data from the EMR to identify patients with severe sepsis throughout the hospital. It was found that the system, which cannot specifically identify the presence of infection or of chronic organ dysfunction, is highly sensitive but sufficiently lacking in specificity to be useful as a tool for warning bedside personnel, although the interjection of surveillance personnel substantially increases the specificity of reports to bedside providers. The research team suggests that the next generation of software should make use of some form of natural language processing to identify these confounding factors and increase the specificity of the computerized system. Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The authors received no financial support for the research, authorship, and/or publication of this article.
References 1. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303-1310. 2. Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med. 2007;35:1244-1250.
3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348:1546-1554. 4. Cohn D, Paul Taylor. Baby boomers approach age 65 glumly. http://www.pewsocialtrends.org/2010/12/20/babyboomers-approach-65-glumly/. Accessed June 4, 2014. 5. Murphy SL, Xu J, Kochanek KD. Deaths: final data for 2010. http://www.cdc.gov/nchs/data/nvsr/nvsr61/ nvsr61_04.pdf. Accessed June 11, 2013. 6. Bone R, Balk R, Cerra F, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101:1644-1655. 7. Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2003;31:1250-1256. 8. Poeze M, Ramsay G, Gerlach H, Rubulotta F, Levy M. An international sepsis survey: a study of doctors’ knowledge and perception about sepsis. Crit Care. 2004;8:R409-R413. 9. Sasse KC, Nauenberg E, Long A, Anton B, Tucker HJ, Hu T-W. Long-term survival after intensive care unit admission with sepsis. Crit Care Med. 1995;23:1040-1047. 10. Kumar A, Roberts D, Wood KE, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med. 2006;34:1589-1596. 11. Rivers E, Nguyen B, Havstad S, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345:1368-1377. 12. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit. Crit Care Med. 2012;40:2096-2101. 13. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39:469-473. 14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57:500-504. 15. Jones AE, Troyer JL, Kline JA. Cost-effectiveness of an emergency department-based early sepsis resuscitation protocol. Crit Care Med. 2011;39:1306-1312. 16. Shorr AF, Micek ST, Jackson WL, Kollef MH. Economic implications of an evidence-based sepsis protocol: can we improve outcomes and lower costs? Crit Care Med. 2007;35:1257-1262. 17. Cerner Corporation. St. John Sepsis agent. https://store. cerner.com/items/1527. Accessed January 6, 2014. 18. Thiel SW, Asghar MF, Micek ST, Reichley RM, Doherty JA, Kollef MH. Hospital-wide impact of a standardized order set for the management of bacteremic severe sepsis. Crit Care Med. 2009;37:819-824.
Downloaded from ajm.sagepub.com at UNIV CALIFORNIA SAN DIEGO on August 21, 2015
