Cardiovascular Engineering and Technology, Vol. 6, No. 3, September 2015 ( 2015) pp. 383–391 DOI: 10.1007/s13239-015-0229-7
Performance Evaluation of New-Generation Pulse Oximeters in the NICU: Observational Study SHERMEEN NIZAMI,1,2 KIM GREENWOOD,2 NICK BARROWMAN,3 and JOANN HARROLD3,4 1 Systems and Computer Engineering, Carleton University, 1125 Colonel By Drive, Ottawa, ON K1S 5B6, Canada; 2Clinical Engineering, The Children’s Hospital of Eastern Ontario, Ottawa, Canada; 3Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada; and 4Division of Neonatology, The Children’s Hospital of Eastern Ontario, Ottawa, Canada
(Received 13 June 2014; accepted 29 May 2015; published online 9 June 2015) Associate Editor Ajit P. Yoganathan oversaw the review of this article.
INTRODUCTION
Abstract—This crossover observational study compares the data characteristics and performance of new-generation Nellcor OXIMAX and Masimo SET SmartPod pulse oximeter technologies. The study was conducted independent of either original equipment manufacturer (OEM) across eleven preterm infants in a Neonatal Intensive Care Unit (NICU). The SmartPods were integrated with Dra¨ger Infinity Delta monitors. The Delta monitor measured the heart rate (HR) using an independent electrocardiogram sensor, and the two SmartPods collected arterial oxygen saturation (SpO2) and pulse rate (PR). All patient data were non-Gaussian. Nellcor PR showed a higher correlation with the HR as compared to Masimo PR. The statistically significant difference found in their median values (1% for SpO2, 1 bpm for PR) was deemed clinically insignificant. SpO2 alarms generated by both SmartPods were observed and categorized for performance evaluation. Results for sensitivity, positive predictive value, accuracy and false alarm rates were Nellcor (80.3, 50, 44.5, 50%) and Masimo (72.2, 48.2, 40.6, 51.8%) respectively. These metrics were not statistically significantly different between the two pulse oximeters. Despite claims by OEMs, both pulse oximeters exhibited high false alarm rates, with no statistically or clinically significant difference in performance. These findings have a direct impact on alarm fatigue in the NICU. Performance evaluation studies can also impact medical device purchase decisions made by hospital administrators.
Original equipment manufacturers (OEM) market new-generation pulse oximeter technology with competing claims of greater accuracy during stable and unstable patient conditions, including motion and low perfusion.16 Generally, the OEM brings the product to market after convincing clinicians that a medical device is sufficiently accurate. However, clinicians debate the reliability of pulse oximeter data produced by reductionist proprietary algorithms.14 These algorithms largely remain unpublished by OEMs;15 making clinical research necessary to assess their effectiveness across different patient populations and pathologies. OEM monitors are built upon relatively simplistic proprietary algorithms for pre-processing artifacts in clinical settings.17,20,25 The high false alarm rates in patient monitors8,19,21 cause high noise levels, resulting in staff disruption and eventual staff desensitization in clinical environments.2,22 The ECRI Institute, a Pennsylvania-based patient-safety organization, listed alarm fatigue as the leading hazard on its annual list of the top 10 health-technology dangers for 2012 and 2013 (https://www.ecri.org/). Neonatal arterial oxygen saturation (SpO2) data acquired by Masimo SET Radical pulse oximeter technology (Masimo Corp., Irvine, CA, USA) have been frequently compared with data acquired by other new-generation pulse oximeters. Such a comparison was made with Datex-Ohmeda TruSat (GE Healthcare, Chalfont St Giles, UK), Siemens SC7000 (Siemens UK, Frimley, UK), Nonin 7500 (Nonin Medical, Plymouth, MN, USA) and Nellcor OxiMax N-600x (Covidien-Nellcor, Boulder, CO, USA).9 OEM-independent research compared Masimo SET data with Philips FAST MP50,29 and with Nellcor OxiMax
Keywords—Arterial oxygen saturation, Patient monitoring systems, False alarm rate, Nellcor OXIMAX, Sensitivity, Masimo SET.
Address correspondence to Shermeen Nizami, Systems and Computer Engineering, Carleton University, 1125 Colonel By Drive, Ottawa, ON K1S 5B6, Canada. Electronic mail: shermeen@sce. carleton.ca
383 1869-408X/15/0900-0383/0
2015 Biomedical Engineering Society
384
NIZAMI et al.
N-600x and Philips Intellivue MP70 (Philips, Germany).12 These studies only analyzed the difference in paired9,29or independent12 SpO2 data generated by each pulse oximeter. These studies did not take into consideration a gold standard, nor did they evaluate the performance of the pulse oximeters. Through a discourse16 on historic developments of the Masimo SET technology, its OEM has replied to van der Eijk et al.29 claiming greater accuracy in unstable conditions such as motion artifact and low perfusion leading to lower false alarm rates. However, the publications cited with these claims1,24 were supported by Masimo. Therefore, bias due to OEM involvement cannot be ruled out. In 2011, research9 revealed that in 2008 Masimo became aware that the Radical pulse oximeters manufactured until that time produce reduced frequency of SpO2 values in the range 87–90%. The software calibrates SpO2 values in this range to above 90%. In 2012, these findings were validated for Masimo SatShare modules integrated with Philips Intellivue MP70 monitors.12 This crossover observational study was conducted at the Neonatal Intensive Care Unit (NICU), Children’s Hospital of Eastern Ontario (CHEO), Ottawa, Canada. CHEO Research Ethics Board approved the data collection and performance evaluation of Masimo SET and Nellcor OXIMAX SmartPods. These SmartPods are integrated with Infinity Delta monitors (Dra¨ger Medical Systems, Inc. Telford, PA, USA). Dra¨ger Medical Systems supplied the latest versions of the SmartPods to the hospital at the time of data collection in 2010. Both pulse oximeters were proven to be consistently accurate during non-clinical testing at the CHEO biomedical engineering department. However, physicians are more interested in the clinical performance of pulse oximeters when used under conditions of patient movement and states of low perfusion. This research conducted in the NICU reports: (i) characteristic discrepancies found in data from the two OEM pulse oximeters; and (ii) performance evaluation to determine false alarm rates of the pulse oximeters. The heart rate (HR) derived from the electrocardiogram (ECG) is used as an independent non-invasive gold standard to estimate which pulse oximeter reading is more accurate when the two OEM devices disagree. This study is conducted independently of involvement from either OEM. The impact of this research reaches clinicians interested in determining whether one or the other OEM pulse oximeter can provide more accurate clinical information for a greater percentage of the time. This research also impacts medical device purchase decisions made by hospital administrators. The false alarm rate findings have a direct impact on alarm fatigue in the NICU.
MATERIALS AND METHODS Study Design All patients admitted to the NICU at CHEO were eligible to enroll in this study, with the following exclusion criteria: (a) parent’s refusal, (b) moribund patients, (c) patients with patent ductus arteriosus (PDA) or cyanotic congenital heart disease. Each infant was simultaneously monitored non-invasively by Masimo and Nellcor pulse oximeters using weight appropriate foot sensors from the respective OEM applied to each foot. The selection of the right or left limb to apply either Masimo or Nellcor sensor was randomized at the beginning of each recording. To minimize bias due to limb selection, the bedside nurse switched each sensor to the other foot midway through the recording. Blue posies were wrapped around both sensors to prevent optical crosstalk between them. Three ECG electrodes by Dra¨ger were appropriately placed on the infant’s chest and abdomen.
Physiologic Data Acquisition The following time-stamped data streams and corresponding alarms were collected simultaneously from each infant at a frequency of one reading every two seconds (0.5 Hz): PR and SpO2 from Masimo SET SmartPod Model # MS16356 (Masimo Corp., Irvine, CA, USA) integrated with an Infinity Delta monitor; PR and SpO2 from Nellcor OxiMax SmartPod Model # MS23997 (Covidien-Nellcor, Boulder, CO, USA) integrated with another Infinity Delta monitor; ECG derived Heart Rate (HR) and Respiratory Rate (RR) from one of the Infinity Delta monitors. The HR, sourced by independent electrical sensors, is used as the gold standard for evaluating the performance of both pulse oximeters that use optical sensors. Masimo and Nellcor SmartPods neither display nor output any Signal Quality Indicators (SQI). The averaging mode of both SmartPods was set to Normal. This setting is less sensitive to artifact, therefore, slower to alarm. Alarm thresholds on the monitors were set according to clinical guidelines in this NICU. Lower and upper alarm limits for PR and HR were 100 and 200 beats per minute (bpm) respectively. Similar to past recommendations,5 lower SpO2 limits ranged between 85 and 88%; and upper SpO2 limits ranged between 94 and 100%. Limits set for each patient depended on the clinical context including age, weight and medical condition. Recordings on each patient lasted approximately 4 h. RS232 serial ports on both Infinity Delta monitors were connected through Digi International Edgeport 4 hardware to a USB port on a computer. Eltima Port Monitor Software
Pulse Oximeter Performance Evaluation NICU
Professional Edition v4.x was installed on that computer for reading and logging data transmitted by each monitor in real-time. Data Annotation The bedside research associate (SN) annotated clinically significant events, perceptible artifacts and monitor alarms in real-time in consultation with the infant’s bedside nurse. Annotations include oxygen desaturation, clinical intervention, patient motion, routine care, sleep stages, feeding, sensor off, and any other events that triggered an alarm. The research associate asked the bedside nurse to assess the patient and report the clinical validity of the alarm whenever feasible. The neonatologist (JH) and SN retrospectively reviewed the data in conjunction with all real-time annotations to categorize SpO2 alarms generated by each pulse oximeter. For the purpose of this study a real clinical event implies a physiologic change in the patient’s condition as determined in real-time by the bedside nurse or retrospectively by the neonatologist. This includes both true desaturation (hypoxia) and violation of the upper saturation limit (hyperoxia). All alarms or clinical events occurring within a 1 min period were counted only once as they were assumed to be related to the same physiological event. Previous studies have considered longer event durations such as 5 min in the Pediatric Intensive Care Unit11 and 2 min in the Intensive Care Unit.19 There is no consensus upon an optimal definition for event duration to discriminate between linked and separate events.3 Clinically significant events of hypoxia or hyperoxia in NICU patients have been observed to last anywhere between 4 and 76 s.13,18 Hence, the selection of a 1 min interval was considered rather appropriate for this study. An alarm associated with a real clinical event was classified as a true positive (TP); silence during a real clinical event was classified as a false negative (FN); an alarm without a clinically significant event associated with it was termed as false positive (FP). FP alarms include not only those induced by motion artifact and other nonphysiological conditions, but also include true, brief breaches of alarm limits without clinical importance.10 These clinically irrelevant alarms have previously been described as clinically false7 or nuisance alarms that add to alarm fatigue.6,31 The HR and its accompanying alarms were taken as gold standard measurements. If there was an alarm associated with a low HR, then an accompanying oxygen desaturation would be considered a true event. JH visually compared the difference in the HR and PR values when determining the validity of an alarm. A difference £12 bpm was considered plausible. A larger difference cast doubt on the signal quality of the pulse oximeter.
385
Data Preparation Data collected from the three OEM monitors were synchronized. OEM-specified artifacts, missing segments and nonsensical data were removed and replaced with a missing value indicator for processing in MATLAB R2013a. Cleaned data from each monitor source were pooled for all patients. The following analysis was conducted on the pooled datasets. Data Characteristics The relative frequency plots of SpO2 data of both pulse oximeters and their box plots were generated. Such exploration can reveal unknown differential characteristics between the two datasets, which can otherwise go unnoticed in higher order statistical abstractions. SpO2 data from both pulse oximeters were tested for normality using Lilliefor test. Both datasets failed the normality test at the 5% significance level. Therefore, the nonparametric Kruskal–Wallis test was selected to compare ranks of the median of SpO2 samples from both pulse oximeters. The relative frequency plots of the Dra¨ger HR, and Nellcor and Masimo PRs were generated. These data were similarly tested for normality and failed. The empirical cumulative distribution function (CDF) for the Dra¨ger HR was explored to determine the cause of this failure. The nonparametric Kruskal–Wallis test compared ranks of the median HR and PRs. Spearman’s Rho (q) was used to determine the effect size of this analysis.4 Peformance Evaluation The performance of each pulse oximeter was evaluated by adding different SpO2 alarm categories across all patients. The following four performance measures were determined for each pulse oximeter: sensitivity (ratio of true alarms and total number of true events), positive predictive value (ratio of true alarms and all alarms), accuracy (ratio of true alarms and sum of all alarms and events), and false alarm rate (ratio of false positive alarms and sum of true positive and false positive alarms). The Nellcor and Masimo alarms classification contingency table was evaluated using the McNemar test.
RESULTS Data Preparation Data were collected from eleven preterm infants with diverse pathologies between February and June of 2010. A total of 79,200 samples of each data type were rendered for analysis. To synchronize data collected
386
NIZAMI et al.
from the three OEM monitors, these samples were interpolated to obtain a sample every second, resulting in 158,400 samples of each data type. The cleaned data consisted of 145,309 samples of each data type. Data Characteristics Patient demographics including weight and gestational age distributions are listed in Table 1. Figures 1 and 2 show the relative frequency of SpO2 acquired by each monitor for individual patients. Datasets for each pulse oximeter were pooled across all patients and explored as follows. Figure 3 depicts the relative frequency of SpO2 data of both pulse oximeters. Masimo produced reduced frequency of SpO2 values between 87 and 90%, similar to results shown by earlier research.9,12,23 Its algorithm calibrates values in this range to above 90%; hence the rise in frequency
TABLE 1. NICU patient demographics. Demographics (N = 11)
Median (range)
Gender (M:F) 7:4 Birth weight (g) 1197 (515–2240) Weight at recording (g) 2390 (677–3410) Gestational age (weeks) 28 (25–35) Corrected gestational age at recording (weeks) 36 (27–42) 88 (85–92) Lower SpO2 limit (%) Upper SpO2 limit (%) 100 (95–100) Heart Rate limit (bpm) (100–200)
between 97 and 99% as compared to Nellcor. This difference is further explored in Fig. 4 through the box plot for SpO2 data of each pulse oximeter. The median SpO2 values are Nellcor (97%) and Masimo (98%). The medians are similar; however, their ranks are different as shown by the range of the box plots. The Kruskal–Wallis test applied to the ranks of these medians results in (p < 0.0001, alpha = 0.05, 95% CI [ 17,372; 16,176]). The two large negative values are the 95% confidence interval (CI) for the difference in the ranks of the two median SpO2 values. Figure 5 compares the relative frequency of HR and PR data of all three OEM monitors, exhibiting some discrepancies at the low and high values. The CDF of the HR, F(x), in Fig. 6, does not satisfy the 68-95-99 rule for a Gaussian (normal) distribution.30 Box plots in Fig. 7 explore the differences in the HR and PR values. Their median values are 161 beats per minute (bpm) for Dra¨ger HR, and 162 bpm for both Nellcor and Masimo PRs. The medians are similar; however, their ranks are different as shown by the range of these box plots. The Kruskal–Wallis test on the ranks of the median HR and PRs results in (p < 0.0001, alpha = 0.05). 95% CIs for the difference in the ranks of their medians are: [ 11,693; 9505] for HR and Nellcor PR; [ 8017, 5829] for HR and Masimo PR; and [2583, 4770] for Nellcor PR and Masimo PR. Spearman’s q = 0.65 for SpO2 data of both pulse oximeters; q = 0.81 for PR data of both pulse oximeters; q = 0.93 for HR and Nellcor PR; q = 0.79 for HR and Masimo PR. The effect size measures the
FIGURE 1. Relative frequency of SpO2 values obtained by Nellcor for each patient.
Pulse Oximeter Performance Evaluation NICU
387
FIGURE 2. Relative frequency of SpO2 values obtained by Masimo for each patient.
FIGURE 3. Relative frequency of pooled SpO2 values obtained by Nellcor and Masimo across all patients.
fraction of variability in one variable (Nellcor SpO2) explained by the other correlated variable (Masimo SpO2). The effect size = q2 = 0.652 = 0.43 shows a medium to large effect.4 Performance Evaluation Categories of alarms were pooled for each pulse oximeter as shown in Table 2. The false alarm rate of
Nellcor was 50% and that of Masimo was 51.8%. Sensitivity of Nellcor was evaluated at 80.3% and that of Masimo at 72.2%. PPV of Nellcor was 50% and that of Masimo was 48.2%. Accuracy of Nellcor was 44.5% and that of Masimo was 40.6%. Table 3 shows the contingency table for Nellcor and Masimo alarm classification. Total number of alarms classified as discrepant was less than 10. Hence, a two-tailed exact McNemar test with significance (alpha/2 = 0.025) is
388
NIZAMI et al.
FIGURE 4. Box plot of pooled SpO2 data obtained by Nellcor and Masimo across all patients.
FIGURE 5. Relative frequency of pooled HR and PR data obtained by Dra¨ger, Nellcor and Masimo across all patients.
Pulse Oximeter Performance Evaluation NICU
389
FIGURE 6. Cumulative distribution function of HR data obtained by Dra¨ger across all patients.
FIGURE 7. Box plot for pooled HR and PR data of Dra¨ger, Nellcor and Masimo across all patients.
applied to this table, based on the cumulative binomial distribution with probability of success = probability of failure = 0.5.27 The proportion of success, i.e. true alarms for Nellcor was 0.44 and that of Masimo was 0.41, with p = 0.22.26
DISCUSSION Statistically significant differences were found in the Nellcor and Masimo SpO2 data with a difference of 1% in their medians. Although, at the time of data
collection in 2010, Dra¨ger Medical Systems supplied the latest versions of the SmartPods to CHEO. However, this research led to the verification that the Masimo SmartPod housed the discontinued version of the calibration algorithm and not the latest version which the OEM had been supplying to other hospitals since 2008.23 Therefore, the 1% higher median of Masimo can be attributed to its calibration algorithm causing a rise in frequency between 97 and 99%. Since the revised calibration algorithm is shown to return a distribution of values that is similar to Nellcor OxiMax oximeters.9 Therefore, it can be inferred that the
390
NIZAMI et al.
TABLE 2. Performance evaluation of Nellcor and Masimo pulse oximeters. Measure
Nellcor
Masimo
TP FP FN Sens PPV Accuracy
57 57 14 80.3% 50% 44.5%
52 56 20 72.2% 48.2% 40.6%
TABLE 3. Contingency table of Nellcor and Masimo alarms classification. Nellcor Pulse oximeter OEM Masimo True False Column total
True
False
Row total
52 5 57
1 71 72
53 76 129
discrepancy in the medians may be essentially nonexistent. Moreover, since this difference did not impact patient management in real-time, it was deemed clinically unimportant. Ranks of the median PRs differed statistically from each other and from that of the gold standard HR; however, a difference of 1 bpm in the actual median value is not clinically relevant. Nellcor PR exhibited the highest correlation with the HR. The McNemar test found no statistically significant difference between the performances of the two pulse oximeters since (p = 0.22) > (alpha/2 = 0.025). Despite competing claims of higher accuracy made by OEMs, this research produces evidence that there is no major advantage to using one technology over the other in preterm infants in the NICU. The high false alarm rates in both Nellcor and Masimo pulse oximeters result in low PPV and accuracy. The majority of FP alarms in this study were issued when physiologic data values hovered close to the lower or higher SpO2 limit. This is a limitation of the simplistic threshold-based alarm generation algorithms in newgeneration pulse oximeters. For greater clinical effectiveness, these algorithms need modification by including more complex clinical rule based reasoning. This may lead to increasing their sensitivity, accuracy and PPV, while decreasing alarm fatigue and staff desensitization in hospitals. Similar to earlier studies,9,12,29 another limitation is the slightly different averaging times applied by each OEM technology to smooth the data. Masimo-supported research28 describes the effect of averaging times on SpO2 readings acquired by Masimo SET.
The study sample size was limited due to hospital logistics. In future, a larger sample size could facilitate sub-group analyses with division based on clinical characteristics, weight, and gestational and chronological ages of infants.
CONCLUSION While the SpO2 and PR data had statistically different characteristics between the two monitors, the differences were not observed to be clinically important. Contrary to claims made by OEM-supported research, this study did not find statistically or clinically significant differences in the sensitivity, PPV and accuracy of SpO2 alarms generated by Nellcor OXIMAX and Masimo SET pulse oximeter technologies. This study produced evidence that both pulse oximeters have high false alarm rates when used in the preterm neonatal population with diverse pathologies. Performance evaluation plays an essential role for hospitals during purchase decisions. Unfortunately, it is often not possible for individual healthcare organizations to conduct detailed side by side evaluation of medical devices during the procurement process to validate vendor performance claims. Most hospitals have to rely on OEM and third party evaluation data in conjunction with their brief high level product assessments. As a result of this research, CHEO is considering updating the pulse oximeter SmartPod technology integrated with the Infinity Delta monitors in its NICU.
ACKNOWLEDGMENTS The authors would like to acknowledge Will Greenwood and Amna Basharat for computing, and Romaissa Saadi for data entry support.
ETHICAL STANDARD All procedures performed in the study involving human subjects were in accordance with the ethical standards of the CHEO REB and approved by it. This study does not involve any animal subjects. This research received no specific Grant from any funding agency in the public, commercial or not-for-profit sectors. CONFLICT OF INTEREST None of the authors have any conflict of interest to declare as per the ICMJE form for disclosure of potential conflict of interest.
Pulse Oximeter Performance Evaluation NICU
REFERENCES 1
Castillo, A., R. Deulofeut, A. Critz, and A. Sola. Prevention of retinopathy of prematurity in preterm infants through changes in clinical practice and SpO2 technology. Acta Paediatr. 100:188–192, 2011. 2 Cvach, M. Monitor alarm fatigue: an integrative review. Biomed. Instrum. Technol. 46:268–277, 2012. 3 Elder, D. E., A. J. Campbell, and D. Galletly. Current definitions for neonatal apnoea: are they evidence based? J. Paediatr. Child Health 49:E388–E396, 2013. 4 Field, A. Discovering statistics using SPSS for Windows. Thousand Oaks, CA: Sage, 2009. 5 Finer, N., and T. Leone. Oxygen saturation monitoring for the preterm infant: the evidence basis for current practice. Pediatr. Res. 65:375–380, 2009. 6 Graham, K. C., and M. Cvach. Monitor alarm fatigue: standardizing use of physiological monitors and decreasing nuisance alarms. Am. J. Crit. Care 19:28–34, 2010. 7 Imhoff, M., and S. Kuhls. Alarm algorithms in critical monitoring. Anesth. Analg. 102:1525–1537, 2006. 8 Imhoff, M., S. Kuhls, U. Gather, and R. Fried. Smart alarms from medical devices in the OR and ICU. Best Pract. Res. Clin. Anaesthesiol. 23:39–50, 2009. 9 Johnston, E. D., B. Boyle, E. Juszczak, A. King, P. Brocklehurst, and B. J. Stenson. Oxygen targeting in preterm infants using the Masimo SET radical pulse oximeter. Arch. Dis. Child. Fetal Neonatal Ed. 96:F429–F433, 2011. 10 Konkani, A., B. Oakley, and T. J. Bauld. Reducing hospital noise: a review of medical device alarm management. Biomed. Instrum. Technol. 46:478–487, 2012. 11 Lawless, S. T. Crying wolf: false alarms in a pediatric intensive care unit. Crit. Care Med. 22:981–985, 1994. 12 Li, D., V. Jeyaprakash, S. Foreman, and A. M. Groves. Comparing oxygen targeting in preterm infants between the Masimo and Philips pulse oximeters. Arch. Dis. Child. Fetal Neonatal Ed. 97:F311–F312, 2012. 13 Lim, K., K. I. Wheeler, T. J. Gale, H. D. Jackson, J. F. Kihlstrand, C. Sand, J. A. Dawson, and P. A. Dargaville. Oxygen saturation targeting in preterm infants receiving continuous positive airway pressure. J. Pediatr. 164:730–736, 2014. 14 Lynn, L. A., and J. P. Curry. Patterns of unexpected in-hospital deaths: a root cause analysis. Patient Saf. Surg. 5:3–24, 2011. 15 Nizami, S., J. R. Green, and C. McGregor. Implementation of artifact detection in critical care: a methodological review. IEEE Rev. Biomed. Eng. 6:127–142, 2013. 16 O’Reilly, M. Reply to ‘‘‘new-generation’ pulse oximeters in extremely low-birth-weight infants’’. J. Perinat. Neonatal Nurs. 26:282–283, 2012.
17
391
Raymer, K. E., J. Bergstro¨m, and J. M. Nyce. Anaesthesia monitor alarms: a theory-driven approach. Ergonomics 55:1487–1501, 2012. 18 Razi, N. M., M. DeLauter, and P. B. Pandit. Periodic breathing and oxygen saturation in preterm infants at discharge. J. Perinatol. 22:442–444, 2002. 19 Schmid, F., M. S. Goepfert, D. Kuhnt, V. Eichhorn, S. Diedrichs, H. Reichenspurner, A. E. Goetz, and D. A. Reuter. The wolf is crying in the operating room: patient monitor and anesthesia workstation alarming patterns during cardiac surgery. Anesth. Analg. 112:78–83, 2011. 20 Schoenberg, R., D. Z. Sands, and C. Safran. Making ICU alarms meaningful: a comparison of traditional vs. trendbased algorithms. Proceedings Annual Symposium AMIA, pp. 379–383, 1999. 21 Siebig, S., W. Sieben, F. Kollmann, M. Imhof, T. Bruennler, F. Rockmann, U. Gather, and C. E. Wrede. Users’ opinions on intensive care unit alarms: a survey of German intensive care units. Anaesth. Intensive Care 37:112–116, 2009. 22 Solet, J. M., and P. R. Barach. Managing alarm fatigue in cardiac care. Prog. Pediatr. Cardiol. 33:85–90, 2012. 23 Stenson, B. J., et al. Oxygen saturation and outcomes in preterm infants. N. Engl. J. Med. 368:2094–2104, 2013. 24 Taenzer, A. H., J. B. Pyke, S. P. McGrath, and G. T. Blike. Impact of pulse oximetry surveillance on rescue events and intensive care unit transfers: a before-and-after concurrence study. Anesthesiology 112:282–287, 2010. 25 Takla, G., J. H. Petre, D. J. Doyle, M. Horibe, and B. Gopakumaran. The problem of artifacts in patient monitor data during surgery: a clinical and methodological review. Anesth. Analg. 103:1196–1204, 2006. 26 Trujillo-Ortiz, A., R. Hernandez-Walls, and A. CastroPerez. McNemarextest: McNemar’s Exact Probability Test. A MATLAB file, 2004. 27 Uebersax, J. McNemar Tests of Marginal Homogeneity, 2009. 28 Vagedes, J., C. F. Poets, and K. Dietz. Averaging time, desaturation level, duration and extent. Arch. Dis. Child. Fetal Neonatal Ed. 98:F265–F266, 2013. 29 Van Der Eijk, A. C., S. Horsch, P. H. C. Eilers, J. Dankelman, and B. J. Smit. ‘‘New-generation’’ pulse oximeters in extremely low-birth-weight infants: How do they perform in clinical practice? J. Perinat. Neonatal Nurs. 26:172–180, 2012. 30 Weaver, A., and S. Goldberg. Clinical Biostatistics made Ridiculously Simple. Miami, FL: MedMaster Inc., 2011. 31 Welch, J. An evidence-based approach to reduce nuisance alarms and alarm fatigue. Biomed. Instrum. Technol. 45:46– 52, 2011.
Performance Evaluation of New-Generation Pulse Oximeters in the NICU: Observational Study SHERMEEN NIZAMI,1,2 KIM GREENWOOD,2 NICK BARROWMAN,3 and JOANN HARROLD3,4 1 Systems and Computer Engineering, Carleton University, 1125 Colonel By Drive, Ottawa, ON K1S 5B6, Canada; 2Clinical Engineering, The Children’s Hospital of Eastern Ontario, Ottawa, Canada; 3Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada; and 4Division of Neonatology, The Children’s Hospital of Eastern Ontario, Ottawa, Canada
(Received 13 June 2014; accepted 29 May 2015; published online 9 June 2015) Associate Editor Ajit P. Yoganathan oversaw the review of this article.
INTRODUCTION
Abstract—This crossover observational study compares the data characteristics and performance of new-generation Nellcor OXIMAX and Masimo SET SmartPod pulse oximeter technologies. The study was conducted independent of either original equipment manufacturer (OEM) across eleven preterm infants in a Neonatal Intensive Care Unit (NICU). The SmartPods were integrated with Dra¨ger Infinity Delta monitors. The Delta monitor measured the heart rate (HR) using an independent electrocardiogram sensor, and the two SmartPods collected arterial oxygen saturation (SpO2) and pulse rate (PR). All patient data were non-Gaussian. Nellcor PR showed a higher correlation with the HR as compared to Masimo PR. The statistically significant difference found in their median values (1% for SpO2, 1 bpm for PR) was deemed clinically insignificant. SpO2 alarms generated by both SmartPods were observed and categorized for performance evaluation. Results for sensitivity, positive predictive value, accuracy and false alarm rates were Nellcor (80.3, 50, 44.5, 50%) and Masimo (72.2, 48.2, 40.6, 51.8%) respectively. These metrics were not statistically significantly different between the two pulse oximeters. Despite claims by OEMs, both pulse oximeters exhibited high false alarm rates, with no statistically or clinically significant difference in performance. These findings have a direct impact on alarm fatigue in the NICU. Performance evaluation studies can also impact medical device purchase decisions made by hospital administrators.
Original equipment manufacturers (OEM) market new-generation pulse oximeter technology with competing claims of greater accuracy during stable and unstable patient conditions, including motion and low perfusion.16 Generally, the OEM brings the product to market after convincing clinicians that a medical device is sufficiently accurate. However, clinicians debate the reliability of pulse oximeter data produced by reductionist proprietary algorithms.14 These algorithms largely remain unpublished by OEMs;15 making clinical research necessary to assess their effectiveness across different patient populations and pathologies. OEM monitors are built upon relatively simplistic proprietary algorithms for pre-processing artifacts in clinical settings.17,20,25 The high false alarm rates in patient monitors8,19,21 cause high noise levels, resulting in staff disruption and eventual staff desensitization in clinical environments.2,22 The ECRI Institute, a Pennsylvania-based patient-safety organization, listed alarm fatigue as the leading hazard on its annual list of the top 10 health-technology dangers for 2012 and 2013 (https://www.ecri.org/). Neonatal arterial oxygen saturation (SpO2) data acquired by Masimo SET Radical pulse oximeter technology (Masimo Corp., Irvine, CA, USA) have been frequently compared with data acquired by other new-generation pulse oximeters. Such a comparison was made with Datex-Ohmeda TruSat (GE Healthcare, Chalfont St Giles, UK), Siemens SC7000 (Siemens UK, Frimley, UK), Nonin 7500 (Nonin Medical, Plymouth, MN, USA) and Nellcor OxiMax N-600x (Covidien-Nellcor, Boulder, CO, USA).9 OEM-independent research compared Masimo SET data with Philips FAST MP50,29 and with Nellcor OxiMax
Keywords—Arterial oxygen saturation, Patient monitoring systems, False alarm rate, Nellcor OXIMAX, Sensitivity, Masimo SET.
Address correspondence to Shermeen Nizami, Systems and Computer Engineering, Carleton University, 1125 Colonel By Drive, Ottawa, ON K1S 5B6, Canada. Electronic mail: shermeen@sce. carleton.ca
383 1869-408X/15/0900-0383/0
2015 Biomedical Engineering Society
384
NIZAMI et al.
N-600x and Philips Intellivue MP70 (Philips, Germany).12 These studies only analyzed the difference in paired9,29or independent12 SpO2 data generated by each pulse oximeter. These studies did not take into consideration a gold standard, nor did they evaluate the performance of the pulse oximeters. Through a discourse16 on historic developments of the Masimo SET technology, its OEM has replied to van der Eijk et al.29 claiming greater accuracy in unstable conditions such as motion artifact and low perfusion leading to lower false alarm rates. However, the publications cited with these claims1,24 were supported by Masimo. Therefore, bias due to OEM involvement cannot be ruled out. In 2011, research9 revealed that in 2008 Masimo became aware that the Radical pulse oximeters manufactured until that time produce reduced frequency of SpO2 values in the range 87–90%. The software calibrates SpO2 values in this range to above 90%. In 2012, these findings were validated for Masimo SatShare modules integrated with Philips Intellivue MP70 monitors.12 This crossover observational study was conducted at the Neonatal Intensive Care Unit (NICU), Children’s Hospital of Eastern Ontario (CHEO), Ottawa, Canada. CHEO Research Ethics Board approved the data collection and performance evaluation of Masimo SET and Nellcor OXIMAX SmartPods. These SmartPods are integrated with Infinity Delta monitors (Dra¨ger Medical Systems, Inc. Telford, PA, USA). Dra¨ger Medical Systems supplied the latest versions of the SmartPods to the hospital at the time of data collection in 2010. Both pulse oximeters were proven to be consistently accurate during non-clinical testing at the CHEO biomedical engineering department. However, physicians are more interested in the clinical performance of pulse oximeters when used under conditions of patient movement and states of low perfusion. This research conducted in the NICU reports: (i) characteristic discrepancies found in data from the two OEM pulse oximeters; and (ii) performance evaluation to determine false alarm rates of the pulse oximeters. The heart rate (HR) derived from the electrocardiogram (ECG) is used as an independent non-invasive gold standard to estimate which pulse oximeter reading is more accurate when the two OEM devices disagree. This study is conducted independently of involvement from either OEM. The impact of this research reaches clinicians interested in determining whether one or the other OEM pulse oximeter can provide more accurate clinical information for a greater percentage of the time. This research also impacts medical device purchase decisions made by hospital administrators. The false alarm rate findings have a direct impact on alarm fatigue in the NICU.
MATERIALS AND METHODS Study Design All patients admitted to the NICU at CHEO were eligible to enroll in this study, with the following exclusion criteria: (a) parent’s refusal, (b) moribund patients, (c) patients with patent ductus arteriosus (PDA) or cyanotic congenital heart disease. Each infant was simultaneously monitored non-invasively by Masimo and Nellcor pulse oximeters using weight appropriate foot sensors from the respective OEM applied to each foot. The selection of the right or left limb to apply either Masimo or Nellcor sensor was randomized at the beginning of each recording. To minimize bias due to limb selection, the bedside nurse switched each sensor to the other foot midway through the recording. Blue posies were wrapped around both sensors to prevent optical crosstalk between them. Three ECG electrodes by Dra¨ger were appropriately placed on the infant’s chest and abdomen.
Physiologic Data Acquisition The following time-stamped data streams and corresponding alarms were collected simultaneously from each infant at a frequency of one reading every two seconds (0.5 Hz): PR and SpO2 from Masimo SET SmartPod Model # MS16356 (Masimo Corp., Irvine, CA, USA) integrated with an Infinity Delta monitor; PR and SpO2 from Nellcor OxiMax SmartPod Model # MS23997 (Covidien-Nellcor, Boulder, CO, USA) integrated with another Infinity Delta monitor; ECG derived Heart Rate (HR) and Respiratory Rate (RR) from one of the Infinity Delta monitors. The HR, sourced by independent electrical sensors, is used as the gold standard for evaluating the performance of both pulse oximeters that use optical sensors. Masimo and Nellcor SmartPods neither display nor output any Signal Quality Indicators (SQI). The averaging mode of both SmartPods was set to Normal. This setting is less sensitive to artifact, therefore, slower to alarm. Alarm thresholds on the monitors were set according to clinical guidelines in this NICU. Lower and upper alarm limits for PR and HR were 100 and 200 beats per minute (bpm) respectively. Similar to past recommendations,5 lower SpO2 limits ranged between 85 and 88%; and upper SpO2 limits ranged between 94 and 100%. Limits set for each patient depended on the clinical context including age, weight and medical condition. Recordings on each patient lasted approximately 4 h. RS232 serial ports on both Infinity Delta monitors were connected through Digi International Edgeport 4 hardware to a USB port on a computer. Eltima Port Monitor Software
Pulse Oximeter Performance Evaluation NICU
Professional Edition v4.x was installed on that computer for reading and logging data transmitted by each monitor in real-time. Data Annotation The bedside research associate (SN) annotated clinically significant events, perceptible artifacts and monitor alarms in real-time in consultation with the infant’s bedside nurse. Annotations include oxygen desaturation, clinical intervention, patient motion, routine care, sleep stages, feeding, sensor off, and any other events that triggered an alarm. The research associate asked the bedside nurse to assess the patient and report the clinical validity of the alarm whenever feasible. The neonatologist (JH) and SN retrospectively reviewed the data in conjunction with all real-time annotations to categorize SpO2 alarms generated by each pulse oximeter. For the purpose of this study a real clinical event implies a physiologic change in the patient’s condition as determined in real-time by the bedside nurse or retrospectively by the neonatologist. This includes both true desaturation (hypoxia) and violation of the upper saturation limit (hyperoxia). All alarms or clinical events occurring within a 1 min period were counted only once as they were assumed to be related to the same physiological event. Previous studies have considered longer event durations such as 5 min in the Pediatric Intensive Care Unit11 and 2 min in the Intensive Care Unit.19 There is no consensus upon an optimal definition for event duration to discriminate between linked and separate events.3 Clinically significant events of hypoxia or hyperoxia in NICU patients have been observed to last anywhere between 4 and 76 s.13,18 Hence, the selection of a 1 min interval was considered rather appropriate for this study. An alarm associated with a real clinical event was classified as a true positive (TP); silence during a real clinical event was classified as a false negative (FN); an alarm without a clinically significant event associated with it was termed as false positive (FP). FP alarms include not only those induced by motion artifact and other nonphysiological conditions, but also include true, brief breaches of alarm limits without clinical importance.10 These clinically irrelevant alarms have previously been described as clinically false7 or nuisance alarms that add to alarm fatigue.6,31 The HR and its accompanying alarms were taken as gold standard measurements. If there was an alarm associated with a low HR, then an accompanying oxygen desaturation would be considered a true event. JH visually compared the difference in the HR and PR values when determining the validity of an alarm. A difference £12 bpm was considered plausible. A larger difference cast doubt on the signal quality of the pulse oximeter.
385
Data Preparation Data collected from the three OEM monitors were synchronized. OEM-specified artifacts, missing segments and nonsensical data were removed and replaced with a missing value indicator for processing in MATLAB R2013a. Cleaned data from each monitor source were pooled for all patients. The following analysis was conducted on the pooled datasets. Data Characteristics The relative frequency plots of SpO2 data of both pulse oximeters and their box plots were generated. Such exploration can reveal unknown differential characteristics between the two datasets, which can otherwise go unnoticed in higher order statistical abstractions. SpO2 data from both pulse oximeters were tested for normality using Lilliefor test. Both datasets failed the normality test at the 5% significance level. Therefore, the nonparametric Kruskal–Wallis test was selected to compare ranks of the median of SpO2 samples from both pulse oximeters. The relative frequency plots of the Dra¨ger HR, and Nellcor and Masimo PRs were generated. These data were similarly tested for normality and failed. The empirical cumulative distribution function (CDF) for the Dra¨ger HR was explored to determine the cause of this failure. The nonparametric Kruskal–Wallis test compared ranks of the median HR and PRs. Spearman’s Rho (q) was used to determine the effect size of this analysis.4 Peformance Evaluation The performance of each pulse oximeter was evaluated by adding different SpO2 alarm categories across all patients. The following four performance measures were determined for each pulse oximeter: sensitivity (ratio of true alarms and total number of true events), positive predictive value (ratio of true alarms and all alarms), accuracy (ratio of true alarms and sum of all alarms and events), and false alarm rate (ratio of false positive alarms and sum of true positive and false positive alarms). The Nellcor and Masimo alarms classification contingency table was evaluated using the McNemar test.
RESULTS Data Preparation Data were collected from eleven preterm infants with diverse pathologies between February and June of 2010. A total of 79,200 samples of each data type were rendered for analysis. To synchronize data collected
386
NIZAMI et al.
from the three OEM monitors, these samples were interpolated to obtain a sample every second, resulting in 158,400 samples of each data type. The cleaned data consisted of 145,309 samples of each data type. Data Characteristics Patient demographics including weight and gestational age distributions are listed in Table 1. Figures 1 and 2 show the relative frequency of SpO2 acquired by each monitor for individual patients. Datasets for each pulse oximeter were pooled across all patients and explored as follows. Figure 3 depicts the relative frequency of SpO2 data of both pulse oximeters. Masimo produced reduced frequency of SpO2 values between 87 and 90%, similar to results shown by earlier research.9,12,23 Its algorithm calibrates values in this range to above 90%; hence the rise in frequency
TABLE 1. NICU patient demographics. Demographics (N = 11)
Median (range)
Gender (M:F) 7:4 Birth weight (g) 1197 (515–2240) Weight at recording (g) 2390 (677–3410) Gestational age (weeks) 28 (25–35) Corrected gestational age at recording (weeks) 36 (27–42) 88 (85–92) Lower SpO2 limit (%) Upper SpO2 limit (%) 100 (95–100) Heart Rate limit (bpm) (100–200)
between 97 and 99% as compared to Nellcor. This difference is further explored in Fig. 4 through the box plot for SpO2 data of each pulse oximeter. The median SpO2 values are Nellcor (97%) and Masimo (98%). The medians are similar; however, their ranks are different as shown by the range of the box plots. The Kruskal–Wallis test applied to the ranks of these medians results in (p < 0.0001, alpha = 0.05, 95% CI [ 17,372; 16,176]). The two large negative values are the 95% confidence interval (CI) for the difference in the ranks of the two median SpO2 values. Figure 5 compares the relative frequency of HR and PR data of all three OEM monitors, exhibiting some discrepancies at the low and high values. The CDF of the HR, F(x), in Fig. 6, does not satisfy the 68-95-99 rule for a Gaussian (normal) distribution.30 Box plots in Fig. 7 explore the differences in the HR and PR values. Their median values are 161 beats per minute (bpm) for Dra¨ger HR, and 162 bpm for both Nellcor and Masimo PRs. The medians are similar; however, their ranks are different as shown by the range of these box plots. The Kruskal–Wallis test on the ranks of the median HR and PRs results in (p < 0.0001, alpha = 0.05). 95% CIs for the difference in the ranks of their medians are: [ 11,693; 9505] for HR and Nellcor PR; [ 8017, 5829] for HR and Masimo PR; and [2583, 4770] for Nellcor PR and Masimo PR. Spearman’s q = 0.65 for SpO2 data of both pulse oximeters; q = 0.81 for PR data of both pulse oximeters; q = 0.93 for HR and Nellcor PR; q = 0.79 for HR and Masimo PR. The effect size measures the
FIGURE 1. Relative frequency of SpO2 values obtained by Nellcor for each patient.
Pulse Oximeter Performance Evaluation NICU
387
FIGURE 2. Relative frequency of SpO2 values obtained by Masimo for each patient.
FIGURE 3. Relative frequency of pooled SpO2 values obtained by Nellcor and Masimo across all patients.
fraction of variability in one variable (Nellcor SpO2) explained by the other correlated variable (Masimo SpO2). The effect size = q2 = 0.652 = 0.43 shows a medium to large effect.4 Performance Evaluation Categories of alarms were pooled for each pulse oximeter as shown in Table 2. The false alarm rate of
Nellcor was 50% and that of Masimo was 51.8%. Sensitivity of Nellcor was evaluated at 80.3% and that of Masimo at 72.2%. PPV of Nellcor was 50% and that of Masimo was 48.2%. Accuracy of Nellcor was 44.5% and that of Masimo was 40.6%. Table 3 shows the contingency table for Nellcor and Masimo alarm classification. Total number of alarms classified as discrepant was less than 10. Hence, a two-tailed exact McNemar test with significance (alpha/2 = 0.025) is
388
NIZAMI et al.
FIGURE 4. Box plot of pooled SpO2 data obtained by Nellcor and Masimo across all patients.
FIGURE 5. Relative frequency of pooled HR and PR data obtained by Dra¨ger, Nellcor and Masimo across all patients.
Pulse Oximeter Performance Evaluation NICU
389
FIGURE 6. Cumulative distribution function of HR data obtained by Dra¨ger across all patients.
FIGURE 7. Box plot for pooled HR and PR data of Dra¨ger, Nellcor and Masimo across all patients.
applied to this table, based on the cumulative binomial distribution with probability of success = probability of failure = 0.5.27 The proportion of success, i.e. true alarms for Nellcor was 0.44 and that of Masimo was 0.41, with p = 0.22.26
DISCUSSION Statistically significant differences were found in the Nellcor and Masimo SpO2 data with a difference of 1% in their medians. Although, at the time of data
collection in 2010, Dra¨ger Medical Systems supplied the latest versions of the SmartPods to CHEO. However, this research led to the verification that the Masimo SmartPod housed the discontinued version of the calibration algorithm and not the latest version which the OEM had been supplying to other hospitals since 2008.23 Therefore, the 1% higher median of Masimo can be attributed to its calibration algorithm causing a rise in frequency between 97 and 99%. Since the revised calibration algorithm is shown to return a distribution of values that is similar to Nellcor OxiMax oximeters.9 Therefore, it can be inferred that the
390
NIZAMI et al.
TABLE 2. Performance evaluation of Nellcor and Masimo pulse oximeters. Measure
Nellcor
Masimo
TP FP FN Sens PPV Accuracy
57 57 14 80.3% 50% 44.5%
52 56 20 72.2% 48.2% 40.6%
TABLE 3. Contingency table of Nellcor and Masimo alarms classification. Nellcor Pulse oximeter OEM Masimo True False Column total
True
False
Row total
52 5 57
1 71 72
53 76 129
discrepancy in the medians may be essentially nonexistent. Moreover, since this difference did not impact patient management in real-time, it was deemed clinically unimportant. Ranks of the median PRs differed statistically from each other and from that of the gold standard HR; however, a difference of 1 bpm in the actual median value is not clinically relevant. Nellcor PR exhibited the highest correlation with the HR. The McNemar test found no statistically significant difference between the performances of the two pulse oximeters since (p = 0.22) > (alpha/2 = 0.025). Despite competing claims of higher accuracy made by OEMs, this research produces evidence that there is no major advantage to using one technology over the other in preterm infants in the NICU. The high false alarm rates in both Nellcor and Masimo pulse oximeters result in low PPV and accuracy. The majority of FP alarms in this study were issued when physiologic data values hovered close to the lower or higher SpO2 limit. This is a limitation of the simplistic threshold-based alarm generation algorithms in newgeneration pulse oximeters. For greater clinical effectiveness, these algorithms need modification by including more complex clinical rule based reasoning. This may lead to increasing their sensitivity, accuracy and PPV, while decreasing alarm fatigue and staff desensitization in hospitals. Similar to earlier studies,9,12,29 another limitation is the slightly different averaging times applied by each OEM technology to smooth the data. Masimo-supported research28 describes the effect of averaging times on SpO2 readings acquired by Masimo SET.
The study sample size was limited due to hospital logistics. In future, a larger sample size could facilitate sub-group analyses with division based on clinical characteristics, weight, and gestational and chronological ages of infants.
CONCLUSION While the SpO2 and PR data had statistically different characteristics between the two monitors, the differences were not observed to be clinically important. Contrary to claims made by OEM-supported research, this study did not find statistically or clinically significant differences in the sensitivity, PPV and accuracy of SpO2 alarms generated by Nellcor OXIMAX and Masimo SET pulse oximeter technologies. This study produced evidence that both pulse oximeters have high false alarm rates when used in the preterm neonatal population with diverse pathologies. Performance evaluation plays an essential role for hospitals during purchase decisions. Unfortunately, it is often not possible for individual healthcare organizations to conduct detailed side by side evaluation of medical devices during the procurement process to validate vendor performance claims. Most hospitals have to rely on OEM and third party evaluation data in conjunction with their brief high level product assessments. As a result of this research, CHEO is considering updating the pulse oximeter SmartPod technology integrated with the Infinity Delta monitors in its NICU.
ACKNOWLEDGMENTS The authors would like to acknowledge Will Greenwood and Amna Basharat for computing, and Romaissa Saadi for data entry support.
ETHICAL STANDARD All procedures performed in the study involving human subjects were in accordance with the ethical standards of the CHEO REB and approved by it. This study does not involve any animal subjects. This research received no specific Grant from any funding agency in the public, commercial or not-for-profit sectors. CONFLICT OF INTEREST None of the authors have any conflict of interest to declare as per the ICMJE form for disclosure of potential conflict of interest.
Pulse Oximeter Performance Evaluation NICU
REFERENCES 1
Castillo, A., R. Deulofeut, A. Critz, and A. Sola. Prevention of retinopathy of prematurity in preterm infants through changes in clinical practice and SpO2 technology. Acta Paediatr. 100:188–192, 2011. 2 Cvach, M. Monitor alarm fatigue: an integrative review. Biomed. Instrum. Technol. 46:268–277, 2012. 3 Elder, D. E., A. J. Campbell, and D. Galletly. Current definitions for neonatal apnoea: are they evidence based? J. Paediatr. Child Health 49:E388–E396, 2013. 4 Field, A. Discovering statistics using SPSS for Windows. Thousand Oaks, CA: Sage, 2009. 5 Finer, N., and T. Leone. Oxygen saturation monitoring for the preterm infant: the evidence basis for current practice. Pediatr. Res. 65:375–380, 2009. 6 Graham, K. C., and M. Cvach. Monitor alarm fatigue: standardizing use of physiological monitors and decreasing nuisance alarms. Am. J. Crit. Care 19:28–34, 2010. 7 Imhoff, M., and S. Kuhls. Alarm algorithms in critical monitoring. Anesth. Analg. 102:1525–1537, 2006. 8 Imhoff, M., S. Kuhls, U. Gather, and R. Fried. Smart alarms from medical devices in the OR and ICU. Best Pract. Res. Clin. Anaesthesiol. 23:39–50, 2009. 9 Johnston, E. D., B. Boyle, E. Juszczak, A. King, P. Brocklehurst, and B. J. Stenson. Oxygen targeting in preterm infants using the Masimo SET radical pulse oximeter. Arch. Dis. Child. Fetal Neonatal Ed. 96:F429–F433, 2011. 10 Konkani, A., B. Oakley, and T. J. Bauld. Reducing hospital noise: a review of medical device alarm management. Biomed. Instrum. Technol. 46:478–487, 2012. 11 Lawless, S. T. Crying wolf: false alarms in a pediatric intensive care unit. Crit. Care Med. 22:981–985, 1994. 12 Li, D., V. Jeyaprakash, S. Foreman, and A. M. Groves. Comparing oxygen targeting in preterm infants between the Masimo and Philips pulse oximeters. Arch. Dis. Child. Fetal Neonatal Ed. 97:F311–F312, 2012. 13 Lim, K., K. I. Wheeler, T. J. Gale, H. D. Jackson, J. F. Kihlstrand, C. Sand, J. A. Dawson, and P. A. Dargaville. Oxygen saturation targeting in preterm infants receiving continuous positive airway pressure. J. Pediatr. 164:730–736, 2014. 14 Lynn, L. A., and J. P. Curry. Patterns of unexpected in-hospital deaths: a root cause analysis. Patient Saf. Surg. 5:3–24, 2011. 15 Nizami, S., J. R. Green, and C. McGregor. Implementation of artifact detection in critical care: a methodological review. IEEE Rev. Biomed. Eng. 6:127–142, 2013. 16 O’Reilly, M. Reply to ‘‘‘new-generation’ pulse oximeters in extremely low-birth-weight infants’’. J. Perinat. Neonatal Nurs. 26:282–283, 2012.
17
391
Raymer, K. E., J. Bergstro¨m, and J. M. Nyce. Anaesthesia monitor alarms: a theory-driven approach. Ergonomics 55:1487–1501, 2012. 18 Razi, N. M., M. DeLauter, and P. B. Pandit. Periodic breathing and oxygen saturation in preterm infants at discharge. J. Perinatol. 22:442–444, 2002. 19 Schmid, F., M. S. Goepfert, D. Kuhnt, V. Eichhorn, S. Diedrichs, H. Reichenspurner, A. E. Goetz, and D. A. Reuter. The wolf is crying in the operating room: patient monitor and anesthesia workstation alarming patterns during cardiac surgery. Anesth. Analg. 112:78–83, 2011. 20 Schoenberg, R., D. Z. Sands, and C. Safran. Making ICU alarms meaningful: a comparison of traditional vs. trendbased algorithms. Proceedings Annual Symposium AMIA, pp. 379–383, 1999. 21 Siebig, S., W. Sieben, F. Kollmann, M. Imhof, T. Bruennler, F. Rockmann, U. Gather, and C. E. Wrede. Users’ opinions on intensive care unit alarms: a survey of German intensive care units. Anaesth. Intensive Care 37:112–116, 2009. 22 Solet, J. M., and P. R. Barach. Managing alarm fatigue in cardiac care. Prog. Pediatr. Cardiol. 33:85–90, 2012. 23 Stenson, B. J., et al. Oxygen saturation and outcomes in preterm infants. N. Engl. J. Med. 368:2094–2104, 2013. 24 Taenzer, A. H., J. B. Pyke, S. P. McGrath, and G. T. Blike. Impact of pulse oximetry surveillance on rescue events and intensive care unit transfers: a before-and-after concurrence study. Anesthesiology 112:282–287, 2010. 25 Takla, G., J. H. Petre, D. J. Doyle, M. Horibe, and B. Gopakumaran. The problem of artifacts in patient monitor data during surgery: a clinical and methodological review. Anesth. Analg. 103:1196–1204, 2006. 26 Trujillo-Ortiz, A., R. Hernandez-Walls, and A. CastroPerez. McNemarextest: McNemar’s Exact Probability Test. A MATLAB file, 2004. 27 Uebersax, J. McNemar Tests of Marginal Homogeneity, 2009. 28 Vagedes, J., C. F. Poets, and K. Dietz. Averaging time, desaturation level, duration and extent. Arch. Dis. Child. Fetal Neonatal Ed. 98:F265–F266, 2013. 29 Van Der Eijk, A. C., S. Horsch, P. H. C. Eilers, J. Dankelman, and B. J. Smit. ‘‘New-generation’’ pulse oximeters in extremely low-birth-weight infants: How do they perform in clinical practice? J. Perinat. Neonatal Nurs. 26:172–180, 2012. 30 Weaver, A., and S. Goldberg. Clinical Biostatistics made Ridiculously Simple. Miami, FL: MedMaster Inc., 2011. 31 Welch, J. An evidence-based approach to reduce nuisance alarms and alarm fatigue. Biomed. Instrum. Technol. 45:46– 52, 2011.
