366
Short report
Reliability of electronic recording of waiting times in the emergency department: a prospective multicenter study Judith Gorlickia,e, Pierre-Alexis Raynalb,e, Agathe Leleuc,f, Bruno Rioua,e, Patrick Rayd,e and Yonathan Freunda,e We aimed to evaluate the reliability of waiting times (WT) measures electronically retrieved. We prospectively collected true WT in four emergency departments during 20 predefined 2-h inclusion periods, and compared them with the electronically retrieved waiting time (ERWT). We assessed agreement with calculation of rate of outliers (difference exceeding 20 min), bias, and its 95% limits of agreements, and associated Bland and Altman plot. We analyzed 274 patients. The mean difference was − 2 min (SD 13) between ERWT and true WT, with a 95% limits of agreements (− 28 to 24 min). Bland and Altman plot showed a good agreement, and we report 7% of outliers. Using ERWT, 14 patients (5%) were misclassified as having their target WT exceeded or not. ERWT agree well with the true WT, although the significant rate of outlier and misclassification calls for caution in their
Introduction Waiting times (WT) in the emergency department (ED) are a matter of great concern because increased WT have been reported to be associated with decreased satisfaction [1], delayed urgent treatment [2], and poorer outcomes [3]. Furthermore, longer WT may have a negative social impact as patients’ satisfaction decreases. In several countries, WT are used as an indicator of quality and efficiency. In Canada, Australia, the UK, and the USA, target WT are used as a measure of quality of emergency care [4]. In France, the ministry of health recently focused on prolonged WT in the ED, and is planning reforms both to reduce WT and to generalize live reports of WT in each ED.
Importance
Either for scientific purpose or for socioeconomic objectives, WT are widely studied and analyzed. These data are primarily collected from electronic records because manual recording of WT requires unrealistic resource allocation in EDs that treat more than 100 patients each day [3,5]. Whether this measure is reliable has never been studied, and a strong agreement between electronic record of waiting time (ERWT) and the true WT is mandatory to extrapolate the conclusions of clinical and administrative studies on this matter. Electronically recorded time is the reported variable described in recent studies or report that estimate WT or length of stay in the ED [5–7]. 0969-9546 Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
interpretation. European Journal of Emergency Medicine 22:366–369 Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved. European Journal of Emergency Medicine 2015, 22:366–369 Keywords: agreement, Bland and Altman, waiting times a Emergency Department, Assistance Publique-Hôpitaux de Paris AP-HP, Hôpital Pitié-Salpêtrière, bEmergency Department, APHP, Hôpital Saint-Antoine, c Emergency Department, APHP, Hôpital Bichat-Claude Bernard, dEmergency Department, APHP, Hôpital Tenon, eSorbonne Universités, UPMC Univ Paris 06 and fUniversité Paris Diderot, Sorbonne Paris Cité, Paris, France
Correspondence to Yonathan Freund, MD, Service d’Accueil des Urgences, Groupe Hospitalier Pitié-Salpêtrière, 83 boulevard de l’hôpital, 75013 Paris, France Tel: + 33 1 84827129; fax: + 33 1 42177001; e-mail: [email protected] Received 22 August 2014 Accepted 6 November 2014
We hypothesize that ERWT and true WT are in close agreement, with a low rate of outliers (i.e. a difference < 20 min).
Materials and methods This is a multicenter prospective study carried out during the months of May–June 2014, in four urban academic ED affiliated with Assistance Publique – Hôpitaux de Paris (AP-HP), with an annual census ranging from 45 000 to 85 000 visits/year. All the participating EDs use the same electronic medical record software (Urqual; McKesson, San Francisco, California, USA). In the participating EDs, following administrative registration, patients are assessed by a triage nurse, who classifies the acuity on a 1 to 5 triage scale, before seeing a boardcertified emergency physician (EP) or a resident [5].
Selection of participants and outcome measure
We included for analysis all patients who were seen by an EP during predefined 2-h periods of recruitment. These periods were set to recruit a sample that would include patients from various days of the week and time of the day as these variables (along with triage level and age) have been reported to be associated with different WT [5]. We excluded patients with triage level 1 (most severe) as they were to be seen immediately and patients with triage level 5 (less severe) as they were often referred to minors unit or ambulatory clinic. DOI: 10.1097/MEJ.0000000000000232
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
Reliability of electronic waiting times in the ED Gorlicki et al. 367
In the predefined 2-h periods, a local investigator (EP) recorded manually the WT of patients who were seen by EP. We defined the true WT as the time to seeing an EP, that is the duration between time of arrival of the patient at the ED (electronically recorded as the time of first administrative contact) and time where the EP and the patient met in the examination cubicle. The ERWT was the delay between the time of arrival at the ED and the time when the first EP electronically logs in the medical record. As the process of care was unchanged, our Institutional Review Board authorized the study without the need for informed consent (Comité de Protection des Personnes – Paris Ile de France 6). Statistical analysis and sample size
Categorical data are expressed as number (%). Gaussian distributed data are expressed as mean (± SD) and nonGaussian distributed data are expressed as median (25–75% interquartile range). Comparisons of proportions were performed using the χ2-test and mean using the Student's t-test. ERWT and true WT are expressed in minutes, and their difference was calculated. We defined a record as an outlier when the absolute difference (ERWT – true WT) was longer than 20 min. This threshold has been set arbitrarily, and corresponds to the recommended time frame to see patients with the highest level of acuity included in our study (level 2). We calculated the mean difference (i.e. the bias) and its 95% limits of agreement, and present our results using a Bland and Altman plot. As mentioned elsewhere [5,8], the raw value of the WT may be difficult to interpret because it does not take into account the severity of the patient. We then calculated the rate of patients seen within the recommended triage time with both ERWT and true WT, and reported the rate of patients misclassified with ERWT. We considered that to be reliable, the rate of outliers should not exceed 10%. After a first evaluation of 100 patients, we estimated the rate of outliers to be of 6%. To have the upper bond of the 95% Confidence Interval (CI) below 10%, we estimated that 250 patients were needed – an estimate of 20 2-h periods. All analyses were two-tailed. Statistical analysis was carried out using SPSS software (version 20.0; IBM, Armonk, New York, USA).
Results During 20 prespecified 2-h inclusion periods, we included 388 patients for analysis [155 (57%) from PitiéSalpêtrière, 57 (21%) from Saint-Antoine, 45 (16%) from Bichat, and 17 (6%) from Tenon]. The mean age of the patients was 51 years (SD 21). The main characteristics are reported in Table 1; there was no significant difference in distribution for day of the week, time of the day, or triage level between our sample and the population that attends these centers [5].
Table 1
Baseline characteristics N (%)
Center PSL SAT BCH TNN Age [mean (SD)] (years) > 75 Triage level 2 3 4 Day of the week Tuesday to Saturday Monday Sunday Time of visit Day (1–6 p.m.) Morning (8 a.m.–1 p.m.) Night (6 p.m.–8 a.m.) Waiting times [mean (SD) median (IQR)] ERWT True WT Difference (true WT-ERWT) Mean (SD) median (IQR) Absolute difference (min) [N (%) (95% CI)] > 10 > 20 Exceeded TWT [N (%) (95% CI)] With ERWT With True WT Discordance in TWT exceeded [N (%) (95% CI)]
274 155 (57) 57 (21) 45 (16) 17 (6) 51 (21) 47 (17) 52 (19) 168 (61) 54 (20) 169 (62) 82 (30) 23 (8) 103 (38) 124 (45) 47 (17) 68 (48) [54 (36–91)] 66 (48) [50 (31–90)] − 2 (13) [ − 1 (− 4 to 0)] 6 (12) [2 (1–6)] 43 (16) (12–21) 19 (7) (5–10) 137 (50) (44–56) 131 (48) (42–54) 14 (5) (3–8)
BCH, Bichat; ERWT, electronically retrieved waiting time; IQR, interquartile range; PSL, Pitié-Salpêtrière; SAT, Saint-Antoine; TNN, Tenon; TWT, target waiting time; WT, waiting times.
The mean WT was 68 min (SD 48) for ERWT and 66 min (SD 48) for true WT (Table 1). The mean difference (bias) was − 2 min (SD 13), with a 95% limits of agreement (− 28 to 24 min). The Bland and Altman plot showed good agreement (Fig. 1). There was a difference greater than 20 min in 19 patients [6.9%, 95% CI (4.5–10.5%)]. Using ERWT, we found that 137 patients (50%) had their target WT exceeded, versus 131 (48%) with the true WT. This outcome was discordant in 14 patients [5%, (95% CI 3–8%)].
Discussion In this prospective multicenter study, we report that in 7% patients (95% CI 5–10%), the difference between the true WT and the ERWT exceeds 20 min. This is, to our knowledge, the first study that assesses the reliability of electronic measurement of WT. The very low bias of 2 min that we report, and the Bland and Altman plot, suggests that electronic measurement and recording and WT is accurate. However, the rate of outliers calls for cautious interpretation of ERWT: we report an upper value of the 95% CI above 10%. In addition, we report a discrepancy of 5% of patients ‘seen within the recommended time frame’. These results question the reliability of WT and advocate for a generalized study of their accuracy.
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
368
European Journal of Emergency Medicine 2015, Vol 22 No 5
Fig. 1
50
Difference
0
−50
−100
−150 0
50 Bias
100 150 Mean waiting time 95% LOA
200
250
Outlier
Bland and Altman plot. Waiting time (min). 95% LOA = ±1.96 × SD. LOA, limits of agreements.
to include patients with different characteristics (especially regarding age, triage level, day of the week, and time of the day – factors that we reported previously to be associated independently with a prolonged WT) [5]. Second, a Hawthorne effect might have reduced the differences between ERWT and true WT: although the inclusion periods were kept blinded, EP were aware of the ongoing study, and some might have noticed that they were being recorded. Third, our results may not be generalized to other health care systems. Although we evaluated four different centers, only one electronic software and process of ERWT has been evaluated. Fourth, we excluded patients with lower severity triage (5 on a 1–5 scale) because they were managed independently in some centers in different parts of the department, such as ‘minors’ or ‘fast track’ units. Therefore, our results may not be generalizable to triage five patients. Finally, we chose to define an outlier as a difference of more than 20 min for the two methods and the maximal acceptable rate of outliers at 10%. We agree that these thresholds are arbitrary, and therefore potentially subject to criticism.
The raw value of WT is a quality indicator and may be associated with financial incentive, but the accuracy of recording and the collection method has never been evaluated. In France, EDs often use electronic support software for administrative tasks and WT analysis [5,6,9]. Locker and Mason [9] have previously reported that the recording of WT or ED length of stay could be unreliable, and may have been manipulated to follow targets such as the ‘4-h’ in EDs from the UK. They suggest, however, that electronic recording instead of manual recording would reduce this bias [10]. We present results that should alert clinicians and managers as to the reliability of reported electronic data. In our opinion, EDs and institutions should first evaluate their method of collecting WT and their accuracy before drawing conclusions from their data.
Conclusion
The reasons for the differences between ERWT and the true WT are not obvious. We anticipated that the differences will be most of the time positive, resulting from the fact that EP may first assess their patients before logging into the software to report their clinical examination and entry orders. However, roughly half of our patients had ERWT that were lower than their true WT, hence a negative difference. This was also the case for those with outlier measures, without any obvious cause.
There are no conflicts of interest.
Limitations
This study presents some limitations. First, measurement of WT is a time-consuming activity; therefore, we could not systematically include every patient who visits our ED for a prolonged inclusion time. To reduce this inclusion bias, we chose prespecified various 2-h periods
In summary, ERWT and true WT agree closely. However, we report a significant rate of outliers: 7% of data with a more than 20 min difference, and a 5% misclassification rate for patients seen within the target WT.
Acknowledgements The authors would like to thank Benjamin Bloom (Royal London Hospital, London, UK) for his help and support. Y.F. and B.R. conceived the study. J.G., P.A.R., and A.L. collected the data. B.R. and Y.F. carried out statistical analysis. Y.F., B.R., and P.R. interpreted the results. Y.F. drafted the manuscript. J.G., B.R., and P.R. made substantial contribution. Conflicts of interest
References 1
2
3
4
5
Spaite DW, Bartholomeaux F, Guisto J, Lindberg E, Hull B, Eyherabide A, et al. Rapid process redesign in a university-based emergency department: decreasing waiting time intervals and improving patient satisfaction. Ann Emerg Med 2002; 39:168–177. McCarthy ML, Zeger SL, Ding R, Levin SR, Desmond JS, Lee J, Aronsky D. Crowding delays treatment and lengthens emergency department length of stay, even among high-acuity patients. Ann Emerg Med 2009; 54:492–503, e4. Guttmann A, Schull MJ, Vermeulen MJ, Stukel TA. Association between waiting times and short term mortality and hospital admission after departure from emergency department: population based cohort study from Ontario, Canada. BMJ 2011; 342:d2983. The National Quality Forum. National Consensus Standards for Emergency Care. A Consensus Report. Available at: http://www.qualityforum.org/ Publications/2009/09/National_Voluntary_Consensus_Standards_for_ Emergency_Care.aspx. [Accessed 27 December 2012]. Freund Y, Vincent-Cassy C, Bloom B, Riou B, Ray P. APHP Emergency Database Study Group. Association between age older than 75 years and exceeded target waiting times in the emergency department: a multicenter
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
Reliability of electronic waiting times in the ED Gorlicki et al. 369
cross-sectional survey in the Paris metropolitan area, France. Ann Emerg Med 2013; 62:449–456. 6 Limkakeng AT Jr, Glickman SW, Shofer F, Mani G, Drake W, Freeman D, et al. Are patients with longer emergency department wait times less likely to consent to research? Acad Emerg Med 2012; 19:396–401. 7 Huang EP, Liu SS, Fang CC, Chou HC, Wang CH, Yen ZS, Chen SC. The impact of adding clinical assistants on patient waiting time in a crowded emergency department. Emerg Med J 2013; 30:1017–1019.
8
Horwitz LI, Bradley EH. Percentage of US emergency department patients seen within the recommended triage time: 1997 to 2006. Arch Intern Med 2009; 169:1857–1865. 9 Locker TE, Mason SM. Are these emergency department performance data real? Emerg Med J 2006; 23:558–559. 10 Keep SL, Locker TE. The impact of a computerised whiteboard system on digit preference bias in the recording of emergency department process times. Eur J Emerg Med 2012; 19:177–180.
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
Short report
Reliability of electronic recording of waiting times in the emergency department: a prospective multicenter study Judith Gorlickia,e, Pierre-Alexis Raynalb,e, Agathe Leleuc,f, Bruno Rioua,e, Patrick Rayd,e and Yonathan Freunda,e We aimed to evaluate the reliability of waiting times (WT) measures electronically retrieved. We prospectively collected true WT in four emergency departments during 20 predefined 2-h inclusion periods, and compared them with the electronically retrieved waiting time (ERWT). We assessed agreement with calculation of rate of outliers (difference exceeding 20 min), bias, and its 95% limits of agreements, and associated Bland and Altman plot. We analyzed 274 patients. The mean difference was − 2 min (SD 13) between ERWT and true WT, with a 95% limits of agreements (− 28 to 24 min). Bland and Altman plot showed a good agreement, and we report 7% of outliers. Using ERWT, 14 patients (5%) were misclassified as having their target WT exceeded or not. ERWT agree well with the true WT, although the significant rate of outlier and misclassification calls for caution in their
Introduction Waiting times (WT) in the emergency department (ED) are a matter of great concern because increased WT have been reported to be associated with decreased satisfaction [1], delayed urgent treatment [2], and poorer outcomes [3]. Furthermore, longer WT may have a negative social impact as patients’ satisfaction decreases. In several countries, WT are used as an indicator of quality and efficiency. In Canada, Australia, the UK, and the USA, target WT are used as a measure of quality of emergency care [4]. In France, the ministry of health recently focused on prolonged WT in the ED, and is planning reforms both to reduce WT and to generalize live reports of WT in each ED.
Importance
Either for scientific purpose or for socioeconomic objectives, WT are widely studied and analyzed. These data are primarily collected from electronic records because manual recording of WT requires unrealistic resource allocation in EDs that treat more than 100 patients each day [3,5]. Whether this measure is reliable has never been studied, and a strong agreement between electronic record of waiting time (ERWT) and the true WT is mandatory to extrapolate the conclusions of clinical and administrative studies on this matter. Electronically recorded time is the reported variable described in recent studies or report that estimate WT or length of stay in the ED [5–7]. 0969-9546 Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.
interpretation. European Journal of Emergency Medicine 22:366–369 Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved. European Journal of Emergency Medicine 2015, 22:366–369 Keywords: agreement, Bland and Altman, waiting times a Emergency Department, Assistance Publique-Hôpitaux de Paris AP-HP, Hôpital Pitié-Salpêtrière, bEmergency Department, APHP, Hôpital Saint-Antoine, c Emergency Department, APHP, Hôpital Bichat-Claude Bernard, dEmergency Department, APHP, Hôpital Tenon, eSorbonne Universités, UPMC Univ Paris 06 and fUniversité Paris Diderot, Sorbonne Paris Cité, Paris, France
Correspondence to Yonathan Freund, MD, Service d’Accueil des Urgences, Groupe Hospitalier Pitié-Salpêtrière, 83 boulevard de l’hôpital, 75013 Paris, France Tel: + 33 1 84827129; fax: + 33 1 42177001; e-mail: [email protected] Received 22 August 2014 Accepted 6 November 2014
We hypothesize that ERWT and true WT are in close agreement, with a low rate of outliers (i.e. a difference < 20 min).
Materials and methods This is a multicenter prospective study carried out during the months of May–June 2014, in four urban academic ED affiliated with Assistance Publique – Hôpitaux de Paris (AP-HP), with an annual census ranging from 45 000 to 85 000 visits/year. All the participating EDs use the same electronic medical record software (Urqual; McKesson, San Francisco, California, USA). In the participating EDs, following administrative registration, patients are assessed by a triage nurse, who classifies the acuity on a 1 to 5 triage scale, before seeing a boardcertified emergency physician (EP) or a resident [5].
Selection of participants and outcome measure
We included for analysis all patients who were seen by an EP during predefined 2-h periods of recruitment. These periods were set to recruit a sample that would include patients from various days of the week and time of the day as these variables (along with triage level and age) have been reported to be associated with different WT [5]. We excluded patients with triage level 1 (most severe) as they were to be seen immediately and patients with triage level 5 (less severe) as they were often referred to minors unit or ambulatory clinic. DOI: 10.1097/MEJ.0000000000000232
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
Reliability of electronic waiting times in the ED Gorlicki et al. 367
In the predefined 2-h periods, a local investigator (EP) recorded manually the WT of patients who were seen by EP. We defined the true WT as the time to seeing an EP, that is the duration between time of arrival of the patient at the ED (electronically recorded as the time of first administrative contact) and time where the EP and the patient met in the examination cubicle. The ERWT was the delay between the time of arrival at the ED and the time when the first EP electronically logs in the medical record. As the process of care was unchanged, our Institutional Review Board authorized the study without the need for informed consent (Comité de Protection des Personnes – Paris Ile de France 6). Statistical analysis and sample size
Categorical data are expressed as number (%). Gaussian distributed data are expressed as mean (± SD) and nonGaussian distributed data are expressed as median (25–75% interquartile range). Comparisons of proportions were performed using the χ2-test and mean using the Student's t-test. ERWT and true WT are expressed in minutes, and their difference was calculated. We defined a record as an outlier when the absolute difference (ERWT – true WT) was longer than 20 min. This threshold has been set arbitrarily, and corresponds to the recommended time frame to see patients with the highest level of acuity included in our study (level 2). We calculated the mean difference (i.e. the bias) and its 95% limits of agreement, and present our results using a Bland and Altman plot. As mentioned elsewhere [5,8], the raw value of the WT may be difficult to interpret because it does not take into account the severity of the patient. We then calculated the rate of patients seen within the recommended triage time with both ERWT and true WT, and reported the rate of patients misclassified with ERWT. We considered that to be reliable, the rate of outliers should not exceed 10%. After a first evaluation of 100 patients, we estimated the rate of outliers to be of 6%. To have the upper bond of the 95% Confidence Interval (CI) below 10%, we estimated that 250 patients were needed – an estimate of 20 2-h periods. All analyses were two-tailed. Statistical analysis was carried out using SPSS software (version 20.0; IBM, Armonk, New York, USA).
Results During 20 prespecified 2-h inclusion periods, we included 388 patients for analysis [155 (57%) from PitiéSalpêtrière, 57 (21%) from Saint-Antoine, 45 (16%) from Bichat, and 17 (6%) from Tenon]. The mean age of the patients was 51 years (SD 21). The main characteristics are reported in Table 1; there was no significant difference in distribution for day of the week, time of the day, or triage level between our sample and the population that attends these centers [5].
Table 1
Baseline characteristics N (%)
Center PSL SAT BCH TNN Age [mean (SD)] (years) > 75 Triage level 2 3 4 Day of the week Tuesday to Saturday Monday Sunday Time of visit Day (1–6 p.m.) Morning (8 a.m.–1 p.m.) Night (6 p.m.–8 a.m.) Waiting times [mean (SD) median (IQR)] ERWT True WT Difference (true WT-ERWT) Mean (SD) median (IQR) Absolute difference (min) [N (%) (95% CI)] > 10 > 20 Exceeded TWT [N (%) (95% CI)] With ERWT With True WT Discordance in TWT exceeded [N (%) (95% CI)]
274 155 (57) 57 (21) 45 (16) 17 (6) 51 (21) 47 (17) 52 (19) 168 (61) 54 (20) 169 (62) 82 (30) 23 (8) 103 (38) 124 (45) 47 (17) 68 (48) [54 (36–91)] 66 (48) [50 (31–90)] − 2 (13) [ − 1 (− 4 to 0)] 6 (12) [2 (1–6)] 43 (16) (12–21) 19 (7) (5–10) 137 (50) (44–56) 131 (48) (42–54) 14 (5) (3–8)
BCH, Bichat; ERWT, electronically retrieved waiting time; IQR, interquartile range; PSL, Pitié-Salpêtrière; SAT, Saint-Antoine; TNN, Tenon; TWT, target waiting time; WT, waiting times.
The mean WT was 68 min (SD 48) for ERWT and 66 min (SD 48) for true WT (Table 1). The mean difference (bias) was − 2 min (SD 13), with a 95% limits of agreement (− 28 to 24 min). The Bland and Altman plot showed good agreement (Fig. 1). There was a difference greater than 20 min in 19 patients [6.9%, 95% CI (4.5–10.5%)]. Using ERWT, we found that 137 patients (50%) had their target WT exceeded, versus 131 (48%) with the true WT. This outcome was discordant in 14 patients [5%, (95% CI 3–8%)].
Discussion In this prospective multicenter study, we report that in 7% patients (95% CI 5–10%), the difference between the true WT and the ERWT exceeds 20 min. This is, to our knowledge, the first study that assesses the reliability of electronic measurement of WT. The very low bias of 2 min that we report, and the Bland and Altman plot, suggests that electronic measurement and recording and WT is accurate. However, the rate of outliers calls for cautious interpretation of ERWT: we report an upper value of the 95% CI above 10%. In addition, we report a discrepancy of 5% of patients ‘seen within the recommended time frame’. These results question the reliability of WT and advocate for a generalized study of their accuracy.
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
368
European Journal of Emergency Medicine 2015, Vol 22 No 5
Fig. 1
50
Difference
0
−50
−100
−150 0
50 Bias
100 150 Mean waiting time 95% LOA
200
250
Outlier
Bland and Altman plot. Waiting time (min). 95% LOA = ±1.96 × SD. LOA, limits of agreements.
to include patients with different characteristics (especially regarding age, triage level, day of the week, and time of the day – factors that we reported previously to be associated independently with a prolonged WT) [5]. Second, a Hawthorne effect might have reduced the differences between ERWT and true WT: although the inclusion periods were kept blinded, EP were aware of the ongoing study, and some might have noticed that they were being recorded. Third, our results may not be generalized to other health care systems. Although we evaluated four different centers, only one electronic software and process of ERWT has been evaluated. Fourth, we excluded patients with lower severity triage (5 on a 1–5 scale) because they were managed independently in some centers in different parts of the department, such as ‘minors’ or ‘fast track’ units. Therefore, our results may not be generalizable to triage five patients. Finally, we chose to define an outlier as a difference of more than 20 min for the two methods and the maximal acceptable rate of outliers at 10%. We agree that these thresholds are arbitrary, and therefore potentially subject to criticism.
The raw value of WT is a quality indicator and may be associated with financial incentive, but the accuracy of recording and the collection method has never been evaluated. In France, EDs often use electronic support software for administrative tasks and WT analysis [5,6,9]. Locker and Mason [9] have previously reported that the recording of WT or ED length of stay could be unreliable, and may have been manipulated to follow targets such as the ‘4-h’ in EDs from the UK. They suggest, however, that electronic recording instead of manual recording would reduce this bias [10]. We present results that should alert clinicians and managers as to the reliability of reported electronic data. In our opinion, EDs and institutions should first evaluate their method of collecting WT and their accuracy before drawing conclusions from their data.
Conclusion
The reasons for the differences between ERWT and the true WT are not obvious. We anticipated that the differences will be most of the time positive, resulting from the fact that EP may first assess their patients before logging into the software to report their clinical examination and entry orders. However, roughly half of our patients had ERWT that were lower than their true WT, hence a negative difference. This was also the case for those with outlier measures, without any obvious cause.
There are no conflicts of interest.
Limitations
This study presents some limitations. First, measurement of WT is a time-consuming activity; therefore, we could not systematically include every patient who visits our ED for a prolonged inclusion time. To reduce this inclusion bias, we chose prespecified various 2-h periods
In summary, ERWT and true WT agree closely. However, we report a significant rate of outliers: 7% of data with a more than 20 min difference, and a 5% misclassification rate for patients seen within the target WT.
Acknowledgements The authors would like to thank Benjamin Bloom (Royal London Hospital, London, UK) for his help and support. Y.F. and B.R. conceived the study. J.G., P.A.R., and A.L. collected the data. B.R. and Y.F. carried out statistical analysis. Y.F., B.R., and P.R. interpreted the results. Y.F. drafted the manuscript. J.G., B.R., and P.R. made substantial contribution. Conflicts of interest
References 1
2
3
4
5
Spaite DW, Bartholomeaux F, Guisto J, Lindberg E, Hull B, Eyherabide A, et al. Rapid process redesign in a university-based emergency department: decreasing waiting time intervals and improving patient satisfaction. Ann Emerg Med 2002; 39:168–177. McCarthy ML, Zeger SL, Ding R, Levin SR, Desmond JS, Lee J, Aronsky D. Crowding delays treatment and lengthens emergency department length of stay, even among high-acuity patients. Ann Emerg Med 2009; 54:492–503, e4. Guttmann A, Schull MJ, Vermeulen MJ, Stukel TA. Association between waiting times and short term mortality and hospital admission after departure from emergency department: population based cohort study from Ontario, Canada. BMJ 2011; 342:d2983. The National Quality Forum. National Consensus Standards for Emergency Care. A Consensus Report. Available at: http://www.qualityforum.org/ Publications/2009/09/National_Voluntary_Consensus_Standards_for_ Emergency_Care.aspx. [Accessed 27 December 2012]. Freund Y, Vincent-Cassy C, Bloom B, Riou B, Ray P. APHP Emergency Database Study Group. Association between age older than 75 years and exceeded target waiting times in the emergency department: a multicenter
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
Reliability of electronic waiting times in the ED Gorlicki et al. 369
cross-sectional survey in the Paris metropolitan area, France. Ann Emerg Med 2013; 62:449–456. 6 Limkakeng AT Jr, Glickman SW, Shofer F, Mani G, Drake W, Freeman D, et al. Are patients with longer emergency department wait times less likely to consent to research? Acad Emerg Med 2012; 19:396–401. 7 Huang EP, Liu SS, Fang CC, Chou HC, Wang CH, Yen ZS, Chen SC. The impact of adding clinical assistants on patient waiting time in a crowded emergency department. Emerg Med J 2013; 30:1017–1019.
8
Horwitz LI, Bradley EH. Percentage of US emergency department patients seen within the recommended triage time: 1997 to 2006. Arch Intern Med 2009; 169:1857–1865. 9 Locker TE, Mason SM. Are these emergency department performance data real? Emerg Med J 2006; 23:558–559. 10 Keep SL, Locker TE. The impact of a computerised whiteboard system on digit preference bias in the recording of emergency department process times. Eur J Emerg Med 2012; 19:177–180.
Copyright r 2015 Wolters Kluwer Health, Inc. All rights reserved.
