Acta Clinica Belgica International Journal of Clinical and Laboratory Medicine

ISSN: 1784-3286 (Print) 2295-3337 (Online) Journal homepage: http://www.tandfonline.com/loi/yacb20

Agreement on urgency assessment between secretaries and general practitioners: an observational study in out-of-hours general practice service in Belgium H. Philips, J. Van Bergen, L. Huibers, A. Colliers, S. Bartholomeeusen, S. Coenen & R. Remmen To cite this article: H. Philips, J. Van Bergen, L. Huibers, A. Colliers, S. Bartholomeeusen, S. Coenen & R. Remmen (2015) Agreement on urgency assessment between secretaries and general practitioners: an observational study in out-of-hours general practice service in Belgium, Acta Clinica Belgica, 70:5, 309-314, DOI: 10.1179/2295333715Y.0000000017 To link to this article: http://dx.doi.org/10.1179/2295333715Y.0000000017

Published online: 30 Mar 2015.

Submit your article to this journal

Article views: 51

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=yacb20 Download by: [University of California, San Diego]

Date: 25 March 2016, At: 21:13

Original Paper

Agreement on urgency assessment between secretaries and general practitioners: an observational study in out-of-hours general practice service in Belgium H. Philips1, J. Van Bergen1, L. Huibers2, A. Colliers1, S. Bartholomeeusen1, S. Coenen1, R. Remmen1 1

Department of Primary and Interdisciplinary Care, University of Antwerp, Antwerp, Wilrijk, Belgium, Research Unit for General Practice, Aarhus University, Denmark and Scientific Institute for Quality of Healthcare, Radboudumc Nijmegen, the Netherlands

Downloaded by [University of California, San Diego] at 21:13 25 March 2016

2

Background: In some European countries telephone triage (TT) during out-of-hours primary care showed to be safe and effective. Other countries, such as Belgium, may not have trained auxiliary personnel while their national health services want to establish TT. Objectives: To compare urgency levels assessed by secretaries and general practitioners in one general practice cooperative in Belgium. Methods: Percentage of correct-, under-, and over-triage were calculated in total and per reason for encounter. Inter-rater agreement was investigated. Results: The secretaries correctly triaged (same urgency level) 77% of the telephone calls, under-triaged 10% and over-triaged 13%.‘Shortness of breath’, ‘skin cuts’, ‘chest pain’, ‘feeling unwell’ and ‘syncope’ were often under-triaged. Conclusion: Before introducing TT, auxiliary staff should be trained and protocols should be used. Keywords: After hours care, Triage, General practitioners, Family practice, Patient safety

Background Globally, out-of-hours (OOH) primary health care services are under pressure because of an increasing demand in combination with a decreasing workforce because reduction in numbers of general practitioners (GPs) and availability for OOH care. To cope with the challenges of providing safe and affordable care during OOH, new models are being introduced in western countries.1,2 There is a trend towards the establishment of large-scale general practice cooperatives (GPCs) and primary care centres integrated into hospitals’ emergency departments (EDs) in OOH care. Often telephone triage (TT) and advice services by auxiliary personnel are established simultaneously.1–3 Countries with a strong organisation of primary care during regular day-care generally have well established OOH primary care.2–4 However, many countries have a less-structured primary care organisation and lack the infrastructure and availability of

Correspondence to: Annelies Colliers, Department of Primary and Interdisciplinary Care, University of Antwerp, Antwerp, Wilrijk, Belgium. Email: [email protected]

ß Acta Clinica Belgica 2015

well trained auxiliary personnel that underpins quality and efficiency.5,6 Auxiliary personnel can perform TT: answer calls, assess the level of urgency and determine the type of health care required.7 Evidence shows that TT can be a safe and efficient way to handle the majority of primary care OOH calls.7–13 The assessment of urgency and type of health care needed by specially trained, auxiliary personnel often shows to be accurate.7,13 Telephone triage by primary care nurses is efficient in reducing centre consultations and home visits in other countries, because of a high number of telephone consultations with self-care advice, which is of common use in other primary care settings.1,7,8,14 The majority of the cited studies, however, evaluate auxiliary staff performance in countries with well established OOH primary care.3,4,7,12,13,15 Countries with less-structured primary health care, such as Belgium, are also looking for ways to reform OOH care.6,16 Out-of-hours primary care in Belgium is in transition: traditionally GPs provided OOH continuity in small rotation groups, but this is evolving towards large-scale OOH services like GPCs. At the

Downloaded by [University of California, San Diego] at 21:13 25 March 2016

Philips et al.

Agreement on urgency assessment between secretaries and general practitioners

time of study, for *20% of the population, OOH primary care was provided by GPCs. This model is becoming more common. In 2014, *45% of the population is covered by GPCs. General practice cooperatives are only supported by administrative assistance but no paramedical personnel. At present, medically trained auxiliary staff for OOH primary care is nonexistent in Belgium, as neither there is a formal training available, nor is it an officially recognised profession. However, the central government has set directives to introduce TT during OOH care. Protocols to assess the level of urgency have been developed and implementation is planned. This study sets out to establish a baseline of how accurate urgency levels are assessed by secretaries at this moment, i.e. before the implementation of their education and the use of protocols. In addition, this study examines a possible mismatch of the urgency levels allocated to telephone calls by secretaries and GPs at one GPC, using the percentage of reasons for encounter (RFEs) that were under-triaged or overtriaged by the secretaries.

Methods Study design This observational, explorative study examined how accurate untrained secretaries at one OOH GPC allocated urgency levels to telephone calls and compared this to the urgency levels of the GP after the face-toface contact with the patient. This comparison enables to estimate safety and efficiency of call handling by untrained secretaries, using percentages of correct-, under- and over-triage.

Setting This study was performed in one OOH GPC (Bruges, Belgium), serving *135 000 inhabitants. At the time 106 GPs worked in the GPC assisted by 5 secretaries. The GPC provides OOH care on weekends and public holidays. The on-call GPs are assisted by secretaries, who answer the telephone calls and record administrative information from the caller. The GPC secretaries in Belgium receive no specific training for TT and have no medical background. Therefore, they are instructed to pass the call to the supervising GP in case of any doubt or a suspected medical emergency. Legally, the GP on duty has final responsibility for the correct call handling. To date, telephone consultations by auxiliary personnel or by the GP are not permitted in Belgium.

Data All telephone calls on weekends and public holidays between 8 a.m. and 21 p.m.to the GPC of the entire year of 2010 were included. The secretaries were asked to assess the urgency level of the call and to record the RFE as an additional registration. The

registration of urgency and RFE started 4 months before the study period and therefore, the secretaries could get used to the additional work. They did not receive additional training or protocols. To estimate urgency a scale based on the Manchester Triage System (MTS) and the Netherlands Triage Standard (NTS) was developed, using five urgency levels (U1–U5) (Table 1).17,18The same scale will be used in future when central dispatching of medical calls will be implemented in Belgium. To record the RFE, a predefined list with the 26 most frequent RFEs was provided in the registration software (Wachtpost Mailer, VOF Brouns & Co, Deurne, Belgium). This list was based on data from other GPCs in Belgium using the same registration software and represented 80% of all OOH primary care contacts. One category was added (‘RFE not present in the list’) in case the RFE could not be categorised according to the list. After the consultation or home visit, the GP on duty also assessed the urgency as an obligatory part of the medical report. The GPs were blinded to the urgency assessment of the secretaries. The GPs’ assessments were regarded as the best reference to compare with the secretaries initial assessment and used as a ‘silver standard’.19 Thus, the data set consisted of the RFE and the urgency assessment by the secretary at the call handling, and the urgency assessment by the GP after the face-to-face contact with the patient. This enabled us to make a rough estimation of safety and efficiency, in order to concentrate on the RFEs where the largest amount of underestimation (possible safety problem) appears. Being an explorative study, for legal and safety reasons, the assessment of the urgency did not influence or change routine call handling or patient management.

Analysis The comparison of urgency assessment by the secretaries and the GPs’ assessment resulted in three triage outcomes. ‘Correct-triage’ was a match between the urgency assessment by the secretary and the GP on duty. ‘Undertriage’ referred to a lower urgency assessment by the secretary compared to the GP. ‘Over-triage’ referred to a more urgent assessment by the secretary compared to the GP. Under-triage is an indicator for safety, whereas over-triage is an indicator for efficiency.19 The triage outcomes were assessed for all patients seen by the GPs as well as by RFE assigned by the secretaries. Crosstabs were created. Crosstabs were created comparing the assessment of secretaries and GPs. A table was constructed for triage outcome versus RFEs (ICPC2).To assess the inter-rater reliability between the estimation of the secretaries and the GPs, the weighted kappa was calculated. Data was analysed using SPSS statistics version 18.0

Philips et al.

Agreement on urgency assessment between secretaries and general practitioners

Table 1 Levels of urgency for telephone calls as defined for the secretaries Levels of urgency

Label

Description

U1

Life-threatening

U2

Urgent

U3

Acute

U4 U5

Routine Not urgent

To provide help immediately, consider alerting an ambulance at the same time; e.g. shock, unconsciousness To provide help as soon as possible; certainly v1 hour. There is a chance the condition will worsen at short term; e.g. unstable angina pectoris To be assessed in the next couple of hours, e.g. suspected pneumonia, severe vertigo Normal use of on-call service, business as usual This complaint could have waited until the next (working) day. No reason to contact the GP on-call.

Downloaded by [University of California, San Diego] at 21:13 25 March 2016

Scale used for the assessment of urgency by both the GPC secretaries and the GPs. The scale was based on MTS and NTS. GP: general practitioner; GPC: general practice cooperative; MTS: Manchester Triage System (MTS); NTS: the Netherlands Triage Standard.

(SPSS Inc., Chicago, Illinois, USA). Weighted kappa calculation was calculated as described by Cohen,20 using the psych add-on package in the statistical software package R (version 3.1.0).

Results Contacts A total of 5628 telephone calls were handled by the five GPC secretaries during the study period in 2010. In 1499 calls (26.6%) the RFE was missing, and in 112 (2.0%) calls no urgency level was given by the GP. Therefore, 4017 calls were included for analysis. The ‘RFE not present in the list’ code was given in 823 calls (20.5%). A total of 106 GPs participated in the study.

Urgency Table 2 shows the urgency assessment by the secretaries compared to the urgency assessment by the GPs. The secretaries never used urgency levels U1 or U2, while 154 calls (2.8%) were graded U3, 5327 calls (96.6%) U4 and 35 calls (0.6%) U5. In the GPs’ assessments, more differentiation was observed, with 12 calls (0.2%) graded U1, 60 calls (1.1%) U2, 531 calls (9.6%) U3, 4271 calls (77.4%) U4 and 642 calls (11.6%) U5. Overall secretaries showed 77% correcttriage, 10% under-triage and 13% over-triage. All calls assessed as U1 or U2 by the GPs (1.3%, n572) were under-triaged by secretaries. Secretaries

assessed 53 of them (73.6%) as U4. In addition, 484 out of 531 (91.2%) U3 calls, as assessed by the GPs, were under-triaged by the secretaries. The majority of U4 calls (n54271) according to the GPs were triaged correctly (97.6%), while almost all (98.1%) of the U5 calls (n5642) according to the GPs’ assessments were over-triaged by the secretaries. For assessment of the inter-rater reliability of the urgency levels the weighted kappa between the urgency assessment of the secretaries and the GPs was estimated to be 0.136, with a 95% confidence interval ranging from 0.102 to 0.17.

Reasons for encounter Table 3 shows the percentages for correct-triage, under-triage and over-triage by RFEs, as provided by the secretaries. Highest frequencies of correcttriage were found for ‘tooth ache’ (95%), ‘itch’ (90%), ‘ear ache’ (88%), ‘eye problems or complaints’ (85%) and ‘diarrhoea’ (84%). The most frequently under-triaged RFEs were ‘shortness of breath’ (28%), ‘skin cuts’ (27%), ‘chest pain’ (23%), ‘feeling unwell’ (22%) and ‘syncope’ (18%). The following RFE were over-triaged most frequently: ‘medication prescriptions, requests and renewals’ (45%), ‘issuance of a medical certificate’ (38%), ‘chest pain’ (32%), ‘animal or human bite’ (25%) and ‘syncope’ (18%).

Table 2 Urgency level assessment of the GPC secretaries compared to the urgency level assessment by the GPs Urgency level by the GP

Urgency level by the GPC secretary

U1 U2 U3 U4 U5 Total

U1

U2

U3

U4

U5

Total

0 (0%) 0 (0%) 3 (0.1%) 9 (0.2%) 0 (0%) 12 (0.2%)

0 (0%) 0 (0%) 16 (0.3%) 44 (0.8%) 0 (0%) 60 (1.1%)

0 (0%) 0 (0%) 47 (0.9%) 481 (8.7%) 3 (0.1%) 531 (9.6%)

0 (0%) 0 (0%) 83 (1.5%) 4168 (75.6%) 20 (0.4%) 4271 (77.4%)

0 (0%) 0 (0%) 5 (0.1%) 625 (11.3%) 12 (0.2%) 642 (11.6%)

0 (0%) 0 (0%) 154 (2.8%) 5327 (96.6%) 35 (0.6%) 5516 (100%)

The figures in bold in the shaded cells represent the correct-triaged calls. In the cells below the shaded cells, the undertriaged calls are found (potentially unsafe); in the cells above the shaded cells, the over-triaged calls are found (potentially inefficient).U1: life-threatening; U2: urgent; U3: acute; U4: routine; U5: not urgent; GP: general practitioner; GPC: general practice cooperative.

Philips et al.

Agreement on urgency assessment between secretaries and general practitioners

Table 3 Percentages of correct-, under- and over-triaged calls for all the different RFE, with corresponding ICPC2 code, by the secretaries

Downloaded by [University of California, San Diego] at 21:13 25 March 2016

GPC secretary RFEs

Correct-triage (%)

Under-triage (%)

Over-triage (%)

Total

RFE not present in the list Fever (A03) Throat ache (R21) Cough (R05) Vomiting (D10) Abdominal pain (D01) Ear ache (H01) Feeling unwell (A05) Diarrhoea (D11) Headache (N01) Lumbar pain (L03) Shortness of breath (R02) Dizziness/vertigo (N17) Dysuria (U01) Eye problems/complaints (F29) Stomach ache (D02) Nausea (D09) Skin cut (S18) Chest pain (K02) Insect bite (S12) Declaration of death (A96) Medication prescription/request/renewal (A50) Tooth ache (D82) Itch (S02) Syncope (A06) Issuance of a medical certificate (A97) Animal or human bite (S13) Total valid RFEs Missing Overall total

73 83 83 77 78 72 88 66 84 83 79 60 73 82 85 76 79 70 46 80 56 52 95 90 64 50 75

12 7 3 7 11 17 2 22 6 3 8 28 9 10 6 10 5 27 23 13 7 3 0 0 18 13 0

15 9 14 16 11 12 10 13 11 13 13 12 18 8 9 13 16 4 32 7 4 45 5 11 18 38 25

75 77

12 10

13 13

823 (20%) 817 (20%) 247 (6%) 223 (6%) 203 (5%) 191 (5%) 188 (5%) 152 (4%) 153 (4%) 157 (4%) 136 (3%) 108 (3%) 88 (2%) 79 (2%) 78 (2%) 67 (2%) 58 (1%) 56 (1%) 44 (1%) 30 (1%) 27 (1%) 29 (1%) 21 (1%) 19 (1%) 11 (v1%) 8 (v1%) 4 (v1%) 4017 (100%) 1499 5516

The total absolute number and the percentage of RFEs are also presented.GPC: general practice cooperative; RFE: reasons for encounter.

Discussion Statement of principal findings In our study, 10% of the calls to a GPC during OOH were under-triaged and 13% over-triaged by the secretaries assessing urgency at call handling as compared to the assessment of the GP after the face-to-face contact with the patient. In addition, specific RFEs could be identified that potentially present safety and/or efficiency issues.

Interpretation of the results Secretaries nearly always assessed calls to be U4 or ‘routine on call use’. They under-triaged 10% and over-triaged 13% of the calls to a GPC. Other studies have investigated the percentage of correct TT in OOH primary care, resulting in a range from 58% to 97% correct assessment of urgency 7,9,10,12,13,15,21,22 levels. Under-triage in these studies varied from 7.1% to 41%.12,13 However, these figures originate from countries with a well established role of auxiliary personnel in primary care. The methodology on used triage scales, time span and setting between both assessments of urgency and education of personnel differs largely between studies and with our setting as well. Results have to be compared with caution, although the broad range of

percentages confirms the difficulty in estimation of urgency, independent from the setting. The probability of urgent medical situations, in a European study, was estimated about 5%.23 This means that secretaries can perform well when assessing calls mainly as U4, but the risk to under-triage and hence miss an important medical situation is also realistic. If the authors consider that in 1 year time 72 patient contacts that were potentially lifethreatening (U1) or urgent enough to provide help as soon as possible (U2) were not recognised by the administrative staff of a GPC covering a region of 135 000 inhabitants, the introduction of TT without training would hamper patient safety. Although this might be an overestimation of the real safety problem (secretaries manage the first contact, without seeing or knowing the patient, compared to the GP, whose urgency estimation is made after the patient contact) the results enable us to define possible dangerous RFEs; ‘shortness of breath’, ‘skin cuts’, and ‘chest pain’ were under-triaged and should typically warrant caution in telephone contact with patients. The most over-triaged RFEs typically present administrative reasons (i.e. ‘medication prescriptions, requests and renewals’, and ‘issuance of a medical certificate’).

Downloaded by [University of California, San Diego] at 21:13 25 March 2016

Philips et al.

The poor agreement identifies relevant issues to improve efficiency. Typically, efficient TT would have filtered out and redirected these administrative problems to regular day-care. The 642 patient contacts in 11year time that could have waited until the next day illustrate the efficiency gain which could be attained when using TT in this region of 135 000 inhabitants. Part of the disagreement between secretaries and GPs is because of unfamiliarity with the concept of TT in Belgium. At present, the secretaries are neither trained nor mandated to give any advice and have been instructed to give every patient an appointment. This may explain that for them nearly all calls are considered routine and assessed as U4. As the triage outcome was not used in the subsequent patient management, this might have led to less motivation for the secretaries to perform well. A study by Li et al.24 in 2002 already stated that a prudent lay person definition may not be an ideal standard by which to judge ambiguous symptoms that could represent serious underlying illness. This statement is confirmed by our study. However, the results remain relevant to the Belgian health care system as the Federal Government will initiate TT. The authors showed that if the secretaries are not trained and have no support (e.g. protocols), this is potentially unsafe and may not increase efficiency of the use of OOH care.

Strengths and limitations of the study This study is the first to describe the match of urgency assessment by secretaries in call handling as compared to GPs after the face-to-face contact with the patient in a country with limited level of organisation of primary care. The use of our predefined list narrowed down the number of possible RFEs. As less frequent RFEs might present triage problems, this is a possible bias. Similar to multiple studies, the authors used the opinion of the GP after the face-to-face consultation, a silver standard was used as a reference for urgency.19 This method assumes that GPs’ assessments are correct. However, as ‘urgency’ is not an objective parameter, no gold standard is available. In our study, GPs on duty had more information to make an assessment because they completed a face-to-face contact, there by providing the best approximation to the callers’ health status. This design has a potential bias because of different moments in the care process of the estimation of the urgency.

Possible explanations and implications for clinicians and policymakers This study confirms the vulnerability of TT as first access to OOH care in a country with limited organisation of primary care.13 The limited safety problems

Agreement on urgency assessment between secretaries and general practitioners

of TT encountered in the Netherlands and Norway are well studied. The authors learn from this study that especially high-urgency calls are most at risk to be of lower quality and to have higher probability to be under-triaged.13,15 Appropriate triage, however, in these countries varies between 78% and 97%. This can most probably be explained by the highly organised structure of OOH primary care settings: nurses answering the telephone and using clearly defined TT protocols. Currently, both are absent in Belgium. In several pilot projects, the Federal Government aims to redirect the patient flow during OOH through the introduction of auxiliary personnel in TT. This small pilot study fits in a larger project aiming to implement a single-access telephone number for all OOH medical services. Our results show that training auxiliary staff specifically for OOH primary care TT seems urgently needed. Furthermore, this personnel needs guidelines or protocols, as this has shown to improve quality.10 In addition, as there is a significant correlation between correct estimation of urgency and specific training on the use of telephone guidelines, this should be introduced as well.9

Unanswered questions and future research More research is needed on OOH primary care TT in systems with limited organisational structure of primary health care, as is the case in many countries. Interventions aimed at education of the current staff or at introduction of trained staff should be considered. Once newly constructed OOH primary care TT protocols are introduced, these should be continuously evaluated for safety and efficiency. In addition, the training of the call takers should be evaluated.

Conclusion This study identified opportunities to improve safety and efficiency of current telephone call handling in OOH primary care in a health care system where OOH primary care is in transition. When considering the introduction of TT in recently established largescale GPCs, specially trained staff and triage guidelines should be introduced because current secretaries suboptimally assesses urgency of calls.

Disclaimer Statements Contributors PH, VBJ, RR and HL were involved in the concept and design of the study. PH and VBJ were responsible for the acquisition of data. PH, VBJ, CA, BS, CS and RR carried out the analysis and interpretation of the data. PH, VBJ, HL, CA, BS, CS and RR revised and helped to draft the manuscript. Funding None.

Philips et al.

Agreement on urgency assessment between secretaries and general practitioners

Conflict-of-interest The authors declare that there are no conflicts of interests. Ethics approval This study was approved by the ethical committee of the University of Antwerp and the Antwerp University Hospital (reference B300201112096).

Downloaded by [University of California, San Diego] at 21:13 25 March 2016

Acknowledgements The authors would like to acknowledge the particular contribution of Johan Brouns (for his IT support of the database and writing the queries for this study), Ilse Druwe´, Koen Steel, Philip Vandevelde and Piet Vermeire (for their involvement in the study design and to motivate and support all participants, secretaries and GPs). The authors would also like to thank all of our registering secretaries and GPs.

References 1 Leibowitz R, Day S, Dunt D. A systematic review of the effect of different models of after-hours primary medical care services on clinical outcome, medical workload, and patient and GP satisfaction. Fam Pract. 2003;20(3):311–7. 2 Grol R, Giesen P, van Uden C, Grol R, Giesen P, van Uden C. After-hours care in the United Kingdom, Denmark, and the Netherlands: new models. Health Aff (Millwood). 2006;25(6):1733–7. 3 Huibers L, Giesen P, Wensing M, Grol R. Out-of-hours care in western countries: assessment of different organizational models. BMC Health Serv Res. 2009;9(1):105. 4 Giesen P, Smits M, Huibers L, Grol R, Wensing M. Quality of after-hours primary care in the netherlands: a narrative review. Ann Intern Med. 2011;155(2):108–13. 5 Boerma WG, van der Zee J, Fleming DM. Service profiles of general practitioners in Europe. European GP task profile study. Br J Gen Pract. 1997;47(421):481–6. 6 Remmen R, Seuntjens L, Paulus D, Pestiaux D, Knops K, Bruel AV. Can we import quality tools? a feasibility study of European practice assessment in a country with less organised general practice. BMC Health Serv Res. 2009;9:183. 7 Huibers L, Smits M, Renaud V, Giesen P, Wensing M. Safety of telephone triage in out-of-hours care: a systematic review. Scand J Prim Health Care. 2011;29(4):198–209. 8 Bunn F, Byrne G, Kendall S. Telephone consultation and triage: effects on health care use and patient satisfaction. Cochrane Database Syst Rev. 2004;4:CD004180.

9 Giesen P, Ferwerda R, Tijssen R, Mokkink H, Drijver R, Bosch W. Safety of telephone triage in general practitioner cooperatives: do triage nurses correctly estimate urgency? Qual Saf Health Care. 2007;16:181–4. 10 Marklund B, Stro¨m M, Ma˚nsson J, Borgquist L, Baigi A, Fridlund B. Computer-supported telephone nurse triage: an evaluation of medical quality and costs. J Nurs Manag. 2007;15(2):180–7. 11 Car J, Koshy E, Bell D, Sheikh A. Telephone triage in out of hours call centres. BMJ. 2008;337:a1167. 12 Derkx HP, Rethans JJ, Muijtjens AM, Maiburg BH, Winkens R, van Rooij HG, et al. Quality of clinical aspects of call handling at Dutch out of hours centres: cross sectional national study. BMJ. 2008;337:a1264. 13 Huibers L, Keizer E, Giesen P, Grol R, Wensing M. Nurse telephone triage: good quality associated with appropriate decisions. Fam Pract. 2012;29(5):547–52. 14 Kaminsky E, Carlsson M, Ho¨glund AT, Holmstro¨m I. Paediatric health calls to Swedish telenurses: a descriptive study of content and outcome. J Telemed Telecare. 2010;16(8):454–7. 15 Hansen EH, Hunskaar S. Telephone triage by nurses in primary care out-of-hours services in Norway: an evaluation study based on written case scenarios. BMJ Qual Saf. 2011;20(5):390–6. 16 Jonckheer P, Borgermans L, Dubois C, Verhoeven E, Rinchard E, Baudewyns A, et al. After-hours in primary care: which solutions? . Brussels: Belgian Health Care Knowledge Centre (KCE); 2011. 17 Manchester Triage Group. Emergency triage. In: Mackway-Jones K, Marsden J, Windle J, editors. London: BMJ Publishing Group; 1997. 18 van Ierland Y, van Veen M, Huibers L, Giesen P, Moll HA. Validity of telephone and physical triage in emergency care: the Netherlands triage system. Fam Pract. 2011;28(3):334–41. 19 Moll HA. Challenges in the validation of triage systems at emergency departments. J Clin Epidemiol. 2010;63(4):384–8. 20 Cohen J. Weighted Kappa - nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213–20. 21 Hildebrandt DE, Westfall JM, Smith PC. After-hours telephone triage affects patient safety. J Fam Pract. 2003;52(3):222–7. 22 Meer A, Gwerder T, Duembgen L, Zumbrunnen N, Zimmermann H. Is computer-assisted telephone triage safe? A prospective surveillance study in walk-in patients with nonlife-threatening medical conditions. Emerg Med J. 2012;29(2):124–8. 23 Huibers LA, Moth G, Bondevik GT, Kersnik J, Huber CA, Christensen MB, et al. Diagnostic scope in out-of-hours primary care services in eight European countries: an observational study. BMC Fam Pract. 2011;12:30. 24 Li J, Galvin HK, Johnson SC. The ‘‘prudent layperson’’ definition of an emergency medical condition. Am J Emerg Med. 2002;20(1):10–13.

Agreement on urgency assessment between secretaries and general practitioners: an observational study in out-of-hours general practice service in Belgium.

In some European countries telephone triage (TT) during out-of-hours primary care showed to be safe and effective. Other countries, such as Belgium, m...
392KB Sizes 2 Downloads 13 Views