G Model
NS-2348; No. of Pages 8 International Journal of Nursing Studies xxx (2014) xxx–xxx
Contents lists available at ScienceDirect
International Journal of Nursing Studies journal homepage: www.elsevier.com/ijns
Review
A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System Nicola Parenti a,*, Maria Letizia Bacchi Reggiani b, Primiano Iannone c, Daniela Percudani d, Dawn Dowding e a
University of Parma, Parma, Italy Biostatistic, University of Bologna, Italy Department of Emergency Medicine of Hospital Lavagna, Genova, Italy d Administrative, University of Parma, Italy e Columbia University School of Nursing and Visiting Nursing Service of New York, New York, USA b c
A R T I C L E I N F O
A B S T R A C T
Article history: Received 18 July 2013 Received in revised form 20 November 2013 Accepted 24 January 2014
Objective: To conduct a systematic review to check the level of validity and reliability of the Manchester Triage System and the quality of reporting of literature on this topic. Design: This is a systematic review based on the PRISMA guideline on reporting systematic reviews. Data sources: The systematic search of the international literature published from 1997 through 30 November 2012 in the PubMed, Embase, Cochrane Library, Cinahl, Web of Knowledge, and Scopus databases. Review methods: This review included quantitative and qualitative research investigating the reliability and validity of the Manchester Triage System for the broad population of adults and children visiting the emergency department. After a systematic selection process, included studies were assessed on their quality by three researchers using the STARD guidelines. Results: Twelve studies were included in the review. The studies investigated the interand intra-rater reliability using the ‘‘kappa’’ statistic; the validity was tested with many measures: validity in predicting mortality, hospital admission, under- and overtriage, used resources, and length of stay in the emergency department, as well as a reference standard rating. Conclusions: In this review, the Manchester Triage System shows a wide inter-rater agreement range with a prevalence of good and very good agreement. Its safety was low because of the high rate of undertriage and the low sensitivity in predicting higher urgency levels. The high rate of overtriage could cause unnecessarily high use of resources in the emergency department. The quality of the reporting in studies of the reliability and validity of the Manchester Triage System is good. ß 2014 Elsevier Ltd. All rights reserved.
Keywords: Triage Emergency care Emergency medical service Literature review Systematic review
* Corresponding author at: Via San Vitale 96, Bologna Zip Code 40125, BO, Italy. Tel.: +39 0519912065; fax: +39 0519921849. E-mail address: [email protected] (N. Parenti). 0020-7489/$ – see front matter ß 2014 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ijnurstu.2014.01.013
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 2
N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx
What is already known about the topic? The Manchester Triage System (MTS) is a widespread inhospital triage system. The main quality indicators of a triage scale are the validity and reliability. Only recent studies tested MTS validity and reliability with different conclusions. What does this paper adds The Manchester Triage System seems to have a wide inter-rater agreement range and there are limited data on its intra-rater reliability. There was a wide range of results for validity measures among studies included. It is likely the triage scale could be improved in its safety. The quality of reporting among studies on this topic is good. 1. Introduction Triage is the first assessment and sorting process used to prioritize patients arriving in the emergency department (ED). The need to prioritize these patients is stressed by the considerable demand for emergency care, frequent ED overcrowding, and limited resources. For this reason many triage scales were developed in the last twenty years. Most current triage tools actually in use are based on a categorical measurement acuity scale and have five levels. The Australasian Triage Scale, ATS (Standards Committee Council, 1994); the Canadian Triage and Acuity Scale, CTAS (Beveridge, 1998); the Manchester Triage System, MTS (Mackway-Jones, 1997); and the Emergency Severity Index, ESI (Eitel et al., 2003; Tanabe et al., 2004; Wuerz et al., 2000, 2001), are all five-level triage tools. Many studies have evaluated the reliability and validity of acuity ratings by a triage nurse (Eitel et al., 2003; Parenti et al., 2009; Taboulet et al., 2009; Van der Wulp, 2008; Worster et al., 2007), probably because a triage scale should meet at least these two criteria to perform as intended (Twomey et al., 2007). In fact, according to most triage experts, (Fernandes et al., 2005) the ‘‘ideal’’ triage scale must demonstrate the characteristics of reliability and validity. For this reason we decided to conduct a systematic review on the state of studies on the reliability and validity of the Manchester Triage Scale, as MTS is one of the mostused triage scales in Europe. To our knowledge there are few reviews on all triage scales actually in use (Christ et al., 2010; Farrohknia et al., 2011; Van Veen and Moll, 2009), but there are no systematic reviews on the validity and reliability of the Manchester Triage System. A previous review on triage methods suggested that the reliability, validity, and safety of triage scales needed to be investigated further (Farrohknia et al., 2011). Van Veen and Moll (2009) concluded that MTS and PaedCTAS seem valid to triage children in emergency care, as well as that reliability of MTS is good, moderate-to-good for ESI, moderate for PaedCTAS, and poor to moderate for ATS.
2. Background The Manchester Triage System is an in-hospital triage scale usually used by nurses in many emergency departments in Great Britain and Europe since 1997 (Mackway-Jones, 1997; Martins et al., 2009; Olofsson et al., 2009; Van Baar et al., 2007). It is based on 52 flowchart diagrams; each of the flow charts depicts six key discriminators (Mackway-Jones, 1997). MTS considers five triage priority levels: level 1, immediate assessment; level 2, very urgent; level 3, urgent; level 4, standard; level 5, non-urgent. In recent years, many studies have been published on MTS reliability and validity. There is an inter-rater reliability and intra-rater reliability for triage systems. They are usually analyzed using the k statistic (Cohen’s kappa). The inter-rater reliability refers to the statistical measurement of agreement obtained by two or more users of the scale. It measures the agreement beyond chance between raters. Kappa equals zero when the amount of agreement is what would be observed by chance alone, and k is equal to one when there is a perfect agreement. Kappa levels are frequently evaluated with the following terminology (Landis and Koch, 1977): poor agreement, k less than 0.20; fair agreement if k is from 0.20 to 0.40; moderate agreement, k from 0.40 to 0, 60; good agreement, k from 0.60 to 0.80; very good agreement, k from 0.80 to 1. The MTS inter-rater reliability is determined by the agreement in triage urgency level if multiple raters triage one patient or patient scenario. The intra-rater agreement presents the agreement in triage urgency level if one triage nurse triages one case scenario at different points in time. Validity refers to the agreement between the value of a measurement and its true value; a triage scale is valid if it measures what it is supposed to measure: the ‘‘true urgency.’’ To assess the triage systems’ validity, a ‘‘gold standard’’ as a proxy for urgency has to be defined. Many outcome measures and reference standard for urgency have been used. The MTS validity can be expressed in sensitivity and specificity. MTS sensitivity presents its ability to identify high-urgency patients; MTS specificity presents its ability to identify patients with low-urgency problems. 3. Methods 3.1. Aim The primary aim was to check the state of studies on the reliability and validity of the Manchester Triage Scale for the broad population of adults and children visiting the ED. The questions for the review were: (1) What is the level of reliability of MTS among the selected studies? (2) How valid is MTS in predicting outcomes and workload among the selected studies? (3) How is the quality of reporting among published studies on reliability and validity of MTS?
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx
3.2. Review protocol
Included
Eligibility
Screening
Identification
This is a systematic review with a narrative synthesis of the main findings on the reliability of MTS. The review is based on the PRISMA guideline on reporting systematic reviews (Liberati et al., 2009) and on the STARD (Standard for Reporting of Diagnostic Accuracy) guidelines (Bossuyt et al., 2003). We used the PRISMA guideline for the first part of the review protocol: the selection of studies. The PRISMA guideline is an evidence-based minimum set of items for reporting in systematic reviews and metaanalyses. Its aim is to help authors improve the reporting of systematic reviews and consists of a 27-item checklist and a four-phase flow chart. Then we used the STARD guidelines to analyze the quality of reporting among studies selected. The STARD guidelines are used to improve the accuracy and completeness of reporting of studies of diagnostic accuracy, to allow readers to assess the potential for bias in the study and to evaluate its generalizability. The STARD guidelines consist of a checklist of 25 items and recommend the use of a flow chart that describes the design of the study and the flow of patients (Bossuyt et al., 2003). The selection of articles included in the review was performed in a three-phase process (Fig. 1) according to PRISMA guideline. In the first phase, one author, an expert in literature research, conducted a literature search of the following databases: PubMed, Embase, Cochrane Library, Cinahl, Web of Knowledge, and Scopus. In the second phase, three researchers independently performed a screening by regarding eligibility criteria of the six lists of articles (title and abstract) selected in phase one. All duplicates were removed. In this way 353 potentially useful articles were selected with their full text. In the third phase, the three researchers
380 records identified through database searching
3
independently examined all 353 full-text articles to select the studies that met the inclusion criteria for the systematic review. They had to justify the reasons for exclusion of studies in this phase. After this phase 15 articles were included, but then three of those articles were excluded with reasons: one study was a descriptive analysis of an implementation project of triage in one ED with MTS; two studies collected the literature’s findings on the reliability and validity of MTS. The 12 studies remaining after this phase were included in the analysis. Three researchers independently examined all 12 full-text articles according STARD guidelines. 3.3. Search methods We included all studies on reliability and validity of MTS conducted on all ages of patients in all languages. We excluded only the duplicates studies. The systematic search of the international literature published from 1997 through 30 November 2012 explored the databases of PubMed, Embase, Cochrane Library, Cinahl, Web of Knowledge, and Scopus. The following key words were used for the literature search: For PubMed ‘‘manchester triage’’ [all fields] OR ‘‘Manchester Triage System’’ [all fields] OR ‘‘manchester triage protocol’’ [all fields] OR ‘‘manchester triage scale’’ [all fields] OR ‘‘manchester triage method’’ [all fields] OR (Manchester [Ti,Ab] AND triage[Ti,Ab]) OR (triage[MeSH Terms no explode] AND Manchester[all fields]), 97 citations; For EMBASE: (Manchester AND triage [Ti,Ab]) OR (Emergency health service [EmTree no explode] AND Manchester[Ti,Ab]), 95 citations;
51 additional records identified through other sources
78 duplicate records removed
338 records excluded 353 records screened
15 full-text articles assessed for eligibility
3 full-text articles excluded, with reasons
12 studies included in qualitative synthesis and quality appraisal
Fig. 1. Review process.
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx
4
For Cinahl (Manchester AND triage [All fields]) OR (‘‘Triage’’[MH] AND (MTS OR Manchester[all fields]), 55 citations; For Web of Knowledge and Scopus: Manchester [all fields] AND triage [all fields], Web of Knowledge 52 citations and Scopus 81 citations; For Cochrane Library: Triage [all fields], no systematic reviews for Manchester triage.
estimation of the quality of the methodology of the studies (statistics, design of study). The three reviewers’ yes/no agreement for each study were entered into an Excel 2010 (Microsoft Corporation) spreadsheet, and the statistics Fleiss’ kappa for observed agreement was performed. We obtained a Fleiss’ kappa score of k = 0.72, equating to a ‘‘substantial’’ level of agreement between the raters (Landis and Koch, 1977).
3.4. Quality assessment
4. Results
Three authors conducted an appraisal of the reporting quality of the 12 studies included. They used the STARD guidelines because they considered the triage scales similar to diagnostic tools. The quality assessment was conducted in a two-phase process. In the first phase the researchers checked how many items of the STARD guidelines were met by each study. A STARD item was considered met when two of the three authors considered it in this way. To have standardization in the data extraction, we used the same checklist derived from the STARD guidelines (Bossuyt et al., 2003). A narrative summary was used to synthetize the data to provide a description and ordering of the evidence with commentary and interpretation. In the second phase the three researchers gave their personal opinion on the quality level of each study included. They used a three-grade scale: low quality (+/ +++), moderate quality (++/+++), and high quality (+++/ +++). The ‘‘overall quality evaluation’’ was a subjective
We collected 12 studies for final analysis (Table 1). Three studies met fewer than 50% of the STARD items and had a very low-quality methodology in the opinion of the authors (Table 2). None of the studies selected met all 25 items of the STARD guidelines (Table 2). Five of the nine studies met more than 80% of the STARD items. The study that met the greatest number of items on the STARD checklist was that of Van der Wulp et al. (2008) (Table 2). In the opinion of authors, the Van Veen et al. (2008) and Storm et al. (2011) papers reach a high grade of quality and a good performance in reporting according to STARD guidelines (Table 2). Of the 12 studies, six tested validity of MTS, four tested reliability, and two tested both indicators (Table 1). Four of the 12 were on pediatric population. There were one audit observational study, seven retrospective studies, and four prospective studies. Nine studies included were published after 2006 in northern Europe (Netherlands and Sweden).
Table 1 Characteristics of studies selected for review. Author, Year, Reference, Country
Study design
Patient Characteristics
Drop out
Outcome (Reliability/Validity)
Goodacre et al. (1999), Australia Roukema et al. (2006), Netherlands
Audit observ
0
Reliability K inter statistic
18,469
Van Veen et al. (2008), Netherlands
Prospective
50 scenarios; 4 reviewers Unknown patients characteristic 1065 pat: Male: 55.2%; Age (yrs, mean): 4.2 Pediatric population 16,735 pat: Age: 3.4 yrs; Female: 42% Pediatric population
Van der Wulp et al. (2008), Netherlands
Prospective
Van der Wulp et al. (2009), Netherlands Olofsson et al. (2009), Sweden
Retrospective
Grouse et al. (2009), Australia Martins et al. (2009), Portugal Storm-Versloot et al. (2009), Netherlands Van Veen et al. (2010), Netherlands Seiger et al. (2011), Netherlands Storm-Versloot (2011), Netherlands
Retrospective Retrospective Retrospective
Validity in predicting resource utilization, hospitalization and reference standard rating; Sensitivity/Spec; Over/undertriage Validity: Sens/Spec/LH ratio for rating respect to a RS Over/undertriage Reliability: K inter (unweighted and weighted) Validity: Sens/Spec for urgent pat; Over/ undertriage Respect to a reference standard Validity in predicting admission and mortality LH ratio (OR) and mortality rate Reliability with K inter (unweighted and weighted and quadratic weighted) Validity with accuracy in predicting a RS triage code; Over/undertriage Reliability with K inter (weighted) Validity: Rate of mortality and admission Reliability: K inter (unweighted and quadratic weighted) Reliability: K inter (weighted and quadratic weighted) Validity: Undertriage respect a RS tested with logistic regression (OR) Validity: undertriage, sens/spec respect a RS; relation between resource use/hospitalization/ length of stay and urgency level
Retrospective
Retrospective
Retrospective Prospective Prospective
50 scenarios; 55 nurses (+2RS); Unknown pat. charact; Mean nurses’ experience in ED: 7–15 yrs 34,258 pat; Age (yrs, mean): 42.4; Male: 51.6% 13 scenarios; 79 raters
50 scenarios; 20 raters 316,622 pat 50 scenarios; 12 raters Information on rater 20 scenarios; 43 raters Pediatric population 13,408/17,600 Pediatric population 872/890; Age (yrs, mean) = 37; Male: 53%
5%
26.4%
4917
4192 18
LH, Likelihood; RS, reference standard; Sens, sensitivity; Spec, specificity.
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx Table 2 Quality assessment of studies selected for review. Author (year)
STARD items Methodology respected % quality evaluation (n)
Outcome
Goodacre et al. (1999) Roukema et al. (2006)
72% (18/25) 84% (21/25)
Reliability Validity
Van Veen et al. (2008) Van der Wulp et al. (2008) al. (2009) Olofsson et Van der Wulp et al. (2009) Grouse et al. (2009) Storm-Versloot et al. (2009) Martins et al. (2009) Van Veen et al. (2010) Storm-Versloot (2011) Seiger et al. (2011)
84% (21/25) 92% (23/25) 76% (19/25)
NS-2348; No. of Pages 8 International Journal of Nursing Studies xxx (2014) xxx–xxx
Contents lists available at ScienceDirect
International Journal of Nursing Studies journal homepage: www.elsevier.com/ijns
Review
A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System Nicola Parenti a,*, Maria Letizia Bacchi Reggiani b, Primiano Iannone c, Daniela Percudani d, Dawn Dowding e a
University of Parma, Parma, Italy Biostatistic, University of Bologna, Italy Department of Emergency Medicine of Hospital Lavagna, Genova, Italy d Administrative, University of Parma, Italy e Columbia University School of Nursing and Visiting Nursing Service of New York, New York, USA b c
A R T I C L E I N F O
A B S T R A C T
Article history: Received 18 July 2013 Received in revised form 20 November 2013 Accepted 24 January 2014
Objective: To conduct a systematic review to check the level of validity and reliability of the Manchester Triage System and the quality of reporting of literature on this topic. Design: This is a systematic review based on the PRISMA guideline on reporting systematic reviews. Data sources: The systematic search of the international literature published from 1997 through 30 November 2012 in the PubMed, Embase, Cochrane Library, Cinahl, Web of Knowledge, and Scopus databases. Review methods: This review included quantitative and qualitative research investigating the reliability and validity of the Manchester Triage System for the broad population of adults and children visiting the emergency department. After a systematic selection process, included studies were assessed on their quality by three researchers using the STARD guidelines. Results: Twelve studies were included in the review. The studies investigated the interand intra-rater reliability using the ‘‘kappa’’ statistic; the validity was tested with many measures: validity in predicting mortality, hospital admission, under- and overtriage, used resources, and length of stay in the emergency department, as well as a reference standard rating. Conclusions: In this review, the Manchester Triage System shows a wide inter-rater agreement range with a prevalence of good and very good agreement. Its safety was low because of the high rate of undertriage and the low sensitivity in predicting higher urgency levels. The high rate of overtriage could cause unnecessarily high use of resources in the emergency department. The quality of the reporting in studies of the reliability and validity of the Manchester Triage System is good. ß 2014 Elsevier Ltd. All rights reserved.
Keywords: Triage Emergency care Emergency medical service Literature review Systematic review
* Corresponding author at: Via San Vitale 96, Bologna Zip Code 40125, BO, Italy. Tel.: +39 0519912065; fax: +39 0519921849. E-mail address: [email protected] (N. Parenti). 0020-7489/$ – see front matter ß 2014 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ijnurstu.2014.01.013
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 2
N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx
What is already known about the topic? The Manchester Triage System (MTS) is a widespread inhospital triage system. The main quality indicators of a triage scale are the validity and reliability. Only recent studies tested MTS validity and reliability with different conclusions. What does this paper adds The Manchester Triage System seems to have a wide inter-rater agreement range and there are limited data on its intra-rater reliability. There was a wide range of results for validity measures among studies included. It is likely the triage scale could be improved in its safety. The quality of reporting among studies on this topic is good. 1. Introduction Triage is the first assessment and sorting process used to prioritize patients arriving in the emergency department (ED). The need to prioritize these patients is stressed by the considerable demand for emergency care, frequent ED overcrowding, and limited resources. For this reason many triage scales were developed in the last twenty years. Most current triage tools actually in use are based on a categorical measurement acuity scale and have five levels. The Australasian Triage Scale, ATS (Standards Committee Council, 1994); the Canadian Triage and Acuity Scale, CTAS (Beveridge, 1998); the Manchester Triage System, MTS (Mackway-Jones, 1997); and the Emergency Severity Index, ESI (Eitel et al., 2003; Tanabe et al., 2004; Wuerz et al., 2000, 2001), are all five-level triage tools. Many studies have evaluated the reliability and validity of acuity ratings by a triage nurse (Eitel et al., 2003; Parenti et al., 2009; Taboulet et al., 2009; Van der Wulp, 2008; Worster et al., 2007), probably because a triage scale should meet at least these two criteria to perform as intended (Twomey et al., 2007). In fact, according to most triage experts, (Fernandes et al., 2005) the ‘‘ideal’’ triage scale must demonstrate the characteristics of reliability and validity. For this reason we decided to conduct a systematic review on the state of studies on the reliability and validity of the Manchester Triage Scale, as MTS is one of the mostused triage scales in Europe. To our knowledge there are few reviews on all triage scales actually in use (Christ et al., 2010; Farrohknia et al., 2011; Van Veen and Moll, 2009), but there are no systematic reviews on the validity and reliability of the Manchester Triage System. A previous review on triage methods suggested that the reliability, validity, and safety of triage scales needed to be investigated further (Farrohknia et al., 2011). Van Veen and Moll (2009) concluded that MTS and PaedCTAS seem valid to triage children in emergency care, as well as that reliability of MTS is good, moderate-to-good for ESI, moderate for PaedCTAS, and poor to moderate for ATS.
2. Background The Manchester Triage System is an in-hospital triage scale usually used by nurses in many emergency departments in Great Britain and Europe since 1997 (Mackway-Jones, 1997; Martins et al., 2009; Olofsson et al., 2009; Van Baar et al., 2007). It is based on 52 flowchart diagrams; each of the flow charts depicts six key discriminators (Mackway-Jones, 1997). MTS considers five triage priority levels: level 1, immediate assessment; level 2, very urgent; level 3, urgent; level 4, standard; level 5, non-urgent. In recent years, many studies have been published on MTS reliability and validity. There is an inter-rater reliability and intra-rater reliability for triage systems. They are usually analyzed using the k statistic (Cohen’s kappa). The inter-rater reliability refers to the statistical measurement of agreement obtained by two or more users of the scale. It measures the agreement beyond chance between raters. Kappa equals zero when the amount of agreement is what would be observed by chance alone, and k is equal to one when there is a perfect agreement. Kappa levels are frequently evaluated with the following terminology (Landis and Koch, 1977): poor agreement, k less than 0.20; fair agreement if k is from 0.20 to 0.40; moderate agreement, k from 0.40 to 0, 60; good agreement, k from 0.60 to 0.80; very good agreement, k from 0.80 to 1. The MTS inter-rater reliability is determined by the agreement in triage urgency level if multiple raters triage one patient or patient scenario. The intra-rater agreement presents the agreement in triage urgency level if one triage nurse triages one case scenario at different points in time. Validity refers to the agreement between the value of a measurement and its true value; a triage scale is valid if it measures what it is supposed to measure: the ‘‘true urgency.’’ To assess the triage systems’ validity, a ‘‘gold standard’’ as a proxy for urgency has to be defined. Many outcome measures and reference standard for urgency have been used. The MTS validity can be expressed in sensitivity and specificity. MTS sensitivity presents its ability to identify high-urgency patients; MTS specificity presents its ability to identify patients with low-urgency problems. 3. Methods 3.1. Aim The primary aim was to check the state of studies on the reliability and validity of the Manchester Triage Scale for the broad population of adults and children visiting the ED. The questions for the review were: (1) What is the level of reliability of MTS among the selected studies? (2) How valid is MTS in predicting outcomes and workload among the selected studies? (3) How is the quality of reporting among published studies on reliability and validity of MTS?
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx
3.2. Review protocol
Included
Eligibility
Screening
Identification
This is a systematic review with a narrative synthesis of the main findings on the reliability of MTS. The review is based on the PRISMA guideline on reporting systematic reviews (Liberati et al., 2009) and on the STARD (Standard for Reporting of Diagnostic Accuracy) guidelines (Bossuyt et al., 2003). We used the PRISMA guideline for the first part of the review protocol: the selection of studies. The PRISMA guideline is an evidence-based minimum set of items for reporting in systematic reviews and metaanalyses. Its aim is to help authors improve the reporting of systematic reviews and consists of a 27-item checklist and a four-phase flow chart. Then we used the STARD guidelines to analyze the quality of reporting among studies selected. The STARD guidelines are used to improve the accuracy and completeness of reporting of studies of diagnostic accuracy, to allow readers to assess the potential for bias in the study and to evaluate its generalizability. The STARD guidelines consist of a checklist of 25 items and recommend the use of a flow chart that describes the design of the study and the flow of patients (Bossuyt et al., 2003). The selection of articles included in the review was performed in a three-phase process (Fig. 1) according to PRISMA guideline. In the first phase, one author, an expert in literature research, conducted a literature search of the following databases: PubMed, Embase, Cochrane Library, Cinahl, Web of Knowledge, and Scopus. In the second phase, three researchers independently performed a screening by regarding eligibility criteria of the six lists of articles (title and abstract) selected in phase one. All duplicates were removed. In this way 353 potentially useful articles were selected with their full text. In the third phase, the three researchers
380 records identified through database searching
3
independently examined all 353 full-text articles to select the studies that met the inclusion criteria for the systematic review. They had to justify the reasons for exclusion of studies in this phase. After this phase 15 articles were included, but then three of those articles were excluded with reasons: one study was a descriptive analysis of an implementation project of triage in one ED with MTS; two studies collected the literature’s findings on the reliability and validity of MTS. The 12 studies remaining after this phase were included in the analysis. Three researchers independently examined all 12 full-text articles according STARD guidelines. 3.3. Search methods We included all studies on reliability and validity of MTS conducted on all ages of patients in all languages. We excluded only the duplicates studies. The systematic search of the international literature published from 1997 through 30 November 2012 explored the databases of PubMed, Embase, Cochrane Library, Cinahl, Web of Knowledge, and Scopus. The following key words were used for the literature search: For PubMed ‘‘manchester triage’’ [all fields] OR ‘‘Manchester Triage System’’ [all fields] OR ‘‘manchester triage protocol’’ [all fields] OR ‘‘manchester triage scale’’ [all fields] OR ‘‘manchester triage method’’ [all fields] OR (Manchester [Ti,Ab] AND triage[Ti,Ab]) OR (triage[MeSH Terms no explode] AND Manchester[all fields]), 97 citations; For EMBASE: (Manchester AND triage [Ti,Ab]) OR (Emergency health service [EmTree no explode] AND Manchester[Ti,Ab]), 95 citations;
51 additional records identified through other sources
78 duplicate records removed
338 records excluded 353 records screened
15 full-text articles assessed for eligibility
3 full-text articles excluded, with reasons
12 studies included in qualitative synthesis and quality appraisal
Fig. 1. Review process.
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx
4
For Cinahl (Manchester AND triage [All fields]) OR (‘‘Triage’’[MH] AND (MTS OR Manchester[all fields]), 55 citations; For Web of Knowledge and Scopus: Manchester [all fields] AND triage [all fields], Web of Knowledge 52 citations and Scopus 81 citations; For Cochrane Library: Triage [all fields], no systematic reviews for Manchester triage.
estimation of the quality of the methodology of the studies (statistics, design of study). The three reviewers’ yes/no agreement for each study were entered into an Excel 2010 (Microsoft Corporation) spreadsheet, and the statistics Fleiss’ kappa for observed agreement was performed. We obtained a Fleiss’ kappa score of k = 0.72, equating to a ‘‘substantial’’ level of agreement between the raters (Landis and Koch, 1977).
3.4. Quality assessment
4. Results
Three authors conducted an appraisal of the reporting quality of the 12 studies included. They used the STARD guidelines because they considered the triage scales similar to diagnostic tools. The quality assessment was conducted in a two-phase process. In the first phase the researchers checked how many items of the STARD guidelines were met by each study. A STARD item was considered met when two of the three authors considered it in this way. To have standardization in the data extraction, we used the same checklist derived from the STARD guidelines (Bossuyt et al., 2003). A narrative summary was used to synthetize the data to provide a description and ordering of the evidence with commentary and interpretation. In the second phase the three researchers gave their personal opinion on the quality level of each study included. They used a three-grade scale: low quality (+/ +++), moderate quality (++/+++), and high quality (+++/ +++). The ‘‘overall quality evaluation’’ was a subjective
We collected 12 studies for final analysis (Table 1). Three studies met fewer than 50% of the STARD items and had a very low-quality methodology in the opinion of the authors (Table 2). None of the studies selected met all 25 items of the STARD guidelines (Table 2). Five of the nine studies met more than 80% of the STARD items. The study that met the greatest number of items on the STARD checklist was that of Van der Wulp et al. (2008) (Table 2). In the opinion of authors, the Van Veen et al. (2008) and Storm et al. (2011) papers reach a high grade of quality and a good performance in reporting according to STARD guidelines (Table 2). Of the 12 studies, six tested validity of MTS, four tested reliability, and two tested both indicators (Table 1). Four of the 12 were on pediatric population. There were one audit observational study, seven retrospective studies, and four prospective studies. Nine studies included were published after 2006 in northern Europe (Netherlands and Sweden).
Table 1 Characteristics of studies selected for review. Author, Year, Reference, Country
Study design
Patient Characteristics
Drop out
Outcome (Reliability/Validity)
Goodacre et al. (1999), Australia Roukema et al. (2006), Netherlands
Audit observ
0
Reliability K inter statistic
18,469
Van Veen et al. (2008), Netherlands
Prospective
50 scenarios; 4 reviewers Unknown patients characteristic 1065 pat: Male: 55.2%; Age (yrs, mean): 4.2 Pediatric population 16,735 pat: Age: 3.4 yrs; Female: 42% Pediatric population
Van der Wulp et al. (2008), Netherlands
Prospective
Van der Wulp et al. (2009), Netherlands Olofsson et al. (2009), Sweden
Retrospective
Grouse et al. (2009), Australia Martins et al. (2009), Portugal Storm-Versloot et al. (2009), Netherlands Van Veen et al. (2010), Netherlands Seiger et al. (2011), Netherlands Storm-Versloot (2011), Netherlands
Retrospective Retrospective Retrospective
Validity in predicting resource utilization, hospitalization and reference standard rating; Sensitivity/Spec; Over/undertriage Validity: Sens/Spec/LH ratio for rating respect to a RS Over/undertriage Reliability: K inter (unweighted and weighted) Validity: Sens/Spec for urgent pat; Over/ undertriage Respect to a reference standard Validity in predicting admission and mortality LH ratio (OR) and mortality rate Reliability with K inter (unweighted and weighted and quadratic weighted) Validity with accuracy in predicting a RS triage code; Over/undertriage Reliability with K inter (weighted) Validity: Rate of mortality and admission Reliability: K inter (unweighted and quadratic weighted) Reliability: K inter (weighted and quadratic weighted) Validity: Undertriage respect a RS tested with logistic regression (OR) Validity: undertriage, sens/spec respect a RS; relation between resource use/hospitalization/ length of stay and urgency level
Retrospective
Retrospective
Retrospective Prospective Prospective
50 scenarios; 55 nurses (+2RS); Unknown pat. charact; Mean nurses’ experience in ED: 7–15 yrs 34,258 pat; Age (yrs, mean): 42.4; Male: 51.6% 13 scenarios; 79 raters
50 scenarios; 20 raters 316,622 pat 50 scenarios; 12 raters Information on rater 20 scenarios; 43 raters Pediatric population 13,408/17,600 Pediatric population 872/890; Age (yrs, mean) = 37; Male: 53%
5%
26.4%
4917
4192 18
LH, Likelihood; RS, reference standard; Sens, sensitivity; Spec, specificity.
Please cite this article in press as: Parenti, N., et al., A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int. J. Nurs. Stud. (2014), http://dx.doi.org/10.1016/ j.ijnurstu.2014.01.013
G Model
NS-2348; No. of Pages 8 N. Parenti et al. / International Journal of Nursing Studies xxx (2014) xxx–xxx Table 2 Quality assessment of studies selected for review. Author (year)
STARD items Methodology respected % quality evaluation (n)
Outcome
Goodacre et al. (1999) Roukema et al. (2006)
72% (18/25) 84% (21/25)
Reliability Validity
Van Veen et al. (2008) Van der Wulp et al. (2008) al. (2009) Olofsson et Van der Wulp et al. (2009) Grouse et al. (2009) Storm-Versloot et al. (2009) Martins et al. (2009) Van Veen et al. (2010) Storm-Versloot (2011) Seiger et al. (2011)
84% (21/25) 92% (23/25) 76% (19/25)
