International Journal for Quality in Health Care, 2015, 27(1), 26–30 doi: 10.1093/intqhc/mzu087 Advance Access Publication Date: 22 November 2014 Article
Article
The association between patient-reported incidents in hospitals and estimated rates of patient harm OYVIND BJERTNAES1, ELLEN TVETER DEILKÅS2,3, KJERSTI EEG SKUDAL1, HILDE HESTAD IVERSEN1, and ANNE METTE BJERKAN1 1
Department for Quality and Patient Safety, Norwegian Knowledge Centre for the Health Services, Boks 7004 St Olavs plass, 0130 Oslo, Norway, 2Akershus University Hospital (Health Services Research Unit), Sykehusveien 25, 1478 Lørenskog, Norway, and 3The Norwegian Directorate of Health (the Norwegian Patient Safety Programme), Universitetsgata 2, 0164 Oslo, Norway
Downloaded from by guest on June 6, 2016
Address reprint requests to: Department for Quality and Patient Safety, Norwegian Knowledge Centre for the Health Services, Boks 7004 St Olavs plass, 0130 Oslo, Norway. Tel: +47-91-17-60-45; Fax: +47-23-25-50-10. E-mail: oyvind.andresen. [email protected] Accepted 15 October 2014
Abstract Objective: The aim of this study was to test the association between the rates of patient-reported incidents and patient harm documented in the patient record. Design: The study was a secondary analysis of two national hospital assessments conducted in 2011. Setting: Hospital services in Norway. Participants: The patient survey was a standard national patient-experience survey conducted at the hospital level for all 63 hospitals in Norway. The medical record review was performed by 47 Global Trigger Tools (GTTs) in all 19 hospital trusts and 4 private hospitals. The two data sets were matched at the unit level, yielding comparable patient experiences and GTT data for 7 departments, 16 hospitals and 11 hospital trusts. Intervention: No intervention. Main Outcome Measures: The correlation at the unit level between the patient-reported incident in hospital instrument (PRIH-I) and estimated rates of patient harm from the GTT. Results: The PRIH-I index was significantly correlated with all patient-reported experience indicators at the individual level, with estimates for all patient harm events (Categories E–I) at the unit level (r = 0.62, P < 0.01), and with estimates of more serious harm events in Categories F–I (r = 0.42, P < 0.05). Conclusions: Patient-reported incidents in hospitals, as measured by the PRIH-I, are strongly correlated with patient harm rates based on the GTT. This indicates that patient-reported incidents are related to patient safety, but more research is needed to confirm the usefulness of patient reporting in the evaluation of patient safety. Key words: patient experiences, questionnaire, patient safety, hospital, GTT
Introduction Patient centredness is an important part of health care quality and is often measured using surveys of patient-reported experiences or satisfaction [1]. Several instruments exist for the measurement of patient-
reported experiences with hospitals, such as the CAHPS (Consumer Assessment of Health-care Providers and Systems) Hospital Survey and the Picker Patient Experience Questionnaire [2, 3]. Such instruments measure important patient-experience domains from the patient
© The Author 2014. Published by Oxford University Press in association with the International Society for Quality in Health Care; all rights reserved
26
Patient-reported incidents • Patient safety
Methods Setting and materials The setting was hospital services in Norway. A secondary analysis of two national measurements from 2011 were conducted, a national patient-experience survey and a national GTT medical record review.
Data collection The national survey has been described previously [17]. The survey included 400 randomly selected adult inpatients who were discharged from each hospital in Norway between 1 March and 22 May 2011. The inclusion period was divided into three 4-week groups, such that patients received the questionnaire approximately 1–5 weeks following discharge. The Data Inspectorate and the Norwegian Ministry of Health and Care Services approved the survey. Forty-seven GTT teams from all 19 public hospital trusts and 4 private hospitals reviewed the medical records at those institutions in 2011 [18]. The reviews were conducted according to the GTT manual [11], which was translated into Norwegian [19], and a national protocol [20]. The Norwegian Ministry of Health and Care Services gave a
juridical approval of using the GTT within the hospital trusts and confirmed that anonymous information on harm rates could be submitted and aggregated to the national level. The reviews conducted by all of the GTT teams included 10 288 hospital admissions that had been randomly selected from a total of 585 648 admissions related to somatic health care. Admissions to rehabilitation, pediatric and psychiatric health care were excluded since the GTT is not adapted for these health care services. The GTT team reported their results for 2011 to the national level in a spreadsheet, listing the admissions where the patient experienced harm, and the incidents of harm according to their severity and type. According to the national protocol, the GTT teams reported the total number of randomly selected medical records that they had reviewed; 240 records annually is the minimum for every hospital trust. Hospital trusts have the opportunity to establish GTT teams at organizational levels lower than the trust level, as long as the population of admissions that the records are selected from do not overlap and cover all admissions that the GTT method is adapted for. The GTT teams also report the total number of patient admissions from which the reviewed records were selected.
Measures The questionnaire used in the national survey comprised 73 closedended items. Most experience items had a five-point response format ranging from 1 (‘not at all’) to 5 (‘to a very large extent’). Thirty-five items related to patient experiences with structures, processes and outcomes of health care were aggregated to the 10 quality indicators in the national report [21]: waiting time (one item), physical hospital standard (six items), next of kin (two items), organization (four items), doctor services (seven items), nursing services (seven items), information (three items), discharge planning (two items), cooperation with other health services (two items) and incorrect treatment (one item). The 13 patientincident questions were included in the patient-experience questionnaire. One of the items relates to performance of the primary care system and the patient and was therefore excluded in the index assessing the hospital. The index have been described previously [9]: it covered safety incidents, type of mistake, safety communication, medicine lists, infections, safety actions by health personnel such as hand hygiene and control of identity, and satisfaction with how the health personnel handled the mistake after it occurred. Seven of 12 patient-safety questions had a five-point response format (‘not at all’ to ‘to very large extent’), 3 questions had a 3-point format (‘no’, ‘once’ and ‘more than once’) and two questions had two response categories (‘yes’ and ‘no’). An incident index consisting of 12 safety items was computed [9]. The incident index varied from 0 to 12, with higher numbers indicating more patient-perceived incidents. The incident index was used in this study, in addition to one general item about incorrect treatment that was scored from ‘not at all’ [1] to ‘to a very large extent’ [5]. Estimates of harm rates were computed based on the GTT-team reports. This study used the percentage of hospital admissions with at least one harm event across all categories of severity (E–I) and across the categories of more serious severity (F–I). The severity categories are as follows: Category E represents temporary harm that requires intervention, Category F is temporary harm that requires initial or prolonged hospitalization, Category G represents permanent patient harm, Category H represents intervention required to sustain life and Category I means that the patient died.
Statistical analysis The patient-experience survey and the GTT measurement were conducted separately, with differences in the study protocol regarding
Downloaded from by guest on June 6, 2016
perspective as a basis for quality improvement and public accountability, with particular focus on the non-medical needs of patients [4]. Patients are also included in the evaluation of the effect of treatment through patient-reported outcome measures [5] but have to a much lesser degree been included in the evaluation of patient safety. However, primary studies and reviews are also starting to emerge on this topic [6–10], even though one of the reviews concluded that patientsafety reporting from an academic perspective is in its infancy [6]. The Patient-Reported Incident in Hospital Instrument (PRIH-I) was developed and validated as a method of measuring patient-perceived safety incidents in Norway [9]. It consists of a set of items pertaining to incidents in hospitals that have been reported by patients after discharge and has the possibility of providing a useful summary score for patient-perceived incidents at different levels. The mean score for the patient-incident index was 1.21 in Norway in 2011 (SD 1.69), on a scale from 0 (minimum) to 12 (maximum) incidents [9]. A national survey conducted in Norway in 2011 revealed that the instrument had good test–retest reliability and hospital-level reliability, with an estimated reliability on 0.61 and 0.85, respectively [9]. However, it has not been compared with other patient-safety measurements. Consequently, it is not clear whether and how the PRIH-I can best be integrated into a broader patient-safety measurement approach at the local and national levels. The Global Trigger Tool (GTT) is a standardized procedure for reviewing medical records to detect and monitor the overall level of patient harm in hospitals [11]. The method has strengths and weaknesses [12– 15] but has been used across Norwegian hospitals since 2011 as part of a national patient-safety campaign. A previous review of the literature found that patient experiences are positively associated with patient safety and effectiveness [16], but we are not aware of any scientific studies comparing patient-perceived incidents in hospitals with estimates based on the GTT. One recent study found a positive association at the ward level between patient-safety reporting and safety culture assessed by staff [10], but that study was restricted to 10 wards, and none of the measures were patient-safety outcome measures. However, these studies led to the formulation of the present study hypothesis: that patient-reported incidents in hospitals are positively and significantly associated with GTT estimates. The aim of this study was to test the association between the PRIH-I index and estimated rates of patient harm, based on a review of medical records according to the GTT definitions and procedure.
27
Bjertnaes et al.
28
Results The response rate in the national patient-experience survey was 46.4%. The mapping of patient-experience data and GTT estimates resulted in the inclusion of 11 units at the enterprise level, 16 units at the hospital level and 6 units at the department level in this study (Table 1). The national patient-experience survey showed that the mean PRIH-I index for the 34 GTT teams was 1.33 and varied from 0.68 for the best unit to 2.20 for the worst unit (Table 1). The percentage of patients reporting being incorrectly treated to a large or a very large extent was 4.95% on average, varying from 1.3 to 24.5% at the unit level. The national GTT measurement showed that total harm rates per 100 hospital stays based on GTT was on average 15.96 (F–I: 8.87), varying from 4.35 to 29.17 at the unit level (F–I: 2.08–15.83), where Category E represents temporary harm that requires intervention, F temporary harm requiring initial or prolonged hospitalization, G permanent patient harm, H intervention to sustain life and I means patient death. The PRIH-I index was significantly correlated with all patientreported experience indicators (Table 2), with the correlation being strongest with organization (0.56**) and weakest with waiting time for elective patients (0.10**). Table 3 shows that the PRIH-I index was significantly correlated with total GTT estimates at the unit level (0.62**), and with GTT estimates for Categories F–I (0.42*). At the unit level, the PRIH-I index was also correlated with the percentage of patients reporting a serious error (0.35*).
Discussion The aim of this study was to test the association between the PRIH-I index and GTT estimates of harm rates. The findings show that
Table 1 Description of organizational units Values Units at the enterprise level, n Units at the hospital level, n Units at the department level, n Patient-reported incidents PRIH-I indexa Incorrectly treated by largeor very large extent (%) Harm rates based on GTT All harm types per 100-patients stay Harm rates in Categories F–I per 100-patients stay
11 16 7 1.33 (0.68–2.20) 4.95 (1.30–24.50)
15.96 (4.35–29.17) 8.87 (2.08–15.83)
Data are mean (minimum–maximum) values except where indicated. a PRIH-I index ranges from 0 to 12, with higher numbers indicating more patient-perceived incidents.
Table 2 Correlation between patient-reported incidents (as quantified by the PRIH-I index) and patient-reported experience indicators at the individual level PRIH-I index Waiting time (elective patients) Doctor services Nursing services Information Contact with next of kin Standard Organization Discharge information Cooperation with other health services
0.10** 0.53** 0.51** 0.50** 0.39** 0.38** 0.56** 0.38** 0.39**
**P < 0.01; Pearson’s r.
patient-reported incidents in hospitals, as measured by the PRIH-I, are strongly correlated with estimates of harm rates from 34 GTT teams in Norway. This indicates that patient-reported incidents are related to patient safety and consequently that patient reporting can play a useful role both in evaluating health care effectiveness and responsiveness and in patient safety. The measurement of patient safety locally and nationally is complex, with several studies pointing to methodological challenges for currently available methods like GTT and patient-safety culture [12–15, 23–25]. The development work for the PRIH-I revealed some problems with medical measurements, such as those related to patient insecurity [9]. Patient reporting is in its infancy [6], but the PRIH-I is one of the first scientifically developed and tested instruments to evaluate safety incidents from the patient perspective. As a consequence of measurement problems with other available methods, patient reporting could be used alongside these other measurements to strengthen the measurement of patient safety locally and nationally. The PRIH-I index has been shown to have excellent data quality and hospital-level reliability [9], and the current study shows a strong association with estimates from GTT estimates from 34 GTT teams across Norway. Together these results indicate that the PRIH-I index is related to patient safety and supports its inclusion in the measurement of patient safety. The PRIH-I was developed in the context of a national patientsafety campaign in Norway as one of several measures to monitor the progress of the campaign at the national level. However, the
Downloaded from by guest on June 6, 2016
aggregation level: the protocol for the national patient-experience survey had hospital level as the main level (sample size reflects this), whereas the protocol for GTT was flexible regarding aggregation level allowing GTT teams at the health enterprise, hospital or department level. However, the patient-experience survey collected data at the patient level about hospital departments and wards, which gave the opportunity of selecting subsamples of patients from the same departments as the GTT teams and obtain estimates for patient experiences and GTT also at the department level. It was not possible to map patient-experience data to 12 GTT teams, most of them (n = 9) coming from a single health enterprise wishing to keep the GTT estimates anonymous outside the institution. We also decided to exclude one hospital that was very different from the other hospitals: a private non-acute hospital, with functions mainly within orthopedics and rheumatology. Consequently, patient-experience data were matched to 34 GTT teams. The mean (minimum–maximum) values for the 34 GTT teams were estimated for the PRIH-I index and the question about incorrect treatment. The PRIH-I index was correlated with GTT estimates at the unit level, using Pearson’s r. Pearson’s r assumes continuous, normally distributed variables and is sensitive to outliers in small samples [22]. The PRIH-I index was scored at the individual level with 12 additive items, giving an index ranging at the individual level from 0 to 12 incidents. The index was aggregated to the hospital level, giving hospital averages with a theoretical range from 0 to 12 incidents. The index has an absolute zero point and similar differences between adjacent values at different levels. Both the PRIH-index and the GTT measure were normally distributed at the hospital level. The item about incorrect treatment was highly skewed; thus, Spearman’s rho was used to assess the association between this item and the other variables.
Patient-reported incidents • Patient safety
29
Table 3 Correlation between patient-reported incidents (as quantified by the PRIH-I index) and GTT estimates at the unit level (n = 34)
PRIH-I index ( patients) Incorrect treatment ( patients) GTT: all types GTT: F–I
PRIH-I index( patients)
Incorrect treatment( patients)
GTT:all types
0.35* 0.62** 0.42*
0.32 0.12
0.76**
GTT:F–I
**P < 0.01; *P < 0.05; Pearson’s r for all correlations except for correlations with incorrect treatment (Spearman’s rho).
The mapping of departments, hospitals and enterprises in the PRIH-I study and the GTT study was a challenge and was not possible for 12 of the 47 units. We also excluded one hospital because it was a speciality hospital. Most of the 12 unmatched units (n = 9) were located in the same health enterprise, and hence, most Norwegian hospitals were covered by this study. However, it is not possible to rule out the presence of bias related to the coverage issue, even though we suspect that it would have been small. Another limitation relates to the patient-experience survey being conducted at the hospital level, and therefore, the statistical certainty of estimates made at the department level could be a problem. The response rate in the patient-experience survey was 46.4%, opening up the possibility for non-response bias. However, the material includes weights to compensate for nonresponse [21]; thus, we do not believe that non-response bias is a major concern. The GTT estimates were treated as a gold standard in this study; however, the method has both strengths and weaknesses [12–15, 33]. The GTT s especially useful for estimating harm rates locally and monitoring trends over time, as basis for learning and improvement of patient safety. The GTT can be applied without much resources and concerns a topic with high clinical face validity. However, concerns with the method have been raised, both relating to poor inter-rater reliability, changes in evaluation methods over time within teams, and the fact that the method does not consider harm prevention [12–15]. Moreover, inter-rater reliability data from the GTT teams were lacking in the present study, which is an area for future research. However, the strong correlation between the PRIH-I and GTT in this study provides scientific support for each of the methods and for the notion that both can be used to measure patient safety. The strong correlation between GTT estimates and the PRIH-I means that hospitals should consider patients as a useful source for identifying and monitoring patient safety. Patients have a unique perspective regarding their experiences with hospitals, and these experiences extend to experiences and incidents related to patient safety. Where possible, patients should be asked to report on or surveyed about experiences with safety and safety incidents, as basis for quality improvement and hospital governance. The inclusion of the PRIH-I in national patient-experience surveys means that patient-safety reports could form external quality indicators. The PRIH-I should be relevant in other countries, but following rigorous scientific translation and validation procedures. Patient-reported incidents in hospitals, as measured by the PRIH-I, are strongly correlated with estimates of harm rates based on the GTT. This indicates that patient-reported incidents are related to patient safety, but more research is needed to confirm the usefulness of patient reporting in evaluation of patient safety.
Acknowledgements Thanks to Tomislav Dimoski for data collection management and for developing the FS-system and carrying out the technical aspects of the survey. Thanks
Downloaded from by guest on June 6, 2016
instrument was included in a national patient-experience survey, the goal of which was to measure patient experiences as a basis for quality improvement, hospital governance, free hospital choice and public accountability. Therefore, an important issue relates to applicability of using the PRIH-I for these goals, both at the local and national levels. One approach would be to interpret and treat the PRIH-I the same way as other national patient-experience indicators, namely as important indicators for patient-reported experiences, with standard casemix adjusted results [26] at the hospital level being reported in the public domain. That would be feasible, but the current study also provided preliminary data to support the inclusion of the PRIH-I index as a measurement of patient safety. While the index might function well for most of the goals mentioned earlier, given appropriate assessment of the case mix, the lack of concrete information might be problematic from a quality-improvement perspective. The PRIH-I index signals patient-safety challenges, but does not point to specific patient-safety problems, for instance infections, wrong medication or wrong diagnosis. In fact, this problem is also encountered with overall measures for GTT, and the possible decomposition of the overall measure for both GTT and the PRIH-I results in much larger measurement errors. The optimal solution would be to increase the sample size for each hospital to the required level. Our previous study showed that most items reached a reliability criterion at the hospital level of 0.8 when 300 responses were used [9]. However, this might not be possible due to economic and practical restrictions, at least not for national or other large-scale patient-experience surveys. If the sample is not sufficiently large, disaggregation of the overall index to single items and to lower hospital levels (e.g. ward or department) should be done with caution and where possible supplemented with other safety indicators. In Norway, the PRIH-I is included in annual national patient-experience surveys and can be used alongside safety indicators produced in the national patient-safety campaign, including GTT indicators and indicators of patient-safety culture. Several studies have assessed the association at the individual level between patient reporting and medical record review [27–30] or compared different reporting systems [31]. These studies support patient reporting as an important supplement to other safety sources. Fewer studies have assessed the association between patient-reported safety and other scientifically based safety indicators at the unit or hospital level. One recent study found a positive association at the ward level between patient-safety reporting and safety culture assessed by staff [10], but the study was restricted to only 10 wards and none of the measures were patient-safety outcome measures. Another study also found positive associations between patient-safety reporting and safety culture [32], but the correlation analysis was restricted to four departments. Our research involving the PRIH-I has indicated that the PRIH-I index is a reliable and valid measure of patient safety at the unit level, but more research is needed, including confirmation studies for the current findings and those of the previous study, tests of the ability of the PRIH-I to measure changes over time, and implementation issues related to benchmarking (case mix) and quality improvement.
30 also to Marit Skarpaas, Sinan Akbas and Ulla Benedicte Funder for administrative help in survey data collection. Lastly, thanks to the GTT teams across Norway for data collection.
Funding This secondary analysis was financed by the Norwegian Knowledge for the Health Services.
References
16. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open 2013;3: pii:e001570. 17. Bjertnaes O, Skudal KE, Iversen HH. Classification of patients based on their evaluation of hospital outcomes: cluster analysis following a national survey in Norway. BMC Health Serv Res 2013;13:73. 18. Deilkås E. Rapport for nasjonal journalundersøkelse med Global Trigger Tool 2011. Report for national assessment with Global Trigger Tool in 2011. Oslo: Nasjonalt kunnskapssenter for helsetjenesten, 2013. 19. Strukturert journalundersøkelse, ved bruk av Global Trigger Tool for å identifisere og måle forekomst av skader i helsetjenesten. http://www. pasientsikkerhetsprogrammet.no/no/Helsepersonell/Materiell/_attachment/ 249?_ts=135a0292a89 (20 May 2014, date last accessed). 20. I trygge hender. Gjennomføring av journalundersøkelse med Global Trigger Tool (GTT) i den norske pasientsikkerhetskampanjen. http://www. pasientsikkerhetsprogrammet.no/no/Helsepersonell/Materiell/_attachment/ 1908?_ts=13e99f095c2 (20 may 2014, date last accessed). 21. Skudal KE, Holmboe O, Iversen H, et al. Patient experiences with hospitals: national results in 2011 and changes from 2006. PasOpp-report 3—2012. Oslo, Norway: The Norwegian Knowledge Centre for the Health Services, 2012. 22. Chen PY, Popovich PM. Correlation: Parametric and Nonparametric Measures. Sage University Papers Series on Quantitative Applications in the Social Sciences, 07-139. Thousand Oaks, CA: Sage, 2002. 23. Morello RT, Lowthian JA, Barker AL, et al. Strategies for improving patient safety culture in hospitals: a systematic review. BMJ Qual Saf 2013; 22:11–8. 24. Singer SJ, Vogus TJ. Safety climate research: taking stock and looking forward. BMJ Qual Saf 2013;22:1–4. 25. Verbakel NJ, van Melle M, Langelaan M, et al. Exploring patient safety culture in primary care. Int J Qual Health Care 2014;pii: mzu074. 26. O’Malley AJ, Zaslavsky AM, Elliott MN, et al. Case-mix adjustment of the CAHPS hospital survey. Health Serv Res 2005;40:2162–81. 27. Weissman JS, Schneider EC, Weingart SN, et al. Comparing patientreported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med 2008;149:100–8. 28. Zhu J, Stuver SO, Epstein AM, et al. Can we rely on patients’ reports of adverse events? Med Care 2011;49:948–55. 29. Weingart SN, Pagovich O, Sands DZ, et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Intern Med 2005;20:830–6. 30. Taylor BB, Marcantonio ER, Pagovich O, et al. Do medical inpatients who report poor service quality experience more adverse events and medical errors? Med Care 2008;46:224–8. 31. Levtzion-Korach O, Frankel A, Alcalai H, et al. Integrating incident data from five reporting systems to assess patient safety: making sense of the elephant. Jt Comm J Qual Patient Saf 2010;36:402–10. 32. Deilkås ET, Hofoss D. Psychometric properties of the Norwegian version of the Safety Attitudes Questionnaire (SAQ), Generic version (Short Form 2006). BMC Health Serv Res 2008;8:191. 33. Najjar S, Hamdan M, Euwema MC, et al. The Global Trigger Tool shows that one out of seven patients suffers harm in Palestinian hospitals: challenges for launching a strategic safety plan. Int J Qual Health Care 2013;25:640–7.
Downloaded from by guest on June 6, 2016
1. Crow R, Gage H, Hampson S, et al. The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature. Health Technol Assess 2002;6:1–244. 2. Keller S, O’Malley AJ, Hays RD, et al. Methods used to streamline the CAHPS Hospital Survey. Health Serv Res 2005;40:2057–77. 3. Jenkinson C, Coulter A, Bruster S. The Picker Patient Experience Questionnaire: development and validation using data from in-patient surveys in five countries. Int J Qual Health Care 2002;14:353–8. 4. Gerteis M, Edgman-Levitan S, Daley J, et al. Through the patient’s eyes: understanding and promoting patient-centered care. San Francisco, CA: Jossey-Bass, 1993. 5. Fitzpatrick R, Davey C, Buxton MJ, et al. Evaluating patient-based outcome measures for use in clinical trials. Health Technol Assess 1998;2: i–iv, 1–74. 6. Ward JK, Armitage G. Can patients report patient safety incidents in a hospital setting? A systematic review. BMJ Qual Saf 2012;21:685–99. 7. Massó Guijarro P, Aranaz Andrés JM, Mira JJ, et al. Adverse events in hospitals: the patient’s point of view. Qual Saf Health Care 2010;19:144–7. 8. King A, Daniels J, Lim J, et al. Time to listen: a review of methods to solicit patient reports of adverse events. Qual Saf Health Care 2010;19:148–57. 9. Bjertnaes O, Skudal KE, Iversen HH, et al. The Patient-Reported Incident in Hospital Instrument (PRIH-I): assessments of data quality, test-retest reliability and hospital-level reliability. BMJ Qual Saf 2013;22:743–51. 10. McEachan RR, Lawton RJ, O’Hara JK, et al.; on behalf of the Yorkshire Quality and Safety Research Group. Developing a reliable and valid patient measure of safety in hospitals (PMOS): a validation study. BMJ Qual Saf 2014;23:565–73. 11. Griffin FA, Resar RK. IHI Global Trigger Tool for Measuring Adverse Events, 2nd edn. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement, 2009. 12. Schildmeijer K, Nilsson L, Arestedt K, et al. Assessment of adverse events in medical care: lack of consistency between experienced teams using the global trigger tool. BMJ Qual Saf 2012;21:307–14. 13. Mattsson TO, Knudsen JL, Lauritsen J, et al. Assessment of the global trigger tool to measure, monitor and evaluate patient safety in cancer patients: reliability concerns are raised. BMJ Qual Saf 2013;22:571–9. 14. von Plessen C, Kodal AM, Anhøj J. Experiences with global trigger tool reviews in five Danish hospitals: an implementation study. BMJ Open 2012; 2: pii:e001324. 15. Schildmeijer K, Nilsson L, Perk J, et al. Strengths and weaknesses of working with the global trigger tool method for retrospective record review: focus group interviews with team members. BMJ Open 2013;3:e003131.
Bjertnaes et al.
Article
The association between patient-reported incidents in hospitals and estimated rates of patient harm OYVIND BJERTNAES1, ELLEN TVETER DEILKÅS2,3, KJERSTI EEG SKUDAL1, HILDE HESTAD IVERSEN1, and ANNE METTE BJERKAN1 1
Department for Quality and Patient Safety, Norwegian Knowledge Centre for the Health Services, Boks 7004 St Olavs plass, 0130 Oslo, Norway, 2Akershus University Hospital (Health Services Research Unit), Sykehusveien 25, 1478 Lørenskog, Norway, and 3The Norwegian Directorate of Health (the Norwegian Patient Safety Programme), Universitetsgata 2, 0164 Oslo, Norway
Downloaded from by guest on June 6, 2016
Address reprint requests to: Department for Quality and Patient Safety, Norwegian Knowledge Centre for the Health Services, Boks 7004 St Olavs plass, 0130 Oslo, Norway. Tel: +47-91-17-60-45; Fax: +47-23-25-50-10. E-mail: oyvind.andresen. [email protected] Accepted 15 October 2014
Abstract Objective: The aim of this study was to test the association between the rates of patient-reported incidents and patient harm documented in the patient record. Design: The study was a secondary analysis of two national hospital assessments conducted in 2011. Setting: Hospital services in Norway. Participants: The patient survey was a standard national patient-experience survey conducted at the hospital level for all 63 hospitals in Norway. The medical record review was performed by 47 Global Trigger Tools (GTTs) in all 19 hospital trusts and 4 private hospitals. The two data sets were matched at the unit level, yielding comparable patient experiences and GTT data for 7 departments, 16 hospitals and 11 hospital trusts. Intervention: No intervention. Main Outcome Measures: The correlation at the unit level between the patient-reported incident in hospital instrument (PRIH-I) and estimated rates of patient harm from the GTT. Results: The PRIH-I index was significantly correlated with all patient-reported experience indicators at the individual level, with estimates for all patient harm events (Categories E–I) at the unit level (r = 0.62, P < 0.01), and with estimates of more serious harm events in Categories F–I (r = 0.42, P < 0.05). Conclusions: Patient-reported incidents in hospitals, as measured by the PRIH-I, are strongly correlated with patient harm rates based on the GTT. This indicates that patient-reported incidents are related to patient safety, but more research is needed to confirm the usefulness of patient reporting in the evaluation of patient safety. Key words: patient experiences, questionnaire, patient safety, hospital, GTT
Introduction Patient centredness is an important part of health care quality and is often measured using surveys of patient-reported experiences or satisfaction [1]. Several instruments exist for the measurement of patient-
reported experiences with hospitals, such as the CAHPS (Consumer Assessment of Health-care Providers and Systems) Hospital Survey and the Picker Patient Experience Questionnaire [2, 3]. Such instruments measure important patient-experience domains from the patient
© The Author 2014. Published by Oxford University Press in association with the International Society for Quality in Health Care; all rights reserved
26
Patient-reported incidents • Patient safety
Methods Setting and materials The setting was hospital services in Norway. A secondary analysis of two national measurements from 2011 were conducted, a national patient-experience survey and a national GTT medical record review.
Data collection The national survey has been described previously [17]. The survey included 400 randomly selected adult inpatients who were discharged from each hospital in Norway between 1 March and 22 May 2011. The inclusion period was divided into three 4-week groups, such that patients received the questionnaire approximately 1–5 weeks following discharge. The Data Inspectorate and the Norwegian Ministry of Health and Care Services approved the survey. Forty-seven GTT teams from all 19 public hospital trusts and 4 private hospitals reviewed the medical records at those institutions in 2011 [18]. The reviews were conducted according to the GTT manual [11], which was translated into Norwegian [19], and a national protocol [20]. The Norwegian Ministry of Health and Care Services gave a
juridical approval of using the GTT within the hospital trusts and confirmed that anonymous information on harm rates could be submitted and aggregated to the national level. The reviews conducted by all of the GTT teams included 10 288 hospital admissions that had been randomly selected from a total of 585 648 admissions related to somatic health care. Admissions to rehabilitation, pediatric and psychiatric health care were excluded since the GTT is not adapted for these health care services. The GTT team reported their results for 2011 to the national level in a spreadsheet, listing the admissions where the patient experienced harm, and the incidents of harm according to their severity and type. According to the national protocol, the GTT teams reported the total number of randomly selected medical records that they had reviewed; 240 records annually is the minimum for every hospital trust. Hospital trusts have the opportunity to establish GTT teams at organizational levels lower than the trust level, as long as the population of admissions that the records are selected from do not overlap and cover all admissions that the GTT method is adapted for. The GTT teams also report the total number of patient admissions from which the reviewed records were selected.
Measures The questionnaire used in the national survey comprised 73 closedended items. Most experience items had a five-point response format ranging from 1 (‘not at all’) to 5 (‘to a very large extent’). Thirty-five items related to patient experiences with structures, processes and outcomes of health care were aggregated to the 10 quality indicators in the national report [21]: waiting time (one item), physical hospital standard (six items), next of kin (two items), organization (four items), doctor services (seven items), nursing services (seven items), information (three items), discharge planning (two items), cooperation with other health services (two items) and incorrect treatment (one item). The 13 patientincident questions were included in the patient-experience questionnaire. One of the items relates to performance of the primary care system and the patient and was therefore excluded in the index assessing the hospital. The index have been described previously [9]: it covered safety incidents, type of mistake, safety communication, medicine lists, infections, safety actions by health personnel such as hand hygiene and control of identity, and satisfaction with how the health personnel handled the mistake after it occurred. Seven of 12 patient-safety questions had a five-point response format (‘not at all’ to ‘to very large extent’), 3 questions had a 3-point format (‘no’, ‘once’ and ‘more than once’) and two questions had two response categories (‘yes’ and ‘no’). An incident index consisting of 12 safety items was computed [9]. The incident index varied from 0 to 12, with higher numbers indicating more patient-perceived incidents. The incident index was used in this study, in addition to one general item about incorrect treatment that was scored from ‘not at all’ [1] to ‘to a very large extent’ [5]. Estimates of harm rates were computed based on the GTT-team reports. This study used the percentage of hospital admissions with at least one harm event across all categories of severity (E–I) and across the categories of more serious severity (F–I). The severity categories are as follows: Category E represents temporary harm that requires intervention, Category F is temporary harm that requires initial or prolonged hospitalization, Category G represents permanent patient harm, Category H represents intervention required to sustain life and Category I means that the patient died.
Statistical analysis The patient-experience survey and the GTT measurement were conducted separately, with differences in the study protocol regarding
Downloaded from by guest on June 6, 2016
perspective as a basis for quality improvement and public accountability, with particular focus on the non-medical needs of patients [4]. Patients are also included in the evaluation of the effect of treatment through patient-reported outcome measures [5] but have to a much lesser degree been included in the evaluation of patient safety. However, primary studies and reviews are also starting to emerge on this topic [6–10], even though one of the reviews concluded that patientsafety reporting from an academic perspective is in its infancy [6]. The Patient-Reported Incident in Hospital Instrument (PRIH-I) was developed and validated as a method of measuring patient-perceived safety incidents in Norway [9]. It consists of a set of items pertaining to incidents in hospitals that have been reported by patients after discharge and has the possibility of providing a useful summary score for patient-perceived incidents at different levels. The mean score for the patient-incident index was 1.21 in Norway in 2011 (SD 1.69), on a scale from 0 (minimum) to 12 (maximum) incidents [9]. A national survey conducted in Norway in 2011 revealed that the instrument had good test–retest reliability and hospital-level reliability, with an estimated reliability on 0.61 and 0.85, respectively [9]. However, it has not been compared with other patient-safety measurements. Consequently, it is not clear whether and how the PRIH-I can best be integrated into a broader patient-safety measurement approach at the local and national levels. The Global Trigger Tool (GTT) is a standardized procedure for reviewing medical records to detect and monitor the overall level of patient harm in hospitals [11]. The method has strengths and weaknesses [12– 15] but has been used across Norwegian hospitals since 2011 as part of a national patient-safety campaign. A previous review of the literature found that patient experiences are positively associated with patient safety and effectiveness [16], but we are not aware of any scientific studies comparing patient-perceived incidents in hospitals with estimates based on the GTT. One recent study found a positive association at the ward level between patient-safety reporting and safety culture assessed by staff [10], but that study was restricted to 10 wards, and none of the measures were patient-safety outcome measures. However, these studies led to the formulation of the present study hypothesis: that patient-reported incidents in hospitals are positively and significantly associated with GTT estimates. The aim of this study was to test the association between the PRIH-I index and estimated rates of patient harm, based on a review of medical records according to the GTT definitions and procedure.
27
Bjertnaes et al.
28
Results The response rate in the national patient-experience survey was 46.4%. The mapping of patient-experience data and GTT estimates resulted in the inclusion of 11 units at the enterprise level, 16 units at the hospital level and 6 units at the department level in this study (Table 1). The national patient-experience survey showed that the mean PRIH-I index for the 34 GTT teams was 1.33 and varied from 0.68 for the best unit to 2.20 for the worst unit (Table 1). The percentage of patients reporting being incorrectly treated to a large or a very large extent was 4.95% on average, varying from 1.3 to 24.5% at the unit level. The national GTT measurement showed that total harm rates per 100 hospital stays based on GTT was on average 15.96 (F–I: 8.87), varying from 4.35 to 29.17 at the unit level (F–I: 2.08–15.83), where Category E represents temporary harm that requires intervention, F temporary harm requiring initial or prolonged hospitalization, G permanent patient harm, H intervention to sustain life and I means patient death. The PRIH-I index was significantly correlated with all patientreported experience indicators (Table 2), with the correlation being strongest with organization (0.56**) and weakest with waiting time for elective patients (0.10**). Table 3 shows that the PRIH-I index was significantly correlated with total GTT estimates at the unit level (0.62**), and with GTT estimates for Categories F–I (0.42*). At the unit level, the PRIH-I index was also correlated with the percentage of patients reporting a serious error (0.35*).
Discussion The aim of this study was to test the association between the PRIH-I index and GTT estimates of harm rates. The findings show that
Table 1 Description of organizational units Values Units at the enterprise level, n Units at the hospital level, n Units at the department level, n Patient-reported incidents PRIH-I indexa Incorrectly treated by largeor very large extent (%) Harm rates based on GTT All harm types per 100-patients stay Harm rates in Categories F–I per 100-patients stay
11 16 7 1.33 (0.68–2.20) 4.95 (1.30–24.50)
15.96 (4.35–29.17) 8.87 (2.08–15.83)
Data are mean (minimum–maximum) values except where indicated. a PRIH-I index ranges from 0 to 12, with higher numbers indicating more patient-perceived incidents.
Table 2 Correlation between patient-reported incidents (as quantified by the PRIH-I index) and patient-reported experience indicators at the individual level PRIH-I index Waiting time (elective patients) Doctor services Nursing services Information Contact with next of kin Standard Organization Discharge information Cooperation with other health services
0.10** 0.53** 0.51** 0.50** 0.39** 0.38** 0.56** 0.38** 0.39**
**P < 0.01; Pearson’s r.
patient-reported incidents in hospitals, as measured by the PRIH-I, are strongly correlated with estimates of harm rates from 34 GTT teams in Norway. This indicates that patient-reported incidents are related to patient safety and consequently that patient reporting can play a useful role both in evaluating health care effectiveness and responsiveness and in patient safety. The measurement of patient safety locally and nationally is complex, with several studies pointing to methodological challenges for currently available methods like GTT and patient-safety culture [12–15, 23–25]. The development work for the PRIH-I revealed some problems with medical measurements, such as those related to patient insecurity [9]. Patient reporting is in its infancy [6], but the PRIH-I is one of the first scientifically developed and tested instruments to evaluate safety incidents from the patient perspective. As a consequence of measurement problems with other available methods, patient reporting could be used alongside these other measurements to strengthen the measurement of patient safety locally and nationally. The PRIH-I index has been shown to have excellent data quality and hospital-level reliability [9], and the current study shows a strong association with estimates from GTT estimates from 34 GTT teams across Norway. Together these results indicate that the PRIH-I index is related to patient safety and supports its inclusion in the measurement of patient safety. The PRIH-I was developed in the context of a national patientsafety campaign in Norway as one of several measures to monitor the progress of the campaign at the national level. However, the
Downloaded from by guest on June 6, 2016
aggregation level: the protocol for the national patient-experience survey had hospital level as the main level (sample size reflects this), whereas the protocol for GTT was flexible regarding aggregation level allowing GTT teams at the health enterprise, hospital or department level. However, the patient-experience survey collected data at the patient level about hospital departments and wards, which gave the opportunity of selecting subsamples of patients from the same departments as the GTT teams and obtain estimates for patient experiences and GTT also at the department level. It was not possible to map patient-experience data to 12 GTT teams, most of them (n = 9) coming from a single health enterprise wishing to keep the GTT estimates anonymous outside the institution. We also decided to exclude one hospital that was very different from the other hospitals: a private non-acute hospital, with functions mainly within orthopedics and rheumatology. Consequently, patient-experience data were matched to 34 GTT teams. The mean (minimum–maximum) values for the 34 GTT teams were estimated for the PRIH-I index and the question about incorrect treatment. The PRIH-I index was correlated with GTT estimates at the unit level, using Pearson’s r. Pearson’s r assumes continuous, normally distributed variables and is sensitive to outliers in small samples [22]. The PRIH-I index was scored at the individual level with 12 additive items, giving an index ranging at the individual level from 0 to 12 incidents. The index was aggregated to the hospital level, giving hospital averages with a theoretical range from 0 to 12 incidents. The index has an absolute zero point and similar differences between adjacent values at different levels. Both the PRIH-index and the GTT measure were normally distributed at the hospital level. The item about incorrect treatment was highly skewed; thus, Spearman’s rho was used to assess the association between this item and the other variables.
Patient-reported incidents • Patient safety
29
Table 3 Correlation between patient-reported incidents (as quantified by the PRIH-I index) and GTT estimates at the unit level (n = 34)
PRIH-I index ( patients) Incorrect treatment ( patients) GTT: all types GTT: F–I
PRIH-I index( patients)
Incorrect treatment( patients)
GTT:all types
0.35* 0.62** 0.42*
0.32 0.12
0.76**
GTT:F–I
**P < 0.01; *P < 0.05; Pearson’s r for all correlations except for correlations with incorrect treatment (Spearman’s rho).
The mapping of departments, hospitals and enterprises in the PRIH-I study and the GTT study was a challenge and was not possible for 12 of the 47 units. We also excluded one hospital because it was a speciality hospital. Most of the 12 unmatched units (n = 9) were located in the same health enterprise, and hence, most Norwegian hospitals were covered by this study. However, it is not possible to rule out the presence of bias related to the coverage issue, even though we suspect that it would have been small. Another limitation relates to the patient-experience survey being conducted at the hospital level, and therefore, the statistical certainty of estimates made at the department level could be a problem. The response rate in the patient-experience survey was 46.4%, opening up the possibility for non-response bias. However, the material includes weights to compensate for nonresponse [21]; thus, we do not believe that non-response bias is a major concern. The GTT estimates were treated as a gold standard in this study; however, the method has both strengths and weaknesses [12–15, 33]. The GTT s especially useful for estimating harm rates locally and monitoring trends over time, as basis for learning and improvement of patient safety. The GTT can be applied without much resources and concerns a topic with high clinical face validity. However, concerns with the method have been raised, both relating to poor inter-rater reliability, changes in evaluation methods over time within teams, and the fact that the method does not consider harm prevention [12–15]. Moreover, inter-rater reliability data from the GTT teams were lacking in the present study, which is an area for future research. However, the strong correlation between the PRIH-I and GTT in this study provides scientific support for each of the methods and for the notion that both can be used to measure patient safety. The strong correlation between GTT estimates and the PRIH-I means that hospitals should consider patients as a useful source for identifying and monitoring patient safety. Patients have a unique perspective regarding their experiences with hospitals, and these experiences extend to experiences and incidents related to patient safety. Where possible, patients should be asked to report on or surveyed about experiences with safety and safety incidents, as basis for quality improvement and hospital governance. The inclusion of the PRIH-I in national patient-experience surveys means that patient-safety reports could form external quality indicators. The PRIH-I should be relevant in other countries, but following rigorous scientific translation and validation procedures. Patient-reported incidents in hospitals, as measured by the PRIH-I, are strongly correlated with estimates of harm rates based on the GTT. This indicates that patient-reported incidents are related to patient safety, but more research is needed to confirm the usefulness of patient reporting in evaluation of patient safety.
Acknowledgements Thanks to Tomislav Dimoski for data collection management and for developing the FS-system and carrying out the technical aspects of the survey. Thanks
Downloaded from by guest on June 6, 2016
instrument was included in a national patient-experience survey, the goal of which was to measure patient experiences as a basis for quality improvement, hospital governance, free hospital choice and public accountability. Therefore, an important issue relates to applicability of using the PRIH-I for these goals, both at the local and national levels. One approach would be to interpret and treat the PRIH-I the same way as other national patient-experience indicators, namely as important indicators for patient-reported experiences, with standard casemix adjusted results [26] at the hospital level being reported in the public domain. That would be feasible, but the current study also provided preliminary data to support the inclusion of the PRIH-I index as a measurement of patient safety. While the index might function well for most of the goals mentioned earlier, given appropriate assessment of the case mix, the lack of concrete information might be problematic from a quality-improvement perspective. The PRIH-I index signals patient-safety challenges, but does not point to specific patient-safety problems, for instance infections, wrong medication or wrong diagnosis. In fact, this problem is also encountered with overall measures for GTT, and the possible decomposition of the overall measure for both GTT and the PRIH-I results in much larger measurement errors. The optimal solution would be to increase the sample size for each hospital to the required level. Our previous study showed that most items reached a reliability criterion at the hospital level of 0.8 when 300 responses were used [9]. However, this might not be possible due to economic and practical restrictions, at least not for national or other large-scale patient-experience surveys. If the sample is not sufficiently large, disaggregation of the overall index to single items and to lower hospital levels (e.g. ward or department) should be done with caution and where possible supplemented with other safety indicators. In Norway, the PRIH-I is included in annual national patient-experience surveys and can be used alongside safety indicators produced in the national patient-safety campaign, including GTT indicators and indicators of patient-safety culture. Several studies have assessed the association at the individual level between patient reporting and medical record review [27–30] or compared different reporting systems [31]. These studies support patient reporting as an important supplement to other safety sources. Fewer studies have assessed the association between patient-reported safety and other scientifically based safety indicators at the unit or hospital level. One recent study found a positive association at the ward level between patient-safety reporting and safety culture assessed by staff [10], but the study was restricted to only 10 wards and none of the measures were patient-safety outcome measures. Another study also found positive associations between patient-safety reporting and safety culture [32], but the correlation analysis was restricted to four departments. Our research involving the PRIH-I has indicated that the PRIH-I index is a reliable and valid measure of patient safety at the unit level, but more research is needed, including confirmation studies for the current findings and those of the previous study, tests of the ability of the PRIH-I to measure changes over time, and implementation issues related to benchmarking (case mix) and quality improvement.
30 also to Marit Skarpaas, Sinan Akbas and Ulla Benedicte Funder for administrative help in survey data collection. Lastly, thanks to the GTT teams across Norway for data collection.
Funding This secondary analysis was financed by the Norwegian Knowledge for the Health Services.
References
16. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open 2013;3: pii:e001570. 17. Bjertnaes O, Skudal KE, Iversen HH. Classification of patients based on their evaluation of hospital outcomes: cluster analysis following a national survey in Norway. BMC Health Serv Res 2013;13:73. 18. Deilkås E. Rapport for nasjonal journalundersøkelse med Global Trigger Tool 2011. Report for national assessment with Global Trigger Tool in 2011. Oslo: Nasjonalt kunnskapssenter for helsetjenesten, 2013. 19. Strukturert journalundersøkelse, ved bruk av Global Trigger Tool for å identifisere og måle forekomst av skader i helsetjenesten. http://www. pasientsikkerhetsprogrammet.no/no/Helsepersonell/Materiell/_attachment/ 249?_ts=135a0292a89 (20 May 2014, date last accessed). 20. I trygge hender. Gjennomføring av journalundersøkelse med Global Trigger Tool (GTT) i den norske pasientsikkerhetskampanjen. http://www. pasientsikkerhetsprogrammet.no/no/Helsepersonell/Materiell/_attachment/ 1908?_ts=13e99f095c2 (20 may 2014, date last accessed). 21. Skudal KE, Holmboe O, Iversen H, et al. Patient experiences with hospitals: national results in 2011 and changes from 2006. PasOpp-report 3—2012. Oslo, Norway: The Norwegian Knowledge Centre for the Health Services, 2012. 22. Chen PY, Popovich PM. Correlation: Parametric and Nonparametric Measures. Sage University Papers Series on Quantitative Applications in the Social Sciences, 07-139. Thousand Oaks, CA: Sage, 2002. 23. Morello RT, Lowthian JA, Barker AL, et al. Strategies for improving patient safety culture in hospitals: a systematic review. BMJ Qual Saf 2013; 22:11–8. 24. Singer SJ, Vogus TJ. Safety climate research: taking stock and looking forward. BMJ Qual Saf 2013;22:1–4. 25. Verbakel NJ, van Melle M, Langelaan M, et al. Exploring patient safety culture in primary care. Int J Qual Health Care 2014;pii: mzu074. 26. O’Malley AJ, Zaslavsky AM, Elliott MN, et al. Case-mix adjustment of the CAHPS hospital survey. Health Serv Res 2005;40:2162–81. 27. Weissman JS, Schneider EC, Weingart SN, et al. Comparing patientreported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med 2008;149:100–8. 28. Zhu J, Stuver SO, Epstein AM, et al. Can we rely on patients’ reports of adverse events? Med Care 2011;49:948–55. 29. Weingart SN, Pagovich O, Sands DZ, et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Intern Med 2005;20:830–6. 30. Taylor BB, Marcantonio ER, Pagovich O, et al. Do medical inpatients who report poor service quality experience more adverse events and medical errors? Med Care 2008;46:224–8. 31. Levtzion-Korach O, Frankel A, Alcalai H, et al. Integrating incident data from five reporting systems to assess patient safety: making sense of the elephant. Jt Comm J Qual Patient Saf 2010;36:402–10. 32. Deilkås ET, Hofoss D. Psychometric properties of the Norwegian version of the Safety Attitudes Questionnaire (SAQ), Generic version (Short Form 2006). BMC Health Serv Res 2008;8:191. 33. Najjar S, Hamdan M, Euwema MC, et al. The Global Trigger Tool shows that one out of seven patients suffers harm in Palestinian hospitals: challenges for launching a strategic safety plan. Int J Qual Health Care 2013;25:640–7.
Downloaded from by guest on June 6, 2016
1. Crow R, Gage H, Hampson S, et al. The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature. Health Technol Assess 2002;6:1–244. 2. Keller S, O’Malley AJ, Hays RD, et al. Methods used to streamline the CAHPS Hospital Survey. Health Serv Res 2005;40:2057–77. 3. Jenkinson C, Coulter A, Bruster S. The Picker Patient Experience Questionnaire: development and validation using data from in-patient surveys in five countries. Int J Qual Health Care 2002;14:353–8. 4. Gerteis M, Edgman-Levitan S, Daley J, et al. Through the patient’s eyes: understanding and promoting patient-centered care. San Francisco, CA: Jossey-Bass, 1993. 5. Fitzpatrick R, Davey C, Buxton MJ, et al. Evaluating patient-based outcome measures for use in clinical trials. Health Technol Assess 1998;2: i–iv, 1–74. 6. Ward JK, Armitage G. Can patients report patient safety incidents in a hospital setting? A systematic review. BMJ Qual Saf 2012;21:685–99. 7. Massó Guijarro P, Aranaz Andrés JM, Mira JJ, et al. Adverse events in hospitals: the patient’s point of view. Qual Saf Health Care 2010;19:144–7. 8. King A, Daniels J, Lim J, et al. Time to listen: a review of methods to solicit patient reports of adverse events. Qual Saf Health Care 2010;19:148–57. 9. Bjertnaes O, Skudal KE, Iversen HH, et al. The Patient-Reported Incident in Hospital Instrument (PRIH-I): assessments of data quality, test-retest reliability and hospital-level reliability. BMJ Qual Saf 2013;22:743–51. 10. McEachan RR, Lawton RJ, O’Hara JK, et al.; on behalf of the Yorkshire Quality and Safety Research Group. Developing a reliable and valid patient measure of safety in hospitals (PMOS): a validation study. BMJ Qual Saf 2014;23:565–73. 11. Griffin FA, Resar RK. IHI Global Trigger Tool for Measuring Adverse Events, 2nd edn. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement, 2009. 12. Schildmeijer K, Nilsson L, Arestedt K, et al. Assessment of adverse events in medical care: lack of consistency between experienced teams using the global trigger tool. BMJ Qual Saf 2012;21:307–14. 13. Mattsson TO, Knudsen JL, Lauritsen J, et al. Assessment of the global trigger tool to measure, monitor and evaluate patient safety in cancer patients: reliability concerns are raised. BMJ Qual Saf 2013;22:571–9. 14. von Plessen C, Kodal AM, Anhøj J. Experiences with global trigger tool reviews in five Danish hospitals: an implementation study. BMJ Open 2012; 2: pii:e001324. 15. Schildmeijer K, Nilsson L, Perk J, et al. Strengths and weaknesses of working with the global trigger tool method for retrospective record review: focus group interviews with team members. BMJ Open 2013;3:e003131.
Bjertnaes et al.
