Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
ORIGINAL RESEARCH
Internal consistency, factor structure and construct validity of the French version of the Hospital Survey on Patient Safety Culture Thomas V Perneger,1 Anthony Staines,2,3 François Kundig3
1
Division of Clinical Epidemiology, University Hospitals of Geneva, Geneva, Switzerland 2 Institute for Education and Research on Social and Health Organizations (IFROSS), University of Lyon 3, Lyon, France 3 Clinical Quality and Patient Safety Unit, Hôpital Neuchâtelois, La Chaux-deFonds, Switzerland Correspondence to Dr Thomas V Perneger, Division of Clinical Epidemiology, University Hospitals of Geneva, CH-1211 Geneva, Switzerland; [email protected] Received 22 March 2013 Revised 30 October 2013 Accepted 4 November 2013 Published Online First 28 November 2013
To cite: Perneger TV, Staines A, Kundig F. BMJ Qual Saf 2014;23:389–397.
ABSTRACT Objective To assess the psychometric properties of the French-language version of the Hospital Survey on Patient Safety Culture (HSOPSC). Methods Data were obtained from a staff survey at a Swiss multisite hospital. We computed descriptive statistics and internal consistency coefficients, then conducted a confirmatory and exploratory factor analysis, and performed construct validity tests. Results 1171 staff members participated (response rate 74%). The internal consistency coefficients of the 12 dimension scores ranged from 0.57 to 0.86 (median 0.73). Confirmatory factor analysis indicated a reasonable but not perfect fit of the hypothesised measurement model (root mean square error of approximation 0.043, comparative fit index 0.89). Exploratory data analysis suggested 10 dimensions instead of 12, grouping items from teamwork across hospital units with those of hospital handoffs and transitions, and items from communication openness with those of feedback and communication about error. However, the loading pattern was clean: 41 of 42 main loadings exceeded 0.40, and only 3 of 378 crossloadings exceeded 0.30. All 10 process scores were higher among respondents who rated the global safety grade as ‘excellent’ or ‘very good’ rather than ‘good’, ‘fair’ or ‘poor’ (effect sizes 0.41–0.79, all p50% of items were missing and 1171 (74.0%) were included. By comparison with the hospital employment roster, the response rate was 73% for nurses, 92% for nurse managers, 80% for physicians, 56% for nursing aides, 68% for pharmacists and 55% for administrative employees and technicians. Questionnaire and key variables
We used the HSOPSC developed by the US Agency for Healthcare Research and Quality,4 translated into French by a Belgian team.18 We made no changes to this instrument. The 42 items were answered on a five-point agreement scale (from ‘strongly agree’ to ‘strongly disagree’) or a five-point frequency scale (from ‘always’ to ‘never’). The items measure seven unit-level process dimensions—supervisor expectations 390
and actions, organisational learning, teamwork within hospital units, communication openness, feedback and communication about error, non-punitive response to error and staffing— three hospital-level process dimensions—hospital management support, teamwork across hospital units and hospital handoffs and transitions—and two outcome dimensions—overall perception of safety and frequency of event reporting. In addition, the survey includes two single-item outcome variables that are not used in the computation of the 12 scores: a global safety grade between poor and excellent (‘Please give your area/unit in this hospital an overall grade on patient safety’) and the number of incidents reported in the past year (‘In the past 12 months, how many event reports have you filled out and submitted?’). Data collection
Hospital staff were sent a notification letter regarding the upcoming survey that described the data collection process and the objectives in February 2009. The questionnaire was distributed 1 week later with a cover letter. Filled questionnaires could be returned in ballot boxes located in several points in every building. Reminders were sent after 2 and 4 weeks. The survey also tested the effect of numbering questionnaires on response patterns; these effects were small to absent and have been published previously.19 Statistical analysis
The survey was intended for hospital employees who worked in contact with patients or who were in leadership positions; there was no determination of sample size related to the psychometric validation. However, the achieved sample size (>1000) is sufficient for a multivariate analysis of 42 items and an expected number of 12 latent variables, both to satisfy the rule of 10 observations per variable20 and more recent suggestions that a minimum of 300–400 observations is required.21 We reversed the coding of negatively worded items and computed dimension scores for each respondent both as the proportion of positive responses (‘agree/ strongly agree’ or ‘most of the time/always’) among the corresponding items and as simple averages of the responses on the original 1–5 scale. The scores of a given dimension computed using these two methods were highly correlated, and because the simple average method uses all available information, we performed the validation analyses using the simple average scores. We reported means, SDs, missing values and correlations between the scores computed by means of the two scoring methods. We examined the similarity of ratings obtained within sectors of activity (eg, medicine, surgery, paediatrics and pharmacy) within hospital sites. For this we used mixed linear models, with each score as the dependent variable and the hospital site and the
Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research activity sector nested within hospital site as random factors. We report the percentage of variance attributable to each random factor. To examine the internal consistency of the scales, we obtained Cronbach α coefficients and compared them with published coefficients of the original version and other published translated versions of the HSOPSC. We obtained the range and the median of published Cronbach α coefficients. To verify the internal structure of the instrument, we performed a CFA of the 42 items (recoded as needed), assigning items to their intended dimensions and allowing non-null correlations between the dimensions. To examine the fit of the measurement model, we obtained the root mean square error of approximation, the comparative fit index, the nonnormed fit index, the standardised root mean residual and the coefficient of determination. We compared our results with recommended cut-off values22 23 and with other published results. To examine possible alternative structures, we performed an EFA of the 42 items. We retained factors that had an eigenvalue >1. We obtained the Kaiser– Meyer–Olkin measure of sampling adequacy (>0.6 is recommended) and the Bartlett test of independence (rejection is recommended), and examined the primary loadings on the intended dimension (higher is better, >0.4 is recommended) as well as any crossloadings >0.3 (fewer is better). To explore construct validity, we examined mean scores of the 10 process dimensions in subgroups of respondents defined by the two single-item outcome variables. Both outcome items were dichotomised: safety grade as ‘excellent/very good’ versus the rest, number of incidents reported as any versus none. The differences between score means were tested by t tests. In all cases, we expected a significant difference, with a higher mean score among those who gave a high safety grade and those who reported at least one incident. In addition, we obtained an effect size for each difference, that is, the difference between means divided by the pooled SD of the scores in the two groups. In all cases, we expected a medium-sized difference,24 that is, approximately 0.5. Finally, we performed an EFA of the 10 process dimension scores. We expected distinct groupings of the unit-level process dimensions and the hospitallevel process dimensions because these sets of dimensions are influenced by unit-level and hospital-level management practices and are therefore expected to covary. Analyses were conducted using SPSS V.18, except CFA, which was performed with Stata V.12. RESULTS Participation
A majority of the 1171 respondents were women, and more than half were under 45 years of age (table 1). Nurses were the largest professional group, and about Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
one-tenth of the respondents were physicians. About 90% of the respondents worked directly with patients. The two main hospital facilities represented 76% of the respondents. HSOPSC scores
Among the respondents, 5.3% (N=60) gave their work unit a global safety grade of ‘excellent’, 46.9% (N=530) said ‘very good’, 39.2% (N=443) said ‘acceptable’, 3.7% (N=42) said ‘poor’ and 4.8% (N=54) said ‘failing’. The majority (61.1%, N=690) of respondents had not reported any event related to patient safety in the previous year; 25.0% (N=282) reported 1 or 2 events, 8.5% (N=96) reported 3–5 events, 3.4% (N=38) reported 6–10 events and 2.1% (N=23) reported 11 or more events. The mean percentage of positive responses ranged from 28.1% (hospital management support) to 79.4% (teamwork within hospital units) (table 2). When the scores were computed as simple means on a scale from 1 to 5, the pattern of the averages was similar. The summary scores obtained by the two computation methods from the same items were highly correlated: the Pearson correlation coefficients ranged from 0.82 (hospital management support) to 0.90 (feedback and Table 1 Demographic and professional characteristics of 1171 employees of a Swiss hospital Characteristics Women, N (%) Age groups, N (%) Up to 35 years 36–45 years 46–55 years 55 years and older Occupational group Nurse Nursing assistant Physician Other healthcare Administrative Technical Other Hospital sector Medicine and paediatrics Surgery and gynaecology Technical (pharmacy, operating rooms, imaging, laboratories) Intensive care, emergency and anaesthesia Several or other Contact with patients, N (%) Hospital facilities A B C, D, E, F, G or several facilities
N (%) 910 (79.1) 329 360 341 126
(28.5) (31.1) (29.5) (10.9)
493 124 118 94 91 97 136
(42.8) (10.8) (10.2) (8.2) (7.9) (8.4) (11.8)
321 (27.9) 211 (18.3) 196 (17.0) 139 (12.1) 285 (24.7) 1038 (89.9) 465 (40.4) 410 (35.6) 276 (24.0)
391
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research Table 2 Means (SD) of dimensions measured by the Hospital Survey on Patient Safety Culture using two computation methods (percentage of positive items and mean score), correlation between the two methods and the proportion of variance attributable to hospital sites and activity sectors within hospital sites among 1171 employees of a Swiss multisite hospital Percent variance attributable to
Outcome dimensions Overall perception of safety Frequency of event reporting Process dimensions (unit level): Supervisor expectations and actions Organisational learning Teamwork within hospital units Communication openness Feedback and communication about error Non-punitive response to error Staffing Process dimensions (hospital level) Hospital management support Teamwork across hospital units Hospital handoffs and transitions
Number of items
Missing scores, N (%)
Percentage of positive responses, mean (SD)
Mean score (1–5), mean (SD)
Correlation between the percentage and the mean score
Hospital site
Sector within hospital site
4 3
4 (0.3) 133 (11.4)
45.4 (33.4) 42.1 (42.1)
3.20 (0.74) 3.22 (0.99)
0.87 0.87
0.0 0.0
14.2 4.1
4
7 (0.6)
64.0 (32.4)
3.72 (0.68)
0.87
4.1
5.4
3 4 3 3
3 (0.3) 2 (0.2) 11 (0.9) 12 (1.0)
66.2 (33.3) 79.4 (29.9) 68.0 (33.5) 57.2 (38.9)
3.73 (0.61) 3.96 (0.68) 3.84 (0.76) 3.59 (0.87)
0.86 0.86 0.89 0.90
0.0 0.2 6.1 0.0
4.7 9.0 3.6 6.3
3 4
6 (0.5) 3 (0.3)
45.1 (35.5) 50.6 (33.4)
3.30 (0.74) 3.36 (0.75)
0.86 0.89
3.3 0.0
3.8 17.1
3 4 4
18 (1.5) 19 (1.6) 26 (2.2)
28.1 (37.1) 42.7 (34.4) 35.3 (34.4)
2.92 (0.82) 3.22 (0.62) 3.02 (0.70)
0.82 0.87 0.85
2.5 2.6 1.1
4.5 0.1 6.9
communication about error). Missing score values were low for all scales except frequency of event reporting, for which 11% of respondents had missing values. For most scores, a greater proportion of variance was attributable to activity sectors (within hospital sites) than to hospital sites (table 2). The dimensions that had the highest levels of agreement within activity sectors were staffing (17.1% of variance) and overall perception of safety (14.2%).
instrument, in which all dimensions were allowed to have non-zero correlations (table 4). The root mean square error of approximation, the standardised root mean residual and the coefficient of determination achieved the recommended values, but the comparative fit index and the non-normed fit index were both below the recommended thresholds. These results were less favourable than those of the original instrument but in line with the performance of other translations.
Internal consistency
Exploratory factor analysis
Cronbach α coefficients ranged from 0.57 (organisational learning) to 0.86 (frequency of event reporting) (table 3). The median value was 0.73, and five coefficients were below 0.70. Most coefficients were lower in the French version than in the original US version, but the difference was 0.02 or less for 5 of the 12 dimensions. The largest differences were observed for organisational learning (0.57 vs 0.76) and nonpunitive response to error (0.60 vs 0.79). The α coefficients of these two dimensions were also lower than the median of values reported for other translations, whereas the other 10 coefficients were as high as or higher than the median of published results.
This analysis of the 42 HSOPSC items yielded 10 dimensions with an eigenvalue greater than 1 (table 5). These dimensions captured 58% of the total variance, and the Kaiser–Meyer–Olkin measure of sampling adequacy was 0.88. The structure identified 10 factors rather than 12 as items belonging to the scales communication openness and feedback and communication about error loaded on the same factor, and similarly for items that belonged to teamwork across hospital units and hospital handoffs and transitions. Besides this, the loading structure was clean: 39 of the 42 primary loadings were ≥0.50, and 41 of 42 were ≥0.40. Of 378 cross-loadings (42 items, 9 secondary factors), only 3 (0.8%) exceeded 0.3 and none exceeded 0.5. Only one cross-loading was higher than the primary loading: the item ‘staff are afraid to ask questions when something does not seem right’, which belongs to the
Confirmatory factor analysis
We estimated a measurement model for the 12 latent dimensions as specified by the authors of the 392
Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research Table 3 Internal consistency (Cronbach α) of dimension scores measured by the Hospital Survey on Patient Safety Culture in this study and other published sources Other studies7–14
25
Dimension
This study
Original4
Range
Median
Overall perception of safety Frequency of event reporting Supervisor expectations and actions Organisational learning Teamwork within hospital units Communication openness Feedback and communication about error Non-punitive response to error Staffing Hospital management support Teamwork across hospital units Hospital handoffs and transitions Median α
0.68 0.86 0.75 0.57 0.80 0.67 0.77 0.60 0.61 0.79 0.71 0.78 0.73
0.74 0.84 0.75 0.76 0.83 0.72 0.78 0.79 0.63 0.83 0.80 0.80 0.78
0.26–0.76 0.79–0.88 0.57–0.78 0.50–0.68 0.66–0.83 0.46–0.72 0.46–0.82 0.53–0.72 0.44–0.65 0.59–0.83 0.59–0.76 0.65–0.76 0.60–0.74
0.62 0.83 0.70 0.64 0.77 0.64 0.76 0.69 0.58 0.69 0.70 0.72 0.69
communication openness scale (loading 0.36), loaded also on the non-punitive response to error scale (loading 0.47). Two items from the staffing scale had moderate cross-loadings on the overall perceptions of safety scale. Associations with single-item outcome variables
All 10 safety culture process dimension scores were significantly higher among respondents who gave a favourable safety grade (table 6). The effect sizes (ie, differences divided by the pooled SDs) ranged from 0.41 to 0.79. In contrast, effect sizes were small and for the most part non-significant when respondents who had reported at least one incident in the past year were compared with those who had not. The only significant differences (for staffing and hospital management support) were in a direction opposite to that postulated by theory: respondents who had reported incidents had lower scores on these dimensions. EFA of process dimension scores
The analysis of the 10 process scores confirmed the postulated model as the seven unit-level process Table 4 sources
dimensions formed one factor and the three hospitallevel process dimensions formed another (table 7). These dimensions captured 50% of the total variance, and the Kaiser–Meyer–Olkin measure of sampling adequacy was 0.82 (ie, higher than the recommended minimum of 0.6).
DISCUSSION This study examined the psychometric properties of the French version of HSOPSC. Overall the performance of the instrument was less satisfactory than that of the original US version. Our results, as well as those published by others, raise several questions regarding the HSOPSC and its underlying theoretical model. The acceptability of the instrument and of the data collection methods was good as attested by a participation rate of 74%. Missing data were rare for all dimensions except frequency of event reporting, for which missing scores represented over 11%. One possibility is that some respondents were not familiar with incident reporting, especially as a formal incident reporting system was not in place in all parts of this
Confirmatory factor analysis of the 42 items of the Hospital Survey on Patient Safety Culture in this study and other published Recommended criteria of good fit
Other studies12–15
Root mean square error of approximation Comparative fit index Non-normed fit index Standardised root mean residual Goodness of fit index
This study
Original4
Range
Median
Kline22
Hu and Bentler23
0.043 0.89 0.88 0.046 >0.99
0.04 0.94 0.93 0.04 –
0.033–0.047 0.89–0.99 0.88–0.98 0.044–0.05 0.88–0.99
0.043 0.91 0.90 0.047 0.94
0.90 >0.90 0.95
0.95 >0.95 0.3 of the 42 items of the Hospital Survey on Patient Safety Factors 1
Cooperation across units (F4) Units work well together (F10) Units do not coordinate well (F2R) Unpleasant to work with staff (F6R) Things fall between cracks (F3R) Information often lost (F5R) Problems in exchange of information (F7R) Shift changes problematic (F11R) Feedback about changes (C1) Informed about errors (C3) Discuss ways to prevent errors (C5) Staff speak freely (C2) Feel free to question authority (C4) Afraid to ask questions (C6R) People support one another (A1) We work together as a team (A3) People treat each other with respect (A4) When busy, others help out (A11) Incident caught and corrected (D1) Mistake has no potential to harm (D2) Mistake could harm (D3) Supervisor says a good word (B1) Supervisor considers staff suggestions (B2) Work faster, take shortcuts (B3R) Supervisor overlooks safety problems (B4R) Patient safety never sacrificed (A15) Systems good at preventing errors (A18) It’s just by chance (A10R) We have safety problems (A17R) Management provides work climate (F1) Patient safety is top priority (F8) Interested only after adverse event (F9R) Mistakes held against them (A8R) Person written up (A12R) Mistakes kept in personnel file (A16R) Doing things to improve safety (A6) Mistakes have led to positive changes (A9) Evaluate effectiveness (A13) Enough staff to handle workload (A2) Staff work longer hours (A5R) Use more agency/temporary staff (A7R) Do too much too quickly (A14R)
2
3
5
6
7
8
9
10
0.65 0.60 0.62 0.60 0.66 0.65 0.76 0.49 0.64 0.68 0.60 0.62 0.68 0.36
0.47 0.70 0.81 0.78 0.64
hospital at the time of the survey. Another possible reason is that the questions ask about what ‘is done’ in three hypothetical situations and not about what the respondent would do. Some respondents may have felt that they cannot say what other people do in a given situation. The internal consistency of 11 of the 12 scales conformed to the standards set in the original 394
4
0.82 0.89 0.84 0.59 0.73 0.68 0.74 0.56 0.69 0.53 0.72 0.73 0.81 0.68 0.65 0.41 0.70 0.62 0.59 0.65 0.32
0.35
0.58 0.67 0.62 0.50
publication,4 where Cronbach α coefficients >0.60 were considered acceptable. However, other sources would demand higher standards—for example, Nunnally and Bernstein recommend a range of 0.7– 0.8.25 In our case, five α coefficients were below 0.7. This may reflect shifts in meaning due to the translation. Another possibility is that true variance was lower in our sample, taken from one hospital, than in
Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research Table 6 Mean scores of process dimensions of the Hospital Survey on Patient Safety Culture in subgroups of respondents who rated the safety grade as higher or lower and among those who have reported or not at least one incident in the past 12 months Safety grade
Supervisor expectations and actions Organisational learning Teamwork within hospital units Communication openness Feedback and communication about error Non-punitive response to error Staffing Hospital management support Teamwork across hospital units Hospital handoffs and transitions
Reported an incident in the past 12 months
Excellent or very good
Good, fair or poor
p Value
Effect size
One or more
None
p Value
Effect size
3.86 3.90 4.12 4.05 3.86
(0.60) (0.52) (0.61) (0.73) (0.80)
3.58 (0.73) 3.55 (0.63) 3.79 (0.70) 3.62 (0.74) 3.27 (0.84)
ORIGINAL RESEARCH
Internal consistency, factor structure and construct validity of the French version of the Hospital Survey on Patient Safety Culture Thomas V Perneger,1 Anthony Staines,2,3 François Kundig3
1
Division of Clinical Epidemiology, University Hospitals of Geneva, Geneva, Switzerland 2 Institute for Education and Research on Social and Health Organizations (IFROSS), University of Lyon 3, Lyon, France 3 Clinical Quality and Patient Safety Unit, Hôpital Neuchâtelois, La Chaux-deFonds, Switzerland Correspondence to Dr Thomas V Perneger, Division of Clinical Epidemiology, University Hospitals of Geneva, CH-1211 Geneva, Switzerland; [email protected] Received 22 March 2013 Revised 30 October 2013 Accepted 4 November 2013 Published Online First 28 November 2013
To cite: Perneger TV, Staines A, Kundig F. BMJ Qual Saf 2014;23:389–397.
ABSTRACT Objective To assess the psychometric properties of the French-language version of the Hospital Survey on Patient Safety Culture (HSOPSC). Methods Data were obtained from a staff survey at a Swiss multisite hospital. We computed descriptive statistics and internal consistency coefficients, then conducted a confirmatory and exploratory factor analysis, and performed construct validity tests. Results 1171 staff members participated (response rate 74%). The internal consistency coefficients of the 12 dimension scores ranged from 0.57 to 0.86 (median 0.73). Confirmatory factor analysis indicated a reasonable but not perfect fit of the hypothesised measurement model (root mean square error of approximation 0.043, comparative fit index 0.89). Exploratory data analysis suggested 10 dimensions instead of 12, grouping items from teamwork across hospital units with those of hospital handoffs and transitions, and items from communication openness with those of feedback and communication about error. However, the loading pattern was clean: 41 of 42 main loadings exceeded 0.40, and only 3 of 378 crossloadings exceeded 0.30. All 10 process scores were higher among respondents who rated the global safety grade as ‘excellent’ or ‘very good’ rather than ‘good’, ‘fair’ or ‘poor’ (effect sizes 0.41–0.79, all p50% of items were missing and 1171 (74.0%) were included. By comparison with the hospital employment roster, the response rate was 73% for nurses, 92% for nurse managers, 80% for physicians, 56% for nursing aides, 68% for pharmacists and 55% for administrative employees and technicians. Questionnaire and key variables
We used the HSOPSC developed by the US Agency for Healthcare Research and Quality,4 translated into French by a Belgian team.18 We made no changes to this instrument. The 42 items were answered on a five-point agreement scale (from ‘strongly agree’ to ‘strongly disagree’) or a five-point frequency scale (from ‘always’ to ‘never’). The items measure seven unit-level process dimensions—supervisor expectations 390
and actions, organisational learning, teamwork within hospital units, communication openness, feedback and communication about error, non-punitive response to error and staffing— three hospital-level process dimensions—hospital management support, teamwork across hospital units and hospital handoffs and transitions—and two outcome dimensions—overall perception of safety and frequency of event reporting. In addition, the survey includes two single-item outcome variables that are not used in the computation of the 12 scores: a global safety grade between poor and excellent (‘Please give your area/unit in this hospital an overall grade on patient safety’) and the number of incidents reported in the past year (‘In the past 12 months, how many event reports have you filled out and submitted?’). Data collection
Hospital staff were sent a notification letter regarding the upcoming survey that described the data collection process and the objectives in February 2009. The questionnaire was distributed 1 week later with a cover letter. Filled questionnaires could be returned in ballot boxes located in several points in every building. Reminders were sent after 2 and 4 weeks. The survey also tested the effect of numbering questionnaires on response patterns; these effects were small to absent and have been published previously.19 Statistical analysis
The survey was intended for hospital employees who worked in contact with patients or who were in leadership positions; there was no determination of sample size related to the psychometric validation. However, the achieved sample size (>1000) is sufficient for a multivariate analysis of 42 items and an expected number of 12 latent variables, both to satisfy the rule of 10 observations per variable20 and more recent suggestions that a minimum of 300–400 observations is required.21 We reversed the coding of negatively worded items and computed dimension scores for each respondent both as the proportion of positive responses (‘agree/ strongly agree’ or ‘most of the time/always’) among the corresponding items and as simple averages of the responses on the original 1–5 scale. The scores of a given dimension computed using these two methods were highly correlated, and because the simple average method uses all available information, we performed the validation analyses using the simple average scores. We reported means, SDs, missing values and correlations between the scores computed by means of the two scoring methods. We examined the similarity of ratings obtained within sectors of activity (eg, medicine, surgery, paediatrics and pharmacy) within hospital sites. For this we used mixed linear models, with each score as the dependent variable and the hospital site and the
Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research activity sector nested within hospital site as random factors. We report the percentage of variance attributable to each random factor. To examine the internal consistency of the scales, we obtained Cronbach α coefficients and compared them with published coefficients of the original version and other published translated versions of the HSOPSC. We obtained the range and the median of published Cronbach α coefficients. To verify the internal structure of the instrument, we performed a CFA of the 42 items (recoded as needed), assigning items to their intended dimensions and allowing non-null correlations between the dimensions. To examine the fit of the measurement model, we obtained the root mean square error of approximation, the comparative fit index, the nonnormed fit index, the standardised root mean residual and the coefficient of determination. We compared our results with recommended cut-off values22 23 and with other published results. To examine possible alternative structures, we performed an EFA of the 42 items. We retained factors that had an eigenvalue >1. We obtained the Kaiser– Meyer–Olkin measure of sampling adequacy (>0.6 is recommended) and the Bartlett test of independence (rejection is recommended), and examined the primary loadings on the intended dimension (higher is better, >0.4 is recommended) as well as any crossloadings >0.3 (fewer is better). To explore construct validity, we examined mean scores of the 10 process dimensions in subgroups of respondents defined by the two single-item outcome variables. Both outcome items were dichotomised: safety grade as ‘excellent/very good’ versus the rest, number of incidents reported as any versus none. The differences between score means were tested by t tests. In all cases, we expected a significant difference, with a higher mean score among those who gave a high safety grade and those who reported at least one incident. In addition, we obtained an effect size for each difference, that is, the difference between means divided by the pooled SD of the scores in the two groups. In all cases, we expected a medium-sized difference,24 that is, approximately 0.5. Finally, we performed an EFA of the 10 process dimension scores. We expected distinct groupings of the unit-level process dimensions and the hospitallevel process dimensions because these sets of dimensions are influenced by unit-level and hospital-level management practices and are therefore expected to covary. Analyses were conducted using SPSS V.18, except CFA, which was performed with Stata V.12. RESULTS Participation
A majority of the 1171 respondents were women, and more than half were under 45 years of age (table 1). Nurses were the largest professional group, and about Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
one-tenth of the respondents were physicians. About 90% of the respondents worked directly with patients. The two main hospital facilities represented 76% of the respondents. HSOPSC scores
Among the respondents, 5.3% (N=60) gave their work unit a global safety grade of ‘excellent’, 46.9% (N=530) said ‘very good’, 39.2% (N=443) said ‘acceptable’, 3.7% (N=42) said ‘poor’ and 4.8% (N=54) said ‘failing’. The majority (61.1%, N=690) of respondents had not reported any event related to patient safety in the previous year; 25.0% (N=282) reported 1 or 2 events, 8.5% (N=96) reported 3–5 events, 3.4% (N=38) reported 6–10 events and 2.1% (N=23) reported 11 or more events. The mean percentage of positive responses ranged from 28.1% (hospital management support) to 79.4% (teamwork within hospital units) (table 2). When the scores were computed as simple means on a scale from 1 to 5, the pattern of the averages was similar. The summary scores obtained by the two computation methods from the same items were highly correlated: the Pearson correlation coefficients ranged from 0.82 (hospital management support) to 0.90 (feedback and Table 1 Demographic and professional characteristics of 1171 employees of a Swiss hospital Characteristics Women, N (%) Age groups, N (%) Up to 35 years 36–45 years 46–55 years 55 years and older Occupational group Nurse Nursing assistant Physician Other healthcare Administrative Technical Other Hospital sector Medicine and paediatrics Surgery and gynaecology Technical (pharmacy, operating rooms, imaging, laboratories) Intensive care, emergency and anaesthesia Several or other Contact with patients, N (%) Hospital facilities A B C, D, E, F, G or several facilities
N (%) 910 (79.1) 329 360 341 126
(28.5) (31.1) (29.5) (10.9)
493 124 118 94 91 97 136
(42.8) (10.8) (10.2) (8.2) (7.9) (8.4) (11.8)
321 (27.9) 211 (18.3) 196 (17.0) 139 (12.1) 285 (24.7) 1038 (89.9) 465 (40.4) 410 (35.6) 276 (24.0)
391
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research Table 2 Means (SD) of dimensions measured by the Hospital Survey on Patient Safety Culture using two computation methods (percentage of positive items and mean score), correlation between the two methods and the proportion of variance attributable to hospital sites and activity sectors within hospital sites among 1171 employees of a Swiss multisite hospital Percent variance attributable to
Outcome dimensions Overall perception of safety Frequency of event reporting Process dimensions (unit level): Supervisor expectations and actions Organisational learning Teamwork within hospital units Communication openness Feedback and communication about error Non-punitive response to error Staffing Process dimensions (hospital level) Hospital management support Teamwork across hospital units Hospital handoffs and transitions
Number of items
Missing scores, N (%)
Percentage of positive responses, mean (SD)
Mean score (1–5), mean (SD)
Correlation between the percentage and the mean score
Hospital site
Sector within hospital site
4 3
4 (0.3) 133 (11.4)
45.4 (33.4) 42.1 (42.1)
3.20 (0.74) 3.22 (0.99)
0.87 0.87
0.0 0.0
14.2 4.1
4
7 (0.6)
64.0 (32.4)
3.72 (0.68)
0.87
4.1
5.4
3 4 3 3
3 (0.3) 2 (0.2) 11 (0.9) 12 (1.0)
66.2 (33.3) 79.4 (29.9) 68.0 (33.5) 57.2 (38.9)
3.73 (0.61) 3.96 (0.68) 3.84 (0.76) 3.59 (0.87)
0.86 0.86 0.89 0.90
0.0 0.2 6.1 0.0
4.7 9.0 3.6 6.3
3 4
6 (0.5) 3 (0.3)
45.1 (35.5) 50.6 (33.4)
3.30 (0.74) 3.36 (0.75)
0.86 0.89
3.3 0.0
3.8 17.1
3 4 4
18 (1.5) 19 (1.6) 26 (2.2)
28.1 (37.1) 42.7 (34.4) 35.3 (34.4)
2.92 (0.82) 3.22 (0.62) 3.02 (0.70)
0.82 0.87 0.85
2.5 2.6 1.1
4.5 0.1 6.9
communication about error). Missing score values were low for all scales except frequency of event reporting, for which 11% of respondents had missing values. For most scores, a greater proportion of variance was attributable to activity sectors (within hospital sites) than to hospital sites (table 2). The dimensions that had the highest levels of agreement within activity sectors were staffing (17.1% of variance) and overall perception of safety (14.2%).
instrument, in which all dimensions were allowed to have non-zero correlations (table 4). The root mean square error of approximation, the standardised root mean residual and the coefficient of determination achieved the recommended values, but the comparative fit index and the non-normed fit index were both below the recommended thresholds. These results were less favourable than those of the original instrument but in line with the performance of other translations.
Internal consistency
Exploratory factor analysis
Cronbach α coefficients ranged from 0.57 (organisational learning) to 0.86 (frequency of event reporting) (table 3). The median value was 0.73, and five coefficients were below 0.70. Most coefficients were lower in the French version than in the original US version, but the difference was 0.02 or less for 5 of the 12 dimensions. The largest differences were observed for organisational learning (0.57 vs 0.76) and nonpunitive response to error (0.60 vs 0.79). The α coefficients of these two dimensions were also lower than the median of values reported for other translations, whereas the other 10 coefficients were as high as or higher than the median of published results.
This analysis of the 42 HSOPSC items yielded 10 dimensions with an eigenvalue greater than 1 (table 5). These dimensions captured 58% of the total variance, and the Kaiser–Meyer–Olkin measure of sampling adequacy was 0.88. The structure identified 10 factors rather than 12 as items belonging to the scales communication openness and feedback and communication about error loaded on the same factor, and similarly for items that belonged to teamwork across hospital units and hospital handoffs and transitions. Besides this, the loading structure was clean: 39 of the 42 primary loadings were ≥0.50, and 41 of 42 were ≥0.40. Of 378 cross-loadings (42 items, 9 secondary factors), only 3 (0.8%) exceeded 0.3 and none exceeded 0.5. Only one cross-loading was higher than the primary loading: the item ‘staff are afraid to ask questions when something does not seem right’, which belongs to the
Confirmatory factor analysis
We estimated a measurement model for the 12 latent dimensions as specified by the authors of the 392
Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research Table 3 Internal consistency (Cronbach α) of dimension scores measured by the Hospital Survey on Patient Safety Culture in this study and other published sources Other studies7–14
25
Dimension
This study
Original4
Range
Median
Overall perception of safety Frequency of event reporting Supervisor expectations and actions Organisational learning Teamwork within hospital units Communication openness Feedback and communication about error Non-punitive response to error Staffing Hospital management support Teamwork across hospital units Hospital handoffs and transitions Median α
0.68 0.86 0.75 0.57 0.80 0.67 0.77 0.60 0.61 0.79 0.71 0.78 0.73
0.74 0.84 0.75 0.76 0.83 0.72 0.78 0.79 0.63 0.83 0.80 0.80 0.78
0.26–0.76 0.79–0.88 0.57–0.78 0.50–0.68 0.66–0.83 0.46–0.72 0.46–0.82 0.53–0.72 0.44–0.65 0.59–0.83 0.59–0.76 0.65–0.76 0.60–0.74
0.62 0.83 0.70 0.64 0.77 0.64 0.76 0.69 0.58 0.69 0.70 0.72 0.69
communication openness scale (loading 0.36), loaded also on the non-punitive response to error scale (loading 0.47). Two items from the staffing scale had moderate cross-loadings on the overall perceptions of safety scale. Associations with single-item outcome variables
All 10 safety culture process dimension scores were significantly higher among respondents who gave a favourable safety grade (table 6). The effect sizes (ie, differences divided by the pooled SDs) ranged from 0.41 to 0.79. In contrast, effect sizes were small and for the most part non-significant when respondents who had reported at least one incident in the past year were compared with those who had not. The only significant differences (for staffing and hospital management support) were in a direction opposite to that postulated by theory: respondents who had reported incidents had lower scores on these dimensions. EFA of process dimension scores
The analysis of the 10 process scores confirmed the postulated model as the seven unit-level process Table 4 sources
dimensions formed one factor and the three hospitallevel process dimensions formed another (table 7). These dimensions captured 50% of the total variance, and the Kaiser–Meyer–Olkin measure of sampling adequacy was 0.82 (ie, higher than the recommended minimum of 0.6).
DISCUSSION This study examined the psychometric properties of the French version of HSOPSC. Overall the performance of the instrument was less satisfactory than that of the original US version. Our results, as well as those published by others, raise several questions regarding the HSOPSC and its underlying theoretical model. The acceptability of the instrument and of the data collection methods was good as attested by a participation rate of 74%. Missing data were rare for all dimensions except frequency of event reporting, for which missing scores represented over 11%. One possibility is that some respondents were not familiar with incident reporting, especially as a formal incident reporting system was not in place in all parts of this
Confirmatory factor analysis of the 42 items of the Hospital Survey on Patient Safety Culture in this study and other published Recommended criteria of good fit
Other studies12–15
Root mean square error of approximation Comparative fit index Non-normed fit index Standardised root mean residual Goodness of fit index
This study
Original4
Range
Median
Kline22
Hu and Bentler23
0.043 0.89 0.88 0.046 >0.99
0.04 0.94 0.93 0.04 –
0.033–0.047 0.89–0.99 0.88–0.98 0.044–0.05 0.88–0.99
0.043 0.91 0.90 0.047 0.94
0.90 >0.90 0.95
0.95 >0.95 0.3 of the 42 items of the Hospital Survey on Patient Safety Factors 1
Cooperation across units (F4) Units work well together (F10) Units do not coordinate well (F2R) Unpleasant to work with staff (F6R) Things fall between cracks (F3R) Information often lost (F5R) Problems in exchange of information (F7R) Shift changes problematic (F11R) Feedback about changes (C1) Informed about errors (C3) Discuss ways to prevent errors (C5) Staff speak freely (C2) Feel free to question authority (C4) Afraid to ask questions (C6R) People support one another (A1) We work together as a team (A3) People treat each other with respect (A4) When busy, others help out (A11) Incident caught and corrected (D1) Mistake has no potential to harm (D2) Mistake could harm (D3) Supervisor says a good word (B1) Supervisor considers staff suggestions (B2) Work faster, take shortcuts (B3R) Supervisor overlooks safety problems (B4R) Patient safety never sacrificed (A15) Systems good at preventing errors (A18) It’s just by chance (A10R) We have safety problems (A17R) Management provides work climate (F1) Patient safety is top priority (F8) Interested only after adverse event (F9R) Mistakes held against them (A8R) Person written up (A12R) Mistakes kept in personnel file (A16R) Doing things to improve safety (A6) Mistakes have led to positive changes (A9) Evaluate effectiveness (A13) Enough staff to handle workload (A2) Staff work longer hours (A5R) Use more agency/temporary staff (A7R) Do too much too quickly (A14R)
2
3
5
6
7
8
9
10
0.65 0.60 0.62 0.60 0.66 0.65 0.76 0.49 0.64 0.68 0.60 0.62 0.68 0.36
0.47 0.70 0.81 0.78 0.64
hospital at the time of the survey. Another possible reason is that the questions ask about what ‘is done’ in three hypothetical situations and not about what the respondent would do. Some respondents may have felt that they cannot say what other people do in a given situation. The internal consistency of 11 of the 12 scales conformed to the standards set in the original 394
4
0.82 0.89 0.84 0.59 0.73 0.68 0.74 0.56 0.69 0.53 0.72 0.73 0.81 0.68 0.65 0.41 0.70 0.62 0.59 0.65 0.32
0.35
0.58 0.67 0.62 0.50
publication,4 where Cronbach α coefficients >0.60 were considered acceptable. However, other sources would demand higher standards—for example, Nunnally and Bernstein recommend a range of 0.7– 0.8.25 In our case, five α coefficients were below 0.7. This may reflect shifts in meaning due to the translation. Another possibility is that true variance was lower in our sample, taken from one hospital, than in
Perneger TV, et al. BMJ Qual Saf 2014;23:389–397. doi:10.1136/bmjqs-2013-002024
Downloaded from http://qualitysafety.bmj.com/ on July 11, 2015 - Published by group.bmj.com
Original research Table 6 Mean scores of process dimensions of the Hospital Survey on Patient Safety Culture in subgroups of respondents who rated the safety grade as higher or lower and among those who have reported or not at least one incident in the past 12 months Safety grade
Supervisor expectations and actions Organisational learning Teamwork within hospital units Communication openness Feedback and communication about error Non-punitive response to error Staffing Hospital management support Teamwork across hospital units Hospital handoffs and transitions
Reported an incident in the past 12 months
Excellent or very good
Good, fair or poor
p Value
Effect size
One or more
None
p Value
Effect size
3.86 3.90 4.12 4.05 3.86
(0.60) (0.52) (0.61) (0.73) (0.80)
3.58 (0.73) 3.55 (0.63) 3.79 (0.70) 3.62 (0.74) 3.27 (0.84)
