Int.J. Behav. Med. DOI 10.1007/s12529-014-9420-8
Investigating Variations in Implementation Fidelity of an Organizational-Level Occupational Health Intervention Hanna Augustsson & Ulrica von Thiele Schwarz & Terese Stenfors-Hayes & Henna Hasson
# International Society of Behavioral Medicine 2014
Abstract Background The workplace has been suggested as an important arena for health promotion, but little is known about how the organizational setting influences the implementation of interventions. Purpose The aims of this study are to evaluate implementation fidelity in an organizational-level occupational health intervention and to investigate possible explanations for variations in fidelity between intervention units. Method The intervention consisted of an integration of health promotion, occupational health and safety, and a system for continuous improvements (Kaizen) and was conducted in a quasi-experimental design at a Swedish hospital. Implementation fidelity was evaluated with the Conceptual Framework for Implementation Fidelity and implementation H. Augustsson (*) : U. von Thiele Schwarz : H. Hasson Department of Learning, Informatics, Management and Ethics, Medical Management Centre, Karolinska Institutet, Tomtebodavägen 18A, 171 77 Stockholm, Sweden e-mail: [email protected] U. von Thiele Schwarz e-mail: [email protected]
factors used to investigate variations in fidelity with the Framework for Evaluating Organizational-level Interventions. A multi-method approach including interviews, Kaizen notes, and questionnaires was applied. Results Implementation fidelity differed between units even though the intervention was introduced and supported in the same way. Important differences in all elements proposed in the model for evaluating organizational-level interventions, i.e., context, intervention, and mental models, were found to explain the differences in fidelity. Conclusion Implementation strategies may need to be adapted depending on the local context. Implementation fidelity, as well as pre-intervention implementation elements, is likely to affect the implementation success and needs to be assessed in intervention research. The high variation in fidelity across the units indicates the need for adjustments to the type of designs used to assess the effects of interventions. Thus, rather than using designs that aim to control variation, it may be necessary to use those that aim at exploring and explaining variation, such as adapted study designs. Keywords Health promotion . Process evaluation . Adherence . Kaizen
H. Hasson e-mail: [email protected] U. von Thiele Schwarz Department of Psychology, Stockholm University, 106 91 Stockholm, Sweden T. Stenfors-Hayes Department of Learning, Informatics, Management and Ethics, Centre of Learning and Knowledge, Karolinska Institutet, 171 77 Stockholm, Sweden e-mail: [email protected] H. Hasson Centre for Epidemiology and Community Medicine, Stockholm County Council, Stockholm, Sweden
Introduction There is no other setting where adults spend so much of their waking hours as the workplace. This has made the workplace an attractive arena for reaching adults with public health interventions and health promotion [1]. Using the workplace as an arena for health promotion has been suggested to have a number of advantages [2], including reaching a greater number of individuals compared to individual-based settings [3], providing opportunities for offering social support [4], and providing cues and reinforcement to help maintain behavior
Int.J. Behav. Med.
change [5]. In addition, the advantages of the workplace as an arena for health promotion can be further enhanced by utilizing the work environment and work conditions as such to improve health, i.e., organizational-level occupational health interventions [6]. In recent years, there has been increasing interest in organizational-level occupational health interventions aimed at improving psychosocial working conditions and employee health and well-being [7]. Organizational-level occupational health interventions have also been emphasized in current European legislation [8] as the preferred way of improving employee health and working conditions. However, these interventions have shown inconsistent results on employee health and wellbeing [9, 10]. Some organizational-level interventions have shown positive effects [11–13], while others have failed to do so [14, 15]. Several authors have highlighted the need for process evaluations in order to understand these inconsistencies [16–18]. Process data on organizational-level interventions are particularly limited [19] and have mainly been based on anecdotal data rather than being subjected to structural analyses [19–22]. One reason for this may be a lack of models and frameworks for use in guiding process evaluations of organizational-level interventions. Nielsen and Randall [18] recently proposed an evidence-based three-level framework on how processes of organizational-level interventions can be evaluated, called Framework for Evaluating Organizational-level Interventions. Two of the elements in the framework, context and participants’ mental models, represent the factors that may moderate or mediate the link between the intervention exposure and its outcomes. The third element covers the intervention design and implementation strategies that determine the level of intervention exposure that can be achieved. This element emphasizes the importance of documenting and measuring what is actually implemented, since this can highlight discrepancies between the planned intervention and its actual implementation. Within evaluation research, this has been referred to as implementation fidelity, which is the degree to which an intervention has been implemented as intended by the program developers [23, 24]. The most comprehensive framework for implementation fidelity, the Conceptual Framework for Implementation Fidelity, suggest that fidelity deals with measuring the degree to which an intervention’s content, frequency, duration, and coverage have been implemented in accordance to the original plan [23]. This is central in understanding fidelity’s contribution to program outcomes. Despite this, implementation fidelity has seldom been measured in relation to occupational health interventions [19]. In fact, to our knowledge, no prior studies have systematically evaluated the implementation fidelity of an organizationallevel occupational health intervention.
Aim The aims of this study are to (1) evaluate initial implementation fidelity of an organizational-level occupational health intervention using the Conceptual Framework for Implementation Fidelity [23] and (2) investigate possible explanations for variations in implementation fidelity between intervention units using the Framework for Evaluating Organizational-level Interventions [18]. The study hypotheses are that (1) initial implementation fidelity will differ between intervention units and (2) that variations in implementation fidelity are explained by contextual and implementation factors at the intervention units.
Methods Setting The study was conducted at a county district hospital in Sweden with approximately 500 employees working at 12 units including surgery, radiology, internal medicine, acute care, intensive care, rehabilitation, hospital-governed home care, and geriatrics. The hospital sought to develop their health promotion work, by moving away from time-limited individual-level interventions in which employee participation was limited to having one to two health representatives at each unit to an organizational-level intervention that would (1) clarify how health promotion was related to organizational factors and (2) ensure that organizational development and quality improvement work were conducted with consideration for employees’ health and work environment. Since the hospital already worked with a structured, participatory system for continuous improvement (Kaizen), an intervention integrating health promotion and occupational safety and health (OSH) with this system was developed. Kaizen is a tool in Lean, the production philosophy that has its origin in the Toyota Production System [25] and is described as a system that allows staff and workplace leaders to cooperate in a continuous strive for improvement [26]. The focus is on providing a structure for small-scale, continuous improvements of such character that decisions can be made directly at the workplace. The idea is that the system should encourage the quick identification of a problem that arises in the work process, an understanding of the reasons behind the problem and testing of solutions [27]. At the hospital, every unit had a great deal of freedom to form its own Kaizen work. However, the general process was the same. All employees at the different units were encouraged to identify problems in their work and document them on specific Kaizen notes. Every unit appointed one to three employees as Kaizen representatives, responsible for managing submitted Kaizen notes and for leading Kaizen meetings where all employees attended. At
Int.J. Behav. Med.
these meetings, problems raised were discussed and solutions proposed. For each solution, an employee or a group of employees were assigned the responsibility for implementing and evaluating the solution. Kaizen representatives from all units met regularly, together with a Kaizen coordinator from the HR department, for support, learning, and development. Intervention The intervention was led by the hospital with support from the research group. Key persons for leading the implementation of the intervention were the Kaizen- and health coordinators at the HR department and Kaizenand health representatives as well as managers at the unit level. The intervention, i.e., the integration of health promotion and OSH with the Kaizen system, consisted of two components: 1. All problems raised on Kaizen notes (regardless of which area the problem concerned) were to be analyzed from a health promotion and OSH perspective. This was to be done by considering possible health effects when deciding on testing and/or making solutions permanent. 2. Specific health-related problems and solutions were to be identified and raised on Kaizen notes. The integration also involved some changes in the roles of Kaizen- and health representatives. First, the Kaizenand health coordinators from the HR department held joint meetings for Kaizen- and health representatives as well as, as earlier, separate meetings for each group. Second, for health representatives, the integration also implicated that instead of mainly passing on information about physical activities available to the employees, they were to direct suggestions and ideas related to health promotion to the Kaizen notes. Beyond this, variation between intervention units in how they performed the integration was expected and allowed. The intervention was initiated in February 2012, with a workshop at which Kaizen representatives, health representatives, and managers from the intervention units participated in order to establish a common view on the intervention. A follow-up workshop was held 2 months later to provide the opportunity to share experiences and discuss any problems that might have occurred. The Kaizen coordinator, the health coordinator, and the managers at the intervention units were offered support during the intervention period by a certified coach (Coaching Healthcare Improvement Teams; Dartmouth Medical School and the Dartmouth Institute). The Kaizen coordinator and the health coordinator in turn coached the Kaizen- and health representatives at the intervention units.
Study Design The intervention was conducted with a quasi-experimental design. All hospital units were matched based on their characteristics (size, acute or non-acute care, day care, or around the clock care) and working processes around Kaizen (frequency of Kaizen meetings). One unit in each matched pair was randomized to the intervention group, and the other to the control. This was done to ensure variation regarding these characteristics within the groups and to increase the similarities between the intervention and the control group. A total of six units with approximately 200 employees in all were included in the intervention group. A case study design [28] was used to evaluate the implementation process using a multimethods approach including interviews, questionnaires, and analysis of Kaizen notes. To match the focus on the initial implementation phase, the present study includes data from the first 8 months of the intervention. Participants The present study concerns the intervention implementation and thus only uses the data collected at the intervention units. Kaizen- and health representatives (n=7) and all unit managers (n=6) were interviewed. Three units had a combined Kaizen and health representative, and therefore, only one person was interviewed at these units. One unit had no appointed Kaizen- or health representative; thus, no interview was conducted there. All employees (with the exception of those employed by the hour or on long-term leave or sick leave) were invited to participate in the questionnaire surveys. The response rate was 88 % (n=183) at baseline and 76 % (n= 161) at the 6-month measurement. The demographic characteristics of the respondents at baseline are described in Table 1. There were significantly more men and physicians working at unit 6 compared to the other units. Otherwise, the units did not differ in respondent characteristics. A total of 141 individuals (69 %) responded on both occasions, and this sample was used to evaluate changes in employee ratings between the two measurements. There were no significant differences between employees responding on one occasion compared to employees responding on both occasions regarding demographics or employees’ beliefs that the integration of health promotion with Kaizen would benefit their health (the item used to evaluate changes between the two measurements) at baseline or at 6-month follow-up. Employees responding on both occasions reported a significantly higher mean value at baseline for their level of working with Kaizen compared to those that only responded at baseline (mean value of 48.4 compared to 38.7 p=0.039). All participants gave their informed consent prior to participating in the study. Ethical approval was granted by the regional research ethics committee (Ref. No. 2011/1420-31/5).
Int.J. Behav. Med. Table 1 Characteristics of the respondents at the intervention units at baseline (n=183) Characteristics n (response rate %)
Unit 1 51 (96)
Unit 2 21 (100)
Unit 3 36 (97)
Unit 4 32 (82)
Unit 5 34 (85)
Unit 6 9 (53)
Differences between units p≥0.05
Age mean (SD) Gender n (%) Female Profession n (%) Physician Reg. nurse Assistant nurse Other Years at current workplace mean (SD)
47 (10)
48 (12)
42 (12)
43 (12)
48 (9)
46 (11)
n.s.
48 (94)
21 (100)
36 (100)
32 (100)
31 (91)
3 (33)
0.00a
0 (0) 29 (57) 18 (35) 4 (8) 9 (11)
0 (0) 9 (43) 4 (19) 8 (38) 10 (10)
0 (0) 18 (50) 12 (33) 6 (17) 9 (9)
0 (0) 12 (38) 18 (56) 2 (6) 8 (5)
0 (0) 19 (56) 13 (38) 2 (6) 12 (8)
8 (89) 0 (0) 0 (0) 1 (11) 6 (6)
0.00a
a
n.s.
Unit 6 differs significantly from all other units
Frameworks for Analysis
Data Collection
Two frameworks were used in the study. Implementation fidelity across the units was evaluated using the Conceptual Framework for Implementation Fidelity [23]. According to the framework, the measurement of implementation fidelity is a measurement of adherence, with its sub-categories—content, frequency, duration, and coverage (dose). Thus, adherence relates to the content and dose of the intervention, i.e., whether the active ingredients of the intervention have been received by the participants as often and for as long as was planned [23, 29–31]. In the current study, only the components of content and frequency were applicable due to the intervention design. Thus, implementation fidelity in the current study referred to the extent to which the units analyzed Kaizen notes from a health promotion and OSH perspective and wrote Kaizen notes with a specific health-related focus. Variations in implementation fidelity were examined according to Nielsen and Randall’s Framework for Evaluating Organizational-level Interventions [18]. The framework includes: context with the sub-categories omnibus context and discrete context; intervention with the sub-categories initiation, intervention activities, and implementation strategy (senior management, middle managers, drivers of change, participation, consultants, communication, and information); and mental models with the sub-categories readiness for change, perception of intervention activities, and changes in mental models. Mental models refer to the appraisals and perceptions of key stakeholders and how these may affect key stakeholders’ behaviors (in this study, unit managers, Kaizen- and health representatives, and employees) and thereby influence intervention outcomes [18]. We also added the sub-category history, which refers to previous experience of working according to the intervention, since this has been suggested to be important to consider when conducting process evaluations and evaluating the implementation of interventions [22, 32].
Interviews Semi-structured interviews were conducted 4 months after the intervention started. This time point was chosen since the focus was on the initial implementation. However, earlier time point was not considered as appropriate since the units needed some time to organize their work. The interviews concerned how the unit worked with the intervention, hindering and facilitating contextual factors, manager support, employees’ participation in the intervention, implementation strategies, readiness for change, and perception of the intervention. The interviews, lasted between 25 and 60 min, were carried out by the first author and were recorded and transcribed verbatim. Kaizen Notes All Kaizen notes (n=202) were collected from the intervention units, copied and registered into a database (unit 1: 28 notes, unit 2: 69 notes, unit 3: 40 notes, unit 4: 31 notes, unit 5: 34 notes, and unit 6: 0 notes). Questionnaires Two items measured implementation fidelity: “At my unit we discuss how employee health may be affected before making changes in work” and “At my unit we integrate health promotion with Kaizen.” Questions measuring employees’ ratings of implementation factors used to investigate variations in fidelity were formed based on items from the Dimension of Learning Organization Questionnaire (DLOQ) [33–35], Intervention Process Measure [32] and Survey of Organizations: a machine-scored standardized questionnaire instrument [36]. Factors measured concerned context, intervention activities, history, middle managers, participation, information and communication, readiness for change, perception of
Int.J. Behav. Med.
intervention activities, and changes in mental models (see Tables 3 and 4). The response alternative was a VAS scale with 0 (Disagree completely) and 10 (Fully agree) as endpoints. Items measuring the contextual factor group process—“In my work group we plan and coordinate our efforts”; “In my work group we solve problems in a constructive way”; “My work group is prepared for new challenges”; and “I regard my work group as efficient”—were grouped as suggested in the original questionnaire [36]. The factor structure was supported by a principal component analysis, and the internal reliability of the index was high (α=0.91) according to Cronbach’s alpha.
note was discussed with the second author until agreement was reached.
Results Implementation Fidelity
Data Analyses
The implementation fidelity differed between the intervention units regarding the delivered content and its frequency (Table 2). Based on these results, the units were grouped: Units 2 and 3 were labeled as a high-fidelity group; units 4, 5, and 6 were grouped into a low-fidelity group, and unit 1 was categorized as a medium-fidelity unit.
Statistical Analysis
Investigation of Variations in Implementation Fidelity
Possible differences in demographics between the units and between measurements were investigated using chi-square statistics, independent sample t test, and one-way analysis of variance (ANOVA). Normality was tested using the KolmogorovSmirnov test. Since seven of the items were not normally distributed, both parametric (ANOVA) and non-parametric (Kruskall-Wallis rank sum test) tests were used, yielding identical results. Thus, differences in mean values between the units were measured using ANOVA with the post hoc tests Bonferroni and Dunnett’s T3. Changes in mean values from baseline to the follow-up measurement were analyzed using repeated measures ANOVA. The SPSS statistical software package (Version 20 for Windows) was used. Statistical significance was set at two-tailed p value of
Investigating Variations in Implementation Fidelity of an Organizational-Level Occupational Health Intervention Hanna Augustsson & Ulrica von Thiele Schwarz & Terese Stenfors-Hayes & Henna Hasson
# International Society of Behavioral Medicine 2014
Abstract Background The workplace has been suggested as an important arena for health promotion, but little is known about how the organizational setting influences the implementation of interventions. Purpose The aims of this study are to evaluate implementation fidelity in an organizational-level occupational health intervention and to investigate possible explanations for variations in fidelity between intervention units. Method The intervention consisted of an integration of health promotion, occupational health and safety, and a system for continuous improvements (Kaizen) and was conducted in a quasi-experimental design at a Swedish hospital. Implementation fidelity was evaluated with the Conceptual Framework for Implementation Fidelity and implementation H. Augustsson (*) : U. von Thiele Schwarz : H. Hasson Department of Learning, Informatics, Management and Ethics, Medical Management Centre, Karolinska Institutet, Tomtebodavägen 18A, 171 77 Stockholm, Sweden e-mail: [email protected] U. von Thiele Schwarz e-mail: [email protected]
factors used to investigate variations in fidelity with the Framework for Evaluating Organizational-level Interventions. A multi-method approach including interviews, Kaizen notes, and questionnaires was applied. Results Implementation fidelity differed between units even though the intervention was introduced and supported in the same way. Important differences in all elements proposed in the model for evaluating organizational-level interventions, i.e., context, intervention, and mental models, were found to explain the differences in fidelity. Conclusion Implementation strategies may need to be adapted depending on the local context. Implementation fidelity, as well as pre-intervention implementation elements, is likely to affect the implementation success and needs to be assessed in intervention research. The high variation in fidelity across the units indicates the need for adjustments to the type of designs used to assess the effects of interventions. Thus, rather than using designs that aim to control variation, it may be necessary to use those that aim at exploring and explaining variation, such as adapted study designs. Keywords Health promotion . Process evaluation . Adherence . Kaizen
H. Hasson e-mail: [email protected] U. von Thiele Schwarz Department of Psychology, Stockholm University, 106 91 Stockholm, Sweden T. Stenfors-Hayes Department of Learning, Informatics, Management and Ethics, Centre of Learning and Knowledge, Karolinska Institutet, 171 77 Stockholm, Sweden e-mail: [email protected] H. Hasson Centre for Epidemiology and Community Medicine, Stockholm County Council, Stockholm, Sweden
Introduction There is no other setting where adults spend so much of their waking hours as the workplace. This has made the workplace an attractive arena for reaching adults with public health interventions and health promotion [1]. Using the workplace as an arena for health promotion has been suggested to have a number of advantages [2], including reaching a greater number of individuals compared to individual-based settings [3], providing opportunities for offering social support [4], and providing cues and reinforcement to help maintain behavior
Int.J. Behav. Med.
change [5]. In addition, the advantages of the workplace as an arena for health promotion can be further enhanced by utilizing the work environment and work conditions as such to improve health, i.e., organizational-level occupational health interventions [6]. In recent years, there has been increasing interest in organizational-level occupational health interventions aimed at improving psychosocial working conditions and employee health and well-being [7]. Organizational-level occupational health interventions have also been emphasized in current European legislation [8] as the preferred way of improving employee health and working conditions. However, these interventions have shown inconsistent results on employee health and wellbeing [9, 10]. Some organizational-level interventions have shown positive effects [11–13], while others have failed to do so [14, 15]. Several authors have highlighted the need for process evaluations in order to understand these inconsistencies [16–18]. Process data on organizational-level interventions are particularly limited [19] and have mainly been based on anecdotal data rather than being subjected to structural analyses [19–22]. One reason for this may be a lack of models and frameworks for use in guiding process evaluations of organizational-level interventions. Nielsen and Randall [18] recently proposed an evidence-based three-level framework on how processes of organizational-level interventions can be evaluated, called Framework for Evaluating Organizational-level Interventions. Two of the elements in the framework, context and participants’ mental models, represent the factors that may moderate or mediate the link between the intervention exposure and its outcomes. The third element covers the intervention design and implementation strategies that determine the level of intervention exposure that can be achieved. This element emphasizes the importance of documenting and measuring what is actually implemented, since this can highlight discrepancies between the planned intervention and its actual implementation. Within evaluation research, this has been referred to as implementation fidelity, which is the degree to which an intervention has been implemented as intended by the program developers [23, 24]. The most comprehensive framework for implementation fidelity, the Conceptual Framework for Implementation Fidelity, suggest that fidelity deals with measuring the degree to which an intervention’s content, frequency, duration, and coverage have been implemented in accordance to the original plan [23]. This is central in understanding fidelity’s contribution to program outcomes. Despite this, implementation fidelity has seldom been measured in relation to occupational health interventions [19]. In fact, to our knowledge, no prior studies have systematically evaluated the implementation fidelity of an organizationallevel occupational health intervention.
Aim The aims of this study are to (1) evaluate initial implementation fidelity of an organizational-level occupational health intervention using the Conceptual Framework for Implementation Fidelity [23] and (2) investigate possible explanations for variations in implementation fidelity between intervention units using the Framework for Evaluating Organizational-level Interventions [18]. The study hypotheses are that (1) initial implementation fidelity will differ between intervention units and (2) that variations in implementation fidelity are explained by contextual and implementation factors at the intervention units.
Methods Setting The study was conducted at a county district hospital in Sweden with approximately 500 employees working at 12 units including surgery, radiology, internal medicine, acute care, intensive care, rehabilitation, hospital-governed home care, and geriatrics. The hospital sought to develop their health promotion work, by moving away from time-limited individual-level interventions in which employee participation was limited to having one to two health representatives at each unit to an organizational-level intervention that would (1) clarify how health promotion was related to organizational factors and (2) ensure that organizational development and quality improvement work were conducted with consideration for employees’ health and work environment. Since the hospital already worked with a structured, participatory system for continuous improvement (Kaizen), an intervention integrating health promotion and occupational safety and health (OSH) with this system was developed. Kaizen is a tool in Lean, the production philosophy that has its origin in the Toyota Production System [25] and is described as a system that allows staff and workplace leaders to cooperate in a continuous strive for improvement [26]. The focus is on providing a structure for small-scale, continuous improvements of such character that decisions can be made directly at the workplace. The idea is that the system should encourage the quick identification of a problem that arises in the work process, an understanding of the reasons behind the problem and testing of solutions [27]. At the hospital, every unit had a great deal of freedom to form its own Kaizen work. However, the general process was the same. All employees at the different units were encouraged to identify problems in their work and document them on specific Kaizen notes. Every unit appointed one to three employees as Kaizen representatives, responsible for managing submitted Kaizen notes and for leading Kaizen meetings where all employees attended. At
Int.J. Behav. Med.
these meetings, problems raised were discussed and solutions proposed. For each solution, an employee or a group of employees were assigned the responsibility for implementing and evaluating the solution. Kaizen representatives from all units met regularly, together with a Kaizen coordinator from the HR department, for support, learning, and development. Intervention The intervention was led by the hospital with support from the research group. Key persons for leading the implementation of the intervention were the Kaizen- and health coordinators at the HR department and Kaizenand health representatives as well as managers at the unit level. The intervention, i.e., the integration of health promotion and OSH with the Kaizen system, consisted of two components: 1. All problems raised on Kaizen notes (regardless of which area the problem concerned) were to be analyzed from a health promotion and OSH perspective. This was to be done by considering possible health effects when deciding on testing and/or making solutions permanent. 2. Specific health-related problems and solutions were to be identified and raised on Kaizen notes. The integration also involved some changes in the roles of Kaizen- and health representatives. First, the Kaizenand health coordinators from the HR department held joint meetings for Kaizen- and health representatives as well as, as earlier, separate meetings for each group. Second, for health representatives, the integration also implicated that instead of mainly passing on information about physical activities available to the employees, they were to direct suggestions and ideas related to health promotion to the Kaizen notes. Beyond this, variation between intervention units in how they performed the integration was expected and allowed. The intervention was initiated in February 2012, with a workshop at which Kaizen representatives, health representatives, and managers from the intervention units participated in order to establish a common view on the intervention. A follow-up workshop was held 2 months later to provide the opportunity to share experiences and discuss any problems that might have occurred. The Kaizen coordinator, the health coordinator, and the managers at the intervention units were offered support during the intervention period by a certified coach (Coaching Healthcare Improvement Teams; Dartmouth Medical School and the Dartmouth Institute). The Kaizen coordinator and the health coordinator in turn coached the Kaizen- and health representatives at the intervention units.
Study Design The intervention was conducted with a quasi-experimental design. All hospital units were matched based on their characteristics (size, acute or non-acute care, day care, or around the clock care) and working processes around Kaizen (frequency of Kaizen meetings). One unit in each matched pair was randomized to the intervention group, and the other to the control. This was done to ensure variation regarding these characteristics within the groups and to increase the similarities between the intervention and the control group. A total of six units with approximately 200 employees in all were included in the intervention group. A case study design [28] was used to evaluate the implementation process using a multimethods approach including interviews, questionnaires, and analysis of Kaizen notes. To match the focus on the initial implementation phase, the present study includes data from the first 8 months of the intervention. Participants The present study concerns the intervention implementation and thus only uses the data collected at the intervention units. Kaizen- and health representatives (n=7) and all unit managers (n=6) were interviewed. Three units had a combined Kaizen and health representative, and therefore, only one person was interviewed at these units. One unit had no appointed Kaizen- or health representative; thus, no interview was conducted there. All employees (with the exception of those employed by the hour or on long-term leave or sick leave) were invited to participate in the questionnaire surveys. The response rate was 88 % (n=183) at baseline and 76 % (n= 161) at the 6-month measurement. The demographic characteristics of the respondents at baseline are described in Table 1. There were significantly more men and physicians working at unit 6 compared to the other units. Otherwise, the units did not differ in respondent characteristics. A total of 141 individuals (69 %) responded on both occasions, and this sample was used to evaluate changes in employee ratings between the two measurements. There were no significant differences between employees responding on one occasion compared to employees responding on both occasions regarding demographics or employees’ beliefs that the integration of health promotion with Kaizen would benefit their health (the item used to evaluate changes between the two measurements) at baseline or at 6-month follow-up. Employees responding on both occasions reported a significantly higher mean value at baseline for their level of working with Kaizen compared to those that only responded at baseline (mean value of 48.4 compared to 38.7 p=0.039). All participants gave their informed consent prior to participating in the study. Ethical approval was granted by the regional research ethics committee (Ref. No. 2011/1420-31/5).
Int.J. Behav. Med. Table 1 Characteristics of the respondents at the intervention units at baseline (n=183) Characteristics n (response rate %)
Unit 1 51 (96)
Unit 2 21 (100)
Unit 3 36 (97)
Unit 4 32 (82)
Unit 5 34 (85)
Unit 6 9 (53)
Differences between units p≥0.05
Age mean (SD) Gender n (%) Female Profession n (%) Physician Reg. nurse Assistant nurse Other Years at current workplace mean (SD)
47 (10)
48 (12)
42 (12)
43 (12)
48 (9)
46 (11)
n.s.
48 (94)
21 (100)
36 (100)
32 (100)
31 (91)
3 (33)
0.00a
0 (0) 29 (57) 18 (35) 4 (8) 9 (11)
0 (0) 9 (43) 4 (19) 8 (38) 10 (10)
0 (0) 18 (50) 12 (33) 6 (17) 9 (9)
0 (0) 12 (38) 18 (56) 2 (6) 8 (5)
0 (0) 19 (56) 13 (38) 2 (6) 12 (8)
8 (89) 0 (0) 0 (0) 1 (11) 6 (6)
0.00a
a
n.s.
Unit 6 differs significantly from all other units
Frameworks for Analysis
Data Collection
Two frameworks were used in the study. Implementation fidelity across the units was evaluated using the Conceptual Framework for Implementation Fidelity [23]. According to the framework, the measurement of implementation fidelity is a measurement of adherence, with its sub-categories—content, frequency, duration, and coverage (dose). Thus, adherence relates to the content and dose of the intervention, i.e., whether the active ingredients of the intervention have been received by the participants as often and for as long as was planned [23, 29–31]. In the current study, only the components of content and frequency were applicable due to the intervention design. Thus, implementation fidelity in the current study referred to the extent to which the units analyzed Kaizen notes from a health promotion and OSH perspective and wrote Kaizen notes with a specific health-related focus. Variations in implementation fidelity were examined according to Nielsen and Randall’s Framework for Evaluating Organizational-level Interventions [18]. The framework includes: context with the sub-categories omnibus context and discrete context; intervention with the sub-categories initiation, intervention activities, and implementation strategy (senior management, middle managers, drivers of change, participation, consultants, communication, and information); and mental models with the sub-categories readiness for change, perception of intervention activities, and changes in mental models. Mental models refer to the appraisals and perceptions of key stakeholders and how these may affect key stakeholders’ behaviors (in this study, unit managers, Kaizen- and health representatives, and employees) and thereby influence intervention outcomes [18]. We also added the sub-category history, which refers to previous experience of working according to the intervention, since this has been suggested to be important to consider when conducting process evaluations and evaluating the implementation of interventions [22, 32].
Interviews Semi-structured interviews were conducted 4 months after the intervention started. This time point was chosen since the focus was on the initial implementation. However, earlier time point was not considered as appropriate since the units needed some time to organize their work. The interviews concerned how the unit worked with the intervention, hindering and facilitating contextual factors, manager support, employees’ participation in the intervention, implementation strategies, readiness for change, and perception of the intervention. The interviews, lasted between 25 and 60 min, were carried out by the first author and were recorded and transcribed verbatim. Kaizen Notes All Kaizen notes (n=202) were collected from the intervention units, copied and registered into a database (unit 1: 28 notes, unit 2: 69 notes, unit 3: 40 notes, unit 4: 31 notes, unit 5: 34 notes, and unit 6: 0 notes). Questionnaires Two items measured implementation fidelity: “At my unit we discuss how employee health may be affected before making changes in work” and “At my unit we integrate health promotion with Kaizen.” Questions measuring employees’ ratings of implementation factors used to investigate variations in fidelity were formed based on items from the Dimension of Learning Organization Questionnaire (DLOQ) [33–35], Intervention Process Measure [32] and Survey of Organizations: a machine-scored standardized questionnaire instrument [36]. Factors measured concerned context, intervention activities, history, middle managers, participation, information and communication, readiness for change, perception of
Int.J. Behav. Med.
intervention activities, and changes in mental models (see Tables 3 and 4). The response alternative was a VAS scale with 0 (Disagree completely) and 10 (Fully agree) as endpoints. Items measuring the contextual factor group process—“In my work group we plan and coordinate our efforts”; “In my work group we solve problems in a constructive way”; “My work group is prepared for new challenges”; and “I regard my work group as efficient”—were grouped as suggested in the original questionnaire [36]. The factor structure was supported by a principal component analysis, and the internal reliability of the index was high (α=0.91) according to Cronbach’s alpha.
note was discussed with the second author until agreement was reached.
Results Implementation Fidelity
Data Analyses
The implementation fidelity differed between the intervention units regarding the delivered content and its frequency (Table 2). Based on these results, the units were grouped: Units 2 and 3 were labeled as a high-fidelity group; units 4, 5, and 6 were grouped into a low-fidelity group, and unit 1 was categorized as a medium-fidelity unit.
Statistical Analysis
Investigation of Variations in Implementation Fidelity
Possible differences in demographics between the units and between measurements were investigated using chi-square statistics, independent sample t test, and one-way analysis of variance (ANOVA). Normality was tested using the KolmogorovSmirnov test. Since seven of the items were not normally distributed, both parametric (ANOVA) and non-parametric (Kruskall-Wallis rank sum test) tests were used, yielding identical results. Thus, differences in mean values between the units were measured using ANOVA with the post hoc tests Bonferroni and Dunnett’s T3. Changes in mean values from baseline to the follow-up measurement were analyzed using repeated measures ANOVA. The SPSS statistical software package (Version 20 for Windows) was used. Statistical significance was set at two-tailed p value of
