Measurement Issues

Validation of Performance Indicators for Evaluation of Workplace Health Promotion

t s r

Seung Hee Ho, PhD; Young Moon Chae, PhD PURPOSE

Abstract Purpose. To validate performance indicators for evaluating workplace health promotion (WHP) programs based on a logic model and to analyze the structural relationships between constructs. Design. The study design is cross-sectional. Setting. Design setting was small manufacturing companies implementing WHP programs provided by the Korea Industrial Health Association. Subjects. Seventeen occupational health experts completed a questionnaire to determine the content validity of indicators. In addition, 58 health care managers completed a questionnaire to determine reliability and construct validation. The response rate was 84.1%. Measures. Based on a logic model, 13 constructs of WHP programs were identified: WHP program input, four activities for workplace environment management, two activities for employee health care management, two outputs, and two short-term outcomes. Analysis. Interrater agreement index was used for testing the content validity of indicators. Confirmatory factor analysis was used to test for the reliabilities, and the convergent and discriminant validities. Structuring equation modeling was also used to analyze the relationships among constructs. Results. A total of 35 performance indicators from 11 constructs showed good reliability and validity. All relationships among WHP input, activities, outputs, and short-term outcomes were significant, except for the relationship between environment outputs and short-term outcome. Conclusion. These findings illustrate that the logic model and structuring equation modeling can be used to develop and validate performance indicators for planning and evaluation of the WHP program. (Am J Health Promot 0000;00[0]:000–000.) Key Words: Indicator Validation, Logic Model, Structural Equation Model, Workplace Health Promotion, Prevention Research. Manuscript format: research; Research purpose: modeling/relationship testing; Study design: quasi-experimental; Outcome measure: morbidity; Setting: workplace; Health focus: social health; Strategy: policy; Target population age: adults; Target population circumstances: education

e n i

l n

o

i F

Seung Hee Ho, PhD, is with the Korea National Rehabilitation Research Institute, Ministry of Health and Welfare, Seoul, Korea. Young Moon Chae, PhD, is with the Department of Health Informatics, Graduate School of Public Health, Institute of Health Service Research, Yonsei University, Seoul, Korea. Send reprint requests to Young Moon Chae, PhD, Department of Health Informatics, Graduate School of Public Health, Institute of Health Service Research, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 120–752, Korea; [email protected]. This manuscript was submitted October 8, 2012; revisions were requested December 17, 2012, April 18 and August 6, 2013, and February 17, February 28, and March 6, 2014; the manuscript was accepted for publication March 7, 2014. Copyright Ó 0000 by American Journal of Health Promotion, Inc. 0890-1171/00/$5.00 þ 0 DOI: 10.4278/ajhp.121008-QUAN-488

American Journal of Health Promotion

With significant increases in health care costs, workplace health promotion (WHP) programs have been identified as potentially effective approaches to reduce risk factors for chronic deseases.1 Several studies have shown correlations between WHP and lower health risks, lower health care costs, and improved productivity for employees.2,3 A 2008 survey of large manufacturing firms in the United States indicates that 77% have a WHP program.4 However, smaller firms were less likely to offer a WHP program because of insufficient resources to support it.5 There is strong evidence to conclude that employees of small enterprises are subject to higher risks to health and safety than the employees of larger ones, and that small enterprises have difficulties in controlling risk.6,7 Accordingly, it is important to develop intervention strategies to improve the health of employees of small enterprises, because small enterprises account for most of the companies and a large share of the employees in most countries; for example, in Denmark a total of 98% of all companies have fewer than 50 employees, covering approximately a third of the total workforce.7 Similarly, most small firms in Korea cannot afford to offer WHP programs. However, such programs are more essential to the smaller firms because of poor work environments and lack of younger workers, whereas larger firms tend to recruit younger and healthier employees.8 A 2004 survey on occupational safety and health was conducted on small firms with fewer than 50 workers. The results conveyed that the injury occurrence rates and occupa-

Month 0000, Vol. 0, No. 0

0

tional disease prevalence rates were two times higher than those at larger firms.9 This is a serious problem for national economy because more than 90% of all firms have fewer than 50 workers, and 70.6% of all workers worked for such small firms in Korea. In order to improve the health of workers and prevent occupational accidents in small firms, since 1990 the Korean government has designated selected health institutes to contract with the firms having fewer than 300 workers in order to provide WHP programs at low cost. These institutes, called Group Occupational Health Services (GOHS), were required to provide five basic services: workplace environment management, health consultation and health examination, chronic disease management, health education, and health information management and reporting.8 Although GOHS has contributed greatly toward improving the health status of workers at small firms in Korea, the government has little control over the quality of such services. Methods should be generated to evaluate and monitor the performance of WHP programs by using standard performance indicators, because there are currently 52 GOHSs offering a variety of services.10 Goldenhar et al.11 indicated that in order for the health management program to be effective, it is important to evaluate the outcome of the program and examine the details of the implementation progress and its elements. There are several assessment tools and indicators for evaluating WHP programs. Oldenburg et al.12 developed and validated indicators for evaluating health promotion environments at workplaces, called the Checklist of Health promotion Environments at Worksites (CHEW). The 112-item CHEW is a direct observation tool used to evaluate a worksite’s physical and ‘‘information distributed’’ environments within the context of physical environment in the immediate surrounding community as related to physical activities, eating habits, alcohol consumption, and smoking levels. The CHEW was validated in terms of reliability testing by using intraclass correlations. Although the CHEW is a useful observational measure that has

0

American Journal of Health Promotion

sources and activities with the desired results in a workable program.17 Logic models have been applied to many areas in the health care programs, such as integrating mental health into chronic disease prevention and health promotion,18 and process evaluations and outcome evaluation of Racial and Ethnic Approaches to Community Health (REACH 2010) project by the Centers for Disease Control and Prevention.19 Rather surprising, however, was the fact that there were only a few explicit examples for the use of logic models in the WHP performances evaluation. And most of the previous studies on logic models conceptually provided measurements or indicators using the knowledge and experiences of researchers without validating them. In this study, a logic model was used to identify indicators on WHP program inputs and activities that were conceptually related to their outcomes, and these indicators were comprehensively validated in terms of content validity, reliability, convergent validity, and discriminant validity. Moreover, the structural equation model (SEM) was used to empirically identify inputs and activities that had clear relations to their program goals or the desired outcomes. The purpose of this study was to develop and validate performance indicators for evaluating WHP programs, indicators that include both environment and employee health based on the logic model, and to analyze the structural relationships between the constructs in order to identify indicators from WHP program inputs and activities that were significantly related to their desired outcomes. Specifically, performance indicators were developed for each component of WHP programs: inputs, activities, outputs, and short-term outcomes for both the WHP environment and employee health management. Indicators were validated in a comprehensive way that included content validation, reliability testing, and construct validation in terms of convergent validation and discriminant validation. Finally, performance scores for the selected indicators were compared with the industry average score in order to identify the relative strengths as well as the weak-

t s r

i F

e n i

l n

o

the potential to assess environmental influences on health behaviors, the scope of the CHEW was limited to environment, excluding measures for health. There is increasing evidence that the work environment and the overall health, safety, and well-being of the workers within it are strongly connected. Total Worker Health is a strategy integrating traditional occupational safety and health protection practices with health promotion strategies to not only prevent injury and illness among workers, but also to advance their health and well-being.13 The National Institute for Occupational Safety and Health first launched an initiative to address total worker health in 2004, and it now conducts research on the integration of health protection and health promotion through both intramural and extramural programs.14 DeJoy et al.15 also developed the Environmental Assessment Tool (EAT) for assessing worksite physical and social environmental supports for obesity prevention. Specifically, the items in the EAT pertain to physical activity, nutrition/weight management, and organizational characteristics and support. Plotnikoff et al.16 developed and validated the Workplace Physical Activity Assessment Tool. They used stage-matched and standard print materials for physical activities at workplaces. Although these were useful to evaluate some aspects of WHP programs, they cannot point out what program inputs and activities need to be more closely monitored to achieve intended results or outcomes, because these tools were not categorized in terms of inputs, activities, outputs, and outcomes. Logic models help more closely examine inputs and activities directly related to outputs and outcomes. For the last 20 years, logic models have been commonly used in planning as well as program evaluations to chart out what should have happened and what did or did not occur as intended. Program evaluators have long advocated the use of logic models to demonstrate the intended flow of program operations, from resources (input) to implementing program activities, to program outputs, to short-term, intermediate, and long-term outcomes. Logic models help focus on the re-

Month 0000, Vol. 0, No. 0

Table 1 Constructs for Workplace Health Promotion (WHP) Program Evaluation Based on the Logic Model Components of Logic Model Input Activity

Output Short-term outcome

Workplace Environment Management

Health Management for Workers

WHP program input Safety education Safety management Environment administration Environment administration follow-up Environment output Injury level

WHP program input Health examination implementation Postexamination activity Health administration Health promotion service Health output Disease prevalence

nesses of the WHP program in a particular workplace.

METHODS Design This was a cross-sectional study with two questionnaire surveys. The first survey was conducted for content validation of the performance indicators for evaluating WHP programs. The second survey was conducted for reliability testing and construct validation of the performance indicators. Each question in the survey instrument was measured on a five-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). Sampling design for the second survey was simple random sampling for a size of 156 worksites. That is, 156 worksites were randomly chosen from the population of 1006 worksites belonging to the four Seoul/ Gyeonggi Metropolitan branches of the largest GOHS, the Korea Industrial Health Association (KIHA).

o

Sample Two types of subjects were used for each survey in this study. First, 17 experts on occupational health were given a questionnaire in order to measure the content validity of performance indicators. Lawshe20 suggested a minimum of four panelists for content validation. In this study, it was decided to include as many experts in the panel as possible. They were all representatives from key industrial health institutes in Korea. Of them, 9 (52.9%) were from universities or research institutes, 4 (23.5%) from government, and 4 (23.5%) from

American Journal of Health Promotion

t s r

i F

KIHA. A total of 12 of them (70.6%) have more than 5 years of experience in industrial health. Second, 156 workplaces were randomly selected out of the 1006 workplaces belonging to the four Seoul/Gyunggi Metropolitan branches of KIHA for testing reliability and constructing validity. Of the 69 health care managers who provided health promotion services to 156 workplaces, 58 completed a questionnaire for the survey, giving a response rate of 84.1%. Of these 58 managers, 43 (74.1%) were industrial health nurses and 13 (25.9%) were industrial hygienists.

e n i

l n

constructs for employee health care management (health examination implementation, health examination follow-up, health administration, and health promotion activities), two output constructs (environmental hazard level and health status of works), and two short-term outcome constructs (injury level and disease prevalence). These indicators were initially tested for content validity in terms of being ‘‘essential’’, ‘‘useful’’, ‘‘feasible’’, and ‘‘reliable’’ by 17 experts in the first survey. Furthermore, they were tested for reliability and construct validity, which were measured on a five-point Likert scale, by 58 field health managers in the second survey.

Measures A logic model was modified to select performance indicators for evaluating WHP programs. Because 1-year data were used as validating indicators, both intermediate outcomes and long-term outcomes were excluded from the analysis. As seen in Table 1, 13 constructs were selected from a logic model for two types of WHP programs based on the previous studies on WHP12,15,16: workplace environment management and health management for workers. Workplace environment management includes factors that affect the safety of the workplace environment and the physical health of an employee. And health management for workers deals with initiatives that have a direct impact on workers’ health. The 13 constructs included: WHP program input, four activity constructs for workplace environment management (safety education, safety management, environmental administration, and environmental follow-up), four activity

Analysis Content Validity. Content validity is the determinant for the content relevance of the item, and it focuses on the assessment of judgments systematically provided by experts.20 Testing for content validity gauges agreement among raters regarding how essential a particular item is. Lawshe20 proposed that each of the expert raters on the judging panel respond to the following question for each item: ‘‘Is the skill or knowledge measured by this item ‘essential,’ ‘useful, but not essential,’ or ‘not necessary’ to the performance of the construct?’’ In addition to ‘‘essential’’ and ‘‘useful’’, we asked the experts whether indicators are ‘‘feasible’’ and ‘‘reliable’’, as suggested by the Organisation for Economic Co-operation and Development Health Care Quality Indicators (HCQI) Project.21 ‘‘Feasible’’ refers to the extent to which the indicator can be constructed from available data at workplaces for low costs, and ‘‘reliable’’ refers to the extent to which an indicator provides stable results across various circumstances in workplaces. Interrater agreement index (Rwg) was used for testing content validity. The interrater agreement index focuses on the interchangeability among raters; it addresses the extent to which raters make essentially the same ratings.22 The interrater agreement is acceptable if Rwg(j) þ .70.23 h i RwgðjÞ ¼ j 1  ðS2xj =r2EU Þ hh i i =j 1  ðS2xj =r2EU Þ þ ðS2xj =r2EU Þ ;

Month 0000, Vol. 0, No. 0

0

a. The internal consistency is acceptable if Cronbach a exceeds .6.27

Table 2 Profile of the Study Workplaces Measure and Items

Frequency

Percent

41 62 53 156

26.3 39.7 40.0 100.0

15 95 30 7 9 156

9.6 60.9 19.2 4.5 5.8 100.0

Industry type (manufacturing) Machinery Electronics Chemistry Total No. of employees Fewer than 50 50–100 100–150 150–200 200 and above Total

where j ¼ number of items rated, S2xj ¼ the mean of the observed variance on j items, and r2EU ¼ the expected variance due to random measurement error.

Discriminant Validity. Discriminant validity tests whether measurements that are supposed to be unrelated are, in fact, unrelated. For discriminant validity, the square root of the average variance extracted (AVE) of each construct should be larger than the correlation of the specific construct with any of the other constructs in the model,28 and should be at least .50.25 The AVE for each specific construct is the ratio of the sum of its measurement item variance as extracted by the construct relative to the measurement error attributed to its items. AVE is calculated as:   X  X X k2i þ ð1  k2i Þ ; k2i =

Convergent Validity. Convergent validity refers to the degree to which a measurement is similar to (converges on) other measurements that it should also be theoretically related. Convergent validity is acceptable if factor loadings from confirmatory factor analysis (CFA) are .60 or higher.24

o

0

American Journal of Health Promotion

t s r

i F

where ki is the loading of each measurement item on its corresponding construct.

e n i

l n

Reliability. Reliability is the extent to which a measuring instrument contains variable errors—that is, errors that appear inconsistently between observations, either during any one measurement procedure or each time a given variable is measured by the same instrument. To test for the reliabilities of the latent variables, composite reliability (CR) from CFA was used based on the procedures outlined by Fornell and Larcker.25 CFA was conducted via the Partial Least Squares (PLS) Graph Version 3.0. PLS places minimal restrictions on sample sizes and residual distributions. Thus, PLS was selected to handle the large number of constructs in this study. The reliability for CR is acceptable if CR is .70 or higher.26 In addition, the internal consistency reliability of the latent variables was also assessed by Cronbach

j. The values of Rwg for 11 constructs were calculated and are summarized in Table 3. Two constructs (postexamination activity and health administration) were not selected because of low values. The selected constructs were: WHP program input, four constructs for workplace environment activities (safety education, safety management, environmental administration, and environmental follow-up), two constructs for health care activities of workers (health examination implementation and health promotion services), two output constructs (environmental hazard level and health status of works), and two shortterm outcome constructs (injury level and disease prevalence). The measure of Rwg for each indicator ranged from .82 to .94, showing that the interrater agreement was high.

Relationship Among Constructs by Structural Equation Model The relationships among constructs in WHP input, activity, output, and short-term outcome were analyzed by the structural equation model (SEM). SEM is a statistical technique for testing and estimating causal relations, using a combination of statistical data and qualitative causal assumptions.24 SEM allows confirmatory modeling, which begins with a hypothesis that becomes represented in a causal model.

RESULTS

Characteristics of Subjects A total of 156 worksites were used for reliability testing and to construct validation of performance indicators. Three industry types were included: machinery (26.3%), electronics (39.7%), and chemistry (40.0%). The sample characteristics were summarized with respect to industry and the number of employees in Table 2. Content Validity Content validity was tested using interrater agreement (Rwg), which measures the extent to which the different raters tend to give exactly the same ratings about the rated subject. Rwg(j) is raters’ mean score on indicator

Convergent Validity The lowest loading of this study was .61, thus satisfying the convergent validity prerequisites. Indicators with factor loading scores lower than .60 were deleted from the validity check.

Reliability As seen in Table 3, the CRs of the constructs exceeded the acceptable values (CR . .7) for reliability, and the Cronbach a exceeded the acceptable values of .6 for internal consistency. Discriminant Validity The discriminant validity of the latent variable is confirmed when AVE is higher than .5 and the square root of the AVE for each construct is higher than the correlations between the constructs. The constructs in Table 3 were selected because their AVEs were higher than .5 and each construct had a greater variance from its indicators than from the constructs representing a different block of measures, therefore satisfying the discriminant validity prerequisites. Based on testing for reliability, convergent validity, and discriminant validity, 35 performance indicators out of 49 indicators from 11 constructs were finally selected. Fourteen indicators from six constructs excluded by the validity testing were: health examination implementation (guideline for health examination, special health examination); health examination fol-

Month 0000, Vol. 0, No. 0

Table 3 Reliability, Convergence Validity, and Discriminant Validity of Performance Indicators* Construct

Item

WHP input

Environment management activities Safety education

Safety management

Environmental administration

Environmental administration follow-up

Health management activities Health examination implementation HP services

Mean

SD

Rwg(j)

FL

a

CR

AVE

Top management support Company health policy Hiring safety manager Hiring health care manager Status of health care facility

3.25 3.18 3.30 3.46 3.33

0.73 0.85 0.67 0.65 0.87

0.852 0.896 0.915 0.916 0.918

0.77 0.74 0.81 0.78 0.78

0.830

0.88

0.60

Routine safety education Training for new workers Training on the process changes Frequency of safety meeting Maintenance of safety manual Maintenance of safety device Maintenance of ventilation system Frequency of environmental monitoring Participation by labor union Management of hazardous material Preparation for material safety data Notification of monitoring results Follow-up of monitoring results Management of hazardous materials Maintenance of statistics

3.19 3.11 3.04 2.83 3.02 3.71 3.69 4.41 3.18 3.71 3.85 3.06 3.35 3.69 3.61

0.87 0.86 0.88 1.06 0.96 0.74 0.81 0.69 1.03 0.71 0.82 1.07 0.84 0.77 0.82

0.882 0.923 0.876 0.850 0.910 0.912 0.881 0.874 0.909 0.939 0.906 0.919 0.902 0.920 0.929

0.87 0.93 0.93 0.68 0.83 0.81 0.68 0.61 0.68 0.86 0.85 0.75 0.90 0.86 0.79

0.897

0.94

0.83

0.729

0.84

0.57

0.730

0.84

0.58

0.834

0.90

0.69

96.57 96.14 1.51 2.02 1.72 2.03 4.40 3.46 58.60 65.10 0.38 1.57 1.34 1.48 2.31

6.24 6.12 1.06 1.07 0.94 1.07 0.78 0.90 18.31 14.92 0.61 2.25 1.62 0.57 1.65

0.890 0.918 0.855 0.914 0.931 0.902 0.943 0.916 0.922 0.878 0.883 0.870 0.872 0.872 0.891

0.87 0.87 0.65 0.91 0.85 0.90 0.81 0.81 0.89 0.86 0.95 0.96 0.88 0.82 0.82

0.794

0.87

0.77

0.847

0.90

0.70

0.723

0.79

0.65

0.801

0.87

0.77

0.827

0.97

0.89

0.758

0.81

0.67

l n

Environment output Health status of workers

Environment short-term outcome

Health short-term outcome

o

i F

e n i

Conducting of health examination Conducting of preexamination Conducting of physical strength test Conducting of HP program Participation of HP program Follow-up of HP program % of safety device use Improvement of environment % of nonsmoking workers % of normal BMI workers Rate of injuries Rate of industrial accidents Loss of work days Rate of occupational diseases Rate of nonoccupational diseases

t s r

* Rwg(j) indicates interrater agreement index; FL, factor loading; a, Cronbach coefficient; CR, composite reliability; AVE, average variance extracted; WHP, workplace health promotion; HP, health promotion; and BMI, body mass index.

low-up (distribution of health examination results to each worker, postexamination health care, postexamination report to government, maintenance of health examination results); health administration (management of emergency system, posting of the major contents of the Labor Act, health promotion education, periodic documentation of health care management); environmental hazard level (exposure to carcinogens, environment assessment results); employee health status (percentage of workers with normal blood pressure);

American Journal of Health Promotion

and disease prevalence (prevalence rate of occupational disease). Relationship Among Constructs by the SEM An analysis of the SEM was conducted to confirm the hypothesized relationships that were identified by the logic model. In the logic model, the boxes are the steps that can be counted or monitored, and the lines connecting the boxes (e.g., inputs, activities, etc.) are the hypothesized linkages or causal relationships that require in-depth study for managers to

determine whether the program is working. As seen in Figure 1, the line showed the sets of input and activity indicators of WHP programs statistically significantly contributed to the environment and health management outputs, and the dotted line showed an insignificant relationship to environment short-term outcomes. Environment management activities (e.g., safety education, environment monitoring, etc.) were a second-order construct that was significantly formed by safety education, safety management, environmental administration,

Month 0000, Vol. 0, No. 0

0

Figure 1 Structural Equation Model for Workplace Health Promotion (WHP) Program Evaluation

i F

e n i

l n

t s r

BMI indicates body mass index. Dotted line indicates an insignificant relationship. **p , .01; ***p , .001.

and environmental follow-up. Health management activities (e.g., health examination, health promotion programs, etc.) were also a second-order construct that was significantly shaped by implementation of health examinations and health promotion services. The WHP inputs (e.g., top management support, company policy, etc.) significantly influenced both environment management activities and health management activities. In addition, environment management activities significantly influenced environment output (e.g., percentage of safety device use, improvement in environment), but they did not influence environment short-term outcomes (e.g., rates of injury and accidents). On the other hand, health management activities significantly influenced health output (e.g., rates of nonsmoking and normal body mass index) and significantly influenced

o

0

American Journal of Health Promotion

health short-term outcomes (e.g., prevalence rates of occupational and nonoccupational diseases). The SEM results show that WHP inputs and health management activities had clear linkages with their intended health outputs and shortterm health outcomes, and therefore health managers at worksites should closely monitor indicators in WHP inputs and activities to achieve health goals. The results also show that WHP inputs and environment management activities had clear linkage with their environment outputs, and therefore managers at worksites should closely examine indicators in WHP inputs and environment management activities to achieve environment outputs. However, additional activities should be carried out to further reduce rates of injury and accidents because there is no clear linkage with environment short-term outcome.

Use of Indicators for Performance Evaluation of WHP Programs at Worksites The indicators can be used in evaluating the performance of health promotion programs at worksites in order to identify the relative strengths as well as the weaknesses of a WHP program at a particular workplace in comparison with those at other firms. For example, the health promotion program at each worksite may be evaluated by an indicator on a 5-point Likert scale compared with other worksites in the same industry. We evaluated health promotion programs for 156 worksites by using indicators and classified them by industry. As seen in Figure 2, the performance of a health promotion program, as indicated by 11 constructs, which were measured on a 5-point scale, for metal workplace No. 20 was compared with the metal industry average score. It was

Month 0000, Vol. 0, No. 0

Figure 2 Evaluation of Workplace Health Promotion (WHP) Program at Workplace No. 20

found that three areas were weaker than the metal industry average: health status, environment hazard, and health promotion.

l n

DISCUSSION

This study has developed and validated 49 performance indicators from 13 constructs for evaluating WHP programs at worksites based on the logic model. The relationships among constructs in WHP inputs, activities, outputs, and short-term outcomes were analyzed by the SEM. All relationships were significant except the linkage between environment output and short-term outcome, perhaps because there were factors other than safety device use contributing to the rates of injuries and accidents. Accordingly, additional inputs and activities are needed to further reduce rates of injury and accidents. The SEM results also showed that WHP inputs and health management activities had clear linkages to their intended health outputs and short-term health outcomes, and therefore health managers at worksites should closely monitor indicators in WHP inputs and activities to achieve health goals.

o

American Journal of Health Promotion

i F

e n i

BMI indicates body mass index.

t s r

Model fit measure was not provided in this study because SEM procedures that have different objective functions allowing for formative measures (e.g., PLS) are unable to provide model fit measures. In reality, models with good fit indices may still be considered poor based on other measures, such as factor loadings and R2 values.28 From the convergent validation, the indicators with the factor loading of .6 were selected, indicating that each indicator was accounting for 60% or more of the variance of the underlying constructs. Summary scores of these indicators provide a way of comparing the performance of the WHP program at a particular workplace with other workplaces in a similar industry using the industry average score, and this provides an indication of strengths and weaknesses for health promotion programs in workplaces. If the industry average score for a certain industry (e.g., the metal industry) is very low compared with other sectors, an adjustment factor can be applied to the industry average score. One way to estimate the adjustment factor is to use the industry ranking in performance scores. These adjusted scores of indicators can be used to track whether WHP initiatives actually result in

changes in the environment and employee health at workplaces. Similarly, the adjusted scores of indicators for each workplace can also be compared over the years to monitor the progress of WHP programs. Health care managers at GOHSs might use the evaluation results as a basis for planning new health promotion initiatives in consultation with workplace health care managers. For example, based on the comparison of summary scores of indicators among workplaces in a similar industry within GOHSs, the weaknesses of the WHP program at a particular workplace, indicated by lower scores than industry average, may be brought to the attention of the workplace health care manager, along with recommendations for further improvement. Governments might use the evaluation results to evaluate the performances of GOHSs by comparing individual summary score with the nationwide GOHS average score. This study has some limitations. First, the relevance of this study is restricted to one area of industry: small businesses in the manufacturing industry in South Korea. Thus, the findings drawn from this study cannot be generalized directly to other industry groups in other countries. However,

Month 0000, Vol. 0, No. 0

0

we would argue that our methods can be transferable to other industries and to countries other than South Korea. To apply them to other countries, indicators from each box of the logic model should be identified and validated according to the methods demonstrated in our study. In addition, even though there are contextual factors that improve the outcomes of health promotion, this study did not consider other factors, such as task structures and social factors within the workplace. There are a number of factors that influence the approach to health promotion that could be effective for other locations. Such factors include the size of the workplace, the type of work performed, the demographics of the workers, and the hourly/salary mix of the population.2 More research is needed to determine

which indicators should be added to accurately assess workplace environment and employee health and to improve the quality of WHP initiatives. Acknowledgments

References

0

American Journal of Health Promotion

1. Faghri PD, Kotejoshyer R, Cherniack M, et al. Assessment of a worksite health promotion readiness checklist. J Occup Environ Med. 2010;52:893–899. 2. O’Donnell MP. Health Promotion in the Workplace. 3rd ed. Albany, NY: Delmar; 2002. 3. Berry LL, Mirabito AM, Baun WB. What’s the hard retrurn on emplyee wellness programs? Harv Bus Rev. 2010;88:105–112. 4. Baicker K, Cutler D, Song Z. Workplace wellness programs can generate savings. Health Aff (Millwood). 2010;29:304–311. 5. Linnan L, Bowling M, Childress J. Results of the 2004 national worksite health promotion survey. Am J Public Health. 2008; 98:1503–1509. 6. European Commission. European Social Statistics: Accidents at Work and Work-Related Health Problems. Luxembourg City, Luxembourg: Office for Official Publications of the European Communities; 2002. 7. Hasle P, Limborg HJ. A review of the literature on preventive occupational health and safety activities in small enterprises. Ind Health. 2006;44:6–12. 8. Han YR, Kim SG, Ha EH. The experience of occupational health care providers in the government-funded subsidized occupational health program for small scale industries. Korean J Occup Environ Med. 2002;14:392–407. 9. Lee KS, Chang SH, Choi KHJ, et al. Relationship between injury and workplace organization in small-size manufacturing factories. Korean J Occup Environ Med. 2006;18:73–86. 10. Kim SM, Cho SH, Kim CY, et al. Quality assessment of Group Occupational Health Service for small and medium scale enterprises in Korea. Korean J Occup Environ Med. 1998;10:71–82. 11. Goldenhar LM, Lamontagne A, Katz T, et al. The intervention research process in occupational safety and health: an overview from the national occupational research agenda intervention effectiveness team. J Occup Environ Med. 2001;43:616– 622. 12. Oldenburg B, Sallis JF, Harris D, Owen N. Checklist of health promotion environments at worksites (CHEW): development and measurement characteristics. Am J Health Promot. 2002; 16:288–299. 13. National Institute for Occupational Safety and Health. Total worker health. Available

16.

t s r

i F

e n i

l n

o

15.

The survey in this study was carried out with the support of the Korea Industrial Health Association. The authors thank those who helped conduct this study.

SO WHAT? Implications for Health Promotion Practitioners and Researchers What is already known on this topic? Many instruments are currently available to assess specific aspects of WHP. Examples include the Checklist of Health promotion Environment at Worksites (CHEW) and the Environmental Assessment Tool (EAT). What does this article add? Although tools such as CHEW and EAT were useful to evaluate selected aspects of WHP programs, they cannot point out what program inputs and activities need to be more closely monitored to achieve the intended results of a WHP program, because these indicators were not categorized. In this study, a logic model and SEM were used to identify indicators from WHP program inputs and activities that were significantly related to their desired outcomes. What are the implications for health promotion practice or research? WHP practitioners might use the performance indicators identified by the logic model and SEM to monitor progress toward the intended outputs and outcomes. Indicators may also be used as a monitoring instrument as well as planning tools for managers.

14.

at: http://wcdc.gov/niosh/TWH/. Accessed November 31, 2013. Schulte P, Pandalai S, Wulsin V, Chun H. Interaction of occupational and personal risk factors in workforce health and safety. Am J Public Health. 2012;102:434–448. DeJoy DM, Wilson MG, Goetzel RZ, et al. Development of the environmental assessment tool (EAT) to measure organizational physical and social support for worksite obesity prevention programs. J Occup Environ Med. 2008;50:126–137. Plotnikoff RC, Brunet S, Courneya KS, et al. The efficacy of stage-matched and standard public health materials for promoting physical activity in the workplace: the Physical Activity Workplace Study (PAWS). Am J Health Promot. 2007;21: 501–509. McLaughlin JA, Jordan JB. Logic models: a tool for telling your program’s performance story. Eval Program Plann. 1999;22:65–72. Lando J, Williams SM, Williams B, Sturgis S. A logic model for the integration of mental health into chronic disease prevention and health promotion. Prev Chronic Dis. 2006;3:1–4. Tucker P, Liao Y, Giles W, Liburd L. The REACH 2010 logic model: an illustration of expected performance. Prev Chronic Dis. 2006;3:1–6. Lawshe CH. A quantitative approach to content validity. Personnel Psychol. 1975;28: 563–575. Mattke SE, Kelley P, Hurst SJ, et al. Health Care Quality Indicators Project: Initial Indicators Report. OECD Health Working Paper No. 22. Paris, France: Organisation for Economic Co-operation and Development; 2006. Kozlowski SSJ, Hattrup K. A disagreement about within-group agreement: disentangling issues of consistency versus consensus. J Appl Psychol. 1992;77:161–167. James LR, Demaree RG, Wolf G. Estimating within-group interrater reliability with and without response bias. J Appl Psychol. 1984;69:85–98. Hair J, Anderson R, Tatham RB. Multivariate Data Analysis. Upper Saddle River, NJ: Prentice Hall; 1998. Fornell C, Larcker DF. Evaluating structural equation models with unobservable variables and measurement error. J Mark Res. 1981;18:39–50. Chin WW. The partial least squares approach to structural equation modeling. In: Marcoulides GA, ed. Modern Methods for Business Research. Mahway, NJ: Lawrence Erlbaum; 1998:295–336. Cortina JM. What is coefficient alpha? An examination of theory and applications. J Appl Psychol. 1993;78:98–104. Chin WW, Gopal A, Salisbury WD. Advancing the theory of adaptive structuration: the development of a scale to measure faithfulness of appropriation. Inf Syst Res. 1997;8:342–367.

17.

18.

19.

20.

21.

22.

23.

24. 25.

26.

27.

28.

Month 0000, Vol. 0, No. 0

Validation of Performance Indicators for Evaluation of Workplace Health Promotion.

To validate performance indicators for evaluating workplace health promotion (WHP) programs based on a logic model and to analyze the structural relat...
310KB Sizes 0 Downloads 3 Views