RESEARCH ARTICLE

The Relationship Between School-Level Characteristics and Implementation Fidelity of a Coordinated School Health Childhood Obesity Prevention Intervention ALYSSA M. LEDERER, MPH, CHESa MINDY H. KING, PhDb DANIELLE SOVINSKI, MPHc DONG-CHUL SEO, PhD, FAAHBd NAYOUNG KIM, MAe

ABSTRACT BACKGROUND: Curtailing childhood obesity is a public health imperative. Although multicomponent school-based programs reduce obesity among children, less is known about the implementation fidelity of these interventions. This study examines process evaluation findings for the Healthy, Energetic Ready, Outstanding, Enthusiastic, Schools (HEROES) Initiative, a tri-state school-based childhood obesity prevention intervention based on the coordinated school health (CSH) model. METHODS: Site visits were conducted that included key stakeholder interviews, observation, and document review. Scores were given for 8 domains, and a total implementation score was calculated. Two-way analyses of variance were conducted to examine the relationship of 4 school-level characteristics: elementary vs. middle/high schools, public vs. private schools, district vs. building level implementation, and socioeconomic status on each implementation area. RESULTS: Overall, schools had high fidelity scores, although some domains were implemented more successfully than others. Three school-level characteristics were associated with 1 or more domains, with elementary schools and schools implementing at the building level consistently having higher implementation scores than their counterparts. CONCLUSIONS: Process evaluation findings provide insight into successes and challenges schools implementing the CSH approach may encounter. Although preliminary, these findings on school-level characteristics establish a new area of research related to school-based childhood obesity prevention programs’ implementation fidelity. Keywords: process evaluation; implementation fidelity; childhood obesity; schools; coordinated school health. Citation: Lederer AM, King MH, Sovinski D, Seo D-C, Kim N. The relationship between school-level characteristics and implementation fidelity of a coordinated school health childhood obesity prevention intervention. J Sch Health. 2015; 85: 8-16. Received on August 6, 2013 Accepted on June 27, 2014

C

hildhood obesity is considered a public health crisis in the United States. Obesity has tripled among American youth since 1980,1 and one third are currently overweight or obese.2 Overweight and obesity in children are especially problematic; not only are these children more likely to have risk factors for chronic disease than children of normal weight, but they are also more likely to suffer from a host of health concerns and face psychological problems such as low self-esteem.3 Overweight and obese children are also likely to become obese adults,4 who disproportionately suffer from chronic diseases,

including diabetes, heart disease, and cancer compared with those of normal weight.3,4 Schools have been consistently identified as an important setting to improve children’s physical activity and nutritious eating.4-7 Correspondingly, multicomponent school-based interventions have been found to decrease obesity among children8,9 and improve their learning outcomes.10,11 However, information about the core components of interventions and schools’ ability to implement them with fidelity has been a more limited area of research.12 Examining implementation fidelity is often done through process

a Doctoral

Candidate, Associate Instructor, ([email protected]), Department of Applied Health Science, Indiana University School of Public Health-Bloomington, 1025 East 7th Street, Room 116, Bloomington, IN 47405. b Research Scientist, ([email protected]), Center on Education and Lifelong Learning, Indiana Institute on Disability and Community, Indiana University, 2853 East Tenth Street, Bloomington, IN 47408-2696. c Research Associate, ([email protected]), Center on Education and Lifelong Learning, Indiana Institute on Disability and Community, Indiana University, 2853 East Tenth Street, Bloomington, IN 47408-2696.

8



Journal of School Health



January 2015, Vol. 85, No. 1



© 2014, American School Health Association

evaluation, which essentially asks ‘‘Was the program carried out as planned?’’12(p5) . Process evaluation is a necessary component of a comprehensive and rigorous evaluation framework,12 yet has not received as much attention as evaluation approaches that assess program effectiveness. Conducting process evaluation is also necessary to avoid type III error, which occurs when one fails to find a program effect only because the program is poorly implemented, not because of the program in and of itself.12 This study responds to the call for more studies to examine processbased outcomes measures9,12-14 by examining schools’ implementation of the HEROES (Healthy, Energetic Ready, Outstanding, Enthusiastic, Schools) Initiative, a school-based childhood obesity prevention intervention in southern indiana, northwestern Kentucky, and southeastern Illinois. Based on the recommended Centers for Disease Control and Prevention’s coordinated school health (CSH) approach,15 the HEROES Initiative provides financial and technical support for data-based schoollevel interventions intended to increase opportunities for physical activity and healthy eating among students and staff, to integrate health and wellness education into the overall academic curriculum, to engage parents and community-based organizations to enhance the healthfulness of the school environment, and to empower the school to develop and implement wellness policies that support healthy lifestyles for students, their families, and the school staff. The HEROES Initiative focuses on 9 domains central to CSH, described in more detail below. Schools are expected to implement multiple activities specific to each of these domains. Each participating school makes a 3-year commitment to the HEROES Initiative. Programming efforts are led by a school wellness coordinator, funded in part by the initiative, and a wellness committee within each school that consists of several stakeholders, including teachers, the school nurse, the guidance counselor, principal, and parents. A more detailed description of the HEROES Initiative is available elsewhere.16 This study provides an overview of how the implementation of the HEROES Initiative was assessed and participating schools’ fidelity to each component of the HEROES intervention. This is one of only a few studies that reports implementation fidelity of a CSH-based intervention. This study also extends the literature by

examining the association between multiple schoollevel characteristics and implementation fidelity scores in order to gain a better understanding of what school-level features may facilitate or hinder effective implementation. The study specifically assessed the relationship between implementation fidelity and elementary vs. middle/high schools, public vs. private schools, implementation coordinated at the district level vs. the school level, and socioeconomic status (SES). To the authors’ knowledge, this is the first study to examine school-level variables in relationship to implementation fidelity.

METHODS Participants The sample consisted of the 17 schools that participated in the HEROES Initiative during the 2011-2012 school year, which included 10 elementary schools, 4 middle schools, and 3 high schools. Five of the schools were in their first year of funding; 9 were in their second year, and 3 were in their third year. A full listing of various school characteristics, including the independent variables for the study, can be found in Table 1. Schools are listed in no particular order, and their names have been removed to protect their confidentiality. The total number of students reached by the program within the 17 schools was 6884. The study was approved by the researchers’ Institutional Review Board. Procedure and Instruments Fidelity to the HEROES intervention was measured during site visits to each participating HEROES school during April and May of 2012 through a university-community partnership. During the site visits, 3 trained researchers familiar with the HEROES Initiative and with backgrounds in school health conducted key stakeholder interviews with the school wellness coordinator, a school administrator (most often the principal), and cafeteria manager based on a semistructured interview protocol. Site visitors also observed various elements of the school environment, such as educational materials posted in the cafeteria and students engaging in physical education (PE) classes. Materials provided by the wellness coordinators were also reviewed, such as school newsletters and pictures and promotional

d Professor, ([email protected]), Department of Applied Health Science, Indiana University School of Public Health-Bloomington, 1025 East 7th Street, Room 116, Bloomington, IN 47405. e Doctoral Candidate, Associate Instructor, ([email protected]), Department of Applied Health Science, Indiana University School of Public Health-Bloomington, 1025 East 7th Street, Room 116, Bloomington, IN 47405.

Address correspondence to: Alyssa M. Lederer, Doctoral Candidate, Associate Instructor, ([email protected]), Department of Applied Health Science, Indiana University School of Public Health-Bloomington, 1025 East 7th Street, Room 116, Bloomington, IN 47405. The project described was supported by grants fromthe Welborn Baptist Foundation (WBF), grant number: 4440311, 4443010, and 4443011. The content is solely the responsibility of the authors and does not necessarily represent the official views of the WBF.

Journal of School Health



January 2015, Vol. 85, No. 1



© 2014, American School Health Association • 9

Table 1. HEROES Schools’ Characteristics School

Funding Year

Enrollment

% Non-White

% Free/ Reduced Lunch

Elementary/Middle/ High School

Public/ Private

District/Building Implementation

School 1 School 2 School 3 School 4 School 5 School 6 School 7 School 8 School 9 School 10 School 11 School 12 School 13 School 14 School 15 School 16 School 17

3 3 3 2 2 2 2 2 2 2 2 2 1 1 1 1 1

668 180 327 768 253 381 391 221 174 325 153 229 423 781 659 613 424

18.6 2.8 4.8 10.2 47.0 71.4 2.1 2.2 4.5 4.9 17.4 1.7 48.2 52.9 5.0 2.8 26.4

53.3 22.8 53.7 44.0 69.9 94.5 44.1 5.9 14.4 8.3 22.4 45.9 77.8 68.4 39.3 33.2 54.2

Middle Elementary Elementary Middle High Elementary Elementary Elementary Elementary Elementary Elementary Elementary Middle High Middle High Elementary

Public Public Public Public Public Public Public Private Private Private Private Public Public Public Public Public Public

Building Building Building District District District Building Building Building Building Building Building District District Building Building Building

HEROES, Healthy, Energetic Ready, Outstanding, Enthusiastic, Schools.

materials from school events. The multiple sources of information were used for data triangulation purposes and to validate the qualitative interviews.17 The interview protocol was developed, organized, and administered based on the 9 domains critical to the HEROES Initiative, which coincide with 5 of 8 CSH components central to obesity prevention in addition to recommendations for CSH program implementation:18,19 (1) invested administrative involvement; (2) engaged and active HEROES coordination within the school; (3) development, awareness, and implementation of district and school wellness policies; (4) display and integration of nutrition education materials into the overall academic curricula; (5) healthy food service in the cafeteria and during school activities; (6) implementation of PE and activity programs; (7) implementation of and participation in staff wellness activities; (8) family and community communication and involvement; and (9) development and engagement of a student-based School Wellness Advocacy Group (SWAG) (for middle and high schools only). Several open-ended questions assessed various programmatic and policy changes that schools were expected to make related to each of the 9 domains. Although the wellness coordinator, principal, and food service manager were asked a subset of similar questions to provide a comparison between responses and an additional layer of trustworthiness, the primary focus of each interview was on the area most relevant to the stakeholder’s involvement in the intervention (eg, the cafeteria manager and food service). Each school site visit typically lasted approximately 3 hours, with the majority of the time spent interviewing the wellness coordinator. 10



Journal of School Health



January 2015, Vol. 85, No. 1



The qualitative data ascertained during the site visits were used to inform quantitative scoring of the 9 domains for each school. Schools could receive a ‘‘4’’ (exists and exceeds standard), ‘‘3’’ (exists and meets standard), ‘‘2’’ (exists but does not meet standard), ‘‘1’’ (does not exist), or ‘‘N/A’’ (based on implementation year or school level) on 2 to 5 indicators associated with each domain. Scores were determined based on a rubric to ensure consistent scoring. The rubric was developed by the study team in consultation with the intervention’s developer/funder and based on the requirements of schools participating in the HEROES Initiative. The study team also communicated throughout the site visits and met once the site visits were complete to discuss the few unanticipated questions that emerged that could have affected particular indicator scores. The study team came to a consensus on how to address these issues and modified any scores impacted from the discussion to assure uniformity. Schools were provided with individualized reports detailing their implementation fidelity levels that included both the quantitative scores and qualitative explanatory comments in order to provide formative feedback to encourage improvements. In a final attempt to ensure accuracy of the scores, once all of the site visit reports were drafted, a member of the study team who was not present at the site visit reviewed each report to confirm that the quantitative scores aligned with the qualitative feedback. Implementation data were aggregated for the purposes of the current analyses. Full copies of the interview protocol, domain indicators, scoring rubric, and report template can be made available by the authors. © 2014, American School Health Association

Measures Domain scores. Eight HEROES domains central to the CSH model were included in the analyses: administrative involvement (indicators related to principal involvement and support), HEROES coordination (indicators related to the establishment, maintenance and composition of a school wellness committee), wellness policy (indicators related to knowledge of a district wellness policy and development of a schoolspecific wellness policy), nutrition education (indicators related to display of educational materials and nutrition education curriculum integration), food service (indicators related to healthy changes to foods served in the cafeteria and during school events), PE/activity (indicators related to using a formalized curriculum in PE classes, use of heart rate monitors, establishment of a walking program and/or other cardiovascular activities outside of PE, high levels of participation, and expansion of physical activity infrastructure), staff wellness (indicators related to offering a fall and spring health fair for staff, regularly scheduled educational and fitness programs, and high levels of participation), and family and community involvement (indicators related to hosting a family health fair with multiple community partners and communication efforts with students’ families). The SWAG domain was not included because it was introduced as a component of the initiative in the middle of the school year, which did not allow ample time for its implementation by the schools. Given that indicator scores with a ‘‘3’’ and ‘‘4’’ were both considered sufficient levels of implementation for HEROES, a ‘‘3’’ was considered the desired score for fidelity and all scores of ‘‘4’’ were converted to ‘‘3.’’ As such, it should be noted that percent scores represent the extent to which schools achieved, but did not necessarily exceed, the standards of desired implementation. Indicator scores for each domain were then averaged to create each domain score. Total implementation score. An overall implementation percentage was calculated for each school based on the aggregated indicator scores the school attained for all domains, divided by the overall score possible. Elementary vs. middle/high schools. Schools were dichotomized into 2 groups: elementary schools (N = 10) and middle/high schools (N = 7). Middle and high schools were combined because, in comparison with elementary schools, they tend to be similar in size and structure, have similar PE and PE teacher certification requirements, and similar availability of competitive foods such as a` la carte items and snack bars. Initial analyses confirmed similar scores between the middle and high schools. Public vs. private schools. Schools were dichotomized as private (N = 4) or public schools (N = 13). Of note, all private schools in the sample were Catholic affiliated. Journal of School Health



District- vs. building-level implementation. Five of the schools were part of the same large urban Indiana district. Because fiscal matters, including grant dollars, are often managed by the district rather than by individual schools, this district elected to pool HEROES funds from several schools to coordinate and centralize school wellness efforts at the district level. These efforts included employing a district-level coordinator to support HEROES activities such as organizing meetings and health fairs, which resulted in fewer responsibilities for the individual school wellness coordinators. In addition, the district purchased a parent wellness newsletter prepared by an outside company, rather than having schools develop newsletters individually. The district also coordinated communication between the program funder and the schools, rather than have school wellness coordinators communicate directly. All other schools (N = 12) implemented the HEROES Initiative at the individual school building level. Socioeconomic status. Student eligibility for free and reduced lunch served as a proxy measure of SES and was dichotomized as students who were eligible for free/reduced-price lunch and students who were not eligible for the program. These numbers were retrieved through the Department of Education for each state and the most recent data available for each school were used (2011-2012 for public schools; 2009-2010 for private schools). According to the United States Department of Agriculture (USDA) income eligibility guidelines,20 students from families with 130% income poverty level are eligible for free meals and those from families with 185% of income poverty level are eligible for reduced price meals. The schools were then dichotomized into ‘‘low’’ percentage of students eligible for free/reduced lunch (N = 7) and ‘‘high’’ percentage of students eligible for free/reduced lunch (N = 10). These groupings were based on a K means cluster analysis, an algorithm approach that identifies relatively homogeneous groups to allow for multiple comparisons.21 Schools in the high percentage group had between 44.0% and 94.5% free/reduced lunch rate, and schools in the low percentage group had between 5.9% and 39.3% free/reduced lunch rate. Data Analysis The 17 schools’ data were aggregated to complete the analyses. Means and SDs were calculated for total and domain implementation scores. Two-way analyses of variance (ANOVAs) were conducted that examined the main effects of each school-level independent variable on the dependent variables, which were total implementation score and the 8 domain scores. The assumption of normality was met after conducting the natural log transformation of the PE/activity, nutrition

January 2015, Vol. 85, No. 1



© 2014, American School Health Association • 11

education, and administrative involvement domains. Interactions were unable to be examined because of the small sample size. All analyses were conducted in SPSS version 20 (IBM, Armonk, NY). Statistical significance was set a priori at p < .05, and given the small sample size marginal significance was reported at p < .10. Bonferroni adjustment to guard against inflated type I error due to multiple tests on the same dependent variable was not made as reduced statistical power due to the small sample size was judged to outweigh the inflated propensity for type I error.

Table 2. Domain Scores Domain

Mean Score

SD

2.83 2.81 2.73 2.65 2.64 2.36 2.25 2.06

0.29 0.24 0.32 0.42 0.36 0.42 0.40 0.63

Administrative involvement Family/community involvement HEROES coordination Nutrition education Physical education/activity Staff wellness Food service Wellness policies

HEROES, Healthy, Energetic, Ready, Outstanding, Enthusiastic Schools.

DISCUSSION RESULTS Schools’ total implementation scores ranged from 71.6% to 96.4% with a mean of 84.19% (SD = 7.58%). As Table 2 shows, mean domain scores ranged from 2.06 (SD = 0.63) (wellness policies) to 2.83 (SD = 0.29) (administrative involvement). Results from the 2-way ANOVAs revealed that elementary vs. middle/high schools was the only characteristic that had a significant impact on total implementation score (F(1,10) = 9.627, p = .009), with elementary schools demonstrating higher levels of implementation (mean = 89.12%, SD = 5.57) than middle/high schools (mean = 77.14%, SD = 2.91). Schoollevel characteristics appeared to be unrelated to the PE/ activity, nutrition education, family/community involvement, and HEROES coordination domains. Elementary schools also scored marginally better (mean = 2.47, SD = 0.37) than middle/high schools (mean = 1.94, SD = 0.20) on the food service domain (F(1,10) = 3.232, p = .097). Three school characteristics significantly differed within the staff wellness domain. Elementary schools (mean = 2.54, SD = 0.39) outperformed middle/high schools (mean = 2.10, SD = 0.32) (F(1,10) = 7.15, p = .020), and public school implementation (mean = 2.39, SD = 0.43) was stronger than private school implementation (mean = 2.25, SD = 0.41) (F(1,10) = 6.96, p = .022). Building level implementation was marginally better (mean = 2.51, SD = 0.39) than implementation at the district level (mean = 2.0; SD = 0.21) (F(1,10) = 3.473, p = .087). For the administrative involvement domain, schools implementing at the building level (mean = 2.95, SD = 0.12) had higher scores than those implementing at the district level (mean = 2.54, SD = 0.39) (F(1,10) = 4.20, p = .063). Private schools implemented the wellness policy domain marginally better (mean = 2.63, SD = 0.48) than public schools (mean = 1.88, SD = 0.58) (F(1,10) = 3.322, p = .063). Table 3 provides each ANOVA table for overall implementation and all of the domains. 12



Journal of School Health



January 2015, Vol. 85, No. 1



This study describes recent process evaluation findings for the HEROES Initiative. A summary and discussion of the schools’ overall implementation of the HEROES Initiative are presented first, followed by possible interpretations of the comparisons between the 4 school-level characteristics results. Overall Implementation Findings Although there was some variability, results demonstrate that schools were generally able to implement the HEROES program with high fidelity. When examining each specific HEROES domain, schools most successfully garnered administrative involvement and family/community involvement and established HEROES coordination. The indicators associated with these domains were fairly prescribed in comparison with the other HEROES domains, which may have been partially responsible for the strong degree of implementation. It is also possible that the administrative involvement domain scores were particularly high due to schools’ self-selection into the intervention and that principals had to be signatories on the grant application and were often involved in the grant writing process. Schools faced the most challenges in the food service, wellness policies, and staff wellness domains. Regarding food service, the HEROES intervention coincided with new USDA requirements for reimbursable breakfast and lunch. Given this, cafeteria staff likely prioritized these changes over the modifications stipulated by HEROES. Some cafeteria managers also reported concerns about food waste if students did not favor changes made, a reality that has emerged at other schools attempting to implement healthier lunch options.22 Another impediment was resistance to reducing or eliminating competitive foods such as vending machines and concession stands and unhealthy food fundraisers, as these products generate revenue for the schools. For the staff wellness domain, efforts targeting school staff may have been perceived as less important than other components of HEROES, perhaps because

© 2014, American School Health Association

Table 3. School Characteristics’ Relationship With HEROES Domains

Total implementation percentage Level Type District vs. building SES Physical education/activity Level Type District vs. building SES Nutrition education Level Type District vs. building SES Food service Level Type District vs. building SES Staff wellness Level Type District vs. building SES Family/community involvement Level Type District vs. building SES HEROES coordination Level Type District vs. building SES Administrative involvement Level Type District vs. building SES Wellness policies Level Type District vs. building SES

df

F

4 1 1 1 1 4 1 1 1 1 4 1 1 1 1 4 1 1 1 1 4 1 1 1 1 4 1 1 1 1 4 1 1 1 1 4 1 1 1 1 4 1 1 1 1

5.825 9.627 0.025 0.636 0.095 0.745 1.933 0.269 1.074 1.328 0.218 0.067 0.004 0.003 0.003 2.93 3.232 1.305 0.062 0.344 6.156 7.15 6.96 3.473 0.279 0.3 0.034 0.483 0.011 0.726 2.729 3.126 1.29 1.624 0.14 2.582 0.531 0.069 4.2 0.03 2.633 1.25 3.322 0.164 1.472

p-Value .008** .009** .878 .441 .763 .58 .19 .613 .321 .272 .923 .8 .949 .956 .956 .066m .097m .276 .808 .568 .006** .020* .022* .087m .607 .873 .857 .5 .919 .411 .080m .102 .278 .227 .715 .091m .48 .797 .063m .866 .087m .285 .093m .693 .248

m p < .10. ∗∗ p < .05. ∗∗ p < .01.

HEROES, Healthy, Energetic Ready, Outstanding, Enthusiastic Schools; SES, socioeconomic status.

not all staff members were invested in the initiative, or they may have been comfortable participating on behalf of their students, but did not want to extend this involvement to their own behavior. In addition, indicators within the staff wellness domain included the amount of staff participation in the activities provided. Because the types of activities offered (eg, fitness classes and challenges, lunch-and-learns) may Journal of School Health



have been new for many staff members, it may take more time to build participation levels. Although staff members generally were aware of their district wellness policy, few schools had created a school-specific wellness policy, a requirement of the HEROES intervention. It seems that this type of policy development must be supported and enforced by the school principal, and despite the administrative involvement domain ranking highly, principals may not have been willing or able to prioritize this particular element of the intervention. In addition, even though HEROES schools were willing to implement additional programming above and beyond their district wellness policy, a number of schools considered policy to be something that was developed and enforced at the district, not school, level. Although other school-based obesity prevention programs have reported process evaluation outcomes, comparing them with the current findings is difficult given the diverse measures used. Furthermore, many of these studies report participation, satisfaction, and implementation of specific strategies23-25 rather than multifaceted intervention areas, as with the domains assessed in the current research. Some scholars have recommended that interventions based on the CSH model use school portfolios as a way to evaluate implementation, but published literature has focused on their use and strengths and weaknesses rather than actual portfolio findings.26-28 Furthermore, although portfolios can be a useful tool for schools with limited evaluation resources and as a means to showcase their CSH efforts, the self-reported nature of this approach may lead to biased findings. This study is unique given it is one of the first interventions explicitly based on the CSH approach to describe process evaluation results and it was conducted by an objective third party. An exception is Valois and Hoyle,29 who reported process evaluation data for 11 critical program elements from the Mariner Project, a 3-year pilot CSH intervention with 7 schools in South Carolina. The program aimed to improve students’ nutrition and physical activity, among several other health issues. The researchers found that implementation met established standards; however, although some program measures were similar to HEROES, they differed enough to preclude comparing findings. For example, the Mariner Project included a community linkages measure, but HEROES combines community with family involvement. This is a common challenge among process evaluation studies, as measures must be specific to the intervention expectations, but programs vary widely.30 Perhaps most alike was the measurement of administrative support/buy-in. Similar to HEROES, the Mariner Project found adequate or above adequate performance in this area, and notably that positive implementation of

January 2015, Vol. 85, No. 1



© 2014, American School Health Association • 13

this element in combination with other personnelbased measures (eg, having a wellness coordinator and council) served as the foundation for the success of the other program components. CSH multisite evaluative studies are forthcoming that should supplement the limited available research;31 however, it is unclear if process evaluation measures will be included. Comparison of School-Level Characteristics Findings Whereas the HEROES implementation evaluation framework uses similar methods to other school-based physical activity and nutrition programs,30 this study extends evaluation findings into a novel area of scholarship by examining the role of school-level characteristics on implementation fidelity. These findings should be considered preliminary, and their interpretation requires some speculation given there is no comparable literature to draw from, but they offer exciting new directions for future evaluation efforts. Three of the independent variables—elementary vs. middle/high school, public vs. private school, and district vs. building level implementation—were associated with implementation of one or more HEROES domain. Elementary vs. middle/high school was related to fidelity most often, with elementary schools consistently having better scores than middle/high schools. Given the relatively smaller size of these schools, it may have been easier to plan activities that included a larger proportion of students and staff. In addition, elementary schools may be accustomed to more centralized coordination efforts and have a more homogenous group of teachers (who teach all subject areas) rather than the academic subject silos typical of middle/high schools. This could lead to greater receptivity to the initiative among school staff, as the elementary school teachers may have a more ‘‘generalist’’ approach to their students compared to the middle/high school teachers. Also, elementary schools did not face some of the same obstacles to implementation as middle/high schools that were identified in the overall process evaluation. For example, given that elementary schools do not typically host athletic events, they were not faced with the opportunity to limit unhealthy options in concession stands. Elementary schools also do not typically serve competitive foods in their cafeteria, as is done in secondary schools.32 Given that elementary schools did not have the option of modifying these existing policies whereas middle/high schools did, they gained an advantage given the often difficult nature of school policy change. The finding that intervention implementation was better at the individual building level rather than the district level is particularly interesting given that CSH programs are typically implemented at the district level.32-34 It is possible that building level implementation allows schools to improve tailoring of programs, a 14



Journal of School Health



January 2015, Vol. 85, No. 1



best practice for intervention development35 and take ownership over their programming, leading to more positive outcomes. In addition, district level implementation of school wellness policy and program changes may inadvertently reduce levels of involvement among school wellness coordinators and other building level staff. SES did not appear to influence implementation fidelity. Interestingly, schools with students from predominantly low SES households have been found to offer fewer nutrition and physical activity opportunities than schools comprised of higher SES students,36,37 so it is promising that SES did not appear to be a factor in the HEROES implementation. Optimistically, this demonstrates that schools can implement HEROES regardless of students’ personal characteristics. However, the fact that funding was provided for many of the school-level changes and the schools’ self-selection into the intervention may have been factors that contributed to consistent implementation among schools serving higher and lower proportions of low SES students. It is also possible that the small difference in mean SES percentages between 2 schools (44.0% vs. 39.3%) in the K means cluster analysis influenced these findings. Limitations Whereas this study has made an important contribution to the literature, it is not without limitations. Although 17 schools is a large number in practice, it is a small sample for statistical analysis, and therefore, more sophisticated statistical analyses could not be conducted. Accordingly, although the differences identified between school-level characteristics can be viewed with confidence given the small sample size, additional differences may have been missed owing to type II error. Additionally, while multiple sources were used to determine the fidelity scores, they were based on 1 day of observation. Data were primarily selfreported, making them subject to social desirability, especially in the administrative involvement domain. While a rubric and regular communication was used to ensure consistency among the study team’s school implementation scores, inter-rater reliability of the rubric was not initially assessed. Furthermore, each private school was religiously affiliated, which is a possible confounder. Finally, it is possible that the Indiana district examined in this study may not be generalizable to other school districts. Conclusions This study adds to the literature regarding implementation fidelity findings for a childhood obesity prevention intervention grounded in the CSH approach and has established a new area of research related to the relationship between school-level characteristics

© 2014, American School Health Association

and implementation fidelity. More research is necessary to substantiate and expand upon the findings presented, especially regarding school-level variables. Future studies should include more rigorous analyses with a larger sample size. They should also examine school-level characteristics that could not be compared in this sample, such as rural vs. urban schools, intervention cohorts, and schools of varying academic achievement levels. Longitudinal studies would also be beneficial. For example, Caballero et al38 found that several process measures improved over the course of a 3-year intervention. However, longitudinal studies could prove challenging for interventions structured similarly to HEROES, in which yearly process evaluation results prompt changes to intervention components and evaluation measures in the interest of quality improvement, limiting their comparison over time. Additionally, future studies should assess the relationship between process and impact measures. Rosas et al11 found that higher levels of CSH program implementation were associated with better schoollevel academic performance, and Seo et al16 found that HEROES schools with higher total implementation scores had greater improvement in children’s vigorous physical activity. Additional research that broadens these behaviors and examines the specific program components that lead to enhanced behavioral outcomes would be worthwhile.

IMPLICATIONS FOR SCHOOL HEALTH This research supplements an emerging area of study on interventions using the CSH approach. This study demonstrates that interventions grounded in CSH can be implemented with fidelity. However, some areas are more easily implemented than others. For practitioners planning to implement CSH in their schools, there are several lessons that can be learned from the HEROES Initiative. School staff may benefit from being given prescribed implementation strategies, as long as they can be tailored to meet individual schools’ needs and cultures. School staff must also be seen as stakeholders in CSH interventions, and their buy in should be established before initiation of CSH efforts. Practitioners should also help staff to recognize that students likely view them as role models39 and enhance staff members’ understanding of the interdependent nature of multilevel interventions.40 This study also revealed that some aspects of the HEROES Initiative implementation varied by school-level characteristics. However, implementation was not affected by SES, which holds promise that CSH interventions may be implemented effectively regardless of students’ demographic characteristics. School health researchers should further explore this new area of inquiry. However, the finding that implementation at the building level may Journal of School Health



yield better fidelity compared to implementation at the district level has immediate implications given current practice. School health practitioners should feel empowered to carry out CSH programs autonomously within their schools. In sum, examining process measures can help school health practitioners garner a comprehensive understanding of schoolbased interventions and ways in which future interventions can be improved, an important step in curtailing childhood obesity. Human Subjects Approval Statement This study was approved by the Indiana University Institutional Review Board.

REFERENCES 1. Ogden CL, Carroll MD, Curtin LR, Lamb MM, Flegal KM. Prevalence of high body mass index in US children and adolescents, 2007-2008. JAMA. 2010;303(3):242-249. 2. Ogden CL, Carroll MD, Kit BK, Flegal KM. Prevalence of obesity and trends in body mass index among US children and adolescents, 1999-2010. JAMA. 2012;307(5):483-490. 3. Centers for Disease Control and Prevention. Childhood obesity facts. Available at: http://www.cdc.gov/healthyyouth/ obesity/facts.htm. Accessed July 23, 2013. 4. United States Department of Health and Human Services. The Surgeon General’s Vision for a Healthy and Fit Nation. Rockville, MD: US Department of Health and Human Services, Office of the Surgeon General; 2010. 5. Centers for Disease Control and Prevention. Nutrition, physical activity, and obesity: school health guidelines to promote healthy eating and physical activity. Available at: http://www. cdc.gov/healthyyouth/npao/strategies.htm. Accessed July 23, 2013. 6. Lobstein T, Baur L, Uauy R. Obesity in children and young people: a crisis in public health. Obes Rev. 2004;5(Supp1):4-85. 7. Physical Activity Guidelines for Americans Midcourse Report Subcommittee of the President’s Council on Fitness, Sports & Nutrition. Physical Activity Guidelines for Americans Midcourse Report: Strategies to Increase Physical Activity among Youth. Washington, DC: US Department of Health and Human Services; 2012. 8. Gonzales-Suarez C, Worley A, Grimmer-Somers K, Dones V. School-based interventions on childhood obesity: a metaanalysis. Am J Prev Med. 2009;37(5):418-427. 9. Khambalia AZ, Dickinson S, Hardy LL, Gill T, Baur LA. A synthesis of existing systematic reviews and meta-analyses of school-based behavioural interventions for controlling and preventing obesity. Obes Rev. 2012;13(3):214-233. 10. Murray NG, Low BJ, Hollis C, Cross AW, Davis SM. Coordinated school health programs and academic achievement: a systematic review of the literature. J Sch Health. 2007;77(9):589-600. 11. Rosas S, Case J, Tholstrup L. A retrospective examination of the relationship between implementation quality of the coordinated school health program model and school-level academic indicators over time. J Sch Health. 2009;79(3):108115. 12. Linnan L, Steckler A. Process Evaluation for Public Health Interventions and Research. San Francisco, CA: Jossey-Bass; 2002. 13. Kropski JA, Keckley PH, Jensen GL. School-based obesity prevention programs: an evidence-based review. Obesity. 2008; 16(5):1009-1018. 14. Lohrmann DK. Process evaluation for school health professionals. J Sch Health. 2006;76(4):154-155.

January 2015, Vol. 85, No. 1



© 2014, American School Health Association • 15

15. Centers for Disease Control and Prevention. Coordinated school health. Available at: http://www.cdc.gov/HealthyYouth/CSHP/. Accessed July 23, 2013. 16. Seo DC, King MH, Kim N, Sovinski D, Meade R, Lederer AM. Predictors for moderate- and vigorous-intensity physical activity during an 18-month coordinated school health intervention. Prev Med. 2014;57(5):466-470. 17. Patton MQ. Qualitative Research & Evaluation Methods. 3rd ed. Thousand Oaks, CA: Sage Publications; 2002. 18. Centers for Disease Control and Prevention. Components of coordinated school health. Available at: http://www.cdc.gov/ healthyyouth/cshp/components.htm. Accessed July 23, 2013. 19. Centers for Disease Control and Prevention. How schools can implement coordinated school health. Available at: http://www. cdc.gov/healthyyouth/cshp/schools.htm. Accessed July 23, 2013. 20. United States Department of Agriculture Food and Nutrition Service. Income eligibility guidelines. Available at: http://www. fns.usda.gov/cnd/governance/notices/iegs/iegs.htm. Accessed March 16, 2013. 21. Scott AJ, Knott M. A cluster analysis method for grouping means in the analysis of variance. Biometrics. 1974;30(3): 507-512. 22. Just D, Price J. Default options, incentives and food choices: evidence from elementary school students. Public Health Nutr. 2013;16(12):2281-2288. 23. Neumark-Sztainer D, Story M, Hannan PJ, Rex J. New moves: a school-based obesity prevention program for adolescent girls. Prev Med. 2003;37(1):41-51. 24. Sahota P, Rudolf MCJ, Dixy R, Hill AJ, Barth JH, Cade J. Evaluation of implementation and effect of primary school based intervention to reduce risk factors for obesity. BMJ. 2001;323(7320):1027-1029. 25. Steckler A, Ethelbah B, Martin CJ, et al. Pathways process evaluation results: a school-based prevention trial to promote healthful diet and physical activity in American Indian third, fourth, and fifth grade students. Prev Med. 2003;37(Supp 1):S80S90. 26. Weiler RM, Pigg RM. The School Health Portfolio System: a new tool for planning and evaluating coordinated school health programs. J Sch Health. 2004;74(9):359-364. 27. Barnes SP, Torrens A, George V, Brown KM. The use of portfolios in coordinated school health programs: benefits and challenges to implementation. J Sch Health. 2007;77(4): 171-179.

16



Journal of School Health



January 2015, Vol. 85, No. 1



28. Shipley M, Lohrmann D, Barnes P, O’Neill J. Advantages of coordinated school health portfolios: documenting and showcasing achievements. J Sch Health. 2013;83(4):243-248. 29. Valois RF, Hoyle TB. Formative evaluation results from the Mariner Project: a coordinated school health pilot program. J Sch Health. 2000;70(3):95-103. 30. McGraw SA, Sellers D, Stone E, et al. Measuring implementation of school programs and policies to promote healthy eating and physical activity among youth. Prev Med. 2000;31(2):S86S97. 31. Centers for Disease Control and Prevention. Multisite evaluation of coordinated school health. Available at: http://www.cdc. gov/healthyyouth/evaluation/rapid/projects/multisite.htm. Accessed July 23, 2013. 32. Story M, Nanney MS, Schwartz MB. Schools and obesity prevention: creating school environments and policies to promote healthy eating and physical activity. Milbank Q. 2009;87(1):71-100. 33. Centers for Disease Control and Prevention. Program success stories. Available at: http://www.cdc.gov/healthyyouth/ stories/index.htm. Accessed July 23, 2013. 34. DeWitt N, Lohrmann DK, O’Neill J, Clark JK. A qualitative analysis of success stories from Michiana Coordinated School Health Leadership Institute participants. J Sch Health. 2011;81(2):727732. ´ 35. Bartholomew LK, Parcel GS, Kok G, Gottlieb NH, Fernandez ME. Planning Health Promotion Programs: An Intervention Mapping Approach. 3rd ed. San Francisco, CA: Jossey- Bass; 2011. 36. Nanney MS, Bohner C, Friedrichs M. Poverty-related factors associated with obesity prevention policies in Utah secondary schools. J Am Diet Assoc. 2008;108(7):1210-1215. 37. Gould R, Russell J, Barker ME. School lunch menus and 11 to 12 year old children’s food choice in three secondary schools in England—are the nutritional standards being met? Appetite. 2006;46(1):86-92. 38. Caballero B, Clay T, Davis SM, et al. Pathways: a schoolbased, randomized controlled trial for the prevention of obesity in American Indian schoolchildren. Am J Clin Nutr. 2003; 78(5):1030-1038. 39. Hendy HM, Raudenbush B. Effectiveness of teacher modeling to encourage food acceptance in preschool children. J Sch Health. 2000;34(1):61-76. 40. McLeroy KR, Bibeau D, Steckler A, Glanz K. An ecological perspective on health promotion programs. Health Educ Behav. 1988;15(4):351-377.

© 2014, American School Health Association

The relationship between school-level characteristics and implementation fidelity of a coordinated school health childhood obesity prevention intervention.

Curtailing childhood obesity is a public health imperative. Although multicomponent school-based programs reduce obesity among children, less is known...
534KB Sizes 2 Downloads 5 Views