obesity reviews
doi: 10.1111/obr.12185
Obesity Prevention
Process evaluation design and tools used in a kindergarten-based, family-involved intervention to prevent obesity in early childhood. The ToyBox-study O. Androutsos1, E. Apostolidou1, V. Iotova2, P. Socha3, J. Birnbaum4, L. Moreno5,6, I. De Bourdeaudhuij7, B. Koletzko4 and Y. Manios1 on behalf of the ToyBox-study group 1
Department of Nutrition and Dietetics,
Harokopio University, Athens, Greece; 2
Department of Pediatrics, Medical University
Varna, Varna, Bulgaria; 3The Children’s Memorial Health Institute, Warsaw, Poland; 4
Dr. von Hauner Children’s Hospital, University
of Munich Medical Centre, Munich, Germany; 5
GENUD (Growth, Exercise, Nutrition and
Development) Research Group, University of Zaragoza, Zaragoza, Spain; 6School of Health Science (EUCS), University of Zaragoza, Zaragoza, Spain; 7Department of Movement and Sport Sciences, Ghent University, Ghent, Belgium
Received 4 April 2014; revised 8 April 2014; accepted 9 April 2014
Address for correspondence: Yannis Manios, Department of Nutrition and Dietetics, Harokopio University, El. Venizelou 70,
Summary Process evaluation (PE) is used for the in-depth evaluation of the implementation process of health promotion programmes. The aim of the current paper was to present the PE design and tools used in the ToyBox-intervention. The PE design was based on a three-step approach, including the identification of ToyBoxspecific PE elements (step 1), the development of PE tools and harmonization of procedures (step 2), and the implementation of PE using standardized protocol and tools across the intervention countries (step 3). Specifically, to evaluate the implementation of the intervention, teachers’ monthly logbooks were recorded (dose delivered, fidelity, dose received); post-intervention questionnaires were completed by parents/caregivers and teachers (dose received); participation and attrition rates were recorded (recruitment, reach); and audit questionnaires and retrospective information on weather conditions were collected (physical and social environment within which the intervention was implemented). Regarding the teachers’ training sessions, the researchers who performed the trainings completed evaluation forms and documented teachers’ attendance after each training (dose delivered, fidelity, dose received) and teachers completed evaluation forms after each training (dose received). The PE performed in the ToyBox-intervention may contribute in the evaluation of its effectiveness, guide the revision of the intervention material and provide insights for future health promotion programmes and public health policy.
Kallithea 17671, Athens, Greece. E-mail: [email protected]
Keywords: Kindergarten, obesity prevention, preschool children, process evaluation. obesity reviews (2014) 15 (Suppl. 3), 74–80
ToyBox-study group: Coordinator: Yannis Manios; Steering committee: Yannis Manios, Berthold Koletzko, Ilse De Bourdeaudhuij, Mai Chin A Paw, Luis Moreno, Carolyn Summerbell, Tim Lobstein, Lieven Annemans, Goof Buijs; External advisors: John Reilly, Boyd Swinburn, Dianne Ward; Harokopio University (Greece): Yannis Manios, Odysseas Androutsos, Eva Grammatikaki, Christina Katsarou, Eftychia Apostolidou, Eirini Efstathopoulou; Ludwig Maximilians Universitaet Muenchen (Germany): Berthold Koletzko, Kristin Duvinage, Sabine Ibrügger, Angelika Strauß, Birgit Herbert, Julia Birnbaum, Annette Payr; Ghent University (Belgium): Department of Movement and Sports Sciences: Ilse De Bourdeaudhuij, Greet Cardon, Marieke De Craemer, Ellen De Decker; Department of Public Health: Lieven Annemans, Stefaan De Henauw, Lea Maes, Carine Vereecken, Jo Van Assche, Lore Pil; VU University Medical Center EMGO Institute for Health and Care Research (the Netherlands): EMGO Institute for Health and Care Research: Mai Chin A Paw, Saskia te Velde; University of Zaragoza (Spain): Luis Moreno, Theodora Mouratidou, Juan Fernandez, Maribel Mesana, Pilar De Miguel-Etayo, Esther M. González-Gil, Luis Gracia-Marco, Beatriz Oves; Oslo and Akershus University College of Applied Sciences (Norway): Agneta Yngve, Susanna Kugelberg, Christel Lynch, Annhild Mosdøl, Bente B Nilsen; University of Durham (UK): Carolyn Summerbell, Helen Moore, Wayne Douthwaite, Catherine Nixon; State Institute of Early Childhood Research (Germany): Susanne Kreichauf, Andreas Wildgruber; Children’s Memorial Health Institute (Poland): Piotr Socha, Zbigniew Kulaga, Kamila Zych, Magdalena Góz´dz´, Beata Gurzkowska, Katarzyna Szott; Medical University of Varna (Bulgaria): Violeta Iotova, Mina Lateva, Natalya Usheva, Sonya Galcheva, Vanya Marinova, Zhaneta Radkova, Nevyana Feschieva; International Association for the Study of Obesity (UK): Tim Lobstein, Andrea Aikenhead; CBO B.V. (the Netherlands): Goof Buijs, Annemiek Dorgelo, Aviva Nethe, Jan Jansen; AOK-Verlag (Germany): Otto Gmeiner, Jutta Retterath, Julia Wildeis, Axel Günthersberger; Roehampton University (UK): Leigh Gibson; University of Luxembourg (Luxembourg): Claus Voegele.
74 15 (Suppl. 3), 74–80, August 2014
© 2014 World Obesity
obesity reviews
ToyBox-intervention: process evaluation O. Androutsos et al.
Introduction The variability observed in the effectiveness of previous school-based interventions may be attributed to the level of their implementation (1,2). One of the key characteristics of the interventions that were previously found to be effective is the use of process evaluation in order to monitor and document the level of their implementation (3). More specifically, process evaluation can be useful to examine the relationship between the design and implementation of the intervention with its impact and outcome, as well as to assess what made it successful or not (4,5). Over and above these, the use of process evaluation is particularly important in multicentre studies where large local, cultural and sociodemographic diversities are observed and may influence their implementation compared with what was initially planned (6). The aim of the current paper was to present the process evaluation design and tools used in the ToyBoxintervention in order to assess how this intervention was actually implemented, which factors influenced its implementation and effectiveness, whether it reached the initially targeted audience, and what did the stakeholders suggest regarding the future improvement of the ToyBoxintervention and material.
75
conduct permanent environmental changes in the classroom/kindergarten (i.e. install water stations and the ‘magic snack plate’ and rearrange the classroom/ kindergarten to create some free space to assist children’s movement); (ii) promote children’s actual lifestyle behaviour on a regular basis (i.e. remind children to drink water regularly and eat healthy snacks daily, as well as to do short movement breaks for interrupting prolonged sitting time twice in the morning and twice afternoon and to perform physical education sessions two times per week with the children); and (iii) implement interactive classroom activities aiming at total class participation, minimum of 1 h per week. Teachers were also instructed to use the kangaroo hand puppet and perform these four lifestyle behaviours themselves, so as to enhance the effects of the intervention via role modelling. Similar approach was followed at home, where parents/caregivers were encouraged and advised, via simple and friendly to read newsletters, tip cards and posters, to apply relevant environmental changes at home, act as role models and implement these lifestyle behaviours together with their children. Three teachers’ training sessions were also conducted to train the teachers on how to implement the ToyBoxintervention. All trainings were performed locally by trained researchers using standardized protocol, presentations and timeplan (8). Only the teachers from the intervention kindergartens were invited to attend these sessions.
Methods Process evaluation in the ToyBox-intervention The ToyBox-intervention The ToyBox-intervention (http://www.toybox-study.eu) was implemented during the academic year 2012–2013 in six European countries (Belgium, Bulgaria, Germany, Greece, Poland, Spain) (7). The study sample at baseline comprised of 309 kindergartens and 7,056 children aged 3.5–5.5 years, their parents/caregivers and their teachers, recruited from three socioeconomic groups. The ToyBoxintervention aimed to promote preschool children’s water consumption, healthy snacking, physical activity and limit/ interrupt their sedentary behaviour by improving children’s physical and social environment both at the kindergarten and at home. More specifically at kindergartens, teachers were provided with five handbooks and were asked to (i)
Step 1 Identification of PE elements to be assessed in the ToyBox-intervention
A three-step approach was used for the design of the process evaluation in the ToyBox-intervention (Fig. 1). More specifically, the next steps were followed: Step 1: identification of process evaluation elements to be assessed in the ToyBox-intervention The first step took place from September to December 2011. Based on the steps and principles described in the conceptual framework by Saunders et al. (9), we identified and assessed the next process evaluation elements as following: Recruitment/reach: the recruitment procedure was monitored against the standardized protocol of the study. Furthermore, the proportions and the characteristics of
Step 2 Development of PE tools and harmonization of procedures
Step 3 Implementation of PE in the six intervention countries
Figure 1 Design of the ToyBox programme process evaluation (PE).
© 2014 World Obesity
15 (Suppl. 3), 74–80, August 2014
76 ToyBox-intervention: process evaluation O. Androutsos et al.
kindergartens/teachers/families agreeing to participate in the intervention and completing or dropping out from the intervention were recorded. Dose delivered (completeness): the intensity of actual implementation of the ToyBox-intervention was assessed. More specifically, the number of training sessions and the duration of each training were recorded. The degree of implementation/retention of the environmental changes and children’s actual behaviour by the kindergarten teachers, as well as the number, type and duration of the classroom activities were also recorded. Moreover, the delivery of the intervention material to the parents/caregivers (nine newsletters, eight tip cards, four posters) and the duration of the time devoted to the ToyBox-intervention by the teachers were assessed. Fidelity: the extent to which the ToyBox-intervention was implemented as initially planned was assessed. More specifically, the degree of adherence to the teachers’ training protocol (8) was recorded. The actual implementation of the environmental changes, children’s actual behaviour and classroom activities and the delivery of the intervention material to the parents/caregivers (newsletters, tip cards, posters) according to the ToyBox-intervention timeplan and material were also recorded. Moreover, the duration of the time devoted to the ToyBox-intervention by the teachers was assessed. Dose received (exposure and satisfaction): the extent to which children/parents/caregivers/teachers were exposed to the ToyBox-intervention, as well as the degree of their satisfaction with the ToyBox-intervention and material were assessed. Regarding the training sessions, the numbers of teachers who were invited and actually participated in these trainings were recorded. Moreover, teachers’ satisfaction with the trainings, with the researchers who implemented the trainings and with the ToyBox-intervention overall, and researchers’ subjective perception of teachers’ satisfaction with the trainings were assessed after the completion of each training. Teachers’ perceived barriers and facilitators for implementing each part of the ToyBoxintervention were also recorded. Also, the extent to which parents/caregivers were exposed to the material (newsletters, tip cards, posters) and to the ToyBox-intervention was recorded. Stakeholders’ (i.e. headmasters’/ management personnel’s, teachers’, parents’/caregivers’, children’s) satisfaction with the ToyBox-intervention and material (comprehensibility, readability, relevance, credibility and attractiveness of the material) was assessed and their suggestions for future improvement of the ToyBoxintervention and material were recorded. Context: several physical (e.g. weather conditions, kindergarten infrastructure/ facilities/equipment/kindergartens’ curriculum/personnel), social (e.g. socioeconomic status of the family, teachers’ health-related behaviours) and contextual (e.g. policies, legislation and rules) factors that may 15 (Suppl. 3), 74–80, August 2014
obesity reviews
have acted as barriers or facilitators to the implementation of the ToyBox-intervention were recorded. Moreover, information on potential contamination of the intervention between the intervention and the control kindergartens/ teachers was collected. Step 2: development of process evaluation tools and harmonization of procedures The second step was implemented from January to March 2012. During the design of the process evaluation, there was thoughtful consideration of various methodological components as those suggested by Saunders et al. (9) that resulted in the development of a series of process evaluation tools, which were tailor made for the ToyBoxintervention. More specifically, considering the issues of data sources and collection procedures, we developed the process evaluation tools in such a way so that they could be applied within the ToyBox-intervention (i.e. in means of personnel, time and budget availability) and at the same time allow gathering as much information as possible. In parallel, all procedures aimed to burden the teachers as little as possible. Based on the aforementioned issues, the process evaluation tools were designed to include mainly quantitative methods for researchers, teachers and parents/caregivers, as these have the advantage of being less time consuming and burdensome to complete while they can provide more objective and quantified information/data. However, some of the process evaluation tools designed for the teachers included open-ended questions to allow for the collection of unanticipated answers and subjective reports (5). All intervention countries used common process evaluation protocol and tools. Initially, the tools were translated into the six local languages and back-translated to English. Translation and back-translation procedures were harmonized and centrally coordinated and evaluated. Common datasets and codebooks were also used to enter the collected process evaluation data across all intervention countries. Step 3: implementation of process evaluation in the six intervention countries The third step was conducted from April 2012 to June 2013 and included the implementation of the process evaluation, which ran in parallel to the baseline and follow-up measurements and the implementation of the ToyBox-intervention. The implementation of the process evaluation was guided from a common protocol for the six intervention countries. This protocol included an explicit description of the purpose and use of the process evaluation tools and a timeplan for the application of each process evaluation tool. © 2014 World Obesity
obesity reviews
Results Based on the three-step approach that was applied for the design of the ToyBox-intervention process evaluation, common process evaluation protocol and tools were used across the six intervention countries. An overview of the process evaluation tools is presented in Table 1. For the process evaluation of the ToyBox-intervention, 12 process evaluation methods and tools were used (Table 1). Regarding the evaluation of the teachers’ training sessions two process evaluation tools were used. More specifically, immediately after the completion of each training, each teacher completed the ‘teacher’s evaluation form’ and the researchers who implemented the training completed the ‘researchers’ evaluation form’ (Supporting Information). The information obtained via the ‘researchers’ evaluation forms’ was used to assess whether the three training sessions were conducted in the six intervention countries and record the actual duration of each training (dose delivered/completeness), assess if the trainings were actually implemented as originally planned in the teachers’ training protocol (fidelity), capture how much did teachers enjoy and participate in each training (dose received/ satisfaction), and record how many of the invited teachers actually participated in each training (dose received/ exposure). Regarding the fidelity in the implementation of the trainings, the venue and the names of researchers who implemented the trainings were recorded, whereas the researchers who implemented the training were also asked to report any deviations from the protocol and the reasons for these deviations in an open-type question. On the other hand, the ‘teachers’ evaluation form’ was used to assess teachers’ satisfaction with the training, with the researchers who implemented the training and the ToyBoxintervention overall, as well as to record teachers’ suggestions for improving the ToyBox-intervention and material in the future. Weekly records of recruitment monitoring and information on participation rate and attrition rate were collected throughout the recruitment and the implementation phase of the ToyBox-intervention. This information was used to assess whether the recruitment in the six intervention countries was conducted according to the protocol and to collect information on the characteristics of the participants (teachers and families) who participated or dropped out from the intervention. Furthermore, the teachers completed a structured questionnaire, the ‘teachers’ monthly logbook’ (Supporting Information), via face-to-face meetings or phone calls with the researchers or via emails. This logbook was completed at the end each month of the intervention phase, resulting in six logbooks per class at the end of the intervention. For example, if one class had two or more teachers, we aimed to collect six logbooks at minimum from the total number © 2014 World Obesity
ToyBox-intervention: process evaluation O. Androutsos et al.
77
of teachers in this class. With this tool, the intensity (dose delivered/completeness) and the fidelity of the actual implementation of the ToyBox-intervention, as well as children’s/parents’/teachers’ exposure to the ToyBoxintervention and satisfaction with the ToyBox-intervention and material were assessed. For the evaluation of children’s, parents’/caregivers’ and teachers’ exposure and satisfaction, information on the attendance of children in the kindergarten was collected and two more process evaluation tools were applied, i.e. the ‘teachers’ post-intervention questionnaire’ and the ‘parents’/caregivers’ post-intervention questionnaire’ (Supporting Information), after the end of the ToyBox-intervention. The physical and social environment within which the ToyBox-intervention was implemented was assessed via a semi-structured interview with the headmaster and the teachers in each intervention and control kindergarten, i.e. the ‘audit questionnaire’ (Supporting Information) before and after the intervention, and information collected retrospectively on the weather conditions. Moreover, information on potential contamination between the intervention and the control kindergartens/teachers was collected.
Discussion A systematic approach was used for the design and implementation of the ToyBox process evaluation. Standardized protocol, methods and tools were used across the six intervention countries and all participants of this study (i.e. teachers, parents/caregivers, children and researchers) contributed with their opinion to the process evaluation. The completeness and fidelity of the ToyBox-intervention, the exposure of teachers/parents/caregivers/children to the ToyBox-intervention and teachers’/parents’/caregivers’/ children’s satisfaction (intelligibility, comprehensibility, readability, relevance, credibility and attractiveness) with the ToyBox-intervention and the material were assessed. The recruitment procedure, the reach and the contextual factors within which the ToyBox-intervention was implemented were also documented and wherever feasible quantified, while stakeholders’ suggestions for the future improvement of the ToyBox-intervention and material were recorded. Even though process evaluation is important for the evaluation of the effectiveness of health promotion programmes (2), many studies do not use process evaluation in their design (1,3). Indeed, only a limited number of studies have previously presented the process evaluation design, methods and tools used (9–13). Based on key learnings from previous studies (6), a three-step approach was for the design of process evaluation and the development of tailor-made process evaluation methods and tools in the 15 (Suppl. 3), 74–80, August 2014
78 ToyBox-intervention: process evaluation O. Androutsos et al.
obesity reviews
Table 1 The process evaluation tools used in the ToyBox-intervention Process evaluation tools
Process evaluation elements
Information/data collected
Monitoring of recruitment
Recruitment procedures/reach
- Weekly monitoring of recruited kindergartens/families per country and SES level - Numbers of kindergartens/families: approached, participated in the baseline evaluation/entered the ToyBox-intervention and participated in the follow-up evaluation - Characteristics of kindergartens/families dropping out
Dose delivered (completeness)
-
Implementation/retention of environmental changes in the classroom/kindergarten Implementation/retention of children’s lifestyle behaviours in the classroom/kindergarten Duration, number and type of classroom activities implemented Delivery of material (newsletters, tip cards, posters) to parents Time devoted to the ToyBox programme in each classroom
Fidelity
-
Implementation of the sequence of four behaviours according to the timeplan Implementation of classroom environmental changes according to the material and timeplan Implementation of children’s actual behaviour according to the material and timeplan Implementation of the classroom activities according to the material and timeplan Delivery of material (newsletters, tip cards, posters) to parents according to the timeplan and assessment on how this material was handed to the parents
Dose received (exposure and satisfaction)
- Teachers’ feedback on the comprehensibility, readability, relevance, credibility, attractiveness of the handbooks - Teachers’ feedback on children’s satisfaction with the ToyBox programme as implemented in the classroom each month - Teachers’ suggestions for improvement of the ToyBox programme and material after the completion of the intervention - Teachers’ feedback on the barriers and facilitators for implementing each part of the intervention
Attendance of children in the kindergarten
Dose received (exposure)
- Children’s attendance in the kindergarten
Teachers’ post-intervention questionnaire
Dose received (exposure and satisfaction)
- Teachers’ feedback on the barriers and facilitators for implementing the ToyBox programme - Teachers’ feedback on the comprehensibility, readability, relevance, credibility, attractiveness of the handbooks - Subjective teachers’ feedback on children’s, parents’, teachers’ and headmasters’/management personnel’s satisfaction with the ToyBox programme
Parent’s postintervention questionnaire
Dose received (exposure and satisfaction)
- Parents’ feedback on if the received and/or read each part of the material (newsletters, tip cards, posters) - Information on the ToyBox programme beyond the material (newsletters, tip cards, posters) given by the teacher to the parents - Parents’ satisfaction with the ToyBox programme - Parents feedback if they implemented as a family the activities included in the material (newsletters, tip cards) with their children at home - Subjective parents’ feedback on their spouse’s/partner’s and child’s satisfaction with the ToyBox programme - Parents’ feedback on the comprehensibility, readability, relevance, credibility, attractiveness of the material (newsletters, tip cards, posters)
Audit questionnaire
Retrospective information on weather conditions
Physical and social environment, context within which the ToyBox-intervention was implemented and assessment of contamination
- Information on kindergartens’ school food and physical activity environment (i.e. infrastructure/facilities/equipment, policies/rules/legislations, curriculum and personnel/health-related programmes in progress, which were related to the four lifestyle behaviours) - Contamination between intervention and control kindergartens/teachers - Weather conditions during the ToyBox-intervention, teachers’ trainings and baseline/follow-up evaluation in each country
Researchers’ evaluation form
Dose delivered (completeness)
- Number of training sessions - Duration of each training
Fidelity
- Venue of training - Researchers who performed each training - Records of deviations from the teachers’ training protocol
Dose received (exposure and satisfaction)
- Numbers of teachers invited and attended - Subjective assessment of teachers’ satisfaction with each training by the researchers who implemented the training
Dose received (satisfaction)
- Teachers’ satisfaction with each training, with the researchers who implemented the training and the ToyBox programme overall - Teachers’ suggestions for improving ToyBox programme and material after the completion of the ToyBox-intervention
Participation rate Attrition rate Teachers’ monthly logbooks
Information on contamination between intervention and control kindergartens/teachers
Teacher’s evaluation form
SES, socioeconomic status.
15 (Suppl. 3), 74–80, August 2014
© 2014 World Obesity
obesity reviews
ToyBox-intervention. All steps related to the implementation of the process evaluation in ToyBox were standardized across the six intervention countries. The ToyBox process evaluation methods and tools that were developed, focused on the collection of a broad variety of data and information related to the recruitment procedures, the actual implementation of the ToyBoxintervention and the context within which this intervention was implemented. Previous studies indicated that the collection of a variety of process evaluation data (i.e. data on dose delivered, dose received, satisfaction, fidelity, contextual factors) can contribute in evaluating the effectiveness of health promotion programmes and provide valuable insights to what made them successful or not (5). For example, Bowes et al. recorded the dose delivered and the contextual factors in a school-based intervention to increase physical activity and reduce bullying and showed that the originally planned intervention was actually implemented only in 39% of the schools because of the existence of barriers such as the lack of appropriate equipment and resources and changes in school teaching personnel (13). Similarly, Bere et al. showed that students who were more satisfied with the intervention material were those who increased their fruit and vegetable consumption the most (14), whereas Thaker et al. showed that higher teachers’ satisfaction with a school-based programme was linked to better sustainability of this programme by the teachers (15). In another study, Young et al. suggested that higher satisfaction with a programme is related to higher fidelity of its implementation and thus exposure of the participants to the intervention (16). Moreover, the DOiT project showed that changing the school food environment was difficult to induce and sustain mainly due to contextual factors such as the inadequate knowledge of the obesity problem by school management and the profit from the school canteens (17). All of the aforementioned process evaluation elements, i.e. dose delivered/completeness, fidelity, exposure, satisfaction, context, have been assessed in the ToyBoxintervention via several process evaluation methods and tools, which were directed to all study participants. More specifically, we aimed to collect multi-perspective opinions from the participating headmasters, the teachers, the parents/caregivers, the children and the researchers regarding the ToyBox-intervention and material. Collecting such information from the study stakeholders has been previously suggested to increase objectivity of process evaluation data collection and interpretation (9). Moreover, in the ToyBox process evaluation tools we aimed to use both quantitative (i.e. Likert-type scale questions) and qualitative methods (i.e. open-type questions, space with prompts for comments, recording of comments received verbally). This approach has been previously suggested to provide © 2014 World Obesity
ToyBox-intervention: process evaluation O. Androutsos et al.
79
more accurate and detailed information for the programmes’ process evaluation (18). The ToyBox process evaluation has also certain limitations. Firstly, data derived from direct observation during the implementation of the classroom activities could have provided triangulation and a more objective documentation of fidelity. However, direct observation would have burdened the researchers and teachers, both time and cost wise, without also ensuring objectivity. Moreover, apart from the assessment of contextual factors via the audit questionnaire, no monthly process evaluation data were collected from the control kindergartens. This would have enabled us to have a more thorough insight of the curriculum followed in the control kindergartens and collect more information on the potential contamination between intervention and control kindergartens/teachers. However, this procedure would also cause burden to the researchers and teachers and might itself contribute to the contamination between the intervention and the control kindergartens/ teachers. Future health promotion programmes should use well-planned and structured schedules for process evaluation, allowing for flexibility to adaptation according to local needs and stakeholders’ opinion and documenting barriers to participation of the intended priority audience. The process evaluation conducted in the ToyBoxintervention aimed to monitor and document the level of implementation of this intervention. Moreover, it aimed to assess the factors that may have influenced its effectiveness and to record stakeholders’ suggestions for future improvements. Overall, the ToyBox process evaluation may also provide valuable insights for future school-based interventions and public health policy makers.
Conflict of interest statement No conflict of interest was declared.
Acknowledgements The ToyBox-study is funded by the Seventh Framework Programme (CORDIS FP7) of the European Commission under grant agreement number 245200. The content of this article reflects only the authors’ views, and the European Community is not liable for any use that may be made of the information contained therein.
Supporting information Additional Supporting Information may be found in the online version of this article, http://dx.doi.org/10.1111/ obr.12185
References 1. De Bourdeaudhuij I, Van Cauwenberghe E, Spittaels H et al. School-based interventions promoting both physical activity and
15 (Suppl. 3), 74–80, August 2014
80
ToyBox-intervention: process evaluation O. Androutsos et al.
healthy eating in Europe: a systematic review within the HOPE project. Obes Rev 2011; 12: 205–216. 2. Flynn MA, McNeil DA, Maloff B et al. Reducing obesity and related chronic disease risk in children and youth: a synthesis of evidence with ‘best practice’ recommendations. Obes Rev 2006; 7(Suppl. 1): 7–66. 3. Summerbell CD, Waters E, Edmunds LD et al. Interventions for preventing obesity in children. Cochrane Database Syst Rev 2005; (3): CD001871. 4. Scheirer MA, Shediac MC, Cassady CE. Measuring the implementation of health promotion programs: the case of the breast and cervical cancer program in Maryland. Health Educ Res 1995; 10: 11–25. 5. Baranowski T, Stables G. Process evaluations of the 5-a-day projects. Health Educ Behav 2000; 27: 157–166. 6. Oakley A, Strange V, Bonell C et al. Process evaluation in randomised controlled trials of complex interventions. BMJ 2006; 332: 413–416. 7. Manios Y, Androutsos O, Katsarou C et al. Designing and implementing a kindergarten-based, family-involved intervention to prevent obesity in early childhood. The ToyBox-study. Obes Rev 2014; 15(Suppl. 3): 5–13. 8. Androutsos O, Katsarou C, Payr A et al. Designing and implementing teachers’ training sessions in a kindergarten-based, family-involved intervention to prevent obesity in early childhood. The ToyΒox-study. Obes Rev 2014; 15(Suppl. 3): 48–52. 9. Saunders RP, Evans MH, Joshi P. Developing a processevaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 2005; 6: 134–147. 10. Sy A, Glanz K. Factors influencing teachers’ implementation of an innovative tobacco prevention curriculum for
15 (Suppl. 3), 74–80, August 2014
obesity reviews
multiethnic youth: project SPLASH. J Sch Health 2008; 78: 264– 273. 11. Feathers JT, Kieffer EC, Palmisano G et al. The development, implementation, and process evaluation of the REACH Detroit partnership’s diabetes lifestyle intervention. Diabetes Educ 2007; 33: 509–520. 12. Schneider M, Hall WJ, Hernandez AE et al. Rationale, design and methods for process evaluation in the HEALTHY study. Int J Obes 2009; 33(Suppl. 4): S60–S67. 13. Bowes D, Marquis M, Young W et al. Process evaluation of a school-based intervention to increase physical activity and reduce bullying. Health Promot Pract 2009; 10: 394–401. 14. Bere E, Veierod MB, Bjelland M et al. Outcome and process evaluation of a Norwegian school-randomized fruit and vegetable intervention: fruits and vegetables make the marks (FVMM). Health Educ Res 2006; 21: 258–267. 15. Thaker S, Steckler A, Sanchez V et al. Program characteristics and organizational factors affecting the implementation of a school-based indicated prevention program. Health Educ Res 2008; 23: 238–248. 16. Young DR, Steckler A, Cohen S et al. Process evaluation results from a school- and community-linked intervention: the trial of activity for adolescent girls (TAAG). Health Educ Res 2008; 23: 976–986. 17. Singh AS, Chinapaw MJ, Brug J et al. Process evaluation of a school-based weight gain prevention program: the Dutch obesity intervention in teenagers (DOiT). Health Educ Res 2009; 24: 772–777. 18. Bouffard JA, Taxman FS, Silverman R. Improving process evaluations of correctional programs by using a comprehensive evaluation methodology. Eval Program Plann 2003; 26: 149– 161.
© 2014 World Obesity
doi: 10.1111/obr.12185
Obesity Prevention
Process evaluation design and tools used in a kindergarten-based, family-involved intervention to prevent obesity in early childhood. The ToyBox-study O. Androutsos1, E. Apostolidou1, V. Iotova2, P. Socha3, J. Birnbaum4, L. Moreno5,6, I. De Bourdeaudhuij7, B. Koletzko4 and Y. Manios1 on behalf of the ToyBox-study group 1
Department of Nutrition and Dietetics,
Harokopio University, Athens, Greece; 2
Department of Pediatrics, Medical University
Varna, Varna, Bulgaria; 3The Children’s Memorial Health Institute, Warsaw, Poland; 4
Dr. von Hauner Children’s Hospital, University
of Munich Medical Centre, Munich, Germany; 5
GENUD (Growth, Exercise, Nutrition and
Development) Research Group, University of Zaragoza, Zaragoza, Spain; 6School of Health Science (EUCS), University of Zaragoza, Zaragoza, Spain; 7Department of Movement and Sport Sciences, Ghent University, Ghent, Belgium
Received 4 April 2014; revised 8 April 2014; accepted 9 April 2014
Address for correspondence: Yannis Manios, Department of Nutrition and Dietetics, Harokopio University, El. Venizelou 70,
Summary Process evaluation (PE) is used for the in-depth evaluation of the implementation process of health promotion programmes. The aim of the current paper was to present the PE design and tools used in the ToyBox-intervention. The PE design was based on a three-step approach, including the identification of ToyBoxspecific PE elements (step 1), the development of PE tools and harmonization of procedures (step 2), and the implementation of PE using standardized protocol and tools across the intervention countries (step 3). Specifically, to evaluate the implementation of the intervention, teachers’ monthly logbooks were recorded (dose delivered, fidelity, dose received); post-intervention questionnaires were completed by parents/caregivers and teachers (dose received); participation and attrition rates were recorded (recruitment, reach); and audit questionnaires and retrospective information on weather conditions were collected (physical and social environment within which the intervention was implemented). Regarding the teachers’ training sessions, the researchers who performed the trainings completed evaluation forms and documented teachers’ attendance after each training (dose delivered, fidelity, dose received) and teachers completed evaluation forms after each training (dose received). The PE performed in the ToyBox-intervention may contribute in the evaluation of its effectiveness, guide the revision of the intervention material and provide insights for future health promotion programmes and public health policy.
Kallithea 17671, Athens, Greece. E-mail: [email protected]
Keywords: Kindergarten, obesity prevention, preschool children, process evaluation. obesity reviews (2014) 15 (Suppl. 3), 74–80
ToyBox-study group: Coordinator: Yannis Manios; Steering committee: Yannis Manios, Berthold Koletzko, Ilse De Bourdeaudhuij, Mai Chin A Paw, Luis Moreno, Carolyn Summerbell, Tim Lobstein, Lieven Annemans, Goof Buijs; External advisors: John Reilly, Boyd Swinburn, Dianne Ward; Harokopio University (Greece): Yannis Manios, Odysseas Androutsos, Eva Grammatikaki, Christina Katsarou, Eftychia Apostolidou, Eirini Efstathopoulou; Ludwig Maximilians Universitaet Muenchen (Germany): Berthold Koletzko, Kristin Duvinage, Sabine Ibrügger, Angelika Strauß, Birgit Herbert, Julia Birnbaum, Annette Payr; Ghent University (Belgium): Department of Movement and Sports Sciences: Ilse De Bourdeaudhuij, Greet Cardon, Marieke De Craemer, Ellen De Decker; Department of Public Health: Lieven Annemans, Stefaan De Henauw, Lea Maes, Carine Vereecken, Jo Van Assche, Lore Pil; VU University Medical Center EMGO Institute for Health and Care Research (the Netherlands): EMGO Institute for Health and Care Research: Mai Chin A Paw, Saskia te Velde; University of Zaragoza (Spain): Luis Moreno, Theodora Mouratidou, Juan Fernandez, Maribel Mesana, Pilar De Miguel-Etayo, Esther M. González-Gil, Luis Gracia-Marco, Beatriz Oves; Oslo and Akershus University College of Applied Sciences (Norway): Agneta Yngve, Susanna Kugelberg, Christel Lynch, Annhild Mosdøl, Bente B Nilsen; University of Durham (UK): Carolyn Summerbell, Helen Moore, Wayne Douthwaite, Catherine Nixon; State Institute of Early Childhood Research (Germany): Susanne Kreichauf, Andreas Wildgruber; Children’s Memorial Health Institute (Poland): Piotr Socha, Zbigniew Kulaga, Kamila Zych, Magdalena Góz´dz´, Beata Gurzkowska, Katarzyna Szott; Medical University of Varna (Bulgaria): Violeta Iotova, Mina Lateva, Natalya Usheva, Sonya Galcheva, Vanya Marinova, Zhaneta Radkova, Nevyana Feschieva; International Association for the Study of Obesity (UK): Tim Lobstein, Andrea Aikenhead; CBO B.V. (the Netherlands): Goof Buijs, Annemiek Dorgelo, Aviva Nethe, Jan Jansen; AOK-Verlag (Germany): Otto Gmeiner, Jutta Retterath, Julia Wildeis, Axel Günthersberger; Roehampton University (UK): Leigh Gibson; University of Luxembourg (Luxembourg): Claus Voegele.
74 15 (Suppl. 3), 74–80, August 2014
© 2014 World Obesity
obesity reviews
ToyBox-intervention: process evaluation O. Androutsos et al.
Introduction The variability observed in the effectiveness of previous school-based interventions may be attributed to the level of their implementation (1,2). One of the key characteristics of the interventions that were previously found to be effective is the use of process evaluation in order to monitor and document the level of their implementation (3). More specifically, process evaluation can be useful to examine the relationship between the design and implementation of the intervention with its impact and outcome, as well as to assess what made it successful or not (4,5). Over and above these, the use of process evaluation is particularly important in multicentre studies where large local, cultural and sociodemographic diversities are observed and may influence their implementation compared with what was initially planned (6). The aim of the current paper was to present the process evaluation design and tools used in the ToyBoxintervention in order to assess how this intervention was actually implemented, which factors influenced its implementation and effectiveness, whether it reached the initially targeted audience, and what did the stakeholders suggest regarding the future improvement of the ToyBoxintervention and material.
75
conduct permanent environmental changes in the classroom/kindergarten (i.e. install water stations and the ‘magic snack plate’ and rearrange the classroom/ kindergarten to create some free space to assist children’s movement); (ii) promote children’s actual lifestyle behaviour on a regular basis (i.e. remind children to drink water regularly and eat healthy snacks daily, as well as to do short movement breaks for interrupting prolonged sitting time twice in the morning and twice afternoon and to perform physical education sessions two times per week with the children); and (iii) implement interactive classroom activities aiming at total class participation, minimum of 1 h per week. Teachers were also instructed to use the kangaroo hand puppet and perform these four lifestyle behaviours themselves, so as to enhance the effects of the intervention via role modelling. Similar approach was followed at home, where parents/caregivers were encouraged and advised, via simple and friendly to read newsletters, tip cards and posters, to apply relevant environmental changes at home, act as role models and implement these lifestyle behaviours together with their children. Three teachers’ training sessions were also conducted to train the teachers on how to implement the ToyBoxintervention. All trainings were performed locally by trained researchers using standardized protocol, presentations and timeplan (8). Only the teachers from the intervention kindergartens were invited to attend these sessions.
Methods Process evaluation in the ToyBox-intervention The ToyBox-intervention The ToyBox-intervention (http://www.toybox-study.eu) was implemented during the academic year 2012–2013 in six European countries (Belgium, Bulgaria, Germany, Greece, Poland, Spain) (7). The study sample at baseline comprised of 309 kindergartens and 7,056 children aged 3.5–5.5 years, their parents/caregivers and their teachers, recruited from three socioeconomic groups. The ToyBoxintervention aimed to promote preschool children’s water consumption, healthy snacking, physical activity and limit/ interrupt their sedentary behaviour by improving children’s physical and social environment both at the kindergarten and at home. More specifically at kindergartens, teachers were provided with five handbooks and were asked to (i)
Step 1 Identification of PE elements to be assessed in the ToyBox-intervention
A three-step approach was used for the design of the process evaluation in the ToyBox-intervention (Fig. 1). More specifically, the next steps were followed: Step 1: identification of process evaluation elements to be assessed in the ToyBox-intervention The first step took place from September to December 2011. Based on the steps and principles described in the conceptual framework by Saunders et al. (9), we identified and assessed the next process evaluation elements as following: Recruitment/reach: the recruitment procedure was monitored against the standardized protocol of the study. Furthermore, the proportions and the characteristics of
Step 2 Development of PE tools and harmonization of procedures
Step 3 Implementation of PE in the six intervention countries
Figure 1 Design of the ToyBox programme process evaluation (PE).
© 2014 World Obesity
15 (Suppl. 3), 74–80, August 2014
76 ToyBox-intervention: process evaluation O. Androutsos et al.
kindergartens/teachers/families agreeing to participate in the intervention and completing or dropping out from the intervention were recorded. Dose delivered (completeness): the intensity of actual implementation of the ToyBox-intervention was assessed. More specifically, the number of training sessions and the duration of each training were recorded. The degree of implementation/retention of the environmental changes and children’s actual behaviour by the kindergarten teachers, as well as the number, type and duration of the classroom activities were also recorded. Moreover, the delivery of the intervention material to the parents/caregivers (nine newsletters, eight tip cards, four posters) and the duration of the time devoted to the ToyBox-intervention by the teachers were assessed. Fidelity: the extent to which the ToyBox-intervention was implemented as initially planned was assessed. More specifically, the degree of adherence to the teachers’ training protocol (8) was recorded. The actual implementation of the environmental changes, children’s actual behaviour and classroom activities and the delivery of the intervention material to the parents/caregivers (newsletters, tip cards, posters) according to the ToyBox-intervention timeplan and material were also recorded. Moreover, the duration of the time devoted to the ToyBox-intervention by the teachers was assessed. Dose received (exposure and satisfaction): the extent to which children/parents/caregivers/teachers were exposed to the ToyBox-intervention, as well as the degree of their satisfaction with the ToyBox-intervention and material were assessed. Regarding the training sessions, the numbers of teachers who were invited and actually participated in these trainings were recorded. Moreover, teachers’ satisfaction with the trainings, with the researchers who implemented the trainings and with the ToyBox-intervention overall, and researchers’ subjective perception of teachers’ satisfaction with the trainings were assessed after the completion of each training. Teachers’ perceived barriers and facilitators for implementing each part of the ToyBoxintervention were also recorded. Also, the extent to which parents/caregivers were exposed to the material (newsletters, tip cards, posters) and to the ToyBox-intervention was recorded. Stakeholders’ (i.e. headmasters’/ management personnel’s, teachers’, parents’/caregivers’, children’s) satisfaction with the ToyBox-intervention and material (comprehensibility, readability, relevance, credibility and attractiveness of the material) was assessed and their suggestions for future improvement of the ToyBoxintervention and material were recorded. Context: several physical (e.g. weather conditions, kindergarten infrastructure/ facilities/equipment/kindergartens’ curriculum/personnel), social (e.g. socioeconomic status of the family, teachers’ health-related behaviours) and contextual (e.g. policies, legislation and rules) factors that may 15 (Suppl. 3), 74–80, August 2014
obesity reviews
have acted as barriers or facilitators to the implementation of the ToyBox-intervention were recorded. Moreover, information on potential contamination of the intervention between the intervention and the control kindergartens/ teachers was collected. Step 2: development of process evaluation tools and harmonization of procedures The second step was implemented from January to March 2012. During the design of the process evaluation, there was thoughtful consideration of various methodological components as those suggested by Saunders et al. (9) that resulted in the development of a series of process evaluation tools, which were tailor made for the ToyBoxintervention. More specifically, considering the issues of data sources and collection procedures, we developed the process evaluation tools in such a way so that they could be applied within the ToyBox-intervention (i.e. in means of personnel, time and budget availability) and at the same time allow gathering as much information as possible. In parallel, all procedures aimed to burden the teachers as little as possible. Based on the aforementioned issues, the process evaluation tools were designed to include mainly quantitative methods for researchers, teachers and parents/caregivers, as these have the advantage of being less time consuming and burdensome to complete while they can provide more objective and quantified information/data. However, some of the process evaluation tools designed for the teachers included open-ended questions to allow for the collection of unanticipated answers and subjective reports (5). All intervention countries used common process evaluation protocol and tools. Initially, the tools were translated into the six local languages and back-translated to English. Translation and back-translation procedures were harmonized and centrally coordinated and evaluated. Common datasets and codebooks were also used to enter the collected process evaluation data across all intervention countries. Step 3: implementation of process evaluation in the six intervention countries The third step was conducted from April 2012 to June 2013 and included the implementation of the process evaluation, which ran in parallel to the baseline and follow-up measurements and the implementation of the ToyBox-intervention. The implementation of the process evaluation was guided from a common protocol for the six intervention countries. This protocol included an explicit description of the purpose and use of the process evaluation tools and a timeplan for the application of each process evaluation tool. © 2014 World Obesity
obesity reviews
Results Based on the three-step approach that was applied for the design of the ToyBox-intervention process evaluation, common process evaluation protocol and tools were used across the six intervention countries. An overview of the process evaluation tools is presented in Table 1. For the process evaluation of the ToyBox-intervention, 12 process evaluation methods and tools were used (Table 1). Regarding the evaluation of the teachers’ training sessions two process evaluation tools were used. More specifically, immediately after the completion of each training, each teacher completed the ‘teacher’s evaluation form’ and the researchers who implemented the training completed the ‘researchers’ evaluation form’ (Supporting Information). The information obtained via the ‘researchers’ evaluation forms’ was used to assess whether the three training sessions were conducted in the six intervention countries and record the actual duration of each training (dose delivered/completeness), assess if the trainings were actually implemented as originally planned in the teachers’ training protocol (fidelity), capture how much did teachers enjoy and participate in each training (dose received/ satisfaction), and record how many of the invited teachers actually participated in each training (dose received/ exposure). Regarding the fidelity in the implementation of the trainings, the venue and the names of researchers who implemented the trainings were recorded, whereas the researchers who implemented the training were also asked to report any deviations from the protocol and the reasons for these deviations in an open-type question. On the other hand, the ‘teachers’ evaluation form’ was used to assess teachers’ satisfaction with the training, with the researchers who implemented the training and the ToyBoxintervention overall, as well as to record teachers’ suggestions for improving the ToyBox-intervention and material in the future. Weekly records of recruitment monitoring and information on participation rate and attrition rate were collected throughout the recruitment and the implementation phase of the ToyBox-intervention. This information was used to assess whether the recruitment in the six intervention countries was conducted according to the protocol and to collect information on the characteristics of the participants (teachers and families) who participated or dropped out from the intervention. Furthermore, the teachers completed a structured questionnaire, the ‘teachers’ monthly logbook’ (Supporting Information), via face-to-face meetings or phone calls with the researchers or via emails. This logbook was completed at the end each month of the intervention phase, resulting in six logbooks per class at the end of the intervention. For example, if one class had two or more teachers, we aimed to collect six logbooks at minimum from the total number © 2014 World Obesity
ToyBox-intervention: process evaluation O. Androutsos et al.
77
of teachers in this class. With this tool, the intensity (dose delivered/completeness) and the fidelity of the actual implementation of the ToyBox-intervention, as well as children’s/parents’/teachers’ exposure to the ToyBoxintervention and satisfaction with the ToyBox-intervention and material were assessed. For the evaluation of children’s, parents’/caregivers’ and teachers’ exposure and satisfaction, information on the attendance of children in the kindergarten was collected and two more process evaluation tools were applied, i.e. the ‘teachers’ post-intervention questionnaire’ and the ‘parents’/caregivers’ post-intervention questionnaire’ (Supporting Information), after the end of the ToyBox-intervention. The physical and social environment within which the ToyBox-intervention was implemented was assessed via a semi-structured interview with the headmaster and the teachers in each intervention and control kindergarten, i.e. the ‘audit questionnaire’ (Supporting Information) before and after the intervention, and information collected retrospectively on the weather conditions. Moreover, information on potential contamination between the intervention and the control kindergartens/teachers was collected.
Discussion A systematic approach was used for the design and implementation of the ToyBox process evaluation. Standardized protocol, methods and tools were used across the six intervention countries and all participants of this study (i.e. teachers, parents/caregivers, children and researchers) contributed with their opinion to the process evaluation. The completeness and fidelity of the ToyBox-intervention, the exposure of teachers/parents/caregivers/children to the ToyBox-intervention and teachers’/parents’/caregivers’/ children’s satisfaction (intelligibility, comprehensibility, readability, relevance, credibility and attractiveness) with the ToyBox-intervention and the material were assessed. The recruitment procedure, the reach and the contextual factors within which the ToyBox-intervention was implemented were also documented and wherever feasible quantified, while stakeholders’ suggestions for the future improvement of the ToyBox-intervention and material were recorded. Even though process evaluation is important for the evaluation of the effectiveness of health promotion programmes (2), many studies do not use process evaluation in their design (1,3). Indeed, only a limited number of studies have previously presented the process evaluation design, methods and tools used (9–13). Based on key learnings from previous studies (6), a three-step approach was for the design of process evaluation and the development of tailor-made process evaluation methods and tools in the 15 (Suppl. 3), 74–80, August 2014
78 ToyBox-intervention: process evaluation O. Androutsos et al.
obesity reviews
Table 1 The process evaluation tools used in the ToyBox-intervention Process evaluation tools
Process evaluation elements
Information/data collected
Monitoring of recruitment
Recruitment procedures/reach
- Weekly monitoring of recruited kindergartens/families per country and SES level - Numbers of kindergartens/families: approached, participated in the baseline evaluation/entered the ToyBox-intervention and participated in the follow-up evaluation - Characteristics of kindergartens/families dropping out
Dose delivered (completeness)
-
Implementation/retention of environmental changes in the classroom/kindergarten Implementation/retention of children’s lifestyle behaviours in the classroom/kindergarten Duration, number and type of classroom activities implemented Delivery of material (newsletters, tip cards, posters) to parents Time devoted to the ToyBox programme in each classroom
Fidelity
-
Implementation of the sequence of four behaviours according to the timeplan Implementation of classroom environmental changes according to the material and timeplan Implementation of children’s actual behaviour according to the material and timeplan Implementation of the classroom activities according to the material and timeplan Delivery of material (newsletters, tip cards, posters) to parents according to the timeplan and assessment on how this material was handed to the parents
Dose received (exposure and satisfaction)
- Teachers’ feedback on the comprehensibility, readability, relevance, credibility, attractiveness of the handbooks - Teachers’ feedback on children’s satisfaction with the ToyBox programme as implemented in the classroom each month - Teachers’ suggestions for improvement of the ToyBox programme and material after the completion of the intervention - Teachers’ feedback on the barriers and facilitators for implementing each part of the intervention
Attendance of children in the kindergarten
Dose received (exposure)
- Children’s attendance in the kindergarten
Teachers’ post-intervention questionnaire
Dose received (exposure and satisfaction)
- Teachers’ feedback on the barriers and facilitators for implementing the ToyBox programme - Teachers’ feedback on the comprehensibility, readability, relevance, credibility, attractiveness of the handbooks - Subjective teachers’ feedback on children’s, parents’, teachers’ and headmasters’/management personnel’s satisfaction with the ToyBox programme
Parent’s postintervention questionnaire
Dose received (exposure and satisfaction)
- Parents’ feedback on if the received and/or read each part of the material (newsletters, tip cards, posters) - Information on the ToyBox programme beyond the material (newsletters, tip cards, posters) given by the teacher to the parents - Parents’ satisfaction with the ToyBox programme - Parents feedback if they implemented as a family the activities included in the material (newsletters, tip cards) with their children at home - Subjective parents’ feedback on their spouse’s/partner’s and child’s satisfaction with the ToyBox programme - Parents’ feedback on the comprehensibility, readability, relevance, credibility, attractiveness of the material (newsletters, tip cards, posters)
Audit questionnaire
Retrospective information on weather conditions
Physical and social environment, context within which the ToyBox-intervention was implemented and assessment of contamination
- Information on kindergartens’ school food and physical activity environment (i.e. infrastructure/facilities/equipment, policies/rules/legislations, curriculum and personnel/health-related programmes in progress, which were related to the four lifestyle behaviours) - Contamination between intervention and control kindergartens/teachers - Weather conditions during the ToyBox-intervention, teachers’ trainings and baseline/follow-up evaluation in each country
Researchers’ evaluation form
Dose delivered (completeness)
- Number of training sessions - Duration of each training
Fidelity
- Venue of training - Researchers who performed each training - Records of deviations from the teachers’ training protocol
Dose received (exposure and satisfaction)
- Numbers of teachers invited and attended - Subjective assessment of teachers’ satisfaction with each training by the researchers who implemented the training
Dose received (satisfaction)
- Teachers’ satisfaction with each training, with the researchers who implemented the training and the ToyBox programme overall - Teachers’ suggestions for improving ToyBox programme and material after the completion of the ToyBox-intervention
Participation rate Attrition rate Teachers’ monthly logbooks
Information on contamination between intervention and control kindergartens/teachers
Teacher’s evaluation form
SES, socioeconomic status.
15 (Suppl. 3), 74–80, August 2014
© 2014 World Obesity
obesity reviews
ToyBox-intervention. All steps related to the implementation of the process evaluation in ToyBox were standardized across the six intervention countries. The ToyBox process evaluation methods and tools that were developed, focused on the collection of a broad variety of data and information related to the recruitment procedures, the actual implementation of the ToyBoxintervention and the context within which this intervention was implemented. Previous studies indicated that the collection of a variety of process evaluation data (i.e. data on dose delivered, dose received, satisfaction, fidelity, contextual factors) can contribute in evaluating the effectiveness of health promotion programmes and provide valuable insights to what made them successful or not (5). For example, Bowes et al. recorded the dose delivered and the contextual factors in a school-based intervention to increase physical activity and reduce bullying and showed that the originally planned intervention was actually implemented only in 39% of the schools because of the existence of barriers such as the lack of appropriate equipment and resources and changes in school teaching personnel (13). Similarly, Bere et al. showed that students who were more satisfied with the intervention material were those who increased their fruit and vegetable consumption the most (14), whereas Thaker et al. showed that higher teachers’ satisfaction with a school-based programme was linked to better sustainability of this programme by the teachers (15). In another study, Young et al. suggested that higher satisfaction with a programme is related to higher fidelity of its implementation and thus exposure of the participants to the intervention (16). Moreover, the DOiT project showed that changing the school food environment was difficult to induce and sustain mainly due to contextual factors such as the inadequate knowledge of the obesity problem by school management and the profit from the school canteens (17). All of the aforementioned process evaluation elements, i.e. dose delivered/completeness, fidelity, exposure, satisfaction, context, have been assessed in the ToyBoxintervention via several process evaluation methods and tools, which were directed to all study participants. More specifically, we aimed to collect multi-perspective opinions from the participating headmasters, the teachers, the parents/caregivers, the children and the researchers regarding the ToyBox-intervention and material. Collecting such information from the study stakeholders has been previously suggested to increase objectivity of process evaluation data collection and interpretation (9). Moreover, in the ToyBox process evaluation tools we aimed to use both quantitative (i.e. Likert-type scale questions) and qualitative methods (i.e. open-type questions, space with prompts for comments, recording of comments received verbally). This approach has been previously suggested to provide © 2014 World Obesity
ToyBox-intervention: process evaluation O. Androutsos et al.
79
more accurate and detailed information for the programmes’ process evaluation (18). The ToyBox process evaluation has also certain limitations. Firstly, data derived from direct observation during the implementation of the classroom activities could have provided triangulation and a more objective documentation of fidelity. However, direct observation would have burdened the researchers and teachers, both time and cost wise, without also ensuring objectivity. Moreover, apart from the assessment of contextual factors via the audit questionnaire, no monthly process evaluation data were collected from the control kindergartens. This would have enabled us to have a more thorough insight of the curriculum followed in the control kindergartens and collect more information on the potential contamination between intervention and control kindergartens/teachers. However, this procedure would also cause burden to the researchers and teachers and might itself contribute to the contamination between the intervention and the control kindergartens/ teachers. Future health promotion programmes should use well-planned and structured schedules for process evaluation, allowing for flexibility to adaptation according to local needs and stakeholders’ opinion and documenting barriers to participation of the intended priority audience. The process evaluation conducted in the ToyBoxintervention aimed to monitor and document the level of implementation of this intervention. Moreover, it aimed to assess the factors that may have influenced its effectiveness and to record stakeholders’ suggestions for future improvements. Overall, the ToyBox process evaluation may also provide valuable insights for future school-based interventions and public health policy makers.
Conflict of interest statement No conflict of interest was declared.
Acknowledgements The ToyBox-study is funded by the Seventh Framework Programme (CORDIS FP7) of the European Commission under grant agreement number 245200. The content of this article reflects only the authors’ views, and the European Community is not liable for any use that may be made of the information contained therein.
Supporting information Additional Supporting Information may be found in the online version of this article, http://dx.doi.org/10.1111/ obr.12185
References 1. De Bourdeaudhuij I, Van Cauwenberghe E, Spittaels H et al. School-based interventions promoting both physical activity and
15 (Suppl. 3), 74–80, August 2014
80
ToyBox-intervention: process evaluation O. Androutsos et al.
healthy eating in Europe: a systematic review within the HOPE project. Obes Rev 2011; 12: 205–216. 2. Flynn MA, McNeil DA, Maloff B et al. Reducing obesity and related chronic disease risk in children and youth: a synthesis of evidence with ‘best practice’ recommendations. Obes Rev 2006; 7(Suppl. 1): 7–66. 3. Summerbell CD, Waters E, Edmunds LD et al. Interventions for preventing obesity in children. Cochrane Database Syst Rev 2005; (3): CD001871. 4. Scheirer MA, Shediac MC, Cassady CE. Measuring the implementation of health promotion programs: the case of the breast and cervical cancer program in Maryland. Health Educ Res 1995; 10: 11–25. 5. Baranowski T, Stables G. Process evaluations of the 5-a-day projects. Health Educ Behav 2000; 27: 157–166. 6. Oakley A, Strange V, Bonell C et al. Process evaluation in randomised controlled trials of complex interventions. BMJ 2006; 332: 413–416. 7. Manios Y, Androutsos O, Katsarou C et al. Designing and implementing a kindergarten-based, family-involved intervention to prevent obesity in early childhood. The ToyBox-study. Obes Rev 2014; 15(Suppl. 3): 5–13. 8. Androutsos O, Katsarou C, Payr A et al. Designing and implementing teachers’ training sessions in a kindergarten-based, family-involved intervention to prevent obesity in early childhood. The ToyΒox-study. Obes Rev 2014; 15(Suppl. 3): 48–52. 9. Saunders RP, Evans MH, Joshi P. Developing a processevaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 2005; 6: 134–147. 10. Sy A, Glanz K. Factors influencing teachers’ implementation of an innovative tobacco prevention curriculum for
15 (Suppl. 3), 74–80, August 2014
obesity reviews
multiethnic youth: project SPLASH. J Sch Health 2008; 78: 264– 273. 11. Feathers JT, Kieffer EC, Palmisano G et al. The development, implementation, and process evaluation of the REACH Detroit partnership’s diabetes lifestyle intervention. Diabetes Educ 2007; 33: 509–520. 12. Schneider M, Hall WJ, Hernandez AE et al. Rationale, design and methods for process evaluation in the HEALTHY study. Int J Obes 2009; 33(Suppl. 4): S60–S67. 13. Bowes D, Marquis M, Young W et al. Process evaluation of a school-based intervention to increase physical activity and reduce bullying. Health Promot Pract 2009; 10: 394–401. 14. Bere E, Veierod MB, Bjelland M et al. Outcome and process evaluation of a Norwegian school-randomized fruit and vegetable intervention: fruits and vegetables make the marks (FVMM). Health Educ Res 2006; 21: 258–267. 15. Thaker S, Steckler A, Sanchez V et al. Program characteristics and organizational factors affecting the implementation of a school-based indicated prevention program. Health Educ Res 2008; 23: 238–248. 16. Young DR, Steckler A, Cohen S et al. Process evaluation results from a school- and community-linked intervention: the trial of activity for adolescent girls (TAAG). Health Educ Res 2008; 23: 976–986. 17. Singh AS, Chinapaw MJ, Brug J et al. Process evaluation of a school-based weight gain prevention program: the Dutch obesity intervention in teenagers (DOiT). Health Educ Res 2009; 24: 772–777. 18. Bouffard JA, Taxman FS, Silverman R. Improving process evaluations of correctional programs by using a comprehensive evaluation methodology. Eval Program Plann 2003; 26: 149– 161.
© 2014 World Obesity
