Am J Community Psychol (2015) 56:120–133 DOI 10.1007/s10464-015-9737-8

ORIGINAL ARTICLE

Planning for Sustainability of an Evidence-Based Mental Health Promotion Program in Canadian Elementary Schools Bonnie J. Leadbeater1

•

Emilie J. Gladstone2 • Paweena Sukhawathanakul1

Published online: 7 July 2015  Society for Community Research and Action 2015

Abstract Substantial research illuminates many factors effecting the implementation of evidence-based mental health promotion programs in schools; however, research on how schools plan for sustaining their investments in these programs is limited. In this qualitative study, we elicited descriptions of opportunities and challenges for sustainability. We interviewed 24 individuals from schools involved in a longitudinal, qualitative research project that followed uptake and implementation of the evidence-based WITS Programs across 2 years (Leadbeater et al. 2012). WITS stands for Walk away, Ignore, Talk it out and Seek help and the online WITS Programs focus on preventing peer victimization (www.witsprograms.ca). Our findings suggest that sustainability planning in schools is not merely a next step following high quality implementation, but rather involves multiple ongoing processes that need to be anticipated and supported by school leadership and program champions and developers in order to realize investments in evidence-based programs. Keywords Mental health promotion
School-based prevention programs
Implementation
Sustainability

& Bonnie J. Leadbeater [email protected] Paweena Sukhawathanakul [email protected] 1

Department of Psychology, University of Victoria, Cornett Building A236, 3800 Finnerty Road (Ring Road), Victoria, BC V8P 5C2, Canada

2

UBC Centre for Health Services and Policy Research, 2012206 East Mall, Vancouver, BC V6T 1Z3, Canada

123

Introduction Considerable research demonstrates that evidence-based mental health promotion programs that are well implemented in schools can have positive impacts on children’s mental health and academic performance, both by increasing competencies and by reducing problems (Durlak and Wells 1997; Durlak and Dupre 2008). Schools are often considered ideal sites for the delivery of prevention programs to large numbers of children across multiple years (Lee and Gortmaker 2012; Leadbeater and Gladstone 2015; Payne and Eckert 2010). However, research on how schools plan for sustaining programs after initial adoption and implementation is limited (Han and Weiss 2005). Considerable research on uptake and implementation of programs in schools exists (see reviews Meyers et al. 2012; Ogden and Fixsen 2014; Ozer et al. 2010). Studies of the components of successfully implemented mental health promotion show the need for programs that are: (a) integrated into children’s daily activities, (b) address both risk and protective factors, (c) engage and address key ecologies that surround children and youth, and importantly, (d) are sustained across multiple years (Domitrovich and Greenberg 2000; Greenberg et al. 2001; Jones and Bouffard 2012). Reviews suggest that characteristics of sustainability may be specific to intervention types and organizational settings (Scheirer 2012) and a better understanding is needed of how schools, in particular, plan for program sustainability. In this paper, we present findings from a longitudinal, qualitative study that tracked program uptake and implementation fidelity across two and a half years (Leadbeater et al. 2012). The current study focuses on staff decision in planning for continued use of the program after this initial period of uptake.

Am J Community Psychol (2015) 56:120–133

Sustainability is frequently conceptualized as an end goal that is subsequent to program evaluation, dissemination and implementation (Kellam and Langevin 2003); however, recent research suggests that it may be better understood as involving more cyclical processes that involve planning for both implementation and renewal (i.e., the maintenance of fidelity and regulation of decay or adaptations; Han and Weiss 2005). Following Scheirer and Dearing (2011, p. 2060), we define sustainability as ‘‘the continued use of program components and activities for the continued achievement of desirable program and population outcomes’’. Our interviews with program users illuminate factors that influence their planning for the continued use of the evidence-based WITS Programs 2 years after their adoption in eight rural Canadians elementary schools. At this point, the schools’ commitment for involvement with the WITS research team was also ending so in each school, the staff were in the process of deciding how or whether to continue the programs independently. While this qualitative study focuses on only one program, we aimed to illustrate, more generally, the issues that schools consider in planning to sustain a program that they have already invested considerable effort to implement. At the point of the interviews, five of the schools had fully implemented WITS and were using all core components; however, in three schools implementation fidelity was already reduced. This already apparent diversity in the schools’ ability to sustain WITS exposed both opportunities for and challenges to sustainability planning not only in schools that had achieved and maintained strong implementation but also in the context of wavering commitments. Background Research on Prevention Program Sustainability The life cycle of implementation of evidence-based interventions cycles and often recycles through adoption, training, implementation, integration, and maintenance (Scheirer 2012). Multiple factors have been identified that affect initial both program adoption and implementation; such as, organizational readiness (Spoth et al. 2015), leadership and openness to change (Chilenski et al. 2015), advocacy of program champions and compatibility with staff values and ongoing initiatives (Rogers 2002). While research on the sustainably phase of this life cycle has lagged behind, research focusing on sustainability of public health interventions—typically operating within community organizations to address an array of large scale health problems including smoking, substance abuse, obesity, or the prevention of chronic diseases—is emerging. Like implementation, sustainability in community health organizations appears to be influenced by characteristics of

121

both programs and settings (Schell et al. 2013), support for the innovation from community organizations and partnerships (King et al. 2010), advanced funding planning (Cooper et al. 2013; Tibbits et al. 2010), university-community alliances that persist in supporting implementation (Spoth et al. 2002), and favorable policy climates (Rhoades et al. 2012). Theoretical frameworks that outline the components of programs, organizations, and communities that enable sustainability are also proposed based on sustainability research on public health innovations conducted over the last decade (Scheirer and Dearing 2011; Schell et al. 2013). For example, Schell et al.’s (2013) framework was based on a review of the public health literature and input from an expert panel. The authors identified nine core domains that affect ongoing sustainability processes, i.e., (1) stability of intervention funding; (2) political support from internal and external environments that influence the intervention funding and acceptance; (3) partnerships connecting program and community; (4) organizational capacity to manage ongoing intervention activities; (5) adaptations to enhance and sustain buy-in and local effectiveness; (6) continued evaluation of intervention activities and outcomes; (7) strategic communications among stakeholders, decision makers and the public; (8) perceived positive impacts; and (9) strategic planning to define ongoing and future goals and direction. As for sustainability, itself, the framework anticipates dynamic changes in these domains over the life course of a program. Focusing more specifically on school-based programs, Han and Weiss (2005) posit a model of sustainability in which a self-sustaining feedback loop is established incrementally by teachers across a programs’ life cycle from adoption to implementation. Situating teachers’ success in favorable administrative and policy climates, the theoretical model suggests that teachers’ views of the fit of the program with teachers’ ongoing practices and beliefs in a pre-implementation phase; and the efficacy, skills, and attributions of positive program effects that they acquired during an implementation phase increase teachers’ intrinsic motivation, skills and ability to retain and modify the program in a sustainability phase—when the external support provided in the implementation phase is no longer available. Han and Weiss (2005, p. 676) argue that ‘‘teachers’ level of skills and motivation demonstrated during this sustainability phase is heavily dependent on the success experienced during the implementation phase…as indicated by their degree of program delivery, fidelity, and generalized use of program strategies’’. Other research also demonstrates the key role of teachers in implementation of mental health promotion programs (e.g., Bierman et al. 2013; Ringwalt et al. 2010); however, teachers are often burdened by curriculum demands. Lee and Gortmaker

123

122

(2012) suggest that program champions and multi-sector approaches may be needed to sustain teacher involvement and program impacts. Despite the lack of work on sustainability in schools, several challenges to sustainability in schools can also be anticipated from theory and research on the diffusion of preventive innovations (Rogers 2002), implementation science (Ogden and Fixsen 2014; Sloboda et al. 2014), and sustainability in community health organizations (Schell et al. 2013). For example, noting that preventive innovations diffuse slowly, Rogers (2002, p. 989) suggests strategies that could ‘‘speed up’’ the adoption of preventive innovations by enhancing their perceived relative advantage over other approaches and their perceived compatibility with existing values and needs of adopters, by supporting program champions, by enhancing norms supporting evidence based preventive innovations, marketing, and by encouraging peer-to-peer communications about the innovation. Whether such process may also function to sustain commitments to and fidelity of programs over time is not known. Challenges to sustainability that emerge during the implementation period are notable. Necessities for local integration and adaptations, drift in the use of core program components, turnover of trained staff, and loss of program champions and leaders are realities that can affect schools’ ability to sustain innovations (Leadbeater and Gladstone 2015). Moreover, community-university research alliances that work in collaboration to develop and implement evidence-based programs typically lack the expertise or funding to support large-scale dissemination efforts or to monitor programs’ long term use (Spoth et al. 2002). Evaluations of impact of long standing programs on the achievement of intended program ‘‘outcomes’’ also disappear as research-driven monitoring moves out and informal self-evaluations take its place. More knowledge is needed to add to our understanding of what helps schools to maintain their investments in adopting and implementing evidence based programs, to prevent program drift, and to evaluate ongoing program value and long term impacts. Despite the extensive theorizing about sustainability, this research has focused on the points of view of implementation experts and program developers, and the decision processes of program users are under studied. Specifically, little is known about how school-staff decide whether or not to sustain their investments in programs that they have already implemented. In this qualitative study, we add to the previous literature by presenting findings from interviews with individuals from schools who had implemented the evidence-based WITS Programs (www.wit sprograms.ca) which have been shown to reduce peer victimization in previous research (Leadbeater and Sukhawathanakul 2011). A recent study demonstrates that high

123

Am J Community Psychol (2015) 56:120–133

implementation of the WITS Programs is related to increases in children’s social responsibility, prosocial leadership, and positive school climates, and that these protective factors, in turn, predict reductions in peer victimization and aggression and emotional problems across a 2 year period (Leadbeater et al. 2015). Challenges to sustainability related to funding, which other research (Tibbits et al. 2010) shows can present overriding problems when research-related funding is withdrawn, were minimized in this study. The WITS Programs are online and open access programs and the small cost for the initial uptake and implementation of the programs and annual renewal of resources were supported financially and by the research project (Leadbeater et al. 2012). Access to training (and retraining) modules, support from program staff, and new resources are also provided. Hence, our interviews focus directly on the schools’ planning for sustainability, rather than on budget concerns. However, at the time of the interviews all schools were facing the withdrawal of research supported funding and monitoring. Interviews were also conducted in late spring, when school staffs typically plan for the next school year; hence their responses reflect real time decision-making processes about sustainability. To assess current fidelity of implementation of the program, we also assessed the use of eight core components of the program. Core Characteristics of the WITS Programs The evidence-based WITS programs aim to create responsive communities that reduce peer victimization and bullying among children in grades one to six. The programs work to increase protective factors that are incompatible with bullying including social responsibility, prosocial competence and leadership, and positive school climates (Leadbeater et al. 2015). Resources for school staff, parents, community leaders, and children are accessible online at www.witsprogams.ca (or, in French, at www.lepro grammeDIRE.ca). Core components are flexible in their implementation but include the following eight activities: (1) The Programs are launched each year in a ‘‘swearing in ceremony’’ lead by a notable ‘‘Community Leader’’—who helped to catalyze program uptake and sustainability (These individuals typically include local members of police forces or other first responder, Aboriginal Elders, and health professional). (2) Children and school staff members pledge to use their WITS to Walk away, Ignore, Talk it out and Seek help to deal with teasing and bullying and to help other children use their WITS. (3) The Programs create a common language to enable all members of the community to talk proactively about ‘‘using your WITS’’ to solve peer conflicts and increase social responsibility. Principals also can use the language to inquire about and respond to

Am J Community Psychol (2015) 56:120–133

disciplinary problems. (4) Each month, teachers choose a book from the website list (a ‘‘WITS PICK’’) and use the book-based lesson plans to facilitate conversations about WITS messages. These lesson plans are integrated with academic learning objectives to reduce time demands on teachers. (5) Teachers, administrators, and playground staff are encouraged to use ‘‘teachable moments’’ to help children use their WITS to solve current conflicts with peers. (6) WITS Community leaders re-visit the school throughout the school year to ask children about their successes in using their WITS and to deliver take home gifts (pencils, rulers, etc.) that help reinforce the WITS message. (7) Parents are also encouraged to use the WITS language at home and are reminded about WITS activities and online resources for parents through school newsletters and school-wide posters and contests. (8) Older children in grades 4–6 are taught leadership skills that enable them to be more effective bystanders using the LEADerS lesson plans to Look and Listen, Explore Points of View, Act, ask Did it Work and when needed to Seek help in order to feel competent and empowered to help younger children in the playground and neighborhood.

Methods Participants The 24 individuals interviewed for this study included 7 principals, 1 vice principal, 9 teachers, 2 counsellors, 2 librarians, 2 RCMP officers (serving as community leaders). All of the interviewees who were not administrators had been designated as liaisons with the research group. All had participated in a least one previous interview, and were well informed about the use of the programs in the school. At least two individuals were interviewed from each school. Participants came from eight rural elementary schools in British Columbia (BC), Canada who responded to a district wide invitation to participate in research following the implementation of the WITS Programs over two and a half years (see Leadbeater et al. 2012 for details). Schools signed onto the study in the spring of 2010 and initiated implementation of the WITS Programs in October of 2010. Program implementers were interviewed three times in 2011 (February n = 10, May n = 20, and November n = 20) and again in October of 2012 (n = 26). The interviews (n = 24) relating to sustainably occurred in April 2013. To create diversity in the schools involved in the study, all British Columbia (BC) school principals were invited to participate in the research through the BC Principals and Vice Principal’s Association newsletter or by invitation from the Rock Solid Foundation. The first 8 schools

123

responding to the invitation served diverse populations. They were located in rural communities with populations ranging from 2700 to 22,000, and served socio-economically diverse populations. The number of children in the schools ranged from 115 to 480. One school was located near a military base and served these families. Six schools came from the same school district, and the other two schools were from separate school districts. Written agreement to implement the programs and to be involved in the research was provided by initial school administrators and all schools continued to be involved in the research over the entire period. To support program start-up and incentivize participation in the research, the schools received all WITS Program books and start up resources (valued at $1000) in August or September of the start-up year in 2010. Each school also received supplies needed to continue the programs annually (worth about $200). Contact with the research team was facilitated by school liaisons (a principal or vie principal and a designated individual) and was limited to conducting the qualitative interview. Interventions to assist the programs or enhance fidelity in the schools occurred only in response to schools’ requests. In order to create ‘‘real world’’ conditions these requests for help were handled by the WITS Programs’ community coordinator, employed by the Rock Solid Foundation. Reflecting this real world approach, schools varied in their use of the core WITS components: Five schools were implementing eight core program components identified above; whereas three schools were implementing only five or six of the eight core components. Method Face-to-face interviews were conducted, individually, by two research assistants (an employee of the research team and a graduate student) who were distinct from the program development team. Research assistants received training and practice in qualitative interviewing techniques and interview quality was monitored by the first author. A semi-structured questionnaire was created to elicit responses directly relevant to sustainability planning. Questions were designed to be open-ended and non-directive and included: Do you think the WITS Programs will continue to be used? How will they be supported? What about the program makes you think it will or will not continue to be used in your school next year? What about in 5 years? Do you think the program would look different in 5 years? What would you need to be able to keep it going next year? What do you think is important for ensuring the sustainability of the program in your school? What kind of support would you need to keep it going in the long term? Following standard procedures for ground

123

124

theory analyses, questions were modified and expanded during the interview period to encourage elaboration and probe emergent themes generated by respondents. Analysis All interviews were transcribed verbatim and coded using QSR International’s NVivo9 (2010) Software. We used grounded theory analysis, specifically line-by-line coding, to facilitate the identification of participant-generated themes and to reveal their ‘‘in vivo’’ experiences with and plans for sustainability of the WITS Programs (Charmaz 2006). Analyses were completed by the second author. Although theoretical frameworks about sustainability are emerging, none have directly investigated program users’ decision making about maintaining prevention programs after implementing them and the specific concerns of sustainability related to rural schools has not been addressed. Thus, a grounded theory approach was used to facilitate the identification of possible additional themes that could advance understanding of sustainability decisions in rural schools.

Results Reviewing the transcribed interviews with school and community staff members, the second author independently identified 203 open codes that represented the speakers’ unique statements about sustainability planning for WITS. Consistent with grounded theory, we next identified and discussed commonalities in meanings across the 203 open code data. We grouped the open codes into focused codes by regularly comparing the open code data within each emerging focused codes to create distinctive groupings that captured their common meanings. Finally, we reviewed the groupings to ensure that the each of the open code statements were fully represented by and were consistent within the final 58 focused code categories. The 58 focus codes, in turn, were discussed and organized into 10 categories by further comparing their similarities (see Table 1 for examples). To also facilitate their presentation, we highlight the context basis of the focus codes using the following broad headings: (1) Within-School Influences, (2) Influences of the External Context (3) Program Characteristics and Support, and (4) Effects of Variations in Implementation.

Am J Community Psychol (2015) 56:120–133

sustaining program use. For example, one principal says he tries to keep the program ‘‘fresh’’ by directly promoting it to all school staff and ensuring new members are trained: I promote it [WITS]. We’ve gone through the [online] training with all the support staff, it’s always a topic of conversation, teachers know that when a kid comes to me that’s the approach I’m gonna’ use with them. When new teachers come to our school I talk to them about it and make sure they are familiar with WITS, I walk them through the process so we do a bunch of little things. In another school, the principal sustains children’s and teachers’ focus on WITS by making monthly visits to each classroom and reads a WITS book and conducts a WITS lesson (normally this is a teacher’s responsibility). Accountability to administrators also motivated teachers or other staff to continue to use the program in the context of competing demands on their time. One librarian received frequent reminders from the principal to organize the WITS resources and to get the program going—she now voluntarily updates him on the status of the program and teachers’ use of the books. Buy-in from teachers was also seen as particularly salient to long-term stability. One principal says: I think the teachers need to take ownership and believe in it [WITS] and I think right now they are kind of, ‘‘we do it…’’ and they enjoy parts of it but I don’t know that they enjoy it enough to champion it, to continue it. So we need to just relook at it and say, ‘‘okay where are we at with that?’’ Clearly defining teacher buy-in as their attitude toward the program and ongoing belief that WITS is ‘‘making a difference’’, another Principal says: Teachers have to buy in, teachers have to believe in the program, like the program, see a difference, making a difference because there’s too many things going on they have to do, too many things and if it’s too easy to drop because it’s kind of like, ‘‘well we do it once in a while and my hearts not in it’’ then they drop it right? So that’s where I think we just need to tweak so that they are more involved. They do like the program, but I think they need to be more involved and take some ownership of it rather than it being given to them, brought to them, they need to be involved in it.

With-In School Influences Leadership and Staff Buy-In

Processes of Embedding the Program and Creating a Common Language

The school leadership teams (variously including administrators, librarian, and teacher champions) play active roles in

Participants planning to continue to use WITS describe how WITS was part of their school culture, codes of

123

Am J Community Psychol (2015) 56:120–133

125

Table 1 Examples of statement categories and focus codes Statement categories

Sample focus codes

Within school Leadership and staff buy-in

Promoting and supervising program use and overcoming resistance Believing or not believing in program Taking ownership of program (staff)

Processes of embedding the program and program language

Formalizing WITS in codes of conduct Embedding, routinizing, language Transferring program knowledge though children

Ongoing review and goal setting

Coordinating program implementation Setting goals for program Communicating about program

Managing organizational change

Transferring responsibility across staff changes Involving multiple stakeholders Re-envisioning school direction and goals

External context Maintaining community connections

Initiating program by community leader Not maintaining community leader Eroding program without community leader

Policy environment change

Formalizing program support Reducing staff curriculum burden Altering context for program delivery

Competing programming

Infringing on or overtaking WITS Meshing with WITS Resisting intrusion from other programs

Program characteristics and support Enhancing perceived value

Providing quality literature Distributing a simple program

Support from WITS staff

Delivering an essential resource Adding fresh resources in response to feedback Ongoing communication with schools Monitoring school satisfaction

Variation in sustainability Differences in implementation

Maintaining program actively or passively Energizing commitment to program Maintaining only select components

conduct, or every-day practice. This also happens as they integrated WITS into their regular curriculum and teaching by using WITS during art lessons, music class, literary strategies, and Aboriginal education. Formally, embedding the WITS language in the schools’ code of conduct or mission statement and overall approach to enhancing social emotional development and reduce problem behaviors sustained the program part of the growth plan. As one participant says: I think it will fit in very well, we just wrote a new mission statement. We’re kind of going through that process this year and we are working on a new code

of conduct and trying to roll the WITS process into all of that so that it’s not a stand-alone. But it’s part of PBS [Problem Behavior Support] and it is part of the school growth plan and all that. WITS provided an overall framework for the social emotional programming in another school: The ‘‘[school name] way’’ is our way of saying be a good citizen and use your steps, and we kind of pulled from that seven pillars [values curriculum] and it’s all a matter of picking and choosing what fits and what supports and I think even though we pick and

123

126

choose a lot of things, the WITS is the underlying foundation for all the things we do. A teacher describes her every-day reference to the program, saying: We are continuing with using it outside, in the hallways. I think - so still using it sort of as our school culture, to always use every day before they go outside it’s ‘‘keep your hands off and use your WITS’’ It’s what I say to them every day and it’s been two years now and they’re kind of getting it so it’s commonplace, it’s just what they do I think. So we are seeing a lot now. All participating schools were maintaining the program language even when they rarely used other program components (i.e., reading WITS books, using lesson plans or having visits from community leaders). ‘‘Using your WITS’’ became a code word for dealing with peer conflict that connected children, teachers, principals, parents and community leaders. One teacher believes the connections made possible by this common language have helped it ‘‘stand the test of time:’’ I’m excited that it stood the test of time because I think this is probably the third or fourth year that we’ve used in our school and that it’s such a natural language out on the playground and I know our lunch monitors, that the duty people love it because it gives them- they only come onto the site for an hour at lunch time so it gives them a common language that they can use and they know we are all using. I don’t see the kids rolling their eyes when you say ‘Have you used your WITS?’ It’s just very natural to everyone. Another participant says: I think that the teachers buy-into it, in that we really like the common language so that everybody knows what the question is ‘‘have you used your WITS?’’, the kids all know what that means, the teachers know what that means, the parents even have quoted to me ‘‘well he used his WITS today’’ and so on. Children also play an essential role in maintaining the program language and transferring knowledge about WITS to new members of the school community. One principal describes the program language as ‘‘self-sustaining’’ because it is propagated by the children. Another participant says: I think part of it would be building from the ground up so kids know it and it has a sustaining nature based on that alone. The kids are bringing it up and it’s enjoyable to them. Teachers go ‘‘Okay this is

123

Am J Community Psychol (2015) 56:120–133

something.’’ And I think we want the common language too…. WITS is the same in every single class, every teacher can say did you use your WITS and every kid knows what he meant, so I think that’s a good reason to keep it on. Ongoing Communication: Review, Planning, Goal Setting and Re-commitment Several participants elaborated on the theme of requiring ongoing communication and renewing of commitments to sustain the program. They described the need for regular discussions at staff meetings to set priorities for the program, envision its future use, and identify what is working and areas for improvement. Believing in the program matters, as one teacher says: I think we just need to take the time and sit down as a staff and go ‘‘Okay, what do we - what are the positives about this, why do we want to continue this, what do we like?’’ Okay, these are the things; ‘‘What are we struggling with?’’ And then looking at those struggles; ‘‘How do we change those?’’ and sort of set out what we want to do. Do we want to do a book a month? How often do we want the community person in? What kind of a visit should that look like?… But I think we need to sit down as a staff and decide those, right? What is it we want to keep? What do we want to change? What do we believe in? What should it look like? Another teacher saw collaboration with a new community leader would be an antidote to both program drift and teacher apathy saying: [WITS] will keep going next year, and it will probably keep going without a community liaison, a community leader. It will probably continue the same way as it is continuing now which is some teachers are doing the books and the lessons and some not, and it just becoming sort of the language that’s used in the school. With the [new] leader, I can see us going back to an assembly and kind of doing more of the kickoff and having him or her in classrooms, I can see more of that. On the other hand, being unable to sustain collaboration with a community leader challenged program continuity in one school that experienced heavy turnover of both staff and children related to military transfers. One participant says: I think there has been probably less community involvement this year than there has been and that’s due to we’re quite a transient group as well, or

Am J Community Psychol (2015) 56:120–133

transient community. We do have some new staff coming in next year, so we will need to start from getting everybody back on board again. Managing Organizational Stability and Change Participating schools experienced high rates of staff turnover, including routine rotations of school principals. Keeping the program alive required transferring the responsibility for implementation from person-to-person and depending on the buy-in of ongoing teachers, children, and parents. One principal says teachers taking ownership of the program and the fact that multiple stakeholders are also involved (i.e., kids, parents, and community leaders) make it likely that WITS would be sustained in her absence. She says: [the WITS program reaches the parents] through conversations and Parent Advisory Councils, and the teachers own it. It’s not just me. It’s letting them know that and they’re very familiar with different principals in a short period of time in the school. That’s just the nature sometimes of the job, it is transitional. And so keeping it consistent, because it’s their school, it’s not just one person. They are all stakeholders within this community; the parents, the kids, the teachers and all the staff, all the adults. So that’s how it continues and keeping it documented and keeping a part of the school code of conduct and having it visible in the school is how it will continue. However, engaging or re-engaging multiple stakeholders over time was challenging and required frequent renewal of commitments. A vice-principal says the program will continue, but says: I do worry if I may be transferred for the next year that who knows what happens to it then? Not that I’m the only one promoting it but I organize it. So, unless the person who came in and replaced me is familiar [with the program], that would be of concern. Another participant says current attitudes towards implementation matter: I think if we have a new vision of how we want to use it before I leave we’ll be okay. I think that if it carried on the way that we’re doing it right now it won’t continue with the change of people. The spread of the program beyond single schools to the entire school district also helped to overcome the movement of children within the district. It’s simple, it’s easy language, kids understand it, it’s consistent, it’s consistent when kids move from one

127

school to another, a lot of our schools are doing it. Those are the strengths of [WITS]. Influences of the External Context Maintaining Community Connections Policy changes coming from outside the educational system could restrict needed collaborations with the community. One school principal reports difficulty recruiting an appropriate community leader. Ideally, he hoped to partner with their community liaison officer, a member of the Royal Canadian Mounted Police, who also implements the DARE program in their school. However, the officer could not devote additional time to participate in the WITS program because of budget cutbacks to school liaison officers. Without this option, one school was left floundering to find new collaborators who fit the needs of their school and the community component of WITS was dropped. As the principals says: I’ve been talking to our liaison officer about him coming in and you know doing some reading of books with kids and I thought you know that would be something that would be really cool to have them. One thing is that they’ve done a real big cutback on these liaison officers. They’re not even here as often. So I think it is something that would be really neat but it has to be the right person and it has to be somebody that our teachers would trust coming in and like the idea. But who that person would be I’m not exactly sure. Policy Environment Change Participants noted that the years in which the WITS programs were being implemented were marked by two significant changes in educational policies that were initiated at the provincial level by the British Columbia Ministry of Education. One policy directive that appeared to enable WITS reduced and integrated the number of required learning objectives in each grade. A principal believed that this would give teachers more flexibility to create their own programming, and allow them to choose the WITS resources. Another participant says that direct provincial support or endorsement from the BC Ministry of Education could enable long-term sustainability for the WITS program. He says I don’t see why it would disappear like I’m wondering at what point something becomes so big that it would never disappear like you’re as big as the alphabet. The alphabet’s here to stay you’re not

123

128

making a new letter. At what point does the WITS become- maybe if the BC Minister of Education says we want WITS at all schools and here’s why. In the second new policy initiative, the Ministry adopted and implemented a strategy to ‘‘Erase Bullying’’ that focused on threat assessment and intervention with offenders, and reduced both time for and emphasis on prevention. This new anti-bullying initiative created uncertainty about the fate of the WITS Programs in some schools. As one principal says. I just heard this afternoon that the Ministry, who provided work through the ‘‘Erase Bullying Campaign’’ with [the Premier’s] initiative last summer, that they now have a four year plan. So originally it was 2 years and now its 4 years. And that’s around training, around level 1 threat assessment - the Erase Bullying part, so it will have a second piece to it. And who knows what other pieces it will bring. Competing Programs Participants cite the continued growth of new programs as a threat to long-term use of WITS. Seasoned staff members report a long history of waxing and waning programs. Even programs that have been used for many years often lose momentum and are replaced by new ‘‘flashy’’ initiatives. One participant says, the programs’ future is not guaranteed rather, It just depends. Things come up down the road. Like I said, there could be something else that goes, not saying replaced, but just, we want things within schools to take another direction around student safety or student behavior. Another teacher describes her experiences with change, saying ‘‘people like difference’’: I was involved with the Lions Quest for years and years and years and after about five or six years I started to see that program wane a bit and I see that happen with lots of programs that come in. They seem to have legs for a certain amount of time and then they seem to fade away and I don’t know that’s because something better comes up or something different comes up and people like difference, because you’re a school maybe you’re doing WITS and I don’t know how many schools in our district are doing WITS. Do you know? Yet, some participants also believe that WITS is resistant to infringing programs. One participant believes that the program has ‘‘stood the test of time’’ and it would continue:

123

Am J Community Psychol (2015) 56:120–133

I think so. I hope so because if it fits our need I can’t really see what new flashy thing might take its place. Because around anti-bullying day I’ll go online and I’ll find some lion named Daren and I think it’s from the DARE program and I’ll go ‘yeah that was all right.’ But it doesn’t making me think ‘I gotta use this now’ so I think WITS has stood the test of time here so it will continue to be part of our program. Program Characteristics and Support Enhancing Perceived Value The continued fit of the WITS Programs resources fostered its sustainability for some schools. Participants identify books, multicultural messaging, portability, and simplicity as program elements that help to sustain the program. One participant describes the resources and messages that fit her needs ‘‘in any situation’’. She says: I think because the books have such a variety, there’s such a variety of topics, they are definitely multicultural, and they’re definitely very multicultural so that’s very supportive. Not that if you walked in our hallways you’d think we were a multicultural school, but it’s nice to have resources that are and I just think it’s so, not generic but it’s so basic and strong in itself, they’ve really honed it down to a short sweet simple message so it fits anywhere in any situation. The perceived utility and impact of the program in achieving teachers’ own goals also motivates them to sustain it. As one teacher says it will continue: Well because the little kids need an acronym to help solve their problems. So it’s a nice – and using your WITS is what it is…. They need the language to deal with conflicts that arise, so the program is necessary. Ongoing Support from Program Developers (WITS Program Staff) Participants suggest that access to support from the program’s staff played a sustaining role through updating and providing program materials and reaching out to communicate with schools. Annual contacts with the research assistants also provided participants with opportunities to give feedback to the program staff for desired improvements. For example, some participants wanted materials (e.g., books) that were compatible with new technologies they used in class (i.e., Smartboards and iPads). Schools with high levels of implementation required more books and lessons plans. One participant describes the need for

Am J Community Psychol (2015) 56:120–133

enhanced communication about new WITS resources saying: Resources are always beneficial and because there’s only so many books and so many times you can share those books with kids. Continued support through giving us the resources needed, whether it’s a newsletter or an email saying check out this new thing that’s added to the WITS [web] site, this is something that you might want to do within your school, here’s a book, or here’s some money that you can purchase these books and here’s a recommended book list that we have. Those ideas I think would be helpful. The opportunity to provide feedback and contribute to program improvement by interacting with the research assistants motivated one participant to sustain the program. Another said that the meetings with the research team held her accountable and caused her to reflect on what she could do to improve WITS in her school. Maybe having an accountability piece from the research team at the University or looking through Rock Solid. I’ve done this interview twice with you guys, but since I met you last time it’s like ‘‘Okay, now I’ve got to come to the table knowing you’ll probably be back. What am I going to do to show that we’re making significant improvement? Was WITS in our building?’’ I’ve seen a difference, but I think it’s also reflected upon how you’re doing your job, so it just makes me think about ways we can continue promoting. So that’s how I see it being sustainable. Variations in Implementation Differences in Implementation In some schools the fidelity of program implementation varied over time and was cited as a concern related to sustainability planning. Rather than an all or nothing concept, sustainability planning was conceptualized in the context of how well or completely the program was currently implemented. Components that could be maintained with little effort such as the WITS language could persist, even in the context of changing environments for program delivery; whereas other core aspects were dropped compromising both program fidelity and the likelihood of its maintenance over time. In one school, a principal reports that they are currently only sustaining the use of the WITS acronym to remind children of skills for managing conflicts after adjusting to their community leader retiring.

129

I think we’re kind of in maintenance mode right now I would say, we don’t have the liaison person with us anymore. So we’re not getting the visits, we’re not getting the WITSUP or the assembly so anything that we’re doing are things that we are doing with the knowledge we had from before I think that’s using the acronym a lot, the kids pretty much to the person, I think, can recite what it means to use your WITS, that’s almost the first thing that would happen when a kid comes into the office and there’s been a kerfuffle on the playground, ‘‘Do you know what the WITS are?’’ and they can almost all recite that so I think what we’re doing with it is continuing to employ the skills we’ve learned over the past few years. Even in schools with high implementation, some participants express concern that without their continued efforts to energize the program it may fade over time. As one participant says: I think we’re probably in that cruising zone of the WITS, like we’ve all kind of learned it, we implemented it, and now we’re just kind of cruising along hoping it maintains but probably not putting the energy into it that we should, and hopefully it doesn’t fall apart because of that.

Discussion This paper reports on the descriptions of the factors that influenced sustainability planning for continuing the WITS Programs given by staff from eight rural Canadian elementary schools that had been using the programs for 2 years. Our assessment of program fidelity at the time of the interviews revealed that five of the schools were fully implementing the program, and three were able to maintain the less complex aspects of the program (program language and day-to-day strategies), but they were having difficulty maintaining the program components that prescribed specific teacher and community leader activities. While sustainability has been conceptualized as a step that follows high level program implementation (Flay et al. 2005; Kellam, and Langevin 2003), recent research draws attention to ongoing cyclical relations between implementation and maintenance across the life cycle of prevention programs (Han and Weiss 2005; Scheirer 2012). Findings from this qualitative study support this life-cycle theory of sustainability and suggest that sustainability can be understood as a function of ongoing planning for renewing implementation in school settings. While our findings are based on qualitative data that are specific to our participants’ experiences with the WITS Program, it is

123

130

likely that they have heuristic value in illuminating processes that are fundamental (beyond budgetary concerns) to sustaining prevention program in most school settings. Overall, the participants in this qualitative study describe both processes that fostered self-sustaining cycles of implementation over time (e.g., building on staff-buy in and embedding the program activities and common language in codes of conduct), as well as the need for ongoing discussions about the renewal of commitments to using the WITS Programs and efforts to avert program decay (e.g., managing staff turnover, renewing goal setting, and maintaining parent and child involvement and collaborations with a community leader). On the one hand, the institutionalization of program language and activities into school codes of conduct and everyday practice ensured that it would ‘‘stand the test of time’’. On the other hand, turnovers in leadership, staff and children, declining stakeholder investments, difficulties in engaging cross-sectorial support, unfavorable policy environments, and perceived advantages of competing programs were identified as potential threats to program continuity. Mediating between these process of sustaining implementation or averting decay; school leadership and program staff involvement were needed to actively promote the program, provide opportunities for training new staff, update and renew resources, and resist encroachment from new initiatives. Schell et al.’s (2013) framework identifies the core domains that affect ongoing sustainability processes in community health organizations; namely, funding stability, political support, partnerships, organizational capacity, program adaptation, program evaluation, communications among stakeholders, perceived impact and strategic planning to define ongoing and future goals and direction. The participants in this study also described several of these key influences on program sustainability and how they operate within the organizational context of schools (e.g., leadership, institutionalization of the program, ongoing review and planning, and managing staff turn overs, and changing school priorities). Participants also noted the effects of changes in the availability of community collaborations, the provincial policy environment, and characteristic of the program that were not under the schools’ control. They gave examples of how the continued interactions among school, community, policy, and programs facilitated ongoing program implementation. Moreover, the enthusiasm of teachers and parents as well as children’s use of the program language encouraged school leaders and program champions to sustain the program in spite of turnovers in staff and children in the schools. Teachers beliefs in the program and ongoing informal evaluations of whether was ‘‘making a difference’’ mattered. Leaders also relied on the participation of community leaders and responded to educational policy shifts.

123

Am J Community Psychol (2015) 56:120–133

Our findings are also consistent with the incremental self-sustaining cycle described by Han and Weiss (2005). Greater fidelity of implementation of core program components aided long-term commitment; yet this was neither inevitable nor necessarily incremental. Teacher attitudes towards the program, beliefs that it had an effect on children, and ongoing investments in implementation did play a role in this self-sustaining cycle. The participants’ also identified the clear need for both school and program staff to anticipate and respond to ongoing changes both within and outside of the school that could erode early investments made in implementing programs over the long term. Changes in school staff, community leaders, and policy contexts challenged program implementation, and necessitated ongoing renewal, recommitments, and retraining to enable implementation and sustainability. Sustainability appeared to depend on how well programs, schools, and policy environments worked together over time to cope with leadership, staff and child turnovers, policy and priority changes, and competition from other innovations. The participants suggested that sustainability planning relied on regular staff meetings to evaluate, from the school’s perspective, what was working and what needed changing; on active leadership to ensure new staff acquire skills to use the program; on support from provincial policies; and on innovation within the program to it relevant and technologically up to date. These findings also suggest that sustainability processes may also be informed by theory and research on implementation and the diffusion preventive innovations. For example, consistent with Roger’s (2002) terminology (shown in brackets), school staff described the need for regular opportunities to evaluate and re-evaluate (re-invent) the perceived compatibility of the WITS programs with their existing and changing values, needs, and beliefs and to weigh its value in relation to new programs (relative advantage). Participants also related sustainability to the programs’ ongoing visibility (observability) as it became entrenched in school culture and everyday language, and formalized in codes of conduct. Program champions (opinion leaders) who included principals, librarians and teachers needed ongoing support. Encouraging peer-to-peer communications about ongoing use of the program (activing peer networks) in staff meetings and fostering annual reviews and goal setting meetings appeared to be required for ongoing commitments to sustaining the WITS Programs. Leaders’ ongoing self-evaluation of the staffs’ beliefs about whether the program was achieving the desired effects and children’s use of the language and strategies also appeared to motivate continued program use. Furthermore, implementation quality appeared to intersect with sustainability such that schools with better adherence to program components were more likely to

Am J Community Psychol (2015) 56:120–133

describe a firm plan that enabled them to continue to use the program. Recipients (children and parents) roles in maintain commitments to the program were also recognized by our participants. Teachers and administrators were more likely to buy into the program in schools where children already understood and used the WITS language. Stable community leaders could also encourage schools to maintain WITS across school staff turnover by maintaining their involvement with the swearing-in ceremony and class visits. Consistent with the life-cycle theory of sustainability, the multi-sectorial approach of the WITS Programs fosters sustainability by embedding stakeholder roles in the programs’ design, uptake and implementation strategies. Program Characteristics that Support Sustainability It is also likely that programs, themselves, must work to avoid becoming static when the nature of social problems changes while balancing fidelity to core components. It is important that programs respond to the need to be implemented in the context of changing technology, and the need for new or updated resources that keep the program relevant while retaining what works. Online programs have an advantage of being updated in ways that packaged programs cannot. To accommodate staff turnover, the WITS programs provides ongoing access to online training modules. Newsletters actively inform schools of new WITS resources three times a year and schools must make annual requests for programs supplies (e.g., posters, gifts for children etc.) and new resources. This requires schools to recommit annually to using the program. Teachers are encouraged to post and share their adaptation with WITS staff and other teachers through social media. Creating a dynamic program that can respond to the needs for new or more technologically sophisticated resources may play a key role in sustaining commitments to the program. For example, new resources (e.g., posters books, gifts, songs) can be created that inspire program use without compromising fidelity. However, sustaining implementation may also require that core program components or activities, central philosophy, and intended mechanisms of action are clearly spelled out. Adaptations that leave out or change core components, ignore intent or overlook mechanisms of action can compromise program fidelity (Ozer et al. 2010).

Limitations As with all qualitative studies, our findings are heuristic and are not generalizable to the experiences of individuals working in all schools or all programs. All participants were from rural Canadian elementary schools that

131

represented considerable diversity in size and demographics of the populations served. Because the purpose of the study was to generate decision-making themes and only a eight schools were engaged, analyses were conducted at the level of individuals and do not speak to potential differences in the school contexts. It is possible that variations in school-level characteristics, such as levels of poverty of parents, transience among the staff and children, organization and management also effect both implementation and sustainability decisions (Chilenski et al. 2015). Sustaining programs in urban settings or on a national scale offers challenges beyond what are described here for these small rural communities. Moreover, the reality is that few evidence-based programs are disseminated by schools or school boards at a level that is sufficiently widespread that they could have an impact on serious social problems (Leadbeater and Gladstone 2015). Nevertheless realizing investments in mental health promotion programs appears to require a broad shift from providing support only for initial uptake toward supporting schools and programs’ capacities to work together across the program’s life cycle. This encompasses how mental health promotion programs are discovered, made accessible, taken up, integrated, and sustained in schools over time.

Conclusions and Recommendations In summary, in this qualitative study, the ongoing ability to sustain a universal prevention program in schools was the product of reviewing and renewing commitments of multiple stakeholders; including, program developers, program implementers (community leaders, teachers and administrators), and program recipients (children and parents). Several strategies were identified that could enhance decision making needed to maintain commitments to programs. Specifically, school leaders and program champions within schools needed to be designated and supported at all stages in the program lifecycle. Embedding program activities in school codes of conduct and in the every-day school activities and language served to enhance the likelihood that it would remain visible. Annual opportunities to discuss and evaluate the program’s ongoing fit and staffs’ beliefs about desired effects are needed to enhance ongoing buy-in for sustaining a program. Long-term support from educational policy makers and community collaborators at the local and provincial level also can be useful in enhancing the sustainability of evidence based programs. Program renewal and responsiveness driven by the program developers also appeared to make long-term use more viable in the context of competing programs and new technologies. It is clear that sustainability planning in schools is not merely a next step following high quality implementation,

123

132

but rather involves ongoing communication, evaluation, and re-commitment processes that need to be anticipated and supported by both school leaders and program developers in order to realize the large investments that are being made in evidence-based programs. More research is needed to identify the practical targets that allow for the collaborations needed to effectively sustain investments in evidence-based programs in schools.

References Bierman, K. L., DeRousie, R., Heinrichs, B., Domitrovich, C. E., Greenberg, M. T., & Gill, S. (2013). Sustaining high-quality teaching and evidence-based curricula: Follow-up assessment of teachers in the REDI project. Early Education and Development, 24, 1194–1213. doi:10.1080/10409289.2013.755457. Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London: Sage. Chilenski, S. M., Olson, J. R., Schulte, J. A., Perkins, D. F., & Spoth, R. (2015). A multi-level examination of how the organizational context relates to readiness to implement prevention and evidence-based programming in community settings. Evaluation and Program Planning,. doi:10.1016/j.evalprogplan.2014.10. 004. Cooper, B., Bumbarger, B. K., & Moore, J. E. (2013). Sustaining evidence-based prevention programs: Correlates in a large-scale dissemination initiative. Prevention Science,. doi:10.1007/ s11121-013-0427-1. Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221. doi:10.1207/S1532768XJEPC1102_04. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0. Durlak, J. A., & Wells, A. M. (1997). Primary prevention mental health programs: The future is exciting. American Journal of Community Psychology, 25, 233–243. doi:10.1023/A: 1024674631189. Flay, B. R., Biglan, A., Boruch, R. F., Castro, F., Gottfredson, D., Kellam, S., & Ji, P. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6(3), 151–175. Greenberg, M. T., Domitrovich, C., & Bumbarger, B. (2001). The prevention of mental disorders in school-aged children: Current state of the field. Prevention & Treatment, 4(1), 1–64. Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Psychology, 33, 665–679. doi:10.1007/s10802-0057666-2. Jones, S. M., & Bouffard, S. M. (2012). Social and emotional learning in schools: From programs to strategies. Social Policy Report. Society for Research in Child Development, 26(4), 1–33. Kellam, S., & Langevin, D. (2003). A framework for understanding ‘‘evidence’’ in prevention research and programs. Prevention Science: The Official Journal of the Society for Prevention Research, 4(3), 137–153.

123

Am J Community Psychol (2015) 56:120–133 King, D., Glasgow, R., & Leeman-Castillo, B. (2010). Reaiming REAIM: Using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. American Journal of Public Health, 100, 2076–2084. doi:10.2105/AJPH.2009.190959. Leadbeater, B. J. & Gladstone, E. (2015). Supporting schools for the widespread implementation of evidence-based mental health promotion programs: What is needed? In R. Shute, & P. Slee, (Eds.), Mental health and wellbeing through schools: The way forward. e-ISBN 978-1-315-76469-6. Leadbeater, B. J., Gladstone, E., Yeung Thompson, R. S., Sukhawathanakul, P., & Desjardins, T. (2012). Getting started: Assimilatory processes of uptake of mental health promotion and primary prevention programmes in elementary schools. Advances in School Mental Health Promotion, 5(4), 258–276. doi:10.1080/1754730X.2012.736790. Leadbeater, B., & Sukhawathanakul, P. (2011). Multicomponent programs for reducing peer victimization in early elementary school: A longitudinal evaluation of the WITS primary program. Journal of Community Psychology, 39(5), 606–620. doi:10.1002/ jcop.20447. Leadbeater, B. J., Thompson, K., & Sukhawathanakul, P. (2015). Enhancing protective processes to prevent bullying and peer victimization: The WITS programs cluster randomized trial. (Submitted). Lee, R., & Gortmaker, S. (2012). Health dissemination and implementation within schools. In R. C. Brownson, G. A. Colditz, & E. A. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 419–436). New York, NY: Oxford University Press. Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The quality implementation framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50, 462–480. doi:10.1007/s10464-012-9522-x. Ogden, T., & Fixsen, D. L. (2014). Implementation science: A brief overview and a look ahead. Zeitschrift Fu¨r Psychologie, 222, 4–11. doi:10.1027/2151-2604/a000160. Ozer, E. J., Wanis, M. G., & Bazell, N. (2010). Diffusion of schoolbased prevention programs in two urban districts: Adaptations, rationales, and suggestions for change. Prevention Science, 11(1), 42–55. doi:10.1007/s11121-009-0148-7. Payne, A., & Eckert, R. (2010). The relative importance of provider, program, school, and community predictors of the implementation quality of school-based prevention programs. Prevention Science, 11, 126–141. doi:10.1007/s11121-009-0157-6. Rhoades, B. L., Bumbarger, B. K., & Moore, J. E. (2012). The role of a state-level prevention support system in promoting highquality implementation and sustainability of evidence-based programs. American Journal of Community Psychology, 50, 386–401. doi:10.1007/s10464-012-9502-1. Ringwalt, C. L., Pankratz, M. M., Jackson-Newsom, J., Gottfredson, N. C., Hansen, W. B., Giles, S. M., & Dusenbury, L. (2010). Three-year trajectory of teachers’ fidelity to a drug prevention curriculum. Prevention Science, 11, 67–76. doi:10.1007/s11121009-0150-0. Rogers, E. M. (2002). Diffusion of preventive innovations. Addictive Behaviors, 27(6), 989–993. doi:10.1016/S0306-4603(02)003003. Scheirer, M. A. (2012). Planning evaluation through the program life cycle. American Journal of Evaluation, 33, 263–294. doi:10. 1177/1098214011434609. Scheirer, M., & Dearing, J. (2011). An agenda for research on the sustainability of public health programs. American Journal of Public Health, 101, 2059–2067. doi:10.2105/AJPH.2011. 300193.

Am J Community Psychol (2015) 56:120–133 Schell, S., Luke, D., Schooley, M., Elliott, M., Herbers, S., Mueller, N., & Bunger, A. (2013). Public health program capacity for sustainability: A new framework. Implementation Science: IS,. doi:10.1186/1748-5908-8-15. Sloboda, Z., Dusenbury, L., & Petras, H. (2014). Implementation science and the effective delivery of evidence-based prevention. In Z. Sloboda & H. Petras (Eds.), Defining prevention science (pp. 293–314). New York: Springer. doi:10.1007/978-1-48997424-2. Spoth, R., Guyll, M., Trudeau, L., & Goldberg-Lillehoj, C. (2002). Two studies of proximal outcomes and implementation quality of universal preventive interventions in a community-university collaboration context. Journal of Community Psychology, 30, 499–518. doi:10.1002/jcop.10021.

133 Spoth, R., Schainker, L. M., Redmond, C., Ralston, E., Yeh, H., & Perkins, D. F. (2015). Mixed picture of readiness for adoption of evidence-based prevention programs in communities: Exploratory surveys of state program delivery systems. American Journal of Community Psychology, 55(3–4), 253–265. doi:10. 1007/s10464-015-9707-1. Tibbits, M. K., Bumbarger, B. K., Kyler, S. J., & Perkins, D. F. (2010). Sustaining evidence-based interventions under realworld conditions: Results from a large-scale diffusion project. Prevention Science, 11, 252–262. doi:10.1007/s11121-0100170-9.

123