Rigorously assessing the fidelity of implementation of programs’ core components will advance our understanding of how mindfulness-based programs for adolescents may or may not work.
4 Assessing fidelity of core components in a mindfulness and yoga intervention for urban youth: Applying the CORE Process Laura Feagans Gould, Tamar Mendelson, Jacinda K. Dariotis, Matthew Ancona, Ali S. R. Smith, Andres A. Gonzalez, Atman A. Smith, Mark T. Greenberg a growing number of studies are documenting both the feasibility and potential benefits of mindfulness-based programs for youth and adolescents.1 As these programs continue to gain in popularity, it becomes increasingly important to understand which aspects or components of these programs may be helpful for adolescents, and how those components should be delivered so as to have the most beneficial impact. In this chapter, we describe our team’s approach to identifying—and beginning to evaluate and refine—the core components of a mindfulness and yoga program for urban early adolescents.
NEW DIRECTIONS FOR YOUTH DEVELOPMENT, NO. 142, SUMMER 2014 © 2014 WILEY PERIODICALS, INC. Published online in Wiley Online Library (wileyonlinelibrary.com) • DOI: 10.1002/yd.20097
59
60
MINDFULNESS IN ADOLESCENCE
Program core components Program core components are “the most essential and indispensable components of an intervention practice or program.”2 They are essentially the “active ingredients” necessary to produce desired outcomes and are directly related to a program’s theory of change, which proposes the mechanisms by which an intervention works.3 Identifying the core components of a particular program improves our understanding of how individual elements of that program, or a combination of those elements, work together to impact intended outcomes. It also helps ascertain which aspects are essential to be delivered as specified in a program manual, and which elements can be adapted across setting or population.4 Over time, identifying core components of evidence-based programs can help us collectively identify evidence-based practices or the specific activities that work across programs in order to produce intended outcomes. Identification of a program’s core components is often a complex and long-term undertaking in which resources are funneled into systematically developing and studying a program’s fidelity of implementation (FOI)—the extent to which intervention delivery adheres to the protocol or program model as intended by the intervention developers.5 Through the systematic study of FOI, researchers and program developers (1) dismantle a program into its constituent parts and integrate them into a coherent theory of change, (2) develop reliable and valid measures of FOI and establish criteria for implementation integrity, and (3) empirically examine whether variation in the implementation of core components is systematically related to particular outcomes across replication trials. Over multiple trials, this systematic study of the efficacy of program elements yields a more refined theory of change, which specifies the integral (core components) and nonessential aspects of a program.6
new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
61
Core components of mindfulness programs for adolescents Discussion of core program elements and mechanisms through which mindfulness programs may impact early adolescent wellbeing is in its infancy, as is much of the research in the field. Several recent reviews of the preliminary impacts of mindfulnessbased programs for youth and adolescents across a range of settings and target populations have been helpful in outlining some of the potential core components or “underlying principles” of these types of programs, as well as some general mechanisms through which mindfulness programs may impact adolescent outcomes.7 For example, mindfulness programs generally include structured activities that require sustained focused attention on an object, the breath, or a sound with a common goal of sharpening concentration or attention, improving emotion regulation skills, and enhancing self-awareness and knowledge.8 The specific techniques and approaches (for example, yoga, sitting meditation) used to focus attention, however, vary widely as do program lengths, settings, and target populations.9 Almost all of these reviews highlight the need for researchers and program evaluators to more rigorously assess the range of potential active ingredients of mindfulness programs for youth and adolescents. Indeed, in a brief review of eighteen studies published from 2005 to 2013 on mindfulness programs for youth and adolescents, we found that the majority of studies included a basic description of the program, espoused general underlying mindfulness practices or principles that guided program development, and spoke generally about program intended outcomes. Rarely, however, were core program components explicitly outlined and integrated into a formal theory of change (for example, logic model). Approximately half of the studies assessed at least one aspect of FOI, namely, dosage or the amount of the intervention received by participants. Nonetheless, there was an absence of the rigorous study of FOI through the development of reliable, valid measures; the inclusion new directions for youth development • doi: 10.1002.yd
62
MINDFULNESS IN ADOLESCENCE
of multiple sources of data for FOI constructs; and the establishment of a priori criteria for program fidelity. Thus, we believe that a discussion of the development of FOI measures and examination of program core components is a timely topic with relevance for the emerging field of mindfulness programs for youth and adolescents. This chapter provides a practical framework for mindfulness researchers and program developers to draw upon in identifying and rigorously assessing FOI of program core components. Specifically, this chapter outlines the process our team undertook in a recent study in which researchers worked with program developers to identify and measure FOI of hypothesized core components of a yoga and mindfulness-based program for youth. This process represents our initial attempt to engage in a learning cycle based on the Plan, Do, Study, Act (PDSA) cycle.10 We adapted the steps in the PDSA cycle to address the development, testing, and refinement of program core components. We call our adapted cycle the CORE Process: (C) Conceptualize core components, (O) Operationalize and measure, (R) Run analyses and review implementation findings, and (E) Enhance and refine. In this chapter, we describe the CORE Process, provide illustrative examples at each stage of the cycle, and discuss lessons learned as well as broader implications for mindfulness research with adolescents.
A community-academic partnership Our team represents a partnership between the founders of the Holistic Life Foundation, Inc. (HLF)—a nonprofit organization dedicated to promoting health and well-being in underserved communities through mindfulness-based practices (www.hlfinc.org)— and a group of university-based prevention researchers interested in understanding how mindfulness-based programs impact urban youth. The three HLF founders were born and raised in Baltimore City and work with urban youth and adolescents, particularly those new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
63
who face chronic stressors associated with poverty, violent neighborhoods, and lack of adequate environmental supports. The HLF founders conceptualize mindfulness as “the combination of awareness, centering, and being present. That is, mindfulness is awareness of your thoughts, emotions, actions, and energy; the ability to get centered and stay centered in all situations; and the ability to be present, not letting internal and external distractions take you from the current moment.”11 HLF’s program teaches mindfulness through yoga-based movement, breathing techniques, and attention training. The yoga-based postures offer a means of practicing mindfulness that is focused on body movements and breath. Sessions are led by an experienced instructor and assistant, and are conducted in spaces large enough to accommodate yoga mats for each child. Each session begins with several minutes of quiet breathing to settle in, then progresses to active yoga-based movements followed by breath work. Sessions end with several minutes of “guided reflection” during which the students lie on their mats and the instructor guides them in an attention training practice, such as following the breath, sending positive energy to others, or imagining being calm while taking a test. Throughout the sessions, the instructors tell the students about health-related benefits of the practices they are learning. They also engage the students in discussions about what stress is, how to recognize it in the body, and how to use breathing and poses to stay calm during times of stress. In 2006, we (the researchers) partnered with HLF (the program developers) to conduct a pilot randomized trial of their schoolbased program with fourth- and fifth-grade students across four Baltimore City schools. During the pilot trial, we worked with the HLF team to develop an initial logic model drawing on their programmatic goals and the empirical literature on the effects of mindfulness and yoga.12 In this initial logic model, the team proposed that the HLF intervention would lead to enhancements in self-regulation (for example, the ability to modulate thoughts and emotions) which would, in turn, promote improvements in social and emotional functioning (for example, mood, behavior). new directions for youth development • doi: 10.1002.yd
64
MINDFULNESS IN ADOLESCENCE
Findings from the trial generally supported this basic logic model: Students who participated in the HLF program reported significant improvements in their capacity to regulate thoughts and emotions in response to stress as compared with control group students, including fewer intrusive thoughts, less rumination, and lower levels of emotional arousal.13 Findings also indicated that the program was feasible and acceptable to students, teachers, and administrators. We are currently conducting a larger randomized controlled trial in which the HLF school-based program was delivered in three urban elementary schools to fifth- and sixth-grade students. As part of this trial, we committed a portion of our resources to identifying program core components, developing FOI measures, and assessing FOI of the HLF program. The CORE Process served as our framework for this work and is described in more detail below. Step 1 (C): Conceptualize core components Our first undertaking was to enhance program theory and identify the core components or potentially active ingredients of the HLF program.14 We drew upon several sources of information during this process, including written materials produced by the program developers, conversations with the program developers, and experiences from our pilot study.15 The HLF school-based program manual written during our pilot trial served as the starting point for our process of identifying core components because it specified all the activities in the program in the sequence they were to be delivered. To select and categorize core components, we reread the manual and engaged in a series of conversations with the HLF program developers about the central goals of the program, the structure of the manual and intervention, the rationale behind inclusion of specific activities and practices, and the intended benefit or result of different program activities for youth. Through these conversations, the team identified two types of HLF core program components: Core Activities and Core Processes. Core Activities are the activities and practices described in new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
65
the curriculum, consistent with what others have laid out as structural program elements.16 HLF Core Activities include: (1) breathing to settle in at the start of each session, (2) practicing yoga poses, (3) practicing breath work, (4) reviewing previous material (that is, previously taught poses and breathing techniques), (5) discussing benefits and relevance of the intervention activities for health and well-being, (6) engaging in a silent reflection involving focused attention, and (7) assigning homework to encourage use of the intervention practices outside the program sessions. Core Processes, however, address the manner in which the instructors engage with youth and the way in which the program is delivered.17 Our conversations with the program developers and observation of HLF sessions during our pilot study highlighted that these Core Processes are essential and sometimes distinguishing aspects of mindfulness instruction, calling on instructors to embody the qualities of mindfulness as a vehicle for teaching it.18 In effect, the HLF instructors model for the youth guides on how to interact mindfully; staying focused on a chosen target; and treating themselves and others with respect, dignity, and compassion. HLF Core Processes include: (1) engaging students in learning through applying material to their own lives, (2) maintaining order and discipline by redirecting negative behaviors and reinforcing on-task behavior, (3) modeling human compassion by treating themselves and others kindly and empathically, and (4) reviewing concepts and poses throughout session activities to reinforce experiential learning and mastery of material. These seven Core Activities and four Core Processes are aimed at helping students be present and regulate their thoughts, feelings, and behaviors. This in-depth examination of program components highlighted for us the extent to which HLF core program activities and processes are developmentally tailored for early adolescents. For example, when engaging students in learning, the instructors use simple and straightforward language, and they reference popular youth culture—including television shows, music, and celebrities—to make their points relevant for students. Instructors also build mastery of the material by encouraging new directions for youth development • doi: 10.1002.yd
66
MINDFULNESS IN ADOLESCENCE
Figure 4.1. Holistic Life Foundation (HLF) initial and revised program theory
different students to model poses and lead class exercises. They increase the teaching role and autonomy of student leaders over the course of the program so that students gain increasing confidence in their own abilities, an important developmental process for adolescents. With respect to Core Activities, more active movements are generally taught in the earlier portion of the sessions, so as to expend students’ restless energy and facilitate the closing period of calm reflection. Despite our previous familiarity with the HLF program and theory of change, identifying core program components was nevertheless a relatively lengthy process involving discussion with the HLF founders as well as detailed review of their curriculum. The resulting refined program theory served as the foundation for our FOI measurement strategy in Step 2. At the end of this first step, we had developed: (1) a set of clearly articulated HLF core program components and (2) a more fully articulated logic model that broke apart the HLF program into these constituent components, as outlined in bold, in the top portion of Figure 4.1. new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
67
Step 2 (O): Operationalize and measure Once the team had articulated the program core components, we addressed the challenge of operationalizing and measuring program FOI. As a foundation for this work, we used the five major dimensions of FOI identified by leaders in the field of implementation science: adherence, dosage, quality, responsiveness, and differentiation.19 This multidimensional approach provides a nuanced representation of fidelity.20 For the current study, we assessed four of the five dimensions of FOI: (1) adherence—the extent to which the Core Activities and Processes were implemented as designed; (2) dosage—the amount of the intervention received by participants; (3) quality—the extent to which a provider approached a theoretical ideal in terms of delivering program content; and (4) responsiveness—the extent to which participants are engaged by and involved in the activities and content of the program. We did not assess the fifth dimension—differentiation, which identifies unique elements of an intervention that are essential for success. When we have more complete outcome data, we will assess differentiation using component analysis in which we will empirically examine the relationship between specific program components and youth outcomes to determine if some are more or less essential to program efffects.21 We created instruments to measure each of the four dimensions and established empirical cutoffs for “high” program fidelity (see Table 4.1 for an overview of our measurement scheme). In creating the instruments, we attempted to balance measurement rigor (obtaining reliable, valid measures from multiple informants) with measurement feasibility (not overburdening instructors and study participants). To promote rigor, we chose to have at least two measures, from different informants, for most FOI dimensions of interest because different informants offer distinct perspectives on the quality and degree of program implementation. We also elected to develop an observational coding system as these are typically more highly correlated with program outcomes than instructor-reported data, which are prone to bias.22 We also attempted to maximize new directions for youth development • doi: 10.1002.yd
Quality
Session Inventory Form
Adherence: Core Processes
Not set
No. of lessons attended 0 = very negative to 4 = 3—Responded very positive positively 0 = poor to 4 = excellent 3—Good responsiveness Mean of seven items: 0 = 2—Most activities none to 3 = all covered Mean of fourteen items: 0 2—Most activities = none to 3 = all covered Mean of four items: 2—Most processes 0 = not at all effective delivered effectively to 3 = very effective Mean of four items: 0 = 2—Process behaviors not displayed at all to displayed most of the 3 = displayed time throughout session 0 = very ineffective to 4 3—Effective = very effective 0 = poor to 4 = excellent 3—Good
Not set
Criteria for “High Fidelity”
No. of lessons delivered
Measure
3.33–3.70 Not reliable
Not reliable
2.43–2.92
3.65
2.71
2.43–2.84
2.28–2.40
2.36 2.63
2.31–2.74
Not reliable
Not reliable 2.45
3.48–3.74
9.2–27.5
24–32
Range across Schools
3.55
15.7
27
Average across Schools
↑ Qualitative data from focus groups with select participants and teachers ↑
Session Inventory Form Observational Rating
Observational Rating
Session Inventory Form Observational Rating
Session Inventory Form Session Inventory Form Session Inventory Form Observational Rating
Data Source
Adherence: Core Activities
Responsiveness
Dosage
FOI Dimension
Table 4.1. Fidelity of implementation measurement scheme and results
ASSESSING FIDELITY OF CORE COMPONENTS
69
the feasibility of the FOI assessment system so that it could be used in future replications of the HLF program.23 For example, we switched from electronically administered instructor measures to hard copy format based on HLF instructor feedback that it was easier and more reliable for them to fill out paper-based forms they could carry with them to sessions. FOI measures. We assessed FOI using measures from three sources of implementation data: (1) Session Inventory Forms completed by HLF instructors, (2) Independent Observations of HLF sessions, and (3) Focus Groups of a select group of study participants and school personnel. HLF instructors filled out a Session Inventory Form at the end of every session. On these forms instructors marked each participant’s attendance, level of engagement and disruption, and homework completion. In addition, instructors indicated the extent to which they covered each of the seven Core Activities (one item for each activity) and executed the four Core Processes (one item for each process) during that particular session. Instructors also rated the overall quality of the sessions (one item) and in general how responsive participants were to the lesson (one item). We also videotaped fifteen sessions, evenly distributed across the three intervention schools, and developed an observational coding scheme to rate the extent to which instructors adhered to Core Activities and Core Processes as well as overall quality and student responsiveness. To do so, we first outlined the specific behaviors or activities that comprised HLF core components and created an observation rubric and training manual for two independent coders to code each of the videotaped sessions. The rubric was fairly straightforward for Core Activities because these were outlined in great detail in the HLF program manual. Core Processes, on the other hand, were more complicated because we had to outline the specific behaviors HLF instructors were supposed to engage in or “embody” when executing each process with competence.24 The program manual provided a few, but not many, of the specific core process behaviors. Therefore, we worked with the HLF developers to further operationalize behaviors that should be executed for new directions for youth development • doi: 10.1002.yd
70
MINDFULNESS IN ADOLESCENCE
each core process. Of note, the process of creating and executing the observational coding system, while too extensive to describe in full detail here, forced us to be more precise about our definitions and operationalization of program core components. For example, our first set of operational definitions for Core Activities was not precise enough to facilitate interrater reliability (IRR). Therefore, we created a more nuanced coding system in which coders rated whether instructors covered a particular Core Activity at all, as well as the exact proportion of content within each Core Activity to be covered in that lesson as outlined in the program manual. Similarly, observers rated instructors on the frequency of behaviors as well as their skill level for each of the four Core Processes. Finally, we conducted focus group interviews with a subset of fifteen participants and eleven teachers to gain their perspective on program implementation successes and challenges and core elements we may have missed. These focus groups were audio recorded, transcribed, and analyzed by thematic analysis by two coders. Establishing FOI benchmarks. Our next challenge was to establish criteria for implementing the HLF program “as intended,” or with high fidelity, for each dimension. Setting these benchmarks was inherently a part of our previous task of developing FOI measures and operationalizing related behaviors. Because setting specific thresholds for high program fidelity needs to be done a priori and is often overlooked in FOI construction, we made this a separate formal step.25 Specifically, the team defined numerical thresholds for “high” program fidelity for each dimension (outlined in Table 4.1) based on “most” of the program content being delivered with “good” quality and “positive” participant responsiveness. For this initial pass at assessing program fidelity, our FOI thresholds were more of a “best guess” than an established standard. Current and future studies will empirically inform appropriate thresholds. Step 3 (R): Run analyses and review implementation findings For the current study, we assessed the initial reliability and convergent validity of our FOI measures as well as the FOI of the HLF new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
71
program within and across intervention schools.26 To assess IRR— the degree of agreement between the two independent coders who rated each HLF instructor on video-recorded sessions—we calculated a two-way mixed, consistency, average-measures ICC. Our choice of ICC was based on the recommendations of two established guides on computing IRR for observational data.27 Based on outlined interpretive guidelines, ICC ranged from “excellent” for Core Activities (ICC = .95) to “good” for Core Processes (ICC = .69), to “poor” for overall quality and participant responsiveness scores (ICC = .15 and −.66, respectively). Both overall quality and participant responsiveness were single items, therefore it was not surprising that their reliability coefficients were fairly low.28 Given the poor interobserver reliability for overall impression of session quality, we created an observational rating of overall session quality using a composite score of discrete ratings for Core Activities and Core Processes.29 This composite score displayed excellent IRR (ICC = .94). We, however, dropped the observational participant responsiveness scale from the current FOI analyses. Such poor IRR was likely the result of our inability to videotape participants during sessions because parental consent would have been too difficult to obtain for such a purpose. Therefore, observers were unable to observe a key set of behaviors indicating participant responsiveness. To better understand how consistent instructor reports and coders were in their ratings of implementation integrity, we also examined interrater agreement (IRA) for Core Activities, Core Processes, student responsiveness, and overall session quality.30 Specifically, we examined correlational coefficients and ICCs on sessions for which we had obtained BOTH observational ratings and instructor ratings (N = 14 sessions). IRA was “good” for student responsiveness (r = .74; ICC = .67), “fair” for Core Activities and Core Processes (r = .33 and .56; ICC = .49 and .40, respectively), and “poor” for overall session quality (r = .08; ICC = .09). Consistent with other studies, instructors rated themselves more highly on adherence and quality than observational coders.31 new directions for youth development • doi: 10.1002.yd
72
MINDFULNESS IN ADOLESCENCE
Our next analyses were aimed at assessing overall FOI of the HLF program within and across intervention schools. We first examined the average program integrity for all schools in the study to give a sense of the extent of implementation fidelity associated with major experimental impact analyses. We also assessed FOI at each of the three intervention schools to examine the variation in implementation fidelity across intervention schools. Using instructor ratings from all 162 sessions and observer-ratings from the fifteen videotaped sessions, we ran descriptive statistics (means and standard deviations) to assess FOI. The right-hand side of Table 4.1 displays results of FOI across all schools. On average, HLF instructors delivered twenty-seven sessions of which students attended a mean of approximately sixteen sessions (58 percent of those offered). Instructors reported that participants were generally well engaged in program activities and content. According to both instructor and observer ratings, HLF instructors “mostly” to “fully” adhered to program Core Activities and Processes across sessions. Although instructors reported they delivered sessions with “good” to “excellent” quality, observers rated the quality of delivery from “neutral” to “good.” Results also indicate a fair amount of variation in program fidelity across school—particularly related to the dosage of the program. While one school offered a total of thirty-two sessions, of which students on average attended 27.5 sessions (86 percent of those offered), another school offered twenty-four sessions of which students on average attended 9.2 sessions (38 percent of those offered). The adherence, quality, and responsiveness were also somewhat lower at this school. Our focus group data and our communications with the instructor and school personnel highlighted multiple implementation challenges occurring at the school. The key challenge was a lack of administrative support for key program needs, including assistance getting participating students from their classes to the program room, provision of consistent appropriate space for holding program sessions, and adequate communication with the instructor regarding changes in academic scheduling and other barriers to holding sessions.32 We new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
73
will likely exclude this school from analyses evaluating the efficacy of the HLF program due to poor FOI. Ideally, we could classify schools into “low,” “medium,” or “high” FOI and examine related impacts on intervention participants relative to their control counterparts.33 The limited number of intervention schools in the current study, however, will not allow us to take such an approach.
Step 4 (E): Enhance and refine In the final step, we set aside time to review our initial approach for each previous step in order to enhance and refine our HLF program theory and FOI measurement strategy. As part of this learning process, we met with the HLF program developers and implementers again to gather their feedback on implementation successes and challenges and revisit the core components and aims of their program. We also met with the observational coders to obtain their input on the coding system and used focus group feedback to inform implementation successes and challenges. Below we outline our initial reflections from this final step. Enhancing program theory. The CORE cycle generally supported the program elements identified in the initial HLF logic model. Yet, several additional elements emerged as relevant during this final step. For instance, contextual factors related to delivering the HLF program in an urban school setting played a critical role in effective implementation. These contextual factors included gaining the buy-in and support of school administrative staff and teachers, obtaining appropriate space for program activities, and scheduling sessions so as not to compete with other popular school activities.34 We incorporated these contextual factors into our enhanced HLF logic model as core implementation components, as shown in the bottom portion of Figure 4.1. A number of related core implementation components have been identified as essential to the effective delivery of programs across multiple settings in other fields.35 In future studies, we will explicitly measure and evaluate these core implementation components as part of our FOI framework. new directions for youth development • doi: 10.1002.yd
74
MINDFULNESS IN ADOLESCENCE
Additionally, two new HLF program mechanisms surfaced as we reflected with program developers: Compassion for Self and Others, and Self-Empowerment. Program developers articulated the importance of modeling human compassion so that adolescents can begin to treat themselves and others with kindness and see the interconnectedness of all people. For example, instructors use the words “love” and “respect” in guided reflection activities and when explaining program concepts to participants. They reported that youth are often uncomfortable with these kinds of words initially but, as they experience what it is like to be treated with and treat others with compassion, they begin to use these words themselves. In addition, many of the HLF Core Activities and Processes are aimed not only at helping youth better regulate their thoughts and feelings but also at empowering them to recognize that they have choices in the way they feel, act, and behave. These additional mechanisms are now included in our enhanced logic model (bottom of Figure 4.1). Refining FOI measurement. Our system for collecting participant attendance and engagement through Session Inventory Forms worked well, in part because we had previously refined this process. The development of an observational coding scheme, while resource intensive, was critical to our FOI framework because this intensive process pushed us to further operationalize specific behaviors that reflected program integrity for each core component. In addition, the moderate correlations among instructor self-report and observational assessments of Core Activities, Processes, and Quality reinforced the need to obtain implementation integrity data from different sources and perspectives.36 Within the observational coding system itself, we were able to reliably assess HLF Core Activities but not Core Processes, particularly the instructor’s skill—something the mindfulness field at large is still wrestling with how to assess.37 Reliably assessing process dimensions of intervention fidelity is more difficult, although process dimensions are sometimes more correlated with youth outcomes.38 To increase IRR, we plan to refine the observational coding scheme to better operationalize the specific behaviors that reflect skilled new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
75
delivery of HLF Core Processes and revamp the related training of independent coders. We also intend to define more specifically with the program developers the cutoffs or behavioral criteria for implementing core components with “high fidelity” (when participants have received the full dose of the intervention) as well as “low fidelity” (when participants have received low or inadequate doses of the intervention). Establishing a priori criteria for fidelity is an important, complex, iterative process. The initial fidelity work described in this study will inform the selection of appropriate cutoffs for future work. Alternate implementation analyses. Our initial implementation analyses provided valuable information about the degree to which the HLF program was implemented across and within each of the intervention schools. Given multiple dimensions of FOI, multiple measures for each dimension, and multiple program delivery sites, there were a variety of ways in which implementation metrics could be calculated and presented. We chose to present the average level of fidelity across all lessons but will explore in future work the utility of other methods (for example, the number or proportion of sessions delivered with adequate fidelity). In addition, we might include more qualitative data from focus groups to effectively convey the complexity of quality implementation, as at least one other study on mindfulness in adolescents has done well.39 A critical next step for the current study will be to assess linkages between core components and participant outcomes in the intervention group, which will inform further refinements to the HLF program core components and program theory.40
Relevance of the CORE Process and future directions The CORE Process provides one framework for how researchers and practitioners can collaborate to better understand effective program components of adolescent mindfulness programs. The new directions for youth development • doi: 10.1002.yd
76
MINDFULNESS IN ADOLESCENCE
central focus of this iterative process is on collaborative learning. In so doing, the CORE Process can help practitioners and program developers become more cognizant of their own beliefs about how their program effects change and how both the components and the change processes should best be measured and tested. As core components become refined, instructors can be trained in how to be faithful to these essential program elements and allowed to make adaptations to less critical aspects based on the specific setting or population. Several aspects relevant to the CORE Process are important to consider carefully as this work progresses. First, research from other fields, including education, mental health, social services, and program evaluation, can inform both rigorous FOI measurement systems as well as the potential core components and contextual variables relevant for mindfulness-based programs.41 For example, school-based drug-abuse programs have a decades-long history of implementing and researching effective core program components. Some of the key ingredients of effective school-based druguse prevention programs include interactive learning approaches, using skilled instructors and older adolescent peer leaders, and higher intensity (sixteen or more hours).42 Many of the core components of adolescent mindfulness-based programs may overlap with effective components of other types of programming, and we encourage others to draw upon these rich sources of knowledge in their own work. On the other hand, certain core components may be unique to mindfulness interventions and require us to address different implementation challenges or forge new measures or methods. For example, we struggled with how to define and observe “quality” and “skill” of mindfulness instructors. Ideally, instructors of mindfulness-based programs embody the qualities of mindfulness and engage students in participatory learning in their own lives.43 Embodiment of such principles, however, is often difficult to operationalize and evaluate with observational rubrics. For instance, it is consistent with the nature of mindfulness to be “in the moment” and to respond flexibly to students at a given point in time, rather new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
77
than adhering rigidly to a manualized curriculum.44 This principle, however, creates a good deal of complexity when applied to the development of a reliable observational coding system. How does one observe and code when an instructor is responding mindfully “in the moment” versus when he or she may simply be deviating from the curriculum? Do observational coders require some grounding in their own mindfulness practice to accurately observe these behaviors? Indeed some studies in educational field have evidenced a positive association between raters’ expertise level and rating accuracy.45 These are important issues for researchers and practitioners to grapple with both for promoting high-quality instructor training and also for assessing fidelity and quality of program delivery.46 Second, we need to articulate and assess both core program components and core implementation components. Core implementation components are factors that facilitate FOI when programs are implemented in less controlled settings, like schools. The challenges of implementing mindfulness programs in school settings have been documented and the negative effects of poor implementation on program efficacy are increasingly appreciated.47 Therefore, within a single trial, it will be important to conceptualize and test both core program components—that when implemented with high fidelity, regardless of setting, account for program impacts as well as core implementation components; that when present facilitate higher levels of program fidelity within a particular setting. Finally, we need to empirically test the relationship between hypothesized core components and key adolescent outcomes. We are in the process of conducting these analyses for the current study and intend to share these results in the future. Our analyses will be correlational in nature and will provide a starting point for identifying effective core components. In larger studies, researchers can use more rigorous tests such as mediational analyses or dismantling strategies to isolate potential active ingredients. Such increased rigor will be a natural outgrowth of several iterations of the CORE new directions for youth development • doi: 10.1002.yd
78
MINDFULNESS IN ADOLESCENCE
Process. Indeed, if researchers and practitioners work together using the CORE Process, over time and across trials, we can build a more coherent theory of the common program elements and contextual variables necessary for mindfulness programs to positively influence adolescent lives.
Notes 1. Black, D. S., Milam, J., & Sussman, S. (2009). Sitting-meditation interventions among youth: A review of treatment efficacy. Pediatrics, 124, 532–541; Greenberg, M. T., & Harris, A. R. (2011). Nurturing mindfulness in children and youth: Current state of research. Child Development Perspectives, 6(2), 161– 166. 2. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication No. 231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. P. 24. 3. Blas´e, K., & Fixsen, D. (2013). Core intervention components: Identifying and operationalizing what makes programs work (ASPE Research Brief). Washington, DC: OHSP, US Department of Health and Human Services. Retrieved from http://aspe.hhs.gov/hsp/13/KeyIssuesforChildrenYouth /CoreIntervention/rb CoreIntervention.cfm 4. Fixsen et al. (2005); Kutash, K., Cross, B., Madias, A., Duchonowski, A. J., & Green, A. L. (2012). Description of a fidelity implementation system: An example from a community-based children’s mental health program. Journal of Child and Family Studies, 21, 1028–1040; Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340. 5. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45; Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221. 6. Fixsen et al. (2005); Mowbray et al. (2003); O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78(1), 33–84. 7. Greenberg & Harris. (2011); Meiklejohn, J., Phillips, C., Freedman, M. L., Griffin, M. L., Biegel, G., Roach, A., . . . Saltzman, A. (2012). Integrating mindfulness training into K-12 education: Fostering the resilience of teachers and students. Mindfulness, 3(4), 291–307; Davidson, R. J., & Mind and Life Education Research Network. (2012). Contemplative practices and new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
79
mental training: Prospects for American education. Child Development Perspectives, 6(2), 146–153. 8. Greenberg & Harris. (2011). 9. Meiklejohn et al. (2012). 10. Deming, W. E. (1986). Out of crisis. Cambridge: MIT Press. 11. Holistic Life Foundation. (2013, August 22). Personal communication. 12. Mendelson, T., Greenberg, M. T., Dariotis, J. K., Feagans Gould, L., Rhoades, B. L., & Leaf, P. J. (2010). Feasibility and preliminary outcomes of a mindfulness intervention for urban youth. Journal of Abnormal Child Psychology, 38, 985–994; Blair, C., & Diamond, A. (2008). Biological processes in prevention and intervention: The promotion of self-regulation as a means of preventing school failure. Development and Psychopathology, 20, 899–911; Greenberg, M. T. (2006). Promoting resilience in children and youth: Preventive interventions and their interface with neuroscience. Annals of the New York Academy of Science, 1094, 139–150. 13. Mendelson et al. (2010). 14. Durlak, J. A. (1998). Why program implementation is important. Journal of Prevention & Intervention in the Community, 17, 5–18. 15. Leithwood, K. A., & Montgomery, D. J. (1980). Evaluating program implementation. Evaluation Review, 4, 193–214. 16. Mowbray et al. (2003); Century, J., Rudnick, M., & Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31(2), 199–218. 17. Mowbray et al. (2003); Century et al. (2010). 18. Kabat-Zinn, J. (2011). Some reflections on the origins of MBSR, skillful means, and the trouble with maps. Contemporary Buddhism, 12(1), 281–306; Crane, R. S., Kuyken, W., Williams, J. M. G., Hastings, R. P., Cooper, L., & Fennell, M. J. V. (2011). Competence in teaching mindfulness-based courses: Concepts, development, and assessment. Mindfulness, 3, 76–84; Cullen, M. (2011). Mindfulness-based interventions: An emerging phenomenon. Mindfulness, 2, 186–193. 19. Domitrovich & Greenberg. (2000); Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350; Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research Theory & Practice, 18(2), 237–256. 20. Kutash et al. (2012). 21. Kutash et al. (2012); Dusenbury et al. (2003). 22. Dane & Schneider. (1998). 23. Century et al. (2010). 24. Kabat-Zinn. (2011); Crane et al. (2011); Cullen. (2011). 25. O’Donnell. (2008). 26. Mowbray et al. (2003). new directions for youth development • doi: 10.1002.yd
80
MINDFULNESS IN ADOLESCENCE
27. Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34; McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. 28. Hallgren. (2012). 29. Thorndike, R. M., Cunningham, G. K., Thorndike, R. L., & Hagen, E. P. (1991). Measurement and evaluation in psychology and education. New York, NY: Macmillan Publishing Company. 30. Mowbray et al. (2003). 31. Dane & Schneider. (1998). 32. Mendelson, T., Dariotis, J. K., Feagans Gould, L., Smith, A. S. R., Smith, A. A., Gonzalez, A. A., & Greenberg, M. T. (2013). Implementing mindfulness and yoga in urban schools: A community-academic partnership. Journal of Children’s Services, 8(4), 276–291. 33. Mowbray et al. (2003). 34. Mendelson et al. (2013). 35. Domitrovich & Greenberg. (2000); Fixsen, D. L., Blas´e, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19, 531–540. 36. Mowbray et al. (2003); O’Donnell. (2008). 37. Crane et al. (2011); Cullen. (2011). 38. Mowbray et al. (2003). 39. Schonert-Reichl, K. A., & Lawlor, M. S. (2010). The effects of a mindfulness-based education program on pre- and early adolescents’ wellbeing and social and emotional competence. Mindfulness, 1(3), 137–151. 40. Blas´e & Fixsen. (2013) 41. Mowbray et al. (2003); Domitrovich & Greenberg. (2000); O’Donnell. (2008); Dusenbury et al. (2003). 42. Gottfredson, D. C., & Wilson, D. B. (2003). Characteristics of effective school-based substance abuse prevention. Prevention Science, 4, 27–38. 43. Crane et al. (2011). 44. Kabat-Zinn. (2011). 45. Graham, M., Milanowski, A., & Miller, J. (2012). Measuring and promoting inter-rater agreement of teacher and principal performance ratings. Center for Educator Compensation Reform. Retrieved from http://cecr.ed .gov/pdfs/Inter Rater.pdf 46. Crane et al. (2011). 47. Meiklejohn et al. (2012); Mendelson et al. (2013).
laura feagans gould is a research scientist at the University of North Carolina at Greensboro SERVE Center. tamar mendelson is an assistant professor in the Department of Mental Health at the Johns Hopkins Bloomberg School of Public Health. new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
81
jacinda k. dariotis is an assistant professor in population, family, and reproductive health at the Johns Hopkins Bloomberg School of Public Health. matthew ancona is a graduate student in industrial and organizational psychology at the University of Baltimore. ali s. r. smith, andres a. gonzalez, and atman a. smith are cofounders of the Holistic Life Foundation, Inc. mark t. greenberg is the Edna Peterson Bennett Endowed Chair in Prevention Research, founding director of the Prevention Research Center for the Promotion of Human Development, and professor of human development and psychology in the College of Health and Human Development at the Pennsylvania State University.
new directions for youth development • doi: 10.1002.yd
4 Assessing fidelity of core components in a mindfulness and yoga intervention for urban youth: Applying the CORE Process Laura Feagans Gould, Tamar Mendelson, Jacinda K. Dariotis, Matthew Ancona, Ali S. R. Smith, Andres A. Gonzalez, Atman A. Smith, Mark T. Greenberg a growing number of studies are documenting both the feasibility and potential benefits of mindfulness-based programs for youth and adolescents.1 As these programs continue to gain in popularity, it becomes increasingly important to understand which aspects or components of these programs may be helpful for adolescents, and how those components should be delivered so as to have the most beneficial impact. In this chapter, we describe our team’s approach to identifying—and beginning to evaluate and refine—the core components of a mindfulness and yoga program for urban early adolescents.
NEW DIRECTIONS FOR YOUTH DEVELOPMENT, NO. 142, SUMMER 2014 © 2014 WILEY PERIODICALS, INC. Published online in Wiley Online Library (wileyonlinelibrary.com) • DOI: 10.1002/yd.20097
59
60
MINDFULNESS IN ADOLESCENCE
Program core components Program core components are “the most essential and indispensable components of an intervention practice or program.”2 They are essentially the “active ingredients” necessary to produce desired outcomes and are directly related to a program’s theory of change, which proposes the mechanisms by which an intervention works.3 Identifying the core components of a particular program improves our understanding of how individual elements of that program, or a combination of those elements, work together to impact intended outcomes. It also helps ascertain which aspects are essential to be delivered as specified in a program manual, and which elements can be adapted across setting or population.4 Over time, identifying core components of evidence-based programs can help us collectively identify evidence-based practices or the specific activities that work across programs in order to produce intended outcomes. Identification of a program’s core components is often a complex and long-term undertaking in which resources are funneled into systematically developing and studying a program’s fidelity of implementation (FOI)—the extent to which intervention delivery adheres to the protocol or program model as intended by the intervention developers.5 Through the systematic study of FOI, researchers and program developers (1) dismantle a program into its constituent parts and integrate them into a coherent theory of change, (2) develop reliable and valid measures of FOI and establish criteria for implementation integrity, and (3) empirically examine whether variation in the implementation of core components is systematically related to particular outcomes across replication trials. Over multiple trials, this systematic study of the efficacy of program elements yields a more refined theory of change, which specifies the integral (core components) and nonessential aspects of a program.6
new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
61
Core components of mindfulness programs for adolescents Discussion of core program elements and mechanisms through which mindfulness programs may impact early adolescent wellbeing is in its infancy, as is much of the research in the field. Several recent reviews of the preliminary impacts of mindfulnessbased programs for youth and adolescents across a range of settings and target populations have been helpful in outlining some of the potential core components or “underlying principles” of these types of programs, as well as some general mechanisms through which mindfulness programs may impact adolescent outcomes.7 For example, mindfulness programs generally include structured activities that require sustained focused attention on an object, the breath, or a sound with a common goal of sharpening concentration or attention, improving emotion regulation skills, and enhancing self-awareness and knowledge.8 The specific techniques and approaches (for example, yoga, sitting meditation) used to focus attention, however, vary widely as do program lengths, settings, and target populations.9 Almost all of these reviews highlight the need for researchers and program evaluators to more rigorously assess the range of potential active ingredients of mindfulness programs for youth and adolescents. Indeed, in a brief review of eighteen studies published from 2005 to 2013 on mindfulness programs for youth and adolescents, we found that the majority of studies included a basic description of the program, espoused general underlying mindfulness practices or principles that guided program development, and spoke generally about program intended outcomes. Rarely, however, were core program components explicitly outlined and integrated into a formal theory of change (for example, logic model). Approximately half of the studies assessed at least one aspect of FOI, namely, dosage or the amount of the intervention received by participants. Nonetheless, there was an absence of the rigorous study of FOI through the development of reliable, valid measures; the inclusion new directions for youth development • doi: 10.1002.yd
62
MINDFULNESS IN ADOLESCENCE
of multiple sources of data for FOI constructs; and the establishment of a priori criteria for program fidelity. Thus, we believe that a discussion of the development of FOI measures and examination of program core components is a timely topic with relevance for the emerging field of mindfulness programs for youth and adolescents. This chapter provides a practical framework for mindfulness researchers and program developers to draw upon in identifying and rigorously assessing FOI of program core components. Specifically, this chapter outlines the process our team undertook in a recent study in which researchers worked with program developers to identify and measure FOI of hypothesized core components of a yoga and mindfulness-based program for youth. This process represents our initial attempt to engage in a learning cycle based on the Plan, Do, Study, Act (PDSA) cycle.10 We adapted the steps in the PDSA cycle to address the development, testing, and refinement of program core components. We call our adapted cycle the CORE Process: (C) Conceptualize core components, (O) Operationalize and measure, (R) Run analyses and review implementation findings, and (E) Enhance and refine. In this chapter, we describe the CORE Process, provide illustrative examples at each stage of the cycle, and discuss lessons learned as well as broader implications for mindfulness research with adolescents.
A community-academic partnership Our team represents a partnership between the founders of the Holistic Life Foundation, Inc. (HLF)—a nonprofit organization dedicated to promoting health and well-being in underserved communities through mindfulness-based practices (www.hlfinc.org)— and a group of university-based prevention researchers interested in understanding how mindfulness-based programs impact urban youth. The three HLF founders were born and raised in Baltimore City and work with urban youth and adolescents, particularly those new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
63
who face chronic stressors associated with poverty, violent neighborhoods, and lack of adequate environmental supports. The HLF founders conceptualize mindfulness as “the combination of awareness, centering, and being present. That is, mindfulness is awareness of your thoughts, emotions, actions, and energy; the ability to get centered and stay centered in all situations; and the ability to be present, not letting internal and external distractions take you from the current moment.”11 HLF’s program teaches mindfulness through yoga-based movement, breathing techniques, and attention training. The yoga-based postures offer a means of practicing mindfulness that is focused on body movements and breath. Sessions are led by an experienced instructor and assistant, and are conducted in spaces large enough to accommodate yoga mats for each child. Each session begins with several minutes of quiet breathing to settle in, then progresses to active yoga-based movements followed by breath work. Sessions end with several minutes of “guided reflection” during which the students lie on their mats and the instructor guides them in an attention training practice, such as following the breath, sending positive energy to others, or imagining being calm while taking a test. Throughout the sessions, the instructors tell the students about health-related benefits of the practices they are learning. They also engage the students in discussions about what stress is, how to recognize it in the body, and how to use breathing and poses to stay calm during times of stress. In 2006, we (the researchers) partnered with HLF (the program developers) to conduct a pilot randomized trial of their schoolbased program with fourth- and fifth-grade students across four Baltimore City schools. During the pilot trial, we worked with the HLF team to develop an initial logic model drawing on their programmatic goals and the empirical literature on the effects of mindfulness and yoga.12 In this initial logic model, the team proposed that the HLF intervention would lead to enhancements in self-regulation (for example, the ability to modulate thoughts and emotions) which would, in turn, promote improvements in social and emotional functioning (for example, mood, behavior). new directions for youth development • doi: 10.1002.yd
64
MINDFULNESS IN ADOLESCENCE
Findings from the trial generally supported this basic logic model: Students who participated in the HLF program reported significant improvements in their capacity to regulate thoughts and emotions in response to stress as compared with control group students, including fewer intrusive thoughts, less rumination, and lower levels of emotional arousal.13 Findings also indicated that the program was feasible and acceptable to students, teachers, and administrators. We are currently conducting a larger randomized controlled trial in which the HLF school-based program was delivered in three urban elementary schools to fifth- and sixth-grade students. As part of this trial, we committed a portion of our resources to identifying program core components, developing FOI measures, and assessing FOI of the HLF program. The CORE Process served as our framework for this work and is described in more detail below. Step 1 (C): Conceptualize core components Our first undertaking was to enhance program theory and identify the core components or potentially active ingredients of the HLF program.14 We drew upon several sources of information during this process, including written materials produced by the program developers, conversations with the program developers, and experiences from our pilot study.15 The HLF school-based program manual written during our pilot trial served as the starting point for our process of identifying core components because it specified all the activities in the program in the sequence they were to be delivered. To select and categorize core components, we reread the manual and engaged in a series of conversations with the HLF program developers about the central goals of the program, the structure of the manual and intervention, the rationale behind inclusion of specific activities and practices, and the intended benefit or result of different program activities for youth. Through these conversations, the team identified two types of HLF core program components: Core Activities and Core Processes. Core Activities are the activities and practices described in new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
65
the curriculum, consistent with what others have laid out as structural program elements.16 HLF Core Activities include: (1) breathing to settle in at the start of each session, (2) practicing yoga poses, (3) practicing breath work, (4) reviewing previous material (that is, previously taught poses and breathing techniques), (5) discussing benefits and relevance of the intervention activities for health and well-being, (6) engaging in a silent reflection involving focused attention, and (7) assigning homework to encourage use of the intervention practices outside the program sessions. Core Processes, however, address the manner in which the instructors engage with youth and the way in which the program is delivered.17 Our conversations with the program developers and observation of HLF sessions during our pilot study highlighted that these Core Processes are essential and sometimes distinguishing aspects of mindfulness instruction, calling on instructors to embody the qualities of mindfulness as a vehicle for teaching it.18 In effect, the HLF instructors model for the youth guides on how to interact mindfully; staying focused on a chosen target; and treating themselves and others with respect, dignity, and compassion. HLF Core Processes include: (1) engaging students in learning through applying material to their own lives, (2) maintaining order and discipline by redirecting negative behaviors and reinforcing on-task behavior, (3) modeling human compassion by treating themselves and others kindly and empathically, and (4) reviewing concepts and poses throughout session activities to reinforce experiential learning and mastery of material. These seven Core Activities and four Core Processes are aimed at helping students be present and regulate their thoughts, feelings, and behaviors. This in-depth examination of program components highlighted for us the extent to which HLF core program activities and processes are developmentally tailored for early adolescents. For example, when engaging students in learning, the instructors use simple and straightforward language, and they reference popular youth culture—including television shows, music, and celebrities—to make their points relevant for students. Instructors also build mastery of the material by encouraging new directions for youth development • doi: 10.1002.yd
66
MINDFULNESS IN ADOLESCENCE
Figure 4.1. Holistic Life Foundation (HLF) initial and revised program theory
different students to model poses and lead class exercises. They increase the teaching role and autonomy of student leaders over the course of the program so that students gain increasing confidence in their own abilities, an important developmental process for adolescents. With respect to Core Activities, more active movements are generally taught in the earlier portion of the sessions, so as to expend students’ restless energy and facilitate the closing period of calm reflection. Despite our previous familiarity with the HLF program and theory of change, identifying core program components was nevertheless a relatively lengthy process involving discussion with the HLF founders as well as detailed review of their curriculum. The resulting refined program theory served as the foundation for our FOI measurement strategy in Step 2. At the end of this first step, we had developed: (1) a set of clearly articulated HLF core program components and (2) a more fully articulated logic model that broke apart the HLF program into these constituent components, as outlined in bold, in the top portion of Figure 4.1. new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
67
Step 2 (O): Operationalize and measure Once the team had articulated the program core components, we addressed the challenge of operationalizing and measuring program FOI. As a foundation for this work, we used the five major dimensions of FOI identified by leaders in the field of implementation science: adherence, dosage, quality, responsiveness, and differentiation.19 This multidimensional approach provides a nuanced representation of fidelity.20 For the current study, we assessed four of the five dimensions of FOI: (1) adherence—the extent to which the Core Activities and Processes were implemented as designed; (2) dosage—the amount of the intervention received by participants; (3) quality—the extent to which a provider approached a theoretical ideal in terms of delivering program content; and (4) responsiveness—the extent to which participants are engaged by and involved in the activities and content of the program. We did not assess the fifth dimension—differentiation, which identifies unique elements of an intervention that are essential for success. When we have more complete outcome data, we will assess differentiation using component analysis in which we will empirically examine the relationship between specific program components and youth outcomes to determine if some are more or less essential to program efffects.21 We created instruments to measure each of the four dimensions and established empirical cutoffs for “high” program fidelity (see Table 4.1 for an overview of our measurement scheme). In creating the instruments, we attempted to balance measurement rigor (obtaining reliable, valid measures from multiple informants) with measurement feasibility (not overburdening instructors and study participants). To promote rigor, we chose to have at least two measures, from different informants, for most FOI dimensions of interest because different informants offer distinct perspectives on the quality and degree of program implementation. We also elected to develop an observational coding system as these are typically more highly correlated with program outcomes than instructor-reported data, which are prone to bias.22 We also attempted to maximize new directions for youth development • doi: 10.1002.yd
Quality
Session Inventory Form
Adherence: Core Processes
Not set
No. of lessons attended 0 = very negative to 4 = 3—Responded very positive positively 0 = poor to 4 = excellent 3—Good responsiveness Mean of seven items: 0 = 2—Most activities none to 3 = all covered Mean of fourteen items: 0 2—Most activities = none to 3 = all covered Mean of four items: 2—Most processes 0 = not at all effective delivered effectively to 3 = very effective Mean of four items: 0 = 2—Process behaviors not displayed at all to displayed most of the 3 = displayed time throughout session 0 = very ineffective to 4 3—Effective = very effective 0 = poor to 4 = excellent 3—Good
Not set
Criteria for “High Fidelity”
No. of lessons delivered
Measure
3.33–3.70 Not reliable
Not reliable
2.43–2.92
3.65
2.71
2.43–2.84
2.28–2.40
2.36 2.63
2.31–2.74
Not reliable
Not reliable 2.45
3.48–3.74
9.2–27.5
24–32
Range across Schools
3.55
15.7
27
Average across Schools
↑ Qualitative data from focus groups with select participants and teachers ↑
Session Inventory Form Observational Rating
Observational Rating
Session Inventory Form Observational Rating
Session Inventory Form Session Inventory Form Session Inventory Form Observational Rating
Data Source
Adherence: Core Activities
Responsiveness
Dosage
FOI Dimension
Table 4.1. Fidelity of implementation measurement scheme and results
ASSESSING FIDELITY OF CORE COMPONENTS
69
the feasibility of the FOI assessment system so that it could be used in future replications of the HLF program.23 For example, we switched from electronically administered instructor measures to hard copy format based on HLF instructor feedback that it was easier and more reliable for them to fill out paper-based forms they could carry with them to sessions. FOI measures. We assessed FOI using measures from three sources of implementation data: (1) Session Inventory Forms completed by HLF instructors, (2) Independent Observations of HLF sessions, and (3) Focus Groups of a select group of study participants and school personnel. HLF instructors filled out a Session Inventory Form at the end of every session. On these forms instructors marked each participant’s attendance, level of engagement and disruption, and homework completion. In addition, instructors indicated the extent to which they covered each of the seven Core Activities (one item for each activity) and executed the four Core Processes (one item for each process) during that particular session. Instructors also rated the overall quality of the sessions (one item) and in general how responsive participants were to the lesson (one item). We also videotaped fifteen sessions, evenly distributed across the three intervention schools, and developed an observational coding scheme to rate the extent to which instructors adhered to Core Activities and Core Processes as well as overall quality and student responsiveness. To do so, we first outlined the specific behaviors or activities that comprised HLF core components and created an observation rubric and training manual for two independent coders to code each of the videotaped sessions. The rubric was fairly straightforward for Core Activities because these were outlined in great detail in the HLF program manual. Core Processes, on the other hand, were more complicated because we had to outline the specific behaviors HLF instructors were supposed to engage in or “embody” when executing each process with competence.24 The program manual provided a few, but not many, of the specific core process behaviors. Therefore, we worked with the HLF developers to further operationalize behaviors that should be executed for new directions for youth development • doi: 10.1002.yd
70
MINDFULNESS IN ADOLESCENCE
each core process. Of note, the process of creating and executing the observational coding system, while too extensive to describe in full detail here, forced us to be more precise about our definitions and operationalization of program core components. For example, our first set of operational definitions for Core Activities was not precise enough to facilitate interrater reliability (IRR). Therefore, we created a more nuanced coding system in which coders rated whether instructors covered a particular Core Activity at all, as well as the exact proportion of content within each Core Activity to be covered in that lesson as outlined in the program manual. Similarly, observers rated instructors on the frequency of behaviors as well as their skill level for each of the four Core Processes. Finally, we conducted focus group interviews with a subset of fifteen participants and eleven teachers to gain their perspective on program implementation successes and challenges and core elements we may have missed. These focus groups were audio recorded, transcribed, and analyzed by thematic analysis by two coders. Establishing FOI benchmarks. Our next challenge was to establish criteria for implementing the HLF program “as intended,” or with high fidelity, for each dimension. Setting these benchmarks was inherently a part of our previous task of developing FOI measures and operationalizing related behaviors. Because setting specific thresholds for high program fidelity needs to be done a priori and is often overlooked in FOI construction, we made this a separate formal step.25 Specifically, the team defined numerical thresholds for “high” program fidelity for each dimension (outlined in Table 4.1) based on “most” of the program content being delivered with “good” quality and “positive” participant responsiveness. For this initial pass at assessing program fidelity, our FOI thresholds were more of a “best guess” than an established standard. Current and future studies will empirically inform appropriate thresholds. Step 3 (R): Run analyses and review implementation findings For the current study, we assessed the initial reliability and convergent validity of our FOI measures as well as the FOI of the HLF new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
71
program within and across intervention schools.26 To assess IRR— the degree of agreement between the two independent coders who rated each HLF instructor on video-recorded sessions—we calculated a two-way mixed, consistency, average-measures ICC. Our choice of ICC was based on the recommendations of two established guides on computing IRR for observational data.27 Based on outlined interpretive guidelines, ICC ranged from “excellent” for Core Activities (ICC = .95) to “good” for Core Processes (ICC = .69), to “poor” for overall quality and participant responsiveness scores (ICC = .15 and −.66, respectively). Both overall quality and participant responsiveness were single items, therefore it was not surprising that their reliability coefficients were fairly low.28 Given the poor interobserver reliability for overall impression of session quality, we created an observational rating of overall session quality using a composite score of discrete ratings for Core Activities and Core Processes.29 This composite score displayed excellent IRR (ICC = .94). We, however, dropped the observational participant responsiveness scale from the current FOI analyses. Such poor IRR was likely the result of our inability to videotape participants during sessions because parental consent would have been too difficult to obtain for such a purpose. Therefore, observers were unable to observe a key set of behaviors indicating participant responsiveness. To better understand how consistent instructor reports and coders were in their ratings of implementation integrity, we also examined interrater agreement (IRA) for Core Activities, Core Processes, student responsiveness, and overall session quality.30 Specifically, we examined correlational coefficients and ICCs on sessions for which we had obtained BOTH observational ratings and instructor ratings (N = 14 sessions). IRA was “good” for student responsiveness (r = .74; ICC = .67), “fair” for Core Activities and Core Processes (r = .33 and .56; ICC = .49 and .40, respectively), and “poor” for overall session quality (r = .08; ICC = .09). Consistent with other studies, instructors rated themselves more highly on adherence and quality than observational coders.31 new directions for youth development • doi: 10.1002.yd
72
MINDFULNESS IN ADOLESCENCE
Our next analyses were aimed at assessing overall FOI of the HLF program within and across intervention schools. We first examined the average program integrity for all schools in the study to give a sense of the extent of implementation fidelity associated with major experimental impact analyses. We also assessed FOI at each of the three intervention schools to examine the variation in implementation fidelity across intervention schools. Using instructor ratings from all 162 sessions and observer-ratings from the fifteen videotaped sessions, we ran descriptive statistics (means and standard deviations) to assess FOI. The right-hand side of Table 4.1 displays results of FOI across all schools. On average, HLF instructors delivered twenty-seven sessions of which students attended a mean of approximately sixteen sessions (58 percent of those offered). Instructors reported that participants were generally well engaged in program activities and content. According to both instructor and observer ratings, HLF instructors “mostly” to “fully” adhered to program Core Activities and Processes across sessions. Although instructors reported they delivered sessions with “good” to “excellent” quality, observers rated the quality of delivery from “neutral” to “good.” Results also indicate a fair amount of variation in program fidelity across school—particularly related to the dosage of the program. While one school offered a total of thirty-two sessions, of which students on average attended 27.5 sessions (86 percent of those offered), another school offered twenty-four sessions of which students on average attended 9.2 sessions (38 percent of those offered). The adherence, quality, and responsiveness were also somewhat lower at this school. Our focus group data and our communications with the instructor and school personnel highlighted multiple implementation challenges occurring at the school. The key challenge was a lack of administrative support for key program needs, including assistance getting participating students from their classes to the program room, provision of consistent appropriate space for holding program sessions, and adequate communication with the instructor regarding changes in academic scheduling and other barriers to holding sessions.32 We new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
73
will likely exclude this school from analyses evaluating the efficacy of the HLF program due to poor FOI. Ideally, we could classify schools into “low,” “medium,” or “high” FOI and examine related impacts on intervention participants relative to their control counterparts.33 The limited number of intervention schools in the current study, however, will not allow us to take such an approach.
Step 4 (E): Enhance and refine In the final step, we set aside time to review our initial approach for each previous step in order to enhance and refine our HLF program theory and FOI measurement strategy. As part of this learning process, we met with the HLF program developers and implementers again to gather their feedback on implementation successes and challenges and revisit the core components and aims of their program. We also met with the observational coders to obtain their input on the coding system and used focus group feedback to inform implementation successes and challenges. Below we outline our initial reflections from this final step. Enhancing program theory. The CORE cycle generally supported the program elements identified in the initial HLF logic model. Yet, several additional elements emerged as relevant during this final step. For instance, contextual factors related to delivering the HLF program in an urban school setting played a critical role in effective implementation. These contextual factors included gaining the buy-in and support of school administrative staff and teachers, obtaining appropriate space for program activities, and scheduling sessions so as not to compete with other popular school activities.34 We incorporated these contextual factors into our enhanced HLF logic model as core implementation components, as shown in the bottom portion of Figure 4.1. A number of related core implementation components have been identified as essential to the effective delivery of programs across multiple settings in other fields.35 In future studies, we will explicitly measure and evaluate these core implementation components as part of our FOI framework. new directions for youth development • doi: 10.1002.yd
74
MINDFULNESS IN ADOLESCENCE
Additionally, two new HLF program mechanisms surfaced as we reflected with program developers: Compassion for Self and Others, and Self-Empowerment. Program developers articulated the importance of modeling human compassion so that adolescents can begin to treat themselves and others with kindness and see the interconnectedness of all people. For example, instructors use the words “love” and “respect” in guided reflection activities and when explaining program concepts to participants. They reported that youth are often uncomfortable with these kinds of words initially but, as they experience what it is like to be treated with and treat others with compassion, they begin to use these words themselves. In addition, many of the HLF Core Activities and Processes are aimed not only at helping youth better regulate their thoughts and feelings but also at empowering them to recognize that they have choices in the way they feel, act, and behave. These additional mechanisms are now included in our enhanced logic model (bottom of Figure 4.1). Refining FOI measurement. Our system for collecting participant attendance and engagement through Session Inventory Forms worked well, in part because we had previously refined this process. The development of an observational coding scheme, while resource intensive, was critical to our FOI framework because this intensive process pushed us to further operationalize specific behaviors that reflected program integrity for each core component. In addition, the moderate correlations among instructor self-report and observational assessments of Core Activities, Processes, and Quality reinforced the need to obtain implementation integrity data from different sources and perspectives.36 Within the observational coding system itself, we were able to reliably assess HLF Core Activities but not Core Processes, particularly the instructor’s skill—something the mindfulness field at large is still wrestling with how to assess.37 Reliably assessing process dimensions of intervention fidelity is more difficult, although process dimensions are sometimes more correlated with youth outcomes.38 To increase IRR, we plan to refine the observational coding scheme to better operationalize the specific behaviors that reflect skilled new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
75
delivery of HLF Core Processes and revamp the related training of independent coders. We also intend to define more specifically with the program developers the cutoffs or behavioral criteria for implementing core components with “high fidelity” (when participants have received the full dose of the intervention) as well as “low fidelity” (when participants have received low or inadequate doses of the intervention). Establishing a priori criteria for fidelity is an important, complex, iterative process. The initial fidelity work described in this study will inform the selection of appropriate cutoffs for future work. Alternate implementation analyses. Our initial implementation analyses provided valuable information about the degree to which the HLF program was implemented across and within each of the intervention schools. Given multiple dimensions of FOI, multiple measures for each dimension, and multiple program delivery sites, there were a variety of ways in which implementation metrics could be calculated and presented. We chose to present the average level of fidelity across all lessons but will explore in future work the utility of other methods (for example, the number or proportion of sessions delivered with adequate fidelity). In addition, we might include more qualitative data from focus groups to effectively convey the complexity of quality implementation, as at least one other study on mindfulness in adolescents has done well.39 A critical next step for the current study will be to assess linkages between core components and participant outcomes in the intervention group, which will inform further refinements to the HLF program core components and program theory.40
Relevance of the CORE Process and future directions The CORE Process provides one framework for how researchers and practitioners can collaborate to better understand effective program components of adolescent mindfulness programs. The new directions for youth development • doi: 10.1002.yd
76
MINDFULNESS IN ADOLESCENCE
central focus of this iterative process is on collaborative learning. In so doing, the CORE Process can help practitioners and program developers become more cognizant of their own beliefs about how their program effects change and how both the components and the change processes should best be measured and tested. As core components become refined, instructors can be trained in how to be faithful to these essential program elements and allowed to make adaptations to less critical aspects based on the specific setting or population. Several aspects relevant to the CORE Process are important to consider carefully as this work progresses. First, research from other fields, including education, mental health, social services, and program evaluation, can inform both rigorous FOI measurement systems as well as the potential core components and contextual variables relevant for mindfulness-based programs.41 For example, school-based drug-abuse programs have a decades-long history of implementing and researching effective core program components. Some of the key ingredients of effective school-based druguse prevention programs include interactive learning approaches, using skilled instructors and older adolescent peer leaders, and higher intensity (sixteen or more hours).42 Many of the core components of adolescent mindfulness-based programs may overlap with effective components of other types of programming, and we encourage others to draw upon these rich sources of knowledge in their own work. On the other hand, certain core components may be unique to mindfulness interventions and require us to address different implementation challenges or forge new measures or methods. For example, we struggled with how to define and observe “quality” and “skill” of mindfulness instructors. Ideally, instructors of mindfulness-based programs embody the qualities of mindfulness and engage students in participatory learning in their own lives.43 Embodiment of such principles, however, is often difficult to operationalize and evaluate with observational rubrics. For instance, it is consistent with the nature of mindfulness to be “in the moment” and to respond flexibly to students at a given point in time, rather new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
77
than adhering rigidly to a manualized curriculum.44 This principle, however, creates a good deal of complexity when applied to the development of a reliable observational coding system. How does one observe and code when an instructor is responding mindfully “in the moment” versus when he or she may simply be deviating from the curriculum? Do observational coders require some grounding in their own mindfulness practice to accurately observe these behaviors? Indeed some studies in educational field have evidenced a positive association between raters’ expertise level and rating accuracy.45 These are important issues for researchers and practitioners to grapple with both for promoting high-quality instructor training and also for assessing fidelity and quality of program delivery.46 Second, we need to articulate and assess both core program components and core implementation components. Core implementation components are factors that facilitate FOI when programs are implemented in less controlled settings, like schools. The challenges of implementing mindfulness programs in school settings have been documented and the negative effects of poor implementation on program efficacy are increasingly appreciated.47 Therefore, within a single trial, it will be important to conceptualize and test both core program components—that when implemented with high fidelity, regardless of setting, account for program impacts as well as core implementation components; that when present facilitate higher levels of program fidelity within a particular setting. Finally, we need to empirically test the relationship between hypothesized core components and key adolescent outcomes. We are in the process of conducting these analyses for the current study and intend to share these results in the future. Our analyses will be correlational in nature and will provide a starting point for identifying effective core components. In larger studies, researchers can use more rigorous tests such as mediational analyses or dismantling strategies to isolate potential active ingredients. Such increased rigor will be a natural outgrowth of several iterations of the CORE new directions for youth development • doi: 10.1002.yd
78
MINDFULNESS IN ADOLESCENCE
Process. Indeed, if researchers and practitioners work together using the CORE Process, over time and across trials, we can build a more coherent theory of the common program elements and contextual variables necessary for mindfulness programs to positively influence adolescent lives.
Notes 1. Black, D. S., Milam, J., & Sussman, S. (2009). Sitting-meditation interventions among youth: A review of treatment efficacy. Pediatrics, 124, 532–541; Greenberg, M. T., & Harris, A. R. (2011). Nurturing mindfulness in children and youth: Current state of research. Child Development Perspectives, 6(2), 161– 166. 2. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication No. 231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. P. 24. 3. Blas´e, K., & Fixsen, D. (2013). Core intervention components: Identifying and operationalizing what makes programs work (ASPE Research Brief). Washington, DC: OHSP, US Department of Health and Human Services. Retrieved from http://aspe.hhs.gov/hsp/13/KeyIssuesforChildrenYouth /CoreIntervention/rb CoreIntervention.cfm 4. Fixsen et al. (2005); Kutash, K., Cross, B., Madias, A., Duchonowski, A. J., & Green, A. L. (2012). Description of a fidelity implementation system: An example from a community-based children’s mental health program. Journal of Child and Family Studies, 21, 1028–1040; Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340. 5. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45; Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221. 6. Fixsen et al. (2005); Mowbray et al. (2003); O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78(1), 33–84. 7. Greenberg & Harris. (2011); Meiklejohn, J., Phillips, C., Freedman, M. L., Griffin, M. L., Biegel, G., Roach, A., . . . Saltzman, A. (2012). Integrating mindfulness training into K-12 education: Fostering the resilience of teachers and students. Mindfulness, 3(4), 291–307; Davidson, R. J., & Mind and Life Education Research Network. (2012). Contemplative practices and new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
79
mental training: Prospects for American education. Child Development Perspectives, 6(2), 146–153. 8. Greenberg & Harris. (2011). 9. Meiklejohn et al. (2012). 10. Deming, W. E. (1986). Out of crisis. Cambridge: MIT Press. 11. Holistic Life Foundation. (2013, August 22). Personal communication. 12. Mendelson, T., Greenberg, M. T., Dariotis, J. K., Feagans Gould, L., Rhoades, B. L., & Leaf, P. J. (2010). Feasibility and preliminary outcomes of a mindfulness intervention for urban youth. Journal of Abnormal Child Psychology, 38, 985–994; Blair, C., & Diamond, A. (2008). Biological processes in prevention and intervention: The promotion of self-regulation as a means of preventing school failure. Development and Psychopathology, 20, 899–911; Greenberg, M. T. (2006). Promoting resilience in children and youth: Preventive interventions and their interface with neuroscience. Annals of the New York Academy of Science, 1094, 139–150. 13. Mendelson et al. (2010). 14. Durlak, J. A. (1998). Why program implementation is important. Journal of Prevention & Intervention in the Community, 17, 5–18. 15. Leithwood, K. A., & Montgomery, D. J. (1980). Evaluating program implementation. Evaluation Review, 4, 193–214. 16. Mowbray et al. (2003); Century, J., Rudnick, M., & Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31(2), 199–218. 17. Mowbray et al. (2003); Century et al. (2010). 18. Kabat-Zinn, J. (2011). Some reflections on the origins of MBSR, skillful means, and the trouble with maps. Contemporary Buddhism, 12(1), 281–306; Crane, R. S., Kuyken, W., Williams, J. M. G., Hastings, R. P., Cooper, L., & Fennell, M. J. V. (2011). Competence in teaching mindfulness-based courses: Concepts, development, and assessment. Mindfulness, 3, 76–84; Cullen, M. (2011). Mindfulness-based interventions: An emerging phenomenon. Mindfulness, 2, 186–193. 19. Domitrovich & Greenberg. (2000); Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350; Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research Theory & Practice, 18(2), 237–256. 20. Kutash et al. (2012). 21. Kutash et al. (2012); Dusenbury et al. (2003). 22. Dane & Schneider. (1998). 23. Century et al. (2010). 24. Kabat-Zinn. (2011); Crane et al. (2011); Cullen. (2011). 25. O’Donnell. (2008). 26. Mowbray et al. (2003). new directions for youth development • doi: 10.1002.yd
80
MINDFULNESS IN ADOLESCENCE
27. Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34; McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. 28. Hallgren. (2012). 29. Thorndike, R. M., Cunningham, G. K., Thorndike, R. L., & Hagen, E. P. (1991). Measurement and evaluation in psychology and education. New York, NY: Macmillan Publishing Company. 30. Mowbray et al. (2003). 31. Dane & Schneider. (1998). 32. Mendelson, T., Dariotis, J. K., Feagans Gould, L., Smith, A. S. R., Smith, A. A., Gonzalez, A. A., & Greenberg, M. T. (2013). Implementing mindfulness and yoga in urban schools: A community-academic partnership. Journal of Children’s Services, 8(4), 276–291. 33. Mowbray et al. (2003). 34. Mendelson et al. (2013). 35. Domitrovich & Greenberg. (2000); Fixsen, D. L., Blas´e, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19, 531–540. 36. Mowbray et al. (2003); O’Donnell. (2008). 37. Crane et al. (2011); Cullen. (2011). 38. Mowbray et al. (2003). 39. Schonert-Reichl, K. A., & Lawlor, M. S. (2010). The effects of a mindfulness-based education program on pre- and early adolescents’ wellbeing and social and emotional competence. Mindfulness, 1(3), 137–151. 40. Blas´e & Fixsen. (2013) 41. Mowbray et al. (2003); Domitrovich & Greenberg. (2000); O’Donnell. (2008); Dusenbury et al. (2003). 42. Gottfredson, D. C., & Wilson, D. B. (2003). Characteristics of effective school-based substance abuse prevention. Prevention Science, 4, 27–38. 43. Crane et al. (2011). 44. Kabat-Zinn. (2011). 45. Graham, M., Milanowski, A., & Miller, J. (2012). Measuring and promoting inter-rater agreement of teacher and principal performance ratings. Center for Educator Compensation Reform. Retrieved from http://cecr.ed .gov/pdfs/Inter Rater.pdf 46. Crane et al. (2011). 47. Meiklejohn et al. (2012); Mendelson et al. (2013).
laura feagans gould is a research scientist at the University of North Carolina at Greensboro SERVE Center. tamar mendelson is an assistant professor in the Department of Mental Health at the Johns Hopkins Bloomberg School of Public Health. new directions for youth development • doi: 10.1002.yd
ASSESSING FIDELITY OF CORE COMPONENTS
81
jacinda k. dariotis is an assistant professor in population, family, and reproductive health at the Johns Hopkins Bloomberg School of Public Health. matthew ancona is a graduate student in industrial and organizational psychology at the University of Baltimore. ali s. r. smith, andres a. gonzalez, and atman a. smith are cofounders of the Holistic Life Foundation, Inc. mark t. greenberg is the Edna Peterson Bennett Endowed Chair in Prevention Research, founding director of the Prevention Research Center for the Promotion of Human Development, and professor of human development and psychology in the College of Health and Human Development at the Pennsylvania State University.
new directions for youth development • doi: 10.1002.yd
