Adm Policy Ment Health DOI 10.1007/s10488-014-0535-7

ORIGINAL ARTICLE

Exploring Knowledge Exchange at the Research–Policy–Practice Interface in Children’s Behavioral Health Services Laurel K. Leslie • Susan Maciolek • Kathleen Biebel • Gifty Debordes-Jackson Joanne Nicholson

•

 Springer Science+Business Media New York 2014

Abstract This case study explored core components of knowledge exchange among researchers, policymakers, and practitioners within the context of the Rosie D. versus Romney class action lawsuit in Massachusetts and the development and implementation of its remedial plan. We identified three distinct, sequential knowledge exchange episodes with different purposes, stakeholders, and knowledge exchanged, as decision-making moved from Federal Medicaid policy to state Medicaid program standards and to community-level practice. The knowledge exchanged included research regarding Wraparound, a key component of the remedial plan, as well as contextual information critical for implementation (e.g., Federal Medicaid policy, managed care requirements, community organizations’ characteristics). Keywords Research use
Policy
Intermediary
Knowledge exchange
Wraparound
Children’s behavioral health

L. K. Leslie (&) Tufts Medical Center/ Floating Hospital for Children, 800 Washington Street, #345, Boston, MA 02111, USA e-mail: [email protected] S. Maciolek Policy and Management Consultant, Boston, MA, USA K. Biebel
G. Debordes-Jackson University of Massachusetts Medical School, Worcester, MA, USA J. Nicholson Geisel School of Medicine at Dartmouth, Hanover, NH, USA

Introduction The last decade has witnessed the publication of major reports identifying significant gaps in mental health services provided in the US (Committee on Quality of Healthcare in America 2001; Institute of Medicine (US) 2006; New Freedom Commission on Mental Health 2003). Despite an expanding body of promising or evidence-based mental health practices, uptake into usual care settings remains limited (Weisz and Addis 2006). Federal agencies, health insurance companies, and accreditation bodies have increasingly called for the adoption of evidence-informed practices in mental health (Hogan 2003). Calls to bridge the gap between ‘‘what is known’’ and ‘‘what is done’’ have led to a small but growing body of literature on how researchers interface with policymakers and practitioners. Early models of this interface focused on the autonomous production and passive transfer of research by researchers to practitioners and policymakers, describing linear, rational processes whereby research was conducted by a research ‘‘producer,’’ usually located within an academic center, and then ‘‘pushed’’ to a ready and waiting research ‘‘user’’ in the policy or practice arena (Lavis et al. 2003). Challenges faced both by researchers attempting to communicate their findings to policymakers and practitioners, and by policymakers and practitioners attempting to access research, prompted the recognition and promotion of intermediaries who act as translators of research findings to potential users (Lavis et al. 2003). More recently, knowledge translation or knowledge exchange frameworks have been promoted as a potentially useful for examining the interactions among researchers, policymakers, and practitioners. There is considerable variability in the terminology used to describe these frameworks, both within health-related research (e.g.,

123

Adm Policy Ment Health

knowledge translation, linkage and exchange, knowledge to action) and in other disciplines (e.g., technology transfer, knowledge management, change management). Despite this lack of consistent terminology, the World Health Organization has adopted knowledge translation as a guiding principle and defined it as ‘‘the synthesis, exchange, and application of knowledge by relevant stakeholders to accelerate the benefits of global and local innovation in strengthening health systems and improving people’s health’’ (World Health Organization 2006). While multiple knowledge translation or exchange (hereafter, ‘‘exchange’’) frameworks have been proposed across publications (Estabrooks et al. 2008; Hogan 2003; Mitton et al. 2007; Pentland et al. 2011), several themes are common. First, these frameworks posit that researchers, practitioners, and policymakers represent different stakeholder groups with distinct cultures and priorities, all of whom have knowledge and expertise that is needed by the other. This addresses what Tseng describes as the risk of ‘‘privileging researchers’ perspectives and relegating practice professionals to the receiving end of research and dissemination efforts’’ (Tseng 2012) by highlighting the multi-directional exchange of knowledge. Second, explicit among these frameworks is the concept that the knowledge being exchanged does not consist only of the available research on an evidence-based practice but also includes other forms of knowing, such as experiential knowledge about the context within which a policy or practice might be embedded (Graham et al. 2006). Third, principles from participatory research processes are incorporated, highlighting the role of relationships and trust among researchers, policymakers, and practitioners (Gagnon 2011; Ko et al. 2005). Last, knowledge translation and exchange frameworks acknowledge the complexities of the sociopolitical context in which practice and policy decisionmaking occurs (Gibbons 2008). However, research regarding knowledge exchange remains limited. Mitton and colleagues’ review of this literature found that only 20 % of studies reported any realworld application and characterized the majority of papers as ‘‘opinion pieces, reviews, or surveys of stakeholders’’ (Mitton et al. 2007). In addition, most papers have originated in Canada or Europe, with only a limited number reflecting decision-making in public systems in other countries with different political structures and processes for providing mental health care, such as the US. The study described below addresses these limitations by exploring the application of knowledge exchange within the context of a major public mental health policy initiative in the US. Reforms in the Commonwealth of Massachusetts’ children’s behavioral health system offer a unique and timely opportunity to examine the application of knowledge translation and exchange to children’s mental health policy,

123

programs, and practice. Specifically, we employed a case study design to ask the following questions: (1) what knowledge was exchanged, in what setting(s), and for what purpose? (2) What role did researchers, policymakers, and practitioners play in the knowledge exchange? (3) What policy, program, and practice decisions and subsequent actions resulted from the exchange? (4) What can this case study add to the range of knowledge exchange frameworks examining the research–policy–practice interface? For these analyses, we defined knowledge as inclusive of published research evidence as well as other types of knowledge reflecting the context of the policy initiative and the experiences of stakeholders (Asen et al. 2011). The Context: The Massachusetts Children’s Behavioral Health System In 2001, a class action lawsuit, Rosie D. v. Romney, was brought by plaintiffs to the Federal Court system under the early periodic screening diagnosis and treatment (EPSDT) provisions of the Medicaid Act. The federal EPSDT provisions entitle Medicaid-eligible individuals under the age of 21 to screening, diagnosis, and treatment that is medically necessary to correct or ameliorate physical and mental health conditions. Plaintiffs sought to compel Massachusetts to provide access to and coordination of home-based behavioral health services to enable children with severe emotional disturbance identified under EPSDT to receive treatment and support in their homes and home communities. In 2006, the Federal Court found Massachusetts in violation of EPSDT, a remedial plan was established, and an independent Court Monitor was appointed to monitor the state’s implementation of the plan. A key provision of the Rosie D. remedial plan was intensive care coordination using Wraparound, a teambased planning process to provide family-driven, intensive care coordination for children involved with public childand family-serving systems (e.g., mental health, child welfare, juvenile justice, special education) (Burchard et al. 2002; Rossman 2002). While there is some debate as to whether the available evidence supports Wraparound’s designation as ‘‘promising’’ or ‘‘evidence-based,’’ (Suter and Bruns 2009) its documented success in increasing community-based care in lieu of institutional care has led to its implementation both within the US as well as in other countries, its mandated use in several Federally supported programs, and its inclusion in the Surgeon General’s reports on Children’s Mental Health and on Youth Violence (Center for Mental Health Services (US), National Center for Injury Prevention and Control, National Institute of Mental Health (US), and Office of the Surgeon General (US) 2001).

Adm Policy Ment Health

Wraparound is a care planning practice in which formal services and informal supports are coordinated across systems. It is not a direct service treatment intervention and, thus, differs from the single treatment designs that are often designated as evidence-based practices. Wraparound builds on system of care principles and prioritizes family voice and choice in care planning and delivery. Manualized education and training resources are available from the National Wraparound Initiative (http://nwi.pdx.edu/) as well as from national purveyors who provide training and technical assistance to organizations seeking to implement Wraparound. As part of the implementation of the Rosie D. remedy plan, 24 community-based organizations in Massachusetts were selected in 2009 through a competitive procurement process to provide Wraparound. The 24 organizations provide services through 32 Community Service Agencies (CSAs) (reflecting 29 geographical service areas and 3 special populations). CSA contracts were established and administered through the Medicaid Program’s five managed care entities (MCEs). The state also selected, through a competitive procurement, a national Wraparound purveyor to provide training and coaching to CSA staff. The MCEs established technical assistance teams to provide consultation to the CSAs and the Wraparound purveyor on the implementation of Wraparound in the context of the state’s Medicaid managed behavioral health care structures and processes. The planning for and implementation of the Rosie D. remedial plan permitted an in-depth explanatory case study to inform theory regarding knowledge exchange at the researcher-policymaker-practitioner interface. Conduct of our study occurred concurrent with community implementation, after the Rosie D. trial ended and the remedial plan was developed. Our primary interest was in knowledge exchange at the community level. However, understanding the socio-political context established by the court ruling, the remedial plan, and the Medicaid standards was essential for framing decisions made at the community level. Consequently, study methodology required relying on retrospective reports; some potential key informants also were not available. Nevertheless, we gained important insights about the nature and extent of knowledge exchange across decision-making episodes across multiple levels.

Methods Because the theoretical understanding of knowledge exchange is at a formative stage, we employed qualitative methods (Denzin and Lincoln 2005). Qualitative methods are particularly appropriate when the questions at hand

relate to dynamic processes, and are useful in laying the groundwork for further theory development (Weiss 1998). We employed a nested case study design, appropriate to theory generation (Eisenhardt 1989) and useful in understanding state health reforms (Brownson et al. 2009). The study was conducted across three linked and overlapping Phases. Phase 1 included the collection and analysis of public documents regarding the Rosie D. court decision and the development and implementation of the remedial plan. Phase 2 consisted of one-on-one and group interviews with state-level key informants engaged in policy and program decisions related to the Rosie D. court case and the remedial plan, including state agency representatives and consultants, the MCE technical assistance teams, and the Wraparound purveyor. Phase 3 consisted of one-on-one and group interviews and document analysis during site visits with four provider agencies serving as CSAs. The document review and interview protocols were guided by concepts drawn from the use of research evidence literature as a priori specification of constructs (Eisenhardt 1989). Specifically, we drew on Gibbons’ knowledge exchange framework (Gibbons 2008) and Tseng and Seidman’s social settings framework to guide purposeful sampling decisions (i.e., the selection of state-level key informants and CSA sites) (Tseng and Seidman 2007). Data were obtained from multiple sources and stakeholders with divergent points of view and experiences with the systems change initiative to allow for the triangulation of data (Brownson et al. 2009; Creswell and Plano Clark 2007). Study procedures were approved by Institutional Review Boards at all three academic institutions affiliated with the study. Phase 1. Setting the Context: Document Procurement and Review Documents procured were publicly available, accessible through state agency or other websites, and included Court documents, progress reports prepared by the Massachusetts Executive Office of Health and Human Services (EOHHS), Medicaid program and procurement documents, reports prepared by the Court Monitor, and presentations to stakeholders. In addition, state-level informants, interviewed in Phase 2, nominated documents. At least two research team members initially reviewed the 87 documents for: (1) Specific descriptions that permitted delineation of the timeline of the Rosie D. lawsuit and the development and implementation of the proposed remedial plan; and (2) any text regarding knowledge that might have been exchanged among stakeholders as well as any guidelines or parameters put into place that informed policy and practice decisions. Eighteen documents were selected for final review by the entire research team.

123

Adm Policy Ment Health

Phase 2. State-Level Policy and Program: Key Informant Interviews A purposeful sampling strategy was used to recruit study participants representing the plaintiffs’ attorneys, the defendants for the state, court-oversight officials, state agency administrators, parent advocates, Medicaid MCE technical assistance teams, and the national Wraparound purveyor, as well as other individuals whom key informants recommended as offering important perspectives to the study. Potential participants were sent an email informing them of the study and inviting their participation. All invitees agreed to participate. Two study investigators conducted each of the face-to-face interviews between December 2010 and February 2012 in a location convenient to participants; the interviews lasted 1–2 h. If several individuals from an organization were identified and requested to be interviewed together, these individuals were interviewed using standardized group interview facilitation techniques (Frey and Fontana 1991; Kitzinger 1995). Prior to the interviews, informed consent was obtained and participants completed a brief background survey (e.g., training and background, years of experience with the agency or organization, years of experience in various professional roles, participation in statewide professional activities, and role in the development and implementation of the Rosie D. remedial plan). Investigators employed a semi-structured interview protocol informed by the knowledge translation and exchange literature, knowledge gained from the document review, and investigators’ previous experiences as public agency staff or clinicians or in mid-career fellowship placements with public agencies or research settings. The protocol also contained a printed timeline, extending from 2006 to 2012; informants were asked to frame their responses within the context of the timeline. All individual interviews (n = 8) and group interviews (2 groups; n = 13) were digitally recorded, transcribed by a professional transcriptionist, and deidentified to maintain study participants’ privacy. Investigator pairs alternated interviews, allowing for the objective perspectives of research team members not assigned to particular interviews (Eisenhardt 1989). Investigators met for peer debriefing following interviews to review their experiences and impressions, and explore any lack of clarity in meaning. Phase 3. Practice Considerations: Community Service Agency (CSA) Key Informant Interviews A purposeful sampling strategy was used to recruit four provider agencies serving as CSAs for inclusion in the study. Investigators emailed a description of the study, an

123

invitation to participate, and a link to an on-line survey to the Executive Directors of the 21 community-based organizations providing CSA services in 29 service areas across Massachusetts. (The decision was made not to include the 3 organizations providing services to specialty populations (e.g., deaf and hearing impaired), given that their purview was statewide rather than geographic-specific). The provider agency survey included questions about their site locations, size, organization of staff and resources, service array, clients served, and budget. Twelve agency surveys were returned to the study team reflecting 17 CSAs. Visual inspection of survey data (e.g., geographic location; urban, suburban or rural; previous experience with care coordination; and range of experience in the children’s mental health experience) regarding responding agencies as compared with non-respondents suggested the respondents provided an adequate representation of the larger pool of CSA agencies. A quantitative ranking system was established to characterize agencies across three domains hypothesized by Tseng and Seidman (2007) to be relevant to processes in agency settings: (1) social organization (i.e., the complexity of overall agency organizational structure and financing); (2) physical organization (i.e., the location of service delivery site(s) for the remedial plan services); and (3) economic/temporal organization (i.e., infrastructure support for training, quality improvement, and/or research) (Tseng and Seidman 2007). Four provider agencies were identified for the study, representing a range of complexity in each domain. All invited sites agreed to participate. Study team members mailed agency Executive Directors or their assistants, if preferred, to schedule site visits with administrators, managers, and direct service staff nominated by Executive Directors as key players in implementing the Rosie D. remedial plan at the agency level. Prior to site visits, research staff compiled agency profiles based on responses from the online survey and the CSAs’ original application to the state, their agency’s organizational chart, their most recent annual report, and other relevant program materials. Site visits were scheduled over 2–3 days and took place between April 2011 and May 2012, at times and in locations convenient for agency staff. CSA profiles were reviewed and confirmed by CSA participants during site visit. A semi-structured interview protocol was developed to explore the agency’s role in Rosie D. activities and services using the same printed timeline as in Phase 2. Other questions addressed the decision to serve as a CSA, barriers and facilitators to the incorporation of Wraparound into the agency’s organization and activities, and the exchange of information (e.g., research, contextual factors) during implementation. All individual interviews (n = 14) and group interviews (9 groups; n = 38) were digitally recorded, professionally

Adm Policy Ment Health

transcribed, and de-identified to maintain study participants’ confidentiality.

Table 1 Study participants Characteristics*

Analysis All responses from individual surveys were manually entered into database software (Microsoft Office Excel 2007) and imported into a statistical software package (IBM SPSS 20.0) for the calculation of all descriptive data. Quantitative data obtained in provider agency surveys were analyzed using the software component of the online survey application to summarize descriptive information. All transcriptions of qualitative interview data were checked against digital audio files for accuracy. Investigators used a summative analysis approach to the qualitative data as outlined by Rapport (2010). Members of the research team began reviewing narrative data as it became available, as earlier phases of the study informed later phases in response to knowledge gained (Weiss 1998). The team read and re-read the raw material, generating memos regarding impressions, emerging concepts and relationships, and reviewing findings that elaborated or corroborated a priori themes of research use previously defined in the literature (Miles and Huberman 1994; Palinkas et al. 2009). Visual displays and tables were developed to manage and present data, including timelines, flow charts, and matrices of themes and relationships (Eisenhardt 1989). All study documents and transcripts were uploaded into Dedoose, a web-based software program that facilitates mixed methods data management (Dedoose 2012), for further coding, elaboration, and specification of knowledge exchange concepts and relationships. Agency case study data were prepared following a template to facilitate the integration of data across agencies (Lincoln and Guba 1985).

Results Participants Twenty-one state-level key informants (i.e., state agency administrators, legal staff, court, the Wraparound purveyor, MCEs) and 55 community-level key informants (i.e., CSA administrators and staff) were interviewed (see Table 1). The majority were white, female, and had a college degree or higher. CSA participants were younger, reported less prior experience with research, and were relatively new additions to their current agency compared to state-level participants.

State-level policy and program participants N = 21 (%)

CSA participants N = 55 (%)

Gender Female

81

85

19

15

White or Caucasian

100

92

Native Hawaiian or other Pacific Islander

–

4

African-American

–

–

Hispanic/Latina(o)

–

11

Puerto Rican

–

9

Other

–

2

–

4

22–34

5

31

35–44 45–54

38 14

31 25

55–64

43

11

65?

–

2

Male Race/ethnicity

Chose not to answer Age

Education High school graduate/GED

–

9

Partial college credit

–

5

Associate’s degree

–

7

Bachelor’s degree

10

15

Master’s degree

67

60

Professional degree

5

–

Doctorate degree

19

4

Yes

62

40

No

33

Prior experience with research

NA 5 Type of agency affiliated with

55 5

State agency

76

Provider agency

–

– 100

NA/other

24

–

1–3 years

33

65

4–7 years

24

20

8–11 years

5

7

12 or more years

29

7

NA

10

–

Number of years with agency

* Race/ethnicity does not sum to 100 % as Hispanic/Latino was asked as a separate question; other characteristics approximate to 100 % due to rounding conventions

Primary Themes Several primary themes were identified regarding knowledge exchange during the development and implementation of the Rosie D. remedial plan. First, three distinct

episodes of activities were identified that occurred at different levels of policy and practice and with different purposes: (1) the development of the Rosie D. remedial plan through court processes at the Federal policy level;

123

Adm Policy Ment Health Fig. 1 Research use over time in the Massachusetts children’s behavioral health initiative

Episode #1: Policy

Episode #2: Program

Episode #3: Practice

2005-2006

2007-2008

2009+

Dates

Scope of Decision Parameters

Federal Medicaid EPSDT Law and Regulations

State Medicaid Program Standards

Judge

Court Monitor

Decision Plaintiff’s Attorneys Makers & State’s Attorneys Intermediaries (in italics)

State Administrators Medicaid Program Consultant

(2) the establishment of Medicaid program standards for remedy services and a CSA request for proposals (RFP) at the state program level; and (3) the CSAs’ implementation of Wraparound at the community practice level. The knowledge exchanged thus moved sequentially from the Federal level to the state level and then to the local community level. (see Fig. 1). The first episode may be better characterized as knowledge transfer, bounded as it was by the Court’s rules of engagement, with attorneys presenting witnesses and evidence to the judge for his consideration. The second and third episodes exemplified knowledge exchange as an interactive process. Second, while some participants transitioned from one episode to another, episodes were notable for markedly different participants as the level (Federal, state, community) and purpose (policy, program, practice) of the knowledge exchange shifted. In addition, at the same time that knowledge exchange narrowed in scope from federal to local, the numbers and diversity of participants increased. Third, each episode resulted in a particular decision made by different decisionmakers that informed the next actions to be taken. Different knowledge was exchanged to inform these decisions, with research playing different roles in each episode. Documents from one episode served to inform the actions taken in the subsequent episode (see Table 2). Below we describe each episode in more detail, highlighting these three themes. Episodes Episode 1: Federal Court Policy Decisions Regarding Need and Scope of Remedial Plan The first episode consisted of a series of formal exchanges typical of a court proceeding, with the Judge serving as the

123

Wraparound Research Evidence

Community Service Agencies Manage Care Entities Wraparound Purveyor

arbiter of the evidence presented by witnesses identified by attorneys for the plaintiffs and the state. The first phase of Episode 1 occurred during a 6-week trial in May and June 2005, during which the parties presented 45 witnesses and introduced over 500 exhibits (Center for Public Representation 2006). The Judge heard testimony from a range of researchers, clinicians, program administrators, family members, and fiscal and other experts. This quote from one attorney describes the types of experts who testified and the knowledge they presented to the Judge: One expert team was essentially the [pause] the story of Wraparound, the story of home-based services, so that was a very, very heavily data-based presentation… they’re the ones who are gonna describe the model and also demonstrate the effectiveness of the model compared to what existed. A second group of experts who we called the ‘programmatic experts’ who were people who came and observed, they analyzed information about Massachusetts mental health services for kids… and then spoke about the extent to which Massachusetts does or doesn’t offer Wraparound services. And then we had a third group of experts who were fiscal, financial, right, and Medicaid data experts… And they were there to speak to, what is Massachusetts spending, and what does this alternative cost, and make a number of projections about that. And then we had a fourth group of experts who were, who we called ‘client experts’. With respect to the types of knowledge presented during this initial episode, the range varied substantially. Federal reports and research studies provided some context for the

Adm Policy Ment Health Table 2 Characteristics of three episodes Episode #1: Federal policy

Episode #2: State Medicaid program standards

Episode #3: Community-level CSA practice

Dates

2005–2006

2007–2008

2009 to date

Decision to be made

Was Massachusetts in violation of Federal EPSDT requirements?

How would the remedial plan be operationalized in Massachusetts Medicaid program standards?

How would CSAs implement Wraparound within Medicaid managed care?

Decision-maker(s)

Federal Judge

State administrators with oversight from Court and input from plaintiffs’ attorneys

CSA administrators and line staff

Decision parameter

Federal

State

Community

Decision/action

Sources of information for decision-making

Memorandum of decision

Medicaid Program standards developed

Judgment and remedial plan

CSA request for proposals developed

Ongoing, iterative decisions drive day-to-day Wraparound implementation

Researchers

Medicaid managers

CSAs

Clinical experts

Family advocacy groups

Managed care entities

Wraparound state pilot managers

Program experts

Wraparound purveyor State administrators

Plaintiff families Intermediaries

Plaintiff and state attorneys

Plaintiff’s attorneys

Managed care entities

Consultant

Wraparound purveyor

scope of the problem regarding rates of children’s behavioral health needs and possible intervention models. Research studies cited during the trial included papers published in the professional literature and evaluation results from demonstration projects funded by the Substance Abuse Mental Health Services Administration (SAMHSA). Additional program evaluation data were also shared, including reports of programs implemented in other states (Center for Public Representation 2006). Families of plaintiff children shared narratives of their experiences regarding receiving services in Massachusetts. As a result of this non-jury trial, the Federal Court ruled that the Massachusetts Medicaid program was in violation of Federal Medicaid EPSDT requirements. Subsequent to this ruling, the Judge ordered a remedial plan that included intensive care coordination, using the Wraparound model, as a key component. The following quote from the Judge’s Memorandum of Decision on January 26, 2006 reflects the impact of the knowledge exchanged during the Court proceedings on the Judge’s formulation of his decision: The evidence showed, time and again, that the Commonwealth’s efforts to comply with these minimum EPSDT assessment and service coordination requirements were woefully inadequate, with detrimental consequences for thousands of vulnerable children … Defendants’ [the state’s] failure to provide adequate assessments, service coordination, and home-based supportive services for Medicaid-eligible children with serious emotional disturbances was

glaring from the evidence and at times shocking in its consequences (Rosie D. v. Romney 2006). The Judge ordered the plaintiffs and the state to confer and negotiate a joint remedial plan. As the plaintiffs developed their proposed remedial plan, they ‘‘incorporated lots of specifics about what the system would look like, including infrastructure, not just services, but also incorporating oversight, data collection, documentation, monitoring of all that information and services.’’ Central to their plan development was identifying ‘‘programs that work, or… programs that have a track record.’’ For example, the plaintiffs gathered information from grants funded by the Robert Wood Johnson Foundation and SAMSHA as well as from other states’ initiatives in Milwaukee, Indianapolis, and New York. In developing their proposed remedial plan, the state listened to parent advocates, looked at research, and examined other states’ programs. The practical consideration of what might work, given the sociopolitical context in Massachusetts, was a high priority: ‘‘New Jersey was a model we looked at a lot, ‘cause we had to bring in, we always had to be mindful of the Medicaid platform.’’ At the end of the imposed deadline, the two parties were unable to reach agreement and presented separate proposals to the Judge. On July 16, 2007, the Judge issued a final Judgment that essentially adopted the state’s proposed remedial plan. Interestingly, despite acknowledged tensions during the court proceedings and the development of the remedial plan, the decision to include Wraparound in

123

Adm Policy Ment Health

the plan was facilitated by consensus, not dispute, between the plaintiffs and the state. One state manager said, ‘‘I think everybody wanted Wraparound. I think we wanted it, they wanted it.’’ Episode 2: Translating the Remedy Provisions into State Medicaid Managed Care Program Standards The second episode focused on translating the provisions of the Court’s Judgment into action regarding the state Medicaid program. Episode 2 differed from Episode 1 in that the decision as to how the remedial plan would be operationalized was negotiated between the plaintiff attorneys and the state administrators directly and not in the Federal Court. While the plaintiffs’ attorneys continued to be actively involved, state leadership shifted from attorneys in the state Attorney General’s office to administrators in the Massachusetts EOHHS and the state Medicaid program. The Judge also appointed a Court Monitor to represent him (see Table 1.) The negotiations provided opportunity for the exchange of additional knowledge, some of which reflected research but most of which related to the managed care and regulatory context specific to Massachusetts. This contextual knowledge was essential for the development of the program standards regarding the implementation of Wraparound within the Massachusetts Medicaid managed care system. Participants during this episode included representatives from other states that had made similar changes in their Medicaid program, national experts, clinicians, and parent advocates. One state-level key informant summarized the negotiations as follows: All of the services were developed through negotiation with the attorneys for the plaintiffs, so every definition was developed in negotiation, the service descriptions that providers used, the program model, the operations manuals and the medical necessity criteria were all a negotiation between the attorneys for the plaintiffs and the state… And, so, they also, you know, they were attorneys and so their expertise was on the legal matters. They had to use a group of external consultants to advise them. We were using external kind of consultants to advise us. And then our Court Monitor was using consultants that she would bring into talk to both of us. So, it was all of that that was coming together to, to develop, kind of, what the service descriptions looked at. This episode was characterized by intense time pressures that affected the knowledge exchange process. The Judge had established tight implementation timeframes, requiring intensive work planning. One state manager described this episode: ‘‘we had to get a product to the

123

attorneys to respond to in three weeks, so that we could negotiate it for two, so we could get it to the MCOs to roll out, so they could go live with providers in three months, so providers could have, you know, six to eight weeks to get ready.’’ When asked to describe the specific role of research in the process, one informant stated: ‘‘research is always in the mix, but where research falls in terms of, is it the first, second, third, or fourth kind of point in the decision-making process, in that type of process does vary.’’ This informant clarified: ‘‘it’s not that, again, research didn’t play a role, ‘cause it did. I’m wondering if it would’ve played a different role if it wasn’t in the context of a lawsuit… it’s just different…the time pressure…was always the number one decider.’’ Managing knowledge exchange among the parties in a manner that produced complex documents reflecting decisions made in a short timeframe was facilitated by a local consultant with a highly regarded national reputation and extensive expertise in Medicaid regulations. The consultant functioned as an intermediary, demonstrating the necessary skills to effectively work with attorneys as well as the ability to generate realistic options for consideration by state program managers. The consultant was not vested with the decision-making authority of an institution (as was the Judge) but rather exercised influence based on the trust accorded him by the parties. One state-level informant commented: ‘‘You know, I think that’s what was critical, was I think [consultant] was trusted by both sides… You know, that’s what mattered. You know, the parties are never, never, ever gonna trust each other… a trustworthy conduit of information…. that was absolutely essential.’’ The consultant thus functioned as an intermediary in the exchange between the plaintiffs and the state, building on both his trusted status as a neutral party as well as his expertise in the knowledge at the heart of the lawsuit and remedial plan (i.e., Medicaid regulations and operations). A number of meetings during this episode were in fact held in his office, creating a neutral space in which to negotiate critical decisions during this episode. Episode 3: Implementing Wraparound Practice Through Community Service Agencies (CSAs) The third episode consisted of numerous ongoing and iterative exchanges to address the two major challenges the state faced: (1) to train rapidly a sufficient workforce of intensive care coordinators and family partners able to deliver Wraparound; and (2) to implement Wraparound within the context of the state Medicaid program standards and managed care reimbursement strategies. This episode also involved a significantly different and diverse set of stakeholders and organizations than in the first two episodes. Stakeholders continued to include state administrators but a number of

Adm Policy Ment Health

new parties were added including the national Wraparound purveyor staff, MCE technical assistance teams, CSA managers, supervisors and, ultimately, frontline workers. The Court Monitor continued to function in providing review of implementation processes. Wraparound requires a skilled workforce of trained care coordinators and family partners, whose job descriptions differ significantly from traditional clinical roles. In order to provide the knowledge to community organizations to move program standards regarding Wraparound into practice, the state initially provided immediate training by a local (Massachusetts) community-based organization that had used Wraparound in a SAMHSA funded program outside the context of Medicaid managed care. The state subsequently established a 2-year contract with a national purveyor to provide intensive training and coaching. Training methods in Wraparound include ‘‘four-day didactic trainings and then there were coaches’ trainings and then there’s individual coaching’’ as well as ‘‘site based coaches’’ to address implementation with specific children and families seen at community organizations. The state also required the Medicaid MCEs (MCEs) to ‘‘have experts within the plans who knew the services, who could kind of make sure that their internal processes were gonna support the service.’’ Unlike the first two episodes that produced decisions documented in writing, the CSAs had to make numerous decisions on a daily basis. The ad-hoc nature of these decisions, at times, led to conflicting advice from the Wraparound purveyor and the MCEs. As one member of an MCE technical assistance team observed: ‘‘First and foremost there’s an inherent rub between Wraparound and Medicaid, so having to have the [pause] you know, having to understand Wraparound so that you can try and make it fit as best as possible into Medicaid is an ongoing challenge.’’ A state administrator tied implementation to the fit between research and the managed care environment in Massachusetts, describing the goal to ‘‘build upon the research that was available at the time about… Wraparound,’’ but highlighting the unique challenge of context: ‘‘… with the exception of say, a New Jersey, who is using a ASO or some other examples of that, you didn’t have this kind of intersection with five different managed care companies by which you had to navigate.’’ Knowledge exchange thus needed to focus not only on the specifics of Wraparound but also on the context in which it was being embedded. How knowledge was exchanged among the Wraparound purveyor, the MCEs, and the CSAs evolved over time. In the beginning, the Wraparound purveyor and the MCE technical assistance teams each consulted separately with the CSAs. After a period of some frustration among all parties due to this arrangement, the state started acting as a mediator between the MCEs and Wraparound purveyor,

arranging direct conversations via conference calls and other mechanisms to facilitate knowledge exchange. One MCE technical assistance team member described the conference calls as: ‘‘…one way that I think that we’ve worked well with the, the [Wraparound] coaches to be able to have … I think direct contact with the coaches worked well.’’ Relationships developed over time, as the Wraparound purveyor and the MCE technical assistance teams worked together with the common purpose of supporting the effective practice of Wraparound by the CSAs within the Medicaid program. The knowledge exchanged across these groups within the context of CSA decision-making informed the perspectives of each of the groups. As one member of the Wraparound purveyor’s team commented, ‘‘And I think that’s one of the things that’s made it smooth is, you know, I can make this sound like we’ve gotten the MCEs to change. Well, we changed together.’’ While knowledge exchange among the CSAs, MCEs, and Wraparound purveyor helped facilitate problem-solving around Wraparound implementation, many of the CSA administrators and staff interviewed described challenges experienced at the interface between Wraparound and managed care. CSA administrators struggled with billing and reimbursement issues: ‘‘Nobody really helps you as much with how to do Wrap in the managed care world. You, know? Like how to manage your billing and time.’’ Similarly, a consistent theme identified by CSA staff members was the conflict between core Wraparound principles of ‘‘family voice, family choice’’ and Medicaid managed care reimbursement requirements. For example, CSA staff commented that MCE requirements to complete a family assessment within 10 days conflicted with Wraparound’s commitment to meaningfully engage families on their own terms and within families’ timeframes. This was particularly challenging for staff working with referred families who were not necessarily invested in Wraparound and were active clients with the Department of Children and Families (the state child welfare/protective services agency). Despite training, a manual, and coaching and knowledge exchange occurring with the MCEs and Wraparound purveyor, CSAs felt additional interaction among the CSAs to exchange their experienced-derived knowledge with each other would be helpful: ‘‘We don’t get to… sit down at a table and say, ‘how did you do that?’… ‘how did you figure that out?’ I think that would be a, a much bigger help, you know?’’

Discussion This case study explored the theoretical constructs of knowledge exchange in one state as policy, program, and practice decisions were made regarding the need for systems change (Episode 1) and the subsequent

123

Adm Policy Ment Health

implementation of Wraparound within a Medicaid managed care platform in response to a Federal court case (Episodes 2–3). Our findings affirm that the translation and application of research findings in policy and practice requires relationships among researchers, intermediaries, policymakers, and practitioners that support the mutual, multi-directional exchange of a broad spectrum of knowledge (Mendel et al. 2008; Tseng 2012). Essential knowledge exchanged in these episodes included research but also knowledge regarding context (e.g., socio-political and regulatory). It should be noted that, within our broader conceptualization of knowledge exchange, we define exchanges that involved intermediaries representing the research (i.e., Wraparound) and context (e.g., the MCEs) in Episode 3 as knowledge exchange. Some might argue that the absence of researchers, even when represented by intermediaries, is not consistent with other published definitions of knowledge exchange. We posit that our inclusive definition of other types of knowledge and the in-depth examination in this case study allowed us to uncover important themes regarding knowledge exchange frameworks. In our review of the literature, few knowledge exchange frameworks explicitly recognize that knowledge exchange may occur within distinct episodes over time, each with different purposes and decisions to be made, requiring the exchange of different types of knowledge, and involving multiple and shifting stakeholders representing research as well as policy, program, and practice. The complexity inherent within the inter-connected decision-making episodes in this case study of mental health policy allows further elaboration of previous frameworks as discussed below. Shifting Diversity of Stakeholders and Knowledge Exchanged Over Time While knowledge exchange frameworks have identified the value of the perspectives and contributions of multiple and diverse participants, this case study highlights the different levels (i.e., federal, state, community) within and across which knowledge was exchanged in order to make and implement mental health policy decisions over time. Rather than more static models of ongoing relationships between research producers and research users, relationships in this case study consisted of multi-directional exchanges among a shifting panoply of research experts; decision-makers at the policy, program, and practice levels; and intermediaries. As the overarching goal of improving mental health services for children in Massachusetts covered by Medicaid falls under the purview of Federal, state, and communitybased entities, decisions and actions at the Federal, state, and community level were necessary; no one level was

123

sufficient by itself to change publicly funded mental health services for children. The Federal Court’s responsibility was to determine if the state had failed to comply with Federal EPSDT requirements and, once finding so, to identify the specific remedial services that were needed, employing research where the Judge deemed appropriate. State administrators were required to establish new state Medicaid program standards that would ensure compliance with the remedial plan, including Wraparound, and to do so in the context of the Massachusetts Medicaid Program’s managed care arrangements. They supplemented research knowledge provided by national experts with their own professional experience as well as that of administrators in other states that had made similar changes in their Medicaid programs and of parent advocates. The communitybased provider agencies that serve as CSAs were responsible for determining how to implement Wraparound using the resources available in their community and within Medicaid rules and rates. They relied on their own expertise and experience in delivering services to their communities as well as knowledge provided by the Wraparound purveyor and by the MCE technical assistance teams. The CSAs also acknowledged the need for sharing information across CSAs with respect to operationalizing the core principles and practices of Wraparound within the context of the Massachusetts Medicaid program standards and a managed care platform. Similarly, stakeholders representing ‘‘research producers’’ changed over time as the nature of the decisions being made changed. In Episode 1, researchers provided data regarding the rates of mental health needs of children and the unmet need of children in Massachusetts; researchers, program developers, and evaluators also shared evidence for potential effective interventions to address these needs. In contrast, in Episode 3, trained purveyors of the Wraparound model provided technical assistance and problem solving regarding implementing Wraparound in a manner consistent with its evidence base. Some knowledge exchange models call for the active engagement of endusers in the prioritization of research questions and the development of study designs, adopting a participatory research approach where researchers and end-users work together from the creation of research to its implementation (Graham et al. 2006). Our findings suggest that the sheer number and diversity of stakeholders engaged in moving research first to policy and then to practice may require other creative approaches for exchanging knowledge that accommodate this potential shifting set of stakeholders. Intermediaries Translate Contexts as well as Research The research literature identifies the critical role of intermediaries in translating research to potential policy and

Adm Policy Ment Health

practice users. Our case study revealed an important variation of this function. In this case study, intermediaries were also needed to translate the complex context (i.e., policy, regulatory, and fiscal environment) in which the research (in this case study, Wraparound) was to be embedded. In Episode 2, this function was filled by a trusted consultant who used his own expertise and knowledge about the Massachusetts Medicaid program and other states’ Medicaid programs to facilitate knowledge exchange among state administrators, plaintiff attorneys and key stakeholders. In Episode 3, the Wraparound purveyor functioned as an intermediary by sharing information about Wraparound. The MCEs also functioned as intermediaries regarding the Massachusetts Medicaid managed care parameters. Without creating a shared knowledge basis that included both research and context, it would have been difficult to develop program standards and operationalize those standards for CSA line staff.

implementation of Wraparound, was achieved. Consistency in some participants across episodes promoted some coherence. For example, the Court Monitor and the plaintiffs’ attorneys continued to be involved in Episode 2 to ensure that decisions were made within the parameters established by the remedial plan in Episode 1. However, while there was some overlap of participants, most participants were not consistent across the episodes requiring some other vehicle for transmitting information. Instead, public documents served an essential function to maintaining coherence. Both Episode 2 and 3 were guided by written documents produced through decisions made during previous episodes. These documents established a public record, readily available on websites, that allowed participants as well as outside observers to track the basis of decisions from policy to program to practice.

Decision-Making Parameters Narrow and Participants Broaden over Time

The context of a Federal class action lawsuit against a public agency might have affected the type and nature of knowledge exchange that occurred. A courtroom has unique rules of engagement and standards of evidence that bound the potential for knowledge exchange. Some might suggest that uni-directional knowledge transfer from witnesses to the judge would be a more accurate characterization of the trial process than knowledge exchange, as knowledge was primarily transferred to the Judge in Episode 1, with knowledge exchange occurring in Episodes 2 and 3. Our findings in this regard are limited by the fact that we did not observe the trial or interview the judge or a diverse set of witnesses, including researchers describing the high rates of mental health needs among children or potential interventions. The use of research evidence in the courts should be directly examined because class action lawsuits are not uncommon with respect to service provision under Medicaid, public mental health, or child welfare/child protective services. Indeed, one review from 1995 to 2005 identified 35 class action lawsuits initiated to improve children’s public services during that decade (Kosanovich and Joseph 2005). However, class action lawsuits are not the only impetus for policy change. The US Congress, state legislatures, and city councils enact policy, all of which must be translated into programs and practice in order to have the intended impact on children, families, and communities. We also recognize the limitations of our case study approach. First, these three episodes occurred over an extended time period; hence, the interviews conducted for Episodes 1 and 2 were retrospective in nature while those for Episode 3 took place as Wraparound was being implemented. In addition, we did not interview the original researchers who developed Wraparound, but instead interviewed a national Wraparound purveyor. Last, case

Over the course of the three episodes, decision-making parameters narrowed in scope, moving from the Federal level to the state level and then to the local community level. Decisions made during each episode occurred within the context of parameters established by decisions made in the previous episode. At the same time that the parameters narrowed in scope, the numbers of participants with different organizational affiliations participating in decision-making increased. (see Fig. 1.) The decisions in Episode 1 were made by the Judge, the sole arbiter and decision-maker, using evidence presented in the Court by attorneys bringing in a variety of individuals to testify and share ‘‘knowledge,’’ broadly defined. Following the Judge’s rulings, decisionmaking responsibility shifted to a small group of state administrators meeting in the neutral space of the trusted intermediary to make detailed changes to the Medicaid program specifications, with the input of the plaintiffs’ attorneys and the Court Monitor. In the third episode, knowledge was exchanged across a number of players representing the state, the MCEs, the Wraparound purveyor, and CSA administrators and staff. These exchanges occurred in multiple locations where state and regional trainings and onsite consultations occurred. Consequently, the setting and mechanisms for fostering knowledge exchange changed. Public Documents Promote Coherence Across Episodes Given the shifts in level (Federal, state, community) and the broadening numbers of stakeholders engaged in knowledge exchange, coherence from one episode to the next was critical for ensuring that the desired goal, the

Study Limitations

123

Adm Policy Ment Health

studies are inherently biased in their examination of any process as they are rooted in a particular context. Additional research should be conducted to confirm our findings regarding multiple, interconnected episodes over time in other contexts.

Conclusions This case study explored common themes across knowledge exchange frameworks within the context of public mental health policy in one state. Practically, with respect to the delivery of mental health services, our paper highlights the complexity of moving research to policy and practice and calls for heightened attention to this complexity. With respect to the delivery of evidence-informed mental health services, translation and application of Wraparound occurred in distinct, sequential episodes as research moved to policy and subsequently practice. Our findings affirm the value of employing a knowledge exchange framework to examine the interface between research, policy, and practice. Findings also suggest that attention must be paid to the changing individuals, organizations, priorities, and informational needs to inform decisions and subsequent action across episodes. In addition, explicit acknowledgement of intermediaries translating research, policy, and practice information and the skill sets needed to serve as trusted intermediaries deserves further exploration. Future research should investigate specific strategies for managing transitions across episodes and promoting effective exchanges across domains (e.g., Federal, state, community) and stakeholders. Acknowledgments This research was conducted as part of the ‘‘Using Evidence to Improve Medicaid Mental Health Services for Massachusetts Children and Youth Study’’ funded by the William T. Grant Foundation. We thank Tully Saunders, Alexandra Caro, and Thomas Mackie for their editorial assistance and review. We gratefully acknowledge the decision-makers who generously shared their time and expertise with the research team, all of whom demonstrated a noteable commitment to the well-being of the children of Massachusetts.

References Asen, R., Gurke, D., Solomon, R., Conners, P., & Gumm, E. (2011). ‘‘The research says’’: Definitions and uses of a key policy term in federal law and local school board deliberations. Argumentation and Advocacy, 47, 195–213. Brownson, R. C., Chriqui, J. F., & Stamatakis, K. A. (2009). Understanding evidence-based public health policy. American Journal of Public Health, 99(9), 1576–1583. doi:10.2105/AJPH. 2008.156224. Burchard, J. D., Bruns, E. J., & Burchard, S. N. (2002). The Wraparound approach. In B. J. Burns & K. Hoagwood (Eds.), Community treatment for youth: Evidence-based interventions

123

for severe emotional and behavioral disorders (1st ed., pp. 69–90). New York, NY: Oxford University Press. Center for Mental Health Services (US), National Center for Injury Prevention and Control, National Institute of Mental Health (US), and Office of the Surgeon General (US). (2001). Youth violence: A report of the surgeon general. Rockville, MD: Office of the Surgeon Genera. Retrieved from http://www.ncbi.nlm.nih. gov/pubmed/20669522. Accessed 14 Oct 2013. Center for Public Representation. (2006). Summary of the Court’s Decision in Rosie D. V. Romney. Retrieved from http://rosied. org/Default.aspx?pageId=526903. Accessed 14 Oct 2013. Committee on Quality of Healthcare in America, and Institute of Medicine (US). (2001). Crossing the quality chasm: A new health system for the 21st century. Health San Francisco (pp. 1–358). National Academy Press. Retrieved from http:// webcache.googleusercontent.com/search?q=cache:WYXMC zaAmCMJ:www.nap.edu/html/quality_chasm/reportbrief. pdf. Accessed 15 Oct 2013. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. In N. Edition (Ed.), Applied linguistics (Vol. 2) (p. 457). Thousand Oaks: Sage. doi:10.1111/j. 1753-6405.2007.00096.x. Dedoose. (2012). Los Angeles, CA: SocioCultural Research Consultants, LLC. Denzin, N. K., & Lincoln, Y. S. (Eds.). (2005). The SAGE handbook of qualitative research (Vol. 3). Sage. Retrieved from http:// www.loc.gov/catdir/toc/ecip053/2004026085.html. Accessed 15 Oct 2013. Eisenhardt, K. M. (1989). Building theories from case study research. Academy of Management Review, 14(4), 532. Estabrooks, C. A., Derksen, L., Winther, C., Lavis, J. N., Scott, S. D., Wallin, L., et al. (2008). The intellectual structure and substance of the knowledge utilization field: A longitudinal author cocitation analysis, 1945 to 2004. Implementation Science, 3, 49. Frey, J. H., & Fontana, A. (1991). The group interview in social research. The Social Science Journal, 28(2), 175–187. Gagnon, M. L. (2011). Moving knowledge to action through dissemination and exchange. Journal of Clinical Epidemiology, 64(1), 25–31. Gibbons, M. (2008). Why is knowledge translation important? Grounding the conversation. Focus: Technical Brief, 21. Retrieved from http://www.uofaweb.ualberta.ca/kusp/pdfs/Cal garyCIHR41.pdf. Accessed 16 Oct 2013. Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J. M., Caswell, W., et al. (2006). Lost in knowledge translation: Time for a map? The Journal of Continuing Education in the Health Professions, 26(1), 13–24. Hogan, M. F. (2003). The President’s New Freedom Commission: recommendations to transform mental health care in America. Psychiatric Services, 54(11), 1467–1474. Institute of Medicine (US). (2006). Improving the quality of health care for mental and substance-use conditions. Washington, DC: National Academies Press. Kitzinger, J. (1995). Introducing focus groups. BMJ, 311, 299–302. Ko, D.-G., Kirsch, L., & King, W. (2005). Antecedents of knowledge transfer from consultants to clients in enterprise system implementations. Management Information Systems Quarterly, 29(1), 59–85. Kosanovich, A., & Joseph, R. M. (2005). Child welfare consent decrees: Analysis of thirty-five Court actions from 1995 to 2005, (October). Retrieved from http://www.cwla.org/advocacy/con sentdecrees.pdf?pagewanted=all. Accessed 16 Oct 2013. Lavis, J. N., Robertson, D., Woodside, J. M., Mcleod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221–248.

Adm Policy Ment Health Lincoln, Y. W., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage. Mendel, P., Meredith, L. S., Schoenbaum, M., Sherbourne, C. D., & Wells, K. B. (2008). Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health, 35(1–2), 21–37. doi:10.1007/s10488-007-0144-9. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Beverly Hills, CA: Sage. Mitton, C., Adair, C. E., McKenzie, E., Patten, S. B., & Waye Perry, B. (2007). Knowledge transfer and exchange: Review and synthesis of the literature. Milbank Quarterly, 85(4), 729–768. New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. Final report. DHHS Pub. No. SMA-03-3832. Rockville, MD. Retrieved from http://govinfo.library.unt.edu/mentalhealthcommission/reports/ FinalReport/downloads/FinalReport.pdf. Accessed 14 Oct 2013. Palinkas, L. A., Aarons, G. A., Chorpita, B. F., Hoagwood, K., Landsverk, J., & Weisz, J. R. (2009). Cultural exchange and the implementation of evidence-based practices: Two case studies. Research on Social Work Practice, 19(5), 602–612. doi:10.1177/ 1049731509335529. Pentland, D., Forsyth, K., Maciver, D., Walsh, M., Murray, R., Irvine, L., et al. (2011). Key characteristics of knowledge transfer and exchange in healthcare: Integrative literature review. Journal of Advanced Nursing, 67(7), 1408–1425. Rapport, F. (2010). Summative analysis: A qualitative method for social science and health research. International Journal of Qualitative Methods, 9(3), 270–290. Rosie D. v. Romney, 410 F.Supp.2d 18. (2006). Retrieved from http:// www.clearinghouse.net/chDocs/public/MH-MA-0005-0012. pdf. Accessed 19 Sept 2013.

Rossman, S. (2002). Services integration: Strengthening offenders and families, while promoting community health and safety. In: From prison to home: The effect of incarceration and reentry on children, families, and communities. The Urban Institute. Retrieved from http://www.urban.org/publications/410625.html. Accessed 15 Oct 2013. Suter, J. C., & Bruns, E. J. (2009). Effectiveness of the wraparound process for children with emotional and behavioral disorders: A meta-analysis. Clinical Child and Family Psychology Review, 12(4), 336–351. Tseng, V. (2012). Social policy report: The uses of research in policy and practice (Vol. 26). New York: Society for Research in Child Development. Tseng, V., & Seidman, E. (2007). A systems framework for understanding social settings. American Journal of Community Psychology, 39(3–4), 217–228. Weiss, C. H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice-Hall Inc. Weisz, J. R., & Addis, M. E. (2006). The research-practice tango and other choreographic challenges: Using and testing evidence-based psychotherapies in clinical care settings. In C. D. Goodheart, A. E. Kazdin, & R. J. Sternberg (Eds.), Evidence-based psychotherapy: Where practice and research meet (1st ed., pp. 179–206). Washington, DC: American Psychological Association. World Health Organization. (2006). Bridging the ‘‘know–do’’ gap meeting on knowledge translation in global health (p. 17). Geneva, Switzerland. Retrieved from http://www.who.int/kms/ WHO_EIP_KMS_2006_2.pdf. Accessed 16 Oct 2013.

123