Using RE-AIM to evaluate implementation of an evidence-based program: a case example from Minnesota.

This article was downloaded by: [Wayne State University] On: 26 November 2014, At: 09:19 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Gerontological Social Work Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wger20

Using RE-AIM to Evaluate Implementation of an Evidence-Based Program: A Case Example From Minnesota a

Deborah Paone a

Health Policy and Management, University of North Carolina–Chapel Hill, Chapel Hill, North Carolina, USA Accepted author version posted online: 01 Apr 2014.Published online: 29 Jul 2014.

To cite this article: Deborah Paone (2014) Using RE-AIM to Evaluate Implementation of an EvidenceBased Program: A Case Example From Minnesota, Journal of Gerontological Social Work, 57:6-7, 602-625, DOI: 10.1080/01634372.2014.907218 To link to this article: http://dx.doi.org/10.1080/01634372.2014.907218

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions

Journal of Gerontological Social Work, 57:602–625, 2014 Copyright © Taylor & Francis Group, LLC ISSN: 0163-4372 print/1540-4048 online DOI: 10.1080/01634372.2014.907218

Using RE-AIM to Evaluate Implementation of an Evidence-Based Program: A Case Example From Minnesota

Downloaded by [Wayne State University] at 09:19 26 November 2014

DEBORAH PAONE Health Policy and Management, University of North Carolina–Chapel Hill, Chapel Hill, North Carolina, USA

The RE-AIM framework was used to evaluate the implementation experience of 14 organizations in Minnesota over 4 years as they adopted an evidence-based intervention for spousal caregivers. Purpose: To examine the implementation experience and describe sustainability issues. Methods: Mixed-methods using data from qualitative interviews, cost data, and caregiver experience data. Findings: Although the intervention was valued, most organizations were not able to continue providing the program. Conclusion: The results indicate a need for further translation or technical support to identify readiness and prepare the interventionist to offer the program, more robust recruitment methods, and additional funding strategies. KEYWORDS implementation, evaluation, translation, sustainability, caregiving, dementia, RE-AIM

BACKGROUND There is often a long time between development of an innovation and sustained implementation in practice settings (Ginexi & Hilton, 2006). Even when practitioners are aware of the innovation and the job of translation for deployment from laboratory to practice setting has been achieved, effective implementation is uncertain (Ginexi & Hilton, 2006; Glasgow, Vogt, & Received 11 May 2013; revised 18 March 2014; accepted 18 March 2014. Address correspondence to Deborah Paone, Paone & Associates, LLC, 10413 Rhode Island, Minneapolis, MN 55438, USA. E-mail: [email protected] Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/wger. 602


Evaluating Implementation of an Evidence-Based Program

603

Boles, 1999; Green & Glasgow, 2006; Klein & Knight, 2005). Innovations require additional organizational and staff capabilities to implement, and protocols may not fit the real-world complexities or challenges of practice settings (Glasgow, Klesges, Dzewaltowski, Bull, & Estabrooks, 2004). Capacity-building at several levels may be a step that is missed in preparing for implementation (Fixsen, Naoon, Blase, Friedman, & Wallace, 2005). Ensuring organizational fit relates to both the program and the environment. Research on organizational change and program adoption typically identifies factors such as leadership, organizational climate, staff buy-in, and acceptability to the consumer, patient, or client as important for success (Klein & Sorra, 1996; Simpson, 2002). Greenhalgh, Robert, MacFarlane, Bate, and Kyriakidou (2004) also described a good “innovation to system fit” as one where the existing values, norms, strategies, goals, skill mix, supporting technologies, and ways of working are aligned (p. 608). Russell Glasgow and colleagues developed a framework for advancing the effectiveness of translating research to practice and to better evaluate the public health impact of an intervention. This RE-AIM framework has five elements: reach, effectiveness/efficacy, adoption, implementation, and maintenance (Glasgow et al., 1999). The RE-AIM elements are described as follows (www.re-aim.org): ●

●

●

●

●

Reach refers to the number and characteristics (representativeness) of individuals who participate in a given program/intervention. Effectiveness refers to the impact or effect observed from the program/ intervention. Adoption refers to the number and characteristics (representativeness) of settings or staff members who initiate the program/intervention. Implementation refers to how the program protocol is followed and how the program is established at the organization or setting level. Maintenance refers to the extent to which the program becomes part of the routine practices of the organization or practitioners.

Glasgow and colleagues recommend that each of these five RE-AIM elements be examined prior to launching intervention efforts. The framework can also be used to evaluate results. It has been used extensively since it was introduced—the RE-AIM website lists 237+ study publications using this framework. The Centers for Disease Control and Prevention (CDC) began promoting the RE-AIM framework in 2007 as a resource to encourage “research and practice communities [to] work together to translate evidence-based programs and policies into widespread practice” (CDC, 2008). In the field of aging services, The National Council on Aging (2007) and the US Administration on Aging (Substance Abuse and Mental Health Services Administration, 2013) have promoted use of the RE-AIM framework to


604

D. Paone

encourage service providers and program leaders to carefully select, plan, implement, and assess effective prevention and health promotion programs and practices for older adults. The RE-AIM framework can be used to evaluate impact at the individual, population, or organizational level (Dzewaltowski, Glasgow, Klesges, Estabrooks, & Brock, 2004). Examining program results using these five RE-AIM elements can also assist in determining whether external validity—that is the degree to which the effect observed from the intervention in a specific study environment can be generalized to different populations or settings—is achieved (Glasgow, Lichtenstein, & Marcus, 2003). The RE-AIM framework was used to evaluate Minnesota’s experience in implementing the New York University Caregiver Intervention (NYUCI) over 5 years. This intervention was designed to support spousal caregivers of persons with Alzheimer’s disease (Mittelman, Epstein, & Pierzhala, 2003). It was developed by clinician researchers Mary Mittelman, DrPH, and Cynthia Epstein, LCSW. The NYUCI was tested through two randomized controlled trials that demonstrated effectiveness in maintaining caregiver well-being, improving reaction to problem behaviors, increasing caregiver satisfaction with social supports, and decreasing depression (Mittelman, Roth, Clay, & Haley, 2007). The program is comprised of one initial caregiver counseling meeting, four family sessions, and a subsequent caregiver counseling meeting, with additional caregiver consultant time for screening, assessment/re-assessment, ad hoc calls, e-mail or telephone communication, information and referral, caregiver support group participation, and other support to the caregiver and family (Mittelman et al., 2003). As part of the Alzheimer’s Disease Support Services Program, the Minnesota Board on Aging produced a manual in 2010 for those implementing this intervention within the State of Minnesota to guide implementation efforts and share learning to date (Mittelman, Epstein, Paone, & Walberg, 2010). According to a collaborative report called the NYUCI Implementation Guide (Rosalynn Carter Institute for Caregiving, 2013), the program (a) provides education and information for caregivers and families to increase their understanding of how to cope with the disease, (b) enhances the caregiver’s support network by involving extended family and friends, and (c) delays nursing home placement, because the spousal caregiver is more supported in his or her caregiving role. In Minnesota, the purposes of the implementation project were to (a) improve the capability and capacity of the Minnesota spousal caregivers to withstand the difficulties of caregiving through improving familial support and (b) embed the intervention into the services provided by the Minnesota project sites and potentially into the Minnesota aging network as a whole. A simplified logic model presents inputs, target population, antecedent or mediating variables, intermediate outputs, and expected outcomes for this intervention (Figure 1). The logic model identifies the NYUCI program



605

FIGURE 1 NYUCI in Minnesota, simplified logic model.

components and inputs, with the primary input being the interventionist— the Memory Care Consultant. Mediating variables that could affect the outcomes observed (at the individual level) were assumed to include: the age and health of the caregiver, the number of family and friends supporting the caregiver and the level and type of support they provided, and the service availability in the area (e.g., access to home care, respite, transportation, or other services to assist the caregiver). In Minnesota, the NYUCI program is called Family Memory Care (FMC). Implementation began in 2007 through a grant from the US Administration on Aging to the Minnesota Board on Aging. The program sites were selected through invitation. From 2007 to 2012 a total of 14 organizations worked to implement the program. Four sites began in 2007, another five sites in 2008, and the final five sites in 2010. The types of organizations serving as program sites included: nonprofit community-based organizations focused on services for older adults and caregivers (N = 7), hospital-sponsored service program for seniors (N = 2), clinic-based programs (N = 2), county-based program (N = 1), nursing facility/assisted living-based program (N = 1), and Alzheimer’s Association chapter (N = 1). Three of the 14 program sites were located in the Twin Cities metropolitan area. Five were near larger cities (Duluth, St. Cloud, Mankato) and the rest were located in rural areas of the state. Eighteen FMC Consultants were trained in the program protocol and in all of the tools and processes. The FMC Consultants came to the program with a variety of educational and professional backgrounds. Four of the Consultants were licensed clinical social workers, two held a master degree in social work, and two were working toward a master degree in social work

606

D. Paone

or gerontology. Two of the Consultants were trained therapists (occupational or physical therapy). One was a registered nurse. One Consultant held a master degree in counseling. The other Consultants held bachelor degrees in a variety of fields. One was certified in life coaching. Many had worked with seniors and caregivers for years in a professional capacity. The criteria for participants (caregivers) to enroll in the FMC program in Minnesota were as follows: ●


●

●

Spouse caregiver (or partner) must live with a spouse (or partner) with Alzheimer’s disease in the community, Spouse must be the primary caregiver and have no physical or mental conditions that would prevent his/her participation in sessions, There must be at least 1 family member at each of the four family sessions.

This article reviews the implementation experience—particularly focusing on how the program was embedded—using the RE-AIM elements. The examination period was from October 2007 through September 2012. The terms program and intervention are used synonymously.

METHODS This was a process evaluation using mixed methods. The purpose of the evaluation was to observe, document, and analyze the implementation experience of 14 Minnesota organizations (program sites), and determine what issues or challenges arose. The primary unit of analysis for this examination was the program site. The evaluation focused on: ●

●

● ●

Whether/how program sites were able to follow the intervention protocol set forth by the researchers who created the NYUCI intervention, Whether/how the intervention was becoming embedded into the program sites and into the practices of the interventionists (maintenance/ sustainability of the program), The cost to produce the intervention in the Minnesota program sites, and The perceived value of and satisfaction with the program by caregivers.

The evaluation was conducted separately from fidelity monitoring of participant outcomes (that work was conducted by the original researcher, Dr. Mittelman). The Administration on Aging had encouraged the use an independent evaluator to study implementation issues and to conduct a cost analysis—such as is described in this Minnesota experience. Prior to program launch, the following assumptions were made by the evaluator about the factors that would impact effective implementation and sustainability at the organizational level:


607


1. The perceived value of the FMC program by spousal caregivers and family members who completed the program would impact implementation success and program sustainability (maintenance). 2. The ease or difficulty of implementing the intervention at the program site level would impact sustainability/maintenance of the program by that site. 3. The acceptance and perceived value of the intervention by the FMC Consultants based on their experiences would impact implementation success as well as sustainability/maintenance of the program by that site. 4. The cost of the intervention and likelihood of reimbursement or funding to cover that cost would impact sustainability/maintenance of the program by that site. Data sources for the examination included (a) baseline site visits to each program site where one-on-one key informant interviews were conducted and other information about infrastructure, usual care processes and tools, and site amenities was observed; (b) quarterly reports generated by the FMC Consultants that provided narrative information on progress, barriers, and strategies using the 5 RE-AIM elements; (c) cost reports provided by each program site; (d) periodic key informant interviews and electronic surveys primarily with/from the FMC Consultants but also with the organizational leadership and with Area Agency on Aging (AAA) managers; (e) data reports generated from the State agency; and (f) state-required regional conference calls with the FMC program sites and AAAs. The FMC Consultants provided their narrative reports both to the evaluator and to the project director for the program. Key informant interview and electronic survey data were provided only to the evaluator. All information was entered into several master spreadsheets, including one for cost information, one for narrative reports, and one for key informant interviews. Qualitative data were grouped and coded by RE-AIM element. Themes emerged from the interviews and reports from each site. Review of responses over time (12 quarters) reinforced these themes. Subsequent data collection via telephone interviews allowed for incorporation of additional questions to explore these themes in more depth.

FINDINGS RE-AIM Element: Reach Between October 2007 and June 2012, 137 Minnesota caregivers were identified as having completed the core components of the FMC program. Caregivers were primarily spouses (as per the selection criteria), although an exception was made for one site that was reaching out to African American caregivers to allow for the many primary caregivers who were not spouses but provided direct care as a live-in unpaid care partner. Caregivers ranged in age from 45 to 89. One site had many younger-onset caregivers enroll in


608

D. Paone

the program. Almost all (95%) of the caregivers were White, non-Hispanic. Sixty-five percent of caregivers had been married to their spouse (the person with Alzheimer’s disease) for 36 years or more. Conversely, 13% had been married less than 5 years. Although all of the program sites actively pursued a variety of marketing and recruitment strategies to make the FMC program known to communities and caregivers, the total enrollment of 137 caregivers was less than was expected by program sites and state agents. Targets for enrollment at program launch were set at 51 caregivers initially (four program sites) with an additional 200 families projected from the added program sites. Key barriers to recruitment included (a) the eligibility criteria and (b) caregiver reluctance to enroll in a program requiring what was perceived as a hefty time commitment. The majority of the caregivers who completed the intervention were enrolled in three of the 14 program sites. These three program sites were in the initial group that began implementation efforts in 2007, and accounted for more than half of all caregivers completing the program in Minnesota during this time period.

RE-AIM Element: Effectiveness Caregivers and family members’ perspectives on value of the program. Caregivers and family members completing the FMC program were invited to participate in an anonymous survey at the end of their final session. Out of the 137 completer caregivers, 105 returned the Caregiver Experience Survey to the evaluator by September, 2012 (77% response rate). Caregivers reported a high level of satisfaction and high degree of perceived value from this program/intervention. Out of 105 responding caregivers, 85 (81%) said that participating in the program was very helpful to them. Another 17 caregivers (16%) said that this was somewhat helpful, and only 2 caregivers were neutral as to the helpfulness of the program (Figure 2). Write-in narrative responses indicated that the caregivers felt the program:

● ●

●

Provided crucial education about Alzheimer’s disease, Improved their understanding and ability to cope with problem behaviors, and Fostered involvement by family members or others to support them in their caregiving role.

Many caregivers reported that they had not received help prior to enrolling into the program (Figure 3). Table 1 provides a sampling of the comments received from the spousal caregivers. A key part of this intervention was


609

Question: Overall, how helpful or valuable would you say participating in the program was to you? Somewhat = 16% Very = 81% Overall Helpfulness of NYUCI (Family Memory Care), N = 105

Somewhat helpful 17 16%

2 0 2% 0%

1 1% Very helpful 85 81%

Very helpful Somewhat helpful Neutral

Not applicable

N = 105 responses as of September 1, 2012

FIGURE 2 Overall caregiver experience with the NYUCI in Minnesota.

Prior Participation in Services Had you participated in or received services (described) prior to this program?

64

70 Number of Responses


Not helpful

60 50 40

41

61 %

30 20

39%

10 0

Yes

No

FIGURE 3 Caregivers’ prior use of similar service components.

increasing the active involvement and supportive help by extended family (particularly the adult children) and close friends. These informal sources of support can be critical to extending the ability of the primary caregiver to continue the caregiving role. There were 153 family members (associated with the 105 spousal caregivers) who also participated in FMC and who returned a Family

610

D. Paone

TABLE 1 Selection of Caregivers’ Written Responses to Caregiver Experience Survey (N = 137)


“The Program’s . . . impact on me was . . .” Helped me realize what was actually happening—and helped me to stay more calm, be more prepared. Made me more aware of how widespread this problem is and the need for more research to lead to a cure. Educated me more about Alzheimer’s Changed work schedules when I could. Needed to rely on son A big help. Being able to bring out and discuss long term care with my family. Which had been taboo. Advised me as to what to expect in the future. Accepting the disease My realizing how my anger over this disease showed and that is changing It helped me look at things more objectively Know I can get help when we need it–appreciated advice, ideas on safety I learned lots. I’m not alone. There are avenues available to seek help or just vent Gave me more understanding of the disease. Helped with coping I had someone to talk to outside my family and learned more about Alzheimer’s. I didn’t feel so alone. I’m able to attend church & some other meetings and I’m hoping to get back into a exercise program. All gained valuable information. It helped me understand what I will have to do in future More understanding It’s a release of emotions to talk to someone who understands. I learned how to cope with the Alzheimer’s. I was a basket case when I first saw _____________ [Caregiver Consultant]. She listened. She cared. She was available to me by phone, by email, when I needed help. She offered many helpful suggestions. She became a “safe island” that I could go to. Helped me manage stress better. Offered encouragement and support to my family & me. I was helped by understanding I was going down in healthcare I learned to accept the behavior associated with this disease and also how to handle it better. Increased my knowledge, gave me a much bigger support group, gave me more confidence, and helped me reach out to others. I became more accepting of my loved ones condition. I learned to understand more about my husband’s disease and why he did things he did do. I learned to let go of anger much more.

Experience Survey by September 2012. Reponses from these family members and friends to fixed-response options indicated that the program was very helpful in: ● ● ●

Discussing problems related to caring for their relative or friend (86%), Handling changes in condition or behavior (65%) and Learning more about Alzheimer’s disease (64%).

Ninety-four percent of these family members said the program was very or somewhat helpful in improving their ability to participate in assisting their relative to care for the person with the disease (Figure 4).


611

How much have the family meetings & this program helped you…improve your ability to participate in helping the primary caregiver of the person 1% w/Alzheimer's? Very helpful

4% 1%

Somewhat helpful Neutral

30%

64%

Not helpful at all


Does not apply

N = 153 responses as of August 31, 2012

FIGURE 4 Family and friends’ responses to the NYUCI in Minnesota.

Program effectiveness for participants was evaluated periodically by Dr. Mittelman and the State project director by comparing the baseline, 4-month, 8-month, and 12-month scores for each caregiver. The consultants collected this information from caregivers at each of these time periods using standardized assessment instruments including: the Global Deterioration Scale, the Center for Epidemiologic Studies Depression Scale (CES-D),1 the Revised Memory and Behavior Problems (RMBC) instrument,2 and the Montgomery Borgotta Burden Scale,3 and by tracking the change in the caregivers’ reported social networks. As of March 2011 (with 93 caregivers completing the program), comparison of these scores showed continued deterioration in the condition of the person with Alzheimer’s disease, but improvement in the caregivers’ assessment scores in depression, reaction to problem behaviors, and stress burden level. Caregivers also reported an increase in their social support networks of family and friends. These data are reported elsewhere in publications of the State of Minnesota Board on Aging and by Dr. Mary Mittelman (Mittelman & Bartels, 2014). FMC Consultant perspective on value of the program. At the close of the study period, all FMC Consultants were interviewed about their perspectives on the effectiveness and value of the program. These consultants universally expressed support for the program, stating that they believed it provided an important service to the caregiver and his/her family and improved the 1

See http://cesd-r.com/ for the current instrument. See https://www.alz.org/national/documents/c_assess-revisedmemoryandbehcheck.pdf for the RMBC instrument. 3 See http://strokenetworkseo.ca/sites/strokenetworkseo.ca/files/useruploads_global/Burden%20 Scale.pdf for the Montgomery Borgatta Caregiver Burden Scale. 2

612

D. Paone


caregivers’ capabilities to cope with the effects of Alzheimer’s disease. The strongest endorsement came from the Consultants who had been providing the program the longest (4+ years). Within their quarterly narrative reports and in a final summary questionnaire, consultants specifically noted the value of engaging extended family members through the defined process as a key component of its success. Consultants described the program as a “strategy,” “method,” or “approach” to help family members work together to support the primary spousal caregiver. They said that this method/approach was more comprehensive than their traditional approach and more robust that the usual care services and supports offered to caregivers.

RE-AIM Element: Adoption As described earlier, four program sites began implementing FMC in 2007, another five sites began in 2008 and an additional five sites began in 2010. Organizations implementing the program differed from one another on a number of characteristics including: primary purpose and mission for the organization, geographic location, size, familiarity with issues presented by caregivers of persons with Alzheimer’s disease, and existence of caregiver services (usual care). Many were small organizations. Twelve out of the 14 program sites had existing caregiver support services that were offered through the organization. These caregiver support services included: information and referral, one-on-one meeting with the caregiver (usually once or twice with follow-up by phone or e-mail), and caregiver support group(s). Only one organization (the Alzheimer’s Association chapter) already had usual care caregiver services that were specifically focused on caregivers of persons with Alzheimer’s disease. Table 2 provides a summary of program site characteristics. At start-up, all of the Minnesota sites had basic tools and resources at the disposal of the FMC Consultant, including office space, computer, telephone, and some administrative support (e.g., receptionist, human resources, payroll accounting, database and I.T. support, and/or supervisory support) for this program. Most organizations did not hire a new staff member (five of the 18 consultants were new hires). Program sites typically reassigned an existing staff member to provide the intervention and participate in the initiative. As mentioned, 18 FMC Consultants were trained in the program protocol. Each FMC Consultant was assigned to this project on a part-time basis, although many worked full time for their organizations and had multiple additional position duties. Across the 18 consultants, the proportion of time allocated to this program ranged from 10% time (4 hr/week) to 80% time (32 hr/week) with most logging about 14–16 hr per week on FMC. Time requirements to implement this program included training in the protocol and assessment tools, participation in ongoing education and mentoring, documentation requirements, required grant-related meetings, reporting requirements and development of skills related to working

613

County health department (1 FMCC trained) Clinic – Memory care program (2 FMCCs trained) Medical center with senior program (1 FMCC trained) Community-based – located in senior housing (1 FMCC trained) Nursing-facility based caregiver support service (2 FMCCs trained) Community-based – located in senior center (1 FMCC trained) Community-based support program (1 FMCC trained) Medical center (1 FMCC trained)

Clinic + home care agency (1 FMCC trained) Nonprofit with services for seniors ∗ (1 FMCC trained) Community-based – located in hospital (1 FMCC trained) Community-based – located in neighborhood (1 FMCC trained) Community-based - Adult day center (2 FMCCs trained)

#2 #3

#10

Urban, Twin Cities

Rural, north region

Rural, north region

Urban, Twin Cities

Semi-rural, central region Semi-rural, south region Rural, south region

Rural, south region

Rural, central region

Rural, central region Semi-rural, central region Semi-rural, south region Rural, north region

Twin Cities (urban, suburban)

Location

Small

Very small

Very small

Medium

Large (for institution) Small

Very small

Very small

Medium

Large (for institution) Very small

Medium Small

Medium

Size

Yes

Yes

Yes

Yes

No

No

Yes

Yes

Yes

Yes

Yes

Yes Yes

Yes

Existing Caregiver Services (At Baseline)

No

No

No

No

No

No

No

No

No

No

Yes, extensive focus on Alzheimer’s disease No All diseases affecting memory targeted No

Alzheimer’s Disease as a Specific Service Focus (At Baseline)

Note. In terms of size of the organization, “very small” is defined as having less than 5 employees, “small” as an organization with 5 to 25 employees, “medium” as having between 26 and 200 employees, and “large” as having more than 200 employees. FMCCs = FMC Consultants.

#14

#13

#12

#11

#9

#8

#7

#6

#5

#4

Alzheimer’s Association Chapter (2 FMCCs trained)

#1

Site

Organizational Type and Number of Memory Care Consultants trained

TABLE 2 Minnesota Organizational Characteristics



614

D. Paone

with families and caregivers of persons with Alzheimer’s disease. It also included the time required to recruit participants, schedule appointments, and meet with caregivers and families (sometimes requiring long travel times). These FMC Consultants received additional training and education (apart from the NYUCI protocol and skills training) from experts contracted by the Minnesota Board on Aging. Topics for this additional training included recognizing the signs and symptoms of Alzheimer’s disease, understanding disease stages, effective strategies and techniques for facilitating groups, understanding family systems theory and family dynamics, effective communication methods and techniques, and care planning and monitoring. The size of the market areas that the FMC Consultants covered varied. Most set a 1-hr driving time radius around their site as the general geographic region to target. Some consultants conducted the caregiver and family sessions at their office location; others met individuals and families in their homes or at other locations (e.g., local restaurant, private meeting space in community center). As expected, the more rural settings required extensive time behind the wheel by the consultant. For the rurallylocated FMC Consultants, this meant less time available to perform other duties.

RE-AIM Element: Implementation In this implementation experience, the availability of the research team to provide technical assistance to the FMC Consultants was invaluable. Consultants initially had questions about eligibility of caregivers who were potentially appropriate for the program. The availability of the research team helped maintain fidelity to the original research, since there was not an implementation manual or written guide at the time.4 Even with clear selection criteria and extensive published information about the intervention (including books, articles, and presentation materials), the Minnesota consultants confronted many nuances related to the selection criteria, implementation issues, and ongoing strategies which required consultation with the research team. Of the 18 trained FMC Consultants, 16 enrolled spousal caregivers into the program and completed the intervention with at least one caregiver by June 2012. One program site (and corresponding consultant) dropped out of the initiative prior to completing the intervention with any caregivers. In a second organization, the consultant discontinued employment, however another trained FMC Consultant in the region picked up that location, thereby covering two sites. 4 An implementation manual was prepared in 2010 and available to program sites in 2011 through the Minnesota Board on Aging.



615

Staff capabilities, resources required. Consultants and their host organization struggled with program implementation requirements. Despite extensive field experience, many of the FMC Consultants had to climb a steep readiness curve to implement the program. The program was more complex and required more time and skills than originally anticipated by either the organizations or the state. Consultants needed knowledge and skills in caregiver assessment, care planning, family systems and group facilitation, communication strategies, understanding Alzheimer’s disease symptoms, accessing local resources, and providing guidance on options for addressing issues at home. During family sessions, the consultants worked with caregivers and families to help them respond to psycho-social, emotional and behavioral issues of the person with the disease, caregiver, and extended family members. Consultants helped caregivers to deal with conflict among family members and provided encouragement to caregivers to take action or make informed decisions about their situations. There were many difficult issues presented by families during the 4- to 8-month course of the intervention. Facilitating the family sessions and handling family dynamics with professional skill and compassion was initially challenging. This was particularly true for those who had little background or training in family systems, group facilitation, or family counseling. As time went on, the breadth/depth of caregiver and family issues became more apparent, necessitating the need for additional training, role-playing, mentoring, and other support for consultants to effectively facilitate the family sessions. The State provided a clinical consultant (clinical neuropsychologist) who provided regular case reviews to monitor and mentor the FMC Consultants. Subsequently, the State suggested that organizations who wished to implement this program hire a licensed social worker with counseling certification or equivalent skills to serve as the interventionist. The State also set a requirement for clinical supervision of the consultant. In response to these and other translation issues, the NYUCI program protocol, assessment, training, and reporting tools were modified or enhanced. Minnesota consultants described the substantial training and clinical consultant mentoring that was provided through the State as “extremely important” to them for program adoption and ongoing implementation and to ensure adherence to the NYUCI program protocol. This training and mentoring by the NYUCI researchers likewise helped consultants “develop skills and build confidence.” Consultants reported that they brought these skills to their usual care practices with other caregivers and families with whom they came in contact—thus enhancing support offered by the consultant to all caregivers they served (whether participating in FMC or usual care). The extended nature of the FMC intervention (lasting 4–8 months for the key components of the program and then up to 3 years for periodic reassessments) required sustained effort by the FMC Consultant. The process was one of building a relationship of trust from which the caregiver and


616

D. Paone

family members were able to see the value of the program, take some steps toward addressing some of their caregiving issues, and move through the components of the intervention. Many FMC Consultants changed their work hours to accommodate client and family schedules. Consultants across sites reported similar experience in terms of time requirements to conduct the components of the program. On average, the initial assessment took between 90 and 120 min (not including travel or scheduling time), the individual initial caregiver counseling visit lasted 60 to 90 min, family sessions took from 90 to 120 min, and ad hoc calls varied by caller and by subject/issue. Overall, most of the consultants reported receiving very few ad hoc calls. Consultants said that they would initiate calls to caregivers to check in with them in between family sessions. On the plus side, the FMC training and skills development assisted in enhancing the usual care offered to caregivers not enrolled in the program. Consultants provided detail in their reports about how FMC was more robust than the usual care offered to caregivers. Consultants reported that FMC required: ● ● ● ●

● ● ●

More comprehensive assessment; Better care planning; More targeted attention to caregivers’ goals and issues; Better written documentation on caregivers’ and families’ challenges, progress, etc.; More structured staff process in working with caregiver and family; Greater family involvement; and Better outcomes tracking.

Costs of implementation. In the initial grant application, costs of the FMC program were projected to be about $3,500 per caregiver. The actual average cost per caregiver was somewhat more than this estimate. Costs of the program were calculated based on quarterly financial reports from program sites provided to the evaluator over 3 years. The program costs were primarily the salary expenses of the FMC Consultants. Program costs differed widely from site to site. The more rural sites had higher travel expenses related to logging more miles to get to participants’ homes, attend trainings, and conduct marketing efforts but tended to have lower salary costs. Aggregating the information from all program sites resulted in an average cost per caregiver for the whole program of $3,488 (Year 1), $3,722 (Year 2) and $4,520 (Year 3) in Minnesota. Year 1 costs represent the average cost experienced across four program sites, pertaining to the efforts to enroll 58 caregivers of whom 42 were being served (16 caregivers disenrolled between January 2008 and April 2009). The Year 2 costs represent the average costs reported by nine program sites, and the costs in Year 3 represents the average costs reported by all of the operating program sites.



617

The cost per caregiver was sensitive to volume (caseload)—most of the costs are fixed (salary of consultant). Consultants needed to focus their initial implementation work on training and marketing/outreach. The time and effort to find caregivers who met the eligibility criteria was extensive. Several program sites enrolled only a few caregivers in their initial year of implementation. Time spent in recruitment, intake, assessment, and counseling of caregivers who started but did not complete the program remained in the cost analysis as well. Therefore, the total program site costs were spread over a small number of people. The first program sites served more caregivers (per consultant) than the later program sites. This is why the average costs rose year to year. In addition to tracking costs, the consultants tracked how they spent their time. Time logs identified activities and hours by (a) direct service and (b) administration, reporting, and training. In analyzing the time reports of the consultants, it was clear that a hefty proportion of time was spent on grant-related activities that would not be experienced in an operational “steady state.” The evaluator determined that from 35% to 50% of the time that the consultants were spending related to start-up efforts in learning the program protocol or to grant requirements (e.g., regional meetings, grant reports, training meetings, fidelity monitoring, and other activities). Discounting total costs to take into account activities that were not operationally related to program delivery resulted in an estimated steady-state average cost per caregiver ranging from $1,744 to $3,164 across the 3 years examined. More details on the cost analysis methodology and results are available through evaluation reports provided to the State of Minnesota. FMC Consultants reported that it is unlikely that the program costs could be paid for out-of-pocket by consumers or families, although some families might be able to contribute at a modest level. The program cost was considered high in relation to the costs of other caregiver services provided by these organizations. However, when compared to the costs of a nursing home stay, the program costs are modest.5 Delay of nursing home placement was one effect found from the original randomized controlled trial of the NYUCI.

RE-AIM Element: Maintenance One grant objective was to implement and sustain the FMC program in Minnesota program sites. However, sites faced a number of challenges with regard to maintaining the program. Challenges included:

5 Minnesota nursing home costs for a semiprivate room ranged from $130 to $150+ per day and would be higher for a special memory care unit. See http://www.minnesotahealthinfo.org/compare/ nursinghomes.html

618

D. Paone


1. Continued struggles in some organizations with the fit between the FMC Program and the organization’s culture, focus/orientation, and typical services; 2. Difficulty in allocating sufficient staff time and other resources required for training, recruitment, program promotion, program reporting, and for scheduling caregiver assessments and family sessions; and 3. Lack of natural demand for the program—caregivers continued to be reluctant to participate. Program fit with organization. Most of the organizations that implemented FMC under this grant initiative had a strong history of providing caregiver and aging services in their communities. The organizations typically had a number of services and programs that were focused on family and spousal caregivers (e.g., caregiver counseling or coaching, caregiver support groups, etc.). However, a few of the program sites did not have this history or service complement. These program sites (home care agencies, clinics, hospital-based programs) were new to providing caregiver support as a defined service or program and struggled with the fit between FMC program service focus and the organization’s typical focus and operational norms. These organizations provided medical services to patients and were accustomed to receiving reimbursement for services rendered through patients’ insurance plans. These medically-focused facilities had little prior experience with caregiver services. The unbillable world of caregiver support and aging services through grant programs was unfamiliar territory. These organizations struggled with how to incorporate FMC into their organizations. For example, one hospital that was participating as a host site organization moved the FMC Consultant from one department to another three times in the course of 18 months, trying to find the right fit for the program within its operational structure. Another home-care/clinic organization found it hard to designate time of a key staff person (a nurse) to the program and eventually dropped out. Staff time. Consultants struggled with the time that this program demanded. Time requirements included the time to be trained in the protocol and assessment tools, as well as participate in ongoing mentoring, the documentation requirements, the development of knowledge and skills related to working with families and addressing some of the more serious issues faced by caregivers of persons with Alzheimer’s disease. It also included the time required to recruit participants, work with caregivers and families (sometimes requiring long travel times), and participate in ongoing grant project related meetings and reporting. These duties were to be performed in addition to juggling their other position responsibilities as defined by their organizations. Recruitment/market demand. Recruitment efforts were primarily at the local program site level. The Memory Care Consultants worked to build referral relationships and program awareness in their respective market areas


619


and conducted outreach to individual caregivers. Marketing and outreach required sustained effort. Most caregivers needed multiple interactions before deciding to enroll. In comparison to the original New York experience, Minnesota program sites found that recruitment was a lengthier and more complex process than originally anticipated or experienced in the randomized control trials. Because many of these regions already had some type of usual care caregiver service (either through the participating organizations or through other community organizations), it is possible that caregivers’ needs were met by these other services. Key barriers included (a) the eligibility criteria and (b) caregiver reluctance to make the time commitment required and/or to involve their adult children. Other reasons for refusal included: ● ● ● ● ● ● ● ● ●

Not interested at this time, will consider in the future; Lived too far away to attend meetings; Believed that they had things “under control;” Don’t want to bother the adult children; they are “too busy;” Worried about being judged on caregiver care; Had sudden health issues or emergencies of their own; Believed that the intervention had come too late; Were too tired or worn out to participate; and Can’t do even this helpful thing—too much to handle.

Two of the program sites were particularly successful in recruitment— the FMC Consultants at these organizations accounted for more than half of all caregivers enrolled in the 5-year period. These organizations had a longstanding market presence around caregiver support services and memory care support. Consultants that were successful appeared to have several traits: understanding of the norms and cultural preferences of caregivers in the area, strong existing relationships with local providers, active and resourceful in identifying potential referral sources, and persistent in promoting the program communitywide. As part of the final review of implementation experience and lessons learned, 11 Memory Care Consultants participated in a brief questionnaire. The consultants were offered response options that had been crafted from a review of their quarterly narrative reports—thus the options represented input of the consultants over 12 quarters (3 years). Catalysts for the program included strong interest and commitment by the FMC Consultants in the intervention, and very strong personal or professional motivation for serving caregivers–particularly those who are caring for a person with Alzheimer’s disease. Catalysts also included the skills and resourcefulness of the consultants for recruiting and outreach and their skills in building a relationship with caregivers. Additional catalysts included the extensive training provided by the Minnesota Board on Aging that offered


620

D. Paone

access to the original research team as well as education and training from a variety of expert sources within the state, including clinical experts in Alzheimer’s disease. In the last 2 years, a Minnesota-based clinician provided regular consultation to the FMC Consultants through regional meetings and follow-up telephone meetings. The biggest challenges that the Memory Care Consultants reported through this survey were: (a) finding caregivers who met the eligibility criteria for the program, and (b) getting caregivers who met the criteria to actually enroll in the program. As of June 1, 2012, 11 sites had discontinued FMC program as a service offered by their organizations. However, some of these organizations were changing how they offered usual care to caregivers to incorporate elements of enhanced assessment, engagement of families, and caregiver education. A few of the trained FMC Consultants left their organizations and reported that they hoped to offer the program again. During this time period, some organizations experienced significant financial difficulties and a few closed their doors entirely. Obviously, this affected program continuity. Despite the challenges, FMC Consultants reported that their skills, knowledge, and abilities related to working with other non-participating caregivers increased as a result of this program. That knowledge would stay with them.

DISCUSSION The translation and implementation of the NYUCI (FMC) program in Minnesota offers valuable insights. In revisiting the initial assumptions made—about value, ease of implementation, and cost experience—we found there was high perceived value, challenges in implementation (primarily related to extensive staff time and recruitment efforts required), and a cost per caregiver that is sensitive to volume (caseload). This examination found caregivers, family members, and consultants involved in the program strongly supportive. These individuals, closest to the program, saw the benefit. Many grant-funded resources (e.g., training, mentoring, monitoring) were provided through the State of Minnesota and by the original research team and this greatly facilitated program adoption and implementation. All Consultants, despite wide differences in organizational setting, professional background, or training/experience with caregiver services, were able to use the protocols. The FMC program had some implementation challenges, especially initially as training and tools were not fully translated. It required a heavy commitment to protocol training, participation in ongoing case consultation and monitoring, and very extensive outreach and recruitment to attract spousal caregivers who met the enrollment criteria. In addition, the extended nature of the program (lasting 4–8 months for the key components of the



621

program and then up to 3 years for periodic reassessments) required sustained effort on the part of the FMC Consultant for each caregiver. The process was one of building a relationship of trust from which the caregiver and family members were able to communicate and move through the intervention effectively. The average cost of the program (across sites) was slightly higher than anticipated, but not high when compared to a short stay in a nursing home. The cost per caregiver was sensitive to caseload, as most of the costs were the salary of the consultant. At the close of the evaluation period there was no natural source of funding for this program. The average costs per caregiver of $3,400 and higher would not likely be able to be covered by caregivers out-of-pocket. The Minnesota implementation experience represents part of the translation of the program for practice in the field. Several translational issues were not fully anticipated in Minnesota. As the program moved from the academic medical center practice setting in an urban environment (New York) to the rural and community-based environment in Minnesota, these came to light. The knowledge, skills, ongoing mentoring, assessment tools to be used, and recruitment tools/strategies all needed to be enhanced for this environment. Program intensity was underestimated, requiring staff and organizations to stretch to meet the requirements. Most of the organizations did not have staff members with the education or background needed. The requirements for this position are now better understood. Translation has been defined as the enhancement of an intervention using knowledge and information from potential users and agents from the field to adapt components for wider and efficacious use (Rabin & Brownson, 2012). Observers note that there is a tension between allowing for adaptation and maintaining fidelity to the tested protocol. In the area of medical research, the National Institutes of Health has identified two translational steps for interventions delivered in community and other delivery settings—bringing together interdisciplinary teams to integrate basic science and treatment approaches, and translating evidence-based treatments into service delivery settings and sectors in local communities (Landsverk et al., 2012; National Institutes of Health, 2006; Zerhouni, 2005). There is also a growing body of literature on implementation effectiveness. Wandersman and colleagues recognizes the quality of implementation as an important ingredient in dissemination success (Meyers, Durlak, & Wandersman, 2012; Wandersman et al., 2008). There are issues to consider in retooling for continued dissemination of this intervention. Some of these issues can be addressed at an organizational level, and some would need changes at a broader state or even federal level. These include:

622 ●

●

●


●

●

D. Paone

The dependency on the Memory Care Consultant staff member raises issues about staff selection, preparation and turnover, as well as how to maintain consistent service. There needs to be greater efforts around organizational selection, capacitybuilding, readiness, and ongoing support for organizations implementing this program. Training, education, and technical assistance needs are extensive, which includes ensuring expert consultation is available to ensure program fidelity. There was little evidence of a natural market demand for this service directly from caregivers (recruitment of spouse caregivers was the most difficult aspect of the program according to the FMC Consultants), nor from providers who had the opportunity to learn about the program but did not make referrals. This could indicate a structural barrier (e.g., no payment source, no referral mechanism) as well as an awareness issue. Cost of the intervention at a steady-state was high as a stand-alone service, but not when considering the cost of even a short stay in the nursing home. To the extent that this can be considered a service substitution for nursing home care (perhaps with additional hands-on help in the home to the primary caregiver) FMC could represent a cost saving to the state or other payers.

CONCLUSION Program, environment, and organizational factors emerged as very important in the implementation of this evidence-based practice in settings within Minnesota. Factors included organizational readiness and capability/capacity, program fit, and market readiness. Some of these factors may be linked. Challenges included lack of market demand and lack of program identity or differentiation from other usual care caregiver support services. The lack of an ongoing funding source to cover the costs of the program following the grant expiration was reported as a significant factor in sustainability. Although the program was highly valued by the consultants and by the caregivers and families who completed the program, this was not sufficient to maintain the program in many program sites. Given these findings, the following suggestions are offered: 1. When selecting program sites, organizational capabilities and program requirements must be well-matched and sites must have a very clear understanding of the staff capacity and resource requirements. Qualifications and requirements for consultants on education, experience, and skills are needed. Program sites must also consider how continuity would be maintained if it experienced consultant staff turnover.


623


2. Because a ready market/demand for the program did not appear to exist, then outreach, marketing, and recruitment efforts must be extensive and fully supported through many levels including at the organizational, regional, and state level. 3. A long-term funding or business strategy is needed. There must be an ongoing discernible funding source to cover direct costs of the program and/or a clear business model that has reachable targets related to service provision and likely payment for services rendered. There are policy implications, such as extending the definition of a covered benefit under Medicaid to include this program—likely an uphill battle. Especially within the final year of program implementation, these issues were paramount in considering sustainability. Each organization considered its capability to support the Memory Care Consultant role, the cost of the program with potential payer sources, and the recruitment challenges in attracting participants to the program. Additional factors that weighed in on the decision to continue included the availability and likelihood of ongoing (i.e., free) training and clinical expertise that would be offered through the state (Glasgow, Green, & Klesges, 2006; Glasgow et al., 2004). These recommendations are consistent with the recommendations offered by others using the RE-AIM framework to guide improved translation of evidence-based programs for practice settings (Dzewaltowski et al., 2004). There is wisdom in creating an implementation strategy that utilizes a variety of organizational settings, professionals, and market areas. One advantage of this approach is that the implementation experience offers a more realistic picture of how the program might operate and be maintained in the field (external validity). This provides a richer understanding of external validity and translational issues as compared to the understanding that would be gleaned when choosing only one type of organization or a single region in which to implement the program. The disadvantage is complexity. It may be harder to tease out the salient factors that impact implementation effectiveness. Because of the commitment by Minnesota to conduct an external implementation evaluation using the RE-AIM framework, this experience offers a view into readiness, implementation, and maintenance activities and issues. Minnesota’s experience provides important lessons for guiding future efforts to implement and embed evidence-based practices and programs in community, health care, and other settings. Finally, use of the RE-AIM framework provided a very useful guide for studying the implementation process. Setting up a logic model and then specifying metrics to assess each component of the framework (specific to the program being implemented) is a recommended approach.

624

D. Paone

FUNDING This project was supported in part by grant #90AE0323/01 and grant 90AZ2909/01 from the U.S. Administration on Aging, U.S. Administration for Community Living, U.S. Department of Health and Human Services. Grantees undertaking projects under government sponsorship are encouraged to express freely their findings and conclusions. Points of view or opinions do not, therefore, necessarily represent official Administration on Aging policy.


REFERENCES CDC. (2008). Assuring healthy caregivers, a public health approach to translating research into practice. Retrieved from http://www.cdc.gov/aging/caregiving/ assuring.htm Dzewaltowski, D., Glasgow, R., Klesges, L., Estabrooks, P., & Brock, E. (2004). REAIM: Evidence-based standards and a web resource to improve translation of research into practice. Annals of Behavioral Medicine, 28, 75–80. Fixsen, D. L., Naoon, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, the National Implementation Research Network. Ginexi, E., & Hilton, T. (2006). What’s next for translation research? Evaluation & the Health Professions, 29, 334–347. Glasgow, R. E., Green, L. W., & Klesges, L. M. (2006). External validity: We need to do more. Annals of Behavioral Medicine, 31, 105–108. Glasgow, R. E., Klesges, L., Dzewaltowski, D., Bull, S., & Estabrooks, P. (2004). The future of health behavior change research: What is needed to improve translation of research into health promotion practice? Annals of Behavioral Medicine, 27, 3–12. Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-toeffectiveness transition. American Journal of Public Health, 93, 1261–1267. Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327. Green, L. W., & Glasgow, R. E. (2006). Evaluating the relevance, generalization, and applicability of research: Issues in translation methodology. Evaluation & the Health Professions, 29, 126–153. Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly, 82, 581–629. Klein, K., & Knight, A. (2005). Innovation implementation: Overcoming the challenge. Current Directions in Psychological Science, 14, 243–246. Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy of Management Review, 21, 1055–1080.



625

Landsverk, J., Brown, C. H., Chamberlain, P., Palinkas, L., Ogihara, M., Czaja, S., . . . Horwitz, S. (2012). Design and analysis in dissemination and implementation research. In R. Brownson, G. Colditz, & E. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 225–260). New York, NY: Oxford University Press. Meyers, D., Durlak, J., & Wandersman, A. (2012). The Quality Implementation Framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50, 462–480. Mittelman, M., & Bartels, S. (2014). Translating research to practice: Case study of a community-based dementia caregiver intervention. Health Affairs, 33, 587–595. Mittelman, M., Epstein, C., Paone, D., & Walberg, D. (2010). A guide to implementing the New York University Caregiver Intervention and the Minnesota experience. St. Paul, MN: Minnesota Board on Aging. Mittelman, M., Epstein, C., & Pierzhala, A. (2003). Counseling the Alzheimer’s caregiver: A resource for health care professionals. Chicago, IL: AMA Press. Mittelman, M., Roth, D., Clay, O., & Haley, W. (2007). Preserving health of Alzheimer caregivers: Impact of a spouse caregiver intervention. American Journal of Geriatric Psychiatry, 15, 780–789. National Council on Aging. (2007). RE-AIM for program planning. Retrieved from http://www.ncoa.org/improve-health/center-for-healthy-aging/content-library/ re-aim-for-program-planning.html National Institutes of Health. (2006). NIH roadmap for clinical research. Retrieved from http://opasi.nih.gov/documents/NIHRoadmap_FactSheet_Aug06.pdf Rabin, B., & Brownson, R. (2012). Developing the terminology for dissemination and implementation research, In R. Brownson, G. Colditz, & E. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 23–51). New York, NY: Oxford University Press. Rosalynn Carter Institute for Caregiving. (2013). NYUCI implementation guide. Retrieved from http://www.rosalynncarter.org/UserFiles/NYUCI(1).pdf Simpson, D. D. (2002). A conceptual framework for transferring research to practice. Journal of Substance Abuse Treatment, 22, 171–182. Substance Abuse and Mental Health Services Administration. (2013). Older Americans Behavioral Health issue brief 7: Using the RE-AIM implementation framework to improve behavioral health. Retrieved from http://www.aoa.gov/ AoARoot/AoA_Programs/HPW/Behavioral/docs2/Issue%20Brief%207%20REAIM.pdf Wandersman, A., Duffy, J. Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., . . . Saul, J. (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41, 171–181. Zerhouni, E. (2005). Translational and clinical science—Time for a new vision. New England Journal of Medicine, 353, 1621–1623.

Using crowdsourcing to evaluate published scientific literature: methods and example.

Agitation Following Traumatic Brain InjuryA Case Example.

Using the likelihood ratio to evaluate allowable total error--an example with glycated hemoglobin (HbA1c).

Using implementation evaluation to uncover a child neglect prevention program.

Using transcriptomic tools to evaluate biological effects across effluent gradients at a diverse set of study sites in Minnesota, USA.

A proposed model to conduct process and outcome evaluations and implementation research of child health programs in Africa using integrated community case management as an example.

The value of improving failures within a cervical cancer screening program: an example from Norway.

Case-Taking, with an Example.

Reporting on the Strategies Needed to Implement Proven Interventions: An Example From a "Real-World" Cross-Setting Implementation Study.

Translating an early childhood obesity prevention program for local community implementation: a case study of the Melbourne InFANT Program.

A Community-Driven Implementation of the Body and Soul Program in Churches in the Twin Cities, Minnesota, 2011-2014.

Process and implementation of Audio Computer Assisted Self-Interviewing (ACASI) assessments in low resource settings: a case example from Zambia.

DNAseq Workflow in a Diagnostic Context and an Example of a User Friendly Implementation.

Using program evaluation to support knowledge translation in an interprofessional primary care team: a case study.

Using quality improvement methods to change surgical practice: a case example of pediatric soft-tissue abscesses.

Lacustrine responses to decreasing wet mercury deposition rates--results from a case study in northern Minnesota.

Adaptation of an Evidence-Based Colorectal Cancer Screening Program Using the Consolidated Framework for Implementation Research.

Early implementation of QbD in biopharmaceutical development: a practical example.

Analytic strategies to evaluate the association of time-varying exposures to HIV-related outcomes: Alcohol consumption as an example.

Using combined evidence from replicates to evaluate ChIP-seq peaks.

Genomic medicine implementation: learning by example.

Implementation of a national nursing standards program.

Improving physical activity program adoption using integrated research-practice partnerships: an effectiveness-implementation trial.

Implementation of a pain medication stewardship program.