This article was downloaded by: [New York University] On: 27 May 2015, At: 15:44 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Journal of Ethnicity in Substance Abuse Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wesa20
Assessing Treatment Fidelity in a Cultural Adaptation of Motivational Interviewing a
a
a
a
Christina S. Lee , Tonya Tavares , Ami Popat-Jain & Pamela Naab a
Northeastern University, Boston, Massachusetts Published online: 18 May 2015.
Click for updates To cite this article: Christina S. Lee, Tonya Tavares, Ami Popat-Jain & Pamela Naab (2015) Assessing Treatment Fidelity in a Cultural Adaptation of Motivational Interviewing, Journal of Ethnicity in Substance Abuse, 14:2, 208-219, DOI: 10.1080/15332640.2014.973628 To link to this article: http://dx.doi.org/10.1080/15332640.2014.973628
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [New York University] at 15:44 27 May 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Journal of Ethnicity in Substance Abuse, 14:208–219, 2015 Copyright # Taylor & Francis Group, LLC ISSN: 1533-2640 print=1533-2659 online DOI: 10.1080/15332640.2014.973628
Assessing Treatment Fidelity in a Cultural Adaptation of Motivational Interviewing CHRISTINA S. LEE, TONYA TAVARES, AMI POPAT-JAIN, and PAMELA NAAB
Downloaded by [New York University] at 15:44 27 May 2015
Northeastern University, Boston, Massachusetts
The efficacy of motivational interviewing (MI) for addictions is well documented. Grounding MI in social=cultural priorities may enhance treatment response. We evaluate the method of assessing competence using the motivational interviewing treatment integrity system (MITI) for standard MI and culturally adapted MI (CAMI) delivered to Latino heavy drinkers. Twenty audiotapes (MI, n ¼ 10; CAMI, n ¼ 10) were MITI coded by two raters unaware of treatment assignment. Inter-rater reliabilities were excellent (.78–.99) except for CAMI complex reflections, global ratings of empathy, and MI spirit. The MITI reliably evaluates MI and CAMI treatment fidelity. Future research should investigate lower reliabilities for MI global and complex reflections cross-culturally. KEYWORDS motivational interviewing, alcohol, cultural adaptation, Latinos, treatment integrity
INTRODUCTION Motivational interviewing is a directive, person-based counseling approach with proven efficacy in the treatment of addiction. The motivational interviewing emphasis on collaboration and on privileging autonomy to elicit thoughts and feelings about change is of great appeal to those who may have experienced oppression or discrimination (Miller & Rose, 2010), such as newly arrived immigrants or minorities. Poor treatment response A portion of the study findings were presented at the annual meeting of the Research Society on Alcoholism, 2013, and at the 2013 Inter-Professional Research Symposium, Bouve´ College of Health Sciences, Northeastern University. Address correspondence to Christina S. Lee, PhD, Dept. of Applied Psychology, Bouve College of Health Sciences, Northeastern University, Boston, MA 02115. E-mail: chr.lee@ neu.edu. 208
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
209
(Blumenthal, Jacobson, & Robinson, 2007) and an increased burden of health and social problems related to alcohol use (Caetano, Ramisetty-Mikler, & Rodriguez, 2008; Campos-Outcalt, Bay, Dellapenna, & Cota, 2002; Chartier & Caetano, 2010; Mulia, Greenfield, & Zemore, 2009; Stinson, Grant, & Dufour, 2001; Voas, Fisher, & Tippetts, 2002) have been noted among Latinos who drink heavily. Grounding empirically based treatments, such as motivational interviewing, in social and cultural realities of minorities may be an important way to improve addiction treatment participation and response (Szapocznik, Lopez, Prado, Schwartz, & Pantin, 2006). In other words, accounting for the multiple sources of risk (i.e., poverty, discrimination, and acculturation stress) that influence drinking behavior among Latinos (Lee et al., 2013; Zemore, Karriker-Jaffe, Keithly, & Mulia, 2011) may enhance treatment relevance and efficacy (Gallardo & Curry, 2009; Mulia, Ye, Zemore, & Greenfield, 2008; Zemore et al., 2011). The current study is a secondary analysis of a randomized trial that compared a version of MI that was culturally adapted to these stressors (Lee et al., 2013) and then compared against an unadapted MI, thereby employing a rarely used but rigorous scientific design (Martinez & Eddy, 2005). Findings suggested preliminary evidence to support the efficacy of the culturally adapted version (Lee et al., 2013). For clarification, while we conceptualize our treatment as being grounded in social and cultural priorities of our participants, we use the word ‘‘adapted’’ interchangeably with being culturally grounded to improve readability. In adapting empirically based treatments, a first priority is to confirm whether active ingredients of the original treatment were preserved (Lau, 2006). Accordingly, data from the pilot study of the adapted MI revealed that participants responded favorably to the collaborative and nonjudgmental counselor attitudes (Lee et al., 2011), suggesting that the underlying precepts of motivational interviewing, such as MI spirit, was delivered as intended. The follow-up randomized trial (Lee et al., 2013) revealed equally high levels of treatment satisfaction and engagement reported by participants who received either an unadapted or culturally adapted version of MI, reconfirming earlier findings. Evaluating the treatment fidelity of addiction treatments is needed to understand whether treatments are being delivered as intended (Moyers, Martin, Manuel, Hendrickson, & Miller, 2005; Rollnick & Miller, 1995), a particularly important question when the treatment is being delivered to a new population. Few studies have evaluated the treatment fidelity of treatments with Hispanic substance abusers (Santa Ana, et al., 2009). Santa Ana, Carroll, Anez et al. (2009) reported the use of the Independent Tape Rater Scale (ITRS, Ball et al., 2002) to evaluate the extent to which Spanish-speaking therapists provided a motivationally enhanced treatment to Spanish-speaking clients with adequate fidelity. Independent ratings from coding 325 sessions revealed that the ITRS had good to excellent inter-reliability, indicating the
Downloaded by [New York University] at 15:44 27 May 2015
210
C. S. Lee et al.
utility of the ITRS to evaluate treatment fidelity of motivationally enhanced treatment delivered in Spanish. The motivational interviewing treatment integrity coding system (MITI; Moyers, Martin, Manual, Hendrickson, & Miller, 2005) was developed as a way to rate therapist behaviors and adherence to MI principles. As such, it includes an assessment of overall therapist empathy and MI spirit, as well as counts of specific behaviors such as making reflections and asking questions (Pierson et al., 2007). The MITI coding system has been used to evaluate treatment fidelity in German (Brueck, Frick, Loessl, et al., 2009), Swedish (Forsberg, Kallmen, Hermansson, Berman, & Helgason, 2007), and in English (Bennett, Roberts, Vaughan, Gibbins, & Rouse, 2007; McCambridge, Day, Thomas, & Strang, 2011). Our study examines the use of the MITI to evaluate the treatment fidelity of MI adapted to account for key social stressors and cultural influences. While the MITI has been used to evaluate translations of MI in other countries (Brueck et al., 2009; Forsberg et al., 2007), this is the first study, to our knowledge, that reports the use of MITI in evaluating MI that has been adapted. Current study goals are to evaluate whether the adaptation of MI was delivered with the same level of treatment fidelity to MI as the unadapted MI. Second, we compare the reliability of MITI scores for the adapted versus unadapted version.
MATERIALS AND METHODS Source of the Tapes The current study is an analysis of MITI data from a randomized trial that compared a version of MI that was culturally adapted (Lee et al., 2013) to standard MI. Participants were non-treatment seeking and were recruited if they wanted to learn about the effects of drinking on their health. Study eligible participants met criteria for hazardous drinking ( 5 drinks=occasion or 14 drinks=week for men; 4 drinks=occasion or 7 drinks for women), were between 18–54 years of age, of Hispanic nationality, and proficient in English. English proficiency was indicated by one’s ability to understand a description of the study in English without the use of an interpreter and the ability to repeat the components of informed consent. Minimum oral English proficiency was required because an aim of the original study was to assess the effects of culturally adapting the treatment separately from translating the treatment. A larger scale study delivering the culturally adapted treatment in the language of the patient’s choice (Spanish or English) is currently underway (Lee, R01AA021136). All study procedures received approval from the Institutional Review Board at the study institution. Fifty-three participants completed the randomized trial (MI, n ¼ 27, adapted, n ¼ 26). The participants’ (55% men, n ¼ 29) average age was 35
Assessing Treatment Fidelity
211
Downloaded by [New York University] at 15:44 27 May 2015
years (age range 18–61 SD ¼ 12.35), the average educational attainment was 12 years, and 94% were employed but working at lower wage jobs, such as domestic care, janitorial services, and teacher aides. Fifty-three percent of participants (n ¼ 28) were born in the United States. Mean weekly drinking rates were 44 drinks (SD ¼ 42.57) for men and 18 drinks (SD ¼ 15.56) for women. Participants met with their therapists once. The motivational interview (MI) and the culturally adapted motivational interview (CAMI) each took the same length of time to complete (1.5 hours). A random sample of 20 tapes (10 CAMI and 10 MI) were coded using the MITI. Each tape was first transcribed by a bilingual professional translator and then coded by one of the two blind raters.
Therapist Training All seven therapists were trained by the first author to deliver both treatment conditions. MI training took up to 16 hours and focused on MI spirit, MI strategies, and recognizing and eliciting change talk. The CAMI training took an additional 16 hours and included a description of the social contextual model of adaptation and a review of how each manualized treatment component was adapted, as well as discussions of potential process issues in delivering the adaptation. Each participant session was audiotaped. The first author reviewed audiotapes in supervision with therapists to ensure the therapists were trained on criteria.
Selection and Training of Raters Following procedures outlined in Pierson et al. (2008), the first author (MINT [Motivational Interviewing Network of Trainers] trainer, MITI trained) taught two PhD-level psychology graduate students the MITI system (14 hours training time). This training included: reviewing the MITI manual and selected readings (e.g., Moyers et al., 2005), reporting MITI development considerations and psychometrics, parsing therapist utterances, reflections (simple and complex), MI adherent and non-adherent behavior codes, and assessing global dimensions of MI spirit. Raters were given uncoded transcripts from an MI tape series (http://www.motivationalinterview.org/trainers/side_bar/training_act_ resources.html) to practice parsing and MITI coding. Raters then submitted their scores to the first author for review. Inter-rater reliability, which measures the degree to which different raters give corresponding or similar ratings of the same behaviors, was calculated using the intraclass correlation coefficient (ICC), a more conservative measure of inter-rater reliability than other measures (Pearson product moment correlation, Cronbach alpha), because it accounts for the systematic and chance differences between raters (Pierson et al., 2007, p. 14). Inter-rater reliability coefficients range from 0–1, with scores of 1 indicating 100% agreement between raters (Mitchell & Jolley, 2010). Using
212
C. S. Lee et al.
the cut-point recommended by Pierson et al (2007), once it was determined that the inter-rater reliabilities for codes given on the practice transcripts showed adequate agreement (at least .7), training ended and the raters were given MI and CAMI transcripts to code on their own. The two raters were unaware of whether their tapes were MI or CAMI.
Downloaded by [New York University] at 15:44 27 May 2015
Motivational Interviewing Treatment Integrity Coding System, 3.1 (Moyers et al., 2010) Motivational interviewing is a collaborative, goal-oriented style of communication (Miller & Rollnick, 2013) with well-documented efficacy in addictions treatment (Hettema, Steele, & Miller, 2005). MI spirit is defined as the counselor’s collaborative, accepting, and empathic attitude towards the patient, intended to promote a therapeutic climate that elicits and evokes the patient’s own thoughts and feelings about change (Miller & Rollnick, 2013). Further, the therapist uses specific behaviors, or therapeutic strategies (open-ended questions, active listening, evocative questions) prescribed by MI (Longabuagh & Magill, 2011) to elicit the patient’s positive self-motivational statements about desired behavior change. The MITI is a coding system that evaluates therapist MI spirit and MI behavior counts. Raters are instructed to review transcripts and give ratings for MI spirit (computed as an average of evocation þ collaboration þ autonomy=3), and empathy. Second, raters give behavior counts for each MI-specific therapist behavior: giving information, MI adherent and non-adherent behaviors, questions (closed=open), and reflections, or therapist statements about the patient’s utterances. Simple reflections essentially repeat what the participant has said. Complex reflections add an interpretation of the patient’s unstated meaning. For example, in response to ‘‘Someday I’d like to quit drinking, I know it affects my health,’’ a simple reflection would be: ‘‘Someday you want to quit drinking because you know it affects your health.’’ A complex reflection would add: ‘‘When you’re ready, you will quit drinking because you’re worried about your health.’’
Coding and Rating Procedures The raters listened to all of the MI and CAMI tapes alone and then met with the first author to discuss their discrepant codes and to control for rater drift. Each tape lasted an average of 1 hour and 15 minutes. Raters coded the same 20-minute segments that occurred towards the end of each session in each tape.
Analytic Plan As the ICC statistic adjusts for chance agreement between raters and systematic differences between raters (Fleiss & Shrout, 1978), it is a more conservative
213
Assessing Treatment Fidelity
estimate of inter-rater reliability than Cronbach alpha or the Pearson product moment correlation (Moyers et al., 2005). ICCs between the two raters were calculated across both conditions, and then for MI and CAMI conditions. According to Moyers et al. (2005), Cichetti’s (1994) categorization system for evaluating the usefulness of ICCs for clinical instruments classifies .75–1.00 ¼ excellent, .60–.74 ¼ good, .40–.59 ¼ fair, and below .40 ¼ poor.
RESULTS
Downloaded by [New York University] at 15:44 27 May 2015
MITI Reliability The ICCs are presented in Table 1. There were 14 total scores for MI behaviors (not including the average scores). All were in the excellent range (.78–.99) with the exception of the ICC for complex reflections, CAMI, in the fair range (.58). The ICCs for MI globals, spirit and empathy, were in the poor to fair range (.10–.52) for both MI and CAMI. The ICC for global MI spirit was lower for CAMI (.23) than for MI (.52). The ICC for global empathy was lower for MI (.10) than for CAMI (.23).
MITI Summary Scores Moyers et al. (2010) suggested the following thresholds to assess clinician beginning MI proficiency using the MITI, to indicate MI therapist beginning proficiency are as follows: global clinician ratings average score 3.5 (highest score ¼ 5), reflection to question ratio 1:1, percent open questions (%OC) 50%, percent complex reflections (%CR) 40%, and percent MI adherent (% MI-A) 90% MI.
TABLE 1 Intraclass Correlation Coefficients (ICC) of Each Item by Condition and Per Group
Item Global empathy Global MI spirit Giving information MI adherent MI non-adherent Closed questions Open questions Simple reflections Complex reflections
Motivational Interviewing (MI)
Culturally Adapted Motivational Interviewing (CAMI)
Average across groups
Level of relationship
ICC .10 .52 .94 .92 .78 .90 .82 .88 .93
ICC .23 .23 .96 .83 .99 .90 .81 .94 .58
ICC .12 .40 .96 .90 .98 .89 .84 .90 .89
Poor Fair Excellent Excellent Excellent Excellent Excellent Excellent Excellent
214
C. S. Lee et al.
Downloaded by [New York University] at 15:44 27 May 2015
Global clinician ratings are informed by expert opinion and based on summary scores that are computed from individual behavior counts. While viewed as potentially important indices of MI functioning, further normative and validity data is needed (Moyers et al., 2010). Across conditions, we found the global clinician rating was 4.1, 43% complex reflections, 44% of questions were open-ended, the reflection to question ratio was 3:1, and 83% were MI adherent. Within CAMI, we found the global clinician rating was 4.1, 41% complex reflections, 37% of questions were open-ended, the reflection to question ratio was 3:1, and 79% were MI adherent. Within MI, we found the global clinician rating was 4.1, 44% complex reflections, 52% of questions were open-ended, the reflection to question ratio was 4:1, and 91% were MI adherent.
DISCUSSION With regards to the study goals, our findings demonstrate that an adapted version of MI delivered to Latino heavy drinkers was delivered with treatment fidelity. Second, we found that the reliability of scores across both conditions (adapted and unadapted) were similar.
MI Behavior Counts The good-to-excellent inter-rater reliabilities for most behavior counts reported in this study are similar to the rater reliability estimates in prior MITI investigations. Inter-rater reliability for complex reflections in the unadapted condition was excellent (.93) but was in the fair range (.58) in the culturally adapted condition. Similarly, the original MITI study (Moyers et al., 2005) reported fair inter-rater reliability for complex reflections (.52). It may be that coding complex reflections, which requires deciding whether interventionists are offering a new interpretation or meaning of the patient’s statement, is a more involved skill that requires a greater level of MITI training to achieve better reliability. Another possible explanation is that there is something more challenging about coding complex reflections in cross-cultural therapeutic situations. Other studies using the MITI in cross-cultural populations have also reported lower reliabilities for complex reflections (Bennett et al., 2007; Brueck et al., 2009) suggesting that further investigation of the cross-cultural validity of the MITI is warranted (McCambridge et al., 2011). Brueck et al. (2009) adapted the MITI in Germany and reported lower ICCs for complex reflections (.56) compared to the other behavior count categories (.75–.97). Different interpretations of conceptual meanings were attributed as one cause, as: ‘‘ . . . the associative fields of some German and American English words used in MI are considerably different’’ (p. 47). In another instance, even when the
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
215
language was similar (England), different cultural interpretations may also challenge reliable coding. For example, English participants often responded to the MI closed-ended question, ‘‘Would you mind telling me,’’ as an open question (Bennett et al., 2007; McCambridge et al., 2011). In another cross-cultural study of the MITI, investigators encountered in-session language that the MITI was not designed to detect and adapted the MITI coding system to accommodate the different linguistic and cultural Thai conventions (Koken et al., 2012). For example, questions worded using the Thai ‘‘mai’’ form did not have any direct English translation, and was described by a Thai speaker as ‘‘a kind of question that seems open but might technically be closed’’ (p. 578), similar to the English question, ‘‘Can you tell me more about that?’’. The eventual MITI coding decision was to code questions using the ‘‘mai’’ form as closed-ended, as analyses of transcripts revealed that Thai clients responded yes=no to over 90% of the occasions in which a ‘‘mai’’ question arose.
MI Global Scores The ICCs for global MI spirit and for empathy in both conditions were in the poor-to-fair range, suggesting that reliably coding more diffuse therapeutic qualities may be more challenging (Brueck et al., 2009; Moyers et al., 2005; Rollnick & Miller, 1995) than counting verbal behaviors like open-ended questions. Coding clinically diffuse, complex categories like empathy requires the ability to detect a clinical tone underlying a conversation and a greater familiarity with MI. Forsberg et al. (2007) suggested that providing extra training on coding MI globals like empathy and offering a more precise definition of MI global variables and their specific behavioral anchors might help to improve coding reliability. Another possibility is that greater articulation is needed on how constructs like empathy and MI global spirit are communicated or interpreted in different cultures and languages. Of note, inter-rater reliabilities for MI globals (empathy and spirit) were in the poor-to-fair range, similar to studies adapting the MITI conducted in Sweden (Forsberg et al., 2007) and in Germany (Brueck et al., 2009). Expressions of empathy may be expressed differently by therapists in cross-cultural settings. The field would benefit from further construct validation in MI delivery (Brueck et al., 2009), as well as continued investigation on how language and culture might influence the interpretation of MI behaviors such as asking questions or making reflections. Such investigations would contribute to the growing literature on the implementation of MI in different countries. A related area of inquiry might be to investigate how raters who are=are not from the same culture as the MI interventionists code clinically diffuse concepts like empathy or MI spirit. The current study presents data from non-Spanish speaking raters who were listening to MI-trained therapists
Downloaded by [New York University] at 15:44 27 May 2015
216
C. S. Lee et al.
working with Latino participants. Building on this study, a future investigation might investigate how Spanish-speaking MITI raters code tapes of MI-trained therapists working with Latino participants. Our study contributes to the literature in several ways. We document the utility of using the MITI to evaluate treatment fidelity in unadapted and in culturally adapted MI. Findings suggest that researchers need to ensure that adapted MI is delivered with fidelity to MI, and the MITI is a useful tool for doing so. The lower inter-rater reliabilities for complex reflections and for MI globals (spirit and empathy) suggest the need for elaboration of these concepts, enhanced MITI training in more complex clinical categories, as well as the need to attend to possible cultural differences in interpreting these categories. Future MITI studies, especially those that involve diverse patient populations, should investigate whether enhancing MITI training in these areas improves inter-rater reliabilities. Finally, our findings converge with a prior study reporting the feasibility of training to achieve a high level of agreement between two raters (Bennett et al., 2007). Our reported reliability between two raters is similar to the reliability levels reported by Moyers et al. (2005) for three raters. Like the Bennett study, our findings demonstrate that using two raters are sufficient to achieve rating consistency. Our study contributes to the field by demonstrating that the MITI can be used to reliably assess an adapted version of MI. Evaluating the treatment fidelity, or whether the addiction treatment is being delivered as intended (Moyers et al., 2005; Rollnick & Miller, 1995), is critical to interpreting study findings (Koken et al., 2012) and thus a necessary part of building the knowledge base on delivering MI to new populations and in different languages (Bellg et al., 2004; Koken et al., 2012). In short, it is necessary to understand if the treatment is being delivered as intended in order to evaluate whether it works. Further, because treatment fidelity to MI predicts positive behavioral outcomes (Miller & Rollnick, 2014), monitoring fidelity should be a key part of the intervention package (Koken et al., 2012). Evaluating the treatment fidelity of treatment with Hispanic substance abusers is rarely done (Santa Ana et al., 2009), and doing so, as in this study, adds to the data base on which addiction treatments are most effective among Latinos (Carroll et al., 2009).
FUNDING This research was supported by a grant from the National Institutes of Health (NIAAA; AAK2314905) awarded to the first author.
REFERENCES Ball, S. A., Martino, S., Corvino, J., Morgenstern, J., & Carroll, K. M. (2002). Independent tape rater guide. Unpublished psychotherapy tape rating manual.
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
217
Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., et al. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 443–451. Bennett, G. A., Roberts, H. A., Vaughan, T. E., Gibbins, J. A., & Rouse, L. (2007). Evaluating a method of assessing competence in Motivational Interviewing: A study using simulated patients in the United Kingdom. Addictive Behaviors, 32, 69–79. Blumenthal, R. N., Jacobson, J. O., & Robinson, P. L. (2007). Are racial disparities in alcohol treatment completion associated with racial differences in treatment modality entry? Comparison of outpatient treatment and residential treatment in Los Angeles County, 1998 to 2000. Alcoholism, Clinical and Experimental Research, 31, 1920–1926. Brueck, R. K., Frick, K., Loessl, B., Kriston, L., Schondelmaeier, S., Go, C., et al. (2009). Psychometric properties of the German version of the Motivational Interviewing Treatment Integrity Code. Journal of Substance Abuse Treatment, 36, 44–48. Caetano, R., Ramisetty-Mikler, S., & Rodriguez, L. A. (2008). The Hispanic Americans Baseline Alcohol Survey (HABLAS): DUI rates, birthplace, and acculturation across Hispanic national groups. Journal of Studies on Alcohol and Drugs, 69, 259–265. Campos-Outcalt, D., Bay, C., Dellapenna, A., & Cota, M. K. (2002). Pedestrian fatalities by race=ethnicity in Arizona, 1990–96. American Journal of Preventative Medicine, 23, 129–135. Carroll, K. M., Martino, S., Ball, S. A., Nich, C., Frankforter, T., Anez, L. M., et al. (2009). A multisite randomized effectiveness trial of motivational enhancement therapy for Spanish-speaking substance users. Journal of Consulting and Clinical Psychology, 77(5), 993–999. Chartier, K., & Caetano, R. (2010). Ethnicity and health disparities in alcohol research. Alcohol Health and Research World, 33, 152–160. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological assessment, 6(4), 284. Fleiss, J. L., & Shrout, P. E. (1978). Approximate interval estimation for a certain intraclass correlation coefficient. Psychometrika, 43, 259–262. Forsberg, L., Kallmen, H., Hermansson, U., Berman, A. H., & Helgason, A. R. (2007). Coding counsellor behavior in motivational interviewing sessions: Inter-rater reliability for the Swedish Motivational Interviewing Treatment Integrity Code (MITI). Cognitive Behaviour Therapy, 36(3), 162–169. Gallardo, M. E., & Curry, S. J. (2009). Shifting perspectives: Culturally responsive interventions with Latino substance abusers. Journal of Ethnicity in Substance Abuse, 8, 314–329. Hettema, J., Steele, J., & Miller, W. R. (2005). Motivational interviewing. Annual Review of Clinical Psychology, 1, 91–111. Koken, J. A., Naar-King, S., Umasa, S., Parsons, J. T., Saengcharnchai, P., Phanuphak, P., et al. (2012). A cross-cultural three step process model for assessing motivational interviewing treatment fidelity in Thailand. Health Education and Behavior, 39(5), 574–582.
Downloaded by [New York University] at 15:44 27 May 2015
218
C. S. Lee et al.
Lau, A. S. (2006). Making the case for selective and directed cultural adaptations of evidenced-based treatments: Examples from parent training. Clinical Psychology: Science and Practice, 13, 295–310. Lee, C. S., Lopez, S. R., Colby, S. M., Hernandez, L., Caetano, R., Borrelli, B., et al. (2011). A cultural adaptation of motivational interviewing to address heavy drinking among Hispanics. Cultural Diversity & Ethnic Minority Psychology, 17(3), 317–324. Lee, C. S., Lopez, S. R., Colby, S. M., Rohsenow, D., Hernandez, L., Borrelli, B., et al. (2013). Culturally adapted motivational interviewing for Latino heavy drinkers: Results from a randomized clinical trial. Journal of Ethnicity in Substance Abuse, 12(4), 356–373. Lee, C. S., Colby, S. M., Rohsenow, D. J., Lopez, S. R., Hernandez, L., & Caetano, R. (2013). Acculturation stress and drinking problems among urban heavy drinking Latinos in the Northeast. Journal of Ethnicity in Substance Abuse, 12(4), 308–320. Longabaugh, R., & Magill, M. (2011). Recent advances in behavioral addiction treatments: Focusing on mechanisms of change. Current Psychiatry Reports, 13(5), 382–289. Martinez, C. R., & Eddy, J. M. (2005). Effects of culturally adapted parent management training on Latino youth behavioral health outcomes. Journal of Consulting and Clinical Psychology, 73, 841–851. McCambridge, J., Day, M., Thomas, B. A., & Strang, J. (2011). Fidelity to Motivational Interviewing and subsequent cannabis cessation among adolescents. Addictive Behaviors, 36, 749–754. Miller, W. R., & Rose, G. S. (2010). Motivational interviewing in relational context. American Psychologist, 65, 298–299. Miller, W. R., & Rollnick, S. (2013). Motivational interviewing: Helping people change (3rd ed.), New York, NY: Guilford. Miller, W. R., & Rollnick, S. (2014). The effectiveness and ineffectiveness of complex behavioral interventions: Impact of treatment fidelity. Contemporary Clinical Trials, 37, 234–241. Mitchell, M. L., & Jolley, J. M. (2010). Research design explained. (7th ed.). Belmont, CA: Wadsworth, Cengage Learning Inc. Moyers, T. B., Martin, T., Manuel, J. K., Hendrickson, S. M. L., & Miller, W. R. (2005). Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment, 28(1), 19–26. Moyers, T. B., Martin, T., Manuel, J. K., Miller, W. R., & Ernst, D. (2010). Revised global scales: Motivational interviewing treatment integrity 3.1. 1 (MITI 3.1.1). University of New Mexico, Albuquerque, NM. Unpublished manuscript. Mulia, N., Ye, Y., Zemore, S. E., & Greenfield, T. K. (2008). Social disadvantage, stress, and alcohol use among Black, Hispanic, and White Americans: Findings from the 2005 U.S. National Alcohol Survey. Journal of Studies on Alcohol and Drugs, 69, 824–833. Mulia, N., Ye, Y., Greenfield, T. K., & Zemore, S. E. (2009). Disparities in alcohol related problems among white, black, and Hispanic Americans. Alcoholism: Clinical and Experimental Research, 33, 654–662.
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
219
Pierson, H. M., Hayes, S. C., Gifford, E. V., Roget, N., Padilla, M., Bissett, R., et al. (2007). An examination of the Motivational Interviewing Treatment Integrity code. Journal of Substance Abuse Treatment, 32, 11–17. Rollnick, S., & Miller, W. R. (1995). What is motivational interviewing? Behavioural and Cognitive Psychotherapy, 23, 325–334. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings—assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413–428. Santa Ana, E. J., Carroll, K. M., Anez, L., Paris, M., Ball, S. A., Nich, C., et al. (2009). Evaluating motivational enhancement therapy adherence and competence among Spanish-speaking therapists. Drug and Alcohol Dependence, 103, 44–51. Stinson, F. S., Grant, B. F., & Dufour, M. C. (2001). The critical dimension of ethnicity in liver cirrhosis and mortality statistics. Alcoholism: Clinical and Experimental Research, 25, 1181–1187. Szapocznik, J., Lopez, B., Prado, G., Schwartz, S. J., & Pantin, H. (2006). Outpatient drug abuse treatment for Hispanic adolescents. Drug and Alcohol Dependence, 84(Suppl.1), S54–S63. Transcripts from MI Videos (2013). Retrieved from http://www.motivationalinterview. org/trainers/side_bar/training_act_resources.html Vader, A. M., Walters, S. T., Prabhu, G. C., Houck, J. M., & Field, C. A. (2010). The language of motivational interviewing and feedback: Counselor language, client language, and client drinking outcomes. Psychology of Addictive Behaviors, 24(2), 190–197. Voas, R. B., Fisher, D. A., & Tippetts, A. S. (2002). Children in fatal crashes: Driver blood alcohol concentration and demographics of child passengers and their drivers. Addiction, 97, 1439–1448. Zemore, S. E., Karriker-Jaffe, K. J., Keithly, S., & Mulia, N. (2011). Racial prejudice and unfair treatment: Interactive effects with poverty and foreign nativity on problem drinking. Journal of Studies on Alcohol and Drugs, 72, 361–370.
Journal of Ethnicity in Substance Abuse Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wesa20
Assessing Treatment Fidelity in a Cultural Adaptation of Motivational Interviewing a
a
a
a
Christina S. Lee , Tonya Tavares , Ami Popat-Jain & Pamela Naab a
Northeastern University, Boston, Massachusetts Published online: 18 May 2015.
Click for updates To cite this article: Christina S. Lee, Tonya Tavares, Ami Popat-Jain & Pamela Naab (2015) Assessing Treatment Fidelity in a Cultural Adaptation of Motivational Interviewing, Journal of Ethnicity in Substance Abuse, 14:2, 208-219, DOI: 10.1080/15332640.2014.973628 To link to this article: http://dx.doi.org/10.1080/15332640.2014.973628
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [New York University] at 15:44 27 May 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Journal of Ethnicity in Substance Abuse, 14:208–219, 2015 Copyright # Taylor & Francis Group, LLC ISSN: 1533-2640 print=1533-2659 online DOI: 10.1080/15332640.2014.973628
Assessing Treatment Fidelity in a Cultural Adaptation of Motivational Interviewing CHRISTINA S. LEE, TONYA TAVARES, AMI POPAT-JAIN, and PAMELA NAAB
Downloaded by [New York University] at 15:44 27 May 2015
Northeastern University, Boston, Massachusetts
The efficacy of motivational interviewing (MI) for addictions is well documented. Grounding MI in social=cultural priorities may enhance treatment response. We evaluate the method of assessing competence using the motivational interviewing treatment integrity system (MITI) for standard MI and culturally adapted MI (CAMI) delivered to Latino heavy drinkers. Twenty audiotapes (MI, n ¼ 10; CAMI, n ¼ 10) were MITI coded by two raters unaware of treatment assignment. Inter-rater reliabilities were excellent (.78–.99) except for CAMI complex reflections, global ratings of empathy, and MI spirit. The MITI reliably evaluates MI and CAMI treatment fidelity. Future research should investigate lower reliabilities for MI global and complex reflections cross-culturally. KEYWORDS motivational interviewing, alcohol, cultural adaptation, Latinos, treatment integrity
INTRODUCTION Motivational interviewing is a directive, person-based counseling approach with proven efficacy in the treatment of addiction. The motivational interviewing emphasis on collaboration and on privileging autonomy to elicit thoughts and feelings about change is of great appeal to those who may have experienced oppression or discrimination (Miller & Rose, 2010), such as newly arrived immigrants or minorities. Poor treatment response A portion of the study findings were presented at the annual meeting of the Research Society on Alcoholism, 2013, and at the 2013 Inter-Professional Research Symposium, Bouve´ College of Health Sciences, Northeastern University. Address correspondence to Christina S. Lee, PhD, Dept. of Applied Psychology, Bouve College of Health Sciences, Northeastern University, Boston, MA 02115. E-mail: chr.lee@ neu.edu. 208
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
209
(Blumenthal, Jacobson, & Robinson, 2007) and an increased burden of health and social problems related to alcohol use (Caetano, Ramisetty-Mikler, & Rodriguez, 2008; Campos-Outcalt, Bay, Dellapenna, & Cota, 2002; Chartier & Caetano, 2010; Mulia, Greenfield, & Zemore, 2009; Stinson, Grant, & Dufour, 2001; Voas, Fisher, & Tippetts, 2002) have been noted among Latinos who drink heavily. Grounding empirically based treatments, such as motivational interviewing, in social and cultural realities of minorities may be an important way to improve addiction treatment participation and response (Szapocznik, Lopez, Prado, Schwartz, & Pantin, 2006). In other words, accounting for the multiple sources of risk (i.e., poverty, discrimination, and acculturation stress) that influence drinking behavior among Latinos (Lee et al., 2013; Zemore, Karriker-Jaffe, Keithly, & Mulia, 2011) may enhance treatment relevance and efficacy (Gallardo & Curry, 2009; Mulia, Ye, Zemore, & Greenfield, 2008; Zemore et al., 2011). The current study is a secondary analysis of a randomized trial that compared a version of MI that was culturally adapted to these stressors (Lee et al., 2013) and then compared against an unadapted MI, thereby employing a rarely used but rigorous scientific design (Martinez & Eddy, 2005). Findings suggested preliminary evidence to support the efficacy of the culturally adapted version (Lee et al., 2013). For clarification, while we conceptualize our treatment as being grounded in social and cultural priorities of our participants, we use the word ‘‘adapted’’ interchangeably with being culturally grounded to improve readability. In adapting empirically based treatments, a first priority is to confirm whether active ingredients of the original treatment were preserved (Lau, 2006). Accordingly, data from the pilot study of the adapted MI revealed that participants responded favorably to the collaborative and nonjudgmental counselor attitudes (Lee et al., 2011), suggesting that the underlying precepts of motivational interviewing, such as MI spirit, was delivered as intended. The follow-up randomized trial (Lee et al., 2013) revealed equally high levels of treatment satisfaction and engagement reported by participants who received either an unadapted or culturally adapted version of MI, reconfirming earlier findings. Evaluating the treatment fidelity of addiction treatments is needed to understand whether treatments are being delivered as intended (Moyers, Martin, Manuel, Hendrickson, & Miller, 2005; Rollnick & Miller, 1995), a particularly important question when the treatment is being delivered to a new population. Few studies have evaluated the treatment fidelity of treatments with Hispanic substance abusers (Santa Ana, et al., 2009). Santa Ana, Carroll, Anez et al. (2009) reported the use of the Independent Tape Rater Scale (ITRS, Ball et al., 2002) to evaluate the extent to which Spanish-speaking therapists provided a motivationally enhanced treatment to Spanish-speaking clients with adequate fidelity. Independent ratings from coding 325 sessions revealed that the ITRS had good to excellent inter-reliability, indicating the
Downloaded by [New York University] at 15:44 27 May 2015
210
C. S. Lee et al.
utility of the ITRS to evaluate treatment fidelity of motivationally enhanced treatment delivered in Spanish. The motivational interviewing treatment integrity coding system (MITI; Moyers, Martin, Manual, Hendrickson, & Miller, 2005) was developed as a way to rate therapist behaviors and adherence to MI principles. As such, it includes an assessment of overall therapist empathy and MI spirit, as well as counts of specific behaviors such as making reflections and asking questions (Pierson et al., 2007). The MITI coding system has been used to evaluate treatment fidelity in German (Brueck, Frick, Loessl, et al., 2009), Swedish (Forsberg, Kallmen, Hermansson, Berman, & Helgason, 2007), and in English (Bennett, Roberts, Vaughan, Gibbins, & Rouse, 2007; McCambridge, Day, Thomas, & Strang, 2011). Our study examines the use of the MITI to evaluate the treatment fidelity of MI adapted to account for key social stressors and cultural influences. While the MITI has been used to evaluate translations of MI in other countries (Brueck et al., 2009; Forsberg et al., 2007), this is the first study, to our knowledge, that reports the use of MITI in evaluating MI that has been adapted. Current study goals are to evaluate whether the adaptation of MI was delivered with the same level of treatment fidelity to MI as the unadapted MI. Second, we compare the reliability of MITI scores for the adapted versus unadapted version.
MATERIALS AND METHODS Source of the Tapes The current study is an analysis of MITI data from a randomized trial that compared a version of MI that was culturally adapted (Lee et al., 2013) to standard MI. Participants were non-treatment seeking and were recruited if they wanted to learn about the effects of drinking on their health. Study eligible participants met criteria for hazardous drinking ( 5 drinks=occasion or 14 drinks=week for men; 4 drinks=occasion or 7 drinks for women), were between 18–54 years of age, of Hispanic nationality, and proficient in English. English proficiency was indicated by one’s ability to understand a description of the study in English without the use of an interpreter and the ability to repeat the components of informed consent. Minimum oral English proficiency was required because an aim of the original study was to assess the effects of culturally adapting the treatment separately from translating the treatment. A larger scale study delivering the culturally adapted treatment in the language of the patient’s choice (Spanish or English) is currently underway (Lee, R01AA021136). All study procedures received approval from the Institutional Review Board at the study institution. Fifty-three participants completed the randomized trial (MI, n ¼ 27, adapted, n ¼ 26). The participants’ (55% men, n ¼ 29) average age was 35
Assessing Treatment Fidelity
211
Downloaded by [New York University] at 15:44 27 May 2015
years (age range 18–61 SD ¼ 12.35), the average educational attainment was 12 years, and 94% were employed but working at lower wage jobs, such as domestic care, janitorial services, and teacher aides. Fifty-three percent of participants (n ¼ 28) were born in the United States. Mean weekly drinking rates were 44 drinks (SD ¼ 42.57) for men and 18 drinks (SD ¼ 15.56) for women. Participants met with their therapists once. The motivational interview (MI) and the culturally adapted motivational interview (CAMI) each took the same length of time to complete (1.5 hours). A random sample of 20 tapes (10 CAMI and 10 MI) were coded using the MITI. Each tape was first transcribed by a bilingual professional translator and then coded by one of the two blind raters.
Therapist Training All seven therapists were trained by the first author to deliver both treatment conditions. MI training took up to 16 hours and focused on MI spirit, MI strategies, and recognizing and eliciting change talk. The CAMI training took an additional 16 hours and included a description of the social contextual model of adaptation and a review of how each manualized treatment component was adapted, as well as discussions of potential process issues in delivering the adaptation. Each participant session was audiotaped. The first author reviewed audiotapes in supervision with therapists to ensure the therapists were trained on criteria.
Selection and Training of Raters Following procedures outlined in Pierson et al. (2008), the first author (MINT [Motivational Interviewing Network of Trainers] trainer, MITI trained) taught two PhD-level psychology graduate students the MITI system (14 hours training time). This training included: reviewing the MITI manual and selected readings (e.g., Moyers et al., 2005), reporting MITI development considerations and psychometrics, parsing therapist utterances, reflections (simple and complex), MI adherent and non-adherent behavior codes, and assessing global dimensions of MI spirit. Raters were given uncoded transcripts from an MI tape series (http://www.motivationalinterview.org/trainers/side_bar/training_act_ resources.html) to practice parsing and MITI coding. Raters then submitted their scores to the first author for review. Inter-rater reliability, which measures the degree to which different raters give corresponding or similar ratings of the same behaviors, was calculated using the intraclass correlation coefficient (ICC), a more conservative measure of inter-rater reliability than other measures (Pearson product moment correlation, Cronbach alpha), because it accounts for the systematic and chance differences between raters (Pierson et al., 2007, p. 14). Inter-rater reliability coefficients range from 0–1, with scores of 1 indicating 100% agreement between raters (Mitchell & Jolley, 2010). Using
212
C. S. Lee et al.
the cut-point recommended by Pierson et al (2007), once it was determined that the inter-rater reliabilities for codes given on the practice transcripts showed adequate agreement (at least .7), training ended and the raters were given MI and CAMI transcripts to code on their own. The two raters were unaware of whether their tapes were MI or CAMI.
Downloaded by [New York University] at 15:44 27 May 2015
Motivational Interviewing Treatment Integrity Coding System, 3.1 (Moyers et al., 2010) Motivational interviewing is a collaborative, goal-oriented style of communication (Miller & Rollnick, 2013) with well-documented efficacy in addictions treatment (Hettema, Steele, & Miller, 2005). MI spirit is defined as the counselor’s collaborative, accepting, and empathic attitude towards the patient, intended to promote a therapeutic climate that elicits and evokes the patient’s own thoughts and feelings about change (Miller & Rollnick, 2013). Further, the therapist uses specific behaviors, or therapeutic strategies (open-ended questions, active listening, evocative questions) prescribed by MI (Longabuagh & Magill, 2011) to elicit the patient’s positive self-motivational statements about desired behavior change. The MITI is a coding system that evaluates therapist MI spirit and MI behavior counts. Raters are instructed to review transcripts and give ratings for MI spirit (computed as an average of evocation þ collaboration þ autonomy=3), and empathy. Second, raters give behavior counts for each MI-specific therapist behavior: giving information, MI adherent and non-adherent behaviors, questions (closed=open), and reflections, or therapist statements about the patient’s utterances. Simple reflections essentially repeat what the participant has said. Complex reflections add an interpretation of the patient’s unstated meaning. For example, in response to ‘‘Someday I’d like to quit drinking, I know it affects my health,’’ a simple reflection would be: ‘‘Someday you want to quit drinking because you know it affects your health.’’ A complex reflection would add: ‘‘When you’re ready, you will quit drinking because you’re worried about your health.’’
Coding and Rating Procedures The raters listened to all of the MI and CAMI tapes alone and then met with the first author to discuss their discrepant codes and to control for rater drift. Each tape lasted an average of 1 hour and 15 minutes. Raters coded the same 20-minute segments that occurred towards the end of each session in each tape.
Analytic Plan As the ICC statistic adjusts for chance agreement between raters and systematic differences between raters (Fleiss & Shrout, 1978), it is a more conservative
213
Assessing Treatment Fidelity
estimate of inter-rater reliability than Cronbach alpha or the Pearson product moment correlation (Moyers et al., 2005). ICCs between the two raters were calculated across both conditions, and then for MI and CAMI conditions. According to Moyers et al. (2005), Cichetti’s (1994) categorization system for evaluating the usefulness of ICCs for clinical instruments classifies .75–1.00 ¼ excellent, .60–.74 ¼ good, .40–.59 ¼ fair, and below .40 ¼ poor.
RESULTS
Downloaded by [New York University] at 15:44 27 May 2015
MITI Reliability The ICCs are presented in Table 1. There were 14 total scores for MI behaviors (not including the average scores). All were in the excellent range (.78–.99) with the exception of the ICC for complex reflections, CAMI, in the fair range (.58). The ICCs for MI globals, spirit and empathy, were in the poor to fair range (.10–.52) for both MI and CAMI. The ICC for global MI spirit was lower for CAMI (.23) than for MI (.52). The ICC for global empathy was lower for MI (.10) than for CAMI (.23).
MITI Summary Scores Moyers et al. (2010) suggested the following thresholds to assess clinician beginning MI proficiency using the MITI, to indicate MI therapist beginning proficiency are as follows: global clinician ratings average score 3.5 (highest score ¼ 5), reflection to question ratio 1:1, percent open questions (%OC) 50%, percent complex reflections (%CR) 40%, and percent MI adherent (% MI-A) 90% MI.
TABLE 1 Intraclass Correlation Coefficients (ICC) of Each Item by Condition and Per Group
Item Global empathy Global MI spirit Giving information MI adherent MI non-adherent Closed questions Open questions Simple reflections Complex reflections
Motivational Interviewing (MI)
Culturally Adapted Motivational Interviewing (CAMI)
Average across groups
Level of relationship
ICC .10 .52 .94 .92 .78 .90 .82 .88 .93
ICC .23 .23 .96 .83 .99 .90 .81 .94 .58
ICC .12 .40 .96 .90 .98 .89 .84 .90 .89
Poor Fair Excellent Excellent Excellent Excellent Excellent Excellent Excellent
214
C. S. Lee et al.
Downloaded by [New York University] at 15:44 27 May 2015
Global clinician ratings are informed by expert opinion and based on summary scores that are computed from individual behavior counts. While viewed as potentially important indices of MI functioning, further normative and validity data is needed (Moyers et al., 2010). Across conditions, we found the global clinician rating was 4.1, 43% complex reflections, 44% of questions were open-ended, the reflection to question ratio was 3:1, and 83% were MI adherent. Within CAMI, we found the global clinician rating was 4.1, 41% complex reflections, 37% of questions were open-ended, the reflection to question ratio was 3:1, and 79% were MI adherent. Within MI, we found the global clinician rating was 4.1, 44% complex reflections, 52% of questions were open-ended, the reflection to question ratio was 4:1, and 91% were MI adherent.
DISCUSSION With regards to the study goals, our findings demonstrate that an adapted version of MI delivered to Latino heavy drinkers was delivered with treatment fidelity. Second, we found that the reliability of scores across both conditions (adapted and unadapted) were similar.
MI Behavior Counts The good-to-excellent inter-rater reliabilities for most behavior counts reported in this study are similar to the rater reliability estimates in prior MITI investigations. Inter-rater reliability for complex reflections in the unadapted condition was excellent (.93) but was in the fair range (.58) in the culturally adapted condition. Similarly, the original MITI study (Moyers et al., 2005) reported fair inter-rater reliability for complex reflections (.52). It may be that coding complex reflections, which requires deciding whether interventionists are offering a new interpretation or meaning of the patient’s statement, is a more involved skill that requires a greater level of MITI training to achieve better reliability. Another possible explanation is that there is something more challenging about coding complex reflections in cross-cultural therapeutic situations. Other studies using the MITI in cross-cultural populations have also reported lower reliabilities for complex reflections (Bennett et al., 2007; Brueck et al., 2009) suggesting that further investigation of the cross-cultural validity of the MITI is warranted (McCambridge et al., 2011). Brueck et al. (2009) adapted the MITI in Germany and reported lower ICCs for complex reflections (.56) compared to the other behavior count categories (.75–.97). Different interpretations of conceptual meanings were attributed as one cause, as: ‘‘ . . . the associative fields of some German and American English words used in MI are considerably different’’ (p. 47). In another instance, even when the
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
215
language was similar (England), different cultural interpretations may also challenge reliable coding. For example, English participants often responded to the MI closed-ended question, ‘‘Would you mind telling me,’’ as an open question (Bennett et al., 2007; McCambridge et al., 2011). In another cross-cultural study of the MITI, investigators encountered in-session language that the MITI was not designed to detect and adapted the MITI coding system to accommodate the different linguistic and cultural Thai conventions (Koken et al., 2012). For example, questions worded using the Thai ‘‘mai’’ form did not have any direct English translation, and was described by a Thai speaker as ‘‘a kind of question that seems open but might technically be closed’’ (p. 578), similar to the English question, ‘‘Can you tell me more about that?’’. The eventual MITI coding decision was to code questions using the ‘‘mai’’ form as closed-ended, as analyses of transcripts revealed that Thai clients responded yes=no to over 90% of the occasions in which a ‘‘mai’’ question arose.
MI Global Scores The ICCs for global MI spirit and for empathy in both conditions were in the poor-to-fair range, suggesting that reliably coding more diffuse therapeutic qualities may be more challenging (Brueck et al., 2009; Moyers et al., 2005; Rollnick & Miller, 1995) than counting verbal behaviors like open-ended questions. Coding clinically diffuse, complex categories like empathy requires the ability to detect a clinical tone underlying a conversation and a greater familiarity with MI. Forsberg et al. (2007) suggested that providing extra training on coding MI globals like empathy and offering a more precise definition of MI global variables and their specific behavioral anchors might help to improve coding reliability. Another possibility is that greater articulation is needed on how constructs like empathy and MI global spirit are communicated or interpreted in different cultures and languages. Of note, inter-rater reliabilities for MI globals (empathy and spirit) were in the poor-to-fair range, similar to studies adapting the MITI conducted in Sweden (Forsberg et al., 2007) and in Germany (Brueck et al., 2009). Expressions of empathy may be expressed differently by therapists in cross-cultural settings. The field would benefit from further construct validation in MI delivery (Brueck et al., 2009), as well as continued investigation on how language and culture might influence the interpretation of MI behaviors such as asking questions or making reflections. Such investigations would contribute to the growing literature on the implementation of MI in different countries. A related area of inquiry might be to investigate how raters who are=are not from the same culture as the MI interventionists code clinically diffuse concepts like empathy or MI spirit. The current study presents data from non-Spanish speaking raters who were listening to MI-trained therapists
Downloaded by [New York University] at 15:44 27 May 2015
216
C. S. Lee et al.
working with Latino participants. Building on this study, a future investigation might investigate how Spanish-speaking MITI raters code tapes of MI-trained therapists working with Latino participants. Our study contributes to the literature in several ways. We document the utility of using the MITI to evaluate treatment fidelity in unadapted and in culturally adapted MI. Findings suggest that researchers need to ensure that adapted MI is delivered with fidelity to MI, and the MITI is a useful tool for doing so. The lower inter-rater reliabilities for complex reflections and for MI globals (spirit and empathy) suggest the need for elaboration of these concepts, enhanced MITI training in more complex clinical categories, as well as the need to attend to possible cultural differences in interpreting these categories. Future MITI studies, especially those that involve diverse patient populations, should investigate whether enhancing MITI training in these areas improves inter-rater reliabilities. Finally, our findings converge with a prior study reporting the feasibility of training to achieve a high level of agreement between two raters (Bennett et al., 2007). Our reported reliability between two raters is similar to the reliability levels reported by Moyers et al. (2005) for three raters. Like the Bennett study, our findings demonstrate that using two raters are sufficient to achieve rating consistency. Our study contributes to the field by demonstrating that the MITI can be used to reliably assess an adapted version of MI. Evaluating the treatment fidelity, or whether the addiction treatment is being delivered as intended (Moyers et al., 2005; Rollnick & Miller, 1995), is critical to interpreting study findings (Koken et al., 2012) and thus a necessary part of building the knowledge base on delivering MI to new populations and in different languages (Bellg et al., 2004; Koken et al., 2012). In short, it is necessary to understand if the treatment is being delivered as intended in order to evaluate whether it works. Further, because treatment fidelity to MI predicts positive behavioral outcomes (Miller & Rollnick, 2014), monitoring fidelity should be a key part of the intervention package (Koken et al., 2012). Evaluating the treatment fidelity of treatment with Hispanic substance abusers is rarely done (Santa Ana et al., 2009), and doing so, as in this study, adds to the data base on which addiction treatments are most effective among Latinos (Carroll et al., 2009).
FUNDING This research was supported by a grant from the National Institutes of Health (NIAAA; AAK2314905) awarded to the first author.
REFERENCES Ball, S. A., Martino, S., Corvino, J., Morgenstern, J., & Carroll, K. M. (2002). Independent tape rater guide. Unpublished psychotherapy tape rating manual.
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
217
Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., et al. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 443–451. Bennett, G. A., Roberts, H. A., Vaughan, T. E., Gibbins, J. A., & Rouse, L. (2007). Evaluating a method of assessing competence in Motivational Interviewing: A study using simulated patients in the United Kingdom. Addictive Behaviors, 32, 69–79. Blumenthal, R. N., Jacobson, J. O., & Robinson, P. L. (2007). Are racial disparities in alcohol treatment completion associated with racial differences in treatment modality entry? Comparison of outpatient treatment and residential treatment in Los Angeles County, 1998 to 2000. Alcoholism, Clinical and Experimental Research, 31, 1920–1926. Brueck, R. K., Frick, K., Loessl, B., Kriston, L., Schondelmaeier, S., Go, C., et al. (2009). Psychometric properties of the German version of the Motivational Interviewing Treatment Integrity Code. Journal of Substance Abuse Treatment, 36, 44–48. Caetano, R., Ramisetty-Mikler, S., & Rodriguez, L. A. (2008). The Hispanic Americans Baseline Alcohol Survey (HABLAS): DUI rates, birthplace, and acculturation across Hispanic national groups. Journal of Studies on Alcohol and Drugs, 69, 259–265. Campos-Outcalt, D., Bay, C., Dellapenna, A., & Cota, M. K. (2002). Pedestrian fatalities by race=ethnicity in Arizona, 1990–96. American Journal of Preventative Medicine, 23, 129–135. Carroll, K. M., Martino, S., Ball, S. A., Nich, C., Frankforter, T., Anez, L. M., et al. (2009). A multisite randomized effectiveness trial of motivational enhancement therapy for Spanish-speaking substance users. Journal of Consulting and Clinical Psychology, 77(5), 993–999. Chartier, K., & Caetano, R. (2010). Ethnicity and health disparities in alcohol research. Alcohol Health and Research World, 33, 152–160. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological assessment, 6(4), 284. Fleiss, J. L., & Shrout, P. E. (1978). Approximate interval estimation for a certain intraclass correlation coefficient. Psychometrika, 43, 259–262. Forsberg, L., Kallmen, H., Hermansson, U., Berman, A. H., & Helgason, A. R. (2007). Coding counsellor behavior in motivational interviewing sessions: Inter-rater reliability for the Swedish Motivational Interviewing Treatment Integrity Code (MITI). Cognitive Behaviour Therapy, 36(3), 162–169. Gallardo, M. E., & Curry, S. J. (2009). Shifting perspectives: Culturally responsive interventions with Latino substance abusers. Journal of Ethnicity in Substance Abuse, 8, 314–329. Hettema, J., Steele, J., & Miller, W. R. (2005). Motivational interviewing. Annual Review of Clinical Psychology, 1, 91–111. Koken, J. A., Naar-King, S., Umasa, S., Parsons, J. T., Saengcharnchai, P., Phanuphak, P., et al. (2012). A cross-cultural three step process model for assessing motivational interviewing treatment fidelity in Thailand. Health Education and Behavior, 39(5), 574–582.
Downloaded by [New York University] at 15:44 27 May 2015
218
C. S. Lee et al.
Lau, A. S. (2006). Making the case for selective and directed cultural adaptations of evidenced-based treatments: Examples from parent training. Clinical Psychology: Science and Practice, 13, 295–310. Lee, C. S., Lopez, S. R., Colby, S. M., Hernandez, L., Caetano, R., Borrelli, B., et al. (2011). A cultural adaptation of motivational interviewing to address heavy drinking among Hispanics. Cultural Diversity & Ethnic Minority Psychology, 17(3), 317–324. Lee, C. S., Lopez, S. R., Colby, S. M., Rohsenow, D., Hernandez, L., Borrelli, B., et al. (2013). Culturally adapted motivational interviewing for Latino heavy drinkers: Results from a randomized clinical trial. Journal of Ethnicity in Substance Abuse, 12(4), 356–373. Lee, C. S., Colby, S. M., Rohsenow, D. J., Lopez, S. R., Hernandez, L., & Caetano, R. (2013). Acculturation stress and drinking problems among urban heavy drinking Latinos in the Northeast. Journal of Ethnicity in Substance Abuse, 12(4), 308–320. Longabaugh, R., & Magill, M. (2011). Recent advances in behavioral addiction treatments: Focusing on mechanisms of change. Current Psychiatry Reports, 13(5), 382–289. Martinez, C. R., & Eddy, J. M. (2005). Effects of culturally adapted parent management training on Latino youth behavioral health outcomes. Journal of Consulting and Clinical Psychology, 73, 841–851. McCambridge, J., Day, M., Thomas, B. A., & Strang, J. (2011). Fidelity to Motivational Interviewing and subsequent cannabis cessation among adolescents. Addictive Behaviors, 36, 749–754. Miller, W. R., & Rose, G. S. (2010). Motivational interviewing in relational context. American Psychologist, 65, 298–299. Miller, W. R., & Rollnick, S. (2013). Motivational interviewing: Helping people change (3rd ed.), New York, NY: Guilford. Miller, W. R., & Rollnick, S. (2014). The effectiveness and ineffectiveness of complex behavioral interventions: Impact of treatment fidelity. Contemporary Clinical Trials, 37, 234–241. Mitchell, M. L., & Jolley, J. M. (2010). Research design explained. (7th ed.). Belmont, CA: Wadsworth, Cengage Learning Inc. Moyers, T. B., Martin, T., Manuel, J. K., Hendrickson, S. M. L., & Miller, W. R. (2005). Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment, 28(1), 19–26. Moyers, T. B., Martin, T., Manuel, J. K., Miller, W. R., & Ernst, D. (2010). Revised global scales: Motivational interviewing treatment integrity 3.1. 1 (MITI 3.1.1). University of New Mexico, Albuquerque, NM. Unpublished manuscript. Mulia, N., Ye, Y., Zemore, S. E., & Greenfield, T. K. (2008). Social disadvantage, stress, and alcohol use among Black, Hispanic, and White Americans: Findings from the 2005 U.S. National Alcohol Survey. Journal of Studies on Alcohol and Drugs, 69, 824–833. Mulia, N., Ye, Y., Greenfield, T. K., & Zemore, S. E. (2009). Disparities in alcohol related problems among white, black, and Hispanic Americans. Alcoholism: Clinical and Experimental Research, 33, 654–662.
Downloaded by [New York University] at 15:44 27 May 2015
Assessing Treatment Fidelity
219
Pierson, H. M., Hayes, S. C., Gifford, E. V., Roget, N., Padilla, M., Bissett, R., et al. (2007). An examination of the Motivational Interviewing Treatment Integrity code. Journal of Substance Abuse Treatment, 32, 11–17. Rollnick, S., & Miller, W. R. (1995). What is motivational interviewing? Behavioural and Cognitive Psychotherapy, 23, 325–334. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings—assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413–428. Santa Ana, E. J., Carroll, K. M., Anez, L., Paris, M., Ball, S. A., Nich, C., et al. (2009). Evaluating motivational enhancement therapy adherence and competence among Spanish-speaking therapists. Drug and Alcohol Dependence, 103, 44–51. Stinson, F. S., Grant, B. F., & Dufour, M. C. (2001). The critical dimension of ethnicity in liver cirrhosis and mortality statistics. Alcoholism: Clinical and Experimental Research, 25, 1181–1187. Szapocznik, J., Lopez, B., Prado, G., Schwartz, S. J., & Pantin, H. (2006). Outpatient drug abuse treatment for Hispanic adolescents. Drug and Alcohol Dependence, 84(Suppl.1), S54–S63. Transcripts from MI Videos (2013). Retrieved from http://www.motivationalinterview. org/trainers/side_bar/training_act_resources.html Vader, A. M., Walters, S. T., Prabhu, G. C., Houck, J. M., & Field, C. A. (2010). The language of motivational interviewing and feedback: Counselor language, client language, and client drinking outcomes. Psychology of Addictive Behaviors, 24(2), 190–197. Voas, R. B., Fisher, D. A., & Tippetts, A. S. (2002). Children in fatal crashes: Driver blood alcohol concentration and demographics of child passengers and their drivers. Addiction, 97, 1439–1448. Zemore, S. E., Karriker-Jaffe, K. J., Keithly, S., & Mulia, N. (2011). Racial prejudice and unfair treatment: Interactive effects with poverty and foreign nativity on problem drinking. Journal of Studies on Alcohol and Drugs, 72, 361–370.
