This article was downloaded by: [University of Nebraska, Lincoln] On: 26 August 2015, At: 05:13 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: 5 Howick Place, London, SW1P 1WG

Psychotherapy Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/tpsr20

How and when feedback works in psychotherapy: Is it the signal? ab

bc

b

a

Ingunn Amble , Tore Gude , Pål Ulvenes , Sven Stubdal & Bruce E. Wampold

bd

a

Outpatient Clinic, Modum Bad Psychiatric Center, Vikersund, Norway

b

Research Institute, Modum Bad Psychiatric Center, Vikersund, Norway

c

Institute of Basic Medical Sciences, Faculty of Medicine, University of Oslo, Oslo, Norway

d

Department of Counseling Psychology, University of Wisconsin-Madison, Madison, WI, USA Published online: 14 Jul 2015.

Click for updates To cite this article: Ingunn Amble, Tore Gude, Pål Ulvenes, Sven Stubdal & Bruce E. Wampold (2015): How and when feedback works in psychotherapy: Is it the signal?, Psychotherapy Research, DOI: 10.1080/10503307.2015.1053552 To link to this article: http://dx.doi.org/10.1080/10503307.2015.1053552

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions

Psychotherapy Research, 2015 http://dx.doi.org/10.1080/10503307.2015.1053552

EMPIRICAL PAPER

How and when feedback works in psychotherapy: Is it the signal?

INGUNN AMBLE1,2, TORE GUDE2,3, PÅL ULVENES2, SVEN STUBDAL1, & BRUCE E. WAMPOLD2,4 Outpatient Clinic, Modum Bad Psychiatric Center, Vikersund, Norway; 2Research Institute, Modum Bad Psychiatric Center, Vikersund, Norway; 3Institute of Basic Medical Sciences, Faculty of Medicine, University of Oslo, Oslo, Norway & 4 Department of Counseling Psychology, University of Wisconsin-Madison, Madison, WI, USA

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

1

(Received 7 August 2014; revised 1 May 2015; accepted 6 May 2015)

Abstract Objective: Monitoring of ongoing psychotherapy is of crucial importance in improving the quality of mental health care, and feedback (FB) about patients’ progress has been established as a viable means. The essential feature of FB models is that patient progress is measured continuously through therapy. Aim: This study investigated the effect of receiving a warning signal when a patient is not achieving expected improvement (not-on-track), monitored with the Norwegian version of the patient FB system OQ®-Analyst. Method: Patients from six psychiatric clinics in Southern Norway (N = 259) were randomized to FB or no feedback (NFB). Results: For the total sample, the FB effects appeared early (session three). Receiving a warning signal did not change the slope of patients’ progress after the signal was given (FB versus NFB). FB seemed to be more effective with more severely distressed patients, although insignificant. Therapists indicated that the graphs imaging patient progress, and the accompanying discussion with the patient, were the most important aspects of FB. Conclusions: The use of OQ®-Analyst should be recommended in psychotherapeutic settings in Norway. Given the inconsistent results regarding the effect of warning signals, definitive conclusions about their effect may depend upon how and for whom it is used. Keywords: psychotherapy monitoring; feedback; Outcome Questionnaire; not-on-track cases

Introduction Feedback (FB) about patient progress has been established as a viable means to improve the quality of psychotherapy services. The essential feature of FB models is that patient progress is measured over the course of therapy, typically every session. The patient’s progress is compared with normative trajectories of change, which typically are adjusted for initial levels of mental distress, and this information is used to guide the therapy. Many systems for using FB to improve quality of mental health care have been developed, including the Clinical Outcomes in Routine Evaluation–Outcome Measure (CORE-OM; Barkham, Gilbert, Connell, Marshall, & Twigg, 2005; Barkham et al., 2001; Evans et al., 2000, 2002; Lutz et al., 2005), the Partners for Change Outcome Management System (PCOMS; Duncan,

2012; Miller, Duncan, Sorrell, & Brown, 2005), and the Outcome Questionnaire Analyst System (OQ®Analyst; Finch, Lambert, & Schaalje, 2001; Lambert, Whipple, Bishop et al., 2002; Lambert et al., 2003; Lutz, 2002). The manner in which the systems provide FB vary. For example, information on patient progress can be provided only to the therapist or to both the therapist and the patient. Furthermore, some FB systems contain algorithms that identify and produce some combination of signals, messages, or alerts for cases that exhibit deterioration or less than expected progress, given normative data. Thus, FB is a generic term and comprised one or more components, including scores on the measures, graphs of patient progress, normative trajectories, signals of various types, and reports. The evidence supports the conjecture that providing FB improves treatment outcome. Two of the

Correspondence concerning this article should be addressed to Ingunn Amble, Outpatient Clinic, Modum Bad Psychiatric Center, N-3370 Vikersund, Norway. Email: [email protected] © 2015 Society for Psychotherapy Research

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

2

I. Amble et al.

FB systems, the OQ®-Analyst and the PCOMS, have been studied thoroughly in clinical trials comparing a FB condition to treatment as usual (i.e., no feedback (NFB)) (Duncan, 2012; Lambert & Shimokawa, 2011). Based on a meta-analysis of Outcome Questionnaire-45 (OQ-45) studies, Lambert and Shimokawa reported an effect of FB versus NFB of d = .28 for all patients enrolled in the trials, but with a larger effect size for the not-on-track patients (d = .53). Despite the encouraging effects of providing FB to therapists about patient progress, there are still issues that need to be considered and investigated. First, the effect of FB may not be as robust as it appears. In a meta-analysis of 10 controlled treatment studies using different FB systems (of which 5 were OQ-45 studies) from the USA, UK, and Germany, Knaup, Koesters, Schoefer, Becker, and Puschner (2009) found a very small effect for FB versus NFB (d = .10). De Jong, van Sluis, Nugter, Heiser, and Spinhoven (2012) found no significant effect of FB on outcome in a large sample of psychiatric outpatients, although they did find that FB was effective for not-on-track cases when the therapists used the OQ®-Analyst. A second issue is whether providing both patient and therapist with FB is more effective than providing FB only to therapist. In Shimokawa, Lambert, and Smart’s (2010) meta analysis, the effect sizes for the two types of FB (i.e., to therapist only and to therapist and patient) were approximately equal. However, Knaup et al. (2009) concluded that FB was more effective when it was given to both patient and therapist versus only to therapist/staff. In studies in which FB is not given directly to the patient, it is unknown whether or not the therapist shares the information with the patient, which makes conclusions about FB to therapist versus FB to both therapist and patient tenuous. Third, it is theorized that FB helps to resolve problems for patients who are predicted to be treatment failures. The evidence suggests that this is the case as the effect appears to occur most robustly for not-ontrack cases (i.e., when progress deviates from the normative trajectory of change) (Shimokawa et al., 2010). This is particularly true for patients remaining in treatment after the warning signal is given, primarily by reducing the likelihood that the case will be classified as a failure (i.e., not making expected progress or deteriorating; Lambert & Shimokawa, 2011). For patients making expected progress (ontrack), FB does not appear to improve outcomes (Crits-Christoph et al., 2012; Harmon et al., 2007; de Jong et al., 2012; Lambert, Whipple, Smart, Vermeersch, & Nielsen, 2001; Lambert et al., 2002; Probst et al., 2013; Simon, Lambert, Harris,

Busath, & Vazquez, 2012; Slade, Lambert, Harmon, Smart, & Bailey, 2008; Whipple et al., 2003), nor does it appear that FB reduces the likelihood of going off track (e.g., Amble, Gude, Stubdal, Andersen, & Wampold, in press).1 Thus, it would appear that the warning signal might play the most critical role in FB. Indeed, Probst et al. (2013) found that trajectories of not-on-track patients were similar in FB and NFB conditions until the therapist in the FB condition was signaled that the patient was off track, but from that point forward the outcomes of patients in the FB condition improved significantly more than those in the NFB condition (i.e., in the condition where the therapist did not receive any FB, including the warning signal). However, Hawkins, Lambert, Vermeersch, Slade, and Tuttle (2004), Simon et al. (2013), and Amble et al. (in press) found the effect of FB to be similar for on-track and not-on-track patients. These three studies have in common that the patients, both in the on-track and not-on-track conditions, had a relatively higher initial level of distress than those in the other studies. This suggests that FB could be more important for patients with more severe dysfunction, which would be consonant with the fact that patients going off track are often initially more severely distressed (e.g., Lambert et al., 2001). A fourth issue is the need to identify how FB improves the quality of care. There is no evidence that therapists who have received FB on their patient’s progress for a period of time develop expertise—that is, they do not appear to achieve better outcomes over time (Tracey, Wampold, Lichtenberg, & Goodyear, 2014). Crits-Christoph et al. (2010) provided therapists FB about the progress of their patients compared with the average progress of patients, but this strategy had no effect: “Clinicians were apparently relatively unmotivated by feedback reports that were oriented toward helping them with patients as a group rather than feedback for specific patients” (2012, p. 302). As they suggested, it appears that the therapist does something for the particular patient in response to FB for that specific patient rather than learning something about treating patients in general (Tracey et al., 2014). Moreover, it appears that FB to therapists is effective through the warning signal, which alerts the therapist to the fact that this specific patient is not making expected progress. If this were true, trajectories prior to the occurrence of the first warning signal would not be affected by receiving information about patient progress, as was found by Probst et al. (2013). Furthermore, if receiving the warning signal is the critical issue, then off-track patients in the FB condition would have better outcomes than in the NFB condition after the point in therapy where the

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

Psychotherapy Research 3 patient went off track (i.e., received the first warning signal) (Probst et al., 2013). This explanation is bolstered by the fact that therapists, in the absence of any FB, tend to be unaware of when their patients are off track (Hannan et al., 2005). Thus it appears that therapists need to be made aware of when their patient goes off track through the use of warning signals so that they can alter therapy in a more helpful direction for the patient. An alternative explanation for how FB works to improve outcomes involves the integration of the patient progress information into treatment. Some therapists receiving information on patient progress may pay little attention to the graph and reports until a warning signal is provided. However, other therapists might discuss the FB report with the patient as part of the therapy independent of the occurrence of a warning signal. This discussion might communicate to the patient the importance of patient progress. Further, a discussion about the process and outcome of treatment might have a beneficial effect on progress. Some systems, such as the PCOMS, emphasize integrating the information into the process of therapy by discussing the patient’s progress in every session (Duncan, 2012), while the OQ®-Analyst leaves it to the therapist to share the report information with the patient or not. In our earlier study (Amble et al., in press), we examined the effect of FB based on data collected in an randomized controlled trial (RCT) using the OQ®-Analyst in Norwegian in- and outpatient clinics. In this trial, FB to therapists by the OQ®Analyst consisted of a report including a graph showing patient progress, superimposed onto curves showing the expected progress in patient groups with the same initial distress level as the actual patient, as well as a signal with a corresponding message. If the patient’s progress was poorer than expected, the report contained a warning signal. Patients who received warning signals were designated as signal cases or not-on-track cases. If the patient’s progress was as expected, they were designated as on-track cases. In this study, patients were assigned to either FB to both therapist and patient or to NFB. In the FB condition, patients completed the OQ-45 prior to each session, and the therapist shared the FB report with the patient in the session. In the NFB condition, patients completed the OQ45, but FB was not available to either the therapist or the patient. The results of this trial showed that receiving FB was effective. We were surprised that the effect of FB did not differ as a function of whether the patient went off track or not (p = .59), meaning that the FB effect was just as large for the on-track cases as for the not-on-track cases. As most of the OQ-45

studies have found an effect for the not-on-track cases only, we were interested in investigating further when the effect of FB occurred and what factors might be identified for when and how the FB worked. The first issue investigated in this study was whether receipt of the warning signal had an impact on the progress of therapy. Lambert et al. (2001) presented the prototypic result for a comparison of the not-on-track FB group to the not-on-track NFB showing that the warning signal was the critical feature. This result has been replicated in many studies, recently and most convincingly by Probst et al. (2013). In this study, they showed that no differences occurred in the conditions (FB versus NFB) before the point at which the patients first went off track (i.e., received a warning signal for the first time), but the signal influenced progress from that point forward. In the FB condition (i.e., the therapist and patient received the warning), the course of therapy changed for the better, whereas in the NFB condition (i.e., the therapist and patient were unaware of the warning signal), the patient may have improved (e.g., due, say, to regression toward the mean), but the rate of improvement was less than that for the FB patients. Similar to Probst et al., our aim was to examine the slopes prior to and after the first point at which the case went off track. Next, we wanted to identify the point in therapy at which receiving information about patient progress yielded an effect. If the integration of FB into therapy is a critical feature (and not only when receiving a warning signal), then it is reasonable to expect that FB effects might occur relatively early in therapy. On the other hand, if receiving a warning signal (an important message to the therapist that the patient’s progress is discrepant from an ideal state, as Sapyta, Riemer, & Bickman, 2005, suggest), then the effect of FB would occur after receiving warning signals, meaning relatively later in therapy. In our analyses, we examined therapy progress for FB and NFB patients among patients who received at least eight sessions. As discussed before, one explanation for the effects due to warning signals is that receipt of such signals is influenced by severity, as those who go off track are relatively more distressed than those who do not (see, e.g., Lambert et al., 2001), and FB seems to be more effective for more severely distressed patients. We do know that differences in outcomes among therapists increases as patient severity increases (Saxon & Barkham, 2012), suggesting that the effects of FB and severity might be related. Accordingly, we examined the association of the effects of FB with initial patient distress. Finally, we

4

I. Amble et al.

conducted a survey of participating therapists to ascertain what aspects of the FB system they rated as most important. In summary, the following research questions were addressed: (1)

(2) (3)

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

(4)

Does receiving a warning signal improve the progress for patients who go off track in the FB group compared to those who go off track in the NFB group? At which point in therapy does the FB effect (FB versus NFB) become significant? Is the effect of FB dependent on the patient’s initial distress level? What do the participating therapists indicate are important aspects of the OQ®-Analyst FB?

The data used in this investigation are part of a multisite RCT, the main outcome of which was reported by Amble et al. (in press). The purpose, as indicated above, was to use these data to explore further important questions that might clarify how FB works. The details of the sample, instruments, and procedures are available (viz., Amble et al., in press) and hence only a brief summary is provided here.

Method

The FB system. The OQ-Analyst software (OQ®-Analyst) provides the therapist and patient with a report showing a graph of the patient’s trajectory up to the current session. The report displays a comparison of the given patient’s rate of improvement with normative and expected rates of improvements based upon samples of patients at the same initial level of distress. The FB report includes a colored signal determined by the empirical method (i.e., by comparing progress to normative trajectories; see Lambert, Kahler, Harmon, & Burlingame, 2011). A white signal indicates that the patient’s score is similar to people in a state of normal functioning (i.e., below the clinical cutoff of 62), and termination of therapy could be considered. A green signal indicates that the progress is as expected but there is still need for more treatment. A yellow signal is given when there is concern about the patient’s progress or that a positive outcome is in doubt. A red signal indicates serious concern about the final outcome and that there is risk of deterioration unless changes are made. If a patient has an OQ-45 score that generates a red or a yellow signal (a warning signal) during therapy, the case is defined as a not-on-track case. No Clinical Support Tools (CST) (a supplement in the OQ®-Analyst to map the not-on-track cases) were used, as this RCT was begun before the CST were routinely added, and they were not available in the Norwegian version.

Instrument and FB System The OQ-45.2. The OQ-45 is a 45-item patient self-report instrument designed to assess experience of psychological distress, interpersonal functioning, and contentment with social role functioning (Lambert et al., 1996). The OQ-45 is typically given prior to the first and each subsequent therapy session, in either a paper/pencil or an electronic format, and takes 5–10 min to complete. The sum of all items gives a Total Distress score, ranging from 0 to 180, with higher scores being indicative of greater levels of psychological distress. The psychometric properties of the OQ-45 have been studied extensively in the USA and the instrument has been found to be reliable and valid (Beretvas, Kearney, & Barón, 2003; Bludworth, Tracey, & Glidden-Tracey, 2010; Chapman, 2003; Kim, Beretvas, & Sherry, 2010; Lambert et al., 1996; Mueller, Lambert, & Burlingame, 1998). The Norwegian version of the OQ-45 used in this study has been shown to have adequate test–retest reliability and internal stability (r = .85 and α = .93, respectively) and validity with other international instruments. The clinical cutoff score for the Norwegian version is 62 and the Reliable Change Index is 16 (Amble et al., 2014).

Procedures This RCT was developed to test the OQ®-Analyst in Norwegian naturalistic psychiatric settings. The project was approved by the Norwegian regional ethics committee. Six clinics signed a cooperation agreement based on the research protocol and agreed to implement the randomization procedures and contribute data to the project (Amble et al., in press). After having signed and returned an informed consent, the patients were randomized into the FB or NFB condition by gender and in blocks of eight. All the patients, in both conditions, filled in the OQ-45 online prior to each session. They were informed about their randomization condition after having completed the initial score. In this study, OQ®-Analyst FB reports were immediately generated by the OQ®-Analyst software on a session-bysession basis each time a patient completed the measure. For the patients in the FB condition, the therapists were instructed to consider the FB report, show it to the patient every session, and discuss the report when useful or necessary. The FB reports on the NFB patients could not be

Psychotherapy Research 5 opened by anyone other than the project leader and were not available to therapists and patients.

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

Participants During the inclusion period (June 2010–September 2013), 340 patients initially consented to participate, and were randomized to the FB or the NFB condition. At one of the sites 19 patients were not randomized correctly and were therefore excluded from the sample, leaving 321 to be distributed to the two conditions. To be included in the analyses, a patient was required to have completed the OQ45 for a minimum of two sessions representing the first and any subsequent session. Over the course of the study, 14 patients in the FB and 15 in the NFB condition never completed an OQ-45; 16 patients in the FB and 17 in the NFB condition completed the OQ-45 only once. Thus, 259 patients consented to participate and fulfilled the criteria for inclusion. The sample had a mean age of 35.8 years (SD: 11.66, range 18–65), and 68% being females (Amble et al., in press). The sample was randomized; 144 in the FB and 115 in the NFB condition. The patients seen in the clinics had a wide range of diagnoses and comorbidity. Forty-three licensed therapists employed at the six clinics participated.

Analyses With regard to detecting whether receipt of the signal affects subsequent patient progress, we conducted a piecewise longitudinal multilevel model, replicating the method used by Probst et al. (2013). In this analysis, we used only data from patients who went off track any time during treatment. We expected, consistent with prior research, that progress (i.e., slope) before the point at which the patient went off track would not differ for the two conditions (viz., FB versus NFB), whereas after this point, patients in the FB condition would improve more rapidly than the patients in the NFB condition. To examine the point at which FB had an effect, we selected patients who had eight or more sessions and graphed their outcomes for each condition. As well, we conducted independent sample t-tests between the OQ-45 scores at each time point.

As discussed previously, there is some empirical support for the conjecture that FB is more important for more severely distressed patients. To investigate whether there was an association between the severity at intake and the effect of FB vis-à-vis NFB, we examined the interaction between condition (FB versus NFB) and initial OQ-45 as a predictor of the final OQ-45 score. Finally, we conducted a survey of the participating therapists asking for their opinion about the importance of various aspects of the FB system (OQ®-Analyst) used during the trial.

Results Effects of Warning Signals To examine the effects of receiving a warning signal, all of the patients who went off track during the course of therapy were selected from the total sample of 259, yielding 59 patients in the FB condition who received one or more warning signals and 51 patients in the NFB condition (Table I). Longitudinal piecewise multilevel models were used to estimate the slopes of the OQ-45 scores before and after the first warning signal was given (Bauer & Curran, n.d.). In these analyses, unstructured covariance models were used because sessions occurred in varying time patterns (e.g., the time between sessions varied both between and within patients) and because commonly used structures (e.g., autoregressive (AR) and autoregressive integrated moving average (ARIMA) models) produced poorer fitting models. To accomplish these analyses, the session at which it was determined that the patient first went off track was identified for each patient, and the longitudinal data were centered at this point. First, an unconditional model with only slopes and intercept was examined. The intercept was 96.53, indicating that the mean OQ-45 score for the 110 off-track cases was above 96 when it was determined the patient went off track. The slope before the session at which the patient went off track was 0.847 indicating that the OQ-45 score was increasing at this rate each session. This value was significantly different from zero (p = .048), indicating that the off-track cases were deteriorating (elevating slope implying more symptoms). The slope following

Table I. Descriptive statistics, estimated values of not-on-track cases in the FB and NFB conditions.

FB NFB Total

N

Initial value

Value at first warning

End value

59 51 110

94.16 (4.53) 95.17 (3.29) 94.87 (2.26)

94.62 (3.96) 98.73 (2.89) 96.53 (1.97)

90.25 (4.52) 92.83 (3.31) 91.44 (2.25)

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

6

I. Amble et al.

Figure 1. The not-on-track cases, FB and NFB. The slopes prior and after the first warning signal.

the first time the patient went off track was −0.477 (p = .014), indicating that patients significantly improved after this point. The difference between the slopes before and after the first warning signal gives, however, limited information, as determination that a patient is not-on-track indicates an extreme score and regression toward the mean could explain decreasing scores after the signal. The second model was conditioned on whether FB, including the warning signal, was received or not (Figure 1). In the FB condition, the therapist and patient received the warning signal, where in the NFB condition they were unaware that the patient had gone off track. Slopes were tested as an interaction with the FB condition. The slopes of the pre-signal OQ-45 scores were not significantly different in the two conditions (p = .327), nor were the slopes after the signal (p = .987), although the effect was in the expected direction as the rate of improvement for the FB patients was greater than for the NFB patients. Therefore, it could not be conclusively demonstrated that receipt of a warning signal for cases in the FB condition resulted in better outcomes than in cases unaware of being off track.

patients. At the third session, there is a significant difference between conditions (t(123) = 2.01, p = .046, d = 0.36). At the eighth session the difference between FB and NFB was only slightly greater (t(123) = 2.21, p = .029, d = 0.40).

Does Initial Distress Level Make a Difference? To test the hypothesis that initial severity was related to the effect of FB, we performed a multiple regression analysis with the OQ-45 last score as the dependent variable. The independent variables were (a) the initial OQ-45 score, (b) condition (FB versus NFB), and (c) the interaction of the first score and condition, which is the test of whether the effect of FB depends on initial severity. The interaction effect did not produce a significant effect, although the anticipated direction approached significance (standardized β = 0.37, t = 1.58, p = .056, one tailed), indicating a possibility that the greater the initial distress, the larger the effect of FB.

Survey of Therapists When Did FB Make a Difference? Figure 2 presents the OQ-45 scores of 124 patients with eight or more sessions. As we found no difference in effect of FB between the on-track and the not-on-track cases (Amble et al., in press), patients from both groups are included, yielding 65 in the FB condition and 59 in the NFB condition (Table II). It appears that the superiority of FB occurs early in the treatment process for these

After the end of the trial, we conducted a survey of participating therapists to determine what they perceived to be the most important aspects of the FB system. In all, 19 of the 43 participating therapists responded (44.2%). However, these 19 therapists treated most of the patients (180/259, 69.5%). Therapists responded to items on a 5-point scale (1 = to a very small extent to 5 = to a very large extent). In response to the item “How often did you open the feedback reports?” the mean was 4.47

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

Psychotherapy Research 7

Figure 2. The progress of the OQ scores for the 124 patients with 8 sessions or more.

Table II. Descriptives for patients with more than eight sessions, FB and NFB condition.

Sample FB NFB Total

N

Mean initial OQ score (SD)

Mean end OQ score (SD)

65 59 124

94.92 (18.76) 95.80 (19.16) 95.34 (18.88)

79.60 (26.82) 89.66 (25.53) 84.39 (26.59)

(SD = 0.84), indicating that the responding therapists reported using the system. As well, they seemed to find it useful (mean response to “Overall Usefulness” was 4.00, SD = 0.67). As shown in Table III, therapists found that the patient’s progress graph was the most important aspect of the OQ®-Analyst system, followed by, in order of importance, the discussion with the patient, the risk questions, comparing the patient progress with the expected progress, the signal, and the computer-generated FB message.

Table III. Therapist survey N = 19 (1 = to a very small extent to 5 = to a very large extent). Questions

Mean

SD

How useful was the patient’s progress graph? How useful was the discussion with the patient? How useful was the risk questions? How useful was the comparison with the expected graph? How useful was the different signals? How useful was the FB message?

4.16 4.05 3.95 3.78

0.60 0.71 1.03 0.73

3.67 3.37

1.14 1.12

All means were above the midpoint of the scale, but the graph and discussion were more valued than the signal and the generated message.

Discussion The evidence from the present RCT suggests that the warning signal itself did not have an effect on the

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

8

I. Amble et al.

patient progress, as progress after the signal was not significantly different in the two conditions (FB versus NFB). This result is contrary to Probst et al. (2013), who examined the effect of the warning signal with piecewise multilevel models in the same way as performed in this study, and two other studies that examined the effect of FB for signal cases (Crits-Christoph et al., 2012; Simon et al., 2012). Examination of Figure 1 and the analysis of the difference of the slopes after the first warning signal show that there is a trend toward an effect favoring the FB condition and greater statistical power may have detected an effect. However, it is important to note that the present study had a number of off-track patients 2.5 times as great as was the case for Probst et al., who did detect an effect. An important difference between the current study and Probst et al. is that the latter used CST, whereas the present study did not. The result was unexpected and surprising, as it is claimed that the purpose of FB research is to maximize outcomes for predicted treatment failures. We would have expected that receiving a warning signal alerting the therapist on a possible treatment failure would have increased the FB effect. Third, there are differences between the present study and several of the other studies, in terms of initial distress level and the nature of the services (e. g., inpatient and outpatient, mental health versus college student samples). The conjecture that FB works most effectively with patients who are more distressed was supported by a trend seen in this study. Generally, when a warning signal is given, patients are relatively more distressed, which might explain why FB is effective only in the not-on-track cases in studies with a generally lower initial level of distress (e.g., Simon et al., 2012). Moreover, studies that have shown an effect overall (i.e., with the entire sample and not just the not-on-track cases) have been conducted with relatively more distressed patients in both the on-track and not-on-track conditions (viz., the current study, Hawkins et al., 2004; Simon et al., 2013). The role of initial distress is ambiguous, but the results in our study suggest that monitoring patient progress is even more important for more distressed patients, whether or not they are not-on-track cases. An important finding in the present study was that the effects of FB occurred early, with significant differences achieved by session three (Figure 2). This appears to be the first study to detect and report effects for FB so early in treatment. Interestingly, even with the early effects of FB, the rates of off-track cases were not different in the two conditions, meaning that FB did not seem to reduce the number of patients going off track.

Prior research is not clear whether the effects of FB are restricted to off-track cases or not, as studies have produced divergent conclusions. The fundamental issue here is whether the warning signal itself makes a difference, which was not found in the present study in contrast to several other studies. It could well be that some therapists are not utilizing the FB until the first warning signal is given, which might be typical in high case load contexts where therapists see a large number of patients and find it difficult to focus on the difficulties of a particular patient. A warning that a given patient is not progressing may be a “wake-up call” to make the therapists more attentive to this special case than they had previously been. Such a perspective is consistent with reports that many therapists do not use the FB information and that FB seems to work only, or to a greater extent, for those therapists who believe FB is useful (de Jong et al., 2012). This result is also consistent with the observation that therapist’s expertise does not seem to increase as a function of receiving FB (Crits-Christoph et al., 2010; Tracey et al., 2014). In the present study, both the patients and therapists were involved with the FB from the first session and the therapists were encouraged to discuss the FB reports with the patients. Therapists indicated that these discussions, as well as the patient’s progress graphs, were the most useful aspects of the FB system, which is consistent with the relatively early effects of FB detected in the present study. Interestingly, the PCOMS, which explicitly involves discussing progress with the patient (Duncan, 2012), has demonstrated effects across all patients (e.g., Anker, Duncan, & Sparks, 2009; Reese, Norsworthy, & Rowlands, 2009; Reese, Toland, Slone, & Norsworthy, 2010), although it is difficult to compare the two systems because of fundamental differences in the measures, procedures for using the measures, algorithms for determining signals, and in the FB messages (see Lambert et al., 2011). There are limitations in the present study. The heterogeneity of the clinics involved, and the relatively high distress of the patients, limits the generalizability of the results. It is difficult to situate the results of the present study in the array of previous results given the dimensions on which studies differ. Despite the desire to understand how FB changes therapist behavior with patients, we were limited to the questionnaire given to the therapists who reported their opinion about what was useful, a limitation further exacerbated by the relatively low response rate and a probable bias toward therapists who had a positive attitude toward using FB in their practice. Given these limitations and inconsistent results of previous studies, definitive conclusions should be avoided.

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

Psychotherapy Research 9 Part of the take-home message may be that these therapists in these settings with these patients may respond as well or better to general FB about client progress as they do to explicit signals. Setting up FB systems is partly about getting therapists to attend to what they are doing in a different way. Despite the limitations, there are some important issues that need to be considered in future research and in the use of FB systems. How FB data are analyzed and reported as in some studies is problematic because the results for the not-on-track patients only are given. For the manager of care who is accountable for the outcomes of all patients, the effect of FB for the entire patient population should be the central focus. Furthermore, whether a patient is not-on-track is not independent of treatment, and as such is not a natural grouping factor (i.e., treatment and the on-track/not-on-track are confounded). As well, the practice of examining the efficacy of FB by eliminating patients in the FB condition who do not have an OQ-45 score post signal, but failing to similarly trim the NFB condition, limits the generalizability of the outcome across all patients and biases the result in favor of the FB condition (Shimokawa et al., 2010). We recommend that the outcomes of all patients randomized to condition be analyzed and effects reported prior to any examination of subsamples. Future research needs to reflect curiosity about what happens in therapy as a result of FB. In treatment trials, it is not uncommon for sessions to be video recorded and analyzed. If the same were done in FB trials, a record of what therapists are doing with the FB in general and specifically when a warning signal occurs would provide insight into the process as well as the outcome of FB. It would also be important to know how the patients experience getting and using the FB in collaboration with their therapists, and which components they find most useful and helpful in their work to achieve a positive outcome in psychotherapy. Based on our findings, it would also be interesting to conduct a trial comparing only receiving the trajectory curve and the progress graph, with receiving the entire OQ®-Analyst FB report with the warning signals and corresponding messages and CST, and to compare a condition in which OQ®-Analyst FB is integrated into the treatment process with a condition in which the FB is given to therapist and patient but no systematic attempt is made for this information to be part of the treatment. Acknowledgements The authors would like to thank patients and staff from the Modum Bad Outpatient Clinic and

Drammen and Baerum District Psychiatric Centers who have delivered clinical data to this study.

Funding This research was funded by the Norwegian Ministry of Health, Extrastiftelsen (The Norwegian Council for Mental Health), and Modum Bad Psychiatric Center.

Note 1

In our trial (Amble et al., in press), we found that the proportion of patients going off track was not influenced by whether FB was provided (41.0%) or not (44.4%). We examined these proportions in 10 previous OQ-45 FB studies, and in none of these 10 studies were there any significant differences in the proportion of patients going off track as a function of FB. Over the 10 studies, 29.2% went off track in the FB conditions and 28.6% went off track in the no FB conditions (χ 2 = 0.71, p = .71).

References Amble, I., Gude, T., Stubdal, S., Andersen, B. J., & Wampold, B. E. (in press). The effect of implementing the Outcome Questionnaire-45.2 feedback system in Norway: A multisite randomized clinical trial in a naturalistic setting. Psychotherapy Research. Advance online publication. 1–9. doi:10.1080/ 10503307.2014.928756 Amble, I., Gude, T., Stubdal, S., Oktedalen, T., Skjorten, A. M., Andersen, B. J., … Wampold, B. E. (2014). Psychometric properties of the Outcome Questionnaire-45.2: The Norwegian version in an international context. Psychotherapy Research, 24 (4), 504–513. Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009). Using client feedback to improve couple therapy outcomes: A randomized clinical trial in a naturalistic setting. Journal of Consulting and Clinical Psychology, 77(4), 693–704. Barkham, M., Gilbert, N., Connell, J., Marshall, C., & Twigg, E. (2005). Suitability and utility of the CORE-OM and CORE-A for assessing severity of presenting problems in psychological therapy services based in primary and secondary care settings. The British Journal of Psychiatry, 186(3), 239–246. Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C., … McGrath, G. (2001). Service profiling and outcomes benchmarking using the CORE-OM: Toward practicebased evidence in the psychological therapies. Journal of Consulting and Clinical Psychology, 69(2), 184–196. Bauer, D. J., & Curran, P. J. (n.d.). Introduction to multilevel modeling: SPSS demonstration notes. Chapel Hill, NC: Curran-Bauer Analytics. Beretvas, S. N., Kearney, L. K., & Barón, A. (2003). A shortened form of the Outcome Questionnaire: A validation of scores across ethnic groups. Austin, TX: The Counseling & Mental Health Center, The University of Texas at Austin. Retrieved from http://cmhc.utexas.edu/pdf/OQ45report.pdf Bludworth, J. L., Tracey, T. J. G., & Glidden-Tracey, C. (2010). The bilevel structure of the Outcome Questionnaire-45. Psychological Assessment, 22(2), 350–355. Chapman, J. E. (2003). Reliability and validity of the progress questionnaire: An adaptation of the Outcome Questionnaire. Drexel University. Crits-Christoph, P., Ring-Kurtz, S., Hamilton, J. L., Lambert, M. J., Gallop, R., McClure, B., … Rotrosen, J. (2012). A

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

10

I. Amble et al.

preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs. Journal of Substance Abuse Treatment, 42(3), 301–309. Crits-Christoph, P., Ring-Kurtz, S., McClure, B., Temes, C., Kulaga, A., Gallop, R., … Rotrosen, J. (2010). A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers. Journal of Substance Abuse Treatment, 38(3), 251–262. Duncan, B. L. (2012). The Partners for Change Outcome Management System (PCOMS): The heart and soul of change project. Canadian Psychology/Psychologie Canadienne, 53(2), 93–104. Evans, C., Connell, J., Barkham, M., Margison, F., McGrath, G., Mellor-Clark, J., & Audin, K. (2002). Towards a standardised brief outcome measure: Psychometric properties and utility of the CORE-OM. The British Journal of Psychiatry, 180(1), 51–60. Evans, C., Mellor-Clark, J., Margison, F., Barkham, M., Audin, K., Connell, J., & McGrath, G. (2000). CORE: Clinical Outcomes in Routine Evaluation. Journal of Mental Health, 9 (3), 247–255. Finch, A. E., Lambert, M. J., & Schaalje, B. G. (2001). Psychotherapy quality control: The statistical generation of expected recovery curves for integration into an early warning system. Clinical Psychology & Psychotherapy, 8(4), 231–242. Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61(2), 155–163. Harmon, S. C., Lambert, M. J., Smart, D. M., Hawkins, E., Nielsen, S. L., Slade, K., & Lutz, W. (2007). Enhancing outcome for potential treatment failures: Therapist–client feedback and clinical support tools. Psychotherapy Research, 17(4), 379–392. Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K. L., & Tuttle, K. C. (2004). The therapeutic effects of providing patient progress information to therapists and patients. Psychotherapy Research, 14(3), 308–327. doi:10.1093/ptr/kph027 de Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research, 22(4), 464–474. doi:10.1080/10503307.2012.673023 Kim, S. H., Beretvas, S. N., & Sherry, A. R. (2010). A validation of the factor structure of OQ-45 scores using factor mixture modeling. Measurement and Evaluation in Counseling and Development, 42(4), 275–295. Knaup, C., Koesters, M., Schoefer, D., Becker, T., & Puschner, B. (2009). Effect of feedback of treatment outcome in specialist mental healthcare: Meta-analysis. The British Journal of Psychiatry, 195(1), 15–22. Lambert, M. J., Burlingame, G. M., Umphress, V., Hansen, N. B., Vermeersch, D. A., Clouse, G. C., & Yanchar, S. C. (1996). The reliability and validity of the Outcome Questionnaire. Clinical Psychology & Psychotherapy, 3(4), 249–258. Lambert, M. J., Kahler, M., Harmon, C., & Burlingame, G. M. (2011). Administration and Scoring Manual Outcome Questionnaire OQ-45.2. OQ Measures LLC. Lambert, M. J., & Shimokawa, K. (2011). Collecting client feedback. Psychotherapy, 48(1), 72–79. doi:10.1037/a0022238 Lambert, M. J., Whipple, J. L., Bishop, M. J., Vermeersch, D. A., Gray, G. V., & Finch, A. E. (2002). Comparison of empiricallyderived and rationally-derived methods for identifying patients at risk for treatment failure. Clinical Psychology & Psychotherapy, 9(3), 149–164.

Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10(3), 288–301. Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., & Nielsen, S. L. (2001). The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11(1), 49–68. Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., & Goates, M. (2002). Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clinical Psychology & Psychotherapy, 9(2), 91–103. Lutz, W. (2002). Patient-focused psychotherapy research and individual treatment progress as scientific groundwork for an empirically based clinical practice. Psychotherapy Research, 12 (3), 251–272. Lutz, W., Leach, C., Barkham, M., Lucock, M., Stiles, W. B., Evans, C., … Iveson, S. (2005). Predicting change for individual psychotherapy clients on the basis of their nearest neighbors. Journal of Consulting and Clinical Psychology, 73(5), 904–913. Miller, S. D., Duncan, B. L., Sorrell, R., & Brown, G. S. (2005). The Partners for Change Outcome Management System. Journal of Clinical Psychology, 61(2), 199–208. Mueller, R. M., Lambert, M. J., & Burlingame, G. M. (1998). Construct validity of the Outcome Questionnaire: A confirmatory factor analysis. Journal of Personality Assessment, 70(2), 248–262. Probst, T., Lambert, M. J., Loew, T. H., Dahlbender, R. W., Göllner, R., & Tritt, K. (2013). Feedback on patient progress and clinical support tools for therapists: Improved outcome for patients at risk of treatment failure in psychosomatic inpatient therapy under the conditions of routine practice. Journal of Psychosomatic Research, 75(3), 255–261. Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). Does a continuous feedback system improve psychotherapy outcome? Psychotherapy: Theory, Research, Practice, Training, 46(4), 418–431. doi:10.1037/a0017901 Reese, R. J., Toland, M. D., Slone, N. C., & Norsworthy, L. A. (2010). Effect of client feedback on couple psychotherapy outcomes. Psychotherapy: Theory, Research, Practice, Training, 47 (4), 616–630. Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of Clinical Psychology, 61(2), 145–153. doi:10.1002/jclp.20107 Saxon, D., & Barkham, M. (2012). Patterns of therapist variability: Therapist effects and the contribution of patient severity and risk. Journal of Consulting and Clinical Psychology, 80(4), 535–546. doi:10.1037/a0028898 Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78(3), 298–311. doi:10.1037/ a0019247 Simon, W., Lambert, M. J., Busath, G., Vazquez, A., Berkeljon, A., Hyer, K., … Berrett, M. (2013). Effects of providing patient progress feedback and clinical support tools to psychotherapists in an inpatient eating disorders treatment program: A randomized controlled study. Psychotherapy Research, 23(3), 287–300. doi:10.1080/10503307.2013.787497 Simon, W., Lambert, M. J., Harris, M. W., Busath, G., & Vazquez, A. (2012). Providing patient progress information and clinical

Psychotherapy Research

Downloaded by [University of Nebraska, Lincoln] at 05:13 26 August 2015

support tools to therapists: Effects on patients at risk of treatment failure. Psychotherapy Research, 22(6), 638–647. doi:10. 1080/10503307.2012.698918 Slade, K., Lambert, M. J., Harmon, S. C., Smart, D. W., & Bailey, R. (2008). Improving psychotherapy outcome: The use of immediate electronic feedback and revised clinical support tools. Clinical Psychology & Psychotherapy, 15(5), 287–303.

11

Tracey, T. J., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014). Expertise in psychotherapy: An elusive goal? American Psychologist, 69, 218–229. Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E. J. (2003). Improving the effects of psychotherapy: The use of early identification of treatment and problem-solving strategies in routine practice. Journal of Counseling Psychology, 50(1), 59–68.