What Is Psychotherapy? Proc. 9th Int. Congr. Psychother., Oslo 1973 Psychother. Psychosom. 25: 236-238 (1975)
Evaluation Criteria and Psychotherapy Research
There are two important problems in psychotherapy research that remain unsolved: (1) the specification of objectives, and (2) the collection of data appropriate to the specified objectives. These two problems have plagued most of the psychotherapy research to date and, in our view, are the principle reason that most of the evidence for or against the efficacy of psychotherapy still seems unconvincing (1). Either the criterion variables are inadequate or the data used to describe them are inappro priate. In fact, outcome criteria are often superimposed on the data by the statistical analysis. Much of the data of psychotherapy research consist of ‘scores’ on various psychological tests and rating scales. These scores are compared for each patient before, during and after therapy in an attempt to assess change attributable to psychotherapy. However, even if the various IQ tests, rating scales and projective tests used are objective and reliable measures of the criterion variables, they are not valid criteria. Standards are not inherent in any scale. No one is really interested in achieving change, per se. Patients usually come to psychotherapy with specific problems and most psychotherapists want to improve the patient’s interpersonal relationships as well as to alleviate his symptoms. It is difficult to imagine a therapeutic contract to achieve a change in score on some of the scales of the MMPI. Ridiculous as this may seem, many psychotherapy research pro jects are designed as if a change in the patient’s score on the MMPI or the Q-Sort or the Rorschach were the objectives of the therapy and valid criterion for the assessment of the effectiveness of treatment. All anyone actually knows about such changes in score is that they occurred. It has yet to be demonstrated that these ‘objective’ measures actually correlate with more substantial measures of behavior change. It seems to us that, if psychotherapy has content, as we believe it does, then
Downloaded by: Thomas Jefferson University Scott Library 147.140.20.32 - 10/16/2017 3:00:00 PM
Ruth Westheimer
( 549 ) 237
we ought to be able to distinguish people who have had therapy from those who have not, much as we might assess the effects of a course in history or chemistry. If people have learned new facts about themselves or a new way of relating their emotions and their experiences or a more adaptive way of handling their prob lems, then we ought to try to find this out. We take the position that the criteria used to evaluate psychotherapy ought to reflect its content and objectives as directly as possible. If we want to know what a patient has gained from treat ment, we believe that the most reliable way to find out is to ask him. Admit tedly, there is a coding problem in this mode of inquiry that does not appear to affect the scoring of the ‘objective’ tests, but we believe that the face validity of our data makes it actually much easier to interpret than the score on a test that has not been shown to reflect our criteria for evaluation. We also recognize that our objectives as therapists may not be the same as everyone else’s and that our criteria for effectiveness may not coincide with those of other researcher’s. However, we feel that as long as we can make our objectives explicit and can specify what changes in the patient’s behavior will have to occur to satisfy these objectives, then we can answer the question of the effectiveness of the type of psychotherapy that we practice. We continue to find that, no matter how objective and reliable some other kinds of data appear to be, if they do not accurately reflect changes in our criterion variables, there is absolutely no point in collecting them. We have taken the position, expressed at the American Psychiatric Association Meetings in 1970 and in 1971, that there is no psychological inventory, rating scale or projective test whose score or change in score will answer our questions about the efficacy of psychotherapy. For the past 5 years, our research group at the Beth Israel Hospital in Boston has been working to evolve and apply explicit outcome criteria for psychotherapy. We intend that our criteria reflect our objectives as therapists and the problems of each individual patient. We have tried, as Bergin recently suggested, to be as specific as possible about the type of patient that we are treating, the type of therapy we are practicing and the type of changes we require in each patient’s behavior to satisfy us that our treatment is effective (2). Our research design is based on the work of Dr. David Malan of the Tavi stock Clinic in London. Malan and his group designed outcome criteria for each patient individually, based on the patient’s presenting symptoms and on the objectives of the psychoanalytically oriented psychotherapists. They interviewed these patients after therapy and compared the results to their outcome criteria to determine the success of each case. We have elaborated on Malan's design in that we have a control group and we have stipulated the type of patient and the type of therapy under investigation. Our evaluation criteria are specified in terms of changes in the patient’s symptoms, his interpersonal relationships, his predisposition to his particular
Downloaded by: Thomas Jefferson University Scott Library 147.140.20.32 - 10/16/2017 3:00:00 PM
Westheimer
Westheimer
( 550 ) 238
type of neurosis and his ability to solve emotional problems. (These criteria will be described more fully with case examples by Dr. Leeman.) The patients are interviewed prior to treatment to establish their current level of functioning in each of our five categories and outcome criteria are established. We try to be as specific as possible about what the patient will have to tell us in follow-up in order to satisfy us that significant change has taken place. We believe that our data will be appropriate to our objectives as psycho therapists in the evaluation of psychotherapy. We are becoming increasingly confident in our use of interviews for collecting these data, in spite of their inherent ‘subjectivity’. Since we insist that the patient give us examples of how his treatment has helped him, if it has, we feel that it would be very difficult for our patients to fabricate good results to please us or for us to mistake their statements of satisfaction with therapy or themselves for evidence of change in behavior or predisposition. We have found that patients are able to document their statements regarding changes in symptoms, interpersonal relations, self understanding and problem solving, when they have taken place. Finally, we would like to emphasize that our research has two objectives. First, we are interested in testing the hypothesis that, for appropriate patients, treatment with short-term anxiety-provoking psychotherapy will enable them to meet our criteria for change. We would not expect untreated patients to be able to do this. Second, we would like to demonstrate the usefulness of patientspecific criteria for evaluation. Our research method is complicated and timeconsuming and we do not pretend to have all the bugs ironed out. Still, we believe that it is possible to specify valid criteria for each patient in advance of treatment and to document changes if they have taken place.
References
2
Meltzoff, J. and Kornreich, M.: Research in psychotherapy (Atherton Press, New York 1970). Rergin, A E. : The evaluation of therapeutic outcomes; in Rergin and Garfield Hand book of psychotherapy and behavior change (Wiley, New York 1971).
Request reprints from: Ruth Westheimer, 741 Gunderson, Oak Park, IL 60304 (USA)
Downloaded by: Thomas Jefferson University Scott Library 147.140.20.32 - 10/16/2017 3:00:00 PM
1
Evaluation Criteria and Psychotherapy Research
There are two important problems in psychotherapy research that remain unsolved: (1) the specification of objectives, and (2) the collection of data appropriate to the specified objectives. These two problems have plagued most of the psychotherapy research to date and, in our view, are the principle reason that most of the evidence for or against the efficacy of psychotherapy still seems unconvincing (1). Either the criterion variables are inadequate or the data used to describe them are inappro priate. In fact, outcome criteria are often superimposed on the data by the statistical analysis. Much of the data of psychotherapy research consist of ‘scores’ on various psychological tests and rating scales. These scores are compared for each patient before, during and after therapy in an attempt to assess change attributable to psychotherapy. However, even if the various IQ tests, rating scales and projective tests used are objective and reliable measures of the criterion variables, they are not valid criteria. Standards are not inherent in any scale. No one is really interested in achieving change, per se. Patients usually come to psychotherapy with specific problems and most psychotherapists want to improve the patient’s interpersonal relationships as well as to alleviate his symptoms. It is difficult to imagine a therapeutic contract to achieve a change in score on some of the scales of the MMPI. Ridiculous as this may seem, many psychotherapy research pro jects are designed as if a change in the patient’s score on the MMPI or the Q-Sort or the Rorschach were the objectives of the therapy and valid criterion for the assessment of the effectiveness of treatment. All anyone actually knows about such changes in score is that they occurred. It has yet to be demonstrated that these ‘objective’ measures actually correlate with more substantial measures of behavior change. It seems to us that, if psychotherapy has content, as we believe it does, then
Downloaded by: Thomas Jefferson University Scott Library 147.140.20.32 - 10/16/2017 3:00:00 PM
Ruth Westheimer
( 549 ) 237
we ought to be able to distinguish people who have had therapy from those who have not, much as we might assess the effects of a course in history or chemistry. If people have learned new facts about themselves or a new way of relating their emotions and their experiences or a more adaptive way of handling their prob lems, then we ought to try to find this out. We take the position that the criteria used to evaluate psychotherapy ought to reflect its content and objectives as directly as possible. If we want to know what a patient has gained from treat ment, we believe that the most reliable way to find out is to ask him. Admit tedly, there is a coding problem in this mode of inquiry that does not appear to affect the scoring of the ‘objective’ tests, but we believe that the face validity of our data makes it actually much easier to interpret than the score on a test that has not been shown to reflect our criteria for evaluation. We also recognize that our objectives as therapists may not be the same as everyone else’s and that our criteria for effectiveness may not coincide with those of other researcher’s. However, we feel that as long as we can make our objectives explicit and can specify what changes in the patient’s behavior will have to occur to satisfy these objectives, then we can answer the question of the effectiveness of the type of psychotherapy that we practice. We continue to find that, no matter how objective and reliable some other kinds of data appear to be, if they do not accurately reflect changes in our criterion variables, there is absolutely no point in collecting them. We have taken the position, expressed at the American Psychiatric Association Meetings in 1970 and in 1971, that there is no psychological inventory, rating scale or projective test whose score or change in score will answer our questions about the efficacy of psychotherapy. For the past 5 years, our research group at the Beth Israel Hospital in Boston has been working to evolve and apply explicit outcome criteria for psychotherapy. We intend that our criteria reflect our objectives as therapists and the problems of each individual patient. We have tried, as Bergin recently suggested, to be as specific as possible about the type of patient that we are treating, the type of therapy we are practicing and the type of changes we require in each patient’s behavior to satisfy us that our treatment is effective (2). Our research design is based on the work of Dr. David Malan of the Tavi stock Clinic in London. Malan and his group designed outcome criteria for each patient individually, based on the patient’s presenting symptoms and on the objectives of the psychoanalytically oriented psychotherapists. They interviewed these patients after therapy and compared the results to their outcome criteria to determine the success of each case. We have elaborated on Malan's design in that we have a control group and we have stipulated the type of patient and the type of therapy under investigation. Our evaluation criteria are specified in terms of changes in the patient’s symptoms, his interpersonal relationships, his predisposition to his particular
Downloaded by: Thomas Jefferson University Scott Library 147.140.20.32 - 10/16/2017 3:00:00 PM
Westheimer
Westheimer
( 550 ) 238
type of neurosis and his ability to solve emotional problems. (These criteria will be described more fully with case examples by Dr. Leeman.) The patients are interviewed prior to treatment to establish their current level of functioning in each of our five categories and outcome criteria are established. We try to be as specific as possible about what the patient will have to tell us in follow-up in order to satisfy us that significant change has taken place. We believe that our data will be appropriate to our objectives as psycho therapists in the evaluation of psychotherapy. We are becoming increasingly confident in our use of interviews for collecting these data, in spite of their inherent ‘subjectivity’. Since we insist that the patient give us examples of how his treatment has helped him, if it has, we feel that it would be very difficult for our patients to fabricate good results to please us or for us to mistake their statements of satisfaction with therapy or themselves for evidence of change in behavior or predisposition. We have found that patients are able to document their statements regarding changes in symptoms, interpersonal relations, self understanding and problem solving, when they have taken place. Finally, we would like to emphasize that our research has two objectives. First, we are interested in testing the hypothesis that, for appropriate patients, treatment with short-term anxiety-provoking psychotherapy will enable them to meet our criteria for change. We would not expect untreated patients to be able to do this. Second, we would like to demonstrate the usefulness of patientspecific criteria for evaluation. Our research method is complicated and timeconsuming and we do not pretend to have all the bugs ironed out. Still, we believe that it is possible to specify valid criteria for each patient in advance of treatment and to document changes if they have taken place.
References
2
Meltzoff, J. and Kornreich, M.: Research in psychotherapy (Atherton Press, New York 1970). Rergin, A E. : The evaluation of therapeutic outcomes; in Rergin and Garfield Hand book of psychotherapy and behavior change (Wiley, New York 1971).
Request reprints from: Ruth Westheimer, 741 Gunderson, Oak Park, IL 60304 (USA)
Downloaded by: Thomas Jefferson University Scott Library 147.140.20.32 - 10/16/2017 3:00:00 PM
1
