Psychological Reports: Mental & Physical Health 2013, 112, 3, 689-693. © Psychological Reports 2013
RELIABILITY AND VALIDITY OF THE OUTCOME QUESTIONNAIRE–45.21 DONALD L. BOSWELL, JASON K. WHITE, WENDY D. SIMS, R. STEVEN HARRIST, AND JOHN S. C. ROMANS Oklahoma State University Summary.—A random sample of 220 counseling center client records at a large Midwestern university, was used to assess reliability and validity of the Outcome Questionnaire–45.2 (OQ–45.2). The center uses the OQ–45.2 at intake. Internal consistency coefficients for the three subscales and the total score were acceptable, with high coefficients for the total score and Symptom Distress subscale and moderate coefficients for the Interpersonal Relations and Social Role subscales. Correlations among clients’ presenting concerns and OQ–45.2 scores were computed and lend strong support for the validity of the OQ–45.2 total score and the Symptom Distress subscale. Weaker support was found for the Interpersonal Relations and Social Role subscales.
The use of outcome measures by psychologists has increased in the last decade. In one survey, 29% of licensed psychologists reported that they used outcome assessments in their clinical practice (Phelps, Eisman, & Kohout, 1998). The original version of the Outcome Questionnaire-45 (OQ–45; Lambert, Lunnen, Umphress, Hansen, & Burlingame, 1994) was the seventh most common standardized outcome measurement reported by practitioners (Hatfield & Ogles, 2004). The OQ–45 provides an efficient measurement tool to assess the efficacy of clinical interventions of patients in therapy. The OQ–45 was designed with several objectives in mind: brevity, assessment of clinical changes over time, and low cost (Wells, Burlingame, Lambert, Hoag, & Hope, 1996). Wells, et al. (1996) reported reliability estimates of .70 to .94. In previous research, a reliability change index (RCI) also identified four categories of outcomes: Deteriorated, indicating a negative change of 14 or more points; No Change, indicating no significant improvement or deterioration; Improved, referring to a positive change of 14 or more points; and Recovered, indicating a shift in scores from the dysfunctional range to the functional range (Hansen, Lambert, & Forman, 2002). The OQ–45 has been used extensively cross-culturally (Jong, Nugter, Polak, Wagenborg, Spinhoven, & Heiser, 2007). The purpose of the current study is to examine the reliability and validity of the current version of this scale using a university counseling Address correspondence to Donald L. Boswell, 434 Willard Hall, School of Applied Health and Educational Psychology, Oklahoma State University, Stillwater, OK 74078 or e-mail ([email protected]). 1
DOI 10.2466/02.08.PR0.112.3.689-693
01_PR_Boswell_April2013.indd 689
ISSN 0033-2941
20/08/13 10:33 AM
690
D. L. BOSWELL, ET AL.
center sample. Concurrent validity of the OQ–45 and OQ–45.2 with other self-report measures is reported as good (Lambert, Hansen, Umphress, Lunnen, Okiishi, Burlingame, et al., 1996; Umphress, Lambert, Smart, Barlow, & Clouse, 1997). OQ–45.2 developers (Lambert, et al., 1996) reported sensitivity to be 84% (i.e., respondents correctly classified as belonging to the non-clinical group) and specificity to be 83% (i.e., respondents correctly classified as belonging to the clinical group). However, one group of researchers questioned the proposed factor structure of the OQ–45, when they failed to find support for three distinct domains in a confirmatory factor analysis using a sample of 1,082 clients from a variety of clinical settings (Mueller, Lambert, & Burlingame, 1998). The present study is important because the benefit of using outcome measures in clinical practice will be undermined if psychologists do not use measures with adequate reliability and validity. Internal consistency of the scale was assessed to further evaluate overall reliability. We also examined the relationship of clients’ presenting concerns and subscale scores to better understand the underlying constructs measured by the OQ–45.2. There has not been adequate research demonstrating the reliability and validity of the subscales. The purpose of the present study is to assess the psychometric properties of the subscales to assess discrimination among types of presenting concerns. METHOD Sample In order to examine the psychometric properties of the OQ–45.2, 220 files from a recent, five-year period containing OQ–45.2 data from a large, Midwestern university counseling center were randomly selected for review. Clients were all students. The sample included 79 men and 141 women (N = 220). The mean age of the clients was 23.0 yr. (SD = 5.81). The inclusion criteria for the study were that clients must have requested services at the university counseling center and completed all intake forms. In addition to the OQ–45.2 scores, data on clients’ presenting concerns and demographic information were also collected. In order to further assess the validity and reliability of the OQ–45.2, internal consistency reliability coefficients were computed for the three subscales. Clients’ presenting concerns were correlated with the three subscales and the Total score to further assess the construct validity of each of the subscales and the Total score. Measures The OQ–45.2 (Lambert, et al., 1996) is a self-report measure designed for assessing client progress with cost effective, brief, and repeated admin-
01_PR_Boswell_April2013.indd 690
20/08/13 10:33 AM
OQ–45.2 RELIABILITY AND VALIDITY
691
istrations. The OQ–45.2 is purported as a three-factor structure assessing Symptom Distress, Interpersonal Relations, and Social Role performance. The test developers, as well as other researchers, have reported psychometric properties of the OQ–45 or OQ–45.2. Test-retest reliability coefficients for the OQ–45.2 in a university sample were .78 for Symptom Distress, .80 for Interpersonal Relations, .82 for Social Role Performance, and .84 for Total score (Lambert, et al., 1996). Internal consistency reliabilities were .92, .74, .70, and .93, respectively, in a university sample; and in a clinical sample, they were .91, .74, .71, and .93, respectively (Lambert, et al., 1996). These findings indicate good reliability for the OQ–45.2 total score and Symptom Distress subscale, while the reliability of the Interpersonal Relations and Social Role subscales are merely sufficient. Procedure An online, random number generator was used to select the year of client file from the last five years; then an online, random letter generator was used to select two letters to represent first and last initials of clients to be included in the study. Files were selected based on these criteria. Due to personnel resource limitations, we discontinued data collection one semester after data were collected from 220 client files. As part of the standard protocol for the counseling center intake procedure, clients were administered the OQ–45.2 on their first visit. All clients indicated presenting concerns from a list of 13 concerns on the intake sheet (Table 1) by answering “Yes” or “No” to each symptom (i.e., dichotomous variable). These 13 presenting concerns were previously chosen and periodically revised according to common areas of concern for clients presenting at this center. The selection of these presenting concerns was based on clinical (not research) purposes and has not been assessed for validity or reliability. RESULTS AND DISCUSSION Cronbach’s alphas computed for the OQ–45.2 subscales and total scores in the present study were: Symptom Distress (.93), Interpersonal Relationships (.78), Social Role (.70), and Total score (.94). In general, the results of the present study indicated good internal consistency reliability for the OQ–45.2 and the OQ–45.2 subscales, although reliability of the Social Role and Interpersonal Relations subscales were weak. Results of the correlations with the clients’ presenting concerns are presented in Table 1. The variables were dummy coded as zero or one (present) and Pearson product moment correlations were computed. Anxiety, depression, and stress correlated the highest with Total score and Symptom Distress subscale, which supports the validity of the Symptom Distress scale as well as the overall score. As can be seen in Table 1, some sup-
01_PR_Boswell_April2013.indd 691
20/08/13 10:33 AM
692
D. L. BOSWELL, ET AL. TABLE 1 CORRELATIONS OF OQ–45.2 TOTAL SCORES AND SUBSCALE SCORES WITH PRESENTING CONCERNS (N = 220) Presenting Concern
Total
Symptom Distress
Interpersonal Relations
Social Role
Career/major indecision
.24†
.21†
.18†
.26†
Relationship issues
.26†
.17†
.38†
.24†
Family issues
.19†
.14*
.27†
.15*
Sexual orientation
.21†
.17*
.25†
.17*
Alcohol/drug problems
.13*
.09
.16*
.13
Body image issues
.33†
.31†
.33†
.18† .07
Eating disorder
.14*
.13
.17*
Sexual assault/rape
.20†
.17*
.24†
Death of friend/loved one
–.05
–.03
–.01
.13* –.07
Anxiety
.41†
.43†
.25†
.30†
Depression
.48†
.45†
.44†
.33†
Stress
.36†
.33†
.29†
.34†
*p < .05. †p < .01.
port was found for the Interpersonal Relations subscale as scores correlated statistically significantly with presenting concerns of relationship issues, family issues, sexual orientation, body image, and sexual assault/ rape, as well as anxiety, depression, and stress. Some support was also foundfor the Social Role subscale, comprised of items tapping work and school problems, as this subscale correlated statistically significantly with concerns about career indecision. In summary, it appears to us that there is clear support for the Symptom Distress subscale of the OQ–45.2; however, the support for the other two subscales of the measure is less clear from these procedures. REFERENCES
HANSEN, N., LAMBERT, M., & FORMAN, E. (2002) The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329-343. HATFIELD, D. R., & OGLES, B. M. (2004) The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research and Practice, 35, 485-491. DOI:10.1037/0735-7028.35.5.485. JONG, K., NUGTER, M., POLAK, M., WAGENBORG, J., SPINHOVEN, P., & HEISER, W. (2007) The Outcome Questionnaire (OQ–45) in a Dutch population: a cross-cultural validation. Clinical Psychology and Psychotherapy, 14, 288-301. LAMBERT, M. J., HANSEN, N. B., UMPHRESS, V., LUNNEN, K., OKIISHI, J., BURLINGAME, G. M., & REISINGER, C. W. (1996) Administration and scoring manual for the Outcome Questionnaire (OQ–45.2). Wilmington, DE: American Professional Credentialing Services.
01_PR_Boswell_April2013.indd 692
20/08/13 10:33 AM
OQ–45.2 RELIABILITY AND VALIDITY
693
LAMBERT, M. J., LUNNEN, K., UMPHRESS, V., HANSEN, N., & BURLINGAME, G. M. (1994) Administration and scoring manual for the Outcome Questionnaire (OQ–45.1). Salt Lake City, UT: IHC Center for Behavioral Healthcare Efficacy. MUELLER, R. M., LAMBERT, M. J., & BURLINGAME, G. M. (1998) Construct validity of the Outcome Questionnaire: a confirmatory factor analysis. Journal of Personality Assessment, 70, 248-262. DOI:10.1207/s15327752jpa7002_5. PHELPS, R., EISMAN, E. J., & KOHOUT, J. (1998) Psychological practice and managed care: results of the CAPP Practitioner Survey. Professional Psychology: Research and Practice, 29, 31-36. DOI:10.1037/0735-7028.29.1.31. UMPHRESS, V. J., LAMBERT, M. J., SMART, D. W., BARLOW, S. H., & CLOUSE, G. (1997) Concurrent and construct validity of the Outcome Questionnaire. Journal of Psychoeducational Assessment, 15, 40-55. DOI:10.1177/073428299701500104. WELLS, M., BURLINGAME, G., LAMBERT, M., HOAG, M., & HOPE, C. (1996) Conceptualization and measurement of patient change during psychotherapy: development of the Outcome Questionnaire and Youth Outcome Questionnaire. Psychotherapy, 33, 275-283. Accepted March 13, 2013.
01_PR_Boswell_April2013.indd 693
20/08/13 10:33 AM
RELIABILITY AND VALIDITY OF THE OUTCOME QUESTIONNAIRE–45.21 DONALD L. BOSWELL, JASON K. WHITE, WENDY D. SIMS, R. STEVEN HARRIST, AND JOHN S. C. ROMANS Oklahoma State University Summary.—A random sample of 220 counseling center client records at a large Midwestern university, was used to assess reliability and validity of the Outcome Questionnaire–45.2 (OQ–45.2). The center uses the OQ–45.2 at intake. Internal consistency coefficients for the three subscales and the total score were acceptable, with high coefficients for the total score and Symptom Distress subscale and moderate coefficients for the Interpersonal Relations and Social Role subscales. Correlations among clients’ presenting concerns and OQ–45.2 scores were computed and lend strong support for the validity of the OQ–45.2 total score and the Symptom Distress subscale. Weaker support was found for the Interpersonal Relations and Social Role subscales.
The use of outcome measures by psychologists has increased in the last decade. In one survey, 29% of licensed psychologists reported that they used outcome assessments in their clinical practice (Phelps, Eisman, & Kohout, 1998). The original version of the Outcome Questionnaire-45 (OQ–45; Lambert, Lunnen, Umphress, Hansen, & Burlingame, 1994) was the seventh most common standardized outcome measurement reported by practitioners (Hatfield & Ogles, 2004). The OQ–45 provides an efficient measurement tool to assess the efficacy of clinical interventions of patients in therapy. The OQ–45 was designed with several objectives in mind: brevity, assessment of clinical changes over time, and low cost (Wells, Burlingame, Lambert, Hoag, & Hope, 1996). Wells, et al. (1996) reported reliability estimates of .70 to .94. In previous research, a reliability change index (RCI) also identified four categories of outcomes: Deteriorated, indicating a negative change of 14 or more points; No Change, indicating no significant improvement or deterioration; Improved, referring to a positive change of 14 or more points; and Recovered, indicating a shift in scores from the dysfunctional range to the functional range (Hansen, Lambert, & Forman, 2002). The OQ–45 has been used extensively cross-culturally (Jong, Nugter, Polak, Wagenborg, Spinhoven, & Heiser, 2007). The purpose of the current study is to examine the reliability and validity of the current version of this scale using a university counseling Address correspondence to Donald L. Boswell, 434 Willard Hall, School of Applied Health and Educational Psychology, Oklahoma State University, Stillwater, OK 74078 or e-mail ([email protected]). 1
DOI 10.2466/02.08.PR0.112.3.689-693
01_PR_Boswell_April2013.indd 689
ISSN 0033-2941
20/08/13 10:33 AM
690
D. L. BOSWELL, ET AL.
center sample. Concurrent validity of the OQ–45 and OQ–45.2 with other self-report measures is reported as good (Lambert, Hansen, Umphress, Lunnen, Okiishi, Burlingame, et al., 1996; Umphress, Lambert, Smart, Barlow, & Clouse, 1997). OQ–45.2 developers (Lambert, et al., 1996) reported sensitivity to be 84% (i.e., respondents correctly classified as belonging to the non-clinical group) and specificity to be 83% (i.e., respondents correctly classified as belonging to the clinical group). However, one group of researchers questioned the proposed factor structure of the OQ–45, when they failed to find support for three distinct domains in a confirmatory factor analysis using a sample of 1,082 clients from a variety of clinical settings (Mueller, Lambert, & Burlingame, 1998). The present study is important because the benefit of using outcome measures in clinical practice will be undermined if psychologists do not use measures with adequate reliability and validity. Internal consistency of the scale was assessed to further evaluate overall reliability. We also examined the relationship of clients’ presenting concerns and subscale scores to better understand the underlying constructs measured by the OQ–45.2. There has not been adequate research demonstrating the reliability and validity of the subscales. The purpose of the present study is to assess the psychometric properties of the subscales to assess discrimination among types of presenting concerns. METHOD Sample In order to examine the psychometric properties of the OQ–45.2, 220 files from a recent, five-year period containing OQ–45.2 data from a large, Midwestern university counseling center were randomly selected for review. Clients were all students. The sample included 79 men and 141 women (N = 220). The mean age of the clients was 23.0 yr. (SD = 5.81). The inclusion criteria for the study were that clients must have requested services at the university counseling center and completed all intake forms. In addition to the OQ–45.2 scores, data on clients’ presenting concerns and demographic information were also collected. In order to further assess the validity and reliability of the OQ–45.2, internal consistency reliability coefficients were computed for the three subscales. Clients’ presenting concerns were correlated with the three subscales and the Total score to further assess the construct validity of each of the subscales and the Total score. Measures The OQ–45.2 (Lambert, et al., 1996) is a self-report measure designed for assessing client progress with cost effective, brief, and repeated admin-
01_PR_Boswell_April2013.indd 690
20/08/13 10:33 AM
OQ–45.2 RELIABILITY AND VALIDITY
691
istrations. The OQ–45.2 is purported as a three-factor structure assessing Symptom Distress, Interpersonal Relations, and Social Role performance. The test developers, as well as other researchers, have reported psychometric properties of the OQ–45 or OQ–45.2. Test-retest reliability coefficients for the OQ–45.2 in a university sample were .78 for Symptom Distress, .80 for Interpersonal Relations, .82 for Social Role Performance, and .84 for Total score (Lambert, et al., 1996). Internal consistency reliabilities were .92, .74, .70, and .93, respectively, in a university sample; and in a clinical sample, they were .91, .74, .71, and .93, respectively (Lambert, et al., 1996). These findings indicate good reliability for the OQ–45.2 total score and Symptom Distress subscale, while the reliability of the Interpersonal Relations and Social Role subscales are merely sufficient. Procedure An online, random number generator was used to select the year of client file from the last five years; then an online, random letter generator was used to select two letters to represent first and last initials of clients to be included in the study. Files were selected based on these criteria. Due to personnel resource limitations, we discontinued data collection one semester after data were collected from 220 client files. As part of the standard protocol for the counseling center intake procedure, clients were administered the OQ–45.2 on their first visit. All clients indicated presenting concerns from a list of 13 concerns on the intake sheet (Table 1) by answering “Yes” or “No” to each symptom (i.e., dichotomous variable). These 13 presenting concerns were previously chosen and periodically revised according to common areas of concern for clients presenting at this center. The selection of these presenting concerns was based on clinical (not research) purposes and has not been assessed for validity or reliability. RESULTS AND DISCUSSION Cronbach’s alphas computed for the OQ–45.2 subscales and total scores in the present study were: Symptom Distress (.93), Interpersonal Relationships (.78), Social Role (.70), and Total score (.94). In general, the results of the present study indicated good internal consistency reliability for the OQ–45.2 and the OQ–45.2 subscales, although reliability of the Social Role and Interpersonal Relations subscales were weak. Results of the correlations with the clients’ presenting concerns are presented in Table 1. The variables were dummy coded as zero or one (present) and Pearson product moment correlations were computed. Anxiety, depression, and stress correlated the highest with Total score and Symptom Distress subscale, which supports the validity of the Symptom Distress scale as well as the overall score. As can be seen in Table 1, some sup-
01_PR_Boswell_April2013.indd 691
20/08/13 10:33 AM
692
D. L. BOSWELL, ET AL. TABLE 1 CORRELATIONS OF OQ–45.2 TOTAL SCORES AND SUBSCALE SCORES WITH PRESENTING CONCERNS (N = 220) Presenting Concern
Total
Symptom Distress
Interpersonal Relations
Social Role
Career/major indecision
.24†
.21†
.18†
.26†
Relationship issues
.26†
.17†
.38†
.24†
Family issues
.19†
.14*
.27†
.15*
Sexual orientation
.21†
.17*
.25†
.17*
Alcohol/drug problems
.13*
.09
.16*
.13
Body image issues
.33†
.31†
.33†
.18† .07
Eating disorder
.14*
.13
.17*
Sexual assault/rape
.20†
.17*
.24†
Death of friend/loved one
–.05
–.03
–.01
.13* –.07
Anxiety
.41†
.43†
.25†
.30†
Depression
.48†
.45†
.44†
.33†
Stress
.36†
.33†
.29†
.34†
*p < .05. †p < .01.
port was found for the Interpersonal Relations subscale as scores correlated statistically significantly with presenting concerns of relationship issues, family issues, sexual orientation, body image, and sexual assault/ rape, as well as anxiety, depression, and stress. Some support was also foundfor the Social Role subscale, comprised of items tapping work and school problems, as this subscale correlated statistically significantly with concerns about career indecision. In summary, it appears to us that there is clear support for the Symptom Distress subscale of the OQ–45.2; however, the support for the other two subscales of the measure is less clear from these procedures. REFERENCES
HANSEN, N., LAMBERT, M., & FORMAN, E. (2002) The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329-343. HATFIELD, D. R., & OGLES, B. M. (2004) The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research and Practice, 35, 485-491. DOI:10.1037/0735-7028.35.5.485. JONG, K., NUGTER, M., POLAK, M., WAGENBORG, J., SPINHOVEN, P., & HEISER, W. (2007) The Outcome Questionnaire (OQ–45) in a Dutch population: a cross-cultural validation. Clinical Psychology and Psychotherapy, 14, 288-301. LAMBERT, M. J., HANSEN, N. B., UMPHRESS, V., LUNNEN, K., OKIISHI, J., BURLINGAME, G. M., & REISINGER, C. W. (1996) Administration and scoring manual for the Outcome Questionnaire (OQ–45.2). Wilmington, DE: American Professional Credentialing Services.
01_PR_Boswell_April2013.indd 692
20/08/13 10:33 AM
OQ–45.2 RELIABILITY AND VALIDITY
693
LAMBERT, M. J., LUNNEN, K., UMPHRESS, V., HANSEN, N., & BURLINGAME, G. M. (1994) Administration and scoring manual for the Outcome Questionnaire (OQ–45.1). Salt Lake City, UT: IHC Center for Behavioral Healthcare Efficacy. MUELLER, R. M., LAMBERT, M. J., & BURLINGAME, G. M. (1998) Construct validity of the Outcome Questionnaire: a confirmatory factor analysis. Journal of Personality Assessment, 70, 248-262. DOI:10.1207/s15327752jpa7002_5. PHELPS, R., EISMAN, E. J., & KOHOUT, J. (1998) Psychological practice and managed care: results of the CAPP Practitioner Survey. Professional Psychology: Research and Practice, 29, 31-36. DOI:10.1037/0735-7028.29.1.31. UMPHRESS, V. J., LAMBERT, M. J., SMART, D. W., BARLOW, S. H., & CLOUSE, G. (1997) Concurrent and construct validity of the Outcome Questionnaire. Journal of Psychoeducational Assessment, 15, 40-55. DOI:10.1177/073428299701500104. WELLS, M., BURLINGAME, G., LAMBERT, M., HOAG, M., & HOPE, C. (1996) Conceptualization and measurement of patient change during psychotherapy: development of the Outcome Questionnaire and Youth Outcome Questionnaire. Psychotherapy, 33, 275-283. Accepted March 13, 2013.
01_PR_Boswell_April2013.indd 693
20/08/13 10:33 AM
