© 2014 American Psychological Association 1040-3590/14/$ 12.00 http://dx.doi.org/10.1037/a0037063

Psychological Assessment 2014, Vol. 26, No. 4, 1116-1126

Comparing Short Forms of the Social Interaction Anxiety Scale and the Social Phobia Scale R. Nicholas Carleton and Michel A. Thibodeau

Justin W. Weeks

University of Regina

Ohio University

Michelle J. N. Teale Sapach

Peter M. McEvoy

University of Regina

Centre for Clinical Interventions, Perth, Australia, and Curtin University

Samantha C. Horswill

Richard G. Heimberg

University of Regina

Temple University

The Social Interaction Anxiety Scale (SIAS) and the Social Phobia Scale (SPS; Mattick & Clarke, 1998) are companion scales developed to measure anxiety in social interaction and performance situations, respectively. The measures have strong discriminant and convergent validity; however, their factor structures remain debated, and furthermore, the combined administration length (i.e., 39 items) can be prohibitive for some settings. There have been 4 attempts to assess the factor structures of the scales and reduce the item content: the 14-item Social Interaction Phobia Scale (SIPS; Carleton et al., 2009), the 12-item SIAS-6/SPS-6 (Peters, Sunderland, Andrews, Rapee, & Mattick, 2012), the 21-item abbreviated SIAS/SPS (ASIAS/ASPS; Kupper & Denollet, 2012), and the 12-item Readability SIAS and SPS (RSIAS/RSPS; Fergus, Valentiner, McGrath, Gier-Lonsway, & Kim, 2012). The current study compared the short forms on (a) factor structure, (b) ability to distinguish between clinical and non-clinical populations, (c) sensitivity to change following therapy, and (d) convergent validity with related measures. Participants included 3,607 undergraduate students (55% women) and 283 patients with social anxiety disorder (43% women). Results of confirmatory factor analyses, sensitivity analyses, and correlation analyses support the robust utility of items in the SIPS and the SPS-6 and SIAS-6 relative to the other short forms; furthermore, the SIPS and the SPS-6 and SIAS-6 were also supported by convergent validity analyses within the undergraduate sample. The RSIAS/RSPS and the ASIAS/ASPS were least supported, based on the current results and the principle of parsimony. Accordingly, researchers and clinicians should consider carefully which of the short forms will best suit their needs. Keywords: social anxiety, short forms, psychometrics, Social Phobia Scale, Social Interaction Anxiety Scale

pending on the specifics of the situation (Heimberg, Brozovich, & Rapee, 2010); however, between 8% and 12% of adults (Furmark, 2002; Kessler, Berglund, et al., 2005; Kessler, Chiu, Demler, Merikangas, & Walters, 2005; McEvoy, Grove, & Slade, 2011) experience sufficient distress or impairment in their daily lives that they meet diagnostic criteria for social anxiety disorder (SAD). People who seek treatment for SAD report impairment in several areas of their lives (e.g., education, employment, family relation­ ships, friendships, marriage/romantic relationships; Schneier et al., 1994; Stein, McQuaid, Laffaye, & McCahill, 1999), high rates of comorbid symptoms (e.g., depression, alcohol use; Kessler, Chiu, et al., 2005; Schneier et al., 1994), and a low quality of life (Safren, Heimberg, Brown, & Holle, 1996). The pervasive and pernicious nature of SAD has prompted substantial efforts to improve theory, research, and assessment related to social anxiety. There are a variety of symptoms commonly associated with social anxiety (e.g., heart palpitations, blushing, trembling, avoid­ ance; Blanco, Nissenson, & Liebowitz, 2001), and the situations

Social anxiety refers to the anxiety experienced before, during, and after social situations, which include both interactions and performances (American Psychiatric Association, 2013). Most people report experiencing varying degrees of social anxiety de-

This article was published Online First June 16, 2014. R. Nicholas Carleton and Michel A. Thibodeau, Department of Psychol­ ogy, University of Regina; Justin W. Weeks, Department of Psychology, Ohio University; Michelle J. N. Teale Sapach, Department of Psychology, University of Regina; Peter M. McEvoy, Centre for Clinical Interventions, Perth, Australia, and the School of Psychology and Speech Pathology, Curtin University; Samantha C. Horswill, Department of Psychology, University of Regina; Richard G. Heimberg, Department of Psychology, Temple University. Correspondence concerning this article should be addressed to R. Nich­ olas Carleton, Anxiety and Illness Behaviours Laboratory, Department of Psychology, University of Regina, Regina, SK, Canada S4S 0A2. E-mail: Nick.Carleton @uregina.ca 1116

COMPARING SHORT FORMS OF THE SIAS AND SPS

within which these symptoms are experienced has often been divided into two broad categories (Blanco et al., 2001; Liebowitz, 1987; Mattick & Clarke, 1998). One of these categories is social performance (e.g., being the focus of attention, particularly in situations where a person expects to be evaluated). The other category is social interactions (i.e., interacting with one or more persons). The Social Phobia Scale (SPS; Mattick & Clarke, 1998) and the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) are companion measures that were developed as measures of social anxiety within each of these two broad situational categories and are among the most commonly used tools for assessing social anxiety and the outcome of psychosocial therapy. The measures exhibit strong convergent and discriminant validity (Heimberg, Mueller, Holt, Hope, & Liebowitz, 1992; Mattick & Clarke, 1998) and have a strong track record for discriminating individuals with SAD from those with other anxiety disorders (Peters, 2000) and from healthy controls (Brown et al., 1997; Heimberg et al., 1992). Despite the overall psychometric strengths of the SPS and SIAS, their factor structures have been debated for some time (Rodebaugh, Woods, Heimberg, Liebowitz, & Schneier, 2006; Safren, Turk, & Heimberg, 1998). In addition, the combined length can be prohibitive for some research and clinical settings (Edwards, Rob­ erts, Sandercock, & Frost, 2004; Jepson, Asch, Hershey, & Ubel, 2005; Kupper & Denollet, 2012; Peters, 2000). The concerns about factor structure and length of the SPS and SIAS have prompted at least four substantive, but distinct, attempts to assess the factor structure of both item sets while simultaneously reducing item content. The first attempt (Carleton et al., 2009) suggested a 14-item, three-factor solution (i.e., the Social Interaction Phobia Scale; SIPS) based on a large undergraduate sample (n = 319, 76% women) and a large clinical sample of patients with a primary diagnosis of SAD (n = 355, 54% women). The SIPS was designed to ensure a stable factor structure based on assessing the items from the SPS and SIAS both simultaneously and separately (i.e., allowing for anxiety related to social performance and social interactions to be represented as one or more factors). The authors used item-total statistics, exploratory factor analyses (EFA) assess­ ing all items together and the SPS and SIAS items separately, confirmatory factor analyses (CFA), and receiver operating char­ acteristic (ROC) analyses to identify a reduced item set. A subset of five SIAS items loaded on the first factor and comprised the SIPS social interaction anxiety subscale; in contrast, a subset of SPS items loaded on two separate but highly correlated factors and comprised the six-item (SIPS-SPS Factor 1) fear of overt evalua­ tion subscale (e.g., “I would get tense if I had to sit facing other people on a bus or a train”) and the three-item (SIPS-SPS Factor 2) fear of attracting attention subscale (e.g., “I worry I might do something to attract the attention of others”). The fear of overt evaluation subscale focuses on the perception of being observed for fear of being evaluated. In contrast, the fear of attracting attention subscale appears to focus on being noticed at all. The psychometric properties of the SIPS were replicated in a large undergraduate sample (n = 512, 68% women; Reilly, Carleton, & Weeks, 2012) and after a French translation (Duranceau, Peluso, Collimore, Asmundson, & Carleton, in press). The second attempt (Peters, Sunderland, Andrews, Rapee, & Mattick, 2012) produced two separate six-item, unifactorial solu­ tions (i.e., one for each of the SIAS-6 and the SPS-6) based on two

1117

large clinical samples (n = 456, 58% women; n = 446, 48% women), a community sample (n = 137, 58% women), and an undergraduate sample (n = 164, 70% women). The authors used item-total statistics, CFA, non-parametric item response theory modeling focused on diagnostic differentiation, ROC analyses, and correlations assessing convergent validity to identify a reduced item set with a stable factor structure for each. The third attempt (Kupper & Denollet, 2012) suggested a 21-item solution (i.e., referred to here as the Abbreviated SIAS and the Abbreviated SPS [ASIAS and ASPS]) based on a large Dutch community sample (N = 1,598, 49% women). The authors used item-total statistics, EFA, CFA, multivariate analyses of variance, and correlations assessing convergent validity to identify the reduced 21-item set with a stable factor structure across age and sex. The results indicated that the SPS would be best represented by 11 items (i.e., general scrutiny concerns) and the SIAS by 10 items (i.e., social interaction anxiety). In the fourth and most recent attempt (Fergus, Valentiner, McGrath, Gier-Lonsway, & Kim, 2012), the authors suggested a 12-item, two-factor solution (i.e., the Readability SPS and the Readability SIAS [RSPS and RSIAS, respectively]) based on a large undergraduate sample (n = 492, 44% women) and a patient sample with a primary anxiety disorder (17% with SAD; n = 145, 47% women). Item readability (see Fergus et al., 2012, for read­ ability metrics) was used as the primary criterion for item selection to ensure that each item could be read appropriately at the fifth- or sixth-grade level, and six items were selected from each of the original SIAS and SPS scales. The authors then used item-total statistics, CFA, ROC analyses, and correlations assessing conver­ gent validity to further assess the item content and factor structure of the RSIAS and RSPS. The four short forms, despite sharing the same initial pool of 39 items, have little item overlap (see Table 1). Indeed, only one item was included in all four short forms, seven items were shared across three of the short forms, 13 items were shared across two of the short forms, and seven items were unique to one short form. In light of the design differences, the short forms all have potential benefits (e.g., fewer items, increased item readability, diagnostic sensitivity) and drawbacks (e.g., less comprehensive symptom assessments), as detailed by the original authors; however, it is important to the study of social anxiety that direct comparisons between the short forms are made. Continued use of divergent measures of performance and interaction anxiety contributes to unnecessary complexity in the literature, adversely affecting the ability to compare outcomes across studies. Therefore, the main aim of this study was to directly compare existing short versions of the SIAS and SPS with respect to (a) factor structure, (b) sensi­ tivity to change following cognitive behavioral group therapy (CBGT) for SAD, and (c) convergent validity with related mea­ sures. M ethod P articipants Undergraduate sample. Archival undergraduate data were collected from 3,838 participants, of whom 3,607 provided com­ plete data on all variables of interest. Among participants in this subset who reported their sex (n = 3,214), 61.7% were women.

CARLETON ET AL.

1118 T able 1

SPS and SIAS Items Comprising the Short Forms and Their Subscale Designations SPSah 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

SIPS

1

SPS-6

1

1 I

1 1

ASPS

1

RSPS

1

1 1 1

1 2 1 2 2 i i

1

1 1 1

1 1 1 1 1 1

1 1

SIAS 20 items’* SIPS 1 2 3 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

SIAS-6

ASIAS

1

1

RSIAS

1 1 1

1 1

1 1

1

1

3

3

1

1 1 3 3

3

1 1 1 1 1

1 1 1

SIAS 19 itemsb

SIPS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

SIAS-6

ASIAS

1

1

RSIAS

1 1 1

1 1

1 1

1

1

3

3

1

1 1 3 3

3

1 1 1 1 1

1 1 1

Note. SPS = Social Phobia Scale; SIAS = Social Interaction Anxiety Scale; SIPS = Social Interaction Phobia Scale; SPS-6 = Social Phobia Scale, short form; ASPS = Abbreviated Social Phobia Scale; RSPS = Readability Social Phobia Scale; SIAS-6 = Social Interaction Anxiety Scale, short form; ASIAS = Abbreviated Social Interaction Phobia Scale; RSIAS = Readability Social Interaction Anxiety Scale. Items with the same numbers within a scale share the same factor. The factor structure denoted for the SIPS is in line with the three-factor structure proposed by Carleton et al. (2009), but in the current data the three-factor structure was not superior to a two-factor structure, one that consists of SIAS items, and the other that consists of SPS items. Dashes indicate the placement of the item “I find it easy to make friends my own age,” which was included in the 1989 20-item version of the SIAS but not in the 1998 19-item version of the SIAS. a Mattick and Clarke (1989). b Mattick and Clarke (1998).

A m ong participants w ho reported age (n = 3,510), m ean age was 19.08 years (SD = 1.87, range = 17 to 53). A m ong participants w ho reported race (n = 3,494), m ost identified as C aucasian (87.9% ), w ith th e rem ainder identifying as A frican A m erican (3.5% ), A sian (3.2% ), H ispanic (1.1% ), A m erican Indian (0.4% ), Pacific Islan d er (0.2% ), o r other (0.7% ). T he predom inance o f C aucasian undergraduate participants is generally com parable to the original A ustralian sam ple from M attick and C larke (1998), as w ell as the C anadian sam ple used by C arleton and colleagues (2009), the A ustralian sam ple used by Peters and colleagues (2012), and the D utch sam ple used by K upper and D enollet (2012). Fergus and colleagues (2012) m ay be the exception w ith respect to ethnicity, having exam ined a m idw estem A m erican sam ple o f w hich only 64% self-identified as C aucasian. T he ar­ chival undergraduate data w ere approved for use in the current research by the R esearch E thics B oard at the third a u th o r’s uni­ versity. Clinical sample. Participants also included 283 patients w ith either a prim ary or com orbid diagnosis o f SA D referred by health professionals (i.e., general practitioners, psychologists, psychia­ trists) to a com m unity clinic specializing in cognitive behavior therapy fo r anxiety and affective disorders. O f participants w ho reported all dem ographics (i.e., n = 281), m ean age o f the clinical sam ple w as 33.34 years (SD = 11.99, range = 1 8 -8 7 ), and 42.8% w ere w om en. M ost patients w ere b o m in A ustralia (72.2% ), w ith 15.3% b o m in E urope or the U nited K ingdom , 4.9% in A sia, 2.4% in N ew Z ealand, 1.0% in South A m erica, 0.7% in N orth A m erica, and 3.5% in o ther regions. L ess than h a lf (43.8% ) o f participants w ere em ployed. E thnicity p e r se w as n o t directly assessed in the

clinical sam ple, as is the historical custom for the clinic and A ustralian researchers (e.g., R apee, G aston, & A bbott, 2009); how ever, based on the predom inant ethnicity from the countries o f birth, and in line w ith the undergraduate sam ple, the overw helm ing m ajority o f participants w ere likely to be C aucasian. T he predom ­ inance o f C aucasian clinical participants is, again, generally com ­ parable to the sam ples used in the research to develop each short form . M o st participants w ere single (60.5% ) o r w ere eith er m arried or cohabitating (28.2% ); how ever, som e w ere separated or di­ vorced (11.0% ), and a sm all num ber (0.3% ) w ere w idow ed. A p ­ proxim ately tw o thirds (65% ) w ere using psychotropic m edication. T he process used to obtain inform ed w ritten consent to use p a ­ tie n ts’ data for research p urposes w as approved by the A rea H ealth S erv ice’s M ental H ealth H um an R esearch E thics C om m ittee. D iagnoses according to the fourth edition o f the Diagnostic and Statistical Manual o f Mental Disorders (DSM-IV: A m erican P sy­ chiatric A ssociation, 1994) w ere determ ined using the M ini Inter­ n ational N europsychiatric Interview (L ecrubier et al., 1997; S hee­ han et al., 1998, 1997), w hich w as adm inistered b y experienced diagnosticians w ith doctoral- o r m aste r’s-level clinical psychology qualifications. C linicians w ere videotaped and observed by a m ore senior clinician for periods o f eith er 12 (doctorate) or 24 (m aster’s) m onths after their qualifications. D iagnoses w ere discussed and discrepancies resolved at w eekly supervision. T hese interview s w ere recorded in line w ith standard training procedures o f the clinic, but given that these assessm ents w ere conducted w ithin a tertiary referral clinic rather than a form al research unit, precise figures on the proportion o f sessions that w ere recorded and then view ed and verified w ere not available; how ever, 100% o f diag-

COMPARING SHORT FORMS OF THE SIAS AND SPS

nostic assessments for participants in the present study were re­ viewed at weekly clinical review and supervision meetings. All patients completed a comprehensive questionnaire battery, includ­ ing but not limited to the questionnaires used in the current study. Diagnosticians had access to the raw responses during the clinical assessment, but the scale scores were not available until after the initial assessment interview. Up to three Axis I disorders were coded in the database, and primary disorders were identified by patients nominating their most debilitating problem. The most common primary diagnoses were SAD (68.6%), re­ current major depressive disorder (MDD, 18.5%), dysthymia (5.2%), panic disorder with or without agoraphobia (PD, 2.7%), and other (5%). Disorders coded as secondary included SAD (24.7%), MDD (19.5%), dysthymia (13.6%), generalized anxiety disorder (GAD, 9.1%), other (8%), and none (25.1%). The most common tertiary disorders were GAD (9.4%), SAD (6.6%), MDD (3.5%), PD (3.1%), other (7.2%), and none (70.2%). Thus, 74.9% of the sample had at least one comorbid disorder, and 29.8% had at least two comorbid disorders. A subsample of 79 of these patients all (1) had a principal diagnosis of SAD, (2) had identified group therapy as their pre­ ferred treatment modality (vs. individual treatment), and (3) could commit to attending scheduled sessions. This subsample com­ pleted a cognitive behavior group therapy (CBGT) program for SAD that has been demonstrated to be efficacious (Rapee et al., 2009) and effective (McEvoy, Nathan, Rapee, & Campbell, 2012). The other 204 patients from the clinical sample attended individual treatment or a different group treatment to address an alternative principal problem (e.g., depression, GAD) or did not proceed with or complete treatment. Of those in the 79-patient subsample, 78 completed all of the demographic and self-report measures, and responses from this clinical subsample at pre- and post-CBGT were examined to evaluate the treatment sensitivity of the various short forms. Mean age of this clinical subsample was 33.09 years (SD = 11.26, range = 18-71), and 33.3% were women. CBGT comprised 12 weekly, 2-hr sessions including psychoeducation, cognitive monitoring and restructuring, in vivo behavioral exper­ iments, elimination of safety behaviors, video feedback, attention training, identification and challenging of negative core beliefs, and relapse prevention (see McEvoy et a l, 2012, for more details). The 79 participants who completed the CBGT program and the self-report measures were compared to the other 204 participants who did not complete CBGT/post-treatment short forms using x2 analyses and independent-sample t tests. The ratio of women to men in the CBGT participants was lower than that of the sub­ sample who did not complete CBGT, at 33% relative to 47%, y2(l, N = 283) = 4.17,p = .041. There were no differences in age (p = .83). The CBGT completers reported slightly higher scores on the SIAS total (p = .018, r2 = 0.019) and the SIAS-6 (p = .013, r2 = 0.021) than did the 204 participants who did not complete the CBGT; however, no other comparisons between the forms were statistically significant (all ps > .05).

Measures Social Phobia Scale (SPS; Mattick & Clarke, 1998). The SPS is a 20-item self-report scale designed to measure anxiety in performance situations, such as public speaking or being the center of attention. The items (e.g., “I get nervous that people are staring

1119

at me as I walk down the street”) are rated on a 5-point Likert-type scale with values ranging from 0 (not at all characteristic or true o f me) to 4 (extremely characteristic or true o f me). The SPS has demonstrated high internal consistency, 4- to 12-week test-retest reliability (i.e., rs range from .66 to .93), convergent validity (Orsillo, 2001), diagnostic sensitivity to SAD (Heimberg et al., 1992), and discriminant validity (Brown et al., 1997; Mattick & Clarke, 1998). Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998). The original SIAS was a 20-item unpublished measure developed by Mattick and Clarke in 1989; however, Mattick and Clarke published a 19-item SIAS in 1998 after removing Item 5 from the original scale. The SIAS was designed to assess anxiety when interacting with others (i.e., in pairs or groups). The SIAS items (e.g., “I am tense mixing in a group”) are rated on a 5-point Likert-type scale with values ranging from 0 (not at all charac­ teristic or true o f me) to 4 (extremely characteristic or true o f me). The SIAS has demonstrated high internal consistency, 4- to 12week test-retest reliability (i.e., rs range from .86 to .92), conver­ gent validity (Orsillo, 2001; Rodebaugh et al., 2006), diagnostic sensitivity to SAD (Heimberg et al., 1992), and discriminant validity (Brown et al., 1997; Mattick & Clarke, 1998). Brief Fear of Negative Evaluation Scale, Straightforward Items (BFNE-S; Rodebaugh et al., 2004). The BFNE-S is an eight-item version of the BFNE (Leary, 1983) that is used for measuring fears of negative evaluation (e.g., “When I am talking to someone, I worry about what they may be thinking about me”). It comprises the eight straightforwardly worded items (i.e., Items 1, 3, 5, 6, 8, 9, 11, 12) from the original BFNE (Leary, 1983). Each item is rated on a 5-point Likert-type scale, ranging from 1 (not at all characteristic o f me) to 5 (extremely characteristic o f me). Two studies have reported these eight straightforwardly worded items as being more reliable and valid indicators of fear of negative evaluation than are the reverse-scored items (Rodebaugh et al., 2004, and Weeks et al., 2005, in undergraduate and clinical sam­ ples, respectively). Consequently, Rodebaugh et al. (2004) and Weeks et al. (2005) suggested utilizing only the eight straightfor­ ward BFNE items found in the BFNE-S to calculate the total score, a position supported by recent comparative analyses conducted by Carleton, Collimore, McCabe, and Antony (2011). The BFNE-S has demonstrated excellent internal consistency (as > .92), facto­ rial validity, and construct validity in undergraduate (Carleton, Collimore, & Asmundson, 2007; Rodebaugh et al., 2004) and clinical (Weeks et al., 2005) samples. In our study, the BFNE was included in the undergraduate data set but not the clinical data set. Fear of Positive Evaluation Scale (FPES; Weeks, Heimberg, & Rodebaugh, 2008). The FPES is a 10-item measure designed to assess fear of positive evaluation (e.g., “I feel uneasy when I receive praise from authority figures”). Items are rated on a 10point Likert-type scale ranging from 0 (not at all true) to 9 {very true). The item content includes two reversed-scored questions to assess acquiescence response tendencies; however, these items are not included in the FPES total score. The FPES has demonstrated good internal consistency and test-retest reliability for 4.5 months (Weeks, Heimberg, Rodebaugh, Goldin, & Gross, 2012) and strong factorial, discriminant, convergent, and criterion-related validity in predicting social anxiety in both undergraduate and clinical samples (Fergus et al., 2009; Weeks, Heimberg, & Rode­ baugh, 2008; Weeks et al., 2012; Weeks, Heimberg, Rodebaugh,

1120

CARLETON ET AL.

& Norton, 2008). In our study, the FPES was included in the undergraduate data set but not the clinical data set.

Analyses Confirming factor structures. CFAs were first conducted to assess fit of previously demonstrated factor structures and to guide subsequent analyses. For comparison purposes only, the fit indices for the full 20-item SPS and the 19-item SIAS have been included, each treated as having a separate factor; however, due to limited size of the clinical sample, only indices from the undergraduate sample were calculated. In previous analyses, the SIPS was found to have three factors (Carleton et al., 2009; Reilly et al., 2012); in contrast, the SPS-6 and SIAS-6 (Peters et al., 2012), the ASPS and ASIAS (Kupper & Denollet, 2012), and the RSPS and the RSIAS (Fergus et al., 2012) were each found to have a unitary structure. Each of these factor structures was tested and compared to unitary solutions. For the sake of comprehensiveness, a two-factor solu­ tion for the SIPS was also tested, placing the SPS and SIAS item sets each on single, correlated factors. The CFAs were performed with Amos 18.0 (Arbuckle, 2007), and data were inputted in a maximum-likelihood estimation pro­ cedure. Each model was evaluated using the following fit indices and 90% confidence intervals (CIs; where applicable): (1) chiSquare (values should not be significant), (2) chi-Square/degrees of freedom ratio (values must be less than 3.0 but should be less than 2.0), (3) comparative fit Index (CFI; values must be greater than .90, and ideal fits approach or are greater than .95), (4) Tucker-Lewis index (TLI; values must be greater than .90, and ideal fits approach or are greater than .95), (5) standardized rootmean-square residual (SRMR; values must be less than .10, and ideal fits approach or are less than .05), and (6) root-mean-square error of approximation (RMSEA; values must be less than .08, and ideal fits approach or are less than .05, with 90% confidence interval values below .10; Browne, & Cudeck, 1989, 1993; MacCallum, Browne, & Sugawara, 1996). Evaluations should empha­ size the latter five indices (Hu & Bentler, 1998). Multivariate normality was assessed using Mardia’s coefficient of multivariate kurtosis (Byrne, 2001) for all models, with results suggesting non-normal data; however, parameter estimates and most model fit indices are robust to non-normality given maximum-likelihood estimation and a sample size of 100 or more participants (Lei & Lomax, 2005). Nonetheless, we used the Bollen-Stine bootstrap chi-square approach and computed bootstrapped parameter esti­ mates from a maximum-likelihood procedure, which has been demonstrated as an adequate method for resolving non-normality (Byrne, 2001; Davison & Hinkley, 2006; Nevitt & Hancock, 2001). In all cases, the statistical significance value for the BollenStine bootstrap chi-square produced results comparable with those from the maximum-likelihood procedure for the CFA, suggesting that non-normality did not substantially impact the overall results. The aforementioned fit indices allowed for assessments of model fit within each short form (i.e., How well does a model built from those short form items fit the data?) as well as comparisons of different models within a short form (e.g., Does a unitary model built from those short form items fit the data better or worse than a two-factor model?)— commonly referred to as nested models. In doing so, models with poor fit can be identified irrespective of which items were included, and such models can be discounted as

poor even without direct comparisons to other models with other item inclusions; however, there is no definitive statistically defen­ sible way to compare two models with different items and different item counts that both meet or exceed the criteria requirements for the fit indices (Hoyle, 2012). Models that use different items are commonly referred to as non-nested models. Some researchers have argued that such mod­ els can be compared using Akaike’s information criterion (AIC), the Bayesian information criterion (BIC), or the expected crossvalidation index (ECVI), which all depend to varying degrees on the competing models having the same number of manifest vari­ ables (Akaike, 1987; Browne & Cudeck, 1989, 1993; Haughton, Oud, & Jansen, 1997; Kumar & Sharma, 1999; Rigdon, 1999). Relative to the AIC and the BIC, the ECVI calculation is less influenced by the number of parameters (Hoyle, 2012); further­ more, the ECVI is linearly related to the AIC (Wicherts & Dolan, 2004), suggesting that although the two will produce comparable categorical rankings, the ECVI will better represent comparisons of those rankings for non-nested models with different variable counts. Recognizing the inherent constraints in comparing non­ nested models with different item counts, but also that researchers depend on having some metrics for comparisons, the ECVI was used in the current study, and differences in the ECVI based on CIs have been noted. That said, the ECVI values are provided (1) for completeness of the presentation of fit indices, (2) to avoid the appearance of selective reporting, and (3) to demonstrate the expected positive relationship between number of items and the ECVI (Hoyle, 2012); however, the ECVI values are not intended as key comparator values across the non-nested models. Sensitivity to treatment change. Data from the subset of clinical participants who completed CBGT for SAD (n = 79) were included in analyses of sensitivity to treatment change. Mean changes from pre-treatment to post-treatment scores for each of the short forms were calculated. Paired-samples t tests were conducted to test for the statistical significance of differences between scores, and the associated r2 scores were calculated as effect sizes of change (Rosenthal, 1991). Greater r2 scores suggest a greater sensitivity to change and represent a form of criterion validity. The r2 scores were then used in Fisher’s r-z transformations, which were performed to facilitate statistical comparisons between the r values. Comparative convergent validity. The undergraduate data set, but not the clinical data set, included the BFNE-S and the FPES. Convergent validity was assessed by calculating boot­ strapped Pearson correlations between each of the short form subscales, the short form total scores, and the BFNE-S and FPES. Fisher’s r-z transformations were performed to facilitate statistical comparisons between the correlations.

Results Descriptive statistics and internal consistencies for both samples are reported in Table 2. Fit indices resulting from the CFAs of the tested factor structures for each of the short forms are reported in Table 3. Again, due to the limited size of the clinical sample, only indices from the undergraduate sample were calculated for the full SPS and SIAS, and even then only to serve as a comparative reference. The best fitting factor structures, as suggested by all of the fit indices, were generally in accordance with those presented

COMPARING SHORT FORMS OF THE SIAS AND SPS

1121

T able 2

Descriptive Statistics and Internal Consistencies Undergraduate sample (n = 3,607)

Clinical sample (n = 283)

Measure

M (SD)

Min/max

a

S

K

r with BFNE-S

r with

AIIC

FPES

M (SD)

Min/max

a

AIIC

S

K

SIAS items SIPS-SIAS SIAS-6 ASIAS RSIAS SPS items SIPS-SPS Aggregated SIPS-SPS Factor 1 SIPS-SPS Factor 2 SPS-6 ASPS RSPS SIPS total score

25.34(14.38) 5.80 (4.44) 5.99 (4.62) 11.70 (8.24) 7.25 (4.90) 20.96 (15.05) 9.06 (7.41) 6.32 (5.13) 2.75 (2.68) 5.71 (5.06) 12.11 (8.80) 6.48 (4.83) 14.86(11.02)

0/77 0/20 0/24 0/40 0/24 0/80 0/36 0/24 0/12 0/24 0/44 0/24 0/56

.93 .88 .84 .92 .85 .94 .90 .86 .80 .87 .90 .81 .93

.40 .60 .47 .53 .48 .43 .51 .51 .57 .52 .46 .42 .49

.30 .52 .63 .43 .38 .53 .63 .64 .78 .70 .50 .56 .52

-.58 -.41 -.37 -.53 -.57 -.65 -.50 -.40 -.29 -.40 -.62 -.51 -.60

.59 .53 .49 .59 .57 .56 .57 .54 .54 .52 .59 .53 .59

.53 .46

55.78 (12.43) 15.87 (3.94) 14.48 (4.78) 28.28 (7.36) 16.84 (4.35) 42.27 (16.23) 21.07 (8.58) 14.12(6.25) 6.95 (3.15) 13.52 (5.97) 25.87 (9.00) 12.58 (5.22) 36.94(10.95)

15/78 3/20 1/24 4/40 3/24 0/80 0/36 0/24 0/12 0/24 0/44 0/24 3/56

.86 .85 .72 .84 .70 .92 .89 .88 .76 .84 .87 .75 .89

.25 .55 .30 .37 .29 .36 .48 .55 .52 .47 .38 .33 .38

-.80 -1.28 -.38 -.88 -.78 -.10 -.33 -.37 -.18 -.26 -.27 .04 -.52

.50 1.48 -.43 .72 .47 -.48 -.65 -.73 -.75 -.79 -.29 -.62 -.06

Al .51 .52 .55 .54 .52 .49 .52 .55 .52 .55

Note. Min/max = minimum/maximum; AHC = average inter-item correlation; S = skew; K = kurtosis; BFNE-S = Brief Fear of Negative Evaluation Scale; FPES = Fear of Positive Evaluation Scale; SIAS = Social Interaction Anxiety Scale; SIPS-SIAS = Social Interaction Phobia Scale-Social Interaction Anxiety Scale, short form; SIAS-6 = Social Interaction Anxiety Scale, short form; ASIAS = Abbreviated Social Interaction Phobia ScaleRSIAS = Readability Social Interaction Anxiety Scale; SPS = Social Phobia Scale; SIPS-SPS Aggregated = SIPS Social Phobia Scale, short form, all items; SPS-6 = Social Phobia Scale, short form; ASPS = Abbreviated Social Phobia Scale; RSPS = Readability Social Phobia Scale.

in the original validation articles. Specifically, the three-factor solution for the SIPS, the tw o unitary solutions for the SPS-6 and SIA S-6, and tw o unitary solutions fo r the RSPS and R SIA S w ere in line w ith those presented in the original articles for each o f these m easures (C arleton et al„ 2009; Fergus et a l , 2012; Peters et al.,

2012). T he fit indices w ere better in the undergraduate sam ple than in the clinical sam ple for all m easures. In all cases, the m ulti-factorial and dual unifactorial solutions produced consistently and substan­ tively better fit indices relative to the alternative solutions. The three-factor structure for the SIPS originally proposed by C arleton et al. (2009) replicated w ell in the current undergraduate and clinical sam ples but w as not significantly different from a tw ofacto r structure (i.e., SIPS-SPS and SIP S-S IA S). B ased on the grow ing literature suggesting that the three-factor solution is su­ perior (D uranceau et al., in press; R eilly e t al., 2012), the follow ing results referring to the SIPS address both the tw o- and three-factor structures. U sing the undergraduate data, the SIPS, the SPS-6 and SIA S-6, and the RSPS and R SIA S all produced excellent fit indices indi­ cating robust solutions; how ever, also using the u ndergraduate data w ith the E C V I and recognizing the lim itations therein as m en ­ tioned in the A nalyses section (i.e., non-nested fit indices will increase w ith item counts), the SPS-6 and SIA S-6 produced the sm allest E C V I scores (i.e., 12 item s), follow ed by the RSPS and R SIA S (i.e., 12 item s), follow ed by the SIPS (i.e., 14 item s), follow ed by the A SPS and A SIA S (i.e., 21 item s), follow ed finally by the original SPS and SIA S (i.e., 39 item s). T he E C V I 90% confidence intervals overlapped for the SP S-6 and SIA S-6 as well as for the RSPS and R SIA S, but not w ith the SIPS-SPS and SIPS-SIA S. T he E C V I 90% confidence intervals for the R SPS and R SIA S did not overlap w ith any o f the o ther short form s. Im por­ tantly, as per the concerns raised in the A nalyses section and expectations fo r the fit index (H oyle, 2012), the E C V I ranking w as in order based on num ber o f item s included in producing the m odel

fit, suggesting that the num ber o f item s still played a significant role by increasing the E C V I scores. U sing the clinical data, the SIPS as w ell as the SPS-6 and SIA S-6 both produced good fit indices; how ever, also using the clinical data and using the E C V I, the SPS-6 and SIA S-6 produced the sm allest E C V I scores (i.e., 12 item s), follow ed by the RSPS and R SIA S (i.e., 12 item s), follow ed by the SIPS (i.e., 14 item s), follow ed finally by the A SPS and A SIA S (i.e., 21 item s). O nce again, the E C V I rankings appear to have been im pacted by the n um ber o f item s (H oyle, 2012) and, as such, should be given little or no w eight in com paring the short form s. P re-treatm ent and post-treatm ent scores o f individuals w ho par­ ticipated in C B G T fo r SA D are reported in T able 4 w ith the associated paired t test statistic. In addition, the table presents the associated effect sizes and r to

z

transform ation com parisons o f

those effect sizes. Scores for the individual short form s reduced substantially from pre-treatm ent to post-treatm ent. T he correlations betw een each o f the short form s and the B FN E -S and FPES w ere all relatively com parable (see T able 2). C orrelations w ith the B FN E -S ranged from .49 to .59 and w ith the FPES ranged from .46 to .55. T he only readily apparent pattern w as that the SPS-6 and SIA S-6 had slightly sm aller correlations w ith fears o f negative evaluation (i.e., B FN E -S r = .49 and r = .52, respectively) relative to the o ther short form subscales and total scores (i.e., BFN E-S rs = .53 to .5 9 ).' D escriptive statistics and internal consistency for each o f the short form s w ithin the undergraduate and the clinical sam ples are

1 The differences in the relative sizes of the dependent correlations would all be statistically significant using a Williams’ T12 statistic because of the large sample size; however, even the largest difference in correlation values (i.e., .49 and .59) results in a difference in the percentage of variance accounted for of BFNE-S scores that is approximately 1%, which is unlikely to make a practical difference with respect to predictive validity. As such, the post hoc Williams’ T2 statistic comparisons were not included.

CARLETON ET AL.

1122 Table 3

Confirmatory Factor Analyses and Fit Indices for the Proposed Factor Structures Measure

SPS and SIAS SIPS SIPS SIPS SPS-6 and SIAS-6 ASPS and ASIAS RSPS and RSIAS

SIPS SIPS SIPS SPS-6 and SIAS-6 ASPS and ASIAS RSPS and RSIAS

RMSEA

RMSEA 90% Cl

ECVI

ECVI 90% Cl

Undergraduate sample (n = 3,607) 0.046 0.887 0.880 0.055 0.820 0.810 0.030 0.955 0.963 0.032 0.950 0.958 0.862 0.067 ' 0.883 0.021 0.977 0.981 0.036 0.947 0.936 0.043 0.909 0.918 0.053 0.873 0.859 0.039 0.938 0.950 0.045 0.901 0.919

0.058 0.076 0.062 0.065 0.108 0.043 0.072 0.071 0.088 0.067 0.085

0.057,0.059 0.075, 0.077 0.059, 0.065 0.062,0.068 0.105,0.111 0.039, 0.047 0.068,0.076 0.069,0.073 0.086, 0.090 0.063,0.071 0.081,0.089

2.628c 4.294a 0.329jk 0.365jj 0.948f 0.131m 0.316^ 1.01 l f 1.541e 0.268, 0.419h

2.542,2.714 4.187,4.402 0.300, 0.360 0.335, 0.398 0.896, 1.100 0.115,0.150 0.287,0.346 0.959, 1.067 1.474, 1.607 0.242,0.296 0.385,0.454

Clinical sample (n = 283) 0.054 0.889 0.910 0.054 0.893 0.910 0.077 0.749 0.788 0.921 0.052 0.937 0.092 0.690 0.746 0.071 0.835 0.852 0.092 0.644 0.680 0.077 0.840 0.797 0.710 0.081 0.763

0.068 0.067 0.103 0.064 0.131 0.077 0.113 0.085 0.102

0.804, 1.075 0.054,0.080 0.010, 0.115 0.048, 0.080 0.118,0.146 0.069,0.085 0.106,0.121 0.070, 0.100 0.087, 0.116

0.926f 0.917f 1.381e 0.670gh 1.378e 2.09l d 3.541b 0.748gf 0.919f

0.804, 1.075 0.795, 1.066 1.204, 1.585 0.572,0.792 1.193, 1.590 1.869, 2.341 3.229, 3.880 0.626, 0.897 0.774, 1.091

Factor

x2

df

X2/df

2 1 3 2 1 2 1 2 1 2 1

9,249.31* 16,481.97* 1,079.40* 1,231.73* 3,337.17* 400.06* 1,067.41* 3,565.10* 5,431.70* 917.51* 1,463.01*

701 702 74 76 77 53 54 188 189 53 54

13.19 23.48 14.49 16.21 43.34 7.54 19.77 18.96 28.74 17.31 27.09

3 2 1 2 1 2 1 2 1

171.08* 172.54* 305.41* 114.50* 316.62* 503.72* 872.54* 160.88* 211.11*

74 76 77 53 54 188 189 53 54

2.31 2.27 3.96 2.16 5.86 2.67 4.61 3.04 3.91

CFI

TLI

SRMR

Note. CFI = comparative fit index; TLI = Tucker-Lewis index; SRMR = standardized root-mean-square residual; RMSEA = root-mean-square error of approximation; Cl = confidence interval; ECVI = expected cross-validation index; SPS = Social Phobia Scale; SIAS = Social Interaction Anxiety Scale; SIPS = Social Interaction Phobia Scale; SPS-6 = Social Phobia Scale, short form; SIAS-6 = Social Interaction Anxiety Scale, short form; ASPS = Abbreviated Social Phobia Scale; ASIAS = Abbreviated Social Interaction Phobia Scale; RSPS = Readability Social Phobia Scale; RSIAS = Readability Social Interaction Anxiety Scale. Values within columns (even across samples) that have different subscripts are significantly different based on the 90% C l.

> < .01. also included in Table 2. According to Clark and Watson (1995), average inter-item correlations for broad constructs (e.g., extraver­ sion) should range from .15 to .25, whereas more narrow con­ structs should have higher average inter-item correlations, ranging from .40 to .50. In both undergraduate and clinical samples, each of the short forms exhibited good internal consistency (as > 0.80; Tabachnick & Fidell, 2013); however, overall, the SIPS items produced the highest internal consistency scores. Similarly, in both undergraduate and clinical samples, the SIPS items produced the highest average inter-item correlations and the RSPS items pro­ duced the lowest. Finally, each of the short forms exhibited ac­ ceptable skew and kurtosis values (Tabachnick & Fidell, 2013) in both undergraduate and clinical samples.

Discussion Measurement of social anxiety symptoms is an important pri­ ority for researchers and clinicians alike. Contemporary theories of SAD suggest two categories of situations that elicit social anxiety symptoms—social performance and social interactions (Blanco et al., 2001; Heimberg et al., 2010; Hofmann, Heinrichs, & Moscovitch, 2004). The SPS and SIAS are popular companion measures of social anxiety symptoms experienced in response to social performance and social interactions, respectively. The measures themselves are well supported psychometrically (Brown et al., 1997; Heimberg et al., 1992; Mattick & Clarke, 1998; Peters, 2000), but the combined factor structure for these scales remains

debated (Rodebaugh et al., 2006; Safren et al., 1998) and the combined 39-item length may be a challenge in some research and clinical settings (Edwards et al., 2004; Jepson et al., 2005; Kupper & Denollet, 2012; Peters, 2000). As such, in 2009, there was an initial attempt to assess the factor structure while simultaneously reducing the item content (Carleton et al., 2009), followed by three distinct additional attempts (Fergus et al., 2012; Kupper & Denol­ let, 2012; Peters et al., 2012). The current study was designed to compare, contrast, and provide an overview of the four short forms with respect to (a) factor structure, (b) sensitivity to change fol­ lowing CBGT for SAD, and (c) convergent validity with related measures. In short, the current results support the robust utility of items in the SIPS and the SPS-6 and SIAS-6 relatively more than the other short forms; that said, there are several caveats associated with that statement that warrant careful attention. The SIPS, the SPS-6 and the SIAS-6, and the RSPS and RSIAS are nearly comparable in length (i.e., 14, 12, and 12 items, respec­ tively). In contrast, the ASPS and the ASIAS are longer (21 items). Based on the principle of parsimony, the SIPS, the SPS-6 and the SIAS-6, and the RSPS and RSIAS should be considered relatively superior. From the original item set, only one item (“I get nervous that people are staring at me as I walk down the street”) was included in all four short forms. A total of seven items were shared across three of the short forms, 13 items were shared across two of the short forms, and seven items were unique to one short form. There was no readily apparent pattern to the item overlap across

COMPARING SHORT FORMS OF THE SIAS AND SPS

1123

Table 4 Changes in Scores From Pre-Treatment to Post-Treatment (N = 79) Pre-treatment Measure SIAS items SIAS SIPS-SIAS SIAS-6 ASIAS RSIAS SPS items SPS SIPS-SPS Aggregated SIPS-SPS Factor 1 SIPS-SPS Factor 2 SPS-6 ASPS RSPS SIPS total score

Post-treatment

M

SD

M

SD

f

Effect size of change (r2)

58.58 16.60 15.53 29.56 17.54

11.07 3.40 4.14 6.38 3.84

43.16 11.66 10.61 20.44 12.30

14.62 4.43 4.70 8.36 4.77

10.29 10.37 9.66 9.56 11.53

.58 „ .58, h .54ah .54 h .63

43.95 22.04 14.80 7.24 14.04 26.56 12.81 38.61

15.01 8.00 5.91 3.04 5.81 8.44 4.93 10.42

25.73 12.84 8.39 4.44 7.77 15.71 7.71 24.49

13.66 7.64 5.43 3.10 5.19 8.13 4.34 11.15

11.53

.63 6K .63 •44h ■6L„ ,62a

11.11

11.56 7.79 10.98 11.32 9.89 12.33

.561 ■66a

Note. SIAS = Social Interaction Anxiety Scale; SIPS = Social Interaction Phobia Scale; SIPS-SIAS = SIPS Social Interaction Anxiety Scale, short form; SIAS-6 - Social Interaction Anxiety Scale, short form; ASIAS = Abbreviated Social Interaction Phobia Scale; RSIAS = Readability Social Interaction Anxiety Scale; SPS = Social Phobia Scale; SIPS-SPS Aggregated = SIPS Social Phobia Scale, short form, all items; SPS-6 = Social Phobia Scale, short form, ASPS — Abbreviated Social Phobia Scale; RSPS - Readability Social Phobia Scale. Effect size values with different subscripts across groups in the column are significantly different (p < .05), with subscript a indicating the largest value(s) and subscript b indicating the smallest value(s). a All fs are statistically significant {p < .001).

short forms; however, the differences may have resulted from different goals of the various research groups. There was, however, one consistent and readily apparent pattern to the absence of items across the short forms; specifically, none of the short forms re­ tained any reverse-scored items. That said, the odds were also not in favor of selecting the reverse-scored items, because only two of the items were reverse-scored. In any case, researchers who do not want to use any of the current short forms could consider scoring all of the original items except the reverse-worded items, as per previous recommendations involving the SIAS (Rodebaugh et a l, 2011; Rodebaugh, Woods, & Heimberg, 2007; note that the SPS contains no reverse-worded items). The relatively robust differ­ ences in item content suggest significant and substantial differ­ ences may exist between the measures; however, based on the theory behind the original scale construction, the general content should be shared across all of the short forms. The distinct goals for each short form (i.e., ensuring a robust factor structure, repli­ cating content, distinguishing social performance and social inter­ action anxiety symptoms, and maximizing readability) may well explain the differences in content and factor structure, but not all factor structures replicated equally in the current study. The three-factor structure for the SIPS originally proposed by Carleton et al. (2009) replicated well in the current undergraduate and clinical samples but was not statistically superior to a twofactor structure (i.e., SIPS-SPS and SIPS-SIAS). Both SIPS solu­ tions showed excellent fit in the current undergraduate sample and acceptable fit in the clinical sample. Similarly, the two unitary factors for the SPS-6 and SIAS-6 replicated excellently in the undergraduate sample and acceptably with the clinical sample data. The two-factor structure for the ASPS and ASIAS replicated acceptably in the undergraduate sample but not well in the clinical sample. The two unitary factors for the RSPS and RSIAS repli­ cated excellently in the undergraduate sample but produced unac­ ceptable fit indices in the clinical sample.

There were several statistically significant differences across the short forms in convergent validity with fears of negative and positive evaluation. The differences need to be considered care­ fully given that the absolute ranges were fairly small and indica­ tions of statistical significance were facilitated by the large sample size. That said, the SPS-6 and SIAS-6 demonstrated the lowest convergent validity with fears of negative evaluation, but the difference was relatively small. Interestingly, the ASPS and ASIAS demonstrated the highest convergent validity with fears of negative evaluation. Associations with fears of positive evaluation were more consistent across the short forms. Overall, convergent validity did not support any one short form, but that may be in part because only measures of fear of evaluation were used. The consistencies may also be the result of range restrictions (Tabachnick & Fidell, 2013). In any case, the extent to which a measure of social anxiety should correlate with a measure of fear of negative evaluation should be considered, since the latter is only part of the former and also related to other emotional states or disorders (Heimberg et al., 2010). There are several limitations to the current study that provide directions for future research. First, although the undergraduate sample was more than amply sized and the clinical sample meets or exceeds minimum size for assessing the four short forms (Costello & Osborne, 2005; Tabachnick & Fidell, 2013), different results may yet be obtained with larger clinical samples. Relatedly, most of the participants were Caucasian who spoke English as their first language. The current samples were generally compara­ ble with respect to language and ethnicity relative to samples used in producing each of the short forms, but different results may be obtained with more ethnically and linguistically diverse samples. Indeed, the factor analytic results associated with the SIPS, the SPS-6, the SIAS-6, the RSPS, and the RSIAS relative to the ASPS and the ASIAS may be indicative of important differences based on language or culture; however, this is highly speculative and

1124

CARLETON ET AL.

impossible to test using the current data. Future research should explore such possibilities using cross-cultural data with different translations of the short forms reviewed. Second, the distinct designs and intentions associated with the development of each short form—that is, diagnostic screening (Peters et al., 2012) versus maximizing readability (Fergus et al., 2012)— suggest that direct statistical comparisons of the four short forms may not account for differences in potential uses, particu­ larly given the relatively absent overlap among items. Indeed, some of the current short forms already have additional psycho­ metric support, such as seen for the SIPS (Duranceau et al., in press; Reilly et al., 2012), and a single short form variant may not be an appropriate or even desirable goal depending on the situation (e.g., in some cases, readability may need to trump other consid­ erations, but given the lower alpha despite the large number of items, such exceptions should be considered very carefully). Third, there is no definitive, statistically defensible way to compare non-nested models with different item counts. The mod­ els that did not meet the criteria associated with any of the fit indices (e.g., CFI, RMSEA) may well be defensibly discounted. Models that met the criteria for fit indices can be accepted as defensible but cannot be readily compared to each other. The ECVI does appear to be the best option among the available indices for non-nested models with different item counts (Hoyle, 2012); however, the ECVI current values are provided for com­ pleteness of the presentation of fit indices rather than as key comparator values across the non-nested models. Fourth, the original article on the SIPS recommended a three-factor structure (Carleton et al., 2009). In the current study, the third factor did not appear to substantively improve the fit of the model, suggest­ ing a two-factor structure may yet be preferred. Despite the statistical comparability of the three- and two-factor structures, there may be clinical, theoretical, and research utility in distinguishing fears of being observed and evaluated from fears of attracting attention. In­ deed, an argument could be made for a two-factor model based on the principle of parsimony; however, parsimony may belie important distinctions between types of performance situations that have been implied by the three-factor model (Duranceau et al., in press; Reilly et al., 2012). Future research should continue to assess the potential implications of formally distinguishing fears of being observed and evaluated from fears of attracting attention. Similarly, the authors of the ASPS and the ASIAS found no statistical improvements by adding a higher order factor (Kupper & Denollet, 2012). Researchers should further explore the robustness of the factor independence, possibly by examining independence by changing the item presenta­ tion order—a technique that has already been demonstrated as suc­ cessful for assessing the robustness of factorial independence (Carle­ ton, Thibodeau, Osborne, & Asmundson, 2012). The replication of the three-factor structure for the SIPS, in which the SPS items are separated into fears of overt evaluation and fears of attracting attention (Duranceau et al., in press; Reilly et al., 2012), may yet warrant further exploration to better facilitate our understanding of different kinds of performance situations and the related anxiety. Fifth, the relative robustness of the SPS-6, SIAS-6, ASPS, ASIAS, RSPS, and RSIAS needs to be further assessed and replicated using independent administrations of the item subsets. To date, only the SIPS item subsets have had such independent replications (Duranceau et al., in press; Reilly et al., 2012). Future research should also explore the transcultural properties of the various item subsets as well as their

capacity for psychometrically supported translations into other lan­ guages. Sixth, the convergent validity analyses were based solely on the BFNE and FPES scales, which were only available for the undergrad­ uate sample. Future research should explore convergent validity with other measures of social anxiety and in clinical samples. Finally, per the helpful feedback from one of the anonymous reviewers for the current article, the distinction between intemal/representational versus external/ elaborative validity considerations remains important (Foster & Cone, 1995). Evidence of superior factor structure, internal consistencies, inter-item correlations, and other issues internal to the measures in the absence of evidence of elaborative validity (e.g., predictive validity, diagnostic utility, treatment sensitivity) is insufficient within the con­ text of measures designed to be applied clinically. Future research should continue to explore predictive validity, diagnostic utility, and treatment sensitivity for each short form.

In summary, the current article provides a review and comparative assessment of the four current short forms of the SPS and SIAS; in particular, it provides comparative sensitivity and specificity analyses with cutoff scores that should benefit researchers and clinicians. The current comparative results suggest strengths and weaknesses associ­ ated with each of the current short forms for the SPS and SIAS. Overall, the SIPS (Carleton et al., 2009) remains the most replicated, supported, and investigated of the short forms to date (Duranceau et al., in press; Reilly et al., 2012; Weeks, Carleton, Asmundson, Mc­ Cabe, & Antony, 2010). The psychometrics for the SIPS (Carleton et al., 2009) and the SPS-6 and SIAS-6 (Peters et al., 2012) appear most robust in the current sample, for the convergent analyses and for the factor analyses, respectively; however, these psychometrics remain to be replicated with independent administrations of each item set. The current data provided relatively less support for the RSPS and RSIAS (Fergus et al., 2012) across all metrics; that said, it is critical to note that the intent of these short forms was markedly different and warrants special consideration for populations that may have reading difficulties. Finally, the ASPS and ASIAS were the least supported measures, based not only on the current psychometric results but also on the principle of parsimony (Kupper & Denollet, 2012). Research­ ers and clinicians should carefully consider their intentions in select­ ing among these short forms, being mindful that, across all of the short forms, the SIAS items appear to provide the best diagnostic differen­ tiation.

References Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52, 317-332. doi: 10.1007/BF02294359 American Psychiatric Association. (1994). Diagnostic and statistical man­ ual o f mental disorders (4tb ed., text rev.). Washington, DC: Author. American Psychiatric Association. (2013). Diagnostic and statistical man­ ual o f mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing. Arbuckle, J. L. (2007). Amos 18.0 [Computer software]. Chicago, IL: SPSS. Blanco, C., Nissenson, K., & Liebowitz, M. R. (2001). Social anxiety disorder: Recent findings in the areas of epidemiology, etiology, and treatment. Current Psychiatry Reports, 3, 273-280. doi:10.1007/sll920001-0019-9 Brown, E. J., Turovsky, J., Heimberg, R. G., Juster, H. R., Brown, T. A., & Barlow, D. H. (1997). Validation of the Social Interaction Anxiety

COMPARING SHORT FORMS OF THE SIAS AND SPS Scale and the Social Phobia Scale across the anxiety disorders. Psycho­ logical Assessment, 9, 21-27. doi: 10.1037/1040-3590.9.1.21 Browne, M. W., & Cudeck, R. (1989). Single sample cross-validation indices for covariance structures. Multivariate Behavioral Research, 24, 445-455. doi: 10.1207/sl5327906mbr2404_4 Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage. Byrne, B. (2001). Structural equation modeling with AMOS: Basic con­ cepts, applications, and programming. Mahwah, NJ: Erlbaum. Carleton, R. N„ Collimore, K. C., & Asmundson, G. J. G. (2007). Social anxiety and fear of negative evaluation: Construct validity of the BFNEII. Journal o f Anxiety Disorders, 21, 131-141. doi:10.1016/j.janxdis .2006.03.010 Carleton, R. N„ Collimore, K. C., Asmundson, G. J. G., McCabe, R. E., Rowa, K., & Antony, M. M. (2009). Refining and validating the Social Interaction Anxiety Scale and the Social Phobia Scale. Depression and Anxiety, 26, E71-E81. doi:10.1002/da.20480 Carleton, R. N., Collimore, K. C., McCabe, R. E., & Antony, M. M. (2011) . Addressing revisions to the Brief Fear of Negative Evaluation Scale: Measuring fear of negative evaluation across anxiety and mood disorders. Journal o f Anxiety Disorders, 25, 822-828. doi: 10.1016/j .janxdis.2011.04.002 Carleton, R. N., Thibodeau, M. A., Osborne, J. W., & Asmundson, G. J. G. (2012) . Exploring item order in anxiety-related constructs: Practical impacts of serial position. Practical Assessment, Research & Evaluation, 17, 1-15. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in scale development. Psychological Assessment, 7, 309-319. doi:10.1037/ 1040-3590.7.3.309 Costello, A. B., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1-9. Davison, A. C., & Hinkley, D. V. (2006). Bootstrap methods and their application. Cambridge, United Kingdom: Cambridge University Press. Duranceau, S., Peluso, D. L„ Collimore, K. C., Asmundson, G. J. G., & Carleton, R. N. (in press). Psychometric properties o f the French version of the Social Interaction Phobia Scale. Canadian Journal o f Behavioural Science. Edwards, P„ Roberts, I., Sandercock, P„ & Frost, C. (2004). Follow-up by mail in clinical trials: Does questionnaire length matter? Controlled Clinical Trials, 25, 31-52. doi:10.1016/j.cct.2003.08.013 Fergus, T. A., Valentiner, D. P., McGrath, P. B., Gier-Lonsway, S. L„ & Kim, H. S. (2012). Short forms of the Social Interaction Anxiety Scale and the Social Phobia Scale. Journal o f Personality Assessment, 94, 310-320. doi: 10.1080/00223891.2012.660291 Fergus, T. A„ Valentiner, D. P., McGrath, P. B„ Stephenson, K„ Gier, S„ & Jencius, S. (2009). The Fear of Positive Evaluation Scale: Psycho­ metric properties in a clinical sample. Journal o f Anxiety Disorders, 23, 1177-1183. doi: 10.1016/j.janxdis.2009.07.024 Foster, S. L„ & Cone, J. D. (1995). Validity issues in clinical assessment. Psychological Assessment, 7, 248-260. doi:10.1037/1040-3590.7.3.248 Furmark, T. (2002). Social phobia: Overview of community surveys. Acta Psychiatrica Scandinavica, 105, 84-93. doi:10.1034/j.1600-0447.2002 •lrl03.x Haughton, D. M. A., Oud, J. H. L., & Jansen, R. A. R. G. (1997). Information and other criteria in structural equation model selection. Communications in Statistics-Simulation and Computation, 26, 14771516. doi: 10.1080/03610919708813451 Heimberg, R. G., Brozovich, F. A., & Rapee, R. M. (2010). A cognitivebehavioral model of social anxiety disorder: Update and extension. In S. G. Hofmann & P. M. DiBartolo (Eds.), Social anxiety: Clinical, developmental, and social perspectives (pp. 395-422). New York, NY: Elsevier. doi:10.1016/B978-0-12-375096-9.00015-8

1125

Heimberg, R. G„ Mueller, G. P., Holt, C. S., Hope, D. A., & Liebowitz, M. R. (1992). Assessment of anxiety in social interaction and being observed by others: The Social Interaction Anxiety Scale and the Social Phobia Scale. Behavior Therapy, 23, 53-73. doi:10.1016/S00057894(05)80308-9 Hofmann, S. G., Heinrichs, N., & Moscovitch, D. A. (2004). The nature and expression of social phobia: Toward a new classification. Clinical Psychology Review, 24, 769-797. doi:10.1016/j.cpr.2004.07.004 Hoyle, R. (Ed.). (2012). Handbook o f structural equation modeling. New York, NY: Guilford Press. Hu, L., & Bender, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model mis-specification. Psychological Methods, 3, 424-453. doi:10.1037/1082-989X.3.4.424 Jepson, C„ Asch, D. A., Hershey, J. C., & Ubel, P. A. (2005). In a mailed physician survey, questionnaire length had a threshold effect on response rate. Journal o f Clinical Epidemiology, 58, 103-105. doi: 10.1016/j jclinepi.2004.06.004 Kessler, R. C., Berglund, P„ Demler, O., Jin, R„ Merikangas, K. R., & Walters, E. E. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Archives o f General Psychiatry, 62, 593-602. doi:10.1001/archpsyc.62 .6.593 Kessler, R. C., Chiu, W. T., Demler, O., Merikangas, K. R., & Walters, E. E. (2005). Prevalence, severity, and comorbidity of 12-month DSM-FV disorders in the National Comorbidity Survey Replication. Archives o f General Psychiatry, 62, 617-627. doi:10.1001/archpsyc.62 .6.617 Kumar, A., & Sharma, S. (1999). A metric measure for direct comparison of competing models in covariance structure analysis. Structural Equa­ tion Modeling, 6, 169-197. doi: 10.1080/10705519909540127 Kupper, N., & Denollet, J. (2012). Social anxiety in the general population: Introducing abbreviated versions of SIAS and SPS. Journal o f Affective Disorders, 136, 90-98. doi:10.1016/j.jad.2011.08.014 Leary, M. R. (1983). A brief version of the Fear of Negative Evaluation Scale. Personality and Social Psychology Bulletin, 9, 371-375. doi: 10.1177/0146167283093007 Lecrubier, Y., Sheehan, D„ Weiller, E„ Amorim, P„ Bonora, I., Sheehan, K., . . . Dunbar, G. (1997). The Mini International Neuropsychiatric Interview (M. I. N. I.), a short diagnostic structured interview: Reliabil­ ity and validity according to the CIDI. European Psychiatry, 12, 2 24231. doi: 10.1016/S0924-9338(97)83296-8 Lei, M., & Lomax, R. G. (2005). The effect of varying degrees of nonnormality in structural equation modeling. Structural Equation Mod­ eling, 12, 1-27. doi:10.1207/sl5328007seml201_l Liebowitz, M. R. (1987). Social phobia. Modern Problems o f Pharmaco­ psychiatry, 22, 141-173. Retrieved from http://psycnet.apa.org/index .cfm?fa=search.displayRecord&uid= 1988-23625-001 MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure mod­ eling. Psychological Methods, I, 130-149. doi:10.1037/1082-989X.1.2 .130 Mattick, R. P., & Clarke, J. C. (1989). Development and validation o f measures o f social phobia scrutiny fea r and social interaction anxiety. Unpublished manuscript. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455-470. doi:10.1016/S00057967(97)10031-6 McEvoy, P. M., Grove, R., & Slade, T. (2011). Epidemiology of anxiety disorders in the Australian general population: Findings of the 2007 Australian National Survey of Mental Health and Wellbeing. Australian and New Zealand Journal o f Psychiatry, 45, 957-967. doi:10.3109/ 00048674.2011.624083

1126

CARLETON ET AL.

McEvoy, P. M„ Nathan, P., Rapee, R. M„ & Campbell, B. N. (2012). Cognitive behavioural group therapy for social phobia: Evidence of transportability to community clinics. Behavior Research and Therapy, 50, 258-265. doi:10.1016/j.brat.2012.01.009 Nevitt, J., & Hancock, G. R. (2001). Performance of bootstrapping ap­ proaches to model test statistics and parameter standard error estimation in structural equation modeling. Structural Equation Modeling, 8, 353— 377. doi:10.1207/S15328007SEM0803_2 Orsillo, S. M. (2001). Measures for social phobia. In M. M. Antony, S. M. Orsillo & L. Roemer (Eds.), Practitioner’s guide to empirically based measures o f anxiety (pp. 165-187). New York, NY: Springer. Peters, L. (2000). Discriminant validity of the Social Phobia and Anxiety Inventory (SPAI), the Social Phobia Scale (SPS) and the Social Inter­ action Anxiety Scale (SIAS). Behavior Research and Therapy, 38, 943-950. doi: 10.1016/S0005-7967(99)00131 -X Peters, L., Sunderland, M., Andrews, G., Rapee, R. M., & Mattick, R. P. (2012). Development of a short form Social Interaction Anxiety (SIAS) and Social Phobia Scale (SPS) using nonparametric item response the­ ory: The SIAS-6 and the SPS-6. Psychological Assessment, 24, 66-76. doi: 10.1037/a0024544 Rapee, R. M„ Gaston, J. E„ & Abbott, M. I. (2009). Testing the efficacy of theoretically derived improvements in the treatment of social phobia. Journal of Consulting and Clinical Psychology, 77, 317-327. doi: 10.1037/a0014800 Reilly, A. R„ Carleton, R. N., & Weeks, J. W. (2012). Psychometric evaluation of the Social Interaction Phobia Scale. Anxiety, Stress, & Coping, 25, 529-542. doi:10.1080/10615806.2011.598150 Rigdon, E. E. (1999). Using the Friedman method of ranks for model comparison in structural equation modeling. Structural Equation Mod­ eling, 6, 219-232. doi: 10.1080/10705519909540131 Rodebaugh, T. L., Heimberg, R. G., Brown, P. J., Fernandez, K. C., Blanco, C., Schneier, F. R., & Liebowitz, M. R. (2011). More reasons to be straightforward: Findings and norms for two scales relevant to social anxiety. Journal of Anxiety Disorders, 25, 623-630. doi: 10.1016/j .janxdis.2011.02.002 Rodebaugh, T. L., Woods, C. M„ & Heimberg, R. G. (2007). The reverse of social anxiety is not always the opposite: The reverse-scored items of the Social Interaction Anxiety Scale do not belong. Behavior Therapy, 38, 192-206. doi: 10.1016/j.beth.2006.08.001 Rodebaugh, T. L„ Woods, C. M., Heimberg, R. G., Liebowitz, M. R„ & Schneier, F. R. (2006). The factor structure and screening utility of the Social Interaction Anxiety Scale. Psychological Assessment, 18, 231— 237. doi: 10.1037/1040-3590.18.2.231 Rodebaugh, T. L„ Woods, C. M., Thissen, D. M., Heimberg, R. G., Chambless, D. L„ & Rapee, R. M. (2004). More information from fewer questions: The factor structure and item properties of the original and Brief Fear of Negative Evaluation Scale. Psychological Assessment, 16, 169 -181. doi: 10.1037/1040-3590.16.2.169 Rosenthal, R. (1991). Meta-analytic procedures for social research (Rev. ed.). Newbury Park, CA: Sage. Safren, S. A., Heimberg, R. G., Brown, E. J., & Holle, C. (1996). Quality of life in social phobia. Depression and Anxiety, 4, 126-133. doi: 10.1002/(SICI)1520-6394(1996)4:33.0.CO;2-E

Safren, S. A., Turk, C. L., & Heimberg, R. G. (1998). Factor structure of the Social Interaction Anxiety Scale and the Social Phobia Scale. Be­ haviour Research and Therapy, 36, 443-453. doi:10.1016/S00057967(98)00032-1 Schneier, F. R., Heckelman, L. R., Garfmkel, R., Campeas, R., Fallon, B. A., Gitow, A., . . . Liebowitz, M. R. (1994). Functional impairment in social phobia. Journal o f Clinical Psychiatry, 55, 322-331. Sheehan, D. V., Lecrubier, Y., Hamett-Sheehan, K., Amorim, P., lanavs, J., Weiller, E., . . . Dunbar, G. (1998). The Mini-International Neuro­ psychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-TV and ICD-10. Journal o f Clinical Psychiatry, 59, 22-33. Sheehan, D. V., Lecrubier, Y., Harnett-Sheehan, K., lanavs, I., Weiller, E., Kesiner, A., . . . Dunbar, G. C. (1997). Reliability and validity of the Mini International Neuropsychiatric Interview (M.I.N.I.) according to the SCID-P. European Psychiatry, 12, 232-241. doi:10.1016/S09249338(97)83297-X Stein, M. B., McQuaid, I. R., Laffaye, C., & McCahill, M. E. (1999). Social phobia in the primary care medical setting. Journal of Family Practice, 48, 514-519. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA: Allyn & Bacon. Weeks, I. W., Carleton, R. N., Asmundson, G. J. G., McCabe, R. E., & Antony, M. M. (2010). “Social anxiety disorder carved at its joints”: Evidence for the taxonicity of social anxiety disorder. Journal of Anxiety Disorders, 24, 734-742. doi:10.1016/j.janxdis.2010.05.006 Weeks, I. W., Heimberg, R. G., Fresco, D. M., Hart, T. A., Turk, C. L., Schneier, F. R., & Liebowitz, M. R. (2005). Empirical validation and psychometric evaluation of the Brief Fear of Negative Evaluation Scale in patients with social anxiety disorder. Psychological Assessment, 17, 179-190. doi: 10.1037/1040-3590.17.2.179 Weeks, I. W„ Heimberg, R. G., & Rodebaugh, T. L. (2008). The Fear of Positive Evaluation Scale: Assessing a proposed cognitive component of social anxiety. Journal o f Anxiety Disorders, 22, 44-55. doi: 10.1016/j .janxdis.2007.08.002 Weeks, I. W„ Heimberg, R. G., Rodebaugh, T. L„ Goldin, P. R„ & Gross, J. J. (2012). Psychometric evaluation of the Fear of Positive Evaluation Scale in patients with social anxiety disorder. Psychological Assessment, 24, 301-312. doi: 10.1037/a0025723 Weeks, I. W., Heimberg, R. G., Rodebaugh, T. L., & Norton, P. J. (2008). Exploring the relationship between fear of positive evaluation and social anxiety. Journal o f Anxiety Disorders, 22, 386—400. doi: 10.1016/j .janxdis.2007.04.009 Wicherts, I. M., & Dolan, C. V. (2004). A cautionary note on the use of information fit indexes in covariance structure modeling with means. Structural Equation Modeling, 11, 45-50. doi:10.1207/S153 28007SEM1101_3

Received June 11, 2013 Revision received April 11, 2014 Accepted April 18, 2014 ■

Copyright of Psychological Assessment is the property of American Psychological Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.