Evaluation and Program Planning 43 (2014) 9–15

Contents lists available at ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

Gender and the relative importance of mental health satisfaction domains Eliza Robillos, Rachel Lale, Jennalee Wooldridge, Richard Heller, Andrew Sarkin * Health Services Research Center, Department of Family and Preventive Medicine, University of California, San Diego, 9500 Gilman Drive #0994, La Jolla, CA 92093-0994, USA

A R T I C L E I N F O

A B S T R A C T

Article history: Received 5 September 2012 Received in revised form 8 October 2013 Accepted 10 October 2013

Consumer-reported satisfaction data is a tool used for measuring and targeting areas for quality improvement in mental healthcare. In this study, we investigated the relationship between gender and the relative importance of mental health service satisfaction domains to overall satisfaction, in addition to gender differences in satisfaction across domains. People receiving mental health services (1765 males and 1950 females) completed questionnaires regarding their overall service satisfaction and satisfaction along six domains: Access to Services, Quality and Appropriateness, Participation in Treatment Planning, Outcome of Services, Social Connectedness, and Functioning. While all were important to overall satisfaction across genders, women reported slightly higher overall satisfaction. Linear regression analyses were used to determine the relative importance of these subscales to overall satisfaction for each gender. While the correlations between each subscale and overall satisfaction were significant for both, gender was found to moderate the relationship between some subscales and overall satisfaction. Although predictive of overall service satisfaction across the sample, we found Functioning, Outcome of Services, Social Connectedness, and Access to Services were relatively more important to overall satisfaction for men than women. Consistent feedback of results and improved access to services may be particularly effective for engaging both men and women in treatment. ß 2013 Elsevier Ltd. All rights reserved.

Keywords: Gender Mental health treatment Consumer satisfaction Community mental health

Consumer-reported satisfaction data is a tool used for measuring and targeting areas for quality improvement in mental healthcare (Bramesfeld, Wedega¨rtner, Elgeti, & Bisson, 2007; Edlund, Young, Kung, Sherbourne, & Wells, 2003; Powell, Holloway, Lee, & Sitzia, 2004; Ruggeri et al., 2007). Although the relationship is small, there is evidence that satisfaction with mental health services is related to successful treatment outcomes (Rey, O’Brien, & Walter, 2002; Turchik, Karpenko, Ogles, Demireva, & Probst, 2010). Patient reported satisfaction with their healthcare has been identified as an important factor related to treatment compliance among medical populations (Jin, Sklar, Min Sen Oh, & Chuen Li, 2008). Similarly, Paykel (1995) found that satisfaction was a strong predictor of treatment adherence among individuals with severe mental illness. Thus, knowledge about the determinants of mental health service satisfaction has the potential to benefit both people receiving services and mental healthcare providers. Moreover, patient satisfaction may be related to the underutilization of mental health services in some populations. Gender

* Corresponding author. Tel.: +1 858 622 1771; fax: +1 858 622 1790. E-mail addresses: [email protected] (E. Robillos), [email protected] (R. Lale), [email protected] (J. Wooldridge), [email protected] (R. Heller), [email protected] (A. Sarkin). 0149-7189/$ – see front matter ß 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.evalprogplan.2013.10.003

appears to be an indicator of mental health service utilization with men being less willing to utilize mental health services than women (Gonzalez, Alegrı´a, Prihoda, Copeland, & Zeber, 2011; Mackenzie, Gekoski, & Knox, 2006). This disparity may be attributed to differences in mental healthcare satisfaction between men and women. However, literature in this area has indicated inconsistent findings regarding gender differences in satisfaction with mental healthcare services. In a study examining satisfaction among adults receiving outpatient psychiatric services, Bjørngaard, Ruud, Garratt, and Hatling (2007) found that women reported having better experiences with services than men. However, among a sample of adolescents in treatment for mental health problems, Godley, Hedges, and Hunter (2011) found that boys reported significantly higher rates of treatment satisfaction than girls. Other studies have found that gender was unrelated to mental health service satisfaction (Blenkiron & Hammill, 2003; Edlund et al., 2003; Gani et al., 2011). Thus, questions remain about the relationship between gender and satisfaction. When mental healthcare systems use client-reported satisfaction data to target improvements, they might be tempted to limit their analysis to identifying which satisfaction domains received the lowest scores, and target those domains for program improvement projects. However, this method does not take into account how important each individual domain is to the overall

10

E. Robillos et al. / Evaluation and Program Planning 43 (2014) 9–15

satisfaction of people receiving mental health services. For example, a smaller improvement in quality for a domain that is very important to people receiving mental health services might result in a greater increase in overall satisfaction with services, when compared to a larger and perhaps more expensive improvement in an area that is not as important to people receiving mental health services (Sohn, 2011). One explanation for these mixed findings is that previous researchers have only examined gender differences in overall service satisfaction. Examining differences in satisfaction among multiple satisfaction domains, and how those domains relate to overall satisfaction within groups, has the potential to provide a richer understanding of the determinants of overall satisfaction with mental health services. For example, men are thought to be more mistrusting of therapists, ambivalent about therapy, and are likely to have lower expectations of therapy than women (Good & Roberts, 2010). These different expectations entering therapy may lead to differences in the perceived value of various treatment aspects. For example, entering treatment with lower expectations may lead men to value the process of therapy less than women, but perhaps to value the direct results of therapy more than women. If this were true, we would expect that satisfaction scale items relating to the direct results of treatment may be more important to overall satisfaction for men than women. Alternatively, we might predict that satisfaction scale items relating to the treatment process will be more important to overall satisfaction for women than men, if women are more receptive to those aspects of treatment. Given these considerations, the current study had two primary aims. First, we sought to examine which components of mental health services are most related to overall service satisfaction in a large community sample of people receiving mental health services in San Diego County. The second purpose of this study was to uncover gender similarities and differences in mental health service satisfaction. Specifically, we sought to compare overall service satisfaction among men and women and, importantly, to determine which components of service are most strongly related to overall service satisfaction in men compared to women.

response options ranged on a 5-point Likert scale from 1 (Strongly disagree) to 5 (Strongly agree). From the 36-items, the following seven domains are assessed: Overall Service Satisfaction, Perception of Access to Services, Perception of Quality and Appropriateness of Care, Perception of Participation in Treatment Planning, Perception of Outcome of Services, Perception of Social Connectedness, and Perception of Functioning as a Result of Services (Table 1). As of 2001, thirty-eight states had implemented a version of the MHSIP consumer survey to assess perception of care from those receiving mental health services (NASMHPD Research Inc., 2002). Reliability of the MHSIP was high in a pilot study (Cronbach’s alpha = .95; Minsky & Lloyd, 1996) and was high in the current sample (Cronbach’s alpha = .97).

1. Methods

Table 1 MHSIP subscale domains and descriptions adapted from Sohn (2011).

1.1. Participants Participants were 3715 male (n = 1765) and female (n = 1950) adults actively receiving psychological and behavioral treatment services at one of 128 mental health provider programs in San Diego County. Data were obtained from a large scale assessment of self-reported satisfaction that occurred twice per year (May and November) where all adults receiving mental health treatment services during a two week data collection period complete an anonymous self-report questionnaire at their provider site. For the current study, we used data collected in May 2009, November 2009, May 2010, November 2010, and May 2011. If a person receiving services completed surveys at multiple times, then the most recent survey was used for that participant.

1.3. Procedure Program staff collected MHSIP satisfaction survey data from people accessing outpatient or case management services during a two-week period in November and May, 2009, 2010, and 2011. People receiving mental health services who were unable to complete the questionnaire due to mental or physical illness were assisted by non-clinical staff members. Completed questionnaires were placed in a sealed envelope and returned for processing to University of California San Diego (UCSD) Health Services Research Center (HSRC). Participant responses to the MHSIP from May 2009 to May 2011 were merged with demographic data. In the case of duplicate surveys, only the most recent survey was retained. The study was approved by the University of California, San Diego, Institutional Review Board and the San Diego County Mental Health Services Research Committee. 1.4. Statistical analyses Data were analyzed using Statistical Package for the Social Sciences (SPSS), Version 19.0 (IBM Corporation, 2010). T-tests were used to determine if mean satisfaction levels differed between males and females. Cohen’s d values were calculated to determine effect size for the differences. Pearson correlations were used to describe the relationship between each MHSIP subscale and Overall

MSHIP subscale

General description

Overall Service Satisfaction

Satisfaction with services in general (i.e., Services were satisfactory, preferable to other choices, and would be recommended to others) Staff availability, the range of service options and how quickly and conveniently services were received Perceived supportiveness of staff, whether services promoted recovery and continuity of care People receiving mental health services’ participation in planning services Services directly provided people receiving mental health services with positive outcomes, particularly changes in areas for which treatment is often sought (e.g., social situations, school/ work, housing, symptom management) Services directly contributed to improved social support and social relationships (i.e., from family and friends) and a sense of belonging in the community Services directly resulted in an increase in meaningful activities, taking care of personal needs, and coping strategies

Perception of Access to Services

Perception of Quality and Appropriateness of Care Perception of Participation in Treatment Planning Perception of Outcome of Services

1.2. Measures 1.2.1. Mental Health Statistics Improvement Program (MHSIP) consumer survey The version of the MHSIP Consumer Survey used in this study consists of 36 items designed to assess the care of persons with mental illness and is widely used in public mental health systems (Center for Mental Health Services, 1996). Each item is a declarative statement (e.g., ‘‘I like the services I receive here’’) and

Perception of Social Connectedness

Perception of Functioning as a Result of Services

E. Robillos et al. / Evaluation and Program Planning 43 (2014) 9–15 Table 2 Sample demographic information.

Age % White % Hispanic % African American % Schizophrenia % Major depression % Bipolar % Outpatient % Full Service Partnership % Case management

All n = 3715

Male n = 1765

Female n = 1950

43.15 (13.54) 52.9 21.1 12.7 36.2 27.7 21.3 72.6 21.4 5.0

41.34 (13.56) 52.4 21.5 13.8 46.1 20.6 17.7 67.8 26.1 4.9

44.79 (13.28)** 53.4 20.7 11.8 27.1** 34.2** 24.5** 77.0** 17.1** 5.0

Note: Demographic information was compiled at individual program sites by program staff. Values in parentheses are standard deviations. *p < 0.05. ** p < 0.001.

Service Satisfaction for each gender and for the whole sample. The moderating effect of gender on these correlations was examined using linear regression analyses to explain the relative importance of the satisfaction domains to Overall Service Satisfaction for males and females. Gender was contrast-coded (males = 1; females = +1) and all continuous variables were deviated to the sample mean to allow for meaningful interpretations of resulting parameter slopes. Six moderation models were tested using multiple linear regressions. In each model, Overall Service Satisfaction was regressed on Gender, a domain of the MHSIP (e.g., Perception of Access to Services), and the interaction of Gender and the domain (e.g., Gender  Perception of Access to Services). Partial eta-squared values were used to determine effect size for the interactions and correlations between continuous variables. 2. Results We obtained demographic and diagnosis information (Table 2) from the San Diego County Adult and Older Adult Medical Information System, a patient record holding database. All participants were older than 18 years of age with an average age of 43.15 (SD = 13.54). Males (M = 41.34 years) were significantly younger than females (M = 44.79), [t (3713) = 7.84, p < 0.001]. The three most common races or ethnicities in the sample were White (53%), Hispanic (21%), and African American (13%). No gender differences were found in percentage of White [x2 = 0.36, p = 0.55], Hispanic [x2 = 0.37, p = 0.55], or African American [x2 = 3.25, p = 0.07] participants. The most common diagnoses were schizophrenia and other schizoaffective disorders (36.2%), major depression (27.7%), and bipolar disorder (21.3%). Additionally, we found that males were significantly more likely to have received a diagnosis of schizophrenia or other schizoaffective

11

disorders (46.1%) than females (27.1%), [x2 = 146, p < 0.001]. Conversely, females were more likely to have received a diagnosis of major depression (34.2%) than males (20.6%), [x2 = 84.46, p < 0.001] and were also more likely to have received a diagnosis of bipolar disorder (24.5%) than males (17.7%), [x2 = 25.6, p < 0.001]. Males were more likely to receive more intensive treatment at Full Service Partnership programs (26.1%) than females (17.1%), [x2 = 44.99, p < 0.001], while females were more likely to have been receiving less intensive services from outpatient treatment (77.0%) than males (67.8%), [x2 = 39.98, p < 0.001]. 2.1. Mean overall service satisfaction and MHSIP domains by gender We examined gender differences in Overall Service Satisfaction and scale domains (Table 3). Our analysis revealed that both females (M = 4.48) and males showed high levels of satisfaction with overall services (M = 4.38), [F(1, 3713) = 18.92, p < 0.001, d = .14]. Additionally, females reported slightly higher scores for Perception of Access to Services [F(1, 3713) = 12.54, p < 0.001, d = .12], Perception of Quality and Appropriateness of Care [F(1, 3713) = 29.76, p < 0.001, d = .18], Perception of Participation in Treatment Planning [F(1, 3713) = 32.51, p < 0.001, d = .19], and Perception of Social Connectedness [F(1, 3713) = 7.67, p < 0.001, d = .09] domains. No gender differences were found in the domains Perception of Outcome of Services [F(1, 3713) = 0.03, p = 0.85, d = .01] or Perception of Functioning as a Result of Services [F(1, 3713) = 0.23, p = 0.64, d = .02]. Effect sizes for significant gender differences ranged from .09 to .19, approaching the conventional cut-off for small practical significance (.2; Cohen, 1992). 2.2. Bivariate correlations between overall service satisfaction and MHSIP domains We examined bivariate correlations between Overall Service Satisfaction scores and MHSIP subscale domains. These correlations were first examined for the entire sample and then individually for females and males (Table 4). Across participants, subscale scores were positively related to Overall Service Satisfaction. Perception of Access to Services was most strongly correlated with Overall Service Satisfaction [r = .740, p < 0.001], followed by Perception of Quality and Appropriateness of Care [r = .714, p < 0.001], Perception of Participation in Treatment Planning [r = .602, p < 0.001], Perception of Outcome of Services [r = .417, p < 0.001], Perception of Social Connectedness [r = .348, p < 0.001], and Perception of Functioning as a Result of Services [r = .336, p < 0.001]. 2.3. Moderation models We used linear regression analyses to examine how gender moderated the relationship between the domain ratings and

Table 3 Mental Health Statistics Improvement Program (MHSIP) consumer survey mean subscale scores and correlations with Overall Service Satisfaction by gender. MHSIP subscale

Mean satisfaction ratings Male

Female

Male

Female

Overall Service Satisfaction Perception of Access to Services Perception of Quality and Appropriateness of Care Perception of Participation in Treatment Planning Perception of Outcome of Services Perception of Social Connectedness Perception of Functioning as a Result of Services

4.38** 4.21** 4.23 4.19** 3.86 3.82** 3.85

4.48** 4.29** 4.34 4.32** 3.86 3.89** 3.84

N/A .75* .73 .59 .48** .38** .40**

N/A .73* .70 .61 .36** .32** .28**

Correlations with overall satisfaction

Note: Possible range scores of mean MHSIP domains 1.00–5.00; actual range of scores 1.00–5.00. Differences in correlation strength are indicated based on interaction of Gender  domain in moderation analyses. * p < 0.01. ** p < 0.001.

12

E. Robillos et al. / Evaluation and Program Planning 43 (2014) 9–15

Table 4 Multiple linear regression analyses models for Overall Service Satisfaction.

b *

t

h2p

Gender Perception of Access to Services Perception of Access to Services  Gender (R2 = .55)

.02 .73*** .03**

2.56 66.87 2.65

.002 .55 .002

Gender Perception of Quality and Appropriateness of Care Perception of Quality and Appropriateness of Care  Gender (R2 = .51)

.01 .76*** .01

.66 61.86 .66

.00 .51 .00

Gender Perception of Participation in Treatment Planning Perception of Participation in Treatment Planning  Gender (R2 = .36)

.01 .57*** 1.11

1.17 45.61 .00

.00 .36

Gender Perception of Outcome of Services Perception of Outcome of Services  Gender (R2 = .18)

.05*** .36*** .06***

4.71 28.25 4.45

.01 .18 .01

Gender Perception of Social Connectedness Perception of Social Connectedness  Gender (R2 = .13)

.04*** .27*** .03***

3.6 22.61 2.44

.003 .12 .002

Gender Perception of Functioning as a Result of Services Perception of Functioning as a Result of Services  Gender (R2 = .12)

.05*** .26*** .06***

4.81 22.19 4.74

.01 .12 .01

Note: Overall treatment satisfaction and treatment domains were measured using the MHSIP. The criterion variable was Overall Service Satisfaction for each model tested above. Gender coded as follows: females (+1), males (1). All satisfaction domain scores were centered to sample mean. * p < 0.05. ** p < 0.005 *** p < 0.001.

Overall Service Satisfaction. We tested six moderation models (Table 4). In each model, Overall Service Satisfaction was regressed on Gender, a domain of the MHSIP (e.g., Perception of Access to Services), and the interaction of Gender and the domain (e.g., Gender  Perception of Access to Services). 2.3.1. Perception of access to services Overall Service Satisfaction was regressed on Access to Services, gender, and their interaction. There was a significant positive relationship between Perception of Access to Services and Overall Service Satisfaction [F(1, 3711) = 4471.17, p < 0.001, h2p ¼ :55], on average across gender. This low, but positive relationship was stronger for men than women [F(1, 3711) = 7.01, p < 0.01]. Gender explained .02% of variance in the relationship between Perception of Access to Services and Overall Service Satisfaction, h2p ¼ :002. 2.3.2. Perception of Quality and Appropriateness of Care Overall Service Satisfaction was regressed on Perception of Quality and Appropriateness of Care, gender, and their interaction. There was a positive relationship between Perception of Quality and Appropriateness of Care and Overall Service Satisfaction [F(1, 3711) = 3826.69, p < 0.001, h2p ¼ :51] averaged across genders. Gender did not moderate the relationship between this subscale and Overall Service Satisfaction [F(1, 3711) = 0.43, p = 0.51, h2p ¼ :001]. 2.3.3. Perception of Participation in Treatment Planning Overall Service Satisfaction was regressed on Perception of Participation in Treatment Planning, gender, and their interaction. There was a positive relationship between Perception of Participation in Treatment Planning and Overall Service Satisfaction [F(1, 3711) = 2080.48, p < 0.001, h2p ¼ :36] on average across gender. Gender did not moderate the relationship between this subscale and Overall Service Satisfaction [F(1, 3711) = 1.23, p = 0.27, h2p ¼ :001]. 2.3.4. Perception of Outcome of Services Overall Service Satisfaction was regressed on Perception of Outcome of Services, gender, and their interaction. There was a significant positive relationship between Perception of Outcome of

Services and Overall Service Satisfaction [F(1, 3711) = 798.34, p < 0.001, h2p ¼ :18], on average across gender. This positive relationship was significantly stronger for men than women [F(1, 3711) = 19.76, p < 0.001]. Gender explained .1% of variance in the relationship between Perception of Outcome of Services and Overall Service Satisfaction, h2p ¼ :01. 2.3.5. Perception of Social Connectedness Overall Service Satisfaction was regressed on Perception of Social Connectedness, gender, and their interaction. There was a significant positive relationship between Perception of Social Connectedness and Overall Service Satisfaction [F(1, 3711) = 511.18, p < 0.001, h2p ¼ :12], on average across gender. This positive relationship was significantly stronger for men than women [F(1, 3711) = 5.96, p < 0.05]. Gender explained .02% of variance in the relationship between Perception of Social Connectedness and Overall Service Satisfaction, h2p ¼ :002. 2.3.6. Perception of Functioning as a Result of Services Overall Service Satisfaction was regressed on Perception of Functioning as a Result of Services, gender, and their interaction. There was a significant positive relationship between Perception of Functioning as a Result of Services and Overall Service Satisfaction [F(1, 3711) = 492.46, p < 0.001, h2p ¼ :12], on average across gender. This positive relationship was significantly stronger for men than women [F(1, 3711) = 22.50, p < 0.001]. Gender explained .1% of the variance in the relationship between Perception of Functioning as a Result of Services and Overall Service Satisfaction, h2p ¼ :01. 3. Discussion The aims of this study were to determine which domains of mental health services satisfaction most influenced overall satisfaction with mental health services and to investigate how satisfaction differed in these domains between men and women. Our results were consistent with previous findings that women tend to report greater satisfaction with mental health services than men (Bjørngaard et al., 2007; Sohn, 2011). In addition to higher

E. Robillos et al. / Evaluation and Program Planning 43 (2014) 9–15

Overall Service Satisfaction, women in the current study had higher scores in Perception of Access to Services, Perception of Quality and Appropriateness of Care, Perception of Participation in Treatment Planning, and Perception of Social Connectedness domains of the MHSIP. Although the effect sizes would be interpreted as very small for these gender differences, with Cohen’s d ranging from .09 to .19, several of the differences might still be considered practically significant in understanding how to increase service satisfaction, which may lead to better retention and recovery. Given that there is more than 85% overlap of the male and female distributions for even the largest mean differences, the genders are certainly more alike than different when it comes to rating satisfaction with mental health services. There were important similarities between men and women on two of the satisfaction scales, indicating that findings about greater satisfaction for women are not universal across domains. Females were not more satisfied in the domains of Perception of Outcomes of Services or Perception of Functioning as a Result of Services, both of which are more associated with the perceived effects of services on the respondent, as opposed to how much they liked the services. It may be that men benefit as much from services as women, but do not have as positive an experience receiving those services. This may be because females give more socially desirable higher ratings, perceive services more positively in some domains, are better at utilizing the healthcare system, or are less stigmatized (Lale, Sarkin, & Sklar, in press). An alternate explanation is that clinics serving predominately or only women may actually provide better services. Because some clinics serve only women, and sampling small numbers across so many clinics results in some clinics having only one gender represented though they serve both genders, a nested design cannot be used in this study, and that may be a significant limitation of interpreting the results. However, most clients receive services at multiple mental health sites within the integrated system and their satisfaction responses are thought to often reflect the services they get in general across multiple sites. The results are intended to be interpreted and acted upon at the system level and not the clinic level, so it could be argued that nesting within the site might mask some real differences that need to be addressed at the system level. If women are more satisfied because they attend better treatment sites, we would not want to ‘‘correct for’’ this real result and conclude that women and men are equal when we correct for site. In this case site is parallel to which office of the medical center they get seen by, rather than which medical center sees them, and we are interested in differences across the whole system. We found significant positive correlations between the MHSIP domains and Overall Service Satisfaction across gender. On average, Overall Service Satisfaction was strongly predicted by Perception of Access to Services, Perception of Quality and Appropriateness of Care, Perception of Participation in Treatment Planning, Perception of Outcome of Services, Perception of Social Connectedness, and Perception of Functioning as a Result of Services domains. Our results overlap with those of Sohn (2011) who found that Perception of Access to Services, Perception of Quality and Appropriateness of Care, and Perception of Participation in Treatment Planning were predictive of Overall Service Satisfaction among people receiving services in community mental health centers in Kentucky. Our findings indicate that because both men and women were satisfied with overall services, focusing improvement efforts in satisfaction based on the importance of these domains may maximize overall client satisfaction with services. Although we expected all MHSIP domains to be indicative of general satisfaction, we were particularly interested in which of these factors were more important to overall satisfaction for men versus women. Furthermore, our analyses revealed that Perception

13

of Access to Services, Perception of Outcome of Services, Perception of Social Connectedness, and Perception of Functioning as a Result of Services were significantly more indicative of Overall Service Satisfaction for men than women. Statistical significance may be attributed to the large sample size of the study, yet these results provide valuable insight as to which aspects of service satisfaction hold particular weight in determining overall satisfaction for men and women. Both genders exhibit a relationship between treatment results and overall satisfaction. Our analyses exposed that there was a significant positive relationship between Overall Service Satisfaction and Perception of Outcome of Services, Perception of Functioning as a Result of Services, and Perception of Social Connectedness for both men and women. Though correlations with overall satisfaction were lowest in these domains compared to Perception of Access to Services, Perception of Quality and Appropriateness, and Perception of Participation in Treatment Planning, these outcome domains were relatively more important for men and are consistent with our prediction that satisfaction would be more strongly central to domains relating to the direct results of services. In fact, all of the individual items within these three subscales explicitly refer to improvements as a direct result of services (e.g., ‘‘As a direct result of the services I have received, I deal more effectively with my daily problems’’). However, it is important to interpret specific subdomains of satisfaction, and this should be done in light of how important they are to the people receiving services, rather than focusing only on the overall satisfaction score. This strategy provides the best guidance for program planning activities to increase satisfaction. Results indicated that some domains of satisfaction that may be particularly relevant to satisfaction among men, a population which has been found to underutilize mental health services (Mackenzie et al., 2006; Gonzalez et al., 2011), such as Perception of Social Connectedness being more related to Overall Service Satisfaction in men than women. Social connectedness may be important for men as they generally report lower levels of social support than women outside of treatment (Taylor et al., 2000; Tamres, Janicki, & Helgeson, 2002; Swickert & Hittner, 2009), though another study in a similar large mental health system found men’s social health to be as high as women (Carlson et al., 2011). Social connectedness as a result of mental health services may be more important for men than previously thought and should be considered in the delivery of services to men. Although the most central domain of Overall Service Satisfaction across our sample, the finding that Perception of Access to Services was more indicative of overall satisfaction for men than women, in particular, is also consistent with our original predictions. Men are less likely to utilize services to begin with and therefore may be more easily deterred by logistical barriers than women. This finding suggests that reducing logistical barriers to services, especially for men, may help people receiving mental health services remain engaged in the treatment process. With the exception of Perception of Access to Services, the domains for which there were differences between men and women were domains that were less related to overall satisfaction for both genders, and these other domains reflected the functioning improvements brought about by treatment, rather than process aspects of the treatment itself. If functioning improvements are more important to males, then efforts to demonstrate these improvements in tangible ways to males might have more positive impact than it would for females on satisfaction. However, these data indicate that functioning changes are not the strongest drivers of overall satisfaction for either gender. The differences were small between men and women on the domains that discriminated between genders, and some satisfaction domains were equally important for men and women. This indicates that men and women may weigh the importance of many

14

E. Robillos et al. / Evaluation and Program Planning 43 (2014) 9–15

aspects of mental health care similarly. Specifically, Perception of Quality and Appropriateness of Care and Perception of Participation in Treatment Planning were highly predictive of Overall Service Satisfaction, and this relationship was not different between men and women. These domains appear to be essential components of mental health service satisfaction for all people receiving services. This finding is in line with previous research indicating that the acceptability of antipsychotic medication among patients with schizophrenia is an important indicator of treatment satisfaction (Naber & Kasper, 2000). This finding is also consistent with those of Bhugra, La Grenade, and Dazzan (2000) who suggest that among psychiatric inpatients, all people receiving mental health services want to be actively involved in their treatment plan. While this study has important clinical implications, there are limitations to interpreting the results. First, data were obtained via self-report and are thus subject to response biases (Furnham, 1986). Additionally, the statistically significant gender differences may be attributed to the large sample size, therefore interpretations pertaining to the clinical significance of these differences must be made with caution. We must exercise caution in interpreting the underlying causes of the relationships revealed in these analyses as such interpretations are outside the scope of the current study. These data are correlational in nature. Therefore, while the relationships between overall satisfaction and individual aspects of treatment were strong and positive, we cannot infer causal direction and we must exercise caution in our interpretations. Moreover, we did not seek to investigate potential confounding variables that might be related to gender that might influence what factors are more important to overall service satisfaction, such as diagnosis and treatment setting. For example, we found that men in our sample were more likely to be diagnosed with schizophrenia whereas women were more likely to be diagnosed with major depression and bipolar disorder. Relative to other mental health disorders, research suggests that schizophrenia may be particularly socially isolating due to heightened social stigma (King et al., 2007). Therefore, it may be the case that the perception of social connectedness is particularly predictive of overall service satisfaction in people receiving mental services with schizophrenia—who were also more likely to be men in this sample. As with diagnosis, specific clinics or providers are not distributed evenly across gender. Some mental health providers having a preponderance of one gender through the natural recruitment process and others having specific missions to serve only one gender or the other. If we hold clinic constant by using it as a nesting variable, then we might be correcting away one of the principle mechanisms by which gender has its effect, the assignment of treatment settings that happens through clinician referral preferences, availability of programs, and funding limitations of the person receiving services. One possible source of satisfaction differences is true differences in service quality that are dependent upon where a person receives services in the integrated system, where people are likely to receive services from multiple providers. Although satisfaction measures can provide meaningful information regarding the quality of mental health services, these instruments do have noted weaknesses. Previous literature has indicated that these measures are not highly correlated with mental health outcomes such as psychiatric symptom reduction (Carlson & Gabriel, 2001; Turchik et al., 2010) and functional improvement (Garland, Haine, & Boxmeyer, 2007; Lunnen & Ogles, 1998; Turchik et al., 2010). Additionally, these measures are typically self-report questionnaires administered after treatment has been completed. This methodology is problematic because satisfaction measures that are administered after patients are finished with treatment may not reflect satisfaction during treatment because of biases and retrospective recall.

Despite its limitations, this study reveals that there may be very small but significant gender differences in which satisfaction domains are more indicative of overall mental health service satisfaction. Although the effect sizes were extremely small for these interactions, they suggest that further investigation may be warranted using these methods to understand differences in treatment satisfaction between groups. These relationships may be particularly informative to service providers interested in costeffective ways of improving satisfaction of people receiving mental health services, which may in turn improve treatment compliance (Paykel, 1995) and aid recovery (Rey et al., 2002; Turchik et al., 2010). Moreover, this study reveals several factors that may be slightly more important to service satisfaction among men—a demographic found to be particularly resistant to seeking mental health services (Gonzalez et al., 2011; Mackenzie et al., 2006). There is also evidence that men are harder to engage in therapy than women and are less likely to remain in therapy than women (Cottone, Drucker, & Javier, 2002). Therefore, treatment satisfaction may be improved (particularly in men) when people receiving mental health services are encouraged to see the positive ‘results’ of treatment. Indeed, service providers who provide explicit feedback regarding client progress during therapy via clinical outcomes tools (e.g., Illness Management and Recovery, the Recovery Markers Questionnaire) may see an improvement in people receiving mental health services’ treatment satisfaction, compliance, and recovery. 4. Conclusions Satisfaction measures have noted weaknesses, but are important tools in evaluating treatment programs. In this study, we investigated which domains of the MHSIP were most strongly associated with overall satisfaction for people receiving mental health services in San Diego County. As a whole, they were satisfied with the mental health services they received. This information gives healthcare providers, planners, and researchers insight on how to maintain quality mental health services, which may lead to improved client treatment engagement and/or adherence. For example, Perception of Outcome of Services was an important satisfaction domain for both men and women. Therefore, explicit feedback regarding individual progress paired with satisfaction data may aid in treatment progress and recovery for people receiving mental health services. In addition, efforts by clinicians to demonstrate these functional improvements might improve satisfaction for people receiving mental health services. As this study shows, satisfaction measures such as the MHSIP are effective in illustrating which domains are most important to people receiving mental health services. Future studies should investigate changes in mental health outcome tools in conjunction with satisfaction measures to investigate whether or not a causal relationship exists. However, these data indicate that the largest impacts on satisfaction might not come from functioning changes due to treatment for either gender, though changes in functioning may be slightly more important to overall satisfaction for men. Acknowledgements The authors acknowledge Todd Gilmer, Steven Tally, Marisa Sklar, Alma Correa, Kimberly Center, and Dara McIntyre for contributions to editing the manuscript and San Diego County Behavioral Health Services for funding the study. References Bhugra, D., La Grenade, J., & Dazzan, P. (2000). Psychiatric inpatients’ satisfaction with services: A pilot study. International Journal of Psychiatry in Clinical Practice, 4(4), 327–332 http://dx.doi.org/10.1080/13651500050517902.

E. Robillos et al. / Evaluation and Program Planning 43 (2014) 9–15 Bjørngaard, J. H., Ruud, T., Garratt, A., & Hatling, T. (2007). Patients’ experiences and clinicians’ ratings of the quality outpatient teams in psychiatric care units in Norway. Psychiatric Services, 58(8), 1102–1107 http://dx.doi.org/10.1176/ appi.ps.58.8.1102. Blenkiron, P., & Hammill, C. A. (2003). What determines patients’ satisfaction with their mental health care and quality of life? Postgraduate Medical Journal, 79(932), 337– 340 http://dx.doi.org/10.1136/pmj.79.932.337. Bramesfeld, A., Wedega¨rtner, F., Elgeti, H., & Bisson, S. (2007). How does mental health care perform in respect to service user’s expectation? Evaluating inpatient and outpatient care in Germany with the WHO responsiveness concept. BMC Health Services Research, 7(99), 1–12 http://dx.doi.org/10.1186/1472-6963/7/99. Carlson, J., Sarkin, A., Levack, A., Sklar, M., Tally, S., Gilmer, T., & Groessl, E. (2011). Evaluating a measure of social health derived from two mental health recovery measures: The California Quality of Life (CA-QOL) and Mental Health Statistics Improvement Program Consumer Survey (MHSIP). Community Mental Health Journal, 47(4), 454–462 http://dx.doi.org/10.1007/s10597-010-9347-8. Carlson, M., & Gabriel, R. (2001). Patient satisfaction, use of services, and one-year outcomes in publicly funded substance abuse treatment. Psychiatric Services, 52(9), 1230–1236 http://dx.doi.org/10.1176/appi.ps.52.9.1230. Center for Mental Health Services. (1996). The MHSIP consumer-oriented mental health report card: The final report of the Mental Health Statistics Improvement Program (MHSIP). Rockville, MD: US Dept of Health and Human Services Substance Abuse and Mental Health Services Administration National Mental Health Information Center. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159 http:// dx.doi.org/10.1037/0033-2909.112.1.155. Cottone, J. G., Drucker, P., & Javier, R. A. (2002). Gender differences in psychotherapy dyads: Changes in psychological symptoms and responsiveness to treatment during 3 months of therapy. Psychotherapy: Theory, Research, Practice, Training, 39(4), 297–308 http://dx.doi.org/10.1037/0033-3204.39.4.297. Edlund, M. J., Young, A. S., Kung, F. Y., Sherbourne, C. D., & Wells, K. B. (2003). Does satisfaction reflect the technical quality of mental health care? Health Services Research, 38(2), 631–645 http://dx.doi.org/10.1111/1475-6773.00137. Furnham, A. (1986). Response bias, social desirability and dissimulation. Personality and Individual Differences, 7(3), 385–400 http://dx.doi.org/10.1016/01918869(86)90014-0. Gani, N., Saeed, K., Minhas, F. A., Anjuman, N., Waleed, M., & Fatima, G. (2011). Assessment of patient satisfaction with mental health services in a tertiary care setting. Journal of Ayub Medical College Abbottabad, 23(1), 43–46. Garland, A., Haine, R., & Boxmeyer, C. (2007). Determinates of youth and parent satisfaction in usual care psychotherapy. Evaluation and Program Planning, 30(1), 45–54 http://dx.doi.org/10.1016/j.evalprogplan.2006.10.003. Godley, S. H., Hedges, K., & Hunter, B. (2011). Gender and racial differences in treatment process and outcome among participants in the adolescent community reinforcement approach. Psychology of Addictive Behaviors: Journal of the Society of Psychologists in Addictive Behaviors, 25(1), 143–154 http://dx.doi.org/10.1037/a0022179. Gonzalez, J. M., Alegrı´a, M., Prihoda, T. J., Copeland, L. A., & Zeber, J. E. (2011). How the relationship of attitudes toward mental health treatment and service use differs by age, gender, ethnicity/race and education. Social Psychiatry & Psychiatric Epidemiology, 46(1), 45–57 http://dx.doi.org/10.1007/s00127-009-0168-4. Good, G. E., & Roberts, J. M. (2010). To accept a pilot? Addressing men’s ambivalence and altering their expectancies about therapy. Psychotherapy Theory, Research, Practice, Training, 47(3), 306–315 http://dx.doi.org/10.1037/a0021162. IBM Corporation. (2010). IBM SPSS statistics for windows (Version 19.0) [Statistical package software]. Armonk, NY: IBM Corp. Jin, J., Sklar, G. E., Min Sen Oh, V., & Chuen Li, S. (2008). Factors affecting therapeutic compliance: A review from the patient’s perspective. Therapeutics and Clinical Risk Management, 4(1), 269–286. King, M., Dinos, S., Shaw, J., Watson, R., Stevens, S., Passetti, F., Weich, S., & Serfaty, M. (2007). The Stigma Scale: development of a standardised measure of the stigma of mental illness. The British Journal of Psychiatry, 190, 248–254 http://dx.doi.org/ 10.1192/bjp.bp.106.024638. Lale R., Sarkin A., Sklar M. Gender moderates social distancing toward stigma toward individuals with alcohol dependence versus depression. Stigma Research and Action 2013, (in press). Lunnen, K., & Ogles, B. (1998). A multiperspective, multivariable evaluation of reliable change. Journal of Consulting and Clinical Psychology, 66(2), 400–410 http:// dx.doi.org/10.1037/0022-006X.66.2.400. Mackenzie, C. S., Gekoski, W. L., & Knox, V. J. (2006). Age, gender, and the underutilization of mental health services: The influence of help-seeking attitudes. Aging & Mental Health, 10(6), 574–582 http://dx.doi.org/10.1080/13607860600641200. Minsky, S., & Lloyd, P. (1996). The use of unique client identifiers in state mental health service systems. Administration and Policy in Mental Health Services Research, 23(3), 231–252 http://dx.doi.org/10.1007/BF02108322. Naber, D., & Kasper, S. (2000). The importance of treatment acceptability to patients. International Journal of Psychiatry in Clinical Practice, 4(Suppl. 1), S25–S34 http:// dx.doi.org/10.1080/13651500050518497.

15

NASMHPD Research Institute Inc. (2002). State profile highlights: Performance measures system: Consumer survey 2000–2001 (June 2002). Retrieved from http://www.nriinc.org/projects/Profiles/Profiles01/03MHSIP2001.pdf. Paykel, E. S. (1995). Psychotherapy, medication combinations, and compliance. Journal of Clinical Psychiatry, 56(Suppl. 1), 24–30. Powell, R. A., Holloway, F., Lee, J., & Sitzia, J. (2004). Satisfaction research and the uncrowned king: Challenges and future directions. Journal of Mental Health, 13(1), 11–20 http://dx.doi.org/10.1080/09638230410001654495. Rey, J. M., O’Brien, M., & Walter, G. (2002). Is the satisfied customer one who also ‘does well’? The relationship between outcome and parent satisfaction in a child and adolescent mental health service. Australasian Psychiatry, 10(3), 246–249 http:// dx.doi.org/10.1046/j.1440-1665.2002.00485.x. Ruggeri, M., Lasalvia, A., Salvi, G., Cristofalo, D., Bonetto, C., & Tansella, M. (2007). Applications and usefulness of routine measurement of patients’ satisfaction with community-based mental health care. Acta Psychiatrica Scandinavica, 116(Suppl. 437), 53–65 http://dx.doi.org/10.1111/j.1600-0447.2007.01093.x. Sohn, M. (2011). What makes people generally satisfied with mental health services?: Findings of 2010 consumer satisfaction survey in Kentucky Community Mental Health Centers (Public Policy Capstone). Swickert, R., & Hittner, J. (2009). Social support coping mediates the relationship between gender and posttraumatic growth. Journal of Health Psychology, 14(3), 387–393 http://dx.doi.org/10.1177/1359105308101677. Tamres, L., Janicki, D., & Helgeson, V. (2002). Sex differences in coping behavior: A meta-analytic review and an examination of relative coping. Personality and Social Psychology Review, 6, 2–30 http://dx.doi.org/10.1207/S15327957PSPR0601_1. Taylor, S., Klein, L., Lewis, B., Gruenwald, T., Gurung, R., & Updegraff, J. (2000). Biobehavioral responses to stress in females: Tend-and-befriend, not flight-or-fight. Psychological Review, 107, 411–429 http://dx.doi.org/10.1037/0033-295X.107.3.411. Turchik, J. A., Karpenko, V., Ogles, B. M., Demireva, P., & Probst, D. R. (2010). Parent and adolescent satisfaction with mental health services: Does it relate to youth diagnosis, age, gender, or treatment outcome? Community Mental Health Journal, 46(3), 282–288 http://dx.doi.org/10.1007/s10597-010-9293-5. Eliza Robillos is a Researcher and Project Coordinator for Health Services Research Center at the University of California, San Diego. She oversees and directs projects from inception to completion, including data management for the Mental Health Statistics Improvement Program (MHSIP) Consumer Survey. She ensures that thousands of people receiving mental health services are able to receive their satisfaction surveys to acquire ample data to analyze and help improve mental health services in San Diego County. She is using her experience in outcomes research toward advocacy for solutions that eliminate health disparities, particularly in gender. Rachel Lale is a doctoral student in Clinical Psychology at Idaho State University and worked as a Community Program Representative at Health Services Research Center at the University of California, San Diego, when this study was conducted. Her previous experience as an alcohol and drug counselor shaped her research interests in clinical psychology with specific emphasis on information processing in psychopathology with comorbid substance use disorders. At HSRC she contributed to projects related to community mental health outcomes, stigma, and barriers to seeking mental health treatment. Jennalee Wooldridge holds a Master’s degree in Psychology emphasizing in physical and mental health research from San Diego State University. She is interested in intervention research as a means for understanding factors that influence health behaviors. Jennalee is a Program Evaluation Specialist at the Health Services Research Center at the University of California, San Diego where she has been responsible for data analysis, reporting, and liaison with community groups related to the Prevention and Early Intervention (PEI) evaluation. Richard Heller is an Assistant Community Health Program Representative for Health Services Research Center at the University of California, San Diego. He has conducted interviews and focus groups with people receiving mental health services to evaluate their needs and the effectiveness of programs. He is mobilizing his personal experience as well as his qualitative research experience to better understand stigma, services, and access to care by people receiving mental health services. Andrew Sarkin, Ph.D. is the Director of Evaluation Research at Health Services Research Center at University of California, San Diego. He uses his training in clinical psychology and outcomes research to develop better ways to measure client recovery while teaching clinicians how to use new recovery measures in clinically meaningful ways. He is a primary organizer of the wide group of stakeholders involved in choosing and implementing new recovery-based assessments. He also directs efforts to evaluate Prevention and Early Intervention projects being implemented in San Diego County to reduce stigma, increase access, and promote knowledge about mental health services.