Involvement of users and relatives in mental health service evaluation.

ORIGINAL ARTICLE

Involvement of Users and Relatives in Mental Health Service Evaluation Angelo Barbato, MD,*Þ Barbara D’Avanzo, PhilD,* Vito D’Anza, MD,þ Emanuele Montorfano, PsyD,§ Monica Savio, PhD,§ and Caterina G. Corbascio, MD, PhD§ Abstract: Although Italian mental health (MH) services are community based, user and relative participation in service evaluation lagged behind until lately. We here review three recent studies involving stakeholder participation in service evaluation: two were quantitative studies, one on 204 users in an MH service in Pistoia (Central Italy) and the other on 2259 relatives, conducted with the National Union of Associations for Mental Health. The third (supported by The Centro per il Controllo delle Malattie, the ministerial Center for Disease Control) was a qualitative study in seven MH services, involving users, relatives, and professionals together, which collected interviews from 136 users, 119 relatives, and 79 professionals. In the quantitative studies, positive evaluations outnumbered negative ones. The qualitative study explored negative aspects in greater depth. Common findings were insufficient information, underinvolvement of users-relatives in planning, no choice of clinician, psychiatrist domination, and limited helpfulness of interventions. With stakeholder participation in service evaluation, the present medical framework will need reshaping. Key Words: Community mental health, Italy, consumers, service evaluation. (J Nerv Ment Dis 2014;202: 479Y486)

G

overnment policies and guidelines in Europe, the United States, and other countries during the past 30 years have promoted user and relative participation in various aspects of mental health (MH) care (Church and Reville, 1989; European Commission, 2005; New Freedom Commission on Mental Health, 2003; National Health Service Health Advisory Service, 1997). Constructive partnerships among patients, relatives, and clinicians are today considered central to the development of recovery-oriented services (Wallcraft, 2012). The involvement of users and their relatives in service planning, delivery, and evaluation was prompted by early evidence of dissatisfaction with services (Rogers and Pilgrim, 1993; Schene and Van Wijngaarden, 1995). By contrast, a positive correlation was consistently found between patients’ and relatives’ perceptions that their needs are being met, the therapeutic alliance, and the outcome of care (Roth and Crane-Ross, 2002). A large body of research is available on various aspects of the consumer perspective: attitudes, satisfaction, preferences, wishes, and expectations of care (Marshall et al., 2007; Noble et al., 2001). However, despite widespread efforts, there is limited material on consumer involvement in service evaluation, quality improvement, and outcome assessment (Weinstein, 2006). Although most studies focused on consumers’satisfaction, a number of researchers suggested that, often, this is not related to well-

*Laboratory of Epidemiology and Social Psychiatry, IRCCS Mario Negri Institute for Pharmacological Research, Milan, Italy; †IRIS Postgraduate School of Psychotherapy, Milan, Italy; ‡Department of Mental Health of Pistoia, Pistoia, Italy; and §Department of Mental Health, Azienda Sanitaria Locale AT, Asti, Italy. Send reprint requests to Angelo Barbato, MD, Laboratory of Epidemiology and Social Psychiatry, IRCCS Mario Negri Institute for Pharmacological Research, Milan via La Masa 19, 20156 Milan, Italy. E-mail: [email protected]. Copyright * 2014 by Lippincott Williams & Wilkins ISSN: 0022-3018/14/20206Y0479 DOI: 10.1097/NMD.0000000000000148

The Journal of Nervous and Mental Disease

defined standards and is not reliable for evaluating health care performance (Avis et al., 1997). We need to move beyond surveys of satisfaction and assess consumers’ judgment about service values, organization, care models, and performance (Groenewegen et al., 2005). Although the distinction between consumer satisfaction and consumer-led evaluation may seem trivial, it is actually quite concrete and relates to a) the passivity implicit in the term satisfaction, where users are required to limit themselves to reacting to the health offer, instead of being actively involved, and b) the overly simple idea that the experience of care in a health system can be faithfully mirrored by a straightforward rating from fully satisfied to not at all satisfied, thus neglecting any interpretation/rationalization about the complexities of the system and people’s own experience (Edwards and Staniszewska, 2000). If researchers do not thoroughly understand the evaluation process, they may question consumers in an inadequate way on issues that are not relevant, thus drawing inappropriate conclusions. From this point of view, qualitative studies may be helpful because they allow for a process-based approach rather than relying on a summary score and may capture the users’ experience, and dynamic interpretation of that experience, better (Edwards and Staniszewska, 2000). Quantitative studies involving representative samples of people and tools developed according to users’ and relatives’ indications can also improve quality assessment, provided that such tools follow a participatory model. Once again, assessment of consumers’ judgment is less subject to a satisfaction bias (Haggerty, 2010). User and relative involvement in MH service evaluation is a challenging endeavor, for a number of reasons: & The power imbalance vis-a`-vis providers is greater than in other health care sectors (Laugharne and Priebe, 2006). & Patients, relatives, and providers see many (although not all) aspects of service quality from different perspectives (Mason et al., 2004; Shumway et al., 2003). Concordance between patients and psychiatrists is out of the question for many of the treatment goals in psychoses (Bridges et al., 2011), whereas disagreement between patients and relatives is apparent from many surveys, with relatives assigning more importance to supportive intervention, hospitalization, compulsory treatments, and medication (De Haan et al., 2001). & Users with mental disorders often belong to marginal groups and experience social exclusion (Jorm and Oh, 2009). A recent worldwide survey found that, very often, people with schizophrenia experienced negative discrimination in making or keeping friends, finding a job, or looking for a close relationship (Thornicroft et al., 2009). The relatives of people with mental illness, to some extent, experience stigma and social isolation as well (Struening et al., 2001). & Most people with mental disorders are considered by clinicians unable to provide meaningful feedback about their health care experience because of lack of insight and impaired reality testing (Buckley et al., 2007). Such challenges are especially important for the subgroup of people with serious mental illness, who often show high needs and require long-term care. However, a number of studies suggest that

& Volume 202, Number 6, June 2014

www.jonmd.com

Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

479


Barbato et al.

even users with a history of severe disorders can be involved in service evaluation, provided they receive adequate support, and report their involvement as an empowering experience (Simpson and House, 2002). Involvement of relatives goes back further than users. It has been suggested that the shift from an institutional toward a communitybased model, with its orientation toward social inclusion of people with mental disorders, should be mirrored by their increasing legitimacy as key stakeholders (Thornicroft and Tansella, 2005). Although this sounds reasonable, it is not necessarily true. In Italy, despite a radical shift toward community care after the closure of mental hospitals in the late 1970s (Barbato, 1998), user participation in service delivery and evaluation lagged behind until a few years ago. Even the latest MH policy guidelines provide no specific indications on the inclusion of consumers in service planning and delivery (Ministero della Salute, 2008). Recent growth of interest in this issue led first to the development of instruments to assess users’ satisfaction, such as the Verona Service Satisfaction Scale (Ruggeri and Dall’Agnola, 1993), and later to a number of Italian studies mainly focusing on satisfaction and needs assessment (Gigantesco et al., 2002; Lasalvia et al., 2005). The tools, however, were developed by professionals and researchers, thus ignoring the consumers’ knowledge. More recently, a number of investigations have examined relatives’ beliefs, attitudes, and opinions (Magliano et al., 2004, 2003), and a few have addressed users’ and relatives’ competence in identifying relevant issues; compared users’, relatives’, and professionals’ views; and designed quality improvement strategies based on a model of participatory research (Bajoni et al., 2010; Barbato et al., 2014; Corbascio and Savio, 2012). We here present a narrative review centering on similarities and differences in three recent studies that mark a step forward within the framework of participation by all stakeholders in service evaluation. In a study conducted in Pistoia (Central Italy), views about the service were gathered as part of a quality improvement project involving users with severe mental disorders in a strongly communityoriented service, based on an outreach care model (Barbato et al., 2014). In a second study, on behalf of the National Union of Associations for Mental Health (UNASAM), the main Italian association of relatives of people with mental illness, a sample of relatives assessed the quality of community MH centers in four regions (Bajoni et al., 2010). The third study, funded by the Center for Disease Control (CCM) under the Italian Ministry of Health, was run in seven MH services, coordinated by the Mental Health Department of Asti. It adopted a qualitative strategy, to improve quality by comparing the views of stakeholders: users, family members, and professionals (Corbascio and Savio, 2012). All studies used a participatory research model (Cornwall and Jewkes, 1995) and aimed to improve the quality of services through the active participation of stakeholders. We will integrate the quantitative and qualitative results in a single framework, highlighting the strengths of both approaches and the information specifically produced by each. The UNASAM study has been previously described (Bajoni et al., 2010), as well as the findings from the first stage of the Pistoia study (Barbato et al., 2014). Preliminary results from the CCM study are reported here for the first time. In summary, we will be answering the following questions: & What are the most critical aspects of strongly community-oriented services in Italy and which factors are most appreciated in users’ and relatives’ opinion? & Do qualitative findings shed light on issues not considered in quantitative studies? How can we integrate indications from these studies? 480

www.jonmd.com


& What can be learned from these experiences in terms of the weaknesses of the MH services in Italy and how can more experiences of participatory evaluation be promoted? & What prevents services from making a more serious appraisal of these findings and applying routine quality evaluation with stakeholder participation?

METHODS The main features of the three studies are summarized in Table 1. All three studies shared a common ground of participatory research, where researchers work in partnership with people directly involved in the issue being studied, bringing their own experience to the foreground, for the purposes of education, action, and social change (Cornwall and Jewkes, 1995). Other equally important goals of participatory research are to answer health questions and benefit the partners in the research process, while developing valid generalizable knowledge. With this framework, in all the studies, a) tools were designed by the stakeholders; b) stakeholders were also active in data collection and in the discussion; c) partnership was kept as strong as possible, to ensure that all choices respected the stakeholders’ preferences; and d) the final aim was to introduce relevant changes in everyday service delivery.

Pistoia Study This was a pre-post study carried out in two services belonging to the same department of MH. A first quality assessment was made; the results were presented to the service and were jointly discussed by users and professionals, to identify the areas most in need of change. A second assessment was subsequently made to check whether quality was perceived as having improved. A questionnaire was prepared by focus groups to identify the main issues and the appropriate wording for the questions. The groups included people belonging to a local users’ association, with researchers as facilitators. The participants preferred answers on a Likert scale and scores expressed in words accompanied by a smile sign. Variables were continuous, and some filter questions were used to select subjects who could give reasonable answers to relevant questions. Users who collected the data were briefly trained to standardize how they contacted people, presented the study, explained the questionnaire, and helped in completing the questionnaire. The subjects to be interviewed were 290 patients defined as heavy users on the basis of diagnosis and use of resources. This choice was intended to specifically address the opinions of users who were most in need of intensive or prolonged care. Data collection took three months and engaged six users paid by project funds. They contacted subjects identified first in day centers and residential facilities, then in MH centers and clinics, and finally at the patients’ homes. Those collecting data worked completely independently in day centers and residential facilities, but they received some logistic support with contacting the sample subjects in other settings. In general, professional involvement was kept to a minimum. In the analysis, to correct for a tendency toward compliance, because answers were on a Likert scale, the very negative and negative ratings were collapsed, and only the very positive evaluations were considered indicators of good quality. More information is given in Table 1.

UNASAM Study This adopted a cross-sectional design, in which opinions were gathered from all relatives accessing 46 community MH centers during an index period. Focus groups were organized to identify the main issues, choose the best wording for the questions, and draw up the questionnaire. Participants were mostly relatives affiliated to associations, with two researchers as facilitators. In the questionnaire, * 2014 Lippincott Williams & Wilkins


* 2014 Lippincott Williams & Wilkins

No. Meetings for Tools Developing

12 in each center: 4 for each stakeholder, aimed to define quality issues (84 focus groups); 2 national meetings to merge quality items and define 3 topic guides (1 for each stakeholder) for peer-to-peer interviews; 4 training sessions in each center for each stakeholder (28 focus groups) to learn recruitment and qualitative interviewing techniques.

CCM (qualitative)

7 community MH services: Asti, Turin, Bologna (2), Reggio Emilia, Livorno, and Bari. 2-year project, data collection and analysis ended in November 2012

4 in 3 months involving 14 users

Pistoia Day centers, (quantitative) residential facilities, community MH centers, patients’ homes.

UNASAM Lombardy (20 centers), Two in each region (quantitative) Emilia-Romagna (13), (Lazio and Sardinia Sardinia (9), Lazio (4) were held together) involving 29 relatives

Study, Type

Place, Setting, and Period

TABLE 1. Main Features of the Three Studies No. Administrations

Administration Modalities

55 yes/no questions, 1, September to In the waiting rooms, self-administered November 2007 relatives approached other relatives. 66 yes/no questions, All opening hours 3 continuous questions, were covered. 15 filter questions Users with F20YF29, 6 users delivered the 11 yes/no filter questions 2 administrations, F30YF39, and F60YF69 and 34 questions with November 2009 questionnaires to ICD diagnoses, who had a Likert scale, to January 2010 the selected subjects been heavy users in the self-administered and October 2011 in the defined settings last year according to an with peer or professional to January 2012. and helped to complete agreed definition. assistance. them if necessary. Peer-to-peer interviews, First step: peer 3 topic guides: 1 for each 3 stakeholders: users, recorded and transcribed recruitment stakeholder group organized relatives, and verbatim (April 2012); in 4 areas: care plan, professionals second step: relationships in the context peer-to-peer of the center, quality of the interviews organization, care model (May to July 2012) of the service

All relatives accessing the community MH services for any reason

Target

Questionnaires: Areas, Items, Variables, Reliability

Analysis

By the researcher. Focus groups discussed the findings and derived indications for change. A general coding structure agreed with the stakeholders Coding carried out by the researcher who also co-coordinated the project. Coding results shared and analyzed with the stakeholders for building the quality improvement plans in each service.

By the researcher. Findings discussed in the focus groups.


& Volume 202, Number 6, June 2014 User-Carer Evaluation of MH Service

www.jonmd.com


481


Barbato et al.

all but two variables were worded as yes/no questions, and some filter questions were used to select people who could reasonably be expected to answer them. A total of 46 MH centers agreed to have the questionnaire distributed by relatives in the waiting rooms. These centers were identified as those where the associations were active. Not all the services allowed data collection. The centers were therefore unevenly distributed in the four regions, and the mean number of questionnaires collected varied greatly across regions and centers. In the analysis of the Pistoia and UNASAM studies, percentages were calculated on the actual numbers of responders to each question, and p-values for repeated measures were computed when appropriate. Issues were considered critical if there were more than 25% of negative evaluations. This criterion draws on a fairly broad basis because various marketing studies take even lower percentages as critical. More information can be found in Table 2.

CCM Study This qualitative study relied upon the active participation of the three main stakeholdersVusers, relatives, and professionalsVin evaluating the quality of the care received. All stakeholders were interviewed by stakeholders from the same group. The study proceeded through seven sequential steps:


1. Training of the professionals. The project coordinators of each service were trained in user involvement issues. 2. Recruiting the stakeholders for focus groups. In each service, 10 stakeholders for each group were involved, reaching a total of 210 (70 patients, 70 family carers, and 70 professionals). 3. Laying the basis for participation and shared knowledge. In each MH service, the three stakeholder groups each met four times, to identify key points of good MH care from a group-specific perspective. These were gathered into a map of quality issues, divided into four main areas: care plan, organization, relationships, and underlying care model. 4. Preparation of three topic guides for semistructured peer-to-peer interviews. The final quality maps were the result of joint meetings among all the MH services. Differences and similarities were analyzed and discussed. The maps were then used as the basis of topic guides for the semistructured interviews specific for each stakeholder group. 5. Training the interviewers. Each group of stakeholders was trained in interviewing techniques and discussed their concerns and expectations in the role of interviewer. The users were paid for their work on the study, to acknowledge the value of their ‘‘expertise by experience.’’ 6. Recruitment of interviewees and interviews. In the seven MH services, 612 users, 284 family carers, and 167 professionals were

TABLE 2. Issues Included in the Common Areas in the Three Studies Specific Quality Issues Quality Areas

Accessibility

Environment Communication and information

Involvement, care plan, choice

UNASAM

Awareness of the care plan, involvement in the definition of the care plan, and involvement in regular assessments

Relationships

Effectiveness

Support and listening when asked, actual improvement of the patient, actual improvement of relations in the family, improvement in treatment compliance, home visits if necessary Professionals’ skills Competence of professionals as a whole Model of care

482

Pistoia

Service location, opening days Waiting time on the telephone, and hours, waiting time on the punctuality, waiting time for telephone, answers on the telephone, non-scheduled meetings help punctuality of professionals, waiting in emergency time for non-scheduled meetings, answer to emergency outside opening hours Size and comfort of the service rooms Size and comfort of the service rooms in each setting Information on users’ financial, Information about illness and welfare, and legal rights treatment, on drug prescription, and on treatments offered by the service

www.jonmd.com

CCM

Location of the service, opening days and hours, home visits, and intervention for the emergencies

Size and comfort of the service rooms

Information about illness and treatment, several aspects of drug prescription and side effects; information about interventions offered by the service Possibility to choose clinicians, Involvement of users in the care plan; professionals’ consideration sharing the plan with family carers and of the user’s opinions nonYmedical professionals; goals, on treatment timing, and monitoring of the plan; housing; job placement support; commitment to recovery Relationship with psychiatrist, Relationship with the psychiatrist and psychologist, rehabilitation the other professionals, presence of a counselor, social worker, nurse, key worker (also during hospital and support worker; support admission, i.e., continuity of care), and listening during acute listening with confidence and availability ward admission; emotional on emergencies. ‘‘Does your key worker climate of the service ask you what you need to be happy?’’ Helpfulness and side effects of Person-centered care, support and hope drugs, psychological interventions, in the recovery process, self-help groups, and self-help groups; satisfaction satisfaction with job placement and social with job placement; and social inclusion interventions, interactions inclusion interventions with the local community, antistigma programs, peer-support workers Competence of each professional Competence of each professional in the care team, availability of support and training for professionals Presence of a shared care model among the professionals





contacted and agreed to be interviewed. Among these, 20 user interviewees were randomly selected, 20 relatives and 12 professionals in each service. In all, 136 users, 119 relatives, and 79 professionals were actually interviewed. The interviews were conducted between peers, audiotaped and transcribed verbatim. The transcripts were analyzed by thematic coding with NVivo software. 7. Outlining improvement plans. The results of thematic coding were shared with all stakeholders at a national meeting. In each service, the three stakeholder groups outlined a shared quality plan identifying the key factors for improving the service.

RESULTS The main findings of the three studies are summarized in Table 3.

UNASAM Study A total of 2259 questionnaires were analyzed. Positive judgments were highest on professionals’ accessibility and kindness (94%) and lowest on information about other care options in the community (25%). Waiting on the telephone had 50% positive judgments, and other indicators of accessibility were rated even better (e.g., opening days and hours, position and transport). Positive evaluations for help during service closing hours totaled 29%. Participation in the care plan was rated more positively when questions were worded in a more general way (83% were aware of the care plan) but less so for more specific questions (55% were involved in

User-Carer Evaluation of MH Service

regular assessments). Collaboration in the team was good for 73% but only 40% when it came to collaborating with other agencies in the community. Support and listening were deemed good by 80%, but only 34% perceived any improvement in the patient; and only 38%, in the relationship between patient and family. Ratings were worse in respondents whose family member had a longer history of contact with the MH services: in particular, these decreased to a statistically significant degree for professionals’ manner, accessibility and competence, the adequacy of opening hours, professionals’ punctuality for appointments, waiting time on the telephone, quality of help received by telephone, support, and attentive listening.

Pistoia Study A total of 204 people responded to the questionnaire. Valid answers ranged from 67 for self-help group helpfulness to 193 for the MH service environment. Positive ratings outnumbered negative ones on almost all counts, although with significant differences from area to area. Positive evaluation ranged between 67% for psychiatrists’ skills and 19% for waiting time before meeting a professional, whereas negative evaluation ranged between 57% for choice of professionals and 4% for social workers’ skills. Questions about the psychiatrist gave the most valid answers, whereas there were fewer valid answers for other professionals. More than 50% gave favorable ratings on all professionals’ competence (from 55% for the support worker to 67% for the psychiatrist) and relationships with each of them (62% for the rehabilitation counselor

TABLE 3. Main Findings Study

No. Subjects

Main Sample Characteristics

Main Results

UNASAM

2368 (2259 questionnaires analyzed)

40% men, 14% G40 yrs, and 15% Q70 yrs; median, 57; 10% more than 20 yrs of contact with the service (median, 6); median of 1 contact of the patient per month

Pistoia

First administration: 204, second administration: 199

CCM

Peer-to-peer qualitative interviews: 136 with users, 119 with family carers, and 79 with professionals selected at random basis among those who agreed to be interviewed

First administration: 54% men, 33% e40 yrs, and 31% 950 yrs; 78% unemployed, 52% schizophrenic disorders; second administration: 58% men, 20% e40 yrs, and 40% 950 yrs. 136 users: 51% men; age: 9% e30 yrs; 55% 31Y50 yrs; 28% 51Y70 yrs; 8% 970 yrs; duration of contact with the service: 15% 0Y2 yrs; 14% 2Y4 yrs; 15% 5Y7 yrs; 55% 97 yrs;

Positive evaluations were highest for staff’s accessibility, kindness, and competencies, lower for the efficacy and the ability to improve family-patient relationship; the lowest was found for information and involvement in the care project. Quality was perceived as moderate for most indicators. The critical issues were choice, information, waiting time, medication side effects, and self-help groups. Best scores were for psychiatrists’ and nurses’ skills. Few changes between the 2 surveys. Users, relatives, and professionals highlighted the need to improve the care plan: a) broad flexible approach, not centered only on medications, with defined objectives to be reached through sequential steps; b) involving users and family carers; and c) ensuring social inclusion and leisure activities. Most users acknowledged good relationship with the key worker (mainly identified with the psychiatrist). Users and relatives asked for more information on drug therapy and side effects. The service model was often poorly defined. Users and family members asked for recovery-oriented approaches. Significant organizational differences were seen concerning access (i.e., length of opening hours ranged between 12 and 6). In all services, the stakeholders produced a joint quality improvement plan.

79 professionals: 77% women; 39% nurses, 25% psychiatrists, 13% educators, 17% support workers, 5% social workers; 119 family members: 58% women; 61% parents, 24% partners, 8% siblings, 5% daughters and sons, 2% other relatives


www.jonmd.com


483


Barbato et al.

and 63% for the nurse), vocational rehabilitation services (61%), and quality of the MH center environment (52%). Negative opinions peaked for choice of professional (57%) and were more than 25% for waiting times (50% to meet a professional, 35% on the telephone), information about illness (29%) and drugs (26%), self-help groups (27%), and medication side effects (27%). No pre-post changes between the two surveys were observed in the case of one of two same-department services, whereas in the second one, statistically significant changes were observed for the environment of the day center (worsened), possibility to choose the professional (improved), and waiting time before meeting a professional (improved). Some critical issues remained: choice of professionals, information, length of waiting, medication side effects, and self-help groups.

CCM Study One specific result of this study hinged upon the methodological approach that allowed users, relatives, and professionals to share their views on ‘‘what makes for quality in our service.’’ Stakeholders’ perspectives are usually not discussed on common ground within services. Users and clinicians pointed out that their relationship was changed by taking account of the other party’s perspective based on the experience of MH care. The interviews generated three types of findings: a) at a national level: what is the common quality demand across services, b) at a local level: what is the quality demand in each service, and c) for the stakeholder group: what is the service quality according to each group perspective. Here, we summarize the results common to all stakeholders across services in the quality areas, as well as the substantial differences among stakeholders. 1. The care plan. Users and family carers agreed that, in many services, the care plan lacked direction in terms of goals, timing, and monitoring. It was described as fixed and lacking a future orientation and was perceived as hindering users’ and professionals’ trust in improvement. The care plan in most cases coincided simply with the drug treatment plan. Moreover, users and relatives were not involved, and professionals did not provide sufficient information. The experiences and feelings generated resignation to the illness being chronic. For users and carers, this was often described as a ‘‘life sentence,’’ whereas for professionals, it was an unspoken defeat. ‘‘If we do not believe in recovery as a goal, we cannot ask patients or family carers to pursue it’’; ‘‘you just get used to never seeing them better, in my career I have seen far more chronic situations than recoveries.’’ It was generally accepted that a care plan not shared with users or family carers and lacking goals could impair the chances of recovery. Most users expressed the need for a written (bona fide, more clear, and dependable) care plan. 2. Relationships in services. The quality of users’ relationships was closely intertwined with other issues, such as the care plan and the service organization. Users defined the relationship with professionals as generally good. However, both users and family carers expressed a general desire to improve connections with the key worker, who was the psychiatrist in more than half the cases. The predominant role of psychiatrists, not only as clinicians but also as care coordinators and the main point of contact for patients, was perceived by most professionals as a factor limiting application of the care plan, which should require all the professionals to be involved. Professionals also paid attention to relationships inside the service, where hierarchies still played a role and were often described as top-down oriented. 3. Quality of organization and access. Most professionals perceived the organization as rigid, lacking flexibility, and tailoring to users’ 484

www.jonmd.com


and relatives’ needs. In one service, all stakeholders strongly objected to the opening hours, which were limited to 6 hours per day. Users and carers asked for greater telephone access and availability over weekends. Relatives consistently and strongly expressed the need for assistance during weekends and emergencies. Overall, the service organization was described as biased toward structural and administrative issues rather than being focused on care. In line with this, the professionals perceived the organization as extraneous and not sufficiently committed to patients’ needs. This was reported in turn to cause lack of motivation and fatigue. 4. The care model followed by the service. This quality area investigated the care models adopted by services and the professionals’ awareness of models. The conceptual basis underlying service interventions was often ill defined and varied widely across services and even among professionals on the same team. Users and relatives often expressed the need for professionals to be trained on a recovery model. Interestingly, users themselves introduced the concept and the term of recovery and also put forward a number of questions on this issue in their topic guide. Professionals also stressed the need for more training on evidencebased practices.

DISCUSSION The three studies we reviewed are different in a number of ways: two were quantitative, one addressing users’ views and the other addressing relatives’ views; the third was a qualitative study addressing users’, relatives’, and professionals’ opinions simultaneously, focusing on dialogue and exchange among the three categories of stakeholders. The first included a representative sample of people with severe mental illness from a single service; and the second, a very large representative sample of relatives from 46 centers in four regions. The third, thanks to the qualitative method adopted, allowed a thorough analysis of a large sample of stakeholders from seven services. These conclusions cannot be generalized to all Italian services but do give some suggestions, to be confirmed by further nationwide research. Despite methodological differences, a number of common conclusions can be drawn. Critical issues were in the areas of information and communication, involvement in the care plan and other core decisions, access, helpfulness, and effectiveness. Users stressed the need for a shared care plan, which was actually tended often to be implicit and based exclusively on medication. Good relationships with professionals did not correspond to involvement, partnership, efficacy, empowerment, and sufficient information. These issues lay on different levels and pertained to personal aspects of the professionals, more than to the quality of the system. Similar conclusions were drawn by Barker et al. (1996): the high scores for personal relationships were not paralleled by satisfaction with communication with physicians or other empowering factors. A qualitative study in several European countries (Van Weeghel et al., 2005) also concluded that all the stakeholder groups interviewed attached the highest value to a respectful and empowering working alliance. Most of the critical issues emerging from these studies relate to the concept of responsiveness, as defined in the World Health Report 2000 (World Health Organization, 2000). Responsiveness is the ability of a service to overcome the negative side effects associated with mental illness and being treated for it. This also means that services must counteract the imbalance of power felt by users and relatives and should work on empowering, strengthening the individual’s rights in the health care system (Bramesfeld et al., 2007; Valentine et al., 2003). The participatory approach, permitting dialogue among stakeholders, highlighted the advantages of this method. Involving * 2014 Lippincott Williams & Wilkins




stakeholders in all steps of studies and empowering them in defining quality issues, language and style in questionnaires, and topic guides produced a high level of motivation. Users acknowledged this procedure as extremely valuable, providing a real opportunity to share their experience of mental illness with peers, as well as being able to point out negative aspects of services that needed to be changed. The fact of users being paid for their research proved fundamental in supporting these feelings because it acknowledged their role as evaluators. Contacts established by peers during questionnaire distribution and peer-to-peer interviews served as a vehicle by which to reach other users and relatives. In some services in the CCM study, it also supported the idea of introducing peer support workers. Entitling the stakeholder groups to be protagonists in rating the quality of services set up a constructive dialogue in which different points of view were acknowledged and valued. Putting all the actors at the center of the process was a powerful tool for changing relationships and dynamics within the service and involving everybody in the common aim of improving the service. This was particularly apparent in the CCM study, which involved all stakeholders in the same task and aim. More difficulties were encountered in the UNASAM study because the professionals endorsed the study in only a limited number of services: the results were shared with professionals in only these few cases and did not form the basis for quality improvement. The CCM study also investigated which care model supported care delivery in services. Professionals commented that the model of care was basically medically oriented and centered on the psychiatrist. Many said that there was no identifiable care model. Care plans and care models could be seen as two faces of the same coin, with users and professionals highlighting similar consequences: paucity of goals and difficulty in measuring outcomes and in sustaining hope and motivation. This seems to be very important common ground for building together the necessary changes in community MH services. Edwards and Staniszewska (2000) said that the advantage of qualitative studies lies in the possibility of overcoming the tendency toward positive evaluations: one gets a more reliable account of users’ experience with the care system. This proved only partly true because quantitative studies here succeeded in pointing out critical aspects too. Even users with severe conditions were reliably able to express their judgment. The Pistoia study, in particular, addressed heavy users’ opinions, whereas the CCM study included users with severe mental illness among both interviewees and interviewers. Whereas the quantitative studies highlighted many positive aspects and negative ones, positive aspects emerged less from the qualitative study, in which there was often more stress on the negative sides. Positive ratings in the quantitative studies were nonetheless high. In the UNASAM study, this surprised the relatives themselves who conducted the study: they had some difficulty in fully understanding the meaning of such positive evaluations. However, the fact is that positive evaluations can represent both the highest point on a quality scale and the achievement of a reasonably acceptable minimum. The complexity of the relationship between experiences, expectations, and evaluations has long been discussed by a number of authors (Sheppard, 1993). The UNASAM study also highlighted a gap between the expectations of relatives affiliated to associations and an unselected sample of relatives: several issues indicated as being critical in focus groups turned out to be less negative than expected. This was the case, for instance, with home visits and emergency help. As a group, affiliated relatives may be more critical toward services, whereas a more heterogeneous sample may tend to dilute what are in fact common experiences and feelings. Several indications can be drawn from these findings. First, the care plan must be a direct target for quality improvement. Some * 2014 Lippincott Williams & Wilkins

User-Carer Evaluation of MH Service

regional governments have addressed the question (Regione Lombardia, 2004), issuing guidelines prescribing that a personal care plan must be written for each user. Although the guidelines represent a step toward quality improvement, there is no easy way to ensure that these are followed, or to check the quality of the plans, and whether these are really shared, personalized, and implemented. A special effort was made in the MH service of Trento (Italy) to address sharing goals, means, and the relational framework of the team defining the care plan (including users, relatives, and professionals; D’Avanzo et al., 2012). Stress was laid on the negotiating potential of the relationship and the sense that it became a contract, where everyone took their own responsibilities. Second, users, family members, and professionals should rate services on a regular basis. One critical point is to provide a mechanism for maintaining participatory evaluation as an ongoing activity within services. In the CCM study, for instance, the stakeholders benefited from external coordination and from the Ministry grant that enabled them to pursue several extra routine tasks. Everyday pressures, shortage of funds, work overload, and the lack of exchange between services may well hinder the dynamics that participation requires. Nevertheless, this project has shown that the first requirement, if practices are to change, is a change in professionals’ perspectives. The experience teaches us that, once a change in relationships has been effected, people are unlikely to revert to previous positions. Third, we also wanted some clues about barriers to implementation of a shared stakeholders’ evaluation in the Italian services. The scenario of Italian psychiatry reflects a somewhat different historical background than other leading European countries. The psychiatric reform stemmed from initiatives mainly led by psychiatrists under the pressure of social claims. The result was a radical shift from a hospital-centered system to community psychiatry. This pioneer experience from the early days of the reform anticipated an interdepartmental organizational model relying on a network of services, closely interconnected and coordinated by psychiatrists. On the one hand, the psychiatrists’ key role in the community services involved a change that meant that they were no longer confined to medical issues. On the other hand, the psychiatrists kept a central position in the framework of these multidisciplinary services. Application of the reform has been a lengthy process that may now be considered completed, but the inclusion of users and families as fully fledged partners within services has not yet been accomplished. Psychiatrists working in community services need to combine different tasks and skills, such as a) being a team leader, b) coordinating facilities and team workers, and c) being an expert clinician (Rosen, 2006). Furthermore, psychiatrists are expected to lead and promote modifications and improvements to service quality. Psychiatrists are consequently overloaded with work, as they themselves complain, and simultaneously affected by the general disappointment felt by the nonYmedical professionals in the team, as clearly emerged from the CCM study. The time is therefore ripe for patients and families to be included in planning MH services at local and national levels. If this is agreed, a partnership of users, relatives, and professionals will require some reshaping of the medical framework in which, although definitely community oriented, Italian services are still embedded. ACKNOWLEDGMENTS Caterina G. Corbascio, Monica Savio, and Emanuele Montorfano thank the coordinators of the local units S. Crosato (Reggio Emilia), M. De Giorgi (Turin), F. Guzzetta (Bologna), P. Pini (Livorno), and M. Squeo (Bari) for their contribution and the patients and families for their invaluable help and outstanding participation. The authors thank the Scientific Committee of the CCM project (G. Tognoni, C. Harding, R. Magliola, P. Henry, and F. Malavasi). www.jonmd.com


485


Barbato et al.

DISCLOSURES The CCM study carried out in Asti was supported by a grant from the Center for Disease Prevention and Control (CCM), Italian Ministry of Health (Rome, Italy), and by the Azienda Sanitaria Locale AT, Asti (Italy). The authors declare no conflict of interest. REFERENCES Avis M, Bond M, Arthur A (1997) Questioning patient satisfaction: An empirical investigation in two outpatient clinics. Soc Sci Med. 44:88Y92. Bajoni A, Barbato A, D’Avanzo B, Muggia E (2010) Family members’ assessment of community mental health services: A survey in four Italian regions. Epidemiol Psichiatr Soc. 19:183Y187. Barbato A (1998) Psychiatry in transition: Outcomes of mental health policy shift in Italy. Aust N Z J Psychiatry. 32:673Y679. Barbato A, Bajoni A, Rapisarda F, D’Anza V, De Luca LF, Inglese C, Iapichino S, Mauriello F, D’Avanzo B (2014) Quality assessment of mental health care by people with severe mental disorders: A participatory research project. Community Ment Health J. 50:402Y408. Barker DA, Shergill SS, Higginson I, Orrell MV (1996) Patients’ views towards care received from psychiatrists. Br J Psychiatry. 168:641Y646. Bramesfeld A, Wedega¨rtner F, Elgeti H, Bisson S (2007) How does mental health care perform in respect to service users’ expectations? Evaluating inpatient and outpatient care in Germany with the WHO responsiveness concept. BMC Health Serv Res. 7:99Y110. Bridges JFP, Slawik L, Schmeding A, Reimer J, Naber D, Kuhnigk O (2013) A test of concordance between patient and psychiatrist valuations of multiple treatment goals for schizophrenia. Health Expect. 16:164Y176. Buckley PF, Wirshing DA, Bhushan P, Pierre JM, Resnick SA, Wirshing WC (2007) Lack of insight in schizophrenia: Impact on treatment adherence. CNS Drugs. 21:129Y141. Church K, Reville D (1989) User involvement in the mental health field in Canada, Can Ment Health. 37:22Y25. Corbascio C, Savio M (2012). Valutazione della qualita` dei Servizi di Salute Mentale: Un percorso di analisi condivisa con pazienti in qualita` di valutatori. Final Report. Rome: Center for Disease Control & Prevention (CCM), Ministero della Salute. Cornwall A, Jewkes R (1995) What is participatory research? Soc Sci Med. 41: 1667Y1676. D’Avanzo B, De Stefani R, Arici S, Bertotti P (2012) A tool to increase share and participation in mental health care. The shared care paths. Presented at: Refocus on Recovery Conference 2012. Poster session. March 5Y7, 2012; London. De Haan L, Van Raaij B, Van den Berg R, Jager M, Houweling P, Stockmann M, Delsing P, Linszen D, Peters B, Wouters L (2001) Preferences for treatment during a first psychotic episode. Eur Psychiatry. 16:83Y89. Edwards CA, Staniszewska S (2000) Accessing the user’s perspective. Health Soc Care Community. 8:417Y424. European Commission (2005) Green Paper. Improving the mental health of the population: Towards a strategy on mental health for the European Union. Brussels, Belgium: Health & Consumer Protection Directorate General. Gigantesco A, Picardi A, Chiaia E, Balbi A, Morosini P (2002) Patients’ and relatives’ satisfaction with psychiatric services in a large catchment area in Rome. Eur Psychiatry. 17:139Y147. Groenewegen PP, Kerssens JJ, Sixma HJ, Van der Eijk I, Boerma WGW (2005) What is important in evaluating health care quality? An international comparison of user views. BMC Health Serv Res. 5:16. Haggerty JL (2010) Are measures of patient satisfaction hopelessly flawed? BMJ. 341:790Y791. Jorm A, Oh E (2009) Desire for social distance from people with mental disorders. Aust N Z J Psychiatry. 43:183Y200. Lasalvia A, Bonetto C, Malchiodi F, Salvi G, Parabiaghi A, Tansella M, Ruggeri M (2005) Listening to patients’ needs to improve their subjective quality of life. Psychol Med. 35:1655Y1665. Laugharne R, Priebe S (2006) Trust, choice and power in mental health. A literature review. Soc Psychiatry Psychiatr Epidemiol. 41:843Y852. Magliano L, Fiorillo A, De Rosa C, Malangone C, Maj M (2004) Beliefs about schizophrenia in Italy: A comparative nationwide survey of the general public, mental health professionals, and patients’ relatives. Can J Psychiatry. 49: 322Y330.

486

www.jonmd.com


Magliano L, Fiorillo A, De Rosa C, Malangone C, Marasco C, Guarneri M, Maj M (2003) The effect of social network on burden and pessimism in relatives of patients with schizophrenia. Am J Orthopsychiatry. 73:302Y309. Marshall SL, Crowe TP, Oades LG, Dean FF, Kavanagh DJ (2007) A review of consumer involvement in evaluations of case management: Consistency with a recovery paradigm. Psychiatr Serv. 58:396Y401. Mason K, Olmos-Gallo AO, Bacon D, McQuilken M, Henley A, Fisher S (2004) Exploring the consumers’ and providers’ perspective on service quality in community mental health care. Community Ment Health J. 40:33Y46. Ministero della Salute (2008) Linee di indirizzo nazionali per la salute mentale. Rome: Ministero della Salute. New Freedom Commission on Mental Health (2003) Achieving the promise: Transforming mental health care in America. Final Report. Rockville, MD. Department of Health and Human Services Pub. No. SMA-03-3832. National Health Service Health Advisory Service (1997) Voices in partnership: Involving users and carers in commissioning and delivering mental health services. London: Stationery Office. Noble LM, Douglas BC, Newman SP (2001) What do patients expect of psychiatric services? A systematic and critical review of empirical studies. Soc Sci Med. 52:985Y998. Regione Lombardia (2004) Direzione Generale Sanita`. Piano Generale Salute Mentale. Milan: Regione Lombardia. Rogers A, Pilgrim D (1993) Service users’ views of psychiatric treatments. Sociol Health Illn. 15:612Y631. Rosen A (2006) The community psychiatrist of the future. Curr Opin Psychiatry. 19:380Y388. Roth D, Crane-Ross D (2002) Impact of services met needs, and service empowerment on consumer outcomes. Ment Health Serv Res. 4:43Y56. Ruggeri M, Dall’Agnola R (1993) The development and use of the Verona Expectations for Care Scale (VECS) and the Verona Service Satisfaction Scale (VSSS) for measuring expectations and satisfaction with community-based psychiatric services in patients, relatives and professionals. Psychol Med. 23:511Y523. Schene AH, van Wijngaarden B (1995) A survey of an organization for families of patients with serious mental illness in The Netherlands. Psychiatr Serv. 46:807Y813. Sheppard M (1993) Client satisfaction, extended intervention and interpersonal skills in community mental health. J Adv Nurs. 18:246Y259. Shumway M, Saunders T, Shern D, Pines E, Downs A, Burbine T, Beller J (2003) Preferences for schizophrenia treatment outcomes among public policy makers, consumers, families, and providers. Psychiatr Serv. 54:1124Y1128. Simpson EL, House AO (2002) Involving users in the delivery and evaluation of mental health services: Systematic review. BMJ. 325:1265Y1268. Struening EL, Perlick DA, Link BG, Hellman F, Herman D, Sirey JA (2001) Stigma as a barrier to recovery: The extent to which caregivers believe most people devalue consumers and their families. Psychiatr Serv. 52:1633Y1638. Thornicroft G, Brohan E, Rose D, Sartorius N, Leese M, for the INDIGO Study Group (2009) Global pattern of experienced and anticipated discrimination against people with schizophrenia: A cross-sectional survey. Lancet. 373:408Y415. Thornicroft G, Tansella M (2005) Growing recognition of the importance of service user involvement in mental health service planning and evaluation. Epidemiol Psichiatr Soc. 14:1Y3. Valentine N, de Silva A, Kawabata K, Darby C, Murray CJL, Evans DB (2003) Health system responsiveness: Concepts, domains, and operationalizations. In Murray CJL, Evans DB (Eds), Health system performance, debates, methods and empiricism (pp 574Y596). Geneva, Switzerland: World Health Organization. Van Weeghel J, van Audenhove C, Colucci M, Garanis-Papadatos T, Lie´geois A, McCulloch A, Muijen M, Norcio B, Ploumbidis D, Bauduin D (2005) The components of good community care for people with severe mental illnesses: Views of stakeholders in five European countries, Psychiatr Rehabil J. 28: 274Y281. Wallcraft J (2012) What has been learned from joint working between mental health professionals, patients and users of psychiatric services, their families and friends? Curr Opin Psychiatry. 25:317Y321. Weinstein J (2006) Involving mental health service users in quality assurance. Health Expect. 9:98Y109. World Health Organization 2000 World Health Report 2000: Health systems: Improving performance. Geneva, Switzerland: World Health Organization.



Service users on interview panels in mental health.

Service user- and carer-reported measures of involvement in mental health care planning: methodological quality and acceptability to users.

Mental health service users' experiences of mental health care: an integrative literature review.

Service Users and Providers Expectations of Mental Health Care in Iran: A Qualitative Study.

Activity and views of service users involved in mental health research: UK survey.

Mental health and social service needs for mental health service users in Japan: a cross-sectional survey of client- and staff-perceived needs.

The experiences of detained mental health service users: issues of dignity in care.

Obstacles to continuity of care in young mental health service users' pathways - an explorative study.

The role of fear in mental health service users' experiences: a qualitative exploration.

Influences on uptake of cancer screening in mental health service users: a qualitative study.

Users' involvement in mental health services: programme logic model of an innovative initiative in integrated care.

'Recovery work' and 'magic' among long-term mental health service-users.

A qualitative study exploring facilitators for improved health behaviors and health behavior programs: mental health service users' perspectives.

More than just a bed: mental health service users' experiences of self-referral admission.

Facilitators and barriers to service user involvement in mental health guidelines: lessons from the Netherlands.

Active Involvement of End Users When Developing Web-Based Mental Health Interventions.

From admission to discharge in mental health services: a qualitative analysis of service user involvement.

Service user involvement in the coproduction of a mental health nursing metric: The Therapeutic Engagement Questionnaire.

Psychobiotics and their involvement in mental health.

Service user involvement in mental health care: an evolutionary concept analysis.

Seeking fluid possibility and solid ground: space and movement in mental health service users' experiences of 'crisis'.

Service user involvement for mental health system strengthening in India: a qualitative study.

The Mental Health Service in 1954.

The Mental Health Service in 1955.