nuance ana management

Consumer satisfaction surveys in mental health Tom Ricketts, Behavioural Psychotherapy Department, Sheffield Health Authority Treating clients of mental health services as consumers is a relatively new concept in Britain. This article seeks to explain the development of consumer satisfaction surveys, reviews the literature, and identifies ways in which the approach can be used to improve services.

T

he debate about what constitutes consumer satisfaction is ongoing (Thomas et al, 1991) and accounts for the wide diver­ sity of methodologies being used. In his re­ view of American consumer satisfaction lit­ erature, Lebow (1982) defines satisfaction as ‘the undifferentiated positive regard for treatment and outcome’ as measured by in­ struments such as the Client Satisfaction Questionnaire of Larsen et al (1979). This scale contains just eight questions. Each of the questions (e.g. how satisfied are you with the amount of help you received?) is scored using a four-point scale, giving an overall score of ‘global satisfaction’ — this concept is discussed later. Mangen and Griffith (1982) argue that the narrowness of view represented by this type of scale ignores the reality of the com­ plicated relationship that clients often have with mental health service staff, does not permit the expression of ambivalence, and has little utility in the evaluation of services and implementation of change. They sup­ port the use of a multi-method approach, and the inclusion of a range of items reflect­ ing on the specific qualities of the thera­ peutic relationship.

Early developments

Mr Ricketts is a trainee Behavioural Psychotherapist at the Behavioural Psychotherapy Department, Psychiatric Unit, N orthern General Hospital, Sheffield

The increasing utilization of satisfaction surveys among clients of mental health ser­ vices in Britain can be traced back to two main sources. The first was the increasingly powerful consumer movement in the USA during the 1970s that culminated in the 1975 Community Mental Health Centres Amendment (Title III of Public Law 94-63). This required mental health centres to perform broad-based evaluation of their treatment programmes in order to receive continued state funding. This evaluation came to include the views of consumers, most commonly through the use of con­ sumer satisfaction surveys. Lebow (1982), in a review of the literature, cited over 60 articles where the views of clients were

British Journal of Nursing, 1992, Voi l,N o 10

sought in this way. In Britain, consumerism in healthcare has been identified as a prior­ ity in the Government’s document The Pa­ tient’s Charter (Department of Health, 1991). The second source of consumer satisfac­ tion approaches in mental health literature can be traced back to moves to develop al­ ternatives to institutional care, culminating in Better Services for the Mentally III (De­ partment of Health and Social Security, 1975). Proposals included in this document were that district-based services centred on district general hospitals should replace the large, separate mental institutions. Raphael and Peers (1972) commissioned by the King’s Fund, surveyed 2000 inpatients in nine large psychiatric hospitals. Their re­ port proposed that evaluation of the ser­ vices by patients should become an integral part of the planning process.

Taking clients’ views seriously British literature relating to clients’ views of treatment is not extensive (Thomas et al, 1991) as the concept of people with mental health problems as consumers of services has met with some scepticism (Schulberg, 1981). This suggests that mental health pro­ fessionals have traditionally minimized the value of client satisfaction as an outcome measure. Godin et al (1987) identify three argu­ ments that have been forwarded to justify this minimization: first, because clients’ mental state is impaired at the point of treatment, they cannot be expected to judge the service they receive; second, a sophisti­ cated knowledge of treatment options in psychiatry is required before judgments can be made, and clients do not possess such knowledge; and third, the role of the client as consumer introduces a spurious supplyand-demand interpretation to the complex relationship between client and therapy agent. Proponents of consumer satisfaction re-

523

Downloaded from magonlinelibrary.com by 130.237.122.245 on January 16, 2019.

Consumer satisfaction surveys in mental health

i As dissatisfaction with services has been related to clients dropping out of treatment (Zastowny et al, 1989), some measure of satisfaction is needed to support efforts to reduce drop-out rates, and thus information can only come from the clients themselves. 5

524

search have attempted to deal with each of these arguments. Lebow (1982) argues that to assume that a distortion of view takes place is to overgeneralize and that occasion­ al distortions do not invalidate the unique perspective of the treatment process that clients have. Indeed, he states that there is evidence that distortions often arise from the therapist rather than from the client. Staff wishing to present services in a good light would be an obvious source of such distortions.

Discrepancy of view Nolan (1989), in a study comparing client and staff perceptions of individual client’s experiences, found a marked degree of dis­ crepancy of view. This study, which asked primary nurses to report what their clients’ experiences of mental health problems, treatment and outcome were, identified that even the staff member working most closely with a client may not fully under­ stand the client’s perspective. To argue that the nurses were right and the clients wrong in their perceptions would seem untenable. The view that sophisticated knowledge of treatment options is required before any judgment of treatment received can be made can be challenged on two points: first, different aspects of treatment regimes and different treatment settings are differential­ ly evaluated (Gould and Click, 1976, Mangen and Griffith, 1982); and second, even an unsophisticated view of treatment from the client is an irreplaceable source of information. As dissatisfaction with ser­ vices has been related to clients dropping out of treatment (Zastowny et al, 1989), some measure of satisfaction is needed to support efforts to reduce drop-out rates, and this information can only come from the clients themselves. Concern about the introduction of a spu­ rious supply-and-demand interpretation to the complex relationship between the client of mental health services and the services themselves receives some support from Mangen and Griffith (1982) and Godin ct al (1987). This issue becomes a problem when consumer satisfaction data are viewed as the primary measure of a treatment pro­ gramme. Garfield (1983) notes that some clients would express satisfaction whatever the nature of the service they received. The primary measure of mental health treatments is their effectiveness in resolving mental health problems and returning the client to effective functioning. Measures of consumer satisfaction add the view of the

client to that of the professional. An over­ reliance on consumer satisfaction data alone would be as unhelpful as ignoring clients’ views of treatment; however, services need to be developed that will actually be used by the target client group. In the absence of any accessible alterna­ tive sources of treatment, promoting satis­ faction with services aims to increase their appropriate utilization (Zastowny et al, 1989). Different approaches to the study of con­ sumer satisfaction with mental health treat­ ment tend to reflect the differing views held by the researchers as to what constitutes satisfaction. Unfortunately, many re­ searchers (Godin et al, 1987; Shields et al, 1988; Thomas et al, 1991; Sharma et al, 1992) fail to state explicitly what they mean by the term, and the reader is left to ascer­ tain this from reading the reports. The main approaches to the study of consumer satis­ faction can be grouped together under the headings short-form standardized ques­ tionnaire and the service-specific survey.

S hort-form standardized questionnaire In response to the view that consumer satis­ faction relates to ‘the undifferentiated posi­ tive regard for treatment and outcome’ (Lebow, 1982), the short-form standardiz­ ed questionnaire has been developed. Re­ searchers utilizing this approach have at­ tempted to develop questionnaires that are reliable, validated and able to be utilized in multiple-treatment centres. This enables quantitative analysis of results, and the comparison of satisfaction scores within the same treatment centre over time, through repeated measures, and between different treatment centres. In attempting to produce a consumer sat­ isfaction scale based on Lebow’s (1982) definition of satisfaction, researchers are at­ tempting to develop a unidimensional scale where all the items relate to a single concept — global satisfaction — but each identify different aspects of that concept. Many pitfalls in the development of sur­ vey and questionnaire approaches arc ident­ ified by Oppcnheim (1966), but unfortu­ nately, many of the studies utilizing the short-form standardized questionnaire ap­ proach appear to have ignored his useful guidelines. For the instrument to be judged as being reliable it needs to be tested in its final form for internal consistency and produce con­ sistent results on a test-retest procedure.

Journal of Nursing, 1992,16, Vol2019. l.N o 10 Downloaded from magonlinelibrary.comBritish by 130.237.122.245 on January

Consumer satisfaction surveys in mental health

c The relative powerlessness o f clients o f mental health services may lead to excessively positive responses (Garfield, 1983) especially i f issues of respondent anonymity and perceived repercussions for staff and services as a result o f the surveys are not addressed. 5

For the instrument to be judged valid, its content needs to be developed empirically with reference to the literature and expert opinion. The content should have been pi­ loted with the target group of respondents, focusing on the specific question of whether its content reflects issues that im­ pinge on the client group’s satisfaction with the service received. Redundant items will then be removed from the questionnaire be­ fore the final draft is repiloted.

C onstruct validity Ultimately, the short-form standardized questionnaire seeks construct validity (Oppenheim, 1966) where all items are agreed by other ‘experts’ to be measuring some aspect of the overall concept of con­ sumer satisfaction; in this case the experts will probably be senior mental health workers. A good example of the development of a short-form standardized questionnaire is that of Larsen et al (1979). Having reviewed the satisfaction literature to identify nine possible categories of determinants of satis­ faction, they generated nine sample items for each of the nine categories that panels of mental health professionals placed in rank order of relevance to the consumer satisfaction construct over two rounds. A preliminary scale was then presented to a pilot group of over 200 mental health cli­ ents. Based on statistical analysis of the rat­ ings of the pilot group of clients, unreliable items were excluded and the final scale pro­ duced and repiloted. Further statistical analysis showed the high internal consist­ ency of the final eight-item scale. Finally, the lack of correlation between the satisfac­ tion scale and measures of clinical improve­ ment were tested, with no correlation being found. Clearly, such large-scale projects are be­ yond the scope of many researchers; how­ ever, some British studies have not taken even minimal care in validating their ques­ tionnaires. Examples include studies of cli­ ent satisfaction with community psychiatric nursing services (Godin et al, 1987; Thomas et al, 1991) and studies of client satisfaction with inpatient services (McIntyre et al, 1989; Sharma et al, 1992). In all these stud­ ies, no information is given as to how the questionnaire content was determined or if it was piloted. Two further problems for the short-form standardized questionnaire are client sam­ pling and the consistently high levels of sat­ isfaction reported in the literature. Lcbow

British Journal of Nursing, 1992, Vol l.N olO

(1982) identifies two sources of sampling bias: selection of the clients contacted and poor response rates. Where categories of clients are excluded from the survey, the validity of the results is threatened. An example of this is the study by Godin et al (1987) where clients who had terminated their contract of care in negative circum­ stances were excluded from the sample. Making sense of the reported high level of satisfaction in this study is made impossible by such systematic bias. Poor response rate is a common problem for satisfaction surveys that utilize postal questionnaires. Lebow (1982) reports that response rates vary between 21% and 90%, and suggests that there is a difference be­ tween the views of respondents and non­ respondents. One study (Ellsworth, 1979) followed up both groups and found that while they were similar in post-treatment functioning, they differed in their satisfac­ tion with the service received. Researchers clearly need to pursue means by which the highest return rates can be achieved. Two suggested approaches are repeated re­ minders (Lebow, 1982) and token pay­ ments accompanying the postal satisfaction questionnaire.

Key flaw Parloff (1983) identifies the high levels of reported satisfaction with all forms of men­ tal health treatment as one of the key flaws of the standardized questionnaire approach. Unless some differentiation can be made between services and treatments, Parloff (1983) argues that the satisfaction survey is performing simply a public relations function. The relative powerlessness of clients of mental health services may lead to excess­ ively positive responses (Garfield, 1983) es­ pecially if issues of respondent anonymity and perceived repercussions for staff and services as a result of the surveys are not addressed. Lebow (1982) identifies the lack of con­ sistent use of a validated instrument over a certain period and between centres as con­ tributing to this problem. If researchers use different instruments, results are therefore not comparable, and their value is reduced. The adoption of a common, validated, short-form standardized questionnaire that is utilized repeatedly over time in a variety of treatment settings is the solution to this problem (Lebow, 1982). Satisfaction scores that stand alone are meaningless. Their change on repeated measurement may be

525

Downloaded from magonlinelibrary.com by 130.237.122.245 on January 16, 2019.

Consumer satisfaction surveys in mental health

( As staff teams develop alternative treatments to hospitalization, the first priority for research needs to be to ensure that the services being developed are actually effective in relieving diseases and promoting full functioning. f

a useful indicator of problems or improve­ ment over time.

Service-specific surveys The service-specific survey approach relates to the view that consumer satisfaction should be investigated in a way that allows individual services to respond to the spe­ cifics of the client’s perspective. This view has led to approaches that are more descrip­ tive and do not result in specific satisfaction scores. Questionnaires are developed to re­ late to the specific service being surveyed and contain many items that do not con­ form to the narrow definition of satisfac­ tion items identified by Lebow (1982). As service-specific surveys are developed to be relevant to a particular service, it is not possible to compare results with other services. One example of this approach was Shields et al’s (1988) study into clients’ views of treatment on a psychiatric ward. A total of 20 inpatients were interviewed using a semi-structured approach by a staff member not directly involved in their care. A content analysis procedure was then undertaken on these interviews. Based on this analysis, a 47-item questionnaire was developed; some of the questions asked for comments while others were multiplechoice with a 4-point scale. The questionnaire was then presented to 50 clients within 3 days of their discharge from the service. All replies were anony­ mous. This method generated descriptive information about clients’ experience of treatment, specifically related to environ­ mental factors, the communication of infor­ mation and the experience of boredom. The problem with the service-specific survey approach to client satisfaction is that each study will develop a unique instru­ ment for evaluating clients’ views. Lebow (1982) identifies this issue as rendering comparison between units impossible. The proponents of the service-specific survey approach argue that what is lost by the ap­ proach in terms of reliability, and the abil­ ity to replicate the approach elsewhere, is balanced by the much greater depth of in­ formation gained, and the empowering na­ ture of an approach that encourages clients to use their own words to describe their experience (Shields et al, 1988).

U nderstanding the client’s view of treatm ent The apparent conflict between the two ap­ proaches to the study of consumer satisfac­ tion with mental health treatment is in fact

526

spurious. The service-specific survey ap­ proach is a hypothesis-generating tool that produces detailed descriptive data on a small scale from which inferences cannot be drawn. The short-form standardized questionnaire has the advantages of being able to be utilized across treatment settings and repeatedly within the same treatment setting; in addition, it generates quantitative data that can be analysed statistically. Information from both approaches must be interpreted by staff and management, to­ gether with treatment outcome informa­ tion. The approaches are complementary and enable both the specific views of small populations of clients and the general views of large populations to be heard. The fact that much of the British consumer satisfac­ tion literature has been methodologically flawed does not detract from the usefulness of the approach. Mental health services in Britain are changing rapidly. There is a national policy emphasis on the development of commu­ nity mental health facilities (DHSS, 1989). As staff teams develop alternative treat­ ments to hospitalization, the first priority for research needs to be to ensure that the services being developed are actually effec­ tive in relieving distress and promoting full functioning. Consumer satisfaction is not an alternative to outcome research; how­ ever, the development of consumer satisfac­ tion approaches, of both the short-form standardized questionnaire and the servicespecific survey approach, may enable men­ tal health services to address such issues as treatment drop-out and treatment refusal (Zastowny et al, 1989). However, this will only occur if a more systematic approach to consumer satisfaction is adopted and the many pitfalls of the British studies to date are avoided.

Conclusion Consumer satisfaction approaches in Brit­ ain, despite a 20-year history, have failed to become a routine part of the evaluation of mental health services. Methodological problems can be identified in a majority of reported studies. All of these can be over­ come but researchers need to provide both a clear statement as to how they define con­ sumer satisfaction and a commitment to the continual striving for validity of the instru­ ments used. Where the regular use of outcome measures for mental health treatment have been instituted, in order to ensure that sat­ isfaction data is not taken as the only

British Journal of Nursing, 1992, Vol No 10 Downloaded from magonlinelibrary.com by 130.237.122.245 on January 16, 1, 2019.

Consumer satisfaction surveys in mental health

isfaction data is not taken as the only measure of a treatment resource, managers should be encouraged to utilize validated consumer satisfaction questionnaires across services, with repeated measures at regular intervals. Qualitative research into the de­ terminants of satisfaction is also required to enable services to identify strategies to raise satisfaction levels. jjflfcl

KEY POINTS •

Consumerism in healthcare, as proposed by the Government in The Patient’s Charter (DoH, 1991), implies that clients’ views of mental health services should be taken seriously.



Attempts to measure consumer satisfaction with mental health services in Britain remain haphazard and uncoordinated.



Consumer satisfaction data is not an alternative to outcome research, but can help services ensure that the targeted client group will actually utilize the services.



The repeated use of validated consumer satisfaction questionnaireswithinservicesenablesproblemsorimprovements to be identified.



Qualitative studies providing information about determinants of client satisfaction are also required to help develop strategies to increase satisfaction with services.

D oH (1991) The Patient's Charter. HMSO, London DHSS (1975) Better Services fo r the Mentally 111. HMSO, London DHSS (1989) Caring fo r People: Community Care in the N ext Decade and Beyond. HM SO, London Ellsworth R (1979) Does follow-up loss reflect poor outcomes? Evaluation and the Health Professions 2: 419-37 Garfield SL (1983) Some comments on consumer satis­ faction in behaviour therapy. Behav Ther 14: 237-41 Godin P, Pearce I, Wilson I (1987) Keeping the cus­ tomer satisfied. Nurs Times 83(38): 35-7 Gould E, Glick ID (1976) Patient-staff judgments of treatment program helpfulness on a psychiatric ward. Br J Med Psychol 49: 23-33 Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD (1979) Assessment of client/patient satisfac­ tion: development of a general scale. Evaluation and Program Planning 2: 197-207 Lebow J (1982) Consumer satisfaction with mental health treatment. Psychol Bull 91: 244-59 McIntyre K, Farrell M, David AS (1989) What do psy­ chiatric inpatients really want? Br Med J 298: 159-60 Mangen SP, Griffith JH (1982) Patient satisfaction with community psychiatric nursing: a prospective controlled study. J A dv Nurs 7: 477—82 Nolan P (1989) Face value. Nurs Times 85 (35): 62-5 Oppenheim A N (1966) Questionnaire Design and A t­ titude Measurement. Heinemann, London Parloff MB (1983) Who will be satisfied by ‘Consumer Satisfaction’ evidence? Behav Ther 14: 242-6 Raphael W, Peers V (1972) Psychiatric Hospitals View­ ed by their Patients. King’s Fund, London Schulberg H C (1981) Outcome evaluations in the mental health field. Community Ment Health J 17(2): 132-142 Sharma T, Carson J, Berry C (1992) Patient voices. Health Serv J 102: 20-1 Shields PJ, Morrison P, H art D (1988) Consumer sat­ isfaction on a psychiatric ward. J A d v Nurs 13: 396-400 Thomas B, Muijen M, Brooking J (1991) Reactions to a new service. Nursing 4 (29): 9-11 Zastowny TR, Roghmann KJ, Cafferata GL (1989) Pa­ tient satisfaction and the use of health services. Med Care 27: 505-23

MAKE SURE OF YOUR COPIES OF BJN If you are having trouble obtaining a copy of British Journal of Nursing from your local newsagent then please get in touch with us immediately and we will ensure that your copy of BJN keeps on coming. Now post this form to: The Publisher BJN, Croxted Mews, 288 Croxted Road, London SE24 9DA. (Tel: 081-671 7521).

N A M E..... ADDRESS

SPECIALTY................................................................................. .TEL No NAME AND ADDRESS OF LOCAL NEWSAGENT..........................

British Journal ol Nursing. 1992, Voi l,N o 10

527

Downloaded from magonlinelibrary.com by 130.237.122.245 on January 16, 2019.

Consumer satisfaction surveys in mental health.

Treating clients of mental health services as consumers is a relatively new concept in Britain. This article seeks to explain the development of consu...
417KB Sizes 0 Downloads 0 Views