Journal of Mental Health
ISSN: 0963-8237 (Print) 1360-0567 (Online) Journal homepage: http://www.tandfonline.com/loi/ijmh20
Lessons from an evaluation of major change in adult mental health services: effects on quality Shanaya Rathod, Avril Lloyd, Carolyn Asher, Jessamy Baird, Enilson Mateus, Eva Cyhlarova, Julie Cameron, Jenny Lieshman, Hesham Elnazer & David Kingdon To cite this article: Shanaya Rathod, Avril Lloyd, Carolyn Asher, Jessamy Baird, Enilson Mateus, Eva Cyhlarova, Julie Cameron, Jenny Lieshman, Hesham Elnazer & David Kingdon (2014) Lessons from an evaluation of major change in adult mental health services: effects on quality, Journal of Mental Health, 23:5, 271-275 To link to this article: http://dx.doi.org/10.3109/09638237.2014.951487
Published online: 15 Sep 2014.
Submit your article to this journal
Article views: 140
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=ijmh20 Download by: [Deakin University Library]
Date: 11 November 2015, At: 11:37
http://informahealthcare.com/jmh ISSN: 0963-8237 (print), 1360-0567 (electronic) J Ment Health, 2014; 23(5): 271–275 ! 2014 Shadowfax Publishing and Informa UK Limited. DOI: 10.3109/09638237.2014.951487
RESEARCH AND EVALUATION
Lessons from an evaluation of major change in adult mental health services: effects on quality Shanaya Rathod1, Avril Lloyd1, Carolyn Asher1, Jessamy Baird1, Enilson Mateus1, Eva Cyhlarova2, Julie Cameron2, Jenny Lieshman3, Hesham Elnazer1, and David Kingdon4* Southern Health NHS Foundation Trust in collaboration with the Princess Royal Trust for Carers, Southampton, UK, 2Mental Health Foundation, London, UK, 3Independent Statistician, New Castle, UK, and 4Southern Health NHS Foundation Trust, University of Southampton, Southampton, UK
Downloaded by [Deakin University Library] at 11:37 11 November 2015
1
Abstract
Keywords
Background: The move from inpatient to community services in mental health has sparked debate internationally but the evidence base for successful service models is sparse. Aim: To evaluate the impact of bed reduction on quality of services when accompanied by redesign of community services. Methods: Qualitative and quantitative data were collected 6 months before and 6 months after the redesign was implemented. Results: Bed numbers reduced by 35%. Number of admissions and occupied bed days (OBD) reduced; bed occupancy and proportion of detained patients increased. Access to community services improved but quality of assessments did not. Transitions across pathways were rated as difficult by clinicians. There was an overall reduction in staff numbers; staff sickness levels and dissatisfaction with working conditions increased. Service users were generally positive about the redesign but GPs and staff were not. Conclusions: Multi-faceted evaluation of change in cost-pressured services is feasible and should guide developments to minimise negative effects on quality of care.
Change, community care, evaluation, mental health beds
Background Over the course of the last half century, across US, Europe and Australasia, the focus of care of severely mentally ill has moved from institutions to community. In the US, state hospital beds have reduced from 339/100 000 in 1955 to 22/100 000 in 2002 (Manderscheid et al., 2002) with continuing decreases since. Similarly in the UK, there has been a steady reduction from a maximum of 155 000 beds in 1954 to 27 000 in 2008 with a national plan to reduce further (Tyrer, 2011). However, simply moving the setting of care is not enough as move of care in the community requires well-resourced community services. Across a number of different countries, specialist community teams have been set up, e.g. Early Intervention in Psychosis (EIP), Assertive Outreach Teams (AOT) in Adult Mental Health (AMH) that have been debated (Kingdon, 2011). Netherlands developed the Flexible Assertive Community Treatment (FACT) model of care which is a European variant on the original Assertive community treatment (ACT). The variety of need is met by providing two distinct levels of service within a single team: high *Current address: Eli Lilly, UK. Correspondence: Dr. Shanaya Rathod, Grove House, Ocean Way, Meridian Cross, Southampton SO14 3TJ, UK. Tel: 02380821230. E-mail: [email protected]
History Received 17 May 2014 Accepted 20 June 2014 Published online 15 September 2014
intensity – following the classic assertive outreach shared caseload approach, the other offering low intensity – more like individual case management. Service users move easily between these levels depending on need, but the staff group remains the same, ensuring continuity of care (Van Veldhuizen, 2007). However, there are few evaluations of models of care in literature and limited success with large-scale change in services (Gilbert et al., 2014). There is a need for planning of services based on evidence, patients’ needs, as well as better integration of services (Pedersen & Kolstad, 2009).
Context of evaluation In Hampshire (UK), Southern Health NHS Foundation Trust serves a population of approximately 1.3 million people. AMH services are delivered through four management units or areas (north, south, east and west) that included six inpatient units with 165 acute beds. The organisation developed a service strategy for 3 years (2010–13) to provide effective services within predicted budgetary constraints. This included a reduction in inpatient beds and units with establishment of new structural components in the community: (1) Access and Assessment Teams (AAT) as a single point of access (SPOA) in each area to provide a ‘‘core assessment’’, which would act as a ‘‘passport’’ around the
272
S. Rathod et al.
system of care, thereby reducing duplication and improving productivity. (2) Integrated Community Treatment Teams (CTT) to ensure that people received timely help. The integrated teams would provide the AOT and EIP function except in one (South) of the four areas where dedicated AOT and EIP teams were retained due to the diversity and morbidity of population. (3) In another area (West), the FACT model was adopted as a pilot project and was named ‘‘the shared care model’’. (4) Acute Care Teams (ACT) – consisting of fewer inpatient units and a strengthened hospital at home model.
Downloaded by [Deakin University Library] at 11:37 11 November 2015
Aim To evaluate the impact of bed reduction and redesigning of community services on quality of care for service users, carers, staff and general practitioners (GPs). Research questions (1) Do the centralised assessment processes lead to improved quality of access and assessment to services? (2) How does integration of EIP and AOT function into community teams impact on care delivery? (3) What is the impact of bed reduction on bed usage, admissions and length of stay? (4) What is the impact on the workforce?
Evaluation methodology This is a descriptive evaluation that used mixed qualitative and quantitative methodology. Findings from the qualitative research were used to triangulate, cross-validate and enrich the results of quantitative research. Data were collected at 6 months before and 6 months post-redesign. Information from the immediate 6 months post-redesign was not collected in order to allow the new teams to form and stabilise.
Evaluation plan Information sources (A) Information on referrals, inpatient data, Critical incident reviews (pre-redesign – 27; post-redesign – 22), workforce was collected from: 1 October 2011 to 31 December 2011 and October 2012 to December 2012 allowing a 3-month follow-up period until 31 March to ensure completeness of data. (B) Patient, Staff and GP satisfaction surveys: Anonymised questionnaires were available through Survey Monkey and printed copies were made available at every team base. Self-addressed envelopes were provided and responses sent to an independent site in the Research and Development (R&D) Department. Analysis of anonymised data was conducted by an independent statistician. The survey was carried out in the first week of December 2012 – identified as ‘‘Census week’’. All service users, staff and GPs connected to the service during this week in AMH could access the questionnaire supported with an information sheet about the evaluation.
J Ment Health, 2014; 23(5): 271–275
(C) Audit of quality of assessments: Randomly selected retrospective EPR audit by senior medical staff was conducted for a period in November 2011 and November 2012 to capture the before and after assessments. The audit tool was devised based on the Royal College of Psychiatrists recommendations of a standard psychiatric and risk assessment (1996). Other sources, e.g. the organisation’s risk policy contributed to the audit tool. Senior medical staff was trained to use the tool in order to increase consistency and reliability across raters and were blind to the time period. 124 assessment audits were completed: 60 pre-redesign and 64 post-redesign. (D) Audit of randomly selected sample from AOT and EIP: Retrospective EPR audit by senior staff was conducted for a period in November 2011 and November 2012 to capture the before and after audit. The audit tool was devised based on commissioned contract. Raters trained for using the tool in order to maintain inter-rater reliability. Records audited: AOT – 68: 41 pre-redesign and 27 post-redesign EIP – 77: 48 pre-redesign and 29 post-redesign. (E) Qualitative interviews with service users (SU), staff, and focus groups with carers: Service users (SU): As this was an evaluation of services before and after the redesign, SU who had used the service prior to April 2012 and after September 2012 were identified through the EPR. They received information sheets from their key workers who assessed their mental capacity to consent and invited them to participate in the interviews. Only where explicit consent was obtained and mental capacity demonstrated and documented, were service users approached for interview. All information obtained was anonymised: person identifiable information was not collected. If at any stage of the interview it was felt that the interview was distressing, it was stopped immediately and the keyworker informed, although this did not occur. The interviews solely focussed on the redesign. The interviewers were trained trust staff including service user staff. Carers: four focus groups – one in each AMH area were facilitated by the Princess Royal Trust for Carers. Information sheets were provided and written informed consent was documented by the facilitators. All information was anonymised. Staff interviews: Information sheets were made available with the questionnaires. Only staff that volunteered and gave written informed consent to be interviewed were contacted. All information remained anonymous. Analysis plan Quantitative data including differences in the before and after cohort were analysed using a variety of methods depending on the distribution of the variable in question. This included confidence interval analysis and descriptive statistics where appropriate. SPSS 18 (Chicago, IL) was used. Qualitative data were transcribed. NVivo (Qualitative Research Solutions) was used for analysis using deductive and inductive analytical approaches to lead to key themes.
Evaluation of a service redesign
DOI: 10.3109/09638237.2014.951487
Ethics The project underwent a proportionate review by the ethics committee (Ref: 12/SW/035). Data were handled by senior trust staff who are CRB checked, complied with the data protection act and Information Governance policy of the trust. All data were anonymised for analysis.
Downloaded by [Deakin University Library] at 11:37 11 November 2015
Results (1) Surveys SU surveys: 992 SU attended appointments in AMH services in the survey week. A total of 482 surveys were returned. A response rate of 48.5% achieved. Among SU who had used the services before, 31.2% respondents were receiving care from CTT; whereas among those who had not used services prior to the redesign 34.1% respondents were cared for by the AAT. Staff surveys: 295 responses received. This is a response rate of 18.44%. GP surveys: 166 responses were received and 146 were included in the analysis as 20 responses were left blank in most questions. Majority of GPs who responded had been practicing for more than 10 years (106; 72.5%). (2) Carer Focus Groups: 29 participants were interviewed through the carer focus groups. (3) Interviews: Clinician interviews: 26 clinicians were interviewed for the study. This was over half of those contacted (n¼43) with two unable to find time to meet and 15 withdrawing after initial interest (reasons for this unknown). Service user interviews: 84 names were provided to contact but only 3 interviews were completed.
Key findings Access and assessment Access Existing SU, clinicians and GPs answered questions relating to accessing the service and movement throughout the service. The results are shown in Table 1. AAT and impact on access: SU were significantly more likely to think access was better or the same than either clinicians (95% CI – 11.98% to 29.48%) or GPs (95% CI – 8.13% to 29.35%). 24/7 access: SU were significantly more likely to think overall access was better or the same than either clinicians (95% CI 8.02% to 29.32%) or GPs (95% CI 3.92% to 32.12%). Time from referral to assessment: SU were significantly more likely to think that time from referral to assessment was better than both clinicians (95% CI 26.17% to 45.67%) and GPs (95%CI 23.63% to 46.12%). Compared to GPs, a significantly higher proportion of clinicians thought that the time from assessment to being assigned a care coordinator was worse since the redesign (95% CI 11.58% to 35.12%). SU were significantly more likely to think that transition from AAT to CTT was better after the redesign than both
273
Table 1. Survey – access and transitions. Better N Single point of access Clinician, N ¼ 214 66 GP, N ¼ 113 35 SU Green, N ¼ 190 67 24/7 access Clinician, N ¼ 207 61 GP, N ¼ 67 19 SU Green, N ¼ 84 35 Referral to assessment Clinician, N ¼ 165 47 GP, N ¼ 99 29 SU Green, N ¼ 177 115 Assessment to care coordinator Clinician, N ¼ 190 19 GP, N ¼ 95 14 Transitions between teams SU Green, N ¼ 167 139 AAT-CTT Clinician, N ¼ 183 20 GP, N ¼ 92 6 SU Green, N ¼ 178 50 CTT-ACT Clinician, N ¼ 178 20 GP, N ¼ 83 15
Same
Worse
%
N
%
N
%
30.8 31.0 35.3
58 33 83
27.1 29.2 43.7
90 45 40
42.1 39.8 21.0
29.5 28.4 41.7
66 23 33
31.9 34.3 39.3
80 25 16
38.6 37.3 19.0
28.5 29.3 65.0
52 34 31
31.5 34.3 17.5
66 36 31
40 36.4 17.5
10 14.7
38 37
20 38.9
133 44
70 46.3
65.9
39
23.5
18
10.8
10.9 6.5 28.1
45 38 89
24.6 41.3 50.0
118 48 39
64.5 52.2 21.9
11.2 18.1
70 33
39.3 39.8
88 35
49.4 42.2
clinicians (95% CI 9.06% to 25.11%) and GPs (95% CI 12.3% to 29.4%). Qualitative data added that the SPOA is seen as an advantage especially for new users. Initial access is generally timely and perceived to be better. Assessment
Compared to clinicians, a significantly lower proportion of SU thought that the focus on strengths and skills in assessments was worse post-redesign (95% CI 7.08% to 23.34%). Compared to clinicians, a significantly lower proportion of SU noted that holistic focus of assessments was worse post-redesign (95% CI 7.08% to 24.34%). Compared to clinicians, a significantly lower proportion of SU noted that the focus on goals of care in assessments was worse post-redesign (95% CI 12.75% to 29.51%). Audit found that the quality of mental health history including mental state, formulation or diagnosis was variable but not improved post-redesign. There was less evidence of balanced risk assessment in the formulation post-redesign. There was some evidence of a social approach and SU involvement in the care plan post-redesign. Reassessment A total of 47.5% of SU who had accessed the services before and after the redesign and 75.9% of SU who had not accessed services before the redesign reported that they were reassessed. Audit and qualitative data supported that reassessments had increased.
274
S. Rathod et al.
J Ment Health, 2014; 23(5): 271–275
Impact on EIP and AOT
morbidity on the ward was perceived to have increased with swifter discharge.
EIP Compared to clinicians, a significantly higher proportion of GPs thought that assessment by EIP had improved access to the AMH service (95% CI 8.6% to 32.3%). The audit and qualitative data supported the finding that EIP function had been diluted following integration into the CTT. Of the four areas, the South had retained a dedicated EIP team. The evaluation showed different population needs and having maintained a dedicated EIP showed better service provision (Table 2).
Staff in AMH reduced by 7.2% from 1724 to 1599 staff (including bank staff). Excluding bank staff, head count reduced (10.1%) from an average of 1431.5 in 2011 to 1287 in 2012. Sick days increased by 381 (7.8%) between the two periods. Staff reported a reduction in non-clinical resources (Table 3).
Discussion
AOT
Downloaded by [Deakin University Library] at 11:37 11 November 2015
Impact of the re-design on workforce
AOT function was not diluted through integration into CTT. Audit identified areas of development although this was consistent before and after the redesign. South population showed different needs due to diversity and morbidity and justified the need for a specialist team. The shared care model (West) based on the FACT model (Van Veldhuizen, 2007) was reported as positive for SU’s. Bed usage
As a result of the redesign, bed numbers reduced from 165 to 107 (35%). The number of admissions to inpatients reduced, with an associated decrease in OBD. Bed occupancy increased following the re-design. There was an increase in proportion of detained patients (95% CI 5.35% to 20.45%) compared to informal admissions between the two time periods. The Health of the Nation Outcome Scales (HoNOS; Wing et al., 1995) scores for admissions were higher for agitation, lower for suicidality, higher for psychosis and higher for social scores post-redesign. SU who had used the service before were more likely than either clinicians (95%CI 11.5% to 37.7%) or GPs (95% CI 3.8% to 23.7%) to state that the clarity of crisis pathways had improved since the redesign. Significantly more clinicians than GPs reported that timely discharge had improved since the redesign (95% CI 3.1% to 19.3%). Significantly more GPs than clinicians reported that the location of inpatient services had improved (95% CI 13.5% to 29.0%). There were no significant differences between those service users who had accessed the service before and those who had not in terms of their views on inpatient services or the hospital at home service. Qualitative data supported home treatment although perception was that reduction in beds had made access to beds more difficult and the severity of admission criteria and Table 2. Survey results on EIP. Better
Same
Worse
Access to EIP
N
%
N
%
N
%
Clinician, N ¼ 141 GP, N ¼ 67
14 20
9.9 29.9
69 31
48.9 46.3
58 16
41.2 23.8
The Royal College of Psychiatrists (2010a), WHO (2007) and Naylor & Bell (2010) highlighted the potential to improve quality and productivity in mental health services. Across the world initiatives have been taken in community services but there are few evaluations of effectiveness or efficiency (Pedersen & Kolstad, 2009). In this study, the number of beds, hence admissions to inpatients reduced, with an associated decrease in the number of OBD but bed occupancy increased following the re-design. This is consistent with the trends to move to community care and demonstrates increased efficiency in the use of inpatient care. The proportion of detained patients increased as could be expected (Keown et al., 2011) and consistent with the national picture (NHS Information centre, 2011/12). While the respondents indicated that access improved through the new teams, the quality of assessment had not improved and number of re-assessments did not reduce. There was concern about difficult transitions and confusion around different roles and teams, e.g. for emergency assessments, and at the interface between them and CTT. Therefore, it was difficult to establish productivity gains. The decision to base integration of the AOT function on local population need was found to be appropriate (Killaspy et al., 2009). Implementation of the FACT model showed early success (Firn et al., 2013). Integration of EIP teams into community teams in Hampshire diluted the EI function. Retention of a distinct team in South preserved this. Literature (McCrone et al., 2010; Stafford et al., 2013) has determined the effectiveness of EIP teams. There is a case for considering the psychosis clinical pathway and extending the benefits of this service,
Table 3. Staff report on non-clinical resources. Improved
Availability of admin support, n ¼ 145 Ability to access information/ records, n ¼ 152 Technology/ access to networks, n ¼ 148 Sufficient office space, n ¼ 152 Location of service for staff, n ¼ 139
Same
Worse
N
%
N
%
N
%
10
6.9
76
52.4
59
40.7
15
9.9
102
67.1
35
23.0
12
8.1
92
62.2
44
29.7
8 6
5.3 4.3
81 77
53.6 55.4
63 56
41.1 40.3
Evaluation of a service redesign
Downloaded by [Deakin University Library] at 11:37 11 November 2015
DOI: 10.3109/09638237.2014.951487
especially in light of the Schizophrenia commission report (Rethink, 2012). One of the important findings from the study was that staff did not rate the redesign as positively as service users. The process of the redesign was perceived as difficult and working conditions, e.g. office accommodation, technology and travelling time, as having deteriorated. Nonmedical staff perception that their caseloads increased may have led to the increased levels of stress and anxiety in staff in terms of whether they could deliver quality care. Staff sickness increased. In the current volatile economic context, change is constant and therefore engaging clinicians is particularly important as the costs of not doing so can be significant. Poorly managed change impacts on quality of care at many levels (Gilbert et al., 2014). Productivity and efficiency can be increased through innovation and integration in clinical services only when needs of staff like safety and belonging are met (Maslow, 1943). The relatively negative response of GPs indicated that better engagement and communication was needed with them. A key learning point from the evaluation was that when implementing large scale change it is important to ensure that the infrastructure for delivering the change is robust (Gilbert et al., 2014). This includes attention to the evidence-base, engagement of key stakeholders and support from corporate partners. Key areas for consideration when implementing large scale change are the need to ensure that service user experience is improved (Francis, 2013). Following a large scale change, benefits can be slow to become apparent but effective evaluation, as in this instance, can be achieved using limited service resources. The service has identified key areas of improvement from this project. Limitations The evaluation project had its limitations. Only three SU were interviewed although considerable time was taken to engage them. The study used a before-and-after methodology. The results only considered a 3-month period from 6 months before and after the redesign. Thus many of the changes were likely to still be ‘‘bedding in’’; including staff getting used to new premises, working arrangements, etc. alongside SU adapting to the changes.
Conclusions Services across the globe are undergoing extensive and challenging change programmes in a climate of on-going austerity. There is a need for a culture of continuous multimethod evaluation and quality improvement with SU experience at the centre of decision making, incorporating learning from services nationally and internationally.
275
Declaration of interest This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
References Firn M, Hindhaugh K, Hubbeling D, et al. (2013). A dismantling study of assertive outreach services: Comparing activity and outcomes following replacement with the FACT model. Soc Psychiatr Psychiatr Epidemiol, 48, 997–1003. Francis R. (2013). Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry. Available at: www.midstaffspublicinquiry.com/ report. Gilbert H, Peck E, Ashton B, et al. (2014). Service transformation lessons from mental health. London: The Kings Fund. Keown P, Scott W, Bhui K, Scott J. (2011). Association between provision of mental illness beds and rate of involuntary admissions in the NHS in England 1988–2008: Ecological study. BMJ (Clinical Research Edition), 343, d3736. Killaspy H, Kingett S, Bebbington P, et al. (2009). Randomised evaluation of assertive community treatment: 3-year outcomes. Br J Psychiatr, 195, 81–2. Kingdon D. (2011). Transforming mental health services. Br J Psychiatr, 199, 1–2. Manderscheid RW, Atay JE, Male A. (2002). Highlights of organized mental health services in 2000 and major national and state trends, in Mental Health US. Washington, DC: Center for Mental Health Services. Maslow AH. (1943). A theory of human motivation. Psychol Rev, 50, 370–96. McCrone P, Craig TK, Power P, Garety PA. (2010). Cost-effectiveness of an early intervention service for people with psychosis. Br J Psychiatr, 196, 377–82. Naylor C, Bell A. (2010). Mental health and the productivity challenge London: The Kings Fund. NHS Information centre. (2011/12). Inpatients formally detained in hospitals under the Mental Health Act 1983, and patients subject to supervised community treatment, Annual figures, England. Pedersen P, Kolstad A. (2009). De-institutionalisation and transinstitutionalisation – Changing trends of inpatient care in Norwegian mental health institutions 1950–2007. Int J Ment Health Syst, 3, 28. Rethink. (2012). The abandoned illness. An independent report from the Schizophrenia commission. Royal College of Psychiatrists. (1996). Assessment and clinical management of risk of harm to other people. Council Report CR 53. London. Royal College of Psychiatrists. (2010a). Looking Ahead. Future development of UK mental health services: Recommendations from a Royal College of Psychiatrists’ enquiry. Occasional Paper OP75. London. Stafford M, Jackson H, Mayo-Wilson E, et al. (2013). Early interventions to prevent psychosis: Systematic review and meta-analysis. Br Med J, 346, f185. Tyrer P. (2011). Has the closure of Psychiatric beds gone too far? Br Med J, 343, d7457. Van Veldhuizen. (2007). The Flexible Assertive Community Treatment model (FACT). Commun Mental Health J, 43(4), 421–33. WHO. (2007). Expert opinion on barriers and facilitating factors for the implementation of existing mental health knowledge in mental health services. Geneva: World Health Organization. Wing J, Beaver AS, Curtis RH, et al. (1995). HoNOS: Health of the nation outcome scales. London: Royal College of Psychiatrists’ Research Unit Report.
ISSN: 0963-8237 (Print) 1360-0567 (Online) Journal homepage: http://www.tandfonline.com/loi/ijmh20
Lessons from an evaluation of major change in adult mental health services: effects on quality Shanaya Rathod, Avril Lloyd, Carolyn Asher, Jessamy Baird, Enilson Mateus, Eva Cyhlarova, Julie Cameron, Jenny Lieshman, Hesham Elnazer & David Kingdon To cite this article: Shanaya Rathod, Avril Lloyd, Carolyn Asher, Jessamy Baird, Enilson Mateus, Eva Cyhlarova, Julie Cameron, Jenny Lieshman, Hesham Elnazer & David Kingdon (2014) Lessons from an evaluation of major change in adult mental health services: effects on quality, Journal of Mental Health, 23:5, 271-275 To link to this article: http://dx.doi.org/10.3109/09638237.2014.951487
Published online: 15 Sep 2014.
Submit your article to this journal
Article views: 140
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=ijmh20 Download by: [Deakin University Library]
Date: 11 November 2015, At: 11:37
http://informahealthcare.com/jmh ISSN: 0963-8237 (print), 1360-0567 (electronic) J Ment Health, 2014; 23(5): 271–275 ! 2014 Shadowfax Publishing and Informa UK Limited. DOI: 10.3109/09638237.2014.951487
RESEARCH AND EVALUATION
Lessons from an evaluation of major change in adult mental health services: effects on quality Shanaya Rathod1, Avril Lloyd1, Carolyn Asher1, Jessamy Baird1, Enilson Mateus1, Eva Cyhlarova2, Julie Cameron2, Jenny Lieshman3, Hesham Elnazer1, and David Kingdon4* Southern Health NHS Foundation Trust in collaboration with the Princess Royal Trust for Carers, Southampton, UK, 2Mental Health Foundation, London, UK, 3Independent Statistician, New Castle, UK, and 4Southern Health NHS Foundation Trust, University of Southampton, Southampton, UK
Downloaded by [Deakin University Library] at 11:37 11 November 2015
1
Abstract
Keywords
Background: The move from inpatient to community services in mental health has sparked debate internationally but the evidence base for successful service models is sparse. Aim: To evaluate the impact of bed reduction on quality of services when accompanied by redesign of community services. Methods: Qualitative and quantitative data were collected 6 months before and 6 months after the redesign was implemented. Results: Bed numbers reduced by 35%. Number of admissions and occupied bed days (OBD) reduced; bed occupancy and proportion of detained patients increased. Access to community services improved but quality of assessments did not. Transitions across pathways were rated as difficult by clinicians. There was an overall reduction in staff numbers; staff sickness levels and dissatisfaction with working conditions increased. Service users were generally positive about the redesign but GPs and staff were not. Conclusions: Multi-faceted evaluation of change in cost-pressured services is feasible and should guide developments to minimise negative effects on quality of care.
Change, community care, evaluation, mental health beds
Background Over the course of the last half century, across US, Europe and Australasia, the focus of care of severely mentally ill has moved from institutions to community. In the US, state hospital beds have reduced from 339/100 000 in 1955 to 22/100 000 in 2002 (Manderscheid et al., 2002) with continuing decreases since. Similarly in the UK, there has been a steady reduction from a maximum of 155 000 beds in 1954 to 27 000 in 2008 with a national plan to reduce further (Tyrer, 2011). However, simply moving the setting of care is not enough as move of care in the community requires well-resourced community services. Across a number of different countries, specialist community teams have been set up, e.g. Early Intervention in Psychosis (EIP), Assertive Outreach Teams (AOT) in Adult Mental Health (AMH) that have been debated (Kingdon, 2011). Netherlands developed the Flexible Assertive Community Treatment (FACT) model of care which is a European variant on the original Assertive community treatment (ACT). The variety of need is met by providing two distinct levels of service within a single team: high *Current address: Eli Lilly, UK. Correspondence: Dr. Shanaya Rathod, Grove House, Ocean Way, Meridian Cross, Southampton SO14 3TJ, UK. Tel: 02380821230. E-mail: [email protected]
History Received 17 May 2014 Accepted 20 June 2014 Published online 15 September 2014
intensity – following the classic assertive outreach shared caseload approach, the other offering low intensity – more like individual case management. Service users move easily between these levels depending on need, but the staff group remains the same, ensuring continuity of care (Van Veldhuizen, 2007). However, there are few evaluations of models of care in literature and limited success with large-scale change in services (Gilbert et al., 2014). There is a need for planning of services based on evidence, patients’ needs, as well as better integration of services (Pedersen & Kolstad, 2009).
Context of evaluation In Hampshire (UK), Southern Health NHS Foundation Trust serves a population of approximately 1.3 million people. AMH services are delivered through four management units or areas (north, south, east and west) that included six inpatient units with 165 acute beds. The organisation developed a service strategy for 3 years (2010–13) to provide effective services within predicted budgetary constraints. This included a reduction in inpatient beds and units with establishment of new structural components in the community: (1) Access and Assessment Teams (AAT) as a single point of access (SPOA) in each area to provide a ‘‘core assessment’’, which would act as a ‘‘passport’’ around the
272
S. Rathod et al.
system of care, thereby reducing duplication and improving productivity. (2) Integrated Community Treatment Teams (CTT) to ensure that people received timely help. The integrated teams would provide the AOT and EIP function except in one (South) of the four areas where dedicated AOT and EIP teams were retained due to the diversity and morbidity of population. (3) In another area (West), the FACT model was adopted as a pilot project and was named ‘‘the shared care model’’. (4) Acute Care Teams (ACT) – consisting of fewer inpatient units and a strengthened hospital at home model.
Downloaded by [Deakin University Library] at 11:37 11 November 2015
Aim To evaluate the impact of bed reduction and redesigning of community services on quality of care for service users, carers, staff and general practitioners (GPs). Research questions (1) Do the centralised assessment processes lead to improved quality of access and assessment to services? (2) How does integration of EIP and AOT function into community teams impact on care delivery? (3) What is the impact of bed reduction on bed usage, admissions and length of stay? (4) What is the impact on the workforce?
Evaluation methodology This is a descriptive evaluation that used mixed qualitative and quantitative methodology. Findings from the qualitative research were used to triangulate, cross-validate and enrich the results of quantitative research. Data were collected at 6 months before and 6 months post-redesign. Information from the immediate 6 months post-redesign was not collected in order to allow the new teams to form and stabilise.
Evaluation plan Information sources (A) Information on referrals, inpatient data, Critical incident reviews (pre-redesign – 27; post-redesign – 22), workforce was collected from: 1 October 2011 to 31 December 2011 and October 2012 to December 2012 allowing a 3-month follow-up period until 31 March to ensure completeness of data. (B) Patient, Staff and GP satisfaction surveys: Anonymised questionnaires were available through Survey Monkey and printed copies were made available at every team base. Self-addressed envelopes were provided and responses sent to an independent site in the Research and Development (R&D) Department. Analysis of anonymised data was conducted by an independent statistician. The survey was carried out in the first week of December 2012 – identified as ‘‘Census week’’. All service users, staff and GPs connected to the service during this week in AMH could access the questionnaire supported with an information sheet about the evaluation.
J Ment Health, 2014; 23(5): 271–275
(C) Audit of quality of assessments: Randomly selected retrospective EPR audit by senior medical staff was conducted for a period in November 2011 and November 2012 to capture the before and after assessments. The audit tool was devised based on the Royal College of Psychiatrists recommendations of a standard psychiatric and risk assessment (1996). Other sources, e.g. the organisation’s risk policy contributed to the audit tool. Senior medical staff was trained to use the tool in order to increase consistency and reliability across raters and were blind to the time period. 124 assessment audits were completed: 60 pre-redesign and 64 post-redesign. (D) Audit of randomly selected sample from AOT and EIP: Retrospective EPR audit by senior staff was conducted for a period in November 2011 and November 2012 to capture the before and after audit. The audit tool was devised based on commissioned contract. Raters trained for using the tool in order to maintain inter-rater reliability. Records audited: AOT – 68: 41 pre-redesign and 27 post-redesign EIP – 77: 48 pre-redesign and 29 post-redesign. (E) Qualitative interviews with service users (SU), staff, and focus groups with carers: Service users (SU): As this was an evaluation of services before and after the redesign, SU who had used the service prior to April 2012 and after September 2012 were identified through the EPR. They received information sheets from their key workers who assessed their mental capacity to consent and invited them to participate in the interviews. Only where explicit consent was obtained and mental capacity demonstrated and documented, were service users approached for interview. All information obtained was anonymised: person identifiable information was not collected. If at any stage of the interview it was felt that the interview was distressing, it was stopped immediately and the keyworker informed, although this did not occur. The interviews solely focussed on the redesign. The interviewers were trained trust staff including service user staff. Carers: four focus groups – one in each AMH area were facilitated by the Princess Royal Trust for Carers. Information sheets were provided and written informed consent was documented by the facilitators. All information was anonymised. Staff interviews: Information sheets were made available with the questionnaires. Only staff that volunteered and gave written informed consent to be interviewed were contacted. All information remained anonymous. Analysis plan Quantitative data including differences in the before and after cohort were analysed using a variety of methods depending on the distribution of the variable in question. This included confidence interval analysis and descriptive statistics where appropriate. SPSS 18 (Chicago, IL) was used. Qualitative data were transcribed. NVivo (Qualitative Research Solutions) was used for analysis using deductive and inductive analytical approaches to lead to key themes.
Evaluation of a service redesign
DOI: 10.3109/09638237.2014.951487
Ethics The project underwent a proportionate review by the ethics committee (Ref: 12/SW/035). Data were handled by senior trust staff who are CRB checked, complied with the data protection act and Information Governance policy of the trust. All data were anonymised for analysis.
Downloaded by [Deakin University Library] at 11:37 11 November 2015
Results (1) Surveys SU surveys: 992 SU attended appointments in AMH services in the survey week. A total of 482 surveys were returned. A response rate of 48.5% achieved. Among SU who had used the services before, 31.2% respondents were receiving care from CTT; whereas among those who had not used services prior to the redesign 34.1% respondents were cared for by the AAT. Staff surveys: 295 responses received. This is a response rate of 18.44%. GP surveys: 166 responses were received and 146 were included in the analysis as 20 responses were left blank in most questions. Majority of GPs who responded had been practicing for more than 10 years (106; 72.5%). (2) Carer Focus Groups: 29 participants were interviewed through the carer focus groups. (3) Interviews: Clinician interviews: 26 clinicians were interviewed for the study. This was over half of those contacted (n¼43) with two unable to find time to meet and 15 withdrawing after initial interest (reasons for this unknown). Service user interviews: 84 names were provided to contact but only 3 interviews were completed.
Key findings Access and assessment Access Existing SU, clinicians and GPs answered questions relating to accessing the service and movement throughout the service. The results are shown in Table 1. AAT and impact on access: SU were significantly more likely to think access was better or the same than either clinicians (95% CI – 11.98% to 29.48%) or GPs (95% CI – 8.13% to 29.35%). 24/7 access: SU were significantly more likely to think overall access was better or the same than either clinicians (95% CI 8.02% to 29.32%) or GPs (95% CI 3.92% to 32.12%). Time from referral to assessment: SU were significantly more likely to think that time from referral to assessment was better than both clinicians (95% CI 26.17% to 45.67%) and GPs (95%CI 23.63% to 46.12%). Compared to GPs, a significantly higher proportion of clinicians thought that the time from assessment to being assigned a care coordinator was worse since the redesign (95% CI 11.58% to 35.12%). SU were significantly more likely to think that transition from AAT to CTT was better after the redesign than both
273
Table 1. Survey – access and transitions. Better N Single point of access Clinician, N ¼ 214 66 GP, N ¼ 113 35 SU Green, N ¼ 190 67 24/7 access Clinician, N ¼ 207 61 GP, N ¼ 67 19 SU Green, N ¼ 84 35 Referral to assessment Clinician, N ¼ 165 47 GP, N ¼ 99 29 SU Green, N ¼ 177 115 Assessment to care coordinator Clinician, N ¼ 190 19 GP, N ¼ 95 14 Transitions between teams SU Green, N ¼ 167 139 AAT-CTT Clinician, N ¼ 183 20 GP, N ¼ 92 6 SU Green, N ¼ 178 50 CTT-ACT Clinician, N ¼ 178 20 GP, N ¼ 83 15
Same
Worse
%
N
%
N
%
30.8 31.0 35.3
58 33 83
27.1 29.2 43.7
90 45 40
42.1 39.8 21.0
29.5 28.4 41.7
66 23 33
31.9 34.3 39.3
80 25 16
38.6 37.3 19.0
28.5 29.3 65.0
52 34 31
31.5 34.3 17.5
66 36 31
40 36.4 17.5
10 14.7
38 37
20 38.9
133 44
70 46.3
65.9
39
23.5
18
10.8
10.9 6.5 28.1
45 38 89
24.6 41.3 50.0
118 48 39
64.5 52.2 21.9
11.2 18.1
70 33
39.3 39.8
88 35
49.4 42.2
clinicians (95% CI 9.06% to 25.11%) and GPs (95% CI 12.3% to 29.4%). Qualitative data added that the SPOA is seen as an advantage especially for new users. Initial access is generally timely and perceived to be better. Assessment
Compared to clinicians, a significantly lower proportion of SU thought that the focus on strengths and skills in assessments was worse post-redesign (95% CI 7.08% to 23.34%). Compared to clinicians, a significantly lower proportion of SU noted that holistic focus of assessments was worse post-redesign (95% CI 7.08% to 24.34%). Compared to clinicians, a significantly lower proportion of SU noted that the focus on goals of care in assessments was worse post-redesign (95% CI 12.75% to 29.51%). Audit found that the quality of mental health history including mental state, formulation or diagnosis was variable but not improved post-redesign. There was less evidence of balanced risk assessment in the formulation post-redesign. There was some evidence of a social approach and SU involvement in the care plan post-redesign. Reassessment A total of 47.5% of SU who had accessed the services before and after the redesign and 75.9% of SU who had not accessed services before the redesign reported that they were reassessed. Audit and qualitative data supported that reassessments had increased.
274
S. Rathod et al.
J Ment Health, 2014; 23(5): 271–275
Impact on EIP and AOT
morbidity on the ward was perceived to have increased with swifter discharge.
EIP Compared to clinicians, a significantly higher proportion of GPs thought that assessment by EIP had improved access to the AMH service (95% CI 8.6% to 32.3%). The audit and qualitative data supported the finding that EIP function had been diluted following integration into the CTT. Of the four areas, the South had retained a dedicated EIP team. The evaluation showed different population needs and having maintained a dedicated EIP showed better service provision (Table 2).
Staff in AMH reduced by 7.2% from 1724 to 1599 staff (including bank staff). Excluding bank staff, head count reduced (10.1%) from an average of 1431.5 in 2011 to 1287 in 2012. Sick days increased by 381 (7.8%) between the two periods. Staff reported a reduction in non-clinical resources (Table 3).
Discussion
AOT
Downloaded by [Deakin University Library] at 11:37 11 November 2015
Impact of the re-design on workforce
AOT function was not diluted through integration into CTT. Audit identified areas of development although this was consistent before and after the redesign. South population showed different needs due to diversity and morbidity and justified the need for a specialist team. The shared care model (West) based on the FACT model (Van Veldhuizen, 2007) was reported as positive for SU’s. Bed usage
As a result of the redesign, bed numbers reduced from 165 to 107 (35%). The number of admissions to inpatients reduced, with an associated decrease in OBD. Bed occupancy increased following the re-design. There was an increase in proportion of detained patients (95% CI 5.35% to 20.45%) compared to informal admissions between the two time periods. The Health of the Nation Outcome Scales (HoNOS; Wing et al., 1995) scores for admissions were higher for agitation, lower for suicidality, higher for psychosis and higher for social scores post-redesign. SU who had used the service before were more likely than either clinicians (95%CI 11.5% to 37.7%) or GPs (95% CI 3.8% to 23.7%) to state that the clarity of crisis pathways had improved since the redesign. Significantly more clinicians than GPs reported that timely discharge had improved since the redesign (95% CI 3.1% to 19.3%). Significantly more GPs than clinicians reported that the location of inpatient services had improved (95% CI 13.5% to 29.0%). There were no significant differences between those service users who had accessed the service before and those who had not in terms of their views on inpatient services or the hospital at home service. Qualitative data supported home treatment although perception was that reduction in beds had made access to beds more difficult and the severity of admission criteria and Table 2. Survey results on EIP. Better
Same
Worse
Access to EIP
N
%
N
%
N
%
Clinician, N ¼ 141 GP, N ¼ 67
14 20
9.9 29.9
69 31
48.9 46.3
58 16
41.2 23.8
The Royal College of Psychiatrists (2010a), WHO (2007) and Naylor & Bell (2010) highlighted the potential to improve quality and productivity in mental health services. Across the world initiatives have been taken in community services but there are few evaluations of effectiveness or efficiency (Pedersen & Kolstad, 2009). In this study, the number of beds, hence admissions to inpatients reduced, with an associated decrease in the number of OBD but bed occupancy increased following the re-design. This is consistent with the trends to move to community care and demonstrates increased efficiency in the use of inpatient care. The proportion of detained patients increased as could be expected (Keown et al., 2011) and consistent with the national picture (NHS Information centre, 2011/12). While the respondents indicated that access improved through the new teams, the quality of assessment had not improved and number of re-assessments did not reduce. There was concern about difficult transitions and confusion around different roles and teams, e.g. for emergency assessments, and at the interface between them and CTT. Therefore, it was difficult to establish productivity gains. The decision to base integration of the AOT function on local population need was found to be appropriate (Killaspy et al., 2009). Implementation of the FACT model showed early success (Firn et al., 2013). Integration of EIP teams into community teams in Hampshire diluted the EI function. Retention of a distinct team in South preserved this. Literature (McCrone et al., 2010; Stafford et al., 2013) has determined the effectiveness of EIP teams. There is a case for considering the psychosis clinical pathway and extending the benefits of this service,
Table 3. Staff report on non-clinical resources. Improved
Availability of admin support, n ¼ 145 Ability to access information/ records, n ¼ 152 Technology/ access to networks, n ¼ 148 Sufficient office space, n ¼ 152 Location of service for staff, n ¼ 139
Same
Worse
N
%
N
%
N
%
10
6.9
76
52.4
59
40.7
15
9.9
102
67.1
35
23.0
12
8.1
92
62.2
44
29.7
8 6
5.3 4.3
81 77
53.6 55.4
63 56
41.1 40.3
Evaluation of a service redesign
Downloaded by [Deakin University Library] at 11:37 11 November 2015
DOI: 10.3109/09638237.2014.951487
especially in light of the Schizophrenia commission report (Rethink, 2012). One of the important findings from the study was that staff did not rate the redesign as positively as service users. The process of the redesign was perceived as difficult and working conditions, e.g. office accommodation, technology and travelling time, as having deteriorated. Nonmedical staff perception that their caseloads increased may have led to the increased levels of stress and anxiety in staff in terms of whether they could deliver quality care. Staff sickness increased. In the current volatile economic context, change is constant and therefore engaging clinicians is particularly important as the costs of not doing so can be significant. Poorly managed change impacts on quality of care at many levels (Gilbert et al., 2014). Productivity and efficiency can be increased through innovation and integration in clinical services only when needs of staff like safety and belonging are met (Maslow, 1943). The relatively negative response of GPs indicated that better engagement and communication was needed with them. A key learning point from the evaluation was that when implementing large scale change it is important to ensure that the infrastructure for delivering the change is robust (Gilbert et al., 2014). This includes attention to the evidence-base, engagement of key stakeholders and support from corporate partners. Key areas for consideration when implementing large scale change are the need to ensure that service user experience is improved (Francis, 2013). Following a large scale change, benefits can be slow to become apparent but effective evaluation, as in this instance, can be achieved using limited service resources. The service has identified key areas of improvement from this project. Limitations The evaluation project had its limitations. Only three SU were interviewed although considerable time was taken to engage them. The study used a before-and-after methodology. The results only considered a 3-month period from 6 months before and after the redesign. Thus many of the changes were likely to still be ‘‘bedding in’’; including staff getting used to new premises, working arrangements, etc. alongside SU adapting to the changes.
Conclusions Services across the globe are undergoing extensive and challenging change programmes in a climate of on-going austerity. There is a need for a culture of continuous multimethod evaluation and quality improvement with SU experience at the centre of decision making, incorporating learning from services nationally and internationally.
275
Declaration of interest This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
References Firn M, Hindhaugh K, Hubbeling D, et al. (2013). A dismantling study of assertive outreach services: Comparing activity and outcomes following replacement with the FACT model. Soc Psychiatr Psychiatr Epidemiol, 48, 997–1003. Francis R. (2013). Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry. Available at: www.midstaffspublicinquiry.com/ report. Gilbert H, Peck E, Ashton B, et al. (2014). Service transformation lessons from mental health. London: The Kings Fund. Keown P, Scott W, Bhui K, Scott J. (2011). Association between provision of mental illness beds and rate of involuntary admissions in the NHS in England 1988–2008: Ecological study. BMJ (Clinical Research Edition), 343, d3736. Killaspy H, Kingett S, Bebbington P, et al. (2009). Randomised evaluation of assertive community treatment: 3-year outcomes. Br J Psychiatr, 195, 81–2. Kingdon D. (2011). Transforming mental health services. Br J Psychiatr, 199, 1–2. Manderscheid RW, Atay JE, Male A. (2002). Highlights of organized mental health services in 2000 and major national and state trends, in Mental Health US. Washington, DC: Center for Mental Health Services. Maslow AH. (1943). A theory of human motivation. Psychol Rev, 50, 370–96. McCrone P, Craig TK, Power P, Garety PA. (2010). Cost-effectiveness of an early intervention service for people with psychosis. Br J Psychiatr, 196, 377–82. Naylor C, Bell A. (2010). Mental health and the productivity challenge London: The Kings Fund. NHS Information centre. (2011/12). Inpatients formally detained in hospitals under the Mental Health Act 1983, and patients subject to supervised community treatment, Annual figures, England. Pedersen P, Kolstad A. (2009). De-institutionalisation and transinstitutionalisation – Changing trends of inpatient care in Norwegian mental health institutions 1950–2007. Int J Ment Health Syst, 3, 28. Rethink. (2012). The abandoned illness. An independent report from the Schizophrenia commission. Royal College of Psychiatrists. (1996). Assessment and clinical management of risk of harm to other people. Council Report CR 53. London. Royal College of Psychiatrists. (2010a). Looking Ahead. Future development of UK mental health services: Recommendations from a Royal College of Psychiatrists’ enquiry. Occasional Paper OP75. London. Stafford M, Jackson H, Mayo-Wilson E, et al. (2013). Early interventions to prevent psychosis: Systematic review and meta-analysis. Br Med J, 346, f185. Tyrer P. (2011). Has the closure of Psychiatric beds gone too far? Br Med J, 343, d7457. Van Veldhuizen. (2007). The Flexible Assertive Community Treatment model (FACT). Commun Mental Health J, 43(4), 421–33. WHO. (2007). Expert opinion on barriers and facilitating factors for the implementation of existing mental health knowledge in mental health services. Geneva: World Health Organization. Wing J, Beaver AS, Curtis RH, et al. (1995). HoNOS: Health of the nation outcome scales. London: Royal College of Psychiatrists’ Research Unit Report.
