QUALITY IMPROVEMENT
Clinical audit: Still an important tool for improving healthcare James Y Paton,1 Rita Ranmal,2 Jan Dudley,3 on behalf of RCPCH Clinical Standards Committee
1
Department of Paediatric Respiratory Medicine, Royal Hospital for Sick Children, Glasgow, UK 2 Royal College of Paediatrics and Child Health, London, UK 3 Department of Paediatric Nephrology, Bristol Royal Hospital for Children, Bristol, UK Correspondence to Dr Jan Dudley, Department of Paediatric Nephrology, Bristol Royal Hospital for Children, Bristol BS28BJ, UK; [email protected] Received 11 November 2013 Revised 13 July 2014 Accepted 21 July 2014 Published Online First 13 August 2014
To cite: Paton JY, Ranmal R, Dudley J, et al. Arch Dis Child Educ Pract Ed 2015;100: 83–88.
ABSTRACT The use of data to challenge and improve healthcare has a long and distinguished history but has often failed to bring about expected improvements. It has never become fully embedded in clinical practice, probably because data alone are insufficient to drive change. There is now a greater appreciation that changing and improving healthcare requires changing behaviours. Clinical audit remains one of the important tools that can be used to facilitate such change.
INTRODUCTION Writing in 2002 in Principles for Best Practice in Clinical Audit, Hine and Rawlins said: ‘The time has come for the NHS to take clinical audit very seriously. Anything less would miss the opportunity we have to re-establish the confidence and trust upon which the NHS is founded’.1 A brief look at the National Health Service (NHS) a decade later would suggest that we have missed that opportunity. The idea that reviewing healthcare, particularly its outcomes, can help to improve care is not new. Florence Nightingale was the heroine of the British soldiers she cared for during the Crimean War (1854–1856). She used health data in a way that would be entirely familiar to us today; she analysed carefully kept mortality records and realised it was not battle injuries but mismanagement of sanitary conditions in the field hospitals and the consequent infectious illnesses that were responsible for the deaths of thousands of soldiers.2 Nightingale was passionate about data and a pioneer in the statistical expression of empirical data (figure 1) and its use to improve care. She argued that The main end of statistics should not be to inform Government as to how many men have died but to enable steps to be
taken to prevent the extension of disease and mortality.
Another early supporter of healthcare reviews was Ernest Codman (1869– 1940), a Boston surgeon who recorded diagnostic and treatment errors and linked these to outcome in order to make improvements. The lack of outcomes evaluation at the hospital where he worked led him to start his own private hospital that he called the ‘End Result Hospital’. All errors in his hospital were recorded and related to outcome. Perhaps, not surprisingly, his efforts brought him ridicule, poverty and censure.3 BRINGING CLINICAL AUDIT TO CENTRE STAGE IN TODAY’S HEALTHCARE SETTINGS Despite such pioneering beginnings in the use of data to argue for improvements in care, clinical audit in the form of outcomes and performance monitoring has failed to make much impact. In an 1984 article about quality assessment in health, Maxwell, the secretary to the King’s Fund, wrote trenchantly that the ‘majority view among British doctors is that assessing and safeguarding the quality of medical care are matters best left to voluntary initiatives among consenting adults in private’.4 It took a public outrage over the management of children receiving complex cardiac surgical care in Bristol between 1984 and 1995 to bring the importance of clinical audit back to centre stage. The resulting Bristol Royal Infirmary Enquiry picked apart the failings of the system, not just in Bristol but throughout the NHS and the published report described ‘a time when there was no agreed means of assessing the quality of care, no standards for evaluating performance and confusion throughout the NHS as to who
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
83
Quality improvement
Figure 1 Diagram of the causes of mortality of the army in the east prepared by Florence Nightingale showing that the introduction of strict sanitary routines and standards of hygiene reduced mortality rates from 40% to 2%.
was responsible for monitoring the quality of care’. The final report recommended that multidisciplinary clinical audit should be at the centre of a system of local performance monitoring by Trusts and that clinical audit should be compulsory for all healthcare professionals.5 There has been some progress in this area; in the UK, commissioners of healthcare services may hold hospital trusts to account through clinical audit of costly medicines that come with an NHS mandate for funding. The General Medical Council has made it a requirement to undertake audit as part of its guidance on ‘Good Medical Practice’. This requires that all doctors in the UK take part in systems of quality assurance and quality improvement (QI; box 1). There is now a mandatory requirement for individual clinicians to produce data about their performance for annual revalidation, a requirement that may finally systematically embed audit within clinical practice. Other countries have taken other approaches to the issue of reviewing and improving care. For example, Scandinavian countries have developed disease registries that through data linkage provide very large datasets of longitudinal data. Such registries have tended to be either procedure specific (eg, hip arthoplasty) or condition specific (eg, cancer registries). These have been used for monitoring health, care quality and health outcomes and exploring geographical variation in care outcomes. They have also provided a base for more local analysis and QI work as well as clinical research. One notable example is the Swedish Hip Arthroplasty Register (SHAR) started in 1979. This 84
has had 100% participation. There has been public involvement in the process, first with public reporting of the data 1999 and from 2002 the incorporation of a patient-reported outcome measure (PROM). The SHAR’s mission is not just to provide a device register but to provide quality control of the whole process of hip arthoplasty. To this end the data have been used to drive improvements in care such that Sweden has a rate of total hip arthoplasty revision considerably lower than in the USA.6
Box 1 General Medical Council guidance (2013) on quality assurance and quality improvement (http://www.gmc-uk.org/guidance/good_medical_ practice/systems_protect.asp) Contribute to and comply with systems to protect patients 22. You must take part in systems of quality assurance and quality improvement to promote patient safety. This includes: A. Taking part in regular reviews and audits of your own work and that of your team, responding constructively to the outcomes, taking steps to address any problems and carrying out further training where necessary; B. Regularly reflecting on your standards of practice and the care you provide; C. Reviewing patient feedback where it is available.
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
Quality improvement Table 1 Some examples of clinical audit and other quality improvement approaches that use data to improve healthcare Approaches that use data to improve healthcare
Example
National/international audit projects
UK Epilepsy 12 (http://www.rcpch.ac.uk/epilepsy12) UK National Paediatric Diabetes Audit (http://www.rcpch.ac.uk/npda) The National Neonatal Audit Programme (http://www.rcpch.ac.uk/nnap). European Chronic Obstructive Pulmonary Disease audit (http://www.erscopdaudit.org)
Confidential enquiries
UK National review of asthma deaths (http://www.rcplondon.ac.uk/nrad)
Clinical registries
UK National joint registry (http://www.njrcentre.org.uk/njrcentre/default.aspx) Swedish Hip Arthoplasty Register (http://www.shpr.se)
Outcomes audit
UK Congenital heart disease outcomes (http://nicor5.nicor.org.uk)
Patient experience surveys
UK National patients’ surveys (http://www.nhssurveys.org)
Medication error reporting systems
UK Hospital medication error reporting systems (http://www.nrls.npsa.nhs.uk) US National Co-ordinating Council for Medication Error Reporting and Prevention (http://www.nccmerp.org)
In the USA, the seminal work on patient safety ‘To Err is Human’ in 2000 showed that medical errors were common and adversely affected patient outcomes.7 More recently in the USA, comparative effective research combines the synthesis of existing evidence with analysis of patient outcomes from data available in insurance claims, electronic medical records or clinical registries to analyse factors that influence clinical outcomes. This has identified8 A. significant geographical variation in healthcare B. lack of evidence for almost 50% of care C. 30% of spending reflecting medical care of uncertain or questionable value.
WHAT IS CLINICAL AUDIT? Clinical audit has had many definitions, of which many resemble the proverbial definition of a camel as a horse designed by a committee. Stripped to its essence, the central purpose of audit is to measure clinical performance against an agreed standard and use the results to improve practice (box 2). Keogh, the NHS Medical Director, recently described clinical audit as one of the ‘tools for quality improvement’ (table 1) and emphasised the importance of the link between national clinical audit and local QI.9 AUDIT OR RESEARCH? Even the experienced can get confused about the difference between audit and research. In simple terms, research is finding out what we ought to do, while audit is finding out if we are doing what we ought to
Box 2
Definition of clinical audit1
Clinical audit is a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change.
do. Research always requires ethical approval and patient consent. The Medical Research Council and the Health Research Authority have provided a decision tool to help users decide (box 3). A characteristic of effective clinical audit is that the data must be sufficiently robust to make a clear case for action that can withstand challenge. Again, Keogh highlighted this point: ‘The status of clinical audit, confidential enquiries and related quality improvement tools should ‘catch up with research’ in their methodological rigour.’9 CLINICAL AUDIT AND OTHER QI TOOLS There is a whole range of methodologies and activities, from large national or international projects to small local surveys that collect and use data to drive QI in healthcare (table 1). Some of these methods would not usually be thought of as clinical audit, for example, patient safety initiatives like medication error reporting systems or the NHS Safety Thermometer (http://www.ic.nhs.uk/thermometer). However, they all share the systematic collection and use of data to review care with a view to improving future care.
Box 3 Medical Research Council/Health Research Authority decision tool to determine whether a project is audit or research (http://www.hra-decisiontools.org.uk/research/) Questions: 1. Are any participants randomised? 2. Does the study protocol demand changing treatment/ patient care from accepted standards for any patients? 3. Are your findings going to be generalisable/ ( publishable)? If the answer to any of these questions is yes, the project or study should be considered as research.
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
85
Quality improvement UNDERTAKING HIGH-QUALITY CLINICAL AUDIT Doing a good clinical audit can be a major undertaking (table 2). Some of the important requirements and steps are summarised below: 1. A supportive environment: A key requirement for any successful audit is that there must be support of audit and a commitment to change within the organisation. If the underlying culture is not supportive, then the outcome may only be a ritual ‘shooting’ of the messenger who brings the news that not all is well. Most institutions now have a local audit department that will facilitate and support clinical audit and these are a useful starting point for advice. 2. A good topic: Good audit topics will focus on important questions arising out of local or national service concerns or priorities. These need to be topics where change is possible. The acronym SMART—specific, measurable, achievable, relevant and time specific—provides a useful mnemonic for evaluating any audit proposal. Auditing without the expectation of being able to change practice is a waste of effort. 3. An evidence base: Audit has in common with clinical diagnosis the fact that it takes place not at the bed side but between bed sides. Actual clinical performance needs to be compared with the best available evidence. This might range from high-quality clinical guidelines from organisations such as the National Institute for Health and Care Excellence to local clinical consensus in situations where the available evidence base is limited. 4. A plan for collecting data: Data collection is often burdensome and time consuming. There should be a clear plan that addresses simple questions such as ‘who, what, where, when, how, how many and by whom?’ Many audits make the mistake of collecting either far too much information or poorly targeted data that will not allow the important questions to be answered clearly. 5. A plan for analysing and presenting data: There needs to be a clear plan and timescale for analysing the data after collection. This should start with thinking about the
quality of the data collected and progress through to how the data are analysed and presented. 6. A plan for feedback and changing behaviours and practice: If audit data are to lead to changes in practice, there needs to be very careful thought paid to how the results are to be fed back and used to promote change. Memorable images such as Florence Nightingale’s diagram of the causes of mortality can have a powerful impact (figure 1). It is increasingly recognised that bringing about change is often the most difficult part of audit. Developing a clear plan for how change is to be achieved is necessary. This will usually involve getting support from a number of key stakeholders. 7. Monitoring to confirm that change has occurred: It is essential to demonstrate that changes have been implemented, are sustainable and result in improvement on what has gone before so a plan for future monitoring will be needed.
For trainee paediatricians and other healthcare professionals wanting to learn more about audit, the Royal College of Paediatrics and Child Health (RCPCH) and partners, with funding from the Healthcare Quality Improvement Partnership, have developed an E-learning resource (available at http:// www.rcpch.ac.uk/e-learning-resources-clinical-audit). The resource is divided into four sessions, which map onto the Plan, Do, Study, Act cycle, describing the stages and processes involved in undertaking clinical audit. IS THERE EVIDENCE THAT CLINICAL AUDIT WORKS? Audit and feedback have been widely promoted as a strategy to improve professional practice. The underpinning belief has been that healthcare professionals will improve their practice in response to information about their performance, particularly when this falls
Table 2 Useful tips for undertaking a good clinical audit Do
Don’t
Be clear of the difference between audit and research.
Submit your work for publication without confirming the need or otherwise for ethics approval and/or patient consent.
Be clear of the objectives and the link to quality improvement.
Be overambitious—remember the SMART objectives.
Link to evidence-based standards.
Store confidential patient data in a way that may be accessible to others (non-encrypted, non-anonymised).
Make sure you have allocated sufficient time.
Collect data that would be ‘nice to have’ that is not essential for the audit.
Involve people likely to be affected by the audit at an early stage.
Forget to register the audit in your organisation. There may be resources available to assist you.
Plan a strategy for presentation and dissemination of audit findings.
Assume that change will occur because the audit has identified areas for improvement—develop a clear plan for how change is to be achieved.
Plan a strategy to demonstrate that improvement has occurred.
86
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
Quality improvement below what are regarded as desirable standards of outcomes. Surprisingly, the evidence that audit is an effective strategy for improving practice is quite weak. A Cochrane review included 140 studies and concluded that audit and feedback led to small, although potentially important, improvements in clinical practice.10 For example, the authors found that there was a 4.3% absolute increase in desired practice (IQR 0.5%–16%) when considering any trial in which audit and feedback was the core aspect of the intervention compared with no audit and feedback. However, the large range of effect sizes led the authors to conclude that when optimally designed, audit and feedback could play a role in improving practice. The authors concluded that audit was more likely to be effective when 1. 2. 3. 4. 5.
baseline performance was low the source of feedback was a supervisor or colleague there was feedback in both verbal and written formats feedback was provided more than once clear, explicit targets and an action plan are included.
WHY HAS CLINICAL AUDIT NOT BEEN MORE SUCCESSFUL IN IMPROVING CARE? In the main, audit has all too often failed because it has not brought about change and improvement. This may have arisen for a variety of reasons; the question may not be clear; data collected may not be objective or valid. There may have been poor and slow feedback of results. Even when issues such as these have been tackled and the information is of high quality and informative, it may be that other barriers to change have not been recognised or addressed. The most common barriers that block change are behavioural. It needs to be clearly understood at the outset of any clinical audit project that change requires behaviours to change and that data alone are usually not sufficient to change behaviour. Audit, and feedback, has a relatively limited effect probably because it needs to engage more realistically and in a more focused way with overcoming resistance to changing behaviours. The relative failure of clinical audit in the way that it has usually been promoted as an audit cycle has led to a more explicit focus on change and QI. Clinical medicine and surgery have increasingly looked to developments in QI in other areas such as aviation or the nuclear industry as well as to the QI techniques developed for manufacturing industries. One obvious example of this shift has been the increasing use of small rapid cycle audits as a method for more quickly implementing change and improving care. CLINICAL AUDIT IN THE FUTURE Currently, clinical audit may be viewed as having a number of roles (box 4). Underpinning them all is recognition that there is a need to evaluate the
Box 4 audit
Current thinking about the role of clinical
1. An approach to obtaining essential data to enable quality improvement 2. A technique for quality improvement 3. A way to get data to support revalidation assurance 4. A set of tools to allow monitoring of quality of healthcare.
performance and outcomes of healthcare systems. This is linked to a widespread acceptance that if healthcare is to improve then reliable data on current performance is a fundamental building block. It is over a hundred years since the great Victorian physicist Lord Kelvin recognised the importance of measuring data when he said: ‘If you cannot measure it you cannot improve it’. It has taken a long time for these insights to become accepted currency in the NHS. Providing data about performance for revalidation is likely to be an important future driver. However, as outlined above, information on its own is rarely effective at changing behaviour. In the future, we need to explore how we might better use the energy and expertise of patients, parents and carers in using audit to deliver change. One example is the increasing use of tools such as PROMS and patient-related experience measures in audits. Audit as a tool for driving improvement also needs to work across healthcare settings, or ‘organisational boundaries’. In the UK, Healthcare Quality Improvement Partnership (HQIP) funds and supports multisite audits that are undertaken in 10 or more different sites, each of which may have different information technology systems. The Virginia Mason Health system in the USA uses ‘value stream mapping’ to analyse current state and design a future state for the series of events that take a product or service from its beginning through to the patient. An example of this is a joint replacement pathway, which starts in primary care.11 Value stream mapping encourages collaboration across organisational boundaries. Reflecting this shift of focus from audit to QI, organisations such as HQIP and NHS Improvement (http://www.improvement.nhs.uk/Default.aspx?alias= www.improvement.nhs.uk/qipp) are making highquality tools available to facilitate QIs in healthcare as well as supporting clinical audit. A list of useful resources can be found in box 5. As part of its aim of raising standards of healthcare for children, the RCPCH Clinical Standards team facilitate a number of high-quality national and multisite audits to help improve patient care and outcomes. These audits enable paediatricians to evaluate their practice against national standards (table 1).
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
87
Quality improvement Box 5
Useful resources for clinical audit
▸ A range of implementation tools to help put National Institute for Health and Care Excellence guidance into practice including audit tools. http://www.nice.org.uk/usingguidance/ implementationtools/implementation_tools.jsp ▸ Local Clinical Audit Handbook for Physicians http://www.hqip.org.uk/assets/Guidance/Local-clinicalaudit-handbook-for-physicians-August-2010-FINAL.pdf ▸ Guide to involving junior doctors in clinical audit http://www.hqip.org.uk/assets/5-HQIP-CA-PD-026Guide-to-Involving-Junior-Doctors-in-Clinical-Audit-19April-2010.pdf ▸ The Royal College of Paediatrics and Child Health E-Learning clinical audit training resource http://www.rcpch.ac.uk/e-learning-resources-clinicalaudit ▸ Online journal of clinical audits: an accessible database and reference area to publish and share audit reports http://www.clinicalaudits.com ▸ Transforming Clinical Audit Data Into Quality Improvements learning package http://www.hqip.org.uk/transforming-clinical-auditdata-into-quality-improvements/
CONCLUSION The ongoing improvements in outcomes following the regular publishing of healthcare results such as in the reporting of Cardiac Surgery outcomes in the UK or in the Swedish Hip Arthoplasty Register provide examples of what clinical audit can achieve. However,
88
using data to improve the quality of care delivered to patients requires a commitment at all organisational levels, from international to local, if the goal of improving the quality of healthcare is to be achieved. Competing interests None. Provenance and peer review Commissioned; externally peer reviewed.
REFERENCES 1 Scrivener R, Morrell C, Baker R, et al. Principles for best practice in clinical audit. Abingdon: Radcliffe Medical Press Ltd, 2002. 2 Gill CJ, Gill CG. Nightingale in Scutari: her legacy re-examined. Clin Infect Dis 2005;40:1799–805. 3 Neuhauser D, Codman EA. Qual Saf Health Care 2002;11:104–5. 4 Maxwell RJ. Quality assessment in health. BMJ (Clin Res Ed) 1984;288:1470–2. 5 Learning from Bristol: The report of the public enquiry into children’s heart surgery at the Bristol Royal Infirmary 1984– 1995. Contract No: Command Paper: CM 5207. 2001. 6 Kurtz SM, Ong KL, Schmier J, et al. Future clinical and economic impact of revision total hip and knee arthroplasty. J Bone Joint Surg Am 2007;89(S3):144–51. 7 Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: MS, 2000. 8 Manchikanti L, Flaco FJ, Boswell MV, et al. Facts, fallacies, and politics of comparative effectiveness research: Part I. Basic considerations. Pain Physician 2010;13:E23–54. 9 Keogh B. Management of the national clinical audit and confidential enquiries programme. London, 2012 [July 2013]. http://www.hqip.org.uk/assets/News-uploads/121127-BKLetter.pdf 10 Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;6:CD000259. 11 Kenney C. Transforming health care: Virginia Mason Medical Center’s pursuit of the perfect patient experience. New York, NY: Productivity Press, 2010.
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
Copyright of Archives of Disease in Childhood -- Education & Practice Edition is the property of BMJ Publishing Group and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Clinical audit: Still an important tool for improving healthcare James Y Paton,1 Rita Ranmal,2 Jan Dudley,3 on behalf of RCPCH Clinical Standards Committee
1
Department of Paediatric Respiratory Medicine, Royal Hospital for Sick Children, Glasgow, UK 2 Royal College of Paediatrics and Child Health, London, UK 3 Department of Paediatric Nephrology, Bristol Royal Hospital for Children, Bristol, UK Correspondence to Dr Jan Dudley, Department of Paediatric Nephrology, Bristol Royal Hospital for Children, Bristol BS28BJ, UK; [email protected] Received 11 November 2013 Revised 13 July 2014 Accepted 21 July 2014 Published Online First 13 August 2014
To cite: Paton JY, Ranmal R, Dudley J, et al. Arch Dis Child Educ Pract Ed 2015;100: 83–88.
ABSTRACT The use of data to challenge and improve healthcare has a long and distinguished history but has often failed to bring about expected improvements. It has never become fully embedded in clinical practice, probably because data alone are insufficient to drive change. There is now a greater appreciation that changing and improving healthcare requires changing behaviours. Clinical audit remains one of the important tools that can be used to facilitate such change.
INTRODUCTION Writing in 2002 in Principles for Best Practice in Clinical Audit, Hine and Rawlins said: ‘The time has come for the NHS to take clinical audit very seriously. Anything less would miss the opportunity we have to re-establish the confidence and trust upon which the NHS is founded’.1 A brief look at the National Health Service (NHS) a decade later would suggest that we have missed that opportunity. The idea that reviewing healthcare, particularly its outcomes, can help to improve care is not new. Florence Nightingale was the heroine of the British soldiers she cared for during the Crimean War (1854–1856). She used health data in a way that would be entirely familiar to us today; she analysed carefully kept mortality records and realised it was not battle injuries but mismanagement of sanitary conditions in the field hospitals and the consequent infectious illnesses that were responsible for the deaths of thousands of soldiers.2 Nightingale was passionate about data and a pioneer in the statistical expression of empirical data (figure 1) and its use to improve care. She argued that The main end of statistics should not be to inform Government as to how many men have died but to enable steps to be
taken to prevent the extension of disease and mortality.
Another early supporter of healthcare reviews was Ernest Codman (1869– 1940), a Boston surgeon who recorded diagnostic and treatment errors and linked these to outcome in order to make improvements. The lack of outcomes evaluation at the hospital where he worked led him to start his own private hospital that he called the ‘End Result Hospital’. All errors in his hospital were recorded and related to outcome. Perhaps, not surprisingly, his efforts brought him ridicule, poverty and censure.3 BRINGING CLINICAL AUDIT TO CENTRE STAGE IN TODAY’S HEALTHCARE SETTINGS Despite such pioneering beginnings in the use of data to argue for improvements in care, clinical audit in the form of outcomes and performance monitoring has failed to make much impact. In an 1984 article about quality assessment in health, Maxwell, the secretary to the King’s Fund, wrote trenchantly that the ‘majority view among British doctors is that assessing and safeguarding the quality of medical care are matters best left to voluntary initiatives among consenting adults in private’.4 It took a public outrage over the management of children receiving complex cardiac surgical care in Bristol between 1984 and 1995 to bring the importance of clinical audit back to centre stage. The resulting Bristol Royal Infirmary Enquiry picked apart the failings of the system, not just in Bristol but throughout the NHS and the published report described ‘a time when there was no agreed means of assessing the quality of care, no standards for evaluating performance and confusion throughout the NHS as to who
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
83
Quality improvement
Figure 1 Diagram of the causes of mortality of the army in the east prepared by Florence Nightingale showing that the introduction of strict sanitary routines and standards of hygiene reduced mortality rates from 40% to 2%.
was responsible for monitoring the quality of care’. The final report recommended that multidisciplinary clinical audit should be at the centre of a system of local performance monitoring by Trusts and that clinical audit should be compulsory for all healthcare professionals.5 There has been some progress in this area; in the UK, commissioners of healthcare services may hold hospital trusts to account through clinical audit of costly medicines that come with an NHS mandate for funding. The General Medical Council has made it a requirement to undertake audit as part of its guidance on ‘Good Medical Practice’. This requires that all doctors in the UK take part in systems of quality assurance and quality improvement (QI; box 1). There is now a mandatory requirement for individual clinicians to produce data about their performance for annual revalidation, a requirement that may finally systematically embed audit within clinical practice. Other countries have taken other approaches to the issue of reviewing and improving care. For example, Scandinavian countries have developed disease registries that through data linkage provide very large datasets of longitudinal data. Such registries have tended to be either procedure specific (eg, hip arthoplasty) or condition specific (eg, cancer registries). These have been used for monitoring health, care quality and health outcomes and exploring geographical variation in care outcomes. They have also provided a base for more local analysis and QI work as well as clinical research. One notable example is the Swedish Hip Arthroplasty Register (SHAR) started in 1979. This 84
has had 100% participation. There has been public involvement in the process, first with public reporting of the data 1999 and from 2002 the incorporation of a patient-reported outcome measure (PROM). The SHAR’s mission is not just to provide a device register but to provide quality control of the whole process of hip arthoplasty. To this end the data have been used to drive improvements in care such that Sweden has a rate of total hip arthoplasty revision considerably lower than in the USA.6
Box 1 General Medical Council guidance (2013) on quality assurance and quality improvement (http://www.gmc-uk.org/guidance/good_medical_ practice/systems_protect.asp) Contribute to and comply with systems to protect patients 22. You must take part in systems of quality assurance and quality improvement to promote patient safety. This includes: A. Taking part in regular reviews and audits of your own work and that of your team, responding constructively to the outcomes, taking steps to address any problems and carrying out further training where necessary; B. Regularly reflecting on your standards of practice and the care you provide; C. Reviewing patient feedback where it is available.
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
Quality improvement Table 1 Some examples of clinical audit and other quality improvement approaches that use data to improve healthcare Approaches that use data to improve healthcare
Example
National/international audit projects
UK Epilepsy 12 (http://www.rcpch.ac.uk/epilepsy12) UK National Paediatric Diabetes Audit (http://www.rcpch.ac.uk/npda) The National Neonatal Audit Programme (http://www.rcpch.ac.uk/nnap). European Chronic Obstructive Pulmonary Disease audit (http://www.erscopdaudit.org)
Confidential enquiries
UK National review of asthma deaths (http://www.rcplondon.ac.uk/nrad)
Clinical registries
UK National joint registry (http://www.njrcentre.org.uk/njrcentre/default.aspx) Swedish Hip Arthoplasty Register (http://www.shpr.se)
Outcomes audit
UK Congenital heart disease outcomes (http://nicor5.nicor.org.uk)
Patient experience surveys
UK National patients’ surveys (http://www.nhssurveys.org)
Medication error reporting systems
UK Hospital medication error reporting systems (http://www.nrls.npsa.nhs.uk) US National Co-ordinating Council for Medication Error Reporting and Prevention (http://www.nccmerp.org)
In the USA, the seminal work on patient safety ‘To Err is Human’ in 2000 showed that medical errors were common and adversely affected patient outcomes.7 More recently in the USA, comparative effective research combines the synthesis of existing evidence with analysis of patient outcomes from data available in insurance claims, electronic medical records or clinical registries to analyse factors that influence clinical outcomes. This has identified8 A. significant geographical variation in healthcare B. lack of evidence for almost 50% of care C. 30% of spending reflecting medical care of uncertain or questionable value.
WHAT IS CLINICAL AUDIT? Clinical audit has had many definitions, of which many resemble the proverbial definition of a camel as a horse designed by a committee. Stripped to its essence, the central purpose of audit is to measure clinical performance against an agreed standard and use the results to improve practice (box 2). Keogh, the NHS Medical Director, recently described clinical audit as one of the ‘tools for quality improvement’ (table 1) and emphasised the importance of the link between national clinical audit and local QI.9 AUDIT OR RESEARCH? Even the experienced can get confused about the difference between audit and research. In simple terms, research is finding out what we ought to do, while audit is finding out if we are doing what we ought to
Box 2
Definition of clinical audit1
Clinical audit is a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change.
do. Research always requires ethical approval and patient consent. The Medical Research Council and the Health Research Authority have provided a decision tool to help users decide (box 3). A characteristic of effective clinical audit is that the data must be sufficiently robust to make a clear case for action that can withstand challenge. Again, Keogh highlighted this point: ‘The status of clinical audit, confidential enquiries and related quality improvement tools should ‘catch up with research’ in their methodological rigour.’9 CLINICAL AUDIT AND OTHER QI TOOLS There is a whole range of methodologies and activities, from large national or international projects to small local surveys that collect and use data to drive QI in healthcare (table 1). Some of these methods would not usually be thought of as clinical audit, for example, patient safety initiatives like medication error reporting systems or the NHS Safety Thermometer (http://www.ic.nhs.uk/thermometer). However, they all share the systematic collection and use of data to review care with a view to improving future care.
Box 3 Medical Research Council/Health Research Authority decision tool to determine whether a project is audit or research (http://www.hra-decisiontools.org.uk/research/) Questions: 1. Are any participants randomised? 2. Does the study protocol demand changing treatment/ patient care from accepted standards for any patients? 3. Are your findings going to be generalisable/ ( publishable)? If the answer to any of these questions is yes, the project or study should be considered as research.
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
85
Quality improvement UNDERTAKING HIGH-QUALITY CLINICAL AUDIT Doing a good clinical audit can be a major undertaking (table 2). Some of the important requirements and steps are summarised below: 1. A supportive environment: A key requirement for any successful audit is that there must be support of audit and a commitment to change within the organisation. If the underlying culture is not supportive, then the outcome may only be a ritual ‘shooting’ of the messenger who brings the news that not all is well. Most institutions now have a local audit department that will facilitate and support clinical audit and these are a useful starting point for advice. 2. A good topic: Good audit topics will focus on important questions arising out of local or national service concerns or priorities. These need to be topics where change is possible. The acronym SMART—specific, measurable, achievable, relevant and time specific—provides a useful mnemonic for evaluating any audit proposal. Auditing without the expectation of being able to change practice is a waste of effort. 3. An evidence base: Audit has in common with clinical diagnosis the fact that it takes place not at the bed side but between bed sides. Actual clinical performance needs to be compared with the best available evidence. This might range from high-quality clinical guidelines from organisations such as the National Institute for Health and Care Excellence to local clinical consensus in situations where the available evidence base is limited. 4. A plan for collecting data: Data collection is often burdensome and time consuming. There should be a clear plan that addresses simple questions such as ‘who, what, where, when, how, how many and by whom?’ Many audits make the mistake of collecting either far too much information or poorly targeted data that will not allow the important questions to be answered clearly. 5. A plan for analysing and presenting data: There needs to be a clear plan and timescale for analysing the data after collection. This should start with thinking about the
quality of the data collected and progress through to how the data are analysed and presented. 6. A plan for feedback and changing behaviours and practice: If audit data are to lead to changes in practice, there needs to be very careful thought paid to how the results are to be fed back and used to promote change. Memorable images such as Florence Nightingale’s diagram of the causes of mortality can have a powerful impact (figure 1). It is increasingly recognised that bringing about change is often the most difficult part of audit. Developing a clear plan for how change is to be achieved is necessary. This will usually involve getting support from a number of key stakeholders. 7. Monitoring to confirm that change has occurred: It is essential to demonstrate that changes have been implemented, are sustainable and result in improvement on what has gone before so a plan for future monitoring will be needed.
For trainee paediatricians and other healthcare professionals wanting to learn more about audit, the Royal College of Paediatrics and Child Health (RCPCH) and partners, with funding from the Healthcare Quality Improvement Partnership, have developed an E-learning resource (available at http:// www.rcpch.ac.uk/e-learning-resources-clinical-audit). The resource is divided into four sessions, which map onto the Plan, Do, Study, Act cycle, describing the stages and processes involved in undertaking clinical audit. IS THERE EVIDENCE THAT CLINICAL AUDIT WORKS? Audit and feedback have been widely promoted as a strategy to improve professional practice. The underpinning belief has been that healthcare professionals will improve their practice in response to information about their performance, particularly when this falls
Table 2 Useful tips for undertaking a good clinical audit Do
Don’t
Be clear of the difference between audit and research.
Submit your work for publication without confirming the need or otherwise for ethics approval and/or patient consent.
Be clear of the objectives and the link to quality improvement.
Be overambitious—remember the SMART objectives.
Link to evidence-based standards.
Store confidential patient data in a way that may be accessible to others (non-encrypted, non-anonymised).
Make sure you have allocated sufficient time.
Collect data that would be ‘nice to have’ that is not essential for the audit.
Involve people likely to be affected by the audit at an early stage.
Forget to register the audit in your organisation. There may be resources available to assist you.
Plan a strategy for presentation and dissemination of audit findings.
Assume that change will occur because the audit has identified areas for improvement—develop a clear plan for how change is to be achieved.
Plan a strategy to demonstrate that improvement has occurred.
86
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
Quality improvement below what are regarded as desirable standards of outcomes. Surprisingly, the evidence that audit is an effective strategy for improving practice is quite weak. A Cochrane review included 140 studies and concluded that audit and feedback led to small, although potentially important, improvements in clinical practice.10 For example, the authors found that there was a 4.3% absolute increase in desired practice (IQR 0.5%–16%) when considering any trial in which audit and feedback was the core aspect of the intervention compared with no audit and feedback. However, the large range of effect sizes led the authors to conclude that when optimally designed, audit and feedback could play a role in improving practice. The authors concluded that audit was more likely to be effective when 1. 2. 3. 4. 5.
baseline performance was low the source of feedback was a supervisor or colleague there was feedback in both verbal and written formats feedback was provided more than once clear, explicit targets and an action plan are included.
WHY HAS CLINICAL AUDIT NOT BEEN MORE SUCCESSFUL IN IMPROVING CARE? In the main, audit has all too often failed because it has not brought about change and improvement. This may have arisen for a variety of reasons; the question may not be clear; data collected may not be objective or valid. There may have been poor and slow feedback of results. Even when issues such as these have been tackled and the information is of high quality and informative, it may be that other barriers to change have not been recognised or addressed. The most common barriers that block change are behavioural. It needs to be clearly understood at the outset of any clinical audit project that change requires behaviours to change and that data alone are usually not sufficient to change behaviour. Audit, and feedback, has a relatively limited effect probably because it needs to engage more realistically and in a more focused way with overcoming resistance to changing behaviours. The relative failure of clinical audit in the way that it has usually been promoted as an audit cycle has led to a more explicit focus on change and QI. Clinical medicine and surgery have increasingly looked to developments in QI in other areas such as aviation or the nuclear industry as well as to the QI techniques developed for manufacturing industries. One obvious example of this shift has been the increasing use of small rapid cycle audits as a method for more quickly implementing change and improving care. CLINICAL AUDIT IN THE FUTURE Currently, clinical audit may be viewed as having a number of roles (box 4). Underpinning them all is recognition that there is a need to evaluate the
Box 4 audit
Current thinking about the role of clinical
1. An approach to obtaining essential data to enable quality improvement 2. A technique for quality improvement 3. A way to get data to support revalidation assurance 4. A set of tools to allow monitoring of quality of healthcare.
performance and outcomes of healthcare systems. This is linked to a widespread acceptance that if healthcare is to improve then reliable data on current performance is a fundamental building block. It is over a hundred years since the great Victorian physicist Lord Kelvin recognised the importance of measuring data when he said: ‘If you cannot measure it you cannot improve it’. It has taken a long time for these insights to become accepted currency in the NHS. Providing data about performance for revalidation is likely to be an important future driver. However, as outlined above, information on its own is rarely effective at changing behaviour. In the future, we need to explore how we might better use the energy and expertise of patients, parents and carers in using audit to deliver change. One example is the increasing use of tools such as PROMS and patient-related experience measures in audits. Audit as a tool for driving improvement also needs to work across healthcare settings, or ‘organisational boundaries’. In the UK, Healthcare Quality Improvement Partnership (HQIP) funds and supports multisite audits that are undertaken in 10 or more different sites, each of which may have different information technology systems. The Virginia Mason Health system in the USA uses ‘value stream mapping’ to analyse current state and design a future state for the series of events that take a product or service from its beginning through to the patient. An example of this is a joint replacement pathway, which starts in primary care.11 Value stream mapping encourages collaboration across organisational boundaries. Reflecting this shift of focus from audit to QI, organisations such as HQIP and NHS Improvement (http://www.improvement.nhs.uk/Default.aspx?alias= www.improvement.nhs.uk/qipp) are making highquality tools available to facilitate QIs in healthcare as well as supporting clinical audit. A list of useful resources can be found in box 5. As part of its aim of raising standards of healthcare for children, the RCPCH Clinical Standards team facilitate a number of high-quality national and multisite audits to help improve patient care and outcomes. These audits enable paediatricians to evaluate their practice against national standards (table 1).
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
87
Quality improvement Box 5
Useful resources for clinical audit
▸ A range of implementation tools to help put National Institute for Health and Care Excellence guidance into practice including audit tools. http://www.nice.org.uk/usingguidance/ implementationtools/implementation_tools.jsp ▸ Local Clinical Audit Handbook for Physicians http://www.hqip.org.uk/assets/Guidance/Local-clinicalaudit-handbook-for-physicians-August-2010-FINAL.pdf ▸ Guide to involving junior doctors in clinical audit http://www.hqip.org.uk/assets/5-HQIP-CA-PD-026Guide-to-Involving-Junior-Doctors-in-Clinical-Audit-19April-2010.pdf ▸ The Royal College of Paediatrics and Child Health E-Learning clinical audit training resource http://www.rcpch.ac.uk/e-learning-resources-clinicalaudit ▸ Online journal of clinical audits: an accessible database and reference area to publish and share audit reports http://www.clinicalaudits.com ▸ Transforming Clinical Audit Data Into Quality Improvements learning package http://www.hqip.org.uk/transforming-clinical-auditdata-into-quality-improvements/
CONCLUSION The ongoing improvements in outcomes following the regular publishing of healthcare results such as in the reporting of Cardiac Surgery outcomes in the UK or in the Swedish Hip Arthoplasty Register provide examples of what clinical audit can achieve. However,
88
using data to improve the quality of care delivered to patients requires a commitment at all organisational levels, from international to local, if the goal of improving the quality of healthcare is to be achieved. Competing interests None. Provenance and peer review Commissioned; externally peer reviewed.
REFERENCES 1 Scrivener R, Morrell C, Baker R, et al. Principles for best practice in clinical audit. Abingdon: Radcliffe Medical Press Ltd, 2002. 2 Gill CJ, Gill CG. Nightingale in Scutari: her legacy re-examined. Clin Infect Dis 2005;40:1799–805. 3 Neuhauser D, Codman EA. Qual Saf Health Care 2002;11:104–5. 4 Maxwell RJ. Quality assessment in health. BMJ (Clin Res Ed) 1984;288:1470–2. 5 Learning from Bristol: The report of the public enquiry into children’s heart surgery at the Bristol Royal Infirmary 1984– 1995. Contract No: Command Paper: CM 5207. 2001. 6 Kurtz SM, Ong KL, Schmier J, et al. Future clinical and economic impact of revision total hip and knee arthroplasty. J Bone Joint Surg Am 2007;89(S3):144–51. 7 Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: MS, 2000. 8 Manchikanti L, Flaco FJ, Boswell MV, et al. Facts, fallacies, and politics of comparative effectiveness research: Part I. Basic considerations. Pain Physician 2010;13:E23–54. 9 Keogh B. Management of the national clinical audit and confidential enquiries programme. London, 2012 [July 2013]. http://www.hqip.org.uk/assets/News-uploads/121127-BKLetter.pdf 10 Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;6:CD000259. 11 Kenney C. Transforming health care: Virginia Mason Medical Center’s pursuit of the perfect patient experience. New York, NY: Productivity Press, 2010.
Paton JY, et al. Arch Dis Child Educ Pract Ed 2015;100:83–88. doi:10.1136/archdischild-2013-305194
Copyright of Archives of Disease in Childhood -- Education & Practice Edition is the property of BMJ Publishing Group and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
