Audit-where do we go from here?
Audit- where do
1107
we go
from here?
What is audit? Although in common use, the term medical audit is poorly understood, and can mean different things to different people. Formally, it constitutes a discipline incorporating a cyclical process of evaluation, improvement, and reevaluation. Evaluation in this context is defined by the World Health Organisation (WHO) as 'the systematic and scientific process of determining the extent to which an action or set of actions was successful in the achievement of predetermined objectives'.' Most clinicians would claim that the critical evaluation of the quality of their work is something they have always taken an interest in. For the majority, this would include reviews of their clinical practice in the light of current medical literature, attendance at postgraduate courses, and regular meetings with colleagues, such as occurs in the presentation of cases at regular clinical meetings. Although such forms of review are helpful and may lead to improvements in clinical practice, they do not fulfil the more rigorous requirements of formal medical audit,' which requires that the objectives are clearly stated before audit begins, and that any changes in policy can be implemented and monitored. The WHO definition clearly requires refinement. Firstly, 'predetermined objectives' must be established by an acceptable body of opinion. Secondly, those taking part must be prepared to review critically their work and make appropriate changes to their practice. Thirdly, there must be some form of continuing review to ensure that the agreed changes are maintained. From the above it becomes clear that the only possible approach to quality review in medicine is peer review.2 Yet while accepting the need for quality control to be governed by clinicians, it is appreciated that different interest groups, such as health care economists, health care professionals, politicians, and patients, will have different understandings as to the meaning of quality of care. These differing views must be taken into account by clinicians developing standards for measuring quality of care. It is also recognised that most existing measurements of quality of care are crude,3 4and there is a real danger that what can be easily measured will be used to draw inappropriate conclusions. What is needed is the laborious but important task of developing more robust measures than are currently available. These should have a high level of consensus among health care professionals as well as being understood by and acceptable to the consumers of health care, namely our patients and their families.
Systematic approaches So if our current measures of quality of care are crude, can they be improved? The first step in such a process must be clearly to define the components of the a9jivities involved.`57 Some type of analysis of the actiylties under review must be carried out before audit can'sensibly be applied. In the commercial world, and in particular in the area of industrial production, the three broad headings that are analysed and quality assured are: (i) raw materials and supply, plant, and personnel, (ii) the production processes themselves, and (iii) the final product.5 These concepts have also been applied to health care. Donabedian in particular has analysed the elements of health care, and uses the terms structure, process, and outcome.5 6 This is the approach
recommended in the recent white paper' and by the Royal College of Physicians own report.' Yet there is some doubt as to how far this approach, taken from the world of industry and commerce, is appropriate in a 'business' driven by health needs and expectations. 'Resource management' techniques have a major role in activities such as estate and capital planning, personnel, and supplies, but their introduction into the assessment of quality of care without further modification may not be appropriate. There is among many health care professionals a considerable unease about the appropriateness of applying an industrial model of audit and resource management to the interaction between carer and patient, and the increasing use of 'performance factors' to represent what is an extremely complex process. 1 '9 Against this background, professional bodies representing medical opinion in the UK have sought to strengthen and extend medical audit by producing guidelines for its introduction and conduct.' 'o " In these recommendations the three major headings outlined above have been maintained, namely structure (or resource), process, and outcome audit. While such headings can be helpful, there is a great deal of overlap between them. The interrelationships between resources and process, or process and outcome, must be carefully explored. Only when these complex relationships are described and properly understood, can valid measurements be derived and used to assess the quality of the service.5 Resource Resource audit is relatively easy to understand and to define. It includes the analysis of all resources that are required for the health care process under review, and their comparison with agreed standards. These resources include staff numbers, level of training and experience, the physical environment where the process is carried out, equipment, laboratory and investigative facilities, and available treatments.5 The organisation and conduct of audit itself should also be included under this broad heading. Resource audit is an area in which business style resource management can be applied with some success.2
Outcome Outcome audit has been more difficult to define. While the need for, and the value of, measuring output is undisputed, the most difficult aspect of such audit is the definition of appropriate measures of outcome. Most commonly used is mortality,'3 14 but this is very crude and presents obvious difficulties in diseases with very low mortality.4 The assessment of outcome after neonatal intensive care is an example where further development is required. Crude mortality figures (for example, survival to discharge) give no indication of morbidity. The inclusion of data on long term handicap and other measures of morbidity, can result in very different conclusions.'5 Measures such as subsequent morbidity, handicap, and quality of life, however, are not easy to define and are consequently less amenable to critical analysis than mortality. Such measures are, however, being developed and refined. Measures of quality of life have been developed for
Ross Russell, Helms
1108
adults,9 16 but few are so far available for children. Questions about perceived quality of life, patient satisfaction, and physical functions allow the construction of a comparative hierarchy against which outcome can be assessed.5 Such methods, however, have major difficulties when applied to a paediatric population in which functional outcome is difficult to measure. One fundamental requirement of outcome audit is its reproducibility both between and within observers. This need for reproducibility also restricts the use of descriptive measures of outcome. Even objective data such as mortality can be subject to criticism if confounding variables are not accounted for. Recent work demonstrating that mortality in cystic fibrosis is lower in groups of patients seen in specialist regional centres,'7 has been criticised as being confounded by variations in social class between the groups cared for locally and at regional centres.'8 Process The most difficult, and consequently most poorly defined area of care is that of the process of health care itself. The current application of audit to this area, at least as far as hospital medicine is concerned, is almost entirely based on reviews of case notes. Reviews of diagnoses or management decisions are based on the retrospective interpretation of the notes, rather than prospective assessment of the physicians' performance. Such prospective assessments are not made because the appropriate measures have not been clearly defined. This is hardly surprising as verifiable and meaningful measures cannot be produced until the clinical process itself has been analysed and defined. Most of the early attempts at such analysis-that is, the detailed description of those activities that are involved in health care-have been undertaken by the nursing profession. Work by Henderson, Orem, and Roper has had a great impact on the nursing approach to health care by attempting to define the role of the nurse and by describing the nursing process in a systematic way. 19-21 Severity scoring systems may also have a place in measuring the process of health care.4 22 These have developed measures of illness severity and treatments given to allow a prognosis to be given for a patient. Although they have generally been used to measure outcome, their adaptation may provide a useful measure of interventions that occur in clinical practice,'2 and this could be used to evaluate the care given to any particular patient. Analysis has only very recently been applied to the medical aspects of health care. This is partly due to the complexity of the subject, but also reflects a failure by many clinicians to understand the importance of a systematic analysis of their activities. No clinician in this age would accept drug treatments that had not been subjected to rigorous safety checks and clinical trials. Yet we are often far less rigorous assessing other aspects of care. For example, the management of a patient's fluid balance, the formulation of a care plan, or the analysis of a radiograph are the kind of activities that are based on personal experience coupled with established medical practice. As such they are consequently difficult to analyse as they may be idiosyncratic and unique to a particular group, or to an individual clinician. Analysis of the process of medical care is needed so that agreed measures can be established and valid comparisons made between different disease states and managements. Such modelling should not enforce a 'standard approach' but rather identify the components of the clinical
process which are held in common across a wide range of disease states and medical management styles.
Conclusion The term audit can mean different things to different people, and therefore its objectives and conduct must be agreed between all parties involved before it is established. It has great importance to practising clinicians, who have an obligation to review critically and maintain the quality of the service they provide. Whether we like it or not, we are all going to become involved in the systematic audit of our work. However, we must continue to develop scientifically sound measures of health care that truthfully reflect the work carried out. With the financial pressure that is being increasingly put on clinicians to justify the escalating cost of health care, there is a real risk that purely economic considerations may be used to apportion funding, which may be to the detriment of the quality of care we can provide. To avoid this we must demonstrate the relationship between quality and quantity of care. The way ahead must be to learn from the experience of systematic clinical audit (with all its limitations), to continue careful and thorough analysis of all the processes of health care, and to develop reliable and reproducible measures of outcome. R I ROSS RUSSELL P J HELMS Respiratory Unit, Hospital for Sick Children, Great Ormond Street, London WCIN 33tH
1 Royal College of Physicians. Medical Audit. A first report. What, why and how? London: Royal College of Physicians, 1989. 2 Quality assurance/medical audit. A methodology produced for the NHS Information Management Centre. London: C International, 1990. (Available from the NHS IMC, 19 Calthorpe Road, Birmingham B15 lRP.) 3 Weinstein MC, Stason WB. Foundations of cost effectiveness analysis for health and medical practices. N Engl J Med 1977;296:716-21. 4 Pollack MM, Puttimann VE, Getson PR. Accurate prediction of the outcome of paediatric intensive care. A new quantitative method. N Engl J Med 1987;316:134-9. 5 Donabedian A. The quality of care. How can it be assessed?JAMA 1988;260: 1743-8. 6 Donabedian A. The definition of quality and approaches to its management. Vol 1: Explorations in quality assessment and monitoring. Ann Arbor, Michigan: Health Administration Press, 1980. 7 Shaw CD. Clinical outcome indicators. Health Trends 1989;21:37. 8 Secretaries of State for Health, Wales, Northern Ireland, and Scotland. WorkIng for patients. London: HMSO, 1989. (Cmnd 555.) 9 Patrick DI, Bush JW, Chen MM. Methods for measuring levels of well-being for a health status index. Health Serv Res 1973;8:228-45. 10 Measurement ofChild Health. Report of a working party of the Executive Committee of the Community Paediatric Group. London: HMSO, 1989. 11 Sheldon MG. Medical audit in general practice. London: Royal College of General Practitioners, 1982. (Occasional paper 20.) 12 Pollack MM, Wilkinson JD, Glass NL. Long stay pediatric intensive care unit patients: outcome and resource utilization. Pediatrics 1987;80:855-60. 13 Rothstein P, Johnson P. Paediatric intensive care: factors that influence outcome. Crit Care Med 1982;10:34-7. 14 Schroeder SA. Outcome assessment 70 years later: are we ready? N Engl . Mcd 1987;316:160. 15 Pharoah POD, Stevenson RC, Cooke RWI, Sandu B. Costs and benefits of neonatal intensive care. Arch Dis C'hild 1988;63:715-8. 16 Stewart AL, Wate JE Jr, Brook RM. Advances in the measurement of functional states: construction of aggregate indexes. Med Care 1981;19:473-88. 17 Hill DJS, Martin AJ, Davidson GP, Smith GS. Survival of cystic fibrosis patients in South Australia. Med I Aust 1985;143:230-2. 18 Britton JR. Effects of social class, sex and region of residence on age at death from cystic fibrosis. Br Med 7 1989;298:483-7. 19 Henderson V. The nature of nursing: a definition and its implications for practice research and education. New York: McMillan, 1966. 20 Orem D. Nursing: concepts of practice. 3rd Ed. New York: McGraw Hill, 1985. 21 Roper N. Principles of nursing in process context. 4th Ed. Edinburgh: Churchill Livingstone, 1988. 22 Price HL, Matthew DJ. Evaluation of pediatric intensive care scoring systems. Intensive Care Med 1989;15:79-83.
Audit- where do
1107
we go
from here?
What is audit? Although in common use, the term medical audit is poorly understood, and can mean different things to different people. Formally, it constitutes a discipline incorporating a cyclical process of evaluation, improvement, and reevaluation. Evaluation in this context is defined by the World Health Organisation (WHO) as 'the systematic and scientific process of determining the extent to which an action or set of actions was successful in the achievement of predetermined objectives'.' Most clinicians would claim that the critical evaluation of the quality of their work is something they have always taken an interest in. For the majority, this would include reviews of their clinical practice in the light of current medical literature, attendance at postgraduate courses, and regular meetings with colleagues, such as occurs in the presentation of cases at regular clinical meetings. Although such forms of review are helpful and may lead to improvements in clinical practice, they do not fulfil the more rigorous requirements of formal medical audit,' which requires that the objectives are clearly stated before audit begins, and that any changes in policy can be implemented and monitored. The WHO definition clearly requires refinement. Firstly, 'predetermined objectives' must be established by an acceptable body of opinion. Secondly, those taking part must be prepared to review critically their work and make appropriate changes to their practice. Thirdly, there must be some form of continuing review to ensure that the agreed changes are maintained. From the above it becomes clear that the only possible approach to quality review in medicine is peer review.2 Yet while accepting the need for quality control to be governed by clinicians, it is appreciated that different interest groups, such as health care economists, health care professionals, politicians, and patients, will have different understandings as to the meaning of quality of care. These differing views must be taken into account by clinicians developing standards for measuring quality of care. It is also recognised that most existing measurements of quality of care are crude,3 4and there is a real danger that what can be easily measured will be used to draw inappropriate conclusions. What is needed is the laborious but important task of developing more robust measures than are currently available. These should have a high level of consensus among health care professionals as well as being understood by and acceptable to the consumers of health care, namely our patients and their families.
Systematic approaches So if our current measures of quality of care are crude, can they be improved? The first step in such a process must be clearly to define the components of the a9jivities involved.`57 Some type of analysis of the actiylties under review must be carried out before audit can'sensibly be applied. In the commercial world, and in particular in the area of industrial production, the three broad headings that are analysed and quality assured are: (i) raw materials and supply, plant, and personnel, (ii) the production processes themselves, and (iii) the final product.5 These concepts have also been applied to health care. Donabedian in particular has analysed the elements of health care, and uses the terms structure, process, and outcome.5 6 This is the approach
recommended in the recent white paper' and by the Royal College of Physicians own report.' Yet there is some doubt as to how far this approach, taken from the world of industry and commerce, is appropriate in a 'business' driven by health needs and expectations. 'Resource management' techniques have a major role in activities such as estate and capital planning, personnel, and supplies, but their introduction into the assessment of quality of care without further modification may not be appropriate. There is among many health care professionals a considerable unease about the appropriateness of applying an industrial model of audit and resource management to the interaction between carer and patient, and the increasing use of 'performance factors' to represent what is an extremely complex process. 1 '9 Against this background, professional bodies representing medical opinion in the UK have sought to strengthen and extend medical audit by producing guidelines for its introduction and conduct.' 'o " In these recommendations the three major headings outlined above have been maintained, namely structure (or resource), process, and outcome audit. While such headings can be helpful, there is a great deal of overlap between them. The interrelationships between resources and process, or process and outcome, must be carefully explored. Only when these complex relationships are described and properly understood, can valid measurements be derived and used to assess the quality of the service.5 Resource Resource audit is relatively easy to understand and to define. It includes the analysis of all resources that are required for the health care process under review, and their comparison with agreed standards. These resources include staff numbers, level of training and experience, the physical environment where the process is carried out, equipment, laboratory and investigative facilities, and available treatments.5 The organisation and conduct of audit itself should also be included under this broad heading. Resource audit is an area in which business style resource management can be applied with some success.2
Outcome Outcome audit has been more difficult to define. While the need for, and the value of, measuring output is undisputed, the most difficult aspect of such audit is the definition of appropriate measures of outcome. Most commonly used is mortality,'3 14 but this is very crude and presents obvious difficulties in diseases with very low mortality.4 The assessment of outcome after neonatal intensive care is an example where further development is required. Crude mortality figures (for example, survival to discharge) give no indication of morbidity. The inclusion of data on long term handicap and other measures of morbidity, can result in very different conclusions.'5 Measures such as subsequent morbidity, handicap, and quality of life, however, are not easy to define and are consequently less amenable to critical analysis than mortality. Such measures are, however, being developed and refined. Measures of quality of life have been developed for
Ross Russell, Helms
1108
adults,9 16 but few are so far available for children. Questions about perceived quality of life, patient satisfaction, and physical functions allow the construction of a comparative hierarchy against which outcome can be assessed.5 Such methods, however, have major difficulties when applied to a paediatric population in which functional outcome is difficult to measure. One fundamental requirement of outcome audit is its reproducibility both between and within observers. This need for reproducibility also restricts the use of descriptive measures of outcome. Even objective data such as mortality can be subject to criticism if confounding variables are not accounted for. Recent work demonstrating that mortality in cystic fibrosis is lower in groups of patients seen in specialist regional centres,'7 has been criticised as being confounded by variations in social class between the groups cared for locally and at regional centres.'8 Process The most difficult, and consequently most poorly defined area of care is that of the process of health care itself. The current application of audit to this area, at least as far as hospital medicine is concerned, is almost entirely based on reviews of case notes. Reviews of diagnoses or management decisions are based on the retrospective interpretation of the notes, rather than prospective assessment of the physicians' performance. Such prospective assessments are not made because the appropriate measures have not been clearly defined. This is hardly surprising as verifiable and meaningful measures cannot be produced until the clinical process itself has been analysed and defined. Most of the early attempts at such analysis-that is, the detailed description of those activities that are involved in health care-have been undertaken by the nursing profession. Work by Henderson, Orem, and Roper has had a great impact on the nursing approach to health care by attempting to define the role of the nurse and by describing the nursing process in a systematic way. 19-21 Severity scoring systems may also have a place in measuring the process of health care.4 22 These have developed measures of illness severity and treatments given to allow a prognosis to be given for a patient. Although they have generally been used to measure outcome, their adaptation may provide a useful measure of interventions that occur in clinical practice,'2 and this could be used to evaluate the care given to any particular patient. Analysis has only very recently been applied to the medical aspects of health care. This is partly due to the complexity of the subject, but also reflects a failure by many clinicians to understand the importance of a systematic analysis of their activities. No clinician in this age would accept drug treatments that had not been subjected to rigorous safety checks and clinical trials. Yet we are often far less rigorous assessing other aspects of care. For example, the management of a patient's fluid balance, the formulation of a care plan, or the analysis of a radiograph are the kind of activities that are based on personal experience coupled with established medical practice. As such they are consequently difficult to analyse as they may be idiosyncratic and unique to a particular group, or to an individual clinician. Analysis of the process of medical care is needed so that agreed measures can be established and valid comparisons made between different disease states and managements. Such modelling should not enforce a 'standard approach' but rather identify the components of the clinical
process which are held in common across a wide range of disease states and medical management styles.
Conclusion The term audit can mean different things to different people, and therefore its objectives and conduct must be agreed between all parties involved before it is established. It has great importance to practising clinicians, who have an obligation to review critically and maintain the quality of the service they provide. Whether we like it or not, we are all going to become involved in the systematic audit of our work. However, we must continue to develop scientifically sound measures of health care that truthfully reflect the work carried out. With the financial pressure that is being increasingly put on clinicians to justify the escalating cost of health care, there is a real risk that purely economic considerations may be used to apportion funding, which may be to the detriment of the quality of care we can provide. To avoid this we must demonstrate the relationship between quality and quantity of care. The way ahead must be to learn from the experience of systematic clinical audit (with all its limitations), to continue careful and thorough analysis of all the processes of health care, and to develop reliable and reproducible measures of outcome. R I ROSS RUSSELL P J HELMS Respiratory Unit, Hospital for Sick Children, Great Ormond Street, London WCIN 33tH
1 Royal College of Physicians. Medical Audit. A first report. What, why and how? London: Royal College of Physicians, 1989. 2 Quality assurance/medical audit. A methodology produced for the NHS Information Management Centre. London: C International, 1990. (Available from the NHS IMC, 19 Calthorpe Road, Birmingham B15 lRP.) 3 Weinstein MC, Stason WB. Foundations of cost effectiveness analysis for health and medical practices. N Engl J Med 1977;296:716-21. 4 Pollack MM, Puttimann VE, Getson PR. Accurate prediction of the outcome of paediatric intensive care. A new quantitative method. N Engl J Med 1987;316:134-9. 5 Donabedian A. The quality of care. How can it be assessed?JAMA 1988;260: 1743-8. 6 Donabedian A. The definition of quality and approaches to its management. Vol 1: Explorations in quality assessment and monitoring. Ann Arbor, Michigan: Health Administration Press, 1980. 7 Shaw CD. Clinical outcome indicators. Health Trends 1989;21:37. 8 Secretaries of State for Health, Wales, Northern Ireland, and Scotland. WorkIng for patients. London: HMSO, 1989. (Cmnd 555.) 9 Patrick DI, Bush JW, Chen MM. Methods for measuring levels of well-being for a health status index. Health Serv Res 1973;8:228-45. 10 Measurement ofChild Health. Report of a working party of the Executive Committee of the Community Paediatric Group. London: HMSO, 1989. 11 Sheldon MG. Medical audit in general practice. London: Royal College of General Practitioners, 1982. (Occasional paper 20.) 12 Pollack MM, Wilkinson JD, Glass NL. Long stay pediatric intensive care unit patients: outcome and resource utilization. Pediatrics 1987;80:855-60. 13 Rothstein P, Johnson P. Paediatric intensive care: factors that influence outcome. Crit Care Med 1982;10:34-7. 14 Schroeder SA. Outcome assessment 70 years later: are we ready? N Engl . Mcd 1987;316:160. 15 Pharoah POD, Stevenson RC, Cooke RWI, Sandu B. Costs and benefits of neonatal intensive care. Arch Dis C'hild 1988;63:715-8. 16 Stewart AL, Wate JE Jr, Brook RM. Advances in the measurement of functional states: construction of aggregate indexes. Med Care 1981;19:473-88. 17 Hill DJS, Martin AJ, Davidson GP, Smith GS. Survival of cystic fibrosis patients in South Australia. Med I Aust 1985;143:230-2. 18 Britton JR. Effects of social class, sex and region of residence on age at death from cystic fibrosis. Br Med 7 1989;298:483-7. 19 Henderson V. The nature of nursing: a definition and its implications for practice research and education. New York: McMillan, 1966. 20 Orem D. Nursing: concepts of practice. 3rd Ed. New York: McGraw Hill, 1985. 21 Roper N. Principles of nursing in process context. 4th Ed. Edinburgh: Churchill Livingstone, 1988. 22 Price HL, Matthew DJ. Evaluation of pediatric intensive care scoring systems. Intensive Care Med 1989;15:79-83.
