745
MEDICAL AUDIT* JOHN GORDON FREYMANN, M.D. Director of Education Hartford Hospital Hartford, Conn.
I SHALL begin by shooting down my title. I think it should be killed and buried. Let us examine the definition of the word audit. According to the Oxford Dictionary, it means: I) a hearing, especially a judicial hearing of complaints; 2) an official examination of accounts; 3) a periodic settlement between a landlord and tenant; or 4) a balance sheet prepared for an auditor. These definitions all have negative implications; they imply that the question we are asking is, "How bad is medical practice?" On the contrary, most doctors believe the general quality of medical practice is good. I think it is. As scientists, however, we should insist on proof. Therefore, the question we pose should be, "How good is medical practice?" Then we should ask, "Are there discrepancies from our standards of excellence and, if so, why?" Let us look for words to reflect this positive approach. Assessment, a term commonly used in the context of my title, is no better than audit. It means either determining the amount of a fine or tax or evaluating for purposes of taxation. Evaluation is better; it has the positive meaning of working out the value of something. But I favor appraisal because it has an even more positive implication. Appraisal means estimating worth or value and comes from the same root as praise and pride. I do not approve of the adjective medical in my title either. This excludes nursing. How can we appraise the care of patients on a coronary care unit, to take one example, without considering the contributions of nursing? Further, if nursing does not join with medicine in appraising the care of patients, a duplicate bureaucracy will soon be set up for nursing audits. Patient-care appraisal is therefore a better *Presented as part of a Symposium on Continuing Medical Education held by the Committee on Medical Education of the New York Academy of Medicine October 10, 1974.
Vol. 51, No. 6, June 1975
74 6
J. G. FREYMANN 746~~ J.G
REMN
term, but I am still not satisfied. Patient is derived from the Latin patior, to suffer, but we should strive to care for people before they reach the stage of suffering. We should be working to keep people healthy. Therefore the title I prefer is "Health-Care Appraisal." The appraisal of health care can be carried out in many ways, but I shall not describe these since many have already been mentioned by other speakers in this symposium. (An excellent summary of the many faces of health-care appraisal appeared in a supplement to the November 1973 issue of the Journal of Medical Education.J) Instead, I shall devote the balance of this paper to seven topics relating to this subject. Some are caveats and some are commentaries. They are not necessarily listed in order of importance. All of them should be considered as medicine in the United States embarks on this new adventure. Two PHILOSOPHIES OF HEALTH-CARE APPRAISAL Health-care appraisal may be based on one of two antithetical philosophies. One of these may be dubbed the welfare approach; the other the Internal Revenue Service (IRS) approach. The underlying philosophy of our welfare system is that everyone is basically dishonest and out to defraud the government. The welfare approach seems to be built into the general philosophy of the Professional Standards Review Organizations (PSROs): namely, requirements that individual physicians conform to standard loci of treatment, lengths of stay, number of visits, services used for specific diagnoses, etc., and that bureaucratic approval must be given for every deviation. If this philosophy is allowed to pervade health-care appraisal, it will mean that every physician must be checked individually. Further, this approach is backward-oriented-based on how things are done today. It would not only proceed on a basic assumption of incompetence but would also limit the practice of medicine to the status quo ante 1975. This has not happened yet, but PSROs could degenerate into this state of affairs. Chapter VII of the PSRO Mamnal gives us a way to avoid the welfare approach. This section of the PSRO regulations specifically states that positively-oriented systems of appraisal (the American Hospital Association's Quality Assurance Program is one example) are acceptable ways to evaluate health care. Such systems take the IRS approach: i.e., the IRS assumes that the vast majority of people are honest. This Bull. N. Y. Acad. Med.
MEDICAL AUDIT
MEDICAL
AUDIT
74 7 747
philosophy permits the practices of a number of physicians to be appraised collectively. Criteria set by the physicians themselves are based on clinical judgment, not on the standards of accountants. Significant discrepancies from these present criteria are noted instead of individual infractions, and these discrepancies are met with corrective programs of continuing education instead of punitive action. Further, since such criteria can be revised to include new advances in care, this philosophy of health-care appraisal is forward-looking-oriented toward a continuous improvement in care.I must caution my fellow physicians that the IRS approach is inconsistent with traditional medical education, which tends to concentrate on exceptions to the rules, and with our Hippocratic concern for individual patients. To avoid the pitfalls of the welfare philosophy, we must re-educate ourselves to view health care as people in the field of public health do. We must concentrate on the peak of the bellshaped distribution curve and assure ourselves that most patients are well treated; we must avoid the temptation to devote all our attention to the few patients who fall at the far ends of the distribution curve. HEALTH-CARE APPRAISAL MUST BE KEPT SEPARATE FROM COST CONTROL Recent federal legislation (PL 92-603) has placed the responsibility for monitoring the costs of health care on physicians instead of on administrators. This is good because if quality care costs more than the public bargains for physicians are in a better position than administrators to say so and to prove it. Proof is vital. If we cannot prove our point we shall have been guilty of an inadequate appraisal of health care. Cost controls are essential, of course, but cost control and healthcare appraisal are quite different. They are the numerator and denominator of the cost/benefit equation. This equation cannot be solvedi.e., we cannot measure benefits derived against the money expendedif cost indicators such as length of hospital stay, number of visits by physicians, and services used are equated with the quality of care. If they were equated, the numerator and denominator would be identical, the equation would equal i, and cost would determine what quality is because benefit could be defined only in terms of how much we can afford. We must not adopt fiscal definitions as the new vocabulary for Vol. 51, No. 6, June 1975
74 8
748
J.
G. FREYMANN G.
FREYMANN
describing health care. If we do, we shall have been co-opted as fiscal agents for fiscal intermediaries and the Social Security Administration. PITFALLS OF HEALTH-CARE APPRAISAL BASED ON DIAGNOSTIC CATEGORIES For 40 years fiscal intermediaries have protected their budgets by categorizing care. The accountants prefer every patient to have a single diagnosis so that they can challenge the treatment of this diagnosis on a service-by-service basis. Lawson3 observed that this is a "restricted view of what disease is humanly about." He has shown that the elderly average five to six diagnoses per person, and Dr. Thomas C. Meyer showed in this symposium that between 20% and 5o% of the patients of the doctors whose practices he studied could be categorized under no pathological entity or nosologic syndrome.4 Health care should be appraised as it is really delivered-in terms of problems. The surgical specialties have an easier job because they can often evaluate the results of specific procedures instituted to treat specific diagnoses, but medicine, pediatrics, and family medicine are not so fortunate. In these fields most patients present problems such as chest pain, failure to thrive, or anxiety, for which the important part of the appraisal is not only what the final diagnosis was but also how the problem was managed. To complicate things further, few patients present a single problem. We must therefore resist the pressures of officialdom and refuse to collapse multiple problem lists into a single principal diagnosis just to satisfy the accountants. What will happen if we appraise health care exclusively by diagnostic categories? First, we shall separate the treatment of disease from the social causes and consequences of disease, which should be the physician's equal concern. Second, we shall make it easier to justify expensive hospitalization than chronic care in extended-care facilities and in the home. Third, preventive medicine will be neglected or entirely ignored. And, finally, we shall limit health care to the treatment of episodic crises and emphasize specialization precisely at the time we are trying to attract 50% of the graduates of our medical schools into primary care.
PROCESS APPRAISAL VERSUS OUTCOME APPRAISAL We physicians have traditionally been preoccupied with the procBull. N. Y. Acad. Med.
MEDICAL AUDIT MEDICAL AUDIT
74 9 749~~~~~~~~~~~~~~~
ess of medical care. It was drummed into us during our education. Rounds and conferences are rituals for worshiping the medical process. As students, house officers, and practicing physicians we have been judged by our superiors and our peers on the way we carry out the processes we were so carefully taught. Perhaps there is a deeper reason for this worship of process. I believe that the ultimate reason is a blind faith that whatever we do will have a good outcome-so long as we do it correctly. One need not turn to the recent literature on health-care appraisal to demonstrate this point. Doctors have always been sure that they were doing good. In i825, for example, Macmichael confidently gave "advances in medicine" credit for the 500o drop in the British death rate between I750 and i825." We now know that this fall in the death rate was due to many causes, among them the invention of the flush toilet and cast-iron pipe, and that all that medicine contributed during this period was vaccination against smallpox. (This preventive practice accounted for only about 6% of the fall in the death rate.6) The public has had the same simple faith in us. Once again, one need not look to recent examples. During the yellow fever epidemic of 1793 Philadelphians lined up before dawn to see Benjamin Rush, the most prominent physician of his day, even though the bleeding and purging he used to treat this dread disease was described by a discerning colleague as "one of those great discoveries which are made from time to time for the depopulation of the earth."7 Going back even further, in iSi8 Erasmus wrote an essay, In Praise of the Healing Arts, which reflected the same touching faith in the power of physicians:8 The special glory of the healing arts is self-sufficient and recommends itself to mankind by its value and utility. . . . Many sicknesses have such a power that death is the sure fate of a patient if the doctor does not interfere immediately. Now let us return to the present and to the process of medical care. Is there any proof that the process has anything to do with its outcome? In a study of appendicitis and acute myocardial infarction in three hospitals of the Kaiser-Permanente system no correlation could be found between the processes by which care was given and the outcome of that care.9 Brook and Appel dropped a bombshell in I973.10 They used five different review methods to evaluate the care of patients with urinary tract infections, hypertension, or ulcerated intestinal disVol. 51, No. 6, June 1975
750
J.
G. FREYMANN
ease. When they used process criteria determined by experts-the method of health-care appraisal which is widely touted as the ultimate ideal-only 2% of the patients were judged to have received adequate care. However, when evaluators were asked if the outcome of treatment would have been improved if the process had been better, 63% of the patients were thought to have received an acceptable quality of care. The authors and an accompanying editorial by Williamson" cautioned that undue emphasis should not be placed on this single experiment, but that it does raise serious questions as to whether an evaluation of the process has any place in the appraisal of health care. Further investigation may show that long laundry lists of procedures have little connection with the quality of the outcome. However, if these lists were standardized they could become blueprints for litigation if they were not followed slavishly-at immense cost.12 Is EVALUATION OF OUTCOME PRACTICAL? If the process does not clearly correlate with the outcome of health care, then it seems logical that health-care appraisal should concentrate on outcomes. However, this is easier said than done. First, physicians in practice have not been accustomed to measuring the outcomes of their treatment except in one area: cancer. More than 8oo hospitals have cancer registries, but for some reason the evaluation of long-term outcome of treatment has not been extended to other diseases except in research projects. Perhaps this reluctance to measure outcomes reflects the blind confidence already mentioned: that whatever we do must be beneficial. Second, factors unrelated to treatment-genetic heritage, family circumstances, economic status, etc.-may be equally, if not more important to ultimate outcomes. Measuring intermediate outcomes is not as complicated, but these may not give accurate appraisals of health care because the points for evaluation must be chosen arbitrarily. For example, among 30I patients on whom ventricular defibrillation was attempted in emergency vehicles, a recent study showed that i99 had reverted to normal rhythms, ioi lived to reach hospitals, 42 were discharged alive, and 2I were still alive 12 months after discharge. Each of the last four figures represented an intermediate outcome, but the value of the initial treatment declines progressively as each is considered. Numerous solutions to the conundrum of the evaluation of results Bull. N. Y. Acad. Med.
MEDICAL AUDIT
MEDICAL
75I 75'
have been proposed. One that seems promising is what Sanazaro and Williamson call the "Six D's."114 Of the first three of these, death, is easily measured, and disease and disability are reasonably easy to weigh. However the second three-discomfort, dissatisfaction, and social disruption-become progressively harder to evaluate. These difficulties in measuring outcomes lead into my next topic. How FAR CAN THE QUANTIFICATION OF HEALTH CARE BE CARRIED? Here we have a paradox. McKeown" showed that before 1920 there was little evidence of medicine's efficacy, at least insofar as can be determined from death rates-yet medicine is a profession as old as the priesthood. Why? I believe it is because society-any societydemands its shaman. A shaman is someone set apart by society because people believe that he has the power to cure, to ward off death. Whether or not the shaman is really so endowed seems less important. (Why else do we still have quacks?) If outcomes had been appraised regularly 50 years ago, medicine would undoubtedly have been proven ineffective. What would have happened? Would medicine have folded its tent? No, it would certainly not have done so then. Further, I do not believe it would now. Medicine must realize that its social mandate is still based largely on its role as shaman, and that this role relates only incidentally to the results of treatment. In other words, the quantification of our efficacy can be carried only so far, but society's need for medicine extends far beyond that point. We must keep our importance to society in historical perspective. If we do not, society may discover that the profession it has honored for millennia has been replaced in the last generation by a class of technologists who still call themselves physicians but are not what the people think physicians should be. Society might then transfer its mandate to others-nurse practitioners or Dr. John W. Williamson's health accountants?"'-who do fulfill the people's primordial need for a shaman.
AVOID ADOPTING ANY FORM OF HEALTH-CARE APPRAISAL UNTIL IT IS VALIDATED American doctors are embarking on the first attempt in 3,000 years of medical history to quantify their actual benefit to society. Nothing Vol. 51, No. 6, June 1975
7 5 2
J. G. FREYMANN
like this has been done before. It will take time and a rigorous application of the scientific method to see if the evaluation of care can be measured and, if it can, how best to do it. Further, the avowed reason for doing it is to enhance the quality of care, but we have not yet proved that health-care appraisal will accomplish this feat. In a study of breast biopsies in Ottawa, Devitt concluded that appraisal followed by remedial education probably improves the quality of care.'5 The data presented by Dr. Williamson here"' looks convincing, at least in one specific diagnostic category. (I cannot help but note, however, that education of the patients, not the physicians, seems to have been responsible for most of the benefit.) Other studies have failed to show any effect. At Yale-New Haven Hospital an appraisal of cholecystectomy and inguinal herniorrhaphy by process and end-results revealed deficiencies; after appropriate educational measures were taken, a reappraisal failed to show any improvement.17 Drs. George E. Miller and Williamson themselves showed in a classic study that the loop between measured discrepancies and improved care is difficult to close with education, even when the problem is as simple as a failure to read urinanalysis reports.'8 Medicine must not overlook the fact that health-care appraisal itself must be validated. Medicine must resist the temptation to grasp at some arbitrary technique because (as one so often hears) "If we don't do something the government will do it for us." If we do this we shall become our own worse enemies, and our patients' as well. It is up to the medical profession not to do as it is told by accountants and bureaucrats but to experiment and develop valid methods of health-care appraisal. Only then will doctors be able to answer the three questions Mager asked in his verse:'9 Where am I going, How shall I get there, And how will I know I've arrived? REFERENCES 1. Barro, A. R.: Survey and evaluation of approaches to physician performance measurements. J. Med. Educ. 48:105193, 1973. 2. Garrell, M. and Hamaty, D.: Professional standards review organizations
and quality insurance programsThey're not the same. Conn. Ned. 38: 430-32, 1974. 3. Lawson, I. R.: Professional standards review organization and care of the elderly. J.A.M.A. 229 :311-13, 1974.
Bull. N. Y. Acad. Med.
MEDICAL AUDIT 4. Meyer, T. C.: Toward a continuum in medical education. Bull. N.Y. Acad. Med. 51:1975. 5. Macmichael, W.: The Gold-Headed Cane. Springfield, Ill., Thomas, 1953, pp. 175-76. 6. McKeown, T.: Medicine in Modern Society. New York, Hafner, 1966, p. 52. 7. Shryock, R. H.: Medicine in America: Historical Essays. Baltimore, Johns Hopkins Press, 1966, p. 156. 8. Erasmus, D.: In Praise of the Healing Arts. Translated in: Desiderius Erasmus (1466-1536), the great humanist. J.A.M.A. 210:1587-88, 1969. 9. Fessel, W. J. and VanBrunt, E. E.: Assessing quality of care from the medical record. New Eng. J. Med. 286: 134-38, 1972. 10. Brook, R. H. and Appel, F. A.: Quality-of-care assessment: Choosing a method for peer review. New Eng. J. Med. 288:1323-29, 1973. 11. Williamson, J. W.: Evaluating the quality of medical care. New Eng. J. Med. 288:1352-53, 1973. 12. Emerson, R. S.: The surgical audit (Yale-New Haven). Conn. Med. 38:
Vol. 51, No. 6, June 1975
7 5 3
116, 1974. 13. Liberthson, R. R., Eugene, N. L., Hirschman, J. C., and Nussenfeld, J. D.: Prehospital ventricular defibrillation. New Eng. J. Med. 291:317-21, 1974. 14. Sanazaro, P. J. and Williamson, J. W.: End result of patient care: A provisional classification based on reports of internists. Med. Care 6:123-30, 1968. 15. Devitt J. E.: Does continuing medical education by peer review really work? Canad. Med. Ass. J. 108:1279-81, 1973. 16. Williamson, J. W.: Patient care assessment. Bull. N.Y. Acad. Med. 51: 1975. 17. Cayton, C. G., Tanner, L. A., Riedel, D. C., and Williams, K. H., Jr.: Surgical audit using predetermined weighted criteria. Conn. Med. 38:117-22, 1974. 18. Williamson, J. W., Alexander, M., and Miller, G. E.: Continuing education and patient care research. J.A.M.A. 201:938-42, 1967. 19. Mager, R. F. (quoted by Brook, R. H.): Quality of care assessment: The role of faculty at academic medical centers. Clin. Res. 22:122-27, 1974.
MEDICAL AUDIT* JOHN GORDON FREYMANN, M.D. Director of Education Hartford Hospital Hartford, Conn.
I SHALL begin by shooting down my title. I think it should be killed and buried. Let us examine the definition of the word audit. According to the Oxford Dictionary, it means: I) a hearing, especially a judicial hearing of complaints; 2) an official examination of accounts; 3) a periodic settlement between a landlord and tenant; or 4) a balance sheet prepared for an auditor. These definitions all have negative implications; they imply that the question we are asking is, "How bad is medical practice?" On the contrary, most doctors believe the general quality of medical practice is good. I think it is. As scientists, however, we should insist on proof. Therefore, the question we pose should be, "How good is medical practice?" Then we should ask, "Are there discrepancies from our standards of excellence and, if so, why?" Let us look for words to reflect this positive approach. Assessment, a term commonly used in the context of my title, is no better than audit. It means either determining the amount of a fine or tax or evaluating for purposes of taxation. Evaluation is better; it has the positive meaning of working out the value of something. But I favor appraisal because it has an even more positive implication. Appraisal means estimating worth or value and comes from the same root as praise and pride. I do not approve of the adjective medical in my title either. This excludes nursing. How can we appraise the care of patients on a coronary care unit, to take one example, without considering the contributions of nursing? Further, if nursing does not join with medicine in appraising the care of patients, a duplicate bureaucracy will soon be set up for nursing audits. Patient-care appraisal is therefore a better *Presented as part of a Symposium on Continuing Medical Education held by the Committee on Medical Education of the New York Academy of Medicine October 10, 1974.
Vol. 51, No. 6, June 1975
74 6
J. G. FREYMANN 746~~ J.G
REMN
term, but I am still not satisfied. Patient is derived from the Latin patior, to suffer, but we should strive to care for people before they reach the stage of suffering. We should be working to keep people healthy. Therefore the title I prefer is "Health-Care Appraisal." The appraisal of health care can be carried out in many ways, but I shall not describe these since many have already been mentioned by other speakers in this symposium. (An excellent summary of the many faces of health-care appraisal appeared in a supplement to the November 1973 issue of the Journal of Medical Education.J) Instead, I shall devote the balance of this paper to seven topics relating to this subject. Some are caveats and some are commentaries. They are not necessarily listed in order of importance. All of them should be considered as medicine in the United States embarks on this new adventure. Two PHILOSOPHIES OF HEALTH-CARE APPRAISAL Health-care appraisal may be based on one of two antithetical philosophies. One of these may be dubbed the welfare approach; the other the Internal Revenue Service (IRS) approach. The underlying philosophy of our welfare system is that everyone is basically dishonest and out to defraud the government. The welfare approach seems to be built into the general philosophy of the Professional Standards Review Organizations (PSROs): namely, requirements that individual physicians conform to standard loci of treatment, lengths of stay, number of visits, services used for specific diagnoses, etc., and that bureaucratic approval must be given for every deviation. If this philosophy is allowed to pervade health-care appraisal, it will mean that every physician must be checked individually. Further, this approach is backward-oriented-based on how things are done today. It would not only proceed on a basic assumption of incompetence but would also limit the practice of medicine to the status quo ante 1975. This has not happened yet, but PSROs could degenerate into this state of affairs. Chapter VII of the PSRO Mamnal gives us a way to avoid the welfare approach. This section of the PSRO regulations specifically states that positively-oriented systems of appraisal (the American Hospital Association's Quality Assurance Program is one example) are acceptable ways to evaluate health care. Such systems take the IRS approach: i.e., the IRS assumes that the vast majority of people are honest. This Bull. N. Y. Acad. Med.
MEDICAL AUDIT
MEDICAL
AUDIT
74 7 747
philosophy permits the practices of a number of physicians to be appraised collectively. Criteria set by the physicians themselves are based on clinical judgment, not on the standards of accountants. Significant discrepancies from these present criteria are noted instead of individual infractions, and these discrepancies are met with corrective programs of continuing education instead of punitive action. Further, since such criteria can be revised to include new advances in care, this philosophy of health-care appraisal is forward-looking-oriented toward a continuous improvement in care.I must caution my fellow physicians that the IRS approach is inconsistent with traditional medical education, which tends to concentrate on exceptions to the rules, and with our Hippocratic concern for individual patients. To avoid the pitfalls of the welfare philosophy, we must re-educate ourselves to view health care as people in the field of public health do. We must concentrate on the peak of the bellshaped distribution curve and assure ourselves that most patients are well treated; we must avoid the temptation to devote all our attention to the few patients who fall at the far ends of the distribution curve. HEALTH-CARE APPRAISAL MUST BE KEPT SEPARATE FROM COST CONTROL Recent federal legislation (PL 92-603) has placed the responsibility for monitoring the costs of health care on physicians instead of on administrators. This is good because if quality care costs more than the public bargains for physicians are in a better position than administrators to say so and to prove it. Proof is vital. If we cannot prove our point we shall have been guilty of an inadequate appraisal of health care. Cost controls are essential, of course, but cost control and healthcare appraisal are quite different. They are the numerator and denominator of the cost/benefit equation. This equation cannot be solvedi.e., we cannot measure benefits derived against the money expendedif cost indicators such as length of hospital stay, number of visits by physicians, and services used are equated with the quality of care. If they were equated, the numerator and denominator would be identical, the equation would equal i, and cost would determine what quality is because benefit could be defined only in terms of how much we can afford. We must not adopt fiscal definitions as the new vocabulary for Vol. 51, No. 6, June 1975
74 8
748
J.
G. FREYMANN G.
FREYMANN
describing health care. If we do, we shall have been co-opted as fiscal agents for fiscal intermediaries and the Social Security Administration. PITFALLS OF HEALTH-CARE APPRAISAL BASED ON DIAGNOSTIC CATEGORIES For 40 years fiscal intermediaries have protected their budgets by categorizing care. The accountants prefer every patient to have a single diagnosis so that they can challenge the treatment of this diagnosis on a service-by-service basis. Lawson3 observed that this is a "restricted view of what disease is humanly about." He has shown that the elderly average five to six diagnoses per person, and Dr. Thomas C. Meyer showed in this symposium that between 20% and 5o% of the patients of the doctors whose practices he studied could be categorized under no pathological entity or nosologic syndrome.4 Health care should be appraised as it is really delivered-in terms of problems. The surgical specialties have an easier job because they can often evaluate the results of specific procedures instituted to treat specific diagnoses, but medicine, pediatrics, and family medicine are not so fortunate. In these fields most patients present problems such as chest pain, failure to thrive, or anxiety, for which the important part of the appraisal is not only what the final diagnosis was but also how the problem was managed. To complicate things further, few patients present a single problem. We must therefore resist the pressures of officialdom and refuse to collapse multiple problem lists into a single principal diagnosis just to satisfy the accountants. What will happen if we appraise health care exclusively by diagnostic categories? First, we shall separate the treatment of disease from the social causes and consequences of disease, which should be the physician's equal concern. Second, we shall make it easier to justify expensive hospitalization than chronic care in extended-care facilities and in the home. Third, preventive medicine will be neglected or entirely ignored. And, finally, we shall limit health care to the treatment of episodic crises and emphasize specialization precisely at the time we are trying to attract 50% of the graduates of our medical schools into primary care.
PROCESS APPRAISAL VERSUS OUTCOME APPRAISAL We physicians have traditionally been preoccupied with the procBull. N. Y. Acad. Med.
MEDICAL AUDIT MEDICAL AUDIT
74 9 749~~~~~~~~~~~~~~~
ess of medical care. It was drummed into us during our education. Rounds and conferences are rituals for worshiping the medical process. As students, house officers, and practicing physicians we have been judged by our superiors and our peers on the way we carry out the processes we were so carefully taught. Perhaps there is a deeper reason for this worship of process. I believe that the ultimate reason is a blind faith that whatever we do will have a good outcome-so long as we do it correctly. One need not turn to the recent literature on health-care appraisal to demonstrate this point. Doctors have always been sure that they were doing good. In i825, for example, Macmichael confidently gave "advances in medicine" credit for the 500o drop in the British death rate between I750 and i825." We now know that this fall in the death rate was due to many causes, among them the invention of the flush toilet and cast-iron pipe, and that all that medicine contributed during this period was vaccination against smallpox. (This preventive practice accounted for only about 6% of the fall in the death rate.6) The public has had the same simple faith in us. Once again, one need not look to recent examples. During the yellow fever epidemic of 1793 Philadelphians lined up before dawn to see Benjamin Rush, the most prominent physician of his day, even though the bleeding and purging he used to treat this dread disease was described by a discerning colleague as "one of those great discoveries which are made from time to time for the depopulation of the earth."7 Going back even further, in iSi8 Erasmus wrote an essay, In Praise of the Healing Arts, which reflected the same touching faith in the power of physicians:8 The special glory of the healing arts is self-sufficient and recommends itself to mankind by its value and utility. . . . Many sicknesses have such a power that death is the sure fate of a patient if the doctor does not interfere immediately. Now let us return to the present and to the process of medical care. Is there any proof that the process has anything to do with its outcome? In a study of appendicitis and acute myocardial infarction in three hospitals of the Kaiser-Permanente system no correlation could be found between the processes by which care was given and the outcome of that care.9 Brook and Appel dropped a bombshell in I973.10 They used five different review methods to evaluate the care of patients with urinary tract infections, hypertension, or ulcerated intestinal disVol. 51, No. 6, June 1975
750
J.
G. FREYMANN
ease. When they used process criteria determined by experts-the method of health-care appraisal which is widely touted as the ultimate ideal-only 2% of the patients were judged to have received adequate care. However, when evaluators were asked if the outcome of treatment would have been improved if the process had been better, 63% of the patients were thought to have received an acceptable quality of care. The authors and an accompanying editorial by Williamson" cautioned that undue emphasis should not be placed on this single experiment, but that it does raise serious questions as to whether an evaluation of the process has any place in the appraisal of health care. Further investigation may show that long laundry lists of procedures have little connection with the quality of the outcome. However, if these lists were standardized they could become blueprints for litigation if they were not followed slavishly-at immense cost.12 Is EVALUATION OF OUTCOME PRACTICAL? If the process does not clearly correlate with the outcome of health care, then it seems logical that health-care appraisal should concentrate on outcomes. However, this is easier said than done. First, physicians in practice have not been accustomed to measuring the outcomes of their treatment except in one area: cancer. More than 8oo hospitals have cancer registries, but for some reason the evaluation of long-term outcome of treatment has not been extended to other diseases except in research projects. Perhaps this reluctance to measure outcomes reflects the blind confidence already mentioned: that whatever we do must be beneficial. Second, factors unrelated to treatment-genetic heritage, family circumstances, economic status, etc.-may be equally, if not more important to ultimate outcomes. Measuring intermediate outcomes is not as complicated, but these may not give accurate appraisals of health care because the points for evaluation must be chosen arbitrarily. For example, among 30I patients on whom ventricular defibrillation was attempted in emergency vehicles, a recent study showed that i99 had reverted to normal rhythms, ioi lived to reach hospitals, 42 were discharged alive, and 2I were still alive 12 months after discharge. Each of the last four figures represented an intermediate outcome, but the value of the initial treatment declines progressively as each is considered. Numerous solutions to the conundrum of the evaluation of results Bull. N. Y. Acad. Med.
MEDICAL AUDIT
MEDICAL
75I 75'
have been proposed. One that seems promising is what Sanazaro and Williamson call the "Six D's."114 Of the first three of these, death, is easily measured, and disease and disability are reasonably easy to weigh. However the second three-discomfort, dissatisfaction, and social disruption-become progressively harder to evaluate. These difficulties in measuring outcomes lead into my next topic. How FAR CAN THE QUANTIFICATION OF HEALTH CARE BE CARRIED? Here we have a paradox. McKeown" showed that before 1920 there was little evidence of medicine's efficacy, at least insofar as can be determined from death rates-yet medicine is a profession as old as the priesthood. Why? I believe it is because society-any societydemands its shaman. A shaman is someone set apart by society because people believe that he has the power to cure, to ward off death. Whether or not the shaman is really so endowed seems less important. (Why else do we still have quacks?) If outcomes had been appraised regularly 50 years ago, medicine would undoubtedly have been proven ineffective. What would have happened? Would medicine have folded its tent? No, it would certainly not have done so then. Further, I do not believe it would now. Medicine must realize that its social mandate is still based largely on its role as shaman, and that this role relates only incidentally to the results of treatment. In other words, the quantification of our efficacy can be carried only so far, but society's need for medicine extends far beyond that point. We must keep our importance to society in historical perspective. If we do not, society may discover that the profession it has honored for millennia has been replaced in the last generation by a class of technologists who still call themselves physicians but are not what the people think physicians should be. Society might then transfer its mandate to others-nurse practitioners or Dr. John W. Williamson's health accountants?"'-who do fulfill the people's primordial need for a shaman.
AVOID ADOPTING ANY FORM OF HEALTH-CARE APPRAISAL UNTIL IT IS VALIDATED American doctors are embarking on the first attempt in 3,000 years of medical history to quantify their actual benefit to society. Nothing Vol. 51, No. 6, June 1975
7 5 2
J. G. FREYMANN
like this has been done before. It will take time and a rigorous application of the scientific method to see if the evaluation of care can be measured and, if it can, how best to do it. Further, the avowed reason for doing it is to enhance the quality of care, but we have not yet proved that health-care appraisal will accomplish this feat. In a study of breast biopsies in Ottawa, Devitt concluded that appraisal followed by remedial education probably improves the quality of care.'5 The data presented by Dr. Williamson here"' looks convincing, at least in one specific diagnostic category. (I cannot help but note, however, that education of the patients, not the physicians, seems to have been responsible for most of the benefit.) Other studies have failed to show any effect. At Yale-New Haven Hospital an appraisal of cholecystectomy and inguinal herniorrhaphy by process and end-results revealed deficiencies; after appropriate educational measures were taken, a reappraisal failed to show any improvement.17 Drs. George E. Miller and Williamson themselves showed in a classic study that the loop between measured discrepancies and improved care is difficult to close with education, even when the problem is as simple as a failure to read urinanalysis reports.'8 Medicine must not overlook the fact that health-care appraisal itself must be validated. Medicine must resist the temptation to grasp at some arbitrary technique because (as one so often hears) "If we don't do something the government will do it for us." If we do this we shall become our own worse enemies, and our patients' as well. It is up to the medical profession not to do as it is told by accountants and bureaucrats but to experiment and develop valid methods of health-care appraisal. Only then will doctors be able to answer the three questions Mager asked in his verse:'9 Where am I going, How shall I get there, And how will I know I've arrived? REFERENCES 1. Barro, A. R.: Survey and evaluation of approaches to physician performance measurements. J. Med. Educ. 48:105193, 1973. 2. Garrell, M. and Hamaty, D.: Professional standards review organizations
and quality insurance programsThey're not the same. Conn. Ned. 38: 430-32, 1974. 3. Lawson, I. R.: Professional standards review organization and care of the elderly. J.A.M.A. 229 :311-13, 1974.
Bull. N. Y. Acad. Med.
MEDICAL AUDIT 4. Meyer, T. C.: Toward a continuum in medical education. Bull. N.Y. Acad. Med. 51:1975. 5. Macmichael, W.: The Gold-Headed Cane. Springfield, Ill., Thomas, 1953, pp. 175-76. 6. McKeown, T.: Medicine in Modern Society. New York, Hafner, 1966, p. 52. 7. Shryock, R. H.: Medicine in America: Historical Essays. Baltimore, Johns Hopkins Press, 1966, p. 156. 8. Erasmus, D.: In Praise of the Healing Arts. Translated in: Desiderius Erasmus (1466-1536), the great humanist. J.A.M.A. 210:1587-88, 1969. 9. Fessel, W. J. and VanBrunt, E. E.: Assessing quality of care from the medical record. New Eng. J. Med. 286: 134-38, 1972. 10. Brook, R. H. and Appel, F. A.: Quality-of-care assessment: Choosing a method for peer review. New Eng. J. Med. 288:1323-29, 1973. 11. Williamson, J. W.: Evaluating the quality of medical care. New Eng. J. Med. 288:1352-53, 1973. 12. Emerson, R. S.: The surgical audit (Yale-New Haven). Conn. Med. 38:
Vol. 51, No. 6, June 1975
7 5 3
116, 1974. 13. Liberthson, R. R., Eugene, N. L., Hirschman, J. C., and Nussenfeld, J. D.: Prehospital ventricular defibrillation. New Eng. J. Med. 291:317-21, 1974. 14. Sanazaro, P. J. and Williamson, J. W.: End result of patient care: A provisional classification based on reports of internists. Med. Care 6:123-30, 1968. 15. Devitt J. E.: Does continuing medical education by peer review really work? Canad. Med. Ass. J. 108:1279-81, 1973. 16. Williamson, J. W.: Patient care assessment. Bull. N.Y. Acad. Med. 51: 1975. 17. Cayton, C. G., Tanner, L. A., Riedel, D. C., and Williams, K. H., Jr.: Surgical audit using predetermined weighted criteria. Conn. Med. 38:117-22, 1974. 18. Williamson, J. W., Alexander, M., and Miller, G. E.: Continuing education and patient care research. J.A.M.A. 201:938-42, 1967. 19. Mager, R. F. (quoted by Brook, R. H.): Quality of care assessment: The role of faculty at academic medical centers. Clin. Res. 22:122-27, 1974.
