Effectiveness of Educational And Administrative Interventions In Medical Outpatient Clinics MICHAEL W. POZEN, MD, SCD PHILIP D. BONNET, MD Abstract: This study examines the popular belief that increased educational supervision and increased administrative support in university outpatient clinics will improve physician performance, which in turn will improve the process and outcome of patient care. Positive effects on house officers' attitudes and better func-
tioning of clinics with respect to follow-up, information retrieval, and prescribing practices were demonstrated. However, no differences in the process and outcome of care were identified by faculty judges using implicit criteria. (Am. J. Public Health, 66:151-155,
Introduction
care. This is especially a problem with chronic conditions, such as heart disease, which are among the most common
There is a commonly held belief that educational and administrative improvements in a medical outpatient department will ameliorate the many well documented shortcomings of medical care in clinics. With the continued expansion of the internist's roles and responsibilities in primary care,1' 7 such interventions have become particularly important in teaching hospitals with the responsibility for training medical house officers in ambulatory care.5 Although much effort has been expended to implement such educational and administrative programs, insuffient documentation as to their effects on physician performance, patient care, or physician education has stymied systematic review and development of these endeavors. The lack of evidence is not a result of professional con&rivance to avoid scrutiny but is a result of an inability to measure the outcomes of care and to relate those outcomes as specific results of the processes of
problems in medical outpatient clinics.
From the Department of Medicine at Baltimore City Hospitals and the Department of Health Care Organization (formerly Medical Care and Hospitals) of the Johns Hopkins University. Address reprint requests to Dr. Pozen at Boston City Hospital, 818 Harrison Ave., Boston, Mass 02118. Supported by a research training grant (5T01 HS 00012) from the National Center for Health Services Research and Development, by the Carnegie-Commonwealth and Robert Wood Johnson Foundation Clinical Scholars Program, and by a Research Award from the Council of Teaching Hospitals of the Association of American Medical Colleges, this paper was submitted to the Journal August 25, 1975. Revision was accepted for publication October 14, 1975.
AJPH February, 1976, Vol. 66, No. 2
1976)
To study some aspects of the problems of effectiveness of medical outpatient clincs, a controlled trial was conducted to examine prospectively the effects of more intensive physician education and of increased administrative support on patient care processes and outcomes. A panel of faculty members retrospectively rated the processes and outcome of care in individual cases using a compilation of information from patients' records from which all identifying information had been removed.
Methods Before July 1, 1975, each of the'35 medical house officers at Baltimore City Hospitals was arbitrarily assigned by the Chief Medical Resident to one of six medical outpatient clinics (each mn eeting weekly) in which the house officers (interns and residents) were principally responsible for patient care. The first 300 new patients eligible for these clinics were assigned to the six clinics in sequence. Sixty per cent came from among the house officers' hospital discharges and 40 per cent were from emergency room referrals. The overall census of the six house officers' clinics remained at approximately 1500 active patients for the duration of the study, distributed in relatively constant proportions among the six clinics. Generally, the patients were 45 to 65 years of age and blue collar workers. Half of the patients were non-white. 151
POZEN AND BONNET
The characteristics of patients, the clinical problems, and the number and sources of cases in each clinic were not significantly different. The distribution of primary clinical problems was typical of most adult outpatient clinics: 20 per cent cardiac, 17 per cent dermatological, 11 per cent GI, 10 per cent pulmonary, 8 per cent psychosomatic, 8 per cent renal, 5 per cent neurological, 5 per cent endocrine, 4 per cent arthritis, 3 per cent hematological, and 9 per cent other. The majority of patients had multiple chronic conditions. The attitudes of the house officers toward outpatient service as identified by a pre-intervention questionnaire were also found to be similiar. At the beginning of the study, three models (the independent variables) were introduced into the six house officers' clinics:
Educational Model To one pair of clinics, the "intensive education" pair, two faculty members were assigned to attend every clinic, conduct weekly conferences, review with the house officers some of the patients, be available for consultation, review all patient records at the end of each clinic, and discuss with the responsible house officers any problems noted. The objective was to make the clinic teaching analogous to that of the inpatient wards.
Administrative Model To another pair of clinics, the "administrative support" pair, an administrative person was employed to see that patient records were available and complete, assure followup of patients, expedite completion and recording of tests, and prepare reports of prescribing patterns and of laboratory tests ordered for monthly feedback to each house officer. Traditional Model No changes were made in the third pair of clinics which continued in the traditional manner with house officers providing the care and with faculty members on call. No information about the study was given to any of the house officers. Nor was any publicity given to the three pairs of clinics. House officers were told that some changes in the range of services would be introduced during the year, but there was no discussion of the objectives or methods of the study in order to minimize "cross-contamination". Judgments Baseline information on the first 100 consecutive new patients admitted to each of these three pairs of clinics was gathered from either hospital discharge sumrmaries or from emergency room referral records. At the end of eight months, the first author, who had no previous knowledge of the patients, obtained a follow-up history and did a physical examination on each patient who could be located [277 of the 300 patients (92 per cent) were seen]. Patients completed a questionnaire regarding their symptoms, functional status, and satisfaction. Clinic records of patients were summarized starting with all baseline information. Any information which identified the clinic, the patient, or the house officer was removed. To the clinic record, summaries of the inde152
pendent clinical assessments and the patients' status reports at the end of the eight months (outcome status) were added. These records were then distributed on a random basis to faculty members serving on the panel of judges who would provide ratings of the processes and of the outcomes. Process was to be judged on a four-point rating scale in accordance with the concept: "the diagnostic, therapeutic, and followup medical care given to this patient was: good, adequate, inadequate, poor." Outcomes were to be judged on three factors: symptoms, function, and health status. A three-point scale was to be used for each factor in accordance with the concept "improved medical care would have altered the patient's condition at the end of the study in the following respects: substantially improvable, improvable at all, not improvable." Every record was rated by two judges. For onethird of the charts distributed to two judges, there were disagreements based on dichotomization of any of the process or outcome judgments ("good" and "adequate" versus "inadequate" and "poor" for process judgments or "substantially improvable" versus "improvable at all" and "not improvable" for outcome judgments). The charts were submitted for judgment to one of the two faculty judges who did not participate in the original round ofjudgments. The majority opinion among the three judges was accepted. To examine the validity of these judgments, a 10 per cent sample of the charts from each clinic was reviewed by all 15judges. After an interval of two months, 20 per cent of each judge's charts were returned to him/her for rating again as a test of reliability (repeatability) ofjudgments. Eachjudge used his own implicit criteria which represented his/her training, experiences, and beliefs about appropriate medical care for each condition with the concomitant expectations for patient improvement or benefit. Explicit criteria involving preset check lists of appropriate procedures, treatments, and expectations were not used. At the study's conclusion, attitudes of house officers toward the outpatient experience were again examined by questionnaires. Detailed explanation of the methods, questionnaires, and analyses are reported elsewhere.4
Results Four principal dependent variables were measured: patient care, house officers' attitudes, patient followup, and patient satisfaction.
Patient Care Table 1 summarizes the process judgments rendered by the faculty using implicit criteria. These, and all subsequent results, are based on the 277 patients available for follow-up. None of these group differences were statistically significant. In the administrative clinics, fewer charts required a third judge, 22 per cent versus 31 per cent in the educational clinics, and 40 per cent in the control clinics, respectively (p = .03). This finding seems to indicate that process quality might have been more readily discernible in the administrative clinics due to the organization of the chart and lab data retrieved by the clinic administrator. AJPH February, 1976, Vol. 66, No. 2
CLINIC INTERVENTIONS
TABLE 1-Judgments of Patient Care Processes Comparing Three Models Process Judgments Clinic Model
Education Clinics Administrative Clinics Control Clinics All Clinics
N
Good
Adequate
Inadequate
Poor
94 90 93 277
10(11%) 17(19%) 12(13%) 39(14%)
50(53%) 51(57%) 50(54%) 151(55%)
27(29%) 20(22%) 29(31%) 76(27%)
7(7%) 2(2%) 2(2%) 11(4%)
Table 2 shows the composites of the outcome ratings reached by averaging the three judgments to a single rating. None of these group differences were statistically significant. Analysis of the specific outcome judgments (symptoms, function, and health status) also failed to reveal any statistically significant difference among the three pairs of clinics. In fact, there was a high degree of consistency in the three outcome judgments rendered by a judge for a given chart. These process and outcome judgments were also analyzed by combining the judgments in all logical combinations. None of these analyses were significant. The judgments reached on the charts reviewed by all 15 judges did not differ significantly from the judgments made by the two or three judge teams. There was an intrajudge reliability rate of 80 percent and the 20 per cent of judgments which were changed were equal in both directions, indicating that intrajudge unreliability did not significantly alter the findings of this study. A review of the charts with process ratings "inadequate" or "poor", revealed three principal deficiencies: incomplete workup, insufficient or inappropriate medication, and poor follow-up. Common examples were: not obtaining gastrointestinal radiographs in the presence of persistent symptoms, inadequate dosages of antihypertensive medication, and loss to follow-up of patients with chronic, recurrent illnesses such as pyelonephritis. Outcomes with ratings "substantially improvable" occurred among a wide spectrum of disease entities in which the patient was judged to be unduly symptomatic relative to the underlying condition. Included among this group were asthmatics who were recycling at frequent intervals, patients with symptomatic peptic ulcers and cholelithiasis, patients with disabling angina pectoris and congestive heart failure on inadequate therapy, and patients with significant, reversible neuropathies.
House Officers' Attitudes The house officers evaluated their outpatient experiences in a questionnaire. Table 3 summarizes the results. The physicians in the educational clinics felt that they were learning more and being better supervised in their ambulatory care activities than did their colleagues in the other clinics. This finding indicates that the educational model was successful in accomplishing some of its goals.
Administrative Support Some effects of the administrative support services are shown in Table 4. These differences are significant at the 0.1 per cent level. These data document that the administrative model was effective in raising the level of available information at clinic visits and in reducing "loss" of follow-up contact. Staff were not available to collect comparable data on laboratory reports for the educational and control clinics. Patient Satisfaction Patients were questioned about many specific aspects of their care. There was a high degree of general satisfaction. Care was considered good and easily accessible. Most patients responded that their doctors took the time to explain the clinical situation and medication schedules and answer questions. Clinic hours were not always convenient, and financial barriers were reported by some. Among the reasons for not returning to the elinic and for seeking care elsewhere (28 per cent of the patients) were financial/geographic barriers and, in second place, dissatisfaction. The findings were uniformly distributed among the three pairs of clinics.
TABLE 2-Judgments of Patient Care Outcomes Comparing Three Models Outcome Judgments Clinic Model
Education Clinics Administrative Clinics Control Clinics All Clinics AJPH February, 1976, Vol. 66, No. 2
N
Not Improvable
Improvable at All
Substantially Improvable
94 90 93 277
52(56%) 43(48%) 47(51%) 142(51%)
37(39%) 44(49%) 44(47%) 125(45%)
5(5%) 3(3%) 2(2%) 10(4%)
153
POZEN AND BONNET
TABLE 3-Mean Ratings* by House Officers of Educational Experience and Administrative Support House Officer Group
Education Experience
Administrative Services
Education Clinics Admin. Clinics Control Clinics
High Intermediate
Low Intermediate Low
Low
*House officers evaluated their experiences using numerical responses on an eight interval ordinal scale. "High" is equivalent to 1-2, "intermediate" to 3-5, and "low" to 6-8. The differences between house officer groups are statistically significant at the 5% level.
TABLE 4-ffects of Administrative Support Services
Clinic Model
Education Clinics Admin. Clinics Control Clinics
% Charts Available at Clinic
Kept Appointment Rate
% Patients Lost to
68% 90% 68%
54% 69% 55%
38% 19% 35%
Follow-Up
% Charts With Lab Data Added
25%
Discussion The expectation that the educational and administrative interventions would result in measurable improvement in patient care was not realized. However, the educational programs were successful in generating positive attitudes toward outpatient experience among the house officers, and the administrative programs did enhance the availability of data, patient follow-up, and overall efficiency in those clinics. The 19 per cent "lost to follow-up" rate is substantially better than the experience in most clinics.2 An implicit assumption in the study was that shortcomings in the process and outcome of medical care are due, in part, to inadequate biomedical information. The corollary, which the study examined, is that improved information will, by changing physician behavior, improve the process and outcome of medical care. Neither the assumption nor its corollary is proven. That physicians lack all sorts of information can be shown easily. What physicians do with specific kinds of information is little understood. The role of information in altering behavior, how behavioral changes come about, and what significance these changes might have for patient management remain problematic. There is also the need to consider the time period necessary for behavioral changes to occur. There may be critical points in a physician's education beyond which some kinds of information input may be ineffectual. A second implicit assumption in the study was that shortcomings in house staff medical care in the outpatient department partially derive from inadequate supervision by faculty members. The house staff's significantly higher ratings of their experience in the educational clinics compared to those in the administrative and control clinics indicate that house staff themselves perceive adequate supervision as a substantial need. The results in terms of patient care, how154
ever, raise important issues about the efficacy of employing a traditional "inpatient" educational approach in the outpatient department, relying on faculty members trained and experienced mostly in inpatient medicine. Perhaps experienced internists from the community would be the more logical source for providing educational supervision in the out-
patient department. Questions about this study
are most likely to revolve around the choice of the independent variables and the use of the implicit judgment criteria. Neither the educational nor the administrative intervention can be considered specific. Both embody broad concepts and many facets. Such interventions are, however, precisely the substance of "innovative programs" being espoused by clinical chiefs and administrators, and as such are appropriate independent variables to be tested. For this study, implicit judgments were used because: (a) explicit criteria for the entire range of diseases among the clinical patients were unavailable; (b) a significant proportion of the patients had combinations of diseases for which explicit criteria have not been formulated; and (c) the implicit judgments paralleled the faculty's method for inpatient chart review. To examine the limitation of implicit judgments, multiple analyses of the process and outcome judgments and the interactions of these judgments with the questions on the scoring form were done. None of these analyses were significant despite adequate variance in the judgments rendered. The interjudge and intrajudge reliability results add further confirmation of these findings. The intrajudge reliability rate of 80 per cent was comparable to that of both the Rochester Perinatal Study3 and the follow-up study in the Central New York Hospital Service Region.6 It seems reasonable to question whether a design employing implicit criteria for groups of patients with many different conditions, mostly chronic, can ever be made sensitive enough to detect the modest degrees of change which are possible within eight months. Initial, uniform measurements of the patient status and a longer follow-up period would obviously enhance the sensitivity of such a design, especially if populations of patients with chronic diseases are to be studied. Thus, the problem of development of a practical, reliable, and sensitive method of evaluation of outcomes of patient care remains. The design would need only to be sensitive enough to discern that level of change which would effect alternative decisions about patient care. But, much more work will need to be done to find such a method.
Conclusion The objectives of this study in a medical outpatient department were to describe and evaluate separately the effects of educational intervention and administrative intervention on the processes and outcomes of patient care. The intervention were shown to have had identifiable effects on some factors believed to improve the performance of physicians, such as their perceptions of their educational experience and the availability of effective support services. When, however, the processes and outcomes of patient care were AJPH February, 1976, Vol. 66, No. 2
CLINIC INTERVENTIONS
rated by the faculty members in the Department of Medicine who are responsible for the care delivered at the hospital, neither intervention-educational or administrativeimproved patient care when compared with the traditional clinic arrangements. It is clear that educational and administrative interventions in the outpatient department can foster improvements. Indeed, the input into the medical care system of performance characteristics of physicians' roles and relationships most certainly does determine in large part the quality of the output from that system. Identifiable changes did occur in house officers' attitudes, in patient follow-up, and in completeness of records. This study, however, reveals a need for more prospective trials of new interventions involving improved organization and management to optimize physician performance. Such studies can provide the necessary information base for rational planning of teaching and patient care innovations in the outpatient department.
REFERENCES 1. Altman, I., Kroeger, H., and Clark, D. The office practice of internists, II. Patient load. JAMA, 193:101-106, 1965. 2. Brook, R. and Stevenson, R. Effectiveness of patient care in an emergency room. N Engl. J. Med., 283:904-907, 1970. 3. Medical Review Project. Empire State Medical, Scientific, and Educational Foundation, Inc. (Richardson, F., Director). 4. Pozen, M. Effects of physician education and administrative support on hospital ambulatory care. Doctoral Dissertation. The Johns Hopkins University, Department of Medical Care and Hospitals, pp. 47-64, May 1974. 5. Rammelkamp, C. and Chester, E. The training of the physician. A new approach to teaching ambulatory medicine. N. Engl. J. Med, 271:349-351, 1964. 6. Richardson, F. Peer review of medical care. Medical Care, 10: 29-39, 1972. 7. White, K. L. Williams, T. F., and Greenberg, B. G. The ecology of medical care. N. Engl. J. Med, 265:885-892, 1961.
I HILLMAN FOUNDATION SEEKS NOMINATIONS FOR 1975 AWARDS The Sidney Hillman Foundation will award prizes of $750 each for outstanding contributions dealing with themes relating to the ideals which Sidney Hillman held throughout his life. Such themes would include the protection of individual civil liberties, improved race relations, a strengthened labor movement, the advancement of social welfare and economic security, greater world understanding, and related problems. Contributions may be in the fields of daily or periodical journalism, fiction, non-fiction, radio and television. All written contributions must have been published in 1975. Radio and television contributions must have been produced under professional auspices in 1975. Submission for a 1975 Prize Award must be received by the Sidney Hillman Foundation, Inc., 15 Union Square, New York, NY 10003, not later than January 31, 1976. Material may be submitted by the author, his publication or publisher, or by anyone connected with it. Unpublished manuscripts of any kind are not eligible for consideration. The Foundation will acknowledge all submissions and will endeavor to return submissions (not before May 16, 1976) if requested. A panel of judges will make the Awards and its decisions will be final.
AJPH February, 1976, Vol. 66, No. 2
155
tioning of clinics with respect to follow-up, information retrieval, and prescribing practices were demonstrated. However, no differences in the process and outcome of care were identified by faculty judges using implicit criteria. (Am. J. Public Health, 66:151-155,
Introduction
care. This is especially a problem with chronic conditions, such as heart disease, which are among the most common
There is a commonly held belief that educational and administrative improvements in a medical outpatient department will ameliorate the many well documented shortcomings of medical care in clinics. With the continued expansion of the internist's roles and responsibilities in primary care,1' 7 such interventions have become particularly important in teaching hospitals with the responsibility for training medical house officers in ambulatory care.5 Although much effort has been expended to implement such educational and administrative programs, insuffient documentation as to their effects on physician performance, patient care, or physician education has stymied systematic review and development of these endeavors. The lack of evidence is not a result of professional con&rivance to avoid scrutiny but is a result of an inability to measure the outcomes of care and to relate those outcomes as specific results of the processes of
problems in medical outpatient clinics.
From the Department of Medicine at Baltimore City Hospitals and the Department of Health Care Organization (formerly Medical Care and Hospitals) of the Johns Hopkins University. Address reprint requests to Dr. Pozen at Boston City Hospital, 818 Harrison Ave., Boston, Mass 02118. Supported by a research training grant (5T01 HS 00012) from the National Center for Health Services Research and Development, by the Carnegie-Commonwealth and Robert Wood Johnson Foundation Clinical Scholars Program, and by a Research Award from the Council of Teaching Hospitals of the Association of American Medical Colleges, this paper was submitted to the Journal August 25, 1975. Revision was accepted for publication October 14, 1975.
AJPH February, 1976, Vol. 66, No. 2
1976)
To study some aspects of the problems of effectiveness of medical outpatient clincs, a controlled trial was conducted to examine prospectively the effects of more intensive physician education and of increased administrative support on patient care processes and outcomes. A panel of faculty members retrospectively rated the processes and outcome of care in individual cases using a compilation of information from patients' records from which all identifying information had been removed.
Methods Before July 1, 1975, each of the'35 medical house officers at Baltimore City Hospitals was arbitrarily assigned by the Chief Medical Resident to one of six medical outpatient clinics (each mn eeting weekly) in which the house officers (interns and residents) were principally responsible for patient care. The first 300 new patients eligible for these clinics were assigned to the six clinics in sequence. Sixty per cent came from among the house officers' hospital discharges and 40 per cent were from emergency room referrals. The overall census of the six house officers' clinics remained at approximately 1500 active patients for the duration of the study, distributed in relatively constant proportions among the six clinics. Generally, the patients were 45 to 65 years of age and blue collar workers. Half of the patients were non-white. 151
POZEN AND BONNET
The characteristics of patients, the clinical problems, and the number and sources of cases in each clinic were not significantly different. The distribution of primary clinical problems was typical of most adult outpatient clinics: 20 per cent cardiac, 17 per cent dermatological, 11 per cent GI, 10 per cent pulmonary, 8 per cent psychosomatic, 8 per cent renal, 5 per cent neurological, 5 per cent endocrine, 4 per cent arthritis, 3 per cent hematological, and 9 per cent other. The majority of patients had multiple chronic conditions. The attitudes of the house officers toward outpatient service as identified by a pre-intervention questionnaire were also found to be similiar. At the beginning of the study, three models (the independent variables) were introduced into the six house officers' clinics:
Educational Model To one pair of clinics, the "intensive education" pair, two faculty members were assigned to attend every clinic, conduct weekly conferences, review with the house officers some of the patients, be available for consultation, review all patient records at the end of each clinic, and discuss with the responsible house officers any problems noted. The objective was to make the clinic teaching analogous to that of the inpatient wards.
Administrative Model To another pair of clinics, the "administrative support" pair, an administrative person was employed to see that patient records were available and complete, assure followup of patients, expedite completion and recording of tests, and prepare reports of prescribing patterns and of laboratory tests ordered for monthly feedback to each house officer. Traditional Model No changes were made in the third pair of clinics which continued in the traditional manner with house officers providing the care and with faculty members on call. No information about the study was given to any of the house officers. Nor was any publicity given to the three pairs of clinics. House officers were told that some changes in the range of services would be introduced during the year, but there was no discussion of the objectives or methods of the study in order to minimize "cross-contamination". Judgments Baseline information on the first 100 consecutive new patients admitted to each of these three pairs of clinics was gathered from either hospital discharge sumrmaries or from emergency room referral records. At the end of eight months, the first author, who had no previous knowledge of the patients, obtained a follow-up history and did a physical examination on each patient who could be located [277 of the 300 patients (92 per cent) were seen]. Patients completed a questionnaire regarding their symptoms, functional status, and satisfaction. Clinic records of patients were summarized starting with all baseline information. Any information which identified the clinic, the patient, or the house officer was removed. To the clinic record, summaries of the inde152
pendent clinical assessments and the patients' status reports at the end of the eight months (outcome status) were added. These records were then distributed on a random basis to faculty members serving on the panel of judges who would provide ratings of the processes and of the outcomes. Process was to be judged on a four-point rating scale in accordance with the concept: "the diagnostic, therapeutic, and followup medical care given to this patient was: good, adequate, inadequate, poor." Outcomes were to be judged on three factors: symptoms, function, and health status. A three-point scale was to be used for each factor in accordance with the concept "improved medical care would have altered the patient's condition at the end of the study in the following respects: substantially improvable, improvable at all, not improvable." Every record was rated by two judges. For onethird of the charts distributed to two judges, there were disagreements based on dichotomization of any of the process or outcome judgments ("good" and "adequate" versus "inadequate" and "poor" for process judgments or "substantially improvable" versus "improvable at all" and "not improvable" for outcome judgments). The charts were submitted for judgment to one of the two faculty judges who did not participate in the original round ofjudgments. The majority opinion among the three judges was accepted. To examine the validity of these judgments, a 10 per cent sample of the charts from each clinic was reviewed by all 15judges. After an interval of two months, 20 per cent of each judge's charts were returned to him/her for rating again as a test of reliability (repeatability) ofjudgments. Eachjudge used his own implicit criteria which represented his/her training, experiences, and beliefs about appropriate medical care for each condition with the concomitant expectations for patient improvement or benefit. Explicit criteria involving preset check lists of appropriate procedures, treatments, and expectations were not used. At the study's conclusion, attitudes of house officers toward the outpatient experience were again examined by questionnaires. Detailed explanation of the methods, questionnaires, and analyses are reported elsewhere.4
Results Four principal dependent variables were measured: patient care, house officers' attitudes, patient followup, and patient satisfaction.
Patient Care Table 1 summarizes the process judgments rendered by the faculty using implicit criteria. These, and all subsequent results, are based on the 277 patients available for follow-up. None of these group differences were statistically significant. In the administrative clinics, fewer charts required a third judge, 22 per cent versus 31 per cent in the educational clinics, and 40 per cent in the control clinics, respectively (p = .03). This finding seems to indicate that process quality might have been more readily discernible in the administrative clinics due to the organization of the chart and lab data retrieved by the clinic administrator. AJPH February, 1976, Vol. 66, No. 2
CLINIC INTERVENTIONS
TABLE 1-Judgments of Patient Care Processes Comparing Three Models Process Judgments Clinic Model
Education Clinics Administrative Clinics Control Clinics All Clinics
N
Good
Adequate
Inadequate
Poor
94 90 93 277
10(11%) 17(19%) 12(13%) 39(14%)
50(53%) 51(57%) 50(54%) 151(55%)
27(29%) 20(22%) 29(31%) 76(27%)
7(7%) 2(2%) 2(2%) 11(4%)
Table 2 shows the composites of the outcome ratings reached by averaging the three judgments to a single rating. None of these group differences were statistically significant. Analysis of the specific outcome judgments (symptoms, function, and health status) also failed to reveal any statistically significant difference among the three pairs of clinics. In fact, there was a high degree of consistency in the three outcome judgments rendered by a judge for a given chart. These process and outcome judgments were also analyzed by combining the judgments in all logical combinations. None of these analyses were significant. The judgments reached on the charts reviewed by all 15 judges did not differ significantly from the judgments made by the two or three judge teams. There was an intrajudge reliability rate of 80 percent and the 20 per cent of judgments which were changed were equal in both directions, indicating that intrajudge unreliability did not significantly alter the findings of this study. A review of the charts with process ratings "inadequate" or "poor", revealed three principal deficiencies: incomplete workup, insufficient or inappropriate medication, and poor follow-up. Common examples were: not obtaining gastrointestinal radiographs in the presence of persistent symptoms, inadequate dosages of antihypertensive medication, and loss to follow-up of patients with chronic, recurrent illnesses such as pyelonephritis. Outcomes with ratings "substantially improvable" occurred among a wide spectrum of disease entities in which the patient was judged to be unduly symptomatic relative to the underlying condition. Included among this group were asthmatics who were recycling at frequent intervals, patients with symptomatic peptic ulcers and cholelithiasis, patients with disabling angina pectoris and congestive heart failure on inadequate therapy, and patients with significant, reversible neuropathies.
House Officers' Attitudes The house officers evaluated their outpatient experiences in a questionnaire. Table 3 summarizes the results. The physicians in the educational clinics felt that they were learning more and being better supervised in their ambulatory care activities than did their colleagues in the other clinics. This finding indicates that the educational model was successful in accomplishing some of its goals.
Administrative Support Some effects of the administrative support services are shown in Table 4. These differences are significant at the 0.1 per cent level. These data document that the administrative model was effective in raising the level of available information at clinic visits and in reducing "loss" of follow-up contact. Staff were not available to collect comparable data on laboratory reports for the educational and control clinics. Patient Satisfaction Patients were questioned about many specific aspects of their care. There was a high degree of general satisfaction. Care was considered good and easily accessible. Most patients responded that their doctors took the time to explain the clinical situation and medication schedules and answer questions. Clinic hours were not always convenient, and financial barriers were reported by some. Among the reasons for not returning to the elinic and for seeking care elsewhere (28 per cent of the patients) were financial/geographic barriers and, in second place, dissatisfaction. The findings were uniformly distributed among the three pairs of clinics.
TABLE 2-Judgments of Patient Care Outcomes Comparing Three Models Outcome Judgments Clinic Model
Education Clinics Administrative Clinics Control Clinics All Clinics AJPH February, 1976, Vol. 66, No. 2
N
Not Improvable
Improvable at All
Substantially Improvable
94 90 93 277
52(56%) 43(48%) 47(51%) 142(51%)
37(39%) 44(49%) 44(47%) 125(45%)
5(5%) 3(3%) 2(2%) 10(4%)
153
POZEN AND BONNET
TABLE 3-Mean Ratings* by House Officers of Educational Experience and Administrative Support House Officer Group
Education Experience
Administrative Services
Education Clinics Admin. Clinics Control Clinics
High Intermediate
Low Intermediate Low
Low
*House officers evaluated their experiences using numerical responses on an eight interval ordinal scale. "High" is equivalent to 1-2, "intermediate" to 3-5, and "low" to 6-8. The differences between house officer groups are statistically significant at the 5% level.
TABLE 4-ffects of Administrative Support Services
Clinic Model
Education Clinics Admin. Clinics Control Clinics
% Charts Available at Clinic
Kept Appointment Rate
% Patients Lost to
68% 90% 68%
54% 69% 55%
38% 19% 35%
Follow-Up
% Charts With Lab Data Added
25%
Discussion The expectation that the educational and administrative interventions would result in measurable improvement in patient care was not realized. However, the educational programs were successful in generating positive attitudes toward outpatient experience among the house officers, and the administrative programs did enhance the availability of data, patient follow-up, and overall efficiency in those clinics. The 19 per cent "lost to follow-up" rate is substantially better than the experience in most clinics.2 An implicit assumption in the study was that shortcomings in the process and outcome of medical care are due, in part, to inadequate biomedical information. The corollary, which the study examined, is that improved information will, by changing physician behavior, improve the process and outcome of medical care. Neither the assumption nor its corollary is proven. That physicians lack all sorts of information can be shown easily. What physicians do with specific kinds of information is little understood. The role of information in altering behavior, how behavioral changes come about, and what significance these changes might have for patient management remain problematic. There is also the need to consider the time period necessary for behavioral changes to occur. There may be critical points in a physician's education beyond which some kinds of information input may be ineffectual. A second implicit assumption in the study was that shortcomings in house staff medical care in the outpatient department partially derive from inadequate supervision by faculty members. The house staff's significantly higher ratings of their experience in the educational clinics compared to those in the administrative and control clinics indicate that house staff themselves perceive adequate supervision as a substantial need. The results in terms of patient care, how154
ever, raise important issues about the efficacy of employing a traditional "inpatient" educational approach in the outpatient department, relying on faculty members trained and experienced mostly in inpatient medicine. Perhaps experienced internists from the community would be the more logical source for providing educational supervision in the out-
patient department. Questions about this study
are most likely to revolve around the choice of the independent variables and the use of the implicit judgment criteria. Neither the educational nor the administrative intervention can be considered specific. Both embody broad concepts and many facets. Such interventions are, however, precisely the substance of "innovative programs" being espoused by clinical chiefs and administrators, and as such are appropriate independent variables to be tested. For this study, implicit judgments were used because: (a) explicit criteria for the entire range of diseases among the clinical patients were unavailable; (b) a significant proportion of the patients had combinations of diseases for which explicit criteria have not been formulated; and (c) the implicit judgments paralleled the faculty's method for inpatient chart review. To examine the limitation of implicit judgments, multiple analyses of the process and outcome judgments and the interactions of these judgments with the questions on the scoring form were done. None of these analyses were significant despite adequate variance in the judgments rendered. The interjudge and intrajudge reliability results add further confirmation of these findings. The intrajudge reliability rate of 80 per cent was comparable to that of both the Rochester Perinatal Study3 and the follow-up study in the Central New York Hospital Service Region.6 It seems reasonable to question whether a design employing implicit criteria for groups of patients with many different conditions, mostly chronic, can ever be made sensitive enough to detect the modest degrees of change which are possible within eight months. Initial, uniform measurements of the patient status and a longer follow-up period would obviously enhance the sensitivity of such a design, especially if populations of patients with chronic diseases are to be studied. Thus, the problem of development of a practical, reliable, and sensitive method of evaluation of outcomes of patient care remains. The design would need only to be sensitive enough to discern that level of change which would effect alternative decisions about patient care. But, much more work will need to be done to find such a method.
Conclusion The objectives of this study in a medical outpatient department were to describe and evaluate separately the effects of educational intervention and administrative intervention on the processes and outcomes of patient care. The intervention were shown to have had identifiable effects on some factors believed to improve the performance of physicians, such as their perceptions of their educational experience and the availability of effective support services. When, however, the processes and outcomes of patient care were AJPH February, 1976, Vol. 66, No. 2
CLINIC INTERVENTIONS
rated by the faculty members in the Department of Medicine who are responsible for the care delivered at the hospital, neither intervention-educational or administrativeimproved patient care when compared with the traditional clinic arrangements. It is clear that educational and administrative interventions in the outpatient department can foster improvements. Indeed, the input into the medical care system of performance characteristics of physicians' roles and relationships most certainly does determine in large part the quality of the output from that system. Identifiable changes did occur in house officers' attitudes, in patient follow-up, and in completeness of records. This study, however, reveals a need for more prospective trials of new interventions involving improved organization and management to optimize physician performance. Such studies can provide the necessary information base for rational planning of teaching and patient care innovations in the outpatient department.
REFERENCES 1. Altman, I., Kroeger, H., and Clark, D. The office practice of internists, II. Patient load. JAMA, 193:101-106, 1965. 2. Brook, R. and Stevenson, R. Effectiveness of patient care in an emergency room. N Engl. J. Med., 283:904-907, 1970. 3. Medical Review Project. Empire State Medical, Scientific, and Educational Foundation, Inc. (Richardson, F., Director). 4. Pozen, M. Effects of physician education and administrative support on hospital ambulatory care. Doctoral Dissertation. The Johns Hopkins University, Department of Medical Care and Hospitals, pp. 47-64, May 1974. 5. Rammelkamp, C. and Chester, E. The training of the physician. A new approach to teaching ambulatory medicine. N. Engl. J. Med, 271:349-351, 1964. 6. Richardson, F. Peer review of medical care. Medical Care, 10: 29-39, 1972. 7. White, K. L. Williams, T. F., and Greenberg, B. G. The ecology of medical care. N. Engl. J. Med, 265:885-892, 1961.
I HILLMAN FOUNDATION SEEKS NOMINATIONS FOR 1975 AWARDS The Sidney Hillman Foundation will award prizes of $750 each for outstanding contributions dealing with themes relating to the ideals which Sidney Hillman held throughout his life. Such themes would include the protection of individual civil liberties, improved race relations, a strengthened labor movement, the advancement of social welfare and economic security, greater world understanding, and related problems. Contributions may be in the fields of daily or periodical journalism, fiction, non-fiction, radio and television. All written contributions must have been published in 1975. Radio and television contributions must have been produced under professional auspices in 1975. Submission for a 1975 Prize Award must be received by the Sidney Hillman Foundation, Inc., 15 Union Square, New York, NY 10003, not later than January 31, 1976. Material may be submitted by the author, his publication or publisher, or by anyone connected with it. Unpublished manuscripts of any kind are not eligible for consideration. The Foundation will acknowledge all submissions and will endeavor to return submissions (not before May 16, 1976) if requested. A panel of judges will make the Awards and its decisions will be final.
AJPH February, 1976, Vol. 66, No. 2
155
