Establishing a Quality Improvement Process for Identification of Psychosocial Problems in a Primary Care Practice RICHARD L. BINGHAM, MSW, DENNIS A. PLANTE, MD, DAVID L. BRONSON, MD, HENRY M. TUFO, MD, KAREN McKNIGHT, MA Objectives: A quality improventent process that will signifi-
cantiy increase the rate o f identification of psychosocial problems through routine use o f case-finding instruments can be established in a general medicine practice. Design: Two groups o f patient examination reports written by physicians were retrospectively compared with the patients" responses on the case-finding database instrumer~ The samples were obtained by s e ~ selection in
f o u r time periods. Setting a n d Patients: The study occurred in a university general internal medicine practice that utilizes the problem-oriented record~ The patients studied were seen f o r first-time comprehensive examinations designed to identify all important health problems, includins psychosocial
prob~ms. I n t e r v e n t i o n : The authors compared performances o f the physicians in identification o f psychosocial problems bef o r e and after t h e intervention, which consisted o f a pilot study audit o f psychosocial problem identification, establishment o f standards f o r interpretation o f the case-find. ing instrumen~ design o f a flow sheet to make case-finding data clearly available to the physician at each comprehensive examination, and feedback o f physician performance according to practice-adopted standards f o r identification o f psychosocial problems. Measurement: The resu/t o f the i ~ was an increase in po~hosocial problem identification f r o m 67% to 9056 o f problems preseng p < 0.05 by chi-square distribution,, or a decrease f r o m 33% to 1056 inpsychosocialproblems missed by the physicians. Conclusion: The quality improvement process f o r identification o f psychosocial problems described in this report significantly i ~ a s e d the rate o f identification o f psychosocial problems by general internists. Key w o r d s : psychosocial problem identification,, primary care; quality improvemen~ case-finding instruments. J GI~I INrvatN M.ED 1990; 5:342-346.
PATIENTS IN PRIMARY CARE n e e d t o d i s c u s s p s y c h o s o c i a l
problems with their physicians. A recent study b y G o o d et al. ~ demonstrated that over 70% o f primary care patients "find it appropriate to turn to their primary care physicians for help with emotional distress, family problems, life stress, behavioral problems, and sexual dysfunction," but only one-fifth to one-third of patients w h o have e x p e r i e n c e d psychosocial difficulties have Received from the Department of Psychiatry (RLB) and the Department of Medicine (DAP, DLB, HMT), University of Vermont, and the Given Health Care Center (KMcK), Burlington, Vermont. Presented at The Treatment of Mental Disorders in General Health Care Settings: A Research Conference, sponsored by the National Institute of Mental Health, University of Pittsburgh School of Medicine, The Upjohn Co. June 15 - 17, 1988. Address correspondence and reprint requests to Mr. Bingham: Given Health Care Center, University Health Center, One South Prospect Street, Burlington, VT 05401.
342
discussed these problems with their primary care providers. Failure to recognize psychosocial problems will lead to inappropriate testing, misuse o f medical resources, and p o o r therapeutic results. 24 Patients w h o have psychosocial problems and are seen in primary care settings include those with serious mental illness as well as those w h o have situational stresses. Estimated prevalences of psychosocial problems in primary care patients range from 30% to 60%, depending on the s o c i o e c o n o m i c level of the population and w h e t h e r identified cases are restricted to conformance with psychiatric DSM-III diagnostic standards. ~'7 However, there are many barriers to successful identification of and treatment planning for psychosocial problems in the primary care setting, s, 9 Patients w h o have psychosocial problems need to be identified within the usual practice environment in a systematic manner that is economically feasible and makes efficient use of the physicians' skills. Given the natural variability among physicians in collecting information, many patients' psychosocial problems will not be detected unless the primary care practice has a system for data collection and organization that includes effective case-finding instruments for psychosocial problems. We accept the distinction made by Coulehan and others,0, t~ that case finding rather than screening is best understood as the aim of a patient-completed questionnaire. The aim of the questionnaire is not diagnosis but to alert the physician, w h o can then determine w h e t h e r a p r o b l e m is present and formulate a diagnosis and treatment plan at his or her level of expertise. An effective system o f practice will utilize standards for identification o f psychosocial problems that are i m p l e m e n t e d with daily practice procedures, and an internal audit with feedback of performance to the adopted standards. ~2-t5 Coulehan et al., 1o based on a study of the efficiency of using questionnaires for the case finding o f depression in primary medical care, state that routine use of such questionnaires is not " a p p r o p r i a t e " because they are too c u m b e r s o m e for total coverage o f the practice. They suggest that case-finding questionnaires be utilized only for high-risk patients. We, on the contrary, maintain that it is feasible to use case-finding instruments for psychosocial p r o b l e m identification in all patients in the practice w h o are seen for c o m p r e h e n sive care. Psychosocial case finding can be efficiently and effectively included in the standard database that is collected from all patients through patient-completed
JOURNALOFGENERALINTERNALMEDICINE,Volume S (July/August), 1990
questionnaires, and through methods that summarize and consolidate the database for efficient utilization by the physician. This report describes a primary care system for the identification of psychosocial problems developed by the Given Health Care Center at the University of Vermont, and how this system increased identification o f psychosocial problems in this practice. METHODS
343
PROMIS Questionnaire Form completed at home by patient
Data Base Office Visit o PROMIS data summarized and tests performed by medical assistant
Practice Setting
The Given Health Care Center, organized in 1971 as a problem-oriented medical practice, serves as the general internal medicine faculty practice site for the University of Vermont School of Medicine. The practice cares for 25,000 patients, who are seen for 48,000 patient visits per year, and is staffed by 13 faculty physicians, primary care medical residents, two nurse practitioners, one physician assistant, and two clinical social workers. Approximately 10% of patient visits are with the resident physicians. Two thousand new patients are seen for first-time complete health examinations each year. These comprehensive examinations are structured to identify all important health problems, including psychosocial problems. Comprehensive Examination
Each patient seen for t h e " Initial Complete Exam" is asked to fill out the PROMIS III Adult Medical History Questionnaire for Problem Oriented Practices. 16 This questionnaire, developed in part by Given, is now utilized by 17 other general medical practices. The questionnaire contains 342 questions, with 106 of these questions pertaining to psychosocial aspects of the patient's life. The questionnaire includes the Popoff Depression Scale, 17-20a Problems of Daily Living Scale, the Holmes Life Events Scale, 21 and a 14-question alcoholism case finder. The psycbosocial database is a major feature of the PROMIS questionnaire, accounting for 31% of the total review-of-systems questions. This database functions as a case-finding instrument, as defined by Coulehan et al., lO for a broad spectrum of psychosocial problems, rather than limiting identification to the problem of depression. The patient completes the PROMIS database as the first step in the sequence described in the flow diagram of Figure 1. Notice that at the second step, the PROMIS data are summarized by the medical assistant. This process includes transfer by the assistant of information from the questionnaire to a database flow sheet. This information is clerically categorized by the medical assistant according to the standards defined by the physicians. This step is crucial to effective use of all the case-finding sectors of the patient-completed database because it makes case-finding information readily available to the physician at every comprehensive examination.
Physician Visit o Physician review of data base o Perform physical examination o Discuss findings (including PROMIS psychosocial data) with patient o Negotiation of goals for care o Formulation of treatment plans
Problem-Oriented Report r e p o r t (including psychosocial problem, if present} sent to patient
o Written
FIGURE 1. Process for identification and treatment planning for psychosocial problems in the primary care setting.
Research Design
For this study, the patient questionnaire answers on the psychosocial sections were audited by one of the clinical social workers and compared with the physicians' written reports derived from the same database with regard to whether the physicians had identified psychosocial problems. We did not attempt to determine whether a physician had made a correct DSM-III diagnosis, but onlywhether the physician had included a psychosocial problem in the problem list when the patient had signaled distress on the questionnaire. The study compared two groups of patient examination reports: a pilot study group, and a group of examination reports reviewed by the practice auditor after completion of the pilot study. The pilot study for psychosocial problem identification and treatment planning was performed between August 1, 1987, and April 30, 1988. Two hundred and forty faculty charts and 79 resident charts of first-time complete health examinations were selected sequentially during three different time periods for retrospective examination by the clinical social worker. 22 Total sample size for the pilot study was 319 patients, or 16% of patients undergoing this type of examination seen in one year. During the first sampling period of the pilot study, the physicians were unaware that a study was occurring. At no time were patients aware of a study, since there was no change in the usual procedures for data collection with
344
Binghamet aL,
PSYCHOSOCIALPROBLEMSIN PRIMARY CARE
patients; all patients having a c o m p r e h e n s i v e examination c o m p l e t e the PROMIS questionnaire. An analysis o f the first group of examinations was r e p o r t e d to the practice by the clinical social w o r k e r in terms of ratios of identified and missed psychosocial problems according to the judgment of the social worker. The practice then selected an audit c o m m i t t e e for psychosocial problems and asked this c o m m i t t e e to review the social worker's audit and r e c o m m e n d practice standards for identification of and treatment planning for psychosocial problems. The standards recomm e n d e d by the c o m m i t t e e consisted of indicators on the PROMIS database that w o u l d mandate inclusion of a psychosocial problem on the patient's p r o b l e m list. The psychosocial p r o b l e m audit committee defined case-finding requirements that w o u l d include a broad spectrum of psychosocial problems, leaving it to the individual physician to determine what specific problem title or diagnosis to apply to the individual patient. Thus, a psychosocial p r o b l e m w o u l d be included in the p r o b l e m list w h e n any patient indicated certain combinations of complaints on the psychosocial section of the database, i.e., a scale score o f 10 or more on the Popoff Depression Scale, a score of 300 or more on the Holmes Scale, or certain other combinations of responses. The audit committee also designed a flow sheet for psychosocial problems, to be c o m p l e t e d by the medical assistant. This flow sheet w o u l d indicate to the physician w h e t h e r or not each patient met the standards for identification of a psychosocial problem, e.g., an elevated depression scale. The audit committee also defined the standard that treatment planning w o u l d be judged adequate w h e n the plan indicated at least one of the following: 1) follow-up with the physician for the problem; 2) referral to a psychotherapist or counselor; 3) statement that the patient is already in treatment for the problem; 4) patient refusal of treatment for the problem; 5) current inactivity of the problem. On the r e c o m m e n d a t i o n of the psychosocial problem audit committee, the practice agreed to accept the case-finding standards. This meant that members of the practice agreed thereafter to use the PROMIS case-finding tool and the flow sheet for psychosocial problems in one standard way so that performances in identifying psychosocial problems could be c o m p a r e d across the practice, and all members of the practice agreed to include psychosocial problems on the p r o b l e m list according to these guidelines w h e n their direct evaluation of the patient s u p p o r t e d the case-finding data (step 3 in Figure 1). If the physician, on investigation with the patient concerning the database responses, found that p r o b l e m identification was not warranted, the physician w o u l d indicate on the database that the case-finding item was "Not a P r o b l e m " (NP). The physician's judgment on interview with the patient was a c c e p t e d as authoritative for presence or absence of a psychosocial problem.
TABLE 1
Patient Demographics Pilot Study Period (n = 319)
Follow-up Audit Period (n = 204)
Gender Male Female
153 (48%) 166 (52%)
78 (38%) 126 (62%)
Age Mean Range
37.6 15-90
46.4 18-84
No attempt was made to judge the accuracy o f the physician's diagnostic acumen, since the goal was simply to define the presence or absence of the class ofpsychosocial problems. After the standards for audit had b e e n established and accepted by the practice, two additional samples were audited during the pilot study period, the flow sheet was tested, and the results for conformance with the standards for identification and treatment planning were discussed with the practice. Performances on specific cases were discussed with individual practitioners so that the standards, the flow sheet, and the audit process were clearly understood. The final step in the study was to place the audit for identification of psychosocial problems into the ongoing regular audit p r o c e d u r e for all medical problems. This audit p r o c e d u r e is c o n d u c t e d internally by a quality assurance/research staff person w h o periodically reports back to the practice on h o w well the members meet the standards they have adopted for c o m m o n medical problems. In this final step of the study, the presence or absence of a psychosocial p r o b l e m on the problem list was audited against the adopted standards in 204 charts in June 1988. In this regular audit, only faculty physician performance is reviewed. The audit does not include resident performance, since that task has already b e e n accomplished by the faculty preceptors. 23 The statistical tool utilized in the study was the chi-square distribution comparing the chart audit during the pilot study with the regular internal practice audit. The hypothesis to be tested was w h e t h e r the audit process yielded a statistically significant increase in identification of and treatment planning for psychosocial problems.
RESULTS Table 1 summarizes the demographic characteristics of the two groups of patients c o m p a r e d for this study. Since there was no significant difference between the three pilot sample groups for identification of psychosocial problems, these were treated as one group and c o m p a r e d with the subsequent regular-audit group one year after beginning the study. The two
JOURNALOFGENERALINTERNALMEDICINE,Volume 5
groups for comparison consisted of 319 patients in the pilot study and 204 patients in the regular-audit group. As shown in Table 2, psychosocial problems identified by the physicians increased from 67% during the pilot study to 90% in the subsequent regular audit (p < 0.05). Treatment planning was not audited by the practice auditor, so the comparison of performances on meeting standards for treatment planning in Table 3 shows data for the first and the last sample in the pilot study. The results listed in Table 3 demonstrate that the physicians improved their overall performance in making treatment plans for patients with psychosocial problems from 79% in the first sample of the pilot study to 100% in the last sample of the pilot study (p < 0.05). An analysis comparing the performance of the PROMIS case-finding instrument with faculty identification of psychosocial problems for the 240 cases seen by faculty is provided by the 2 X 2 diagram in Table 4. The case-finding tool had a sensitivity of 0.71 and a specificity of 0.82. This means that the PROMIS instrument had relatively low false-positive and false-negative rates when compared with how this group of internists judged the presence or absence of a psychosocial problem. It is our contention that internists skilled in
TABLE 2
No.
No. of Patients Pilot study
319
114
Regularaudit
204
45
TOTAL
523
159
Physician and Case Finder
Case Finder Only
76 (67%)-- 7 38 t 41 (90%)--J 4
*Patients who scoredpositiveon the casefinder pluspatients identified by the physicianwho did not score positive on the casefinder (false negatives). tp
cantiy increase the rate o f identification of psychosocial problems through routine use o f case-finding instruments can be established in a general medicine practice. Design: Two groups o f patient examination reports written by physicians were retrospectively compared with the patients" responses on the case-finding database instrumer~ The samples were obtained by s e ~ selection in
f o u r time periods. Setting a n d Patients: The study occurred in a university general internal medicine practice that utilizes the problem-oriented record~ The patients studied were seen f o r first-time comprehensive examinations designed to identify all important health problems, includins psychosocial
prob~ms. I n t e r v e n t i o n : The authors compared performances o f the physicians in identification o f psychosocial problems bef o r e and after t h e intervention, which consisted o f a pilot study audit o f psychosocial problem identification, establishment o f standards f o r interpretation o f the case-find. ing instrumen~ design o f a flow sheet to make case-finding data clearly available to the physician at each comprehensive examination, and feedback o f physician performance according to practice-adopted standards f o r identification o f psychosocial problems. Measurement: The resu/t o f the i ~ was an increase in po~hosocial problem identification f r o m 67% to 9056 o f problems preseng p < 0.05 by chi-square distribution,, or a decrease f r o m 33% to 1056 inpsychosocialproblems missed by the physicians. Conclusion: The quality improvement process f o r identification o f psychosocial problems described in this report significantly i ~ a s e d the rate o f identification o f psychosocial problems by general internists. Key w o r d s : psychosocial problem identification,, primary care; quality improvemen~ case-finding instruments. J GI~I INrvatN M.ED 1990; 5:342-346.
PATIENTS IN PRIMARY CARE n e e d t o d i s c u s s p s y c h o s o c i a l
problems with their physicians. A recent study b y G o o d et al. ~ demonstrated that over 70% o f primary care patients "find it appropriate to turn to their primary care physicians for help with emotional distress, family problems, life stress, behavioral problems, and sexual dysfunction," but only one-fifth to one-third of patients w h o have e x p e r i e n c e d psychosocial difficulties have Received from the Department of Psychiatry (RLB) and the Department of Medicine (DAP, DLB, HMT), University of Vermont, and the Given Health Care Center (KMcK), Burlington, Vermont. Presented at The Treatment of Mental Disorders in General Health Care Settings: A Research Conference, sponsored by the National Institute of Mental Health, University of Pittsburgh School of Medicine, The Upjohn Co. June 15 - 17, 1988. Address correspondence and reprint requests to Mr. Bingham: Given Health Care Center, University Health Center, One South Prospect Street, Burlington, VT 05401.
342
discussed these problems with their primary care providers. Failure to recognize psychosocial problems will lead to inappropriate testing, misuse o f medical resources, and p o o r therapeutic results. 24 Patients w h o have psychosocial problems and are seen in primary care settings include those with serious mental illness as well as those w h o have situational stresses. Estimated prevalences of psychosocial problems in primary care patients range from 30% to 60%, depending on the s o c i o e c o n o m i c level of the population and w h e t h e r identified cases are restricted to conformance with psychiatric DSM-III diagnostic standards. ~'7 However, there are many barriers to successful identification of and treatment planning for psychosocial problems in the primary care setting, s, 9 Patients w h o have psychosocial problems need to be identified within the usual practice environment in a systematic manner that is economically feasible and makes efficient use of the physicians' skills. Given the natural variability among physicians in collecting information, many patients' psychosocial problems will not be detected unless the primary care practice has a system for data collection and organization that includes effective case-finding instruments for psychosocial problems. We accept the distinction made by Coulehan and others,0, t~ that case finding rather than screening is best understood as the aim of a patient-completed questionnaire. The aim of the questionnaire is not diagnosis but to alert the physician, w h o can then determine w h e t h e r a p r o b l e m is present and formulate a diagnosis and treatment plan at his or her level of expertise. An effective system o f practice will utilize standards for identification o f psychosocial problems that are i m p l e m e n t e d with daily practice procedures, and an internal audit with feedback of performance to the adopted standards. ~2-t5 Coulehan et al., 1o based on a study of the efficiency of using questionnaires for the case finding o f depression in primary medical care, state that routine use of such questionnaires is not " a p p r o p r i a t e " because they are too c u m b e r s o m e for total coverage o f the practice. They suggest that case-finding questionnaires be utilized only for high-risk patients. We, on the contrary, maintain that it is feasible to use case-finding instruments for psychosocial p r o b l e m identification in all patients in the practice w h o are seen for c o m p r e h e n sive care. Psychosocial case finding can be efficiently and effectively included in the standard database that is collected from all patients through patient-completed
JOURNALOFGENERALINTERNALMEDICINE,Volume S (July/August), 1990
questionnaires, and through methods that summarize and consolidate the database for efficient utilization by the physician. This report describes a primary care system for the identification of psychosocial problems developed by the Given Health Care Center at the University of Vermont, and how this system increased identification o f psychosocial problems in this practice. METHODS
343
PROMIS Questionnaire Form completed at home by patient
Data Base Office Visit o PROMIS data summarized and tests performed by medical assistant
Practice Setting
The Given Health Care Center, organized in 1971 as a problem-oriented medical practice, serves as the general internal medicine faculty practice site for the University of Vermont School of Medicine. The practice cares for 25,000 patients, who are seen for 48,000 patient visits per year, and is staffed by 13 faculty physicians, primary care medical residents, two nurse practitioners, one physician assistant, and two clinical social workers. Approximately 10% of patient visits are with the resident physicians. Two thousand new patients are seen for first-time complete health examinations each year. These comprehensive examinations are structured to identify all important health problems, including psychosocial problems. Comprehensive Examination
Each patient seen for t h e " Initial Complete Exam" is asked to fill out the PROMIS III Adult Medical History Questionnaire for Problem Oriented Practices. 16 This questionnaire, developed in part by Given, is now utilized by 17 other general medical practices. The questionnaire contains 342 questions, with 106 of these questions pertaining to psychosocial aspects of the patient's life. The questionnaire includes the Popoff Depression Scale, 17-20a Problems of Daily Living Scale, the Holmes Life Events Scale, 21 and a 14-question alcoholism case finder. The psycbosocial database is a major feature of the PROMIS questionnaire, accounting for 31% of the total review-of-systems questions. This database functions as a case-finding instrument, as defined by Coulehan et al., lO for a broad spectrum of psychosocial problems, rather than limiting identification to the problem of depression. The patient completes the PROMIS database as the first step in the sequence described in the flow diagram of Figure 1. Notice that at the second step, the PROMIS data are summarized by the medical assistant. This process includes transfer by the assistant of information from the questionnaire to a database flow sheet. This information is clerically categorized by the medical assistant according to the standards defined by the physicians. This step is crucial to effective use of all the case-finding sectors of the patient-completed database because it makes case-finding information readily available to the physician at every comprehensive examination.
Physician Visit o Physician review of data base o Perform physical examination o Discuss findings (including PROMIS psychosocial data) with patient o Negotiation of goals for care o Formulation of treatment plans
Problem-Oriented Report r e p o r t (including psychosocial problem, if present} sent to patient
o Written
FIGURE 1. Process for identification and treatment planning for psychosocial problems in the primary care setting.
Research Design
For this study, the patient questionnaire answers on the psychosocial sections were audited by one of the clinical social workers and compared with the physicians' written reports derived from the same database with regard to whether the physicians had identified psychosocial problems. We did not attempt to determine whether a physician had made a correct DSM-III diagnosis, but onlywhether the physician had included a psychosocial problem in the problem list when the patient had signaled distress on the questionnaire. The study compared two groups of patient examination reports: a pilot study group, and a group of examination reports reviewed by the practice auditor after completion of the pilot study. The pilot study for psychosocial problem identification and treatment planning was performed between August 1, 1987, and April 30, 1988. Two hundred and forty faculty charts and 79 resident charts of first-time complete health examinations were selected sequentially during three different time periods for retrospective examination by the clinical social worker. 22 Total sample size for the pilot study was 319 patients, or 16% of patients undergoing this type of examination seen in one year. During the first sampling period of the pilot study, the physicians were unaware that a study was occurring. At no time were patients aware of a study, since there was no change in the usual procedures for data collection with
344
Binghamet aL,
PSYCHOSOCIALPROBLEMSIN PRIMARY CARE
patients; all patients having a c o m p r e h e n s i v e examination c o m p l e t e the PROMIS questionnaire. An analysis o f the first group of examinations was r e p o r t e d to the practice by the clinical social w o r k e r in terms of ratios of identified and missed psychosocial problems according to the judgment of the social worker. The practice then selected an audit c o m m i t t e e for psychosocial problems and asked this c o m m i t t e e to review the social worker's audit and r e c o m m e n d practice standards for identification of and treatment planning for psychosocial problems. The standards recomm e n d e d by the c o m m i t t e e consisted of indicators on the PROMIS database that w o u l d mandate inclusion of a psychosocial problem on the patient's p r o b l e m list. The psychosocial p r o b l e m audit committee defined case-finding requirements that w o u l d include a broad spectrum of psychosocial problems, leaving it to the individual physician to determine what specific problem title or diagnosis to apply to the individual patient. Thus, a psychosocial p r o b l e m w o u l d be included in the p r o b l e m list w h e n any patient indicated certain combinations of complaints on the psychosocial section of the database, i.e., a scale score o f 10 or more on the Popoff Depression Scale, a score of 300 or more on the Holmes Scale, or certain other combinations of responses. The audit committee also designed a flow sheet for psychosocial problems, to be c o m p l e t e d by the medical assistant. This flow sheet w o u l d indicate to the physician w h e t h e r or not each patient met the standards for identification of a psychosocial problem, e.g., an elevated depression scale. The audit committee also defined the standard that treatment planning w o u l d be judged adequate w h e n the plan indicated at least one of the following: 1) follow-up with the physician for the problem; 2) referral to a psychotherapist or counselor; 3) statement that the patient is already in treatment for the problem; 4) patient refusal of treatment for the problem; 5) current inactivity of the problem. On the r e c o m m e n d a t i o n of the psychosocial problem audit committee, the practice agreed to accept the case-finding standards. This meant that members of the practice agreed thereafter to use the PROMIS case-finding tool and the flow sheet for psychosocial problems in one standard way so that performances in identifying psychosocial problems could be c o m p a r e d across the practice, and all members of the practice agreed to include psychosocial problems on the p r o b l e m list according to these guidelines w h e n their direct evaluation of the patient s u p p o r t e d the case-finding data (step 3 in Figure 1). If the physician, on investigation with the patient concerning the database responses, found that p r o b l e m identification was not warranted, the physician w o u l d indicate on the database that the case-finding item was "Not a P r o b l e m " (NP). The physician's judgment on interview with the patient was a c c e p t e d as authoritative for presence or absence of a psychosocial problem.
TABLE 1
Patient Demographics Pilot Study Period (n = 319)
Follow-up Audit Period (n = 204)
Gender Male Female
153 (48%) 166 (52%)
78 (38%) 126 (62%)
Age Mean Range
37.6 15-90
46.4 18-84
No attempt was made to judge the accuracy o f the physician's diagnostic acumen, since the goal was simply to define the presence or absence of the class ofpsychosocial problems. After the standards for audit had b e e n established and accepted by the practice, two additional samples were audited during the pilot study period, the flow sheet was tested, and the results for conformance with the standards for identification and treatment planning were discussed with the practice. Performances on specific cases were discussed with individual practitioners so that the standards, the flow sheet, and the audit process were clearly understood. The final step in the study was to place the audit for identification of psychosocial problems into the ongoing regular audit p r o c e d u r e for all medical problems. This audit p r o c e d u r e is c o n d u c t e d internally by a quality assurance/research staff person w h o periodically reports back to the practice on h o w well the members meet the standards they have adopted for c o m m o n medical problems. In this final step of the study, the presence or absence of a psychosocial p r o b l e m on the problem list was audited against the adopted standards in 204 charts in June 1988. In this regular audit, only faculty physician performance is reviewed. The audit does not include resident performance, since that task has already b e e n accomplished by the faculty preceptors. 23 The statistical tool utilized in the study was the chi-square distribution comparing the chart audit during the pilot study with the regular internal practice audit. The hypothesis to be tested was w h e t h e r the audit process yielded a statistically significant increase in identification of and treatment planning for psychosocial problems.
RESULTS Table 1 summarizes the demographic characteristics of the two groups of patients c o m p a r e d for this study. Since there was no significant difference between the three pilot sample groups for identification of psychosocial problems, these were treated as one group and c o m p a r e d with the subsequent regular-audit group one year after beginning the study. The two
JOURNALOFGENERALINTERNALMEDICINE,Volume 5
groups for comparison consisted of 319 patients in the pilot study and 204 patients in the regular-audit group. As shown in Table 2, psychosocial problems identified by the physicians increased from 67% during the pilot study to 90% in the subsequent regular audit (p < 0.05). Treatment planning was not audited by the practice auditor, so the comparison of performances on meeting standards for treatment planning in Table 3 shows data for the first and the last sample in the pilot study. The results listed in Table 3 demonstrate that the physicians improved their overall performance in making treatment plans for patients with psychosocial problems from 79% in the first sample of the pilot study to 100% in the last sample of the pilot study (p < 0.05). An analysis comparing the performance of the PROMIS case-finding instrument with faculty identification of psychosocial problems for the 240 cases seen by faculty is provided by the 2 X 2 diagram in Table 4. The case-finding tool had a sensitivity of 0.71 and a specificity of 0.82. This means that the PROMIS instrument had relatively low false-positive and false-negative rates when compared with how this group of internists judged the presence or absence of a psychosocial problem. It is our contention that internists skilled in
TABLE 2
No.
No. of Patients Pilot study
319
114
Regularaudit
204
45
TOTAL
523
159
Physician and Case Finder
Case Finder Only
76 (67%)-- 7 38 t 41 (90%)--J 4
*Patients who scoredpositiveon the casefinder pluspatients identified by the physicianwho did not score positive on the casefinder (false negatives). tp
