ORIGINAL RESEARCH * NOUVEAUTES EN RECHERCHE
Use of reminders for preventive procedures in family medicine Walter W. Rosser,* MD; Ian McDowell,t PhD; Claire Newell,t MA Objective: To compare the effectiveness of three computerized reminder systems in the delivery of five preventive procedures in family practice. Design: Prospective, randomized, controlled study. Setting: Ottawa Civic Hospital Family Medicine Centre. Participants: Of 8502 patients 15 years of age or more who were not in a hospital or institution 5883 were randomly assigned, by family, to a control group, a physician reminder group (passive) or a telephone or letter reminder group (active). The remaining 2619 patients were not included in the randomized portion of the study but were monitored. Intervention: During 1 year the patients in the active reminder groups received a telephone call or letter reminding them of any overdue preventive procedures; for those in the passive reminder group the physician was reminded at an office visit to provide any overdue service. Outcome measure: Rates of completion of the preventive procedures required. Main results: All three reminder systems significantly improved the delivery of preventive services (p < 0.001). The procedure completion rates were 42.0% in the letter reminder group, 42.0% in the telephone reminder group, 33.7% in the physician reminder group and 14. 1% in the randomized control group. The use of a letter was more cost-effective than the telephone system, but the physician reminder system was the most cost-effective. Conclusion: Computerized reminder systems do improve the delivery of preventive services in family practice.
Objectif: Comparer l'efficacite de trois systemes informatises de rappel dans la prestation de cinq interventions preventives en medecine familiale. Conception: Etude prospective, aleatoire et contr6lee. Contexte: Centre de medecine familiale de l'H6pital Civic d'Ottawa. Participants: Parmi 8 502 patients de 15 ans ou plus non confines a un h6pital ou a un etablissement, 5 883 ont ete affectes au hasard, selon la famille, a un groupe de contr6le, a un groupe de rappel par les medecins (passif) ou a un groupe de rappel par telephone ou par courrier (actif). Les 2 619 autres patients n'ont pas ete inclus a la partie aleatoire de l'etude, mais ont ete suivis. Intervention: Pendant 1 an, les patients des groupes de rappel actif ont recu un appel telephonique ou une lettre leur rappelant toute mesure preventive echue. Dans le cas des membres du groupe de rappel passif, on a rappele aux medecins d'effectuer toute intervention echue a l'occasion d'une consultation. Mesure des resultats : Taux d'execution des mesures de prevention requises.
From *the Department of Family and Community Medicine, University of Toronto, Toronto, Ont., and tthe Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Ont.
Reprint requests to: Dr. Walter W. Rosser, Department ofFamily and Community Medicine, University of Toronto, 444 Yonge St., Toronto, ON M5B 2H4 -
For prescribing information see page 875
CAN MED ASSOC J
1991; 145 (7)
807
Principaux resultats: Les trois systemes de rappel ont ameliore considerablement la prestation de mesures preventives (p < 0,001). Les taux d'intervention se sont etablis a 42,0 % dans le cas du groupe de rappel par courrier, 42,0 % dans celui du groupe de rappel par telephone, 33,7 % dans celui du groupe de rappel aux medecins et 14,1 % dans celui du groupe de contr6le aleatoire. Le courrier s'est revele plus rentable que le telephone, mais le systeme de rappel aux medecins a et le plus rentable. Conclusion: Le systeme de rappel informatise ameliore la prestation de mesures preventives en medecine familiale.
S ince 1979 the Canadian Task Force on the Periodic Health Examination has identified specific preventive procedures that meet criteria for effectiveness in family practice.' Work of the US Preventive Services Task Force, appointed in 1985, further supports the concept of critically assessing preventive screening procedures. In 1987 a joint meeting of the Canadian and US task forces concluded that despite their work large numbers of Canadians and Americans were not receiving appropriate preventive screening.2 A joint publication from the two groups gives a comprehensive set of guidelines for primary prevention in family practice.3 Physicians believe that most of their patients receive appropriate preventive screening in their practices.4'5 However, studies have reported that actual levels of such screening are low in practices and the community. Four studies of practice populations revealed that only 12% to 19% of patients over 65 years of age had been vaccinated against influenza.6-9 Surveys of hypertension screening in practice populations or the community showed that 50% to 90% of patients receive appropriate blood pressure checks. '10' In two practices the smoking status of patients was recorded in only 18% and 23% of the medical records.16"17 Less than 30% of the women in six studies received appropriate screening for cervical cancer.'8-23 Passive or active reminders may be used to improve the rates of delivery of preventive services. Passive reminders occur during regular visits and alert the physician or nurse that preventive procedures are overdue. With active reminders the patients are contacted by telephone or mail to encourage them to come in for overdue procedures. One evaluation of passive reminder systems reported significant improvements in the rates of delivery of preventive procedures when the reminders remain in place for a number of years. A passive reminder system for the physicians in more than 20 family practices in western Ontario resulted in 87% of the patients having their blood pressure checked once in 5 years." However, those who seldom visit the practice, often including those at high risk, may not be reached. Active reminder systems have been found to improve the rate of influenza vaccination in the elderly from 18% to 53%.6-9 Active reminders, in808
CAN MED ASSOC J 1991; 145 (7)
cluding home visits, have been used with some success to improve rates of delivery of preventive services among those not responding to a reminder to undergo a Papanicolaou test.20-23 Several surveys of patients with cancer of the cervix have concluded that women at highest risk are the least likely to have been screened. 1-23 The growing use of computers in primary care2425 prompted us to evaluate the costeffectiveness of both passive and active reminder systems. To our knowledge no one has compared the two systems for more than one preventive procedure in a randomized trial. The purpose of this paper was to analyse the effect of age and sex variables and practice patterns on the effectiveness of reminder systems and to determine whether a computerized reminder system improves the delivery of preventive services. We compared a passive physician reminder with active telephone and letter reminders to measure the improvement in the number of preventive services delivered for influenza vaccination, blood pressure measurement, determination of smoking status, Papanicolaou testing and tetanus vaccination, as recommended by the Canadian Task Force on the Periodic Health Examination. Papers on three of the procedures have been published elsewhere.26 28 Three hypotheses were tested: (a) active (telephone or letter) reminders are more effective than passive (physician) reminders in the delivery of preventive services, (b) the rate of delivery of preventive procedures decreases with increasing complexity (number of steps required) to administer the procedure and (c) telephone reminders (oral) are more effective than letter reminders (written). The number of steps required to complete a procedure determined our rating of complexity. Smoking assessment required the answer to "Do you smoke?" and, if Yes, "How many cigarettes per day?" Blood pressure readings were more complex, requiring access to the instrument and time to take the measurement. Influenza or tetanus vaccination required a series of questions to determine need, an explanation of the risks and retrieval of the appropriate vaccine; for the patient there is the inconvenience and discomfort of having an injection. Papanicolaou testing was the most complex, because the procedure is uncomfortable, requires the most time and may be done only when the woman is not menstruating. LE ler OCTOBRE 1991
..M
Methods Since 1976 all patients attending the Ottawa Civic Hospital Family Medicine Centre have been registered on a computer database.24,25 At the time of our study the practice was organized into six teams, each comprising a staff physician, a nurse and three to five residents and providing primary care to approximately 1200 patients. For regular office visits, by appointment, the patients always visited the same team. During off hours or in emergencies patients could visit any of the physicians or residents who were on call. In 1984 steps were taken to determine the active practice population. Letters were sent to 3722 patients who had not attended the practice in the 18 months before the start of the study to determine their status in the practice. Nonresponders or those stating that they no longer belonged to the practice were excluded. We included the remaining 8502 patients 15 years of age or more who were not in a hospital or an institution. Two of the six teams (representing 2619 patients) did not participate in the randomized component of the study; however, these patients were monitored. With the use of a standard randomization computer program the remaining 5883 patients were assigned, by family, to the control group (1056 families, 1403 people), the physician reminder group (1122 families, 1471 people), the telephone reminder group (1104 families, 1468 people) or the letter reminder group (1168 families, 1541 people). Patients were allocated by family to avoid the contamination that would occur if individuals in a family were assigned to a different team.
Five screening procedures involving different and sex groups and different complexities were included. Procedures required annually were influenza vaccination of those over 65 years of age, blood pressure measurement for those 18 years or more, assessment of smoking status for patients 15 years or more and Papanicolaou testing for all sexually active women 18 to 35 years old. The administration of tetanus toxoid is recommended once every 10 years for everyone over 18 years of age. Since influenza vaccination is seasonal and since we wished to do our follow-up 1 year later the influenza portion of the study was conducted between October 1984 and January 1985, the follow-up occurring during the same months in 1985-86. The rest of the study was performed between Apr. 1, 1985, and Mar. 31, 1986. To establish a starting point before Oct. 1, 1984, and Apr. 1, 1985, the computer scanned all of the patients records to determine whether the procedures had been carried out within the required time. Of the 8502 patients 38 required no procedure and
age
OCTOBER. 1, 1991
thus were excluded from further analysis. The age and sex groups corresponded to those recommended by the North American Primary Care Research Group.29 For patients assigned to the randomized control group no action was taken to remind the physicians or the patients that a procedure was overdue. The families in the two nonparticipating groups were used as a reference point to assess the extent of contamination caused by increased preventive activity in the control group. For families assigned to the physician reminder group a computer-generated reminder was included on the routinely printed encounter form before any visit to the office to remind the physician of outstanding preventive procedures. To validate the reliability of recording preventive actions on the computer 30 consultations with patients known to require one or more procedure were directly monitored by another physician or a research assistant through one-way windows without the knowledge of the study physician. The monitoring determined whether the physician had dealt with the required procedure during the visit and verified that services provided were accurately recorded on the computer record. This validation applied to the physician reminder group only; an internal audit was done in the two active reminder groups. Families assigned to the two active reminder groups were randomly assigned to 24 equal-size groups for ease of providing reminders and evenly distributing the demand for preventive services. At the beginning of each 2-week study period the computer printed a list of the names and telephone numbers of patients with overdue screening procedures. For the telephone reminders the practice nurse attempted to contact the family, making a maximum of five calls during working hours, and completed an action form for each listed patient. Once contact was made the nurse advised the patient about the indicated procedures and then attempted to arrange for them to be performed. The person answering the telephone was asked to relay the message to other family members. Families assigned to the letter reminder group received a computer-generated letter, signed by their physician and nurse, describing the procedures that were overdue for each member of the family and the importance of having them performed. After 21 days a second reminder was sent out to nonrespondents. In all three reminder groups the patients were asked if they had undergone the procedure elsewhere in the required timeframe and if this could be verified in writing. If Yes, the procedure was recorded as being completed. The time required by the physicians to ask about the procedures and the time required by the clerical staff to run the computer to CAN MED ASSOC J 1991; 145 (7)
809
generate letters and issue reminders were recorded and verified. Nurses kept a record of the number of calls made and timed each call with a stopwatch. Outcome was expressed as the proportion of patients in each study group for whom the recommended procedure was performed, as indicated in the computer database.
The 30 visits monitored and validated by an observer revealed that 15% of the completed procedures that the physician was reminded to perform were not recorded. Of the 1471 patients assigned to the physician reminder group only 766 (52.1%) visited the practice during the trial year. Of the families assigned to the telephone reminder group 1104 required a reminder for one or more procedures. Sixty-seven were not contacted because they did not have a telephone, were deaf or did not understand English or French. Of the remaining families 684 (66.0%) were contacted after 2.1 calls on average, each requiring 2 minutes and 48 seconds. Of those not contacted 235 (56.0%) had obsolete telephone numbers, and 185 (44.0%) could not be contacted after five attempts. Letters were issued to the 1442 people who needed one or more procedures updated. Although 164 (14.0%) of the letters were returned as undeliverable we assumed that the other 1004 (86.0%) were received. A second letter had to be sent to 653 (65.0%) of the families. Table 1 lists the proportion of required procedures that were performed during the study period. For those in the physician reminder group who visited the physician 60.4% of the required procedures were completed. However, this represented only 1471 (33.3%) of the 4413 procedures required by those in that group. The other two reminder systems were equally effective (Table 2); they were associated with significantly more completed procedures (42.0%) than the physician reminder system was (p < 0.05). The older the patient the more likely he or she was to visit the physician during the year. This contributed to the more than twofold increase in the completion rates for passive reminders among men
Statistical analysis The significance of the differences in proportions was determined with the use of formula 9.4, as described by Fleiss.30 If the difference was found to be significant further analysis, with the use of Fleiss's formulas 9.10 and 9.11, identified the groups that contributed to that difference.
Results To assess the adequacy of the randomization we compared the study groups in terms of family size, age and sex. There was no significant difference in sex (X2 = 3.3, 4 degrees of freedom [dfJ). Family size differed significantly between the groups, but after the patients in the nonparticipating practices (which tended to have larger families) were excluded the association disappeared. An analysis of variance of age showed no significant difference between the four randomized groups. Thus, we were confident that any differences in the study outcomes were unlikely to have resulted from differences in the group characteristics. The number of procedures required in each group at the trial start was compared with the number completed by the end of the study. Because of the Papanicolaou testing the average number of procedures required for women was higher than that for men (3.04 v. 2.61).
Table 1: Proportion of procedures required at start of trai that were performed during the study period. by patient sex, age and study group
Group: Sex. age. yr Male 15-34 35-64 65 All Female
15--34 35-64 65 All
of procedures performed
Nonrandomized
Randomized
Physicarn
Letter
Telephone
control
control
re minder
re rnnder.
reminder
664) 20.5 34.7 44.7 30.3 nr ----- 807)
(rin. 660)
in
(n
1143) 7.3 10.4 1i5.7
9.9 1476)
(n
9.7 11.2
11.5 10.6
622)
(n
8.3 14.9 13.7 12.3 (on - 781) 13.6 14.5 10.7
35
3i
(n --
31.3*`
37
49.4---
43
52.
*
43.0* fri
--
881;
623) /< :t
43 0741
(Il
845)
45.1
38.4
35.8*. 45.8* . 4a.1*
33.7
42.0*
42
26.7` 38.8&
39
9t
OM..
*Significantly greater than proportion irn randomized control group (p. n.O1. tSignificantly greater than proportion in physician reminder group (p 0.051 Significantly different from proportion in letter reminder group (p O 05)
810
CAN MED ASSOC J 1991; 145 (7)
LE 1 er OCTOBRE 1991
65 years, as compared with the rates among aged 15 to 34 years (Table 1). The telephone reminder was significantly more effective than the letter reminder among men 15 to 44 years (p < 0.05), whereas the letter reminder was significantly more effective than the telephone reminder among older men (p < 0.05). Women had a higher response rate to passive reminders than men because they visited their physicians more often. Among women under 64 years the effectiveness of the letter reminder was similar to that of the telephone reminder, but among women 65 years or more the letter was significantly more effective than the telephone reminder (p < 0.05). The first hypothesis, that either active reminder would be more effective than the passive reminder, was supported in all age and sex groups. When patients did visit the physician 60.4% of the required procedures were completed, as compared with 42.0% of the procedures in the two active reminder groups; this demonstrated the importance of the visit rate for the physician reminder group. The hypothesis that completion rates decrease as a procedure's complexity increases was not supported by our results (Table 2). The complexity of the procedure was unrelated to completion rates in the randomized control group. The hypothesis was supported in the physician reminder group by a gradient in response from the least to the most complex procedure. It was supported to a lesser extent in the two active reminder groups, in which the gradient was less clear. In the cost analysis we assumed that the computer system costs were the same for all of the interventions. A detailed cost breakdown for each of the five preventive procedures has been reported elsewhere.26-28 The physician reminder was the most cost-effective method of improving preventive services, because the time required to perform the procedure once the patient attended the clinic was minimal. Given the clinical and mailing costs the letter reminder was the next most cost-effective method. Cost-effectiveness was calculated by determining the cost of each procedure completed in
over men
excess of the number completed in the control group. The telephone reminder was the least cost-effective, mostly because the cost of having the nursing staff do the telephoning to obtain the extra procedures was high. The hourly wage of the person telephoning the families played a. significant role in the estimates of cost-effectiveness. We did not assess whether there would be any difference in outcome between a clerical person and a nurse delivering the telephone message.
Discussion Despite our preliminary efforts to determine the eligible practice population 17% of the people contacted in the telephone reminder group stated that they were no longer part of the practice. In all, 14% of the letters were not deliverable, and 3% of the patients who received a letter stated that they were no longer part of the practice. The overestimate by about 18% of the denominator may have accounted in part for the low visit rate (52.1%). The highest turnover rate occurred in the lowest age group, which may account in part for the weakness of the intervention in this group. Informal reports from Canadian practices suggest an annual loss rate of 9% to 12% in urban family practices, and since our study occurred over 20 months the rate of attrition is reasonable. The effect this loss of patients had in each group was to diminish the completion rate, which was lower than that in other studies.813-161921 However, we included only people who had not already voluntarily obtained preventive services. We also provided only a single oral reminder and two written reminders, whereas Bass," for example, showed a cumulative effect of repeated reminders over 5 years. Improving the level of preventive services in a practice population requires overcoming a number of obstacles. At the beginning of our trial the nurses and the physicians expressed concern that the patients might be irritated by receiving letters or telephone calls. Most of the patients, however, were pleased that their physician was concerned enough to
Table 2: Distribution of required procedures that were performed, by study group
Nonrandomized Procedure
Administer influenza vaccine Measure blood pressure Assess smoking status
Obtain Papanicolaou smear Administer tetanus vaccine
OCTOBER 1, 1991
control
Group; % of procedures performed Letter Randomized Physician reminder reminder control
Telephone reminder
3.8 18.6 9.5 11.2
9.8 21.1 11.9 13.7
22.9 30.7 37.9 16.5
35.2 40.5 49.1 29.7
37.0 37.2 55.8 30.0
2.3
3.2
22.8
30.6
24.0
CAN MED ASSOC J 1991; 145 (7)
811
contact them about overdue preventive procedures. Middle-aged people were the most responsive to the reminders. If we wish to realize maximum benefit from early detection in terms of life-years saved, people less than 45 years of age remain the most important target group. The lack of response among those more than 75 years may be secondary to both a declining interest in prevention as well as an increasing infirmity resulting in more frequent house calls and less frequent office visits. A number of approaches (e.g., media campaigns, population or community clinics, and home visits by public health nurses) to improve compliance with preventive procedures among those under 40 years should be evaluated. Workplace clinics may also be a more effective approach to achieving higher levels of preventive screening among people less than 45 years provided that appropriate linkage with family physicians is maintained for follow-up purposes. The complexity of the procedures was found to be a less important variable than we had expected. The more complex the procedure the less likely the physician was to complete the procedure during a regularly scheduled office visit. However, when the patient was recalled and prepared for the required intervention complexity was not a barrier. The recommendation that any preventive procedure is required annually reduces the population exposed to the physician, as compared with that for a procedure required every 2 or 3 years. This fact should be carefully considered when guidelines for frequency of procedures are being developed. The finding that telephone reminders were more effective than letter reminders in the lower age groups and the reverse in the older groups was unexpected. A number of respondents over 65 years of age who received a letter telephoned the nurse about how they should respond to it; this may have influenced the results. The preventive procedure completion rate in the physician reminder group indicates that this approach is the most powerful. Most studies have found that 70% of patients in a practice population visit their physician annually, the proportion rising to between 75% and 80% for visits once every 2 years. Although the cost analysis was simplistic we believe that with the use of the physician reminder system a significant improvement in preventive services can be achieved for less than $5 per extra procedure completed. Our analysis of the cost of the telephone reminder system was influenced by the use of the practice nurses, who were paid at a relatively high hospital rate. The use of a clerk or a part-time student at minimum wage could substantially reduce the cost of this approach, although an assessment of the response to telephone calls from a person unknown to the patient would be required. 812
CAN MED ASSOC J 1991; 145 (7)
We postulated that patients become dependent on being reminded after they have responded to an active reminder. Follow-up 1 year later revealed that the improvement in influenza vaccination rates achieved by reminders was not sustained. Some did not show up for vaccination 1 year after the reminder apparently because they were waiting for a reminder.28 One approach to reducing patient dependence on reminder systems would be to empower patients to assume responsibility for their preventive health. This could be achieved through the recording of the status of preventive procedures on a patient-held "smart card"; the patient could then be asked to assess his or her preventive health status by reading the card on arrival for any office visit and reminding the physician of the procedures needed during the visit. On the basis of our results a further study is needed to evaluate the impact of a computergenerated reminder system in primary care practices that would continually use passive (physician) reminders for preventive procedures, personalized letters being generated annually, on a patient's birthday, if recommended procedures were overdue and telephone reminders being used only for patients who did not respond to the letter.
Conclusions Computer-generated reminder systems significantly improve the delivery of preventive services in family practice. Active (telephone and letter) reminders yield significantly higher rates of delivery than passive (physician) reminders. The more complex the procedure the less likely the physician will provide the service during a routine visit. There was no significant difference between the two active systems in the likelihood that the procedure would be completed. This study was funded by the Department of National Health and Welfare (grant 6606-2374), by the Ontario Ministry of Health through the Health Care Research Unit and by a Career Health Scientist Award to Dr. McDowell.
References 1. Canadian Task Force on the Periodic Health Examination: The periodic health examination. Can Med Assoc J 1979; 121: 1193-1254 2. Battista R, Lawrence RS (eds): Implementing preventive services. Am J Prev Med 1988; 4 (suppl): 1-194 3. Goldbloom RB, Lawrence RS (eds): Preventing Disease. Beyond the Rhetoric, Springer, New York, 1990 4. Weschsler H, Levine S, Idelson RK: The physician's role in
continued on page 814 LE lerOCTOBRE 1991
Use of reminders continued from page 812 health promotion
-
a survey of primary care practitioners. N
EnglJ.Ifed 1983: 308: 97-100
ACTI M
acetaminophen oral suspensions
* * * *
-.4;I .I .1
I
T 5. Battista R.N. Palmer CS. Marchand BM et al: Patterns of preventive practice in New Brunswick. Can MVed 4ssoc, J 1985: 132: 1013 1015 6. McDowell 1. Newell C, Rosser W: Comparison of three methods of recalling patients for influenza vaccination. Can .lied.Assoc J 1986: 135: 991-997 7. Anderson C. Martin H: Effectiveness of a patient's recall system on immunization rates for influenza. J Earn Pract 1979: 9: 727-730 8. Frank JW. McMurray L. Henderson L: Influenza vaccination in the elderly. can .Mfd Alssoc J 1986: 135: 989-991 9. Gerace TM. Sangster JF: Influenza vaccination: a comparison of two outreach strategies. Earnm .Med 1988: 20: 43 -45 10. Evans CE. Haynes B3 Birkett N et al: Does a mailed continuing education program improve physician performance? J.IMA-1 1985: 255: 501-503 11. Bass MJ: Organizing the office for effective detection and management of hypertension. Can Earn Phy'sician 1985: 31: 35 1-354 12. Herbert CP. Moore D: A study of health maintenance protocols in family practice. Ibid: 47-54 13. Frame PS: Periodic health screening in a rural family practice. J E-arn Pratt 1979: 9: 57- 64 14. Brownbridge G. Evans A. Fitter M et al: An interactive computerized protocol for the management of hypertension: effects on the general practitioner's clinical behaviour. J R (Coll Gen Prac t 1986: 36: 198-202 15. MacDonald CJ. Hui SL. Smith DM et al: Reminders to physicians from an introspective computer medical record. ,-lRAnn Intern MI.ed 1984: 100: 130-138 16. Chuz FZ. Dau RG: Smoking recognition by family physicians. l J ar PBract 1981: 12: 657-660 17. Fleming DM. Lawrence MSTA: An evaluation of recorded information about preventativse measures in 38 practices. J R Coll Gen Pract 1981: 31: 615-620 18. Standing P. Mercer S: Quinquennial cervical smears: even' woman's right and everv general practitioner's responsibilit. BAIIJ 1984: 289: 883-886 19. Hodes C: Cervical screening - refusal in general practice. J R ("oil Gen PBract 1972: 22: 172-175 20. Spenser JT: A survey of cervical screening in general practice. Practitioner 1967. 198: 274-280 2 1. Kleinman JC. Kopstein A: Who is being screened for cervical cancer? .4tin J Pub/lic Iealtet 1 98 1; 7 1: 73 -76 22. Fidler HK. Boyes DA, Worth AJ: Cervical cancer detection in British Columbia. BrJ Obstet Gvnaecol 1968: 75: 392-404 23. ICTRF Coordinating Comnmiittee on Cervical Screening: Organization of a program for cervical cancer screening. B.IJ 1984: 289: 894-895 24. Rosser WNW. Fluker G: Softw!are for family practice - a decade of development. (an 1-am BIPhn sic lan 1984: 30: 25672571 5. Elmslie T. Rosser WW: Computerization of familyr practice. Can I1ed .ssoc]J 1986: 134: 221 -224 " 6. McDowell 1. Newell C. Rosser WW: A randomized trial of computerized reminders for blood pressure screening in primary care. 1Ied C(arc 1989: 27: 297-305 27. Idem: Computerized reminders to encourage cervical screening in family practice. J I-am Pract 1989: 28: 420-424 28. Idem: A follow-up studx of patients advised to obtain an influenza immunization. Earn lIed 1990: 22: 303-306 29. A glossary for primary care. Report of the North American Primarv Care Research Group (NAPCRG) Committee on Standard Terminology. .I Ear Pract 1977: 5: 633-638 30. Fleiss J L: Statistical Methods fimr Rates and Proportionl.s. Wiley. New York. 1981
LE I er OCTOBRE 1991
Use of reminders for preventive procedures in family medicine Walter W. Rosser,* MD; Ian McDowell,t PhD; Claire Newell,t MA Objective: To compare the effectiveness of three computerized reminder systems in the delivery of five preventive procedures in family practice. Design: Prospective, randomized, controlled study. Setting: Ottawa Civic Hospital Family Medicine Centre. Participants: Of 8502 patients 15 years of age or more who were not in a hospital or institution 5883 were randomly assigned, by family, to a control group, a physician reminder group (passive) or a telephone or letter reminder group (active). The remaining 2619 patients were not included in the randomized portion of the study but were monitored. Intervention: During 1 year the patients in the active reminder groups received a telephone call or letter reminding them of any overdue preventive procedures; for those in the passive reminder group the physician was reminded at an office visit to provide any overdue service. Outcome measure: Rates of completion of the preventive procedures required. Main results: All three reminder systems significantly improved the delivery of preventive services (p < 0.001). The procedure completion rates were 42.0% in the letter reminder group, 42.0% in the telephone reminder group, 33.7% in the physician reminder group and 14. 1% in the randomized control group. The use of a letter was more cost-effective than the telephone system, but the physician reminder system was the most cost-effective. Conclusion: Computerized reminder systems do improve the delivery of preventive services in family practice.
Objectif: Comparer l'efficacite de trois systemes informatises de rappel dans la prestation de cinq interventions preventives en medecine familiale. Conception: Etude prospective, aleatoire et contr6lee. Contexte: Centre de medecine familiale de l'H6pital Civic d'Ottawa. Participants: Parmi 8 502 patients de 15 ans ou plus non confines a un h6pital ou a un etablissement, 5 883 ont ete affectes au hasard, selon la famille, a un groupe de contr6le, a un groupe de rappel par les medecins (passif) ou a un groupe de rappel par telephone ou par courrier (actif). Les 2 619 autres patients n'ont pas ete inclus a la partie aleatoire de l'etude, mais ont ete suivis. Intervention: Pendant 1 an, les patients des groupes de rappel actif ont recu un appel telephonique ou une lettre leur rappelant toute mesure preventive echue. Dans le cas des membres du groupe de rappel passif, on a rappele aux medecins d'effectuer toute intervention echue a l'occasion d'une consultation. Mesure des resultats : Taux d'execution des mesures de prevention requises.
From *the Department of Family and Community Medicine, University of Toronto, Toronto, Ont., and tthe Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Ont.
Reprint requests to: Dr. Walter W. Rosser, Department ofFamily and Community Medicine, University of Toronto, 444 Yonge St., Toronto, ON M5B 2H4 -
For prescribing information see page 875
CAN MED ASSOC J
1991; 145 (7)
807
Principaux resultats: Les trois systemes de rappel ont ameliore considerablement la prestation de mesures preventives (p < 0,001). Les taux d'intervention se sont etablis a 42,0 % dans le cas du groupe de rappel par courrier, 42,0 % dans celui du groupe de rappel par telephone, 33,7 % dans celui du groupe de rappel aux medecins et 14,1 % dans celui du groupe de contr6le aleatoire. Le courrier s'est revele plus rentable que le telephone, mais le systeme de rappel aux medecins a et le plus rentable. Conclusion: Le systeme de rappel informatise ameliore la prestation de mesures preventives en medecine familiale.
S ince 1979 the Canadian Task Force on the Periodic Health Examination has identified specific preventive procedures that meet criteria for effectiveness in family practice.' Work of the US Preventive Services Task Force, appointed in 1985, further supports the concept of critically assessing preventive screening procedures. In 1987 a joint meeting of the Canadian and US task forces concluded that despite their work large numbers of Canadians and Americans were not receiving appropriate preventive screening.2 A joint publication from the two groups gives a comprehensive set of guidelines for primary prevention in family practice.3 Physicians believe that most of their patients receive appropriate preventive screening in their practices.4'5 However, studies have reported that actual levels of such screening are low in practices and the community. Four studies of practice populations revealed that only 12% to 19% of patients over 65 years of age had been vaccinated against influenza.6-9 Surveys of hypertension screening in practice populations or the community showed that 50% to 90% of patients receive appropriate blood pressure checks. '10' In two practices the smoking status of patients was recorded in only 18% and 23% of the medical records.16"17 Less than 30% of the women in six studies received appropriate screening for cervical cancer.'8-23 Passive or active reminders may be used to improve the rates of delivery of preventive services. Passive reminders occur during regular visits and alert the physician or nurse that preventive procedures are overdue. With active reminders the patients are contacted by telephone or mail to encourage them to come in for overdue procedures. One evaluation of passive reminder systems reported significant improvements in the rates of delivery of preventive procedures when the reminders remain in place for a number of years. A passive reminder system for the physicians in more than 20 family practices in western Ontario resulted in 87% of the patients having their blood pressure checked once in 5 years." However, those who seldom visit the practice, often including those at high risk, may not be reached. Active reminder systems have been found to improve the rate of influenza vaccination in the elderly from 18% to 53%.6-9 Active reminders, in808
CAN MED ASSOC J 1991; 145 (7)
cluding home visits, have been used with some success to improve rates of delivery of preventive services among those not responding to a reminder to undergo a Papanicolaou test.20-23 Several surveys of patients with cancer of the cervix have concluded that women at highest risk are the least likely to have been screened. 1-23 The growing use of computers in primary care2425 prompted us to evaluate the costeffectiveness of both passive and active reminder systems. To our knowledge no one has compared the two systems for more than one preventive procedure in a randomized trial. The purpose of this paper was to analyse the effect of age and sex variables and practice patterns on the effectiveness of reminder systems and to determine whether a computerized reminder system improves the delivery of preventive services. We compared a passive physician reminder with active telephone and letter reminders to measure the improvement in the number of preventive services delivered for influenza vaccination, blood pressure measurement, determination of smoking status, Papanicolaou testing and tetanus vaccination, as recommended by the Canadian Task Force on the Periodic Health Examination. Papers on three of the procedures have been published elsewhere.26 28 Three hypotheses were tested: (a) active (telephone or letter) reminders are more effective than passive (physician) reminders in the delivery of preventive services, (b) the rate of delivery of preventive procedures decreases with increasing complexity (number of steps required) to administer the procedure and (c) telephone reminders (oral) are more effective than letter reminders (written). The number of steps required to complete a procedure determined our rating of complexity. Smoking assessment required the answer to "Do you smoke?" and, if Yes, "How many cigarettes per day?" Blood pressure readings were more complex, requiring access to the instrument and time to take the measurement. Influenza or tetanus vaccination required a series of questions to determine need, an explanation of the risks and retrieval of the appropriate vaccine; for the patient there is the inconvenience and discomfort of having an injection. Papanicolaou testing was the most complex, because the procedure is uncomfortable, requires the most time and may be done only when the woman is not menstruating. LE ler OCTOBRE 1991
..M
Methods Since 1976 all patients attending the Ottawa Civic Hospital Family Medicine Centre have been registered on a computer database.24,25 At the time of our study the practice was organized into six teams, each comprising a staff physician, a nurse and three to five residents and providing primary care to approximately 1200 patients. For regular office visits, by appointment, the patients always visited the same team. During off hours or in emergencies patients could visit any of the physicians or residents who were on call. In 1984 steps were taken to determine the active practice population. Letters were sent to 3722 patients who had not attended the practice in the 18 months before the start of the study to determine their status in the practice. Nonresponders or those stating that they no longer belonged to the practice were excluded. We included the remaining 8502 patients 15 years of age or more who were not in a hospital or an institution. Two of the six teams (representing 2619 patients) did not participate in the randomized component of the study; however, these patients were monitored. With the use of a standard randomization computer program the remaining 5883 patients were assigned, by family, to the control group (1056 families, 1403 people), the physician reminder group (1122 families, 1471 people), the telephone reminder group (1104 families, 1468 people) or the letter reminder group (1168 families, 1541 people). Patients were allocated by family to avoid the contamination that would occur if individuals in a family were assigned to a different team.
Five screening procedures involving different and sex groups and different complexities were included. Procedures required annually were influenza vaccination of those over 65 years of age, blood pressure measurement for those 18 years or more, assessment of smoking status for patients 15 years or more and Papanicolaou testing for all sexually active women 18 to 35 years old. The administration of tetanus toxoid is recommended once every 10 years for everyone over 18 years of age. Since influenza vaccination is seasonal and since we wished to do our follow-up 1 year later the influenza portion of the study was conducted between October 1984 and January 1985, the follow-up occurring during the same months in 1985-86. The rest of the study was performed between Apr. 1, 1985, and Mar. 31, 1986. To establish a starting point before Oct. 1, 1984, and Apr. 1, 1985, the computer scanned all of the patients records to determine whether the procedures had been carried out within the required time. Of the 8502 patients 38 required no procedure and
age
OCTOBER. 1, 1991
thus were excluded from further analysis. The age and sex groups corresponded to those recommended by the North American Primary Care Research Group.29 For patients assigned to the randomized control group no action was taken to remind the physicians or the patients that a procedure was overdue. The families in the two nonparticipating groups were used as a reference point to assess the extent of contamination caused by increased preventive activity in the control group. For families assigned to the physician reminder group a computer-generated reminder was included on the routinely printed encounter form before any visit to the office to remind the physician of outstanding preventive procedures. To validate the reliability of recording preventive actions on the computer 30 consultations with patients known to require one or more procedure were directly monitored by another physician or a research assistant through one-way windows without the knowledge of the study physician. The monitoring determined whether the physician had dealt with the required procedure during the visit and verified that services provided were accurately recorded on the computer record. This validation applied to the physician reminder group only; an internal audit was done in the two active reminder groups. Families assigned to the two active reminder groups were randomly assigned to 24 equal-size groups for ease of providing reminders and evenly distributing the demand for preventive services. At the beginning of each 2-week study period the computer printed a list of the names and telephone numbers of patients with overdue screening procedures. For the telephone reminders the practice nurse attempted to contact the family, making a maximum of five calls during working hours, and completed an action form for each listed patient. Once contact was made the nurse advised the patient about the indicated procedures and then attempted to arrange for them to be performed. The person answering the telephone was asked to relay the message to other family members. Families assigned to the letter reminder group received a computer-generated letter, signed by their physician and nurse, describing the procedures that were overdue for each member of the family and the importance of having them performed. After 21 days a second reminder was sent out to nonrespondents. In all three reminder groups the patients were asked if they had undergone the procedure elsewhere in the required timeframe and if this could be verified in writing. If Yes, the procedure was recorded as being completed. The time required by the physicians to ask about the procedures and the time required by the clerical staff to run the computer to CAN MED ASSOC J 1991; 145 (7)
809
generate letters and issue reminders were recorded and verified. Nurses kept a record of the number of calls made and timed each call with a stopwatch. Outcome was expressed as the proportion of patients in each study group for whom the recommended procedure was performed, as indicated in the computer database.
The 30 visits monitored and validated by an observer revealed that 15% of the completed procedures that the physician was reminded to perform were not recorded. Of the 1471 patients assigned to the physician reminder group only 766 (52.1%) visited the practice during the trial year. Of the families assigned to the telephone reminder group 1104 required a reminder for one or more procedures. Sixty-seven were not contacted because they did not have a telephone, were deaf or did not understand English or French. Of the remaining families 684 (66.0%) were contacted after 2.1 calls on average, each requiring 2 minutes and 48 seconds. Of those not contacted 235 (56.0%) had obsolete telephone numbers, and 185 (44.0%) could not be contacted after five attempts. Letters were issued to the 1442 people who needed one or more procedures updated. Although 164 (14.0%) of the letters were returned as undeliverable we assumed that the other 1004 (86.0%) were received. A second letter had to be sent to 653 (65.0%) of the families. Table 1 lists the proportion of required procedures that were performed during the study period. For those in the physician reminder group who visited the physician 60.4% of the required procedures were completed. However, this represented only 1471 (33.3%) of the 4413 procedures required by those in that group. The other two reminder systems were equally effective (Table 2); they were associated with significantly more completed procedures (42.0%) than the physician reminder system was (p < 0.05). The older the patient the more likely he or she was to visit the physician during the year. This contributed to the more than twofold increase in the completion rates for passive reminders among men
Statistical analysis The significance of the differences in proportions was determined with the use of formula 9.4, as described by Fleiss.30 If the difference was found to be significant further analysis, with the use of Fleiss's formulas 9.10 and 9.11, identified the groups that contributed to that difference.
Results To assess the adequacy of the randomization we compared the study groups in terms of family size, age and sex. There was no significant difference in sex (X2 = 3.3, 4 degrees of freedom [dfJ). Family size differed significantly between the groups, but after the patients in the nonparticipating practices (which tended to have larger families) were excluded the association disappeared. An analysis of variance of age showed no significant difference between the four randomized groups. Thus, we were confident that any differences in the study outcomes were unlikely to have resulted from differences in the group characteristics. The number of procedures required in each group at the trial start was compared with the number completed by the end of the study. Because of the Papanicolaou testing the average number of procedures required for women was higher than that for men (3.04 v. 2.61).
Table 1: Proportion of procedures required at start of trai that were performed during the study period. by patient sex, age and study group
Group: Sex. age. yr Male 15-34 35-64 65 All Female
15--34 35-64 65 All
of procedures performed
Nonrandomized
Randomized
Physicarn
Letter
Telephone
control
control
re minder
re rnnder.
reminder
664) 20.5 34.7 44.7 30.3 nr ----- 807)
(rin. 660)
in
(n
1143) 7.3 10.4 1i5.7
9.9 1476)
(n
9.7 11.2
11.5 10.6
622)
(n
8.3 14.9 13.7 12.3 (on - 781) 13.6 14.5 10.7
35
3i
(n --
31.3*`
37
49.4---
43
52.
*
43.0* fri
--
881;
623) /< :t
43 0741
(Il
845)
45.1
38.4
35.8*. 45.8* . 4a.1*
33.7
42.0*
42
26.7` 38.8&
39
9t
OM..
*Significantly greater than proportion irn randomized control group (p. n.O1. tSignificantly greater than proportion in physician reminder group (p 0.051 Significantly different from proportion in letter reminder group (p O 05)
810
CAN MED ASSOC J 1991; 145 (7)
LE 1 er OCTOBRE 1991
65 years, as compared with the rates among aged 15 to 34 years (Table 1). The telephone reminder was significantly more effective than the letter reminder among men 15 to 44 years (p < 0.05), whereas the letter reminder was significantly more effective than the telephone reminder among older men (p < 0.05). Women had a higher response rate to passive reminders than men because they visited their physicians more often. Among women under 64 years the effectiveness of the letter reminder was similar to that of the telephone reminder, but among women 65 years or more the letter was significantly more effective than the telephone reminder (p < 0.05). The first hypothesis, that either active reminder would be more effective than the passive reminder, was supported in all age and sex groups. When patients did visit the physician 60.4% of the required procedures were completed, as compared with 42.0% of the procedures in the two active reminder groups; this demonstrated the importance of the visit rate for the physician reminder group. The hypothesis that completion rates decrease as a procedure's complexity increases was not supported by our results (Table 2). The complexity of the procedure was unrelated to completion rates in the randomized control group. The hypothesis was supported in the physician reminder group by a gradient in response from the least to the most complex procedure. It was supported to a lesser extent in the two active reminder groups, in which the gradient was less clear. In the cost analysis we assumed that the computer system costs were the same for all of the interventions. A detailed cost breakdown for each of the five preventive procedures has been reported elsewhere.26-28 The physician reminder was the most cost-effective method of improving preventive services, because the time required to perform the procedure once the patient attended the clinic was minimal. Given the clinical and mailing costs the letter reminder was the next most cost-effective method. Cost-effectiveness was calculated by determining the cost of each procedure completed in
over men
excess of the number completed in the control group. The telephone reminder was the least cost-effective, mostly because the cost of having the nursing staff do the telephoning to obtain the extra procedures was high. The hourly wage of the person telephoning the families played a. significant role in the estimates of cost-effectiveness. We did not assess whether there would be any difference in outcome between a clerical person and a nurse delivering the telephone message.
Discussion Despite our preliminary efforts to determine the eligible practice population 17% of the people contacted in the telephone reminder group stated that they were no longer part of the practice. In all, 14% of the letters were not deliverable, and 3% of the patients who received a letter stated that they were no longer part of the practice. The overestimate by about 18% of the denominator may have accounted in part for the low visit rate (52.1%). The highest turnover rate occurred in the lowest age group, which may account in part for the weakness of the intervention in this group. Informal reports from Canadian practices suggest an annual loss rate of 9% to 12% in urban family practices, and since our study occurred over 20 months the rate of attrition is reasonable. The effect this loss of patients had in each group was to diminish the completion rate, which was lower than that in other studies.813-161921 However, we included only people who had not already voluntarily obtained preventive services. We also provided only a single oral reminder and two written reminders, whereas Bass," for example, showed a cumulative effect of repeated reminders over 5 years. Improving the level of preventive services in a practice population requires overcoming a number of obstacles. At the beginning of our trial the nurses and the physicians expressed concern that the patients might be irritated by receiving letters or telephone calls. Most of the patients, however, were pleased that their physician was concerned enough to
Table 2: Distribution of required procedures that were performed, by study group
Nonrandomized Procedure
Administer influenza vaccine Measure blood pressure Assess smoking status
Obtain Papanicolaou smear Administer tetanus vaccine
OCTOBER 1, 1991
control
Group; % of procedures performed Letter Randomized Physician reminder reminder control
Telephone reminder
3.8 18.6 9.5 11.2
9.8 21.1 11.9 13.7
22.9 30.7 37.9 16.5
35.2 40.5 49.1 29.7
37.0 37.2 55.8 30.0
2.3
3.2
22.8
30.6
24.0
CAN MED ASSOC J 1991; 145 (7)
811
contact them about overdue preventive procedures. Middle-aged people were the most responsive to the reminders. If we wish to realize maximum benefit from early detection in terms of life-years saved, people less than 45 years of age remain the most important target group. The lack of response among those more than 75 years may be secondary to both a declining interest in prevention as well as an increasing infirmity resulting in more frequent house calls and less frequent office visits. A number of approaches (e.g., media campaigns, population or community clinics, and home visits by public health nurses) to improve compliance with preventive procedures among those under 40 years should be evaluated. Workplace clinics may also be a more effective approach to achieving higher levels of preventive screening among people less than 45 years provided that appropriate linkage with family physicians is maintained for follow-up purposes. The complexity of the procedures was found to be a less important variable than we had expected. The more complex the procedure the less likely the physician was to complete the procedure during a regularly scheduled office visit. However, when the patient was recalled and prepared for the required intervention complexity was not a barrier. The recommendation that any preventive procedure is required annually reduces the population exposed to the physician, as compared with that for a procedure required every 2 or 3 years. This fact should be carefully considered when guidelines for frequency of procedures are being developed. The finding that telephone reminders were more effective than letter reminders in the lower age groups and the reverse in the older groups was unexpected. A number of respondents over 65 years of age who received a letter telephoned the nurse about how they should respond to it; this may have influenced the results. The preventive procedure completion rate in the physician reminder group indicates that this approach is the most powerful. Most studies have found that 70% of patients in a practice population visit their physician annually, the proportion rising to between 75% and 80% for visits once every 2 years. Although the cost analysis was simplistic we believe that with the use of the physician reminder system a significant improvement in preventive services can be achieved for less than $5 per extra procedure completed. Our analysis of the cost of the telephone reminder system was influenced by the use of the practice nurses, who were paid at a relatively high hospital rate. The use of a clerk or a part-time student at minimum wage could substantially reduce the cost of this approach, although an assessment of the response to telephone calls from a person unknown to the patient would be required. 812
CAN MED ASSOC J 1991; 145 (7)
We postulated that patients become dependent on being reminded after they have responded to an active reminder. Follow-up 1 year later revealed that the improvement in influenza vaccination rates achieved by reminders was not sustained. Some did not show up for vaccination 1 year after the reminder apparently because they were waiting for a reminder.28 One approach to reducing patient dependence on reminder systems would be to empower patients to assume responsibility for their preventive health. This could be achieved through the recording of the status of preventive procedures on a patient-held "smart card"; the patient could then be asked to assess his or her preventive health status by reading the card on arrival for any office visit and reminding the physician of the procedures needed during the visit. On the basis of our results a further study is needed to evaluate the impact of a computergenerated reminder system in primary care practices that would continually use passive (physician) reminders for preventive procedures, personalized letters being generated annually, on a patient's birthday, if recommended procedures were overdue and telephone reminders being used only for patients who did not respond to the letter.
Conclusions Computer-generated reminder systems significantly improve the delivery of preventive services in family practice. Active (telephone and letter) reminders yield significantly higher rates of delivery than passive (physician) reminders. The more complex the procedure the less likely the physician will provide the service during a routine visit. There was no significant difference between the two active systems in the likelihood that the procedure would be completed. This study was funded by the Department of National Health and Welfare (grant 6606-2374), by the Ontario Ministry of Health through the Health Care Research Unit and by a Career Health Scientist Award to Dr. McDowell.
References 1. Canadian Task Force on the Periodic Health Examination: The periodic health examination. Can Med Assoc J 1979; 121: 1193-1254 2. Battista R, Lawrence RS (eds): Implementing preventive services. Am J Prev Med 1988; 4 (suppl): 1-194 3. Goldbloom RB, Lawrence RS (eds): Preventing Disease. Beyond the Rhetoric, Springer, New York, 1990 4. Weschsler H, Levine S, Idelson RK: The physician's role in
continued on page 814 LE lerOCTOBRE 1991
Use of reminders continued from page 812 health promotion
-
a survey of primary care practitioners. N
EnglJ.Ifed 1983: 308: 97-100
ACTI M
acetaminophen oral suspensions
* * * *
-.4;I .I .1
I
T 5. Battista R.N. Palmer CS. Marchand BM et al: Patterns of preventive practice in New Brunswick. Can MVed 4ssoc, J 1985: 132: 1013 1015 6. McDowell 1. Newell C, Rosser W: Comparison of three methods of recalling patients for influenza vaccination. Can .lied.Assoc J 1986: 135: 991-997 7. Anderson C. Martin H: Effectiveness of a patient's recall system on immunization rates for influenza. J Earn Pract 1979: 9: 727-730 8. Frank JW. McMurray L. Henderson L: Influenza vaccination in the elderly. can .Mfd Alssoc J 1986: 135: 989-991 9. Gerace TM. Sangster JF: Influenza vaccination: a comparison of two outreach strategies. Earnm .Med 1988: 20: 43 -45 10. Evans CE. Haynes B3 Birkett N et al: Does a mailed continuing education program improve physician performance? J.IMA-1 1985: 255: 501-503 11. Bass MJ: Organizing the office for effective detection and management of hypertension. Can Earn Phy'sician 1985: 31: 35 1-354 12. Herbert CP. Moore D: A study of health maintenance protocols in family practice. Ibid: 47-54 13. Frame PS: Periodic health screening in a rural family practice. J E-arn Pratt 1979: 9: 57- 64 14. Brownbridge G. Evans A. Fitter M et al: An interactive computerized protocol for the management of hypertension: effects on the general practitioner's clinical behaviour. J R (Coll Gen Prac t 1986: 36: 198-202 15. MacDonald CJ. Hui SL. Smith DM et al: Reminders to physicians from an introspective computer medical record. ,-lRAnn Intern MI.ed 1984: 100: 130-138 16. Chuz FZ. Dau RG: Smoking recognition by family physicians. l J ar PBract 1981: 12: 657-660 17. Fleming DM. Lawrence MSTA: An evaluation of recorded information about preventativse measures in 38 practices. J R Coll Gen Pract 1981: 31: 615-620 18. Standing P. Mercer S: Quinquennial cervical smears: even' woman's right and everv general practitioner's responsibilit. BAIIJ 1984: 289: 883-886 19. Hodes C: Cervical screening - refusal in general practice. J R ("oil Gen PBract 1972: 22: 172-175 20. Spenser JT: A survey of cervical screening in general practice. Practitioner 1967. 198: 274-280 2 1. Kleinman JC. Kopstein A: Who is being screened for cervical cancer? .4tin J Pub/lic Iealtet 1 98 1; 7 1: 73 -76 22. Fidler HK. Boyes DA, Worth AJ: Cervical cancer detection in British Columbia. BrJ Obstet Gvnaecol 1968: 75: 392-404 23. ICTRF Coordinating Comnmiittee on Cervical Screening: Organization of a program for cervical cancer screening. B.IJ 1984: 289: 894-895 24. Rosser WNW. Fluker G: Softw!are for family practice - a decade of development. (an 1-am BIPhn sic lan 1984: 30: 25672571 5. Elmslie T. Rosser WW: Computerization of familyr practice. Can I1ed .ssoc]J 1986: 134: 221 -224 " 6. McDowell 1. Newell C. Rosser WW: A randomized trial of computerized reminders for blood pressure screening in primary care. 1Ied C(arc 1989: 27: 297-305 27. Idem: Computerized reminders to encourage cervical screening in family practice. J I-am Pract 1989: 28: 420-424 28. Idem: A follow-up studx of patients advised to obtain an influenza immunization. Earn lIed 1990: 22: 303-306 29. A glossary for primary care. Report of the North American Primarv Care Research Group (NAPCRG) Committee on Standard Terminology. .I Ear Pract 1977: 5: 633-638 30. Fleiss J L: Statistical Methods fimr Rates and Proportionl.s. Wiley. New York. 1981
LE I er OCTOBRE 1991
