STATISTICS IN MEDICINE, VOL. 10,541-557 (1991)

PRACTICE DATABASES AND THEIR USES IN CLINICAL RESEARCH WILLIAM M. TIERNEY AND CLEMENT J. McDONALD Department of Medicine, Indiana University School of Medicine. the Richard L . Roudebush Veterans Administration Hospital. and the Computer Science Research Section. Regenstrief Institute for Health Care, Indianapolis, Indiana 46202, U.S.A.

SUMMARY

A few large clinical information databases have been established within larger medical information systems. Although they are smaller than claims databases, these clinical databases offer several advantages: accurate and timely data, rich clinical detail, and continuous parameters (for example, vital signs and laboratory results). However, the nature of the data vary considerably, which affects the kinds of secondary analyses that can be performed. These databases have been used to investigate clinical epidemiology, risk assessment, post-marketing surveillance of drugs, practice variation, resource use, quality assurance, and decision analysis. In addition, practice databases can be used to identify subjects for prospective studies. Further methodologic developments are necessary to deal with the prevalent problems of missing data and various forms of bias if such databases are to grow and contribute valuable clinical information.

INTRODUCTION Despite modest beginnings, several large practice databases now exist. Although the data were collected during the delivery of routine medical care, such databases often serve two purposes: they can facilitate health care delivery, and they can be used in research. Practice databases may serve the general medical needs of a patient population or they may be organ system specific. The largest and most extensively studied general medical computer information systems with comprehensive databases are COSTAR,’ TMR,2 HELP,3 STOR: the computing system at Beth Israel Hospital,’ and the Regenstrief Medical Record System.6 Speciality systems include the Duke Cardiovascular Data Bank’ and the American Rheumatism Association Medical Information System (ARAMIS).8These eight systems contain data generated during the routine delivery of care. All but ARAMIS serve the real-time data needs of clinicians, but to varying degrees. Practice databases differ from other medical databases such as claims or investigational databases in the following ways: 1. They are compiled by health care institutions, rather than outside agencies such as third party payors, quality assurance organizations, research programmes, or the government. 2. The data are generated during the delivery of care to patients. 3. The data usually come from various institutional sources: the clinical laboratory, the pharmacy, and medical records. Research accomplishments of practice databases have been reviewed previously.’ This paper updates these reviews and discusses the advantages of, and methodologic issues arising from, research involving practice databases. 0277-67 15/9 l/O40541-17$08.50 0 1991 by John Wiley & Sons, Ltd.

542

W. M. TIERNEY AND C. J. McDONALD Table I. Data that might be contained in a complete computerized clinical database

History: symptoms prior procedures functional status behavioural risk factors allergies demographics, living arrangements Physical examination: vital signs blood pressure pulse temperature height clinical findings (for example, rales, murmurs, skin lesions) Diagnoses: inpatient (hospital or long-term care facilities) admission discharge (multiple) outpatient primary care offices consulting offices emergency room Clinical activity (that is, visits): hospitalizations admission date diagnoses length of stay outpatient offices/clinics dates reason for visit type of visit scheduled new scheduled return unscheduled (urgent) emergency room dates reason level of care

Diagnostic tests: inpatient and outpatient clinical laboratory cytology/pathology imaging studies radiology nuclear medicine diagnostic cardiology neurology Therapeutics: drugs dates prescribers (multiple) source (in/outpatient) dosages sigs amounts dispensed prescription drugs and OTCs procedures (surgery) physical therapy prosthetics rehabilitation programmes nursing interventions dressings education devices (for example, catheters) others (for example, radiation therapy) Other data: death certificate data prospective outcome data collected from patients

THE CONTENTS OF PRACTICE DATABASES Existing computerized practice databases vary greatly in the information they contain.’ This variability is due to differences in availability of data and the clinical and research purposes that motivated their creation. A perfect, clinical database, would contain accurate information about all aspects of patient care as exemplified in Table I. However, collecting all of the data in Table I for each contact made with a medical care system by even a small number of patients is not currently possible. Each data type can be obtained only by a separate development effort, and some kinds of data are difficult to obtain under any circumstances. Moreover, the costs for obtaining and storing such data must be balanced against the benefits from their expected clinical and research uses. The most difficult data to capture accurately in a codified, retrievable form are from the clinical history, the physical examination, and the physicians’ impressions, assessments of risk and differential diagnoses. Yet the physician’s overall impression about a patient, his/her ‘gestalt’,

PRACTICE DATABASES USED IN CLINICAL RESEARCH

543

may be the most potent predictor of future events." Also important yet difficult to capture are some important outcomes of diseases and therapy, such as functional status, mental status, and quality of life. ADVANTAGES OF PRACTICE DATABASES By 'claims databases' we mean databases whose data is used primarily for billing and payment and include utilization data and diagnoses. The Medicaid Management Information System and the Medicare database are examples, as are various ad hoc databases at least partially comprising data from hospital case abstracts. Practice databases have several advantages over claims databases for conducting research. These advantages are: accurate timely data; rich clinical detail; and continuous parameters.

Accurate data Data in practice databases are reviewed frequently by physicians and other care-givers who know their patients and use the data in delivering care. Thus, errors are more likely to be identified than in claims databases where quality control can only be performed on a limited set of clinical variables on a subset of patients. For example, a physician is likely to know that a haemoglobin concentration that falls by 50 per cent is erroneous if it occurs in a patient with no active bleeding or evidence of haemolysis. For this reason tests with unexpected abnormal values are often repeated before action is taken." Health care providers are not likely to tolerate a high error rate for their clinical data.

Timely data Since the data in a practice database are primarily used for the day-to-day delivery of health care, they must be available soon after they are generated. This allows researchers and others to identify individual patients with selected clinical characteristics as they appear. Potential subjects for collecting additional data or for acute interventional studies can be found readily.

Richness of clinical detail Claims databases contain diagnostic information obtained from physicians or chart abstracters. This is often secondary - representing an interpretation of raw data within the patient record. Abstractors may not identify diabetes as a problem or not code it if another diagnosis is the primary focus of a patient encounter or yields greater reimbursement. In practice databases, patients with diabetes can be identified not only by recorded diagnoses but also by their blood glucose concentrations, glycosylated haemoglobin results, or medication use (insulin or oral hypoglycaemicdrugs). Similarly,patients with congestive heart failure can be identified according to the stored results of chest roentgenograms and/or echocardiograms. Relying on traditional diagnostic codes can be fraught with errors: the event must be severe enough to be noticed by the clinician who must then record it properly, and the person abstracting the data must identify the diagnosis correctly and code it appropriately. At each point this process can break down. Indeed, in one study, coding errors were common and were usually errors of omission." In addition to their richer clinical content, practice databases also have dates attached to each stored observation. This enables research and quality assurance efforts to link a procedure or a specifiedoutcome with other clinical results, procedures, drug use, etc. using retrospective cohort designs or case-control studies.

544

W. M. TIERNEY AND C. J. McDONALD

Continuous parameters Having the actual results of tests and clinical parameters (for example, weight, blood pressure) allows researchers to analyse actual values and differences rather than simple (and often arbitrary) categories.They can then study the effect of a therapeutic intervention on the change of a continuous variable, for example, haemoglobin, rather than a coder's binary interpretation, for example, anaemia. For example, by using serum creatinine results to assess the effect of a drug on renal function, researchers can identify subtle, subclinical effects. These are more likely to occur and less likely to be recognized than such disastrous complications as hospital admission for renal failure or instituting dialysis. Likewise, researchers can rely on the results of cardiac enzyme tests and electrocardiograms to confidently identify patients with acute myocardial infarctions rather than relying on the vagaries of coding procedures. Researchers using continuous variables can also give greater weight to greater degrees of abnormality. This allows full use of the information contained in a laboratory result. It is not surprising, then, that one study found continuous variables to be better predictors of hospital mortality than qualitative data.' RESEARCH USING PRACTICE DATABASES Although most practice databases were primarily established to help transmit and store information to augment clinical care, they have also been used for r e ~ e a r c hExamining .~ clinical experience is the oldest method of acquiring medical knowledge. The size of many existing databases and the power of epidemiologic and statistical methods now allow researchers to use large volumes of practice data to study a wide range of clinical problems. The data, and the patient populations that the systems serve, have also been used in more traditional clinical research projects.

Clinical epidemiology Practice ddtabases extend the oldest means by which physicians have learned about diseases and the effects of treatments: clinical observation. Computer-storage of practice information permits observing large numbers of patients and the quick retrieval and analyses of their data. In epidemiologic research the most useful databases combine information acquired during routine care with prospectively gathered data. The two most prominent examples are the Duke Cardiovascular Diseases Databank' and ARAMK8 Collecting data prospectively allows researchers to control for many of the common biases of observational studies. The Duke database serves as both a clinical and research tool. It supplies information to physicians caring for patients with cardiovascular diseases, primarily ischemic heart disease, while shedding light on the clinical course and evolution of care for patients with coronary artery disease. For example, the Duke researchers have reported the mortality rate and subsequent acute coronary events after emergent coronary angioplasty following acute myocardial infarction.I4 These researchers have also studied the clinical outcomes of patients admitted to rule out a myocardial infarction who had normal or non-specifically abnormal admitting electrocardiograms. ARAMIS has been used by researchers from many sites to store data on patients with rheumatologic diseases. As at Duke, follow-up data are regularly collected from registered patients and their providers. Studies of ARAMIS patients with rheumatoid arthritis have shed light on the clinical course of the disease, including the frequency of hospital admission, causes for morbidity, and variation in health care costs.'6- l9

''

PRACTICE DATABASES USED IN CLINICAL RESEARCH

545

In addition to the clinical course of diseases, practice databases can be used to study changes in therapy for selected problems. For example, the Duke researchers have documented the trend towards performing coronary artery bypass surgery on patients with compromised left ventricular function.20Duke data and data from other studies (such as the CASS investigations) have shown that, despite an increased operative mortality, patients with coronary artery disease and left ventricular failure do better with bypasses than with medical therapy alone.z1 Assessing patient risk Most clinical tests are ordered to search for abnormalities for which patients are perceived to be at increased risk. Moreover, much of the care delivered to patients is aimed at preventing some future event. Unfortunately, assessing risk -identifying patients who are likely to have a certain condition and apt to benefit from treatment - is difficult. Thus physicians’ testing often becomes excessive and indiscriminate.z2This is not a trivial problem. Diagnostic testing consumes 25 per cent of all outpatient and inpatient health care dollars.23 Items with small individual costs are most of this costz4 but often contribute little to patient If patients at increased risk for a particular abnormality could be identified, then testing could be targeted towards these higher risk patients. Costs could be avoided for patients at low risk, and prophylactic treatment could be instituted for the patient at the highest risk. Pryor and his colleagues used the Duke Cardiovascular Data Bank to generate risk profiles of patients with chest pain. These profiles have been able to accurately identify patients at risk for subsequent myocardial infarctions and death.26 Their models have performed well when compared with the predictions of academicz9and community 30 cardiologists and have been used to control other retrospective analyses for severity of illness. Researchers at Latter Day Saints Hospital used the HELP system to identify patients likely to have hospital-acquired infections3’ and have transmitted this information electronically to patients-treating physician^.^^ The ARAMIS database has been used to identify risk factors for death33and disability” in patients with rheumatoid arthritis. Regenstrief researchers used more than ten years of computer-stored data from a large academic primary care medicine practice to predict abnormalities on commonly ordered outpatient diagnostic tests.34These models were good predictors of these abnormalities and were stable when validated prospectively. Moreover, physicians ordered significantly fewer tests when they were shown their patients’ risk profiles.35 By identifying risk factors for certain adverse outcomes, these studies can suggest means to help avert these outcomes. For example, researchers using the TMR system at Duke have identified hypertension as a risk factor for progression of chronic renal in~ufficiency.~~ Levkoff and his colleagues used data from the Beth Israel Hospital information system to identify urinary tract infections and hypoalbuminaemia as independent risk factors associated with delirium in geriatric patient^.^' Regenstrief investigators found that type I1 diabetes mellitus, black race, elevated systolic blood pressure, age, heart failure, and liver failure were highly predictive of the development of renal insufficiency in patients with h y p e r t e n ~ i o nThese . ~ ~ data suggest controlling diabetes and lowering systolic blood pressure may ameliorate the development of renal insufficiency in hypertensive patients. Post-marketing drug surveillance Most drugs are released for general use after being tested in fewer than 3000 patients. Hence, premarketing studies are not likely to uncover uncommon adverse events. Furthermore, most premarketing trials exclude patients with any significant co-morbid conditions, those taking many (or any) other drugs, and the elderly. Patients who would be excluded from pre-marketing drug

546

W. M. TIERNEY AND C. J. McDONALD

studies are commonly found in primary care practices, and such patients are at risk for developing adverse drug reactions. Thus, surveillance for adverse reactions must occur after a drug is marketed, and most post-marketing surveillance relies upon reports generated by individual practising physicians. Unfortunately, this process can lead to the underreporting of adverse reactions. Although so far they are generally underutilized for post-marketing drug surveillance, practice databases can contribute to this effort because they do not rely upon their treating physicians to recognize and report adverse reactions. For the more commonly prescribed drugs, researchers can examine small changes in continuous indicators of organ function (for example, serum creatinine concentration as an indicator of renal function) in large numbers of patients. They can also compare effects of different drugs given for the same indication, controlling for co-morbid conditions and severity of illness, and they can screen for drug interactions. For example, Singh et al. used ARAMIS to study azathioprine, documenting its low incidence of adverse effects in patients with rheumatoid arthriti~.~’ ARAMIS researchers identified drugs that are associated with development of purpura and abdominal pain4’ and studied the association between non-steroidal anti-inflammatory drugs and ga~tropathy.~’ The investigators at Duke used their cardiovascular database to document that amiodarone therapy in patients with coronary artery disease did not reduce survival, when other prognostic factors were ~ontrolled.~’ The Regenstrief database has been used to identify primary care patients at risk for h y p ~ k a l a e m i aand ~ ~ cardiac arrhythmia^^^ in patients taking diuretics. They also documented that elderly patients and patients with coronary artery disease taking the non-steroidal antiinflammatory drug ibuprofen were at risk for developing renal d y s f ~ n c t i o n . ~ ~ Compared to claims databases, practice databases have fewer patients but more observations and more sensitive data on individual patients. Post-marketing surveillance using practice databases should exercise this strength by focusing on the adverse effects of commonly used drugs. Serious complications of less commonly used drugs and adverse events with a very low incidence should be evaluated by studying the larger claims databases. Despite methodologic questions such as selection bias, practice databases can and should be used to investigate questions about particular drugs and to identify for further study drugs with possible adverse effects. Thus, practice databases can perform an important role in monitoring drug therapy especially when there is no other adequate means for investigating adverse drug effects and for patients excluded from pre-marketing studies. Variation in physician practice Variation in clinical care delivery and in patient outcomes is an expanding focus of health services research. Large claims databases can identify differences over wide geographic areas and suggest possible causes for these differences, such as physician training and experience or availability of health care facilities. In contrast, practice databases can be used to measure differences between individual physicians. Moreover, because the data for individual patients is rich, practice databases can help quantify what portion of the variation is attributable to patient differences and what is due to differences in physician practice patterns. For example, Dittus and his co-workers used Regenstrief data to examine how soon physicians see outpatients in follow-up after visits to a large academic general medicine practice.46 They found a four-fold difference in return visit intervals between the physicians with the shortest follow-up intervals and those with the longest. This difference was not explained by the patients’ diagnoses or clinical status. Interestingly, they found that physicians with the shortest follow-up intervals also ordered significantly more diagnostic tests.

PRACTICE DATABASES USED IN CLINICAL RESEARCH

547

Studies of test ordering by Regenstrief researchers found a seven-fold difference between the physicians with the fewest tests ordered per patient visit and those with the most.47This difference remained after controlling for differences in patients’ clinical characteristics. ARAMIS researchers studied differences in hospitalization rates and costs incurred in treating patients with rheumatoid arthritis, adjusting for severity of illness.l6- I 9 They found, for example, that although therapy with gold compounds was effectivein relieving the symptoms of rheumatoid arthritis, it resulted in twice the number of physician visits and higher costs for laboratory m~nitoring.~~ These types of analyses help identify areas where costs may be saved by targeting low risk patients and high cost physicians. To date, practice databases have been underused in quantifying differences in physician practice patterns. Practice data also hold promise for prescriptive studies of clinical care, such as suggesting appropriate intervals for monitoring stable patients with common conditions and suggesting which diagnostic procedures and therapeutic manoeuvres, if any, are likely to benefit patients. Resource consumption and quality assurance

The United States has entered an era of fiscal concern and constraint in health care delivery. The broadest example of this is the prospective payment system for Medicare centred on Diagnosis Related Groups (DRGs). This system assumes that the risk for individual patients can be estimated by assigning diagnoses at hospital discharge. DRG hospital cost prediction is refined by adding data for a limited number of co-morbid conditions. Understandably, this approach has led to large variation among patients within particular DRGs. It has also produced ‘DRG drift’ where better paying diagnoses are selected as primary over lesser paying ones when patients have more than one active condition (which is often the case). For general health care planning and for identifying patients towards whom cost reducing interventions might be aimed, practice databases might be used to predict individual patients’ health care use. This might be done by assessing ‘harder’ indicators of diagnosis and of severity of illness. For example, the diagnosis of pneumonia can be assigned if there is an infiltrate on the chest roentgenogram. Severity of this illness can be measured, by the patient’s age, number of involved lobes, blood gas results, serum bicarbonate values, leukocyte counts, and by the results of blood cultures. In addition, pre-admission co-morbid conditions can be identified and taken into consideration. Studies comparing the predictive power of clinical data with DRGs and with other indices ) sparse to date but based on resource consumption (for example, that of Horn et ~ z l . 4 ~are encouraging. A group at the San Francisco Veterans Administration Hospital used a practice data system called GP-I to predict hospital costs as well as or better than DRGs.~’Researchers at the University of Michigan used laboratory data to predict inpatient costs. These indices outperformed DRGs for selected diagnoses.” Analyses of Beth Israel data showed that certain DRG co-morbidity indicators had no value in predicting This is an exciting and active area of research because practice data are more sensitive and specific for assigning diagnoses and estimating severity of illness, and because it is more difficult to manipulate these data to inflate reimbursement. Fiscal restraint may have untoward effects on the quality of care. Reductions in resource use may come at the expense of the patient, as Fitzgerald et al. demonstrated for elderly patients with fractured hipss3 Efforts to reduce costs will necessarily lead to studies of their effects on quality of care. Such studies will require accurate clinical data. Using practice databases, supplemented if necessary with outcome data from other sources (death certificate files, Medicare data, or patient

548

W. M. TIERNEY AND C. J. McDONALD

questionnaires), will reduce the time and cost of these studies. In addition to reviewing the quality of health care, practice databases may help improve it. The HELP system has been used to identify patients likely to have nosocomial infections 3 2 and to improve the appropriateness of inpatient antibiotic use.32’5 4 Federal support for research into health care costs, patient outcomes, and quality of care is growing rapidly. So is the technology for analysing these data and combining them with data from claims databases. This will help ‘close the loop’ on tracking patients’ resource consumption, at least for large subpopulations such as those insured by Medicaid and Medicare. Using practice databases to investigate physician and patient factors associated with costs and otucomes is apt to be a fertile area of research in the near future. Physician decision making Studies of how physicians make decisions have used the information stored in practice databases to describe and compare the consequences of physicians’ decisions. For descriptive analyses to be helpful, they must have accurate information on the prevalence of the condition(s) being studied and on the incidence of the different outcomes in treated and untreated patients. These estimates are often missing from the medical literature, and they seldom reflect a particular patient population of interest. Practice databases can help these efforts. For example, Safran and Phillips used data from the Beth Israel system to assess the cost-effectiveness of an intervention designed to lower hospital readmission^.^^ Not only can practice databases illuminate the consequences of physicians’ decisions, but the information can be used to affect health care delivery. Investigators using the STOR system showed that physicians using well-formatted data summaries could more accurately predict patient symptoms and laboratory results than when they used the paper chart.56 Fries showed that a computer flow sheet increased physicians’ ability to find necessary data and decreased the time needed to do so when compared to the paper chart.” Researchers at Regenstrief showed that when physicians were given paper summariess8 or CRT displays of past test results,59 they ordered fewer diagnostic tests. At a higher level of complexity, there are programs that can search a database and identify patients eligible for certain interventions. They can also point out laboratory abnormalities and identify potential drug-drug and drug-diagnosis interactions. Researchers using COSTAR showed that physicians managed syphilis better,60 managed hypertension more appropriately,61 and were able to track drug interactions better62 when they were reminded by the computer of patients needing attention. Computer-generated reminders from the Regenstrief system, given to physicians during visits of scheduled patients, doubled the amount of preventive care given63and performed better than monthly feedback reports.64 Investigators at Duke using TMR data summarized prescribing patterns for physicians. This resulted in more generic drug use.65 The HELP system helped improve physicians’ ordering of antibiotics for hospitalized patients.32*5 4 A number of smaller pharmacy programs identify patients taking drugs which have potentially serious interactions.66 Indeed, surveillance for uncommonly occurring clinical events or monitoring for potential drug interactions, activities that are boring and low yield and are performed poorly by physician^,^' may be one of the most reasonable uses for computerized practice data management systems. Collaborative clinical research Practice databases can be a resource for identifying and contacting subjects for clinical research protocols, such as studies investigating new drugs. The computer can identify potential patients

PRACTICE DATABASES USED IN CLINICAL RESEARCH

549

by extracting inclusion and exclusion criteria and retrieve preliminary or baseline descriptive data from patients’ computer-stored data. Researchers traditionally rely on individual physicians to identify potential subjects, a process often hampered by poor co-operation and poor communication between practising physicians and researchers. Moreover, several kinds of selection bias are intrinsic to this process. Using practice databases allows more comprehensive identification and recruitment of potential subjects and does not depend on the patient’s physician remembering a study and knowing its criteria. However, the treating physician should approve the enrolment of patients identified from a practice database. Epidemiologic studies of practice databases can support arguments that an intervention may be helpful by documenting the prevalence of a problem, its severity, and its effect on prognosis. Preliminary data can also aid in making sample size estimations. Finally, once a study has been initiated, using the practice database to monitor study outcomes, such as death, hospital admission, or costs, is less expensive than reviewing paper charts. METHODOLOGIC ISSUES Despite the advantages of practice databases and their use to date, important methodologic issues need to be resolved if the research scope involving them is to grow. These issues fall into three categories: data; subjects; and bias. Data issues

Except for the Duke Cardiovascular Data Bank and ARAMIS, which attempt to collect a set of data prospectively on all registered patients, practice databases usually record information about patients only when they contact the health care facilities served by the system. Data may thus be missing for two reasons. First, providers outside of the system served by a database may also be treating the patients. This may be due either to patients’ wishes or to changing eligibility criteria that force patients to use certain systems. Thus, one cannot tell whether patient information is absent because the patient is not visiting the system or because he/she is being treated elsewhere. Similar intermittent patient eligibility problems also occur with claims databases (for example, those that use Medicaid inf~rmation’~). Follow-up problems may be less where the population being treated has fewer options in choosing health care providers, such as in Veterans Administration hospitals, HMOs, and public hospitals. A second bias occurs because patients who are less ill may visit a health care facility less frequently. Thus, patients for whom there are more data will tend to be sicker than patients with sparser records. Moreover, relatively benign conditions that require frequent visits (for example, hypertension) may be falsely identified as ‘low risk’ or ‘protective’ since outpatients with such conditions will have a relatively favourable prognosis compared with outpatients with more serious, symptomatic illnesses. Certain types of data can also be missing for patients who do visit a health care system. This is most likely to occur for subjective data, such as diagnoses that must be recognized by the provider and properly recorded, than for objective data, such as test results and prescription information. Not only can data be too sparse, they can be too rich. There may be multiple measures of the parameter of interest, and one may have to decide which value to use: The first? The last? The mean? The worst? The researcher must understand which of these choices would most likely answer the question at hand. A related problem is defining significant changes in a parameter. For example, to examine the effect of a drug on renal function, how much change in creatinine or creatinine clearance is clinically important? How does one assess changes in disease status in patients with congestive heart failure or chronic lung disease?

5 50

W. M. TIERNEY A N D C. J. McDONALD

Defining a clinical condition may vary depending on which of a number of indicators (which have variable accuracy) are used. For example, renal function can be measured with a 24 hour urine collection (which is not done routinely), estimated from the blood urea nitrogen or serum creatinine concentration alone, or calculated using one of more than a half-dozen equations.73 Each method may identify different patients with renal insufficiency. Likewise, because of known differences in the sensitivity and specificity of tests for myocardial infarction (for example, electrocardiograms, cardiac enzyme^^^-^^), how one defines myocardial infarction will have a large effect on the results of any investigation. There are also issues related to the reliability of data in the practice databases. Data gathered from one site may not be as accurate as the same data from another site. For example, blood pressure readings from a hypertension clinic may be more accurate (and less often omitted) than from a primary care clinic which, in turn, may have more accurate and complete data than blood pressure readings from an orthopaedic clinic. Electrocardiograms from the coronary care unit may be of higher quality, and therefore more readable and reliable, than those done in the emergency room. Quantifying these differences and controlling for them is difficult, and researchers using practice databases must constantly make hard decisions about which data to include in their analyses. Data in claims databases suffer similarly from the problem of reproducibility, but the sources of these data are few and quality control and intervention may be easier to accomplish. Solutions

Measures can be taken to reduce the amount of missing data. Supplementary data from claims databases can be merged into patient records in practice databases. Although claims data currently exist only for selected patient populations (for example, Medicaid and Medicare patients), the opportunities for sharing data will expand as advances in software and reductions in hardware costs foster development of newer practice and claims databases. Specialized data are also available from other sources, such as hospital discharge case abstracting systems, state board of health death certificate systems, and the National Death Index. More complete patient followup may be easier to accomplish in the future as regional databases are established, or as data become easier to transfer from one institution or office to another. There are mechanisms that can correct, to some extent, problems with missing data. Objective data can often serve as a proxy for missing subjective impressions. For example, one can use the results of chest radiographs and echocardiograms to assign the diagnosis of congestive heart failure. Patients taking insulin or an oral hypoglycaemic agent can be reliably labelled as diabetic. There are also schemes that can estimate values for missing objective parameters so that important variables need not be excluded from analy~is.~' Problems with patient follow-up can be minimized by limiting studies to short term outcomes for patients with known activity within the health care system being studied. How to take advantage of the number of repeated measurements of many parameters is also an area of on-going development. Weighting schemes are being generated that will fully use the plethora of data generated during clinical care.79 For selected patient sub-populations, one can prospectively collect missing data critical to the question being asked. This is more expensive than either excluding cases with missing values or substituting an estimated result. However, it is the most complete and accurate means of assessing patient outcomes, it minimizes bias, and it has been used very successfully by researchers with the cardiovascular database at Duke and with ARAMIS. Establishing a minimum acceptable set of data to be gathered at baseline (that is, at the time of registration in the data system) and at follow-up would go a long way to improve generalizability

PRACTICE DATABASES USED IN CLINICAL RESEARCH

55 1

of results and exportability of predictive models. Currently, only a very small set of data, mostly demographic information, is routinely stored for patients in most active practice databases. In addition, standardizing definitions of diseases (for example, what level of blood pressure denotes hypertension, and how many readings are necessary), symptoms (what descriptors of chest pain are necessary), physical findings (what are rales and wheezes), and results of diagnostic tests (which abnormalities on the electrocardiogram should be reported) would give database researchers a common language. This would also help develop, research, and disseminate practice database research technology. With more experience using these data for different applications, researchers will be better able to agree on issues such as which of many repeated measures to include or how to define clinicaIIy significant changes in continuous parameters. This and an established minimum dataset will be necessary to realize the full potential of co-operative studies between different practices. Patient issues Practice databases tend to contain data for fewer patients than claims databases. Practice databases are limited to the institution(s) housing the systems. Thus the larger claims databases follow many times the number of patients as the largest practice database. Claims databases are potentially more sensitive to low frequency events by having more patients, but practice databases are more sensitive for clinical changes within individual patients. Currently, most practice databases, and virtually all that have been used for research, are located in academic medical centres. Yet both physicians and patients at these sites may be quite different from their counterparts in general practice. Physicians with data in practice databases tend to be residents and their teachers. The patients more often are indigent or were referred for specialized care. As a result, conclusions from descriptive research or interventions from practices with computerized medical record systems must be generalized cautiously. Solutions

Having fewer patients with data stored in practice databases may be offset to a large extent by the considerably greater number of different observations (both different parameters and repeated measures of individual parameters) and the more sensitive indicators of disease than claims databases. Thus, researchers using practice databases may be most successful by limiting their studies to common conditions and/or treatments that are frequently monitored. Generalization problems will be solved by exporting computer technology and research expertise from academia to general practice. Collaboration between academic and private practice physicians may be augmented by co-operating in managed care systems or sharing regional data management facilities. The economic and technologic underpinnings for such co-operation are currently evolving.

Bias issues When physicians select patients for diagnostic procedures and treatments, they introduce bias that can grossly affect studies of the natural history of disease and its treatment. This has generated controversy over whether epidemiological studies using practice databases can, in some instances, substitute for, or at least complement, classic clinical trials. Califf and his colleagues give several arguments for using stored clinical data for such studies:*'

552

W. M. TIERNEY AND C. J. McDONALD

1. Clinical trials are very expensive, especially for intensive procedures which must be performed on a large number of patients or over a long time. The effect of coronary artery bypass surgery on survival is an example. 2. Many therapies are ‘moving targets’, often changing significantly by the time prospective studies finish evaluating their effects. 3. Many patients (especially elderly patients and patients with significant co-morbid conditions) for whom a treatment or procedure would be used in practice are often excluded from clinical trials. 4. Clinical trials are usually performed at selected centres which demonstrate excellence in performing the procedure of interest. This affects the generalizability of many clinical trial results. 5. Clinical trials are usually studies of ‘efficacy’,designed to maximize the chance of observing a positive clinical result, whereas the clinician is interested in ‘effectiveness’ of a drug or procedure in usual clinical practice.

Byar has argued against substituting practice database analysis for clinical trials.” He argues that: 1. Bias in treatment assignment may not be overcome by controlling for covariates known to affect the treatment outcome. 2. Often rigorous definitions of disease and even of the treatment under study do not exist and frequently change with time. For example, studies of coronary artery disease therapy often divide patients into those treated with medicine and those treated with surgery. The definition of disease and treatment may not be stable between patients. For example, surgical procedures may differ depending upon anatomy, and individual patients may have relative contraindications for using certain medications. Therapy may change with time as surgical technique evolves and as new medications such as calcium channel blockers are developed, or as there are new indications for old medications, such as aspirin. 3. Data are often missing from practice databases, leading to difficulties in defining the disease or treatment of interest and in assessing outcomes. Moreover, patients with missing data are apt to differ from those without missing data. Often they are less ill or they seek additional health care at other sites. S o h tions

It would be unethical to study some clinical problems, such as adverse drug reactions, using clinical trials. Supplying a drug to a patient likely to have an adverse reaction is an example. Other important questions, such as how the Medicare DRG-based prospective payment system affects quality of care, may be impossible to study in a randomized controlled trial. Analysing observational data in practice databases may be the only way to study such problems. That is, imperfect data with known flaws may be better than no information at all. There are several ways to minimize the biases inherent to practice databases. First, prognostic models can be derived to control the analyses for severity of illness. Researchers using the Duke Cardiovascular Databank have employed this method to control for prognosis in several of epidemiologic studies of the natural history and treatments for ischemic heart d i s e a ~ e . In ~~.~~ fact, their results compared quite favourably with data from the prospective, randomized, controlled clinical trials.84

PRACTICE DATABASES USED IN CLINICAL RESEARCH

553

For bias encountered when determining patient outcomes, one can supplement data in practice databases with other data. Researchers can collect it prospectively (an expensive option) or extract it from other sources- the Medicare tapes, state boards of health (death information), or the National Death Index. FUTURE DEVELOPMENT O F PRACTICE DATABASES Computers pervade many aspects of medicine. For example, most laboratories use computers to print ward and chart reports; most pharmacies enter prescriptions into automated systems that print bottle labels and monitor inventory; and most business offices receive hospital information for billing purposes in electronic form. Development of computerized databases has lagged, impeded by: 1. The lack of standards for communication between hardware and software developed by different vendors. 2. Difficulties in assessing the costs and benefits of such systems. 3. The sticky methodologic issues discussed earlier - standard definitions and how to deal with missing values-for which solutions have not yet been accepted. A solution to the first problem, communication of data between systems, has been proposed. The American Society of Testing Materials has published a standard format for medical data t r a n ~ r n i s s i o n . ~Wide ~.~~ acceptance of this communications formatting standard will overcome the major barrier to establishing usable clinical databases in many settings. Assessing the cost of computerizing medical records is an interesting problem. Direct comparisons with paper charting systems are difficult. It is like comparing a telephone to a typewriter: the new equipment may not be more economical at performing the main chore of the system, but it offers many new possibilities, and the benefits are harder to measure and compare. The computer does much more than store and retrieve data. Its availability leads to opportunities that are difficult to measure or even enumerate. For example, one evaluation of COSTAR used in a primary care practice showed that, if purely medical record functions were studied, the system added $0437 to the cost of a patient visit.' However, if additional reporting and monitoring functions of COSTAR were included, the system saued $0.72 per visit. When Dambro and his colleagues assessed the cost of COSTAR after installing it in a family practice centre they considered only its recording of outpatient encounter^.'^ They estimated the additional personnel costs alone at $4.93 per visit and subsequently removed the system. The changing medical environment will foster development of computerized practice databases. Focusing on quality assurance has forced all health care organizations to retrieve and review data. Most have added staff to do so. It takes fewer people to assess quality using computer-stored practice data, and the data are more reliable than paper chart review^.^' Recent emphasis on the outcomes of procedures and other therapies has also increased the need for detailed, timely data. These new forces, with continuing computer hardware and software evolution and emerging communication standards, will foster rapid practice database development. It falls on our shoulders to establish standards for quality data and research so these databases can yield good results that help efficiently deliver quality health care. ACKNOWLEDGEMENT

This work was sponsored by grant number HS-05626 from the Agency for Health Care Policy and Research.

554

W. M. TIERNEY AND C. J. McDONALD

REFERENCES 1. Barnett, G. 0.‘The application of computer-based medical-record systems in ambulatory practice’, New England Journal of Medicine, 310, 1643-1650 (1984). 2. Stead, W. W. ‘Using computers to care for patients with renal disorders’, MD Computing, 1,42-50 (1984). 3. Pryor, T. A,, Gardner, R. M., Clayton, P. D. and Warner, H. R. ‘The HELP System’, Journal ofMedica1 Systems, 7 , 87-102 (1983). 4. Whiting-OKeefe, Q. E., Whiting, A. and Henke, J. ‘The STOR clinical information system’, M D Computing, 5, 8-21 (1988). 5. Bleich, H. L., Beckley, R. F., Horowitz, G. L., Jackson, J. D., Moody, E. S., Franklin, C., Goodman, S. R., McKay, M. W., Pope, R. A., Walden, T., Bloom, S. M. and Slack, W. V. ‘Clinical computing in a teaching hospital’, New England Journal of Medicine, 312, 756764 (1985). 6. McDonald, C. J., Blevins, L., Tierney, W. M. and Martin, D. K. ‘The Regenstrief Medical Records’, MD Computing, 5, 34-47 (1988). 7. Rosati, R. A., McNeer, J. F., Starmer, C. F., Mittler, B. S., Morris, J. J. Jr. and Wallace, A. G . ‘A new information system for medical practice’, Archives of Internal Medicine, 135, 1017-1024 (1975). 8. Fries, J. F. and McShane, D. J. ‘ARAMIS:a protocypical national chronic-disease databank in medical information’, Western Journal of Medicine, 145, 798-804 (1986). 9. Pryor, D. B., Califf, R. M., Harrell, F. E. Jr., Hlatky, M. A., Lee, K. L., Mark, D. B. and Rosati, R. A. ‘Clinical data bases: Accomplishments and unrealized potential’, Medical Care, 23, 623-647 (1985). 10. Tierney, W. M., Fitzgerald, J., McHenry, R., Roth, B. J., Psaty, B. M., Stump, D. L. and Anderson, F. K. ‘Physicians’estimates of the probability of myocardial infarction in emergency room patients with chest pain’, Medical Decision Making, 6, 12-17 (1986). 11. Durbridge, T. C., Edwards, F., Edwards, R. G. and Atkinson, M. ‘Evaluation of benefits of screening tests done immediately on admission to hospital’, Clinical Chemistry, 22, 968-971 (1976). 12. Lloyd, S. S. and Rissing, J. P. ‘Physician and coding errors in patient records’, Journal of the American Medical Association, 254, 1330-1 336 (1985). 13. Daley, J., Jencks, S., Draper, D., Lenhart, L., Thomas, N. and Walker, J. ‘Predicting hospital-associated mortality for Medicare patients’, Journal of the American Medical Association, 260, 3617-3624 (1988). 14. Stack, R. S., Califf, R. M., Hinohara, T., Phillips, H. R., Pryor, D. B., Simonton, C. A., Carlson, E. B., Morris, K. G., Behar, V. S., Kong, Y., Peter, R. H., Hlatky, M. A., O’Connor, C . M. and Mark, D. B. ‘Survival and cardiac event rates in the first year after emergency coronary angioplasty for acute myocardial infarction’, Journal of the American College of Cardiology, 11, 1141-1 149 (1988). 15. Slater, D. K., Hlatky, M. A., Mark, D. B., Harrell, F. E., Pryor, D. B. and Califf, R. M. ‘Outcome in suspected acute myocardial infarction with normal or minimally abnormal admission electrocardiographic findings’, American Journal of Cardiology, 60, 766-770 (1987). 16. Wolfe, F., Kleinheksel, S. M., Spitz, P. W., Lubeck, D. P., Fries, J. F., Young, D. Y., Mitchell, D. M. and Roth, S. H. ‘A multicenter study of hospitalization in rheumatoid arthritis: Effect of health care system, severity, and regional difference’, Journal of Rheumatology, 13, 277-284 (1986). 17. Lubeck, D. P., Spitz, P. W., Fries, J. F., Wolfe, R., Mitchell, D. M. and Roth, S. H. ‘A multicenter study of annual health service utilization and costs in rheumatoid arthritis’, Arthritis and Rheumatism, 29, 488-493 (1986). 18. Wolfe, F., Kleinheksel, S. M., Spitz, P. W., Lubeck, D. P., Fries, J. F., Young, D. Y., Mitchell, D. and Roth, S. ‘A multicenter study of hospitalization in rheumatoid arthritis: Frequency, medical-surgical admissions, and charges’, Arthritis and Rheumatism, 29, 614619 (1986). 19. Sherrer, Y. S., Block, D. A., Mitchell, D. M., Roth, S. H., Wolfe, F. and Freis, J. F. ‘Disability in rheumatoid arthritis: Comparison of prognostic factors across three populations’, Journal of Rheumatology, 14, 705-709 (1987). 20. Califf, R. M., Harrell, F. E. Jr., Lee, K. L., Rankin, J. S., Hlatky, M. A., Mark, D. B., Jones, R. H., Muhlbaier, L. H., Oldham, H. N. Jr. and Pryor, D. B. ‘The evolution of medical and surgical therapy for coronary artery disease: A 15-year perspective’, Journal of the American Medical Association, 261, 2077-2086 (1989). 21. Bounous, E. P., Mark, D. B., Pollock, B. G., Hlatky, M. A., Harrell, F. E. Jr., Lee, K. L., Rankin, J. S., Wechsler, A. S., Pryor, D. B. and Califf, R. M. ‘Surgical survival benefits for coronary disease patients with left ventricular dysfunction’, Circulation, 78 (supplement I), 1151-1157 (1988). 22. Olsen, D. M., Kane, R. L. and Proctor, P. H. ‘A controlled trial of multiphasic screening’, New England Journal of Medicine, 294, 925-930 (1976).

PRACTICE DATABASES USED IN CLINICAL RESEARCH

555

23. Fineberg, H. V. ‘Clinical chemistries: The high cost of low-cost diagnostic tests’, in Altman, S. H. and Bierdon, R. (eds), Medical Technology: The Culprit Behind Health Care Costs?, DHEW Publication no. (PHS) 79-3216, 1979. 24. Maloney, T. W. and Rogers, D. E. ‘Medical technology-a different view of the contentious debate over costs’, New England Journal of Medicine, 301, 1413-1419 (1979). 25. Williamson, J. W., Alexander, M. and Miller, G. E. ‘Continuing education and patient care research: Physician response to screening test results’, Journal ofthe American Medical Association, 201, 118-122 (1967). 26. Shapiro, M. F., Hatch, R. L. and Greenfield, S. ‘Cost containment and labor-intensive tests: The case of the leukocyte differential count’, Journal ofthe American Medical Association, 252, 23 1-234 (1984). 27. Hubbell, F. A. , Greenfield, S., Tyler, J. L., Chetty, K. and Wyle, F. A. ‘The impact of routine admission chest X-ray films on patient care’, New England Journal of Medicine, 312, 209-212 (1985). 28. Califf, R. M., Mark, D. B., Harrell, F. E. Jr., Hlatky, M. A., Lee, K. L., Rosati, R. A. and Pryor, D. B. ‘Importance of clinical measures of ischemia in the prognosis of patients with documented coronary artery disease’, Journal ofthe American College of Cardiology, 11, 2&26 (1988). 29. Lee, K. L., Pryor, D. B., Harrell, F. E. Jr., Califf, R. M., Behar, V. S., Floyd, W. L., Morris, J. J., Waugh, R. A., Whalen, R. E. and Rosati, R. A. ‘Predicting outcome in coronary disease: Statistical models versus expert clinicians’, American Journal of Medicine, 80, 553-650 (1986). 30. Kong, D. F., Lee, K. L., Harrell, F. E. Jr., Boswick, J. M., Mark, D. B., Hlatky, M. A., Califf, R. M. and Pryor, D. B. ‘Clinical experience and predicting survival in coronary disease’, Archioes of Internal Medicine, 149, 1177-1181 (1989). 31. Evans, R. S., Gardner, R. M., Bush, A. R., Burke, J. P., Jacobson, J. A., Larsen, R. A., Meier, F. A. and Warner, H. R. ‘Development of computerized infectious disease monitor’, Computers in Biomedical Research, 18, 103-1 13 (1985). 32. Evans, R. S., Larsen, R. A., Burke, J. P., Gardner, R. M., Meier, F. A,, Jacobson, J. A., Conti, M. T., Jacobson, J. T. and Hulse, R. K. ‘Computer surveillance of hospital-acquired infections and antibiotic use’, Journal of the American Medical Association, 265, 1007-101 1 (1986). 33. Mitchell, D. M., Spitz, P. W., Young, D. Y., Bloch, D. A., McShane, D. J. and Fries, J. F. ‘Survival, prognosis, and causes of death in rheumatoid arthritis’, Arthritis and Rheumatism, 29, 706-714 (1986). 34. Tierney, W. M., Martin, D. K., Hui, S. L. and McDonald, C. J. ‘Using clinical data to predict abnormal serum electrolytes and blood cell profiles’, Journal of General fnternal Medicine, 4, 375-383 (1989). 35. Tierney. W. M., McDonald, C. J., Hui, S. L. and Martin, D. K. ‘Computer predictions of abnormal test results: Effects on outpatient testing’, Journal of the American Medical Association, 259, 1194-1 I98 (1988). 36. Brazy, P. C., Stead, W. W. and Fitzwilliam, J. F. ‘Progression of renal insufficiency: Role of blood pressure’, Kidney International, 35, 670-674 (1989). 37. Levkoff, S. E., Safran, C., Cleary, P. D., Gallop, J. and Phillips, R. S. ‘Identification of factors associated with the diagnosis of delirium in elderly hospitalized patients’, Journal of the American Geriatrics Society, 36, 1099-1 104 (1988). 38. Tierney, W. M., McDonald, C. J. and Luft, F. C. ‘Renal disease in hypertensive adults: Effect of race and type-I1 diabetes mellitus’, American Journal of Kidney Diseases, 13, 485-493 (1989). 39. Singh, G., Fries, J. F., Spitz, P. and Williams, C. A. ‘Toxic effects of azathioprine in rheumatoid arthritis: A national post-marketing perspective’, Arthritis and Rheumatism, 32, 837-843 (1989). 40. Fries, J. F., Bloch, D. A., Segal, M. R., Spitz, P. W., Williams, C. and Lane, N. ‘Postmarketing surveillance in rheumatology: Analysis of purpura and upper abdominal pain’, Journal ofRheumatology, 15, 348-355 (1988). 41. Fries, J . F., Miller, S. R., Spitz, P. W., Williams, C. A., Hubert, H. B. and Bloch, D. A. ‘Toward an epidemiology of gastropathy associated with nonsteroidal antiinflammatory drug use’, Gastroenterology, 96, 647-655 (1989). 42. Kay, G. N., Pryor, D. B., Lee, K. L., Harrell, F. E. Jr., Pressley, J. C., Gilbert, M. R. and German, L. D. ‘Comparison of survival of amiodarone-treated patients with coronary artery disease and malignant ventricular arrhythmias with that of a control group with coronary artery disease’, Journal of the American College of Cardiology, 9, 877-881 (1987). 43. Tierney, W. M., McDonald, C. J. and McCabe, G. P. ‘Serum potassium testing in diuretic-treated outpatients: a multivariate approach’, Medical Decision Making, 5, 89-104 (1985). 44. McDonald, C. J., Hui, S. L. and Tierney, W. M. ‘Diuretic-induced laboratory abnormalities that predict ventricular ectopyz’, Journal of Chronic Diseases, 39, 127-135 (1986).

556

W. M. TIERNEY AND C. J. McDONALD

45. Murray, M. D., Brater, D. C., Tierney, W. M., Hui, S . L. and McDonald, C. J. ‘Ibuprofen-associated renal dysfunction in a large general internal medicine practice’, American Journal of the Medical Sciences, 299, 222-229 (1990). 46. Dittus, R. S. and Tierney, W. M. ‘Scheduling follow-up office visits: Physician variability’, Clinical Research, 35, 738A (1987). 47. Tierney, W. M., Hiu, S. L., Martin, D. K. and McDonald, C. J. ‘Variance in clinical testing in a general medicine clinic’, Clinical Research, 34, 842A (1986). 48. Fries, J. F., Spitz, P. W., Mitchell, D. M., Roth, S. H., Wolfe, F. and Bloch, D. A. ‘Impact of specific therapy upon rheumatoid arthritis’, Arthritis and Rheumatism, 29, 620-627 (1986). 49. Horn, S. D., Horn, R. A., Sharkey, P. D. and Chambers, A. F. ‘Severity of illness within DRGs: Homogeneity study’, Medical Care, 24 , 225-235 (1986). 50. Rogerson, C. L., Stimson, D. H., Simborg, D. W. and Charles, G. ‘Classification of ambulatory care using patient-based, time-oriented indexes’, Medical Care, 23, 780-788 (1985). 51. McMahon, L. F., Hayward, R. H., Bernard, A. M. and Rosevear, J. S. ‘A new severity of illness measure for inpatient care’, Clinical Research, 38, 322A (1990). 52. Safran, C., Porter, D., Slack, W. V. and Bleich, H. L. ‘Diagnosis-related Groups: A critical assessment of the provision of comorbidity’, Medical Care, 25, 1011-1014 (1987). 53. Fitzgerald, J. F., Moore, P. S. and Dittus R. S. ‘The care of elderly patients with hip fracture: Changes since implementation of the prospective payment system’, New England Journal of Medicine, 319, 1392-1397 (1988). 54. Larsen, R. A., Evans, R. S., Burke, J. P., Pestontnik, S. L., Gardner, R. M. and Classen, D. C. ‘Improved perioperative antibiotic use and reduced surgical wound infections through use of computer decision analysis’, Infection Control and Hospital Epidemiology, 10, 3 16-320 (1989). 55. Safran, C. and Phillips, R. S. ‘Interventions to prevent readmission: The constraints of cost and efficacy’, Medical Care, 27, 204-211 (1989). 56. Whiting-O’Keefe, Q. E., Simborg, D. W., Epstein, W. V. and Warger, A. ‘A computerized summary medical record system can provide more information than the standard medical record’, Journal of the American Medical Association, 254, 1185-1 192 (1985). 57. Fries, J. F. ‘Alternatives in medical record formats’, Medical Care, 12, 871-881 (1974). 58. Wilson, G. A., McDonald, C. J. and McCabe, G. P. ‘The effect of immediate access to a computerized medical record on physician test ordering: A controlled clinical trial in the emergency room’, American Journal of Public Health, 72, 698-702 (1982). 59. Tierney, W. M., McDonald, C. J., Martin, D. K., Hui, S. L. and Rogers, M. P. ‘Computerized display of past test results: Effects on outpatient testing’, Annals of Internal Medicine, 107, 569-574 (1987). 60. Winickoff, R. N., Coltin, K. L., Fleishman, S. J. and Barnett, G. 0.‘Semiautomated reminder system for improving syphilis management’, Journal of General Internal Medicine, 1, 78-84 (1986). 61. Barnett, G. O., Winickoff, R. N., Morgan, M. M. and Zielstorff, R. D. ‘A computer-based monitoring system for follow-up of elevated blood pressure’, Medical Care, 21, 400-409 (1983). 62. Dambro, M. R. and Kallgren, M. A. ‘Drug interactions in a clinic using COSTAR, Computers in Biology and Medicine, 18, 31-38 (1988). 63. McDonald, C. J., Hui, S. L., Smith, D. M., Tierney, W. M., Cohen, S. J., Weinberger, M. and McCabe, G. P. ‘Reminders to physicians from an introspective computer medical record a two-year randomized trial’, Annals of Internal Medicine, 100, 130-138 (1984). 64. Tierney, W. M., Hui, S. L. and McDonald, C . J. ‘Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance’, Medical Care, 24, 659-666 (1986). 65. Gehlbach, S. H., Wilkinson, W. E., Hammond, W. E., Clapp, N. E., Finn, A. L., Taylor, W. J. and Rodell, M. ‘Improving drug prescribing in a primary care practice’, Medical Care, 22, 193-201 (1984). 66. Pinciroli, F. and Pozzi, G. ‘A drug-to-drug interaction package’, Computing Methods Programs Biomed., 28, 183-189 (1989). 67. Davidson, K. W., Kahn, A. and Price, R. D. ‘Reduction of adverse drug reactions by computerized drug interaction screening’, Journal of Family Practice, 25, 371-375 (1987). 68. Haumschild, M. J., Ward, E. S., Bishop, J. M. and Haumschild, M. S. ‘Pharmacy-based computer system for monitoring and reporting drug interactions’, American Journal of Hospital Pharmacy, 4, 345-348 ( 1 987). 69. Edmond, E. D., Al-Hamouz, S., Tallis, R. C. and Vellodi, C. ‘Determining drug-drug interactions and related effects on microcomputers’, Computers in Biomedical Research, 20, 85-98 (1987).

PRACTICE DATABASES USED IN CLINICAL RESEARCH

557

70. Ishizuka, H., Hara, S. and Ishikura, C. ‘Computerized prescription checking system’, International Journal of Biomedical Computing, 19, 195-200 (1986). 71. Kirking, D. M., Thomas, J. W., Ascione, F. J. and Boyd, E. L. ‘Detecting and preventing adverse drug interactions: the potential contribution of computers in pharmacies’, Social Science and Medicine, 22, 1-8 (1986). 72. Strom, B. L., Carson, J. L., Morse, L. and LeRoy, A. A. ‘The computerized on-line Medicaid pharmaceutical analysis and surveillance system: A new resource for postmarketing drug surveillance’, Clinical Pharmacology and Therapeutics, 38, 359-364 (1985). 73. Rhodes, P. J., Rhodes, R. S., McClelland, G. H., Culbertson, V. L., Jahnigen, D. W. and Bloedow, D. C. ‘Evaluation of eight methods for estimating creatinine clearance in men’, Clinical Pharmacology, 6, 399-406 (1987). 74. Behar, S., Shcor, S., Kariv, I., Barell, V. and Modan, B. ‘Evaluation of electrocardiogram in emergency room as a decision-making tool’, Chest, 71, 486-491 (1977). 75. Zarling, E. J., Sexton, H. and Milnor, P. ‘Failure to diagnose acute myocardial infarction’, Journal ofthe American Medical Association, 250, 1177-1 181 (1983). 76. Lott, J. A. and Stang, J. M. ‘Serum enzymes and isoenzymes in the diagnosis and differential diagnosis of myocardial infarction and necrosis’, Clinical Chemistry, 26, 1241-1250 (1980). 77. Wagner, G. S., Roe, C. R., Limbird, L. E., Rosati, R. A. and Wallace, A. G . ‘The importance of identification of the myocardial-specific isoenzyme of creatine phosphokinase (MB form) in the diagnosis of acute myocardial infarction’, Circulation, 47, 263-269 (1973). 78. Albridge, K. M., Standish, J. and Fries, J. F. ‘Hierarchical time-oriented approaches to missing data inference’, Computers in Biomedical Research, 21, 349-366 (1988). 79. Hui. S. L. and Berger, J. 0. ‘Empirical Bayes estimation of rates in longitudinal studies’, Journal ofthe American Statistical Association, 78, 753-760 (1 983). 80. Califf, R. M., Pryor, D. B. and Greenfield, J. C. ‘Beyond randomized clinical trials: Applying clinical experience in the treatment of patients with coronary artery disease’, Circulation, 74, 1191-1 194 (1986). 81. Byar, D. P. ‘Why data bases should not replace randomized clinical trials’, Biometrics, 36, 337-342 (1980). 82. Pryor, D. B., Harrell, F. E. Jr., Rankin, J. S.,Lee, K. L., Muhlbaier, L. H., Oldham, H. N., Hlatky, M. A., Mark, D. B., Reves, J. G. and Califf, R. M. ‘The changing survival benefits of coronary revascularization over time’, Circulation, 76 (supplement V), V13-V21 (1987). 83. Califf, R. M., Harrell, F. E. Jr., Lee, K. L., Rankin, J. S., Mark, D. B., Hlatky, M. A., Muhlbaier, L. H., Wechsler, A. S., Jones, R. H., Oldham, H. N. Jr. and Pryor, D. B. ‘Changing efficacy of coronary revascularization: Implications for patient selection’, Circulation, 78 (Supplement I), 1185-1191 (1988). 84. Hlatky, M. A., Califf, R. M., Harrell, F. E. Jr., Lee, K. L., Mark, D. B. and Pryor, D. B. ‘Comparison of predictions based on observational data with the results of randomized controlled trials of coronary artery bypass surgery’, Journal of the American College of Cardiology, 11, 237-245 (1988). 85. ‘A Standard Specification for Transferring Clinical Laboratory Data Messages Between Independent Computer Systems’, ASTM 1238-1288, American Society for Testing Materials, Philadelphia, 1988. 86. McDonald, C. J. and Hammond, W. E. ‘Standard formats for electronic transfer of clinical data’, Annals offnternal Medicine, 110, 333-335 (1989). 87. Dambro, M. R., Weiss, B. D. and McClure, C. L. ‘An unsuccessful experience with computerized medical records in an academic medical center’, Journal of Medical Education. 63, 617-623 (1988).