Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
647
Review “G5
‘-
.
.
.
-
- - .
.
..
..
.
.
Technology Assessment: Organizations William
..
v
,
4\.i
“3 ..
. .
.
.
-
..
.
.
The Contribution
..
of Professional
A. Hendee1
demand. For the first time in history, delivery of health care became a profitable enterprise rather than a charitable service. In the exuberance over research advances, new medical technologies, and expansion of health care services, one question lurked unanswered: How were these changes to be paid for?
Assessment and accountability are becoming the watchwords of medical practice and reimbursement. Medical technologies, especially “big ticket” technologies such as those used in radiology, are obvious targets for increased attention. Providing enhanced accountability while preserving access to technologies for patients who need them requires expert knowledge and medical judgment. No more important role can be envisioned for professional medical organizations and technically expert physicians in practice.
Historical
Article
Era of Cost Containment Partly because of trends introduced during the era of expansion, health care costs rose inexorably in the 1 970s and 1980s. In response, health care payers (principally businesses, third-party carriers, and the government) introduced measures to slow the rising costs. Two decades after introduction of Medicare, health care costs had risen from less than 4% to more than 1 1 % of the United States gross national product. The rise continues today. Enhanced intensity and improved access were not the only factors contributing to rising costs of health care [2]. Other factors included population growth; population aging; inflation, especially labor costs; and liability coverage against malpractice claims. These factors all increased the costs of health care services in the United States. Early in the cost-containment era, front-end controls (e.g., health systems agencies and technology review commissions) were introduced to control the expansion of medical facilities and technologies. When these measures proved ineffective, back-end controls (e.g., diagnosis-related groups [DRGs], preferred provider services [PPSs], and health maintenance
Perspective
The past 40 years have witnessed momentous improvements in the quality of health care. They also have seen major shifts in the ways that the delivery of health care is evaluated and reimbursed. This period has been divided by Relman [1] into three eras: expansion, cost containment, and assessment and accountability.
Era of Expansion High enthusiasm for biomedical research and its clinical benefits, including advances in diagnostic imaging, was exhibited in the 1 950s and 1 960s. It also was a time for recognizing the rights of all persons to access to quality health care. Employer-subsidized health benefits, private health insurance, Medicare, and Medicaid increased the demand for health services. Hospitals, medical schools, and research institutes grew in size and number to meet the Received August 25, 1989; accepted after revision October 12, 1989. I Vice President for Science and Technology, American Medical Association, AJR 154:647-651,
March
1990 0361-803X/90/1543-0647
© American
535 N. Dearborn St. , Chicago,
Roentgen
Ray Society
IL 6061 0. Address reprint requests to W. A. Hendee.
HENDEE
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
648
organizations [HMOs]) were developed to Constrain expenditures by controlling the reimbursement process for medical services. Reimbursement for physicians’ services was included in these efforts through measures such as the resource-based relative value system (RBRVS) and expenditure targets (ETs). Cost control measures of the era of cost containment had little effect on the rising costs of health care. As with many other societal problems, rising health care costs cannot be addressed successfully by trying to constrain the supply of services. The rise in costs is driven principally by the increasing demand for more health services by more people. The only sure way to reduce costs in a demand-dominated situation is to deny services to some who need them (i.e., rationing of health care) and to limit the intensity of services (i.e., the use of technology) when they are provided. Today that is precisely the course of action being pursued by payers of health services, including governmental agencies. This course creates a conflict between government and social policy on the one hand and the expectations and rights of individuals on the other. Efforts to avoid visible conflict are largely responsible for the failure of front-end and back-end efforts to control health care expenditures. Now a major new initiative is evolving to move American medicine into a different era of health care delivery. This initiative offers major opportunities for involvement of health care providers in its planning and implementation. Era of Assessment
and Accountability
The basic challenge to any medical procedure is, does it contribute positively to the health and well-being of the patient? This challenge is the foundation of concepts such as “outcome measures” [3], “outcomes management” [4], the “assessment of medical practice” [5], and “quality assurance.” It underlies efforts of the Health Care Financing Administration (HCFA) to characterize the effectiveness of health care by examining morbidity and mortality statistics from the Medicare data base. It explains the current interest in data about adverse reactions, problems during patients’ recoveries, durations of the recovery period, and rates of readmission. It also explains present concerns over geographic, institutional, and practitioner variations in health care delivery. And it enhances interest in medical technology assessment, where technology is defined by the Office ofTechnology Assessment (OTA) ofthe U.S. Congress as “techniques, drugs, equipment and procedures used by health care professionals in delivering medical care to individuals, and the systems within which such care is delivered” [61. No medical discipline is spared from facing this challenge. But some, such as radiology, must confront it more directly than others because oftheir dependence on “big-ticket” technologies. Radiology is one of the principal targets of the era of assessment and accountability.
Methods
of Technology
Assessment
Various approaches are available for evaluating ness of medical technologies, including imaging
the usefulprocedures.
AJA:154,
Each approach has certain strengths explicitness, universality, and clinical
Randomized
Controlled
Clinical
March
and weaknesses meaningfulness.
1990
in its
Trials
Randomized controlled clinical trials (RCCTs) are the most rigorous approach to evaluation of medical technologies. When properly designed and executed, an RCCT applies the full power of statistical analysis to determination of the clinical usefulness of a technology. In some applications, such as evaluation of a new pharmaceutical that competes with existing drugs for treatment of conditions that are not life-threatening, an RCCT can be designed and implemented in a relatively straightforward manner. But when a drug provides hope to patients who have a life-threatening or untreatable disease, profound ethical concerns are raised by insistence on an RCCT with patients assigned randomly to treatment and control groups. These concerns have caused a relaxation of the insistence on RCCTs in several instances, including the recent introduction of the “Modified Treatment Investigational New Drug” application process of the U.S. Food and Drug Administration (FDA) for drugs intended for patients’ with terminal and untreatable diseases such as human immunodeficiency virus infection and certain forms of cancer. In the evaluation of imaging technologies, RCCTs are particularly difficult to implement. They depend on well-defined protocols, large numbers of patients, and random assignment of patients to study and control groups. They are expensive and time-consuming and usually require multiple institutions and investigators to eliminate bias introduced by specific groups of patients, equipment capabilities, and diagnostic skills. An institution with a new imaging technology is understandably inclined to encourage its use for all eligible patients, especially when the costs of the technology are recaptured from patient fees. Physicians face an ethical dilemma in depriving patients of a technology that could influence treatment of a particular disease or injury. Few imaging technologies have been evaluated by RCCTs. Exceptions include investigations of the usefulness of imaging procedures to detect occult disease in asymptomatic patients. Examples are attempts to identify lung cancer in asymptomatic male smokers [7, 8] and breast cancer in asymptomatic women without risk factors for the disease [9, 10].
Decision
Modeling
An essential first step in evaluating an imaging procedure is to determine its technical performance. Characteristics that define performance include precision (reproducibility), safety, and reliability. Often reliability is expressed in terms of sensitivity and specificity, where sensitivity connotes the ability of a procedure to detect disease when it is present, and specificity describes the ability of a test to correctly exclude disease when it is absent. Graphical presentation of the sensitivity and specificity of a diagnostic test yields a receiver-operatingcharacteristics (ROC) curve for the test [1 1].
AJR:154,
March
1990
TECHNOLOGY
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
Case Series A case series is a collection of studies in which a technology is used for a specific condition of a patient. Studies can be compiled into a case series ifthey are conducted longitudinally over time and if all eligible patients are admitted to the studies. The series may be prospective or retrospective; the former is considered more reliable because of greater assurance that all eligible patients have entered the series. The value of a case series is enhanced by comparison with a control group, even though patients are not assigned randomly to control or study populations. Often a case series reflects the experience of a single institution and may be biased by the skill of the investigators, particular model of the technology used, procedures followed in application of the technology, and population of patients drawn to the institution. Many case series are presented as technology assessments in the radiologic literature, and some exemplify the biases intrinsic in this evaluative method. The presence of such biases in clinical series recently has attracted attention in the radiologic literature [12, 13].
Case Studies A case study is a comprehensive review of several features of a particular technology, such as its clinical applications, economic and ethical implications, and potential for future research and development. Examples include the reports of the Congressional Office of Technology Assessment (OTA) [14] and the guideline reports of the hospital technology series of the American Hospital Association. The AMA/Trends reports of the American Medical Association are another example, as represented by a case study report on sonography currently in preparation. Recent reports of the AMA’s Council on Scientific Affairs concerning imaging technologies such as mammography, MR imaging, and positron emission tomography are further examples of case studies. These reports have been published in the Journal of the American Medical Association (JAMA) in the past couple of years. Case studies are designed as wide-ranging “snapshots in time” of the state of development and clinical applications of a particular medical technology. They tend to draw heavily on the literature, with discretion exercised by authors chosen for their expertise and objectivity. If carefully conducted and extensively documented, case series are useful “state-of-theart” references to particular technologies, especially for those contemplating the purchase and use of the technologies.
649
ASSESSMENT
address questions that may not be answerable with any one study [15]. Extraction of definitive answers by combining several equivocal studies is appealing. The methods of metanalysis are still rather primitive, however, and many challenges remain. Combined studies must be comparable enough to permit “pooling” of data, and the literature search must be sufficiently comprehensive to capture an adequate number of representative and relevant studies so that selection bias is eliminated. Negative studies are often not published but may be important to evaluation of a particular technology. Metanalysis is a developing field with considerable promise not only for technology assessment, but also for many applications in scientific research and clinical medicine [1 6]. Metanalysis has been used only rarely for technology assessments in diagnostic imaging. One example is its recent use to determine the clinical usefulness of mammographic screening in younger asymptomatic women [10].
Consensus
Development
Achievement of a collective opinion about the value of a particular technology through interaction of a group of experts is termed consensus development. The process is often informal, such as that used in the consensus development conferences of the National Institutes of Health (NIH). In this approach, a small panel of experts is assembled to hear testimony about the clinical usefulness and potential of a particular technology. Then the panel drafts responses to specific questions about the technology, and these responses are discussed with those offering testimony and with others in the audience. Through repeated exchanges, a final document is prepared that purportedly reflects the consensus of the assembly. Although useful, this approach is vulnerable to subjective influence by the small number of experts on the panel and by forceful persons in the audience. Several NIH consensus conferences have addressed imaging technologies, including mammographic screening, sonography in pregnancy, CT, and MR imaging. One approach to reduction of potential bias in the groupjudgment approach is use of the DELPHI technique [1 7]. In this method, expert panelists are isolated and unknown to each other. Initial opinions about a technology are obtained from each panelist, and a summary is prepared and distributed to the panel. Each panelist is then asked to respond to the original questions with knowledge of the collective opinion of the panel. This process is repeated several times until a final report can be prepared that reflects a panel consensus.
Metanalysis Metanalysis is a statistical technique for extracting answers to well-defined questions about a subject through analysis of the scientific literature concerning the subject. Review of the scientific literature is an essential feature of all approaches to technology assessment. Published studies often reflect limited experience, however, and may not provide definitive answers to questions about efficacy, effectiveness, and safety of a particular technology. Metanalysis permits the results of several studies to be combined in an effort to
Opinion
Surveys
A straightforward approach to assessing the value of a particular technology is solicitation of opinions from experts and the compilation of these opinions into a report of collective opinion. The experts do not interact with each other, and they are unable to modify their opinions, once stated, except possibly through review of the final report. The most widely known opinion surveys related to medical technologies are performed by the Diagnostic and Therapeutic Technology
HENDEE
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
650
Assessment (DATTA) program, functioning under the Council of Scientific Affairs of the AMA. These surveys use highly structured questions and a choice of specific answers to arrive at a group decision about the safety and efficacy of a particular technology. Answers to questions of safety and efficacy are categorized as established, investigational, promising, doubtful, and unacceptable. The process permits expression of dissenting viewpoints from a majority decision and provides a rapid method for assessment of the clinical value of a particular technology. Several DATTA opinions have examined imaging technologies, including mammographic screening and transrectal sonography for examinations of the prostate. The panel of potential contributors to a DATTA opinion currently consists of more than 1 700 physicians nominated by medical specialty organizations (44%), medical schools (36%), state medical societies (1 2%), and others (8%). When representatives of this panel are selected for a particular DATTA opinion, physicians are chosen who are expert in the technology or who refer patients to it. Typically, 30-100 physicians contribute to a DATTA opinion. As with any technology assessment effect, the opinion survey suffers from certain limitations, including presentation of a point of view that reflects the clinical use of a technology rather than the “cutting edge” of the technology’s development. The point of view also may, at times, reflect the opinions principally of advocates of a technology, despite efforts to eliminate bias through involvement of many contributors.
Organizations
Performing
Technology
Assessment
Several organizations are involved in medical technology assessment, and more are becoming interested as the topic gains visibility in government, business, and medical circles. The OTA was established by the U.S. Congress to study the ways that technology affects people’s lives. The Division of Health and Life Sciences of OTA conducts case studies of technologies under the guidance of an advisory board of experts and with counsel from institutions and organizations within the private sector. The time for completion of a study and preparation of a report averages 1 8 months. Questions about coverage of new technologies under Medicare are addressed by thefederal Office of Health Technology Assessment (OHTA). This office is under the jurisdiction of the National Center for Health Services Research and Technology Assessment which also has responsibility for health services research studies on concerns such as geographic variations in practice patterns. Technology assessments are performed by OHTA for HCFA, the federal agency responsible for Medicare reimbursement policies. A panel of physicians and members of the OHTA staff determine priorities for assessments, often in response to queries from Medicare contractors (fiscal intermediaries), manufacturers, and insurers. Notices of pending assessments are listed in the Federal Register, and input is requested from interested organizations and individuals. The process is time-consuming, with assessments often requiring 2-3 years from the time a request is made to HCFA to culmination in an OHTA report.
AJA:154,
March
1990
Several factors influence a decision by HCFA and OHTA to consider a particular technology for a coverage decision to guide Medicare contractors [1 8]. These factors include the following: the service is a significant medical advance; the service is a “new product,” and Medicare currently covers no similar technology; the service is likely to be used in more than one region of the country; the service is likely to be a significant expense for Medicare; the service has potential for rapid diffusion and application; experts disagree about the safety, efficacy, and appropriateness of the service; fiscal intermediaries have treated the service inconsistently; and the service is outmoded and it is possible that coverage should be withdrawn. Any one of these factors is considered by HCFA and OHTA as sufficient to initiate review of a technology. This position is a contentious issue within the medical community. In deciding about coverage and reimbursement, privatesector third-party carriers frequently must evaluate the clinical usefulness of particular technologies. Criteria for these decisions are illustrated by those of the National Blue Cross and Blue Shield Association [1 9]. A recommendation for coverage by this agency usually requires that a technology satisfy five criteria: (1) the technology must have final approval from appropriate governmental regulatory bodies, (2) the scientific evidence must permit conclusions concerning the effect of the technology on health outcome, (3) the technology must improve the net health outcome, (4) the technology must be at least as beneficial as all established alternatives, and (5) the benefit must be attainable outside an investigational setting. Application of these criteria, and acquisition of the requisite data to satisfy them, can significantly delay coverage and reimbursement of a technology after it is approved by the FDA for commercial release [20]. At the Institute of Medicine of the National Academy of Sciences, The Council on Health Care Technology was established in 1 986 to promote technology assessment in health care. The council consists of 1 6 members and serves as a clearinghouse for information related to medical technologies and their assessment. It also intends to improve the procedures of technology assessment; identify needs in technology assessment; stimulate the assessment of medical technologies; and promote education, training, and assistance related to technology assessment. In 1981 the American College of Physicians established the Clinical Efficacy and Assessment Project (CEAP) for evaluation of medical technologies [21 Technologies are selected for assessment on the basis of the degree of interest to internists, potential for widespread application, and anticipation of significant benefit or risk. From 1 0 to 1 2 topics are selected each year; the time for completion ranges from 9 to 1 5 months. The reports are reviewed extensively by expert advisers and ultimately become official positions of the college. A task force on assessment of diagnostic and therapeutic cardiovascular procedures develops guidelines on diagnosis and management of cardiovascular disease under the aegis ofthe American College of Cardiology and the American Heart Association [22]. Topics are selected by the task force from suggestions of members of both organizations, and reports ,
].
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
AJR:154,
TECHNOLOGY
March 1990
examine several features of a technology, including sensitivity and specificity, indications and contraindications, and costeffectiveness. Reports also consider the training, credentialing, and facilities recommended for use of the technology. The average time from selection of a topic to release of a report is 1 8 months. The AMA has several activities related to technology assessment. The text Drug Evaluations [23], now nearing its seventh edition, is the most comprehensive sourcebook available worldwide on the comparative use of pharmaceuticals in medical practice. The AMA’s Council on Scientific Affairs publishes several reports on medical technologies each year in JAMA. In 1982, the AMA formed the DAUA program described earlier. Ten to 1 2 assessments are performed each year, and the reports are published in JAMA. The average time from selection of a DATTA topic to submission of a report to JAMA for publication is about 3 months. Recently the American College of Radiology started a multiinstitutional study with certain features of technology assessment. This effort, termed the Radiologic Diagnostic Oncology Group (RDOG) study and supported with funds from the National Cancer Institute, is directed first at comparison of the effectiveness of CT with that of MR imaging for staging lung cancer. A parallel study focuses on a comparison of MR imaging and transrectal sonography for staging prostatic cancer. Additional studies are being designed for identification of preferred methods for staging colorectal and pancreatic cancer. These studies are coordinated by the Department of Health Policy Analysis at Harvard University.
Conclusions The importance of technology assessments by professional organizations such as the American College of Radiology, American College of Cardiology, American College of Physicians, and AMA cannot be overemphasized. Organizations such as these have access to the practice environment and the clinical judgment required to evaluate the effectiveness and appropriateness of particular technologies in a clinical setting. This judgment is crucial to continued development and use of medical technologies in an efficacious and costconscious manner for improvement of care and well-being of patients. As Kent and Larson [24] have emphasized, the effectiveness oftechnology assessment efforts often can be enhanced through multidisciplinary efforts involving physicians from several specialties. If an effective counterbalance to efforts to control costs through restrictions on coverage and reimbursement of medical technologies is not developed, further limitations on the access of patients to quality health care can be anticipated. In providing this counterbalance, medical organizations can help physicians advocate the rights of patients to continued access to the quality of health care that they need and deserve. No purpose is closer than that of
The reader’s
attenon
is directed
to thecommentaryon
ASSESSMENT
651
advocacy of patients organizations.
to the true role of professional
medical
REFERENCES 1 . Aelman AS. Assessment and accountability: the third revolution in medical care. N EngIJ Med 1989;319:1220-1222 2. Schwartz WB. The inevitable failure of current cost-containment strategies. JAMA 1987:257:220-224 3. Fineberg HV. Effects of clinical evaluation on the diffusion of medical technology. In: Institute of Medicine. Assessing medical technologies.
Washington,
DC: National Academy Press, 1985:176-210
4. Ellwood PM. Outcomes management: a technology of patient experience. N EnglJ Med 1980:318:1549-1556 5. Roper WG, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care: an initiative to evaluate and improve medical practice. N EngI J Med 1988;319:1197-1202 6. Institute of Medicine. Assessing
medical technologies.
Washington.
DC:
National Academy Press, 1985:1 7. Melamed MA, Flehinger BJ, Zaman MB, Heelan AT, Hallerman ET, Martin N. Detection of true pathologic stage 1 lung cancer in the screening program and the effect on survival. cancer 1981;47(suppl.j:1182-1187 8. Wooner LB, Fontana AS, Sanderson DA, et al. Mayo lung project: evaluation of lung cancer screening through December 1979. Mayo Olin Proc 1981;56:544-555 9. Shapiro S. Evidence on screening for breast cancer from a randomized trial. Cancer 1977:39:2722-2782 10. Eddy DM, Hasseiblad V, McGivney WT. Hendee WA. The value of mammography screening in women under 50 years. JAMA 1988:259:15121519 1 1 . Metz CE. Basic principles of ROC analysis. Semin NucI Med 1978;7: 283-298 12. Wolf GL. Poor quality of clinical research in radiology: another indictment. Radiology 1989;170:31 1-312 13. Cooper LS, Chalmers TC, McCarty M, Berrier J, Sacks HS. The poor quality of early evaluations of magnetic resonance imaging. JAMA 1988;259:3277-3280 14. OffIce of Technology Assessment. Strategies for medical technology assessment. Washington. DC: U.S. Government Printing Office, 1982 15. Louis TA, Fineberg H, Mosteller F. Findings for public health from metan-
alysis. In: Annual review: Reviewslnc., 1986 16. Goodman Assessing
public
health,
1985.
C. Quantitative synthesis methods. medical technologies. Washington.
Palo
Alto,
In: Institute DC:
National
CA:
Annual
of Medicine. Academy
Press, 1985:125-126 1 7. Dalkey NC. The DELPHI method: An experimental study of group opinion. Santa Monica, CA: Rand Corp., 1969 18. Glenn KJ. Assessing medical technology. Medicine and Health Perspectives. Washington. DC: Medicine and Health. June 5, 1989 19. Blue Cross/Blue Shield. Technology evaluation criteria. Chicago: Blue Cross and Blue Shield Association, 1987 20. McGivney WT, Hendee WR. Regulation, coverage and reimbursement of medical technologies. Int J Radiat Once! Biol Phys (in press) 21 . American College of Physicians. clinical Efficacy Assessment Project (CEAPj: Fact sheet, procedural manual and final report. Philadelphia: American College of Physicians, 1983 22. American College of Cardiology. Statements of charge: Joint ACC/AHA Task Force on Assessment of Cardiovascular Procedures (and Subcommittees), Cardiovascular Procedures committee, and Cardiovascular Norms Committee. Bethesda, MD: American College of Cardiology, 1985 23. American Medical Association. Drug evaluations, 6th ad. Chicago: AMA,
1988 24.
Kent DL, Larson
EP. Magnetic
Is clinical efficacy
established
resonance
imaging
of the brain and spine. Ann Intern Med
after the first decade?
1988:108:402-424
thisartkIe,whichappears
onthefoflowuigpages.
]
647
Review “G5
‘-
.
.
.
-
- - .
.
..
..
.
.
Technology Assessment: Organizations William
..
v
,
4\.i
“3 ..
. .
.
.
-
..
.
.
The Contribution
..
of Professional
A. Hendee1
demand. For the first time in history, delivery of health care became a profitable enterprise rather than a charitable service. In the exuberance over research advances, new medical technologies, and expansion of health care services, one question lurked unanswered: How were these changes to be paid for?
Assessment and accountability are becoming the watchwords of medical practice and reimbursement. Medical technologies, especially “big ticket” technologies such as those used in radiology, are obvious targets for increased attention. Providing enhanced accountability while preserving access to technologies for patients who need them requires expert knowledge and medical judgment. No more important role can be envisioned for professional medical organizations and technically expert physicians in practice.
Historical
Article
Era of Cost Containment Partly because of trends introduced during the era of expansion, health care costs rose inexorably in the 1 970s and 1980s. In response, health care payers (principally businesses, third-party carriers, and the government) introduced measures to slow the rising costs. Two decades after introduction of Medicare, health care costs had risen from less than 4% to more than 1 1 % of the United States gross national product. The rise continues today. Enhanced intensity and improved access were not the only factors contributing to rising costs of health care [2]. Other factors included population growth; population aging; inflation, especially labor costs; and liability coverage against malpractice claims. These factors all increased the costs of health care services in the United States. Early in the cost-containment era, front-end controls (e.g., health systems agencies and technology review commissions) were introduced to control the expansion of medical facilities and technologies. When these measures proved ineffective, back-end controls (e.g., diagnosis-related groups [DRGs], preferred provider services [PPSs], and health maintenance
Perspective
The past 40 years have witnessed momentous improvements in the quality of health care. They also have seen major shifts in the ways that the delivery of health care is evaluated and reimbursed. This period has been divided by Relman [1] into three eras: expansion, cost containment, and assessment and accountability.
Era of Expansion High enthusiasm for biomedical research and its clinical benefits, including advances in diagnostic imaging, was exhibited in the 1 950s and 1 960s. It also was a time for recognizing the rights of all persons to access to quality health care. Employer-subsidized health benefits, private health insurance, Medicare, and Medicaid increased the demand for health services. Hospitals, medical schools, and research institutes grew in size and number to meet the Received August 25, 1989; accepted after revision October 12, 1989. I Vice President for Science and Technology, American Medical Association, AJR 154:647-651,
March
1990 0361-803X/90/1543-0647
© American
535 N. Dearborn St. , Chicago,
Roentgen
Ray Society
IL 6061 0. Address reprint requests to W. A. Hendee.
HENDEE
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
648
organizations [HMOs]) were developed to Constrain expenditures by controlling the reimbursement process for medical services. Reimbursement for physicians’ services was included in these efforts through measures such as the resource-based relative value system (RBRVS) and expenditure targets (ETs). Cost control measures of the era of cost containment had little effect on the rising costs of health care. As with many other societal problems, rising health care costs cannot be addressed successfully by trying to constrain the supply of services. The rise in costs is driven principally by the increasing demand for more health services by more people. The only sure way to reduce costs in a demand-dominated situation is to deny services to some who need them (i.e., rationing of health care) and to limit the intensity of services (i.e., the use of technology) when they are provided. Today that is precisely the course of action being pursued by payers of health services, including governmental agencies. This course creates a conflict between government and social policy on the one hand and the expectations and rights of individuals on the other. Efforts to avoid visible conflict are largely responsible for the failure of front-end and back-end efforts to control health care expenditures. Now a major new initiative is evolving to move American medicine into a different era of health care delivery. This initiative offers major opportunities for involvement of health care providers in its planning and implementation. Era of Assessment
and Accountability
The basic challenge to any medical procedure is, does it contribute positively to the health and well-being of the patient? This challenge is the foundation of concepts such as “outcome measures” [3], “outcomes management” [4], the “assessment of medical practice” [5], and “quality assurance.” It underlies efforts of the Health Care Financing Administration (HCFA) to characterize the effectiveness of health care by examining morbidity and mortality statistics from the Medicare data base. It explains the current interest in data about adverse reactions, problems during patients’ recoveries, durations of the recovery period, and rates of readmission. It also explains present concerns over geographic, institutional, and practitioner variations in health care delivery. And it enhances interest in medical technology assessment, where technology is defined by the Office ofTechnology Assessment (OTA) ofthe U.S. Congress as “techniques, drugs, equipment and procedures used by health care professionals in delivering medical care to individuals, and the systems within which such care is delivered” [61. No medical discipline is spared from facing this challenge. But some, such as radiology, must confront it more directly than others because oftheir dependence on “big-ticket” technologies. Radiology is one of the principal targets of the era of assessment and accountability.
Methods
of Technology
Assessment
Various approaches are available for evaluating ness of medical technologies, including imaging
the usefulprocedures.
AJA:154,
Each approach has certain strengths explicitness, universality, and clinical
Randomized
Controlled
Clinical
March
and weaknesses meaningfulness.
1990
in its
Trials
Randomized controlled clinical trials (RCCTs) are the most rigorous approach to evaluation of medical technologies. When properly designed and executed, an RCCT applies the full power of statistical analysis to determination of the clinical usefulness of a technology. In some applications, such as evaluation of a new pharmaceutical that competes with existing drugs for treatment of conditions that are not life-threatening, an RCCT can be designed and implemented in a relatively straightforward manner. But when a drug provides hope to patients who have a life-threatening or untreatable disease, profound ethical concerns are raised by insistence on an RCCT with patients assigned randomly to treatment and control groups. These concerns have caused a relaxation of the insistence on RCCTs in several instances, including the recent introduction of the “Modified Treatment Investigational New Drug” application process of the U.S. Food and Drug Administration (FDA) for drugs intended for patients’ with terminal and untreatable diseases such as human immunodeficiency virus infection and certain forms of cancer. In the evaluation of imaging technologies, RCCTs are particularly difficult to implement. They depend on well-defined protocols, large numbers of patients, and random assignment of patients to study and control groups. They are expensive and time-consuming and usually require multiple institutions and investigators to eliminate bias introduced by specific groups of patients, equipment capabilities, and diagnostic skills. An institution with a new imaging technology is understandably inclined to encourage its use for all eligible patients, especially when the costs of the technology are recaptured from patient fees. Physicians face an ethical dilemma in depriving patients of a technology that could influence treatment of a particular disease or injury. Few imaging technologies have been evaluated by RCCTs. Exceptions include investigations of the usefulness of imaging procedures to detect occult disease in asymptomatic patients. Examples are attempts to identify lung cancer in asymptomatic male smokers [7, 8] and breast cancer in asymptomatic women without risk factors for the disease [9, 10].
Decision
Modeling
An essential first step in evaluating an imaging procedure is to determine its technical performance. Characteristics that define performance include precision (reproducibility), safety, and reliability. Often reliability is expressed in terms of sensitivity and specificity, where sensitivity connotes the ability of a procedure to detect disease when it is present, and specificity describes the ability of a test to correctly exclude disease when it is absent. Graphical presentation of the sensitivity and specificity of a diagnostic test yields a receiver-operatingcharacteristics (ROC) curve for the test [1 1].
AJR:154,
March
1990
TECHNOLOGY
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
Case Series A case series is a collection of studies in which a technology is used for a specific condition of a patient. Studies can be compiled into a case series ifthey are conducted longitudinally over time and if all eligible patients are admitted to the studies. The series may be prospective or retrospective; the former is considered more reliable because of greater assurance that all eligible patients have entered the series. The value of a case series is enhanced by comparison with a control group, even though patients are not assigned randomly to control or study populations. Often a case series reflects the experience of a single institution and may be biased by the skill of the investigators, particular model of the technology used, procedures followed in application of the technology, and population of patients drawn to the institution. Many case series are presented as technology assessments in the radiologic literature, and some exemplify the biases intrinsic in this evaluative method. The presence of such biases in clinical series recently has attracted attention in the radiologic literature [12, 13].
Case Studies A case study is a comprehensive review of several features of a particular technology, such as its clinical applications, economic and ethical implications, and potential for future research and development. Examples include the reports of the Congressional Office of Technology Assessment (OTA) [14] and the guideline reports of the hospital technology series of the American Hospital Association. The AMA/Trends reports of the American Medical Association are another example, as represented by a case study report on sonography currently in preparation. Recent reports of the AMA’s Council on Scientific Affairs concerning imaging technologies such as mammography, MR imaging, and positron emission tomography are further examples of case studies. These reports have been published in the Journal of the American Medical Association (JAMA) in the past couple of years. Case studies are designed as wide-ranging “snapshots in time” of the state of development and clinical applications of a particular medical technology. They tend to draw heavily on the literature, with discretion exercised by authors chosen for their expertise and objectivity. If carefully conducted and extensively documented, case series are useful “state-of-theart” references to particular technologies, especially for those contemplating the purchase and use of the technologies.
649
ASSESSMENT
address questions that may not be answerable with any one study [15]. Extraction of definitive answers by combining several equivocal studies is appealing. The methods of metanalysis are still rather primitive, however, and many challenges remain. Combined studies must be comparable enough to permit “pooling” of data, and the literature search must be sufficiently comprehensive to capture an adequate number of representative and relevant studies so that selection bias is eliminated. Negative studies are often not published but may be important to evaluation of a particular technology. Metanalysis is a developing field with considerable promise not only for technology assessment, but also for many applications in scientific research and clinical medicine [1 6]. Metanalysis has been used only rarely for technology assessments in diagnostic imaging. One example is its recent use to determine the clinical usefulness of mammographic screening in younger asymptomatic women [10].
Consensus
Development
Achievement of a collective opinion about the value of a particular technology through interaction of a group of experts is termed consensus development. The process is often informal, such as that used in the consensus development conferences of the National Institutes of Health (NIH). In this approach, a small panel of experts is assembled to hear testimony about the clinical usefulness and potential of a particular technology. Then the panel drafts responses to specific questions about the technology, and these responses are discussed with those offering testimony and with others in the audience. Through repeated exchanges, a final document is prepared that purportedly reflects the consensus of the assembly. Although useful, this approach is vulnerable to subjective influence by the small number of experts on the panel and by forceful persons in the audience. Several NIH consensus conferences have addressed imaging technologies, including mammographic screening, sonography in pregnancy, CT, and MR imaging. One approach to reduction of potential bias in the groupjudgment approach is use of the DELPHI technique [1 7]. In this method, expert panelists are isolated and unknown to each other. Initial opinions about a technology are obtained from each panelist, and a summary is prepared and distributed to the panel. Each panelist is then asked to respond to the original questions with knowledge of the collective opinion of the panel. This process is repeated several times until a final report can be prepared that reflects a panel consensus.
Metanalysis Metanalysis is a statistical technique for extracting answers to well-defined questions about a subject through analysis of the scientific literature concerning the subject. Review of the scientific literature is an essential feature of all approaches to technology assessment. Published studies often reflect limited experience, however, and may not provide definitive answers to questions about efficacy, effectiveness, and safety of a particular technology. Metanalysis permits the results of several studies to be combined in an effort to
Opinion
Surveys
A straightforward approach to assessing the value of a particular technology is solicitation of opinions from experts and the compilation of these opinions into a report of collective opinion. The experts do not interact with each other, and they are unable to modify their opinions, once stated, except possibly through review of the final report. The most widely known opinion surveys related to medical technologies are performed by the Diagnostic and Therapeutic Technology
HENDEE
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
650
Assessment (DATTA) program, functioning under the Council of Scientific Affairs of the AMA. These surveys use highly structured questions and a choice of specific answers to arrive at a group decision about the safety and efficacy of a particular technology. Answers to questions of safety and efficacy are categorized as established, investigational, promising, doubtful, and unacceptable. The process permits expression of dissenting viewpoints from a majority decision and provides a rapid method for assessment of the clinical value of a particular technology. Several DATTA opinions have examined imaging technologies, including mammographic screening and transrectal sonography for examinations of the prostate. The panel of potential contributors to a DATTA opinion currently consists of more than 1 700 physicians nominated by medical specialty organizations (44%), medical schools (36%), state medical societies (1 2%), and others (8%). When representatives of this panel are selected for a particular DATTA opinion, physicians are chosen who are expert in the technology or who refer patients to it. Typically, 30-100 physicians contribute to a DATTA opinion. As with any technology assessment effect, the opinion survey suffers from certain limitations, including presentation of a point of view that reflects the clinical use of a technology rather than the “cutting edge” of the technology’s development. The point of view also may, at times, reflect the opinions principally of advocates of a technology, despite efforts to eliminate bias through involvement of many contributors.
Organizations
Performing
Technology
Assessment
Several organizations are involved in medical technology assessment, and more are becoming interested as the topic gains visibility in government, business, and medical circles. The OTA was established by the U.S. Congress to study the ways that technology affects people’s lives. The Division of Health and Life Sciences of OTA conducts case studies of technologies under the guidance of an advisory board of experts and with counsel from institutions and organizations within the private sector. The time for completion of a study and preparation of a report averages 1 8 months. Questions about coverage of new technologies under Medicare are addressed by thefederal Office of Health Technology Assessment (OHTA). This office is under the jurisdiction of the National Center for Health Services Research and Technology Assessment which also has responsibility for health services research studies on concerns such as geographic variations in practice patterns. Technology assessments are performed by OHTA for HCFA, the federal agency responsible for Medicare reimbursement policies. A panel of physicians and members of the OHTA staff determine priorities for assessments, often in response to queries from Medicare contractors (fiscal intermediaries), manufacturers, and insurers. Notices of pending assessments are listed in the Federal Register, and input is requested from interested organizations and individuals. The process is time-consuming, with assessments often requiring 2-3 years from the time a request is made to HCFA to culmination in an OHTA report.
AJA:154,
March
1990
Several factors influence a decision by HCFA and OHTA to consider a particular technology for a coverage decision to guide Medicare contractors [1 8]. These factors include the following: the service is a significant medical advance; the service is a “new product,” and Medicare currently covers no similar technology; the service is likely to be used in more than one region of the country; the service is likely to be a significant expense for Medicare; the service has potential for rapid diffusion and application; experts disagree about the safety, efficacy, and appropriateness of the service; fiscal intermediaries have treated the service inconsistently; and the service is outmoded and it is possible that coverage should be withdrawn. Any one of these factors is considered by HCFA and OHTA as sufficient to initiate review of a technology. This position is a contentious issue within the medical community. In deciding about coverage and reimbursement, privatesector third-party carriers frequently must evaluate the clinical usefulness of particular technologies. Criteria for these decisions are illustrated by those of the National Blue Cross and Blue Shield Association [1 9]. A recommendation for coverage by this agency usually requires that a technology satisfy five criteria: (1) the technology must have final approval from appropriate governmental regulatory bodies, (2) the scientific evidence must permit conclusions concerning the effect of the technology on health outcome, (3) the technology must improve the net health outcome, (4) the technology must be at least as beneficial as all established alternatives, and (5) the benefit must be attainable outside an investigational setting. Application of these criteria, and acquisition of the requisite data to satisfy them, can significantly delay coverage and reimbursement of a technology after it is approved by the FDA for commercial release [20]. At the Institute of Medicine of the National Academy of Sciences, The Council on Health Care Technology was established in 1 986 to promote technology assessment in health care. The council consists of 1 6 members and serves as a clearinghouse for information related to medical technologies and their assessment. It also intends to improve the procedures of technology assessment; identify needs in technology assessment; stimulate the assessment of medical technologies; and promote education, training, and assistance related to technology assessment. In 1981 the American College of Physicians established the Clinical Efficacy and Assessment Project (CEAP) for evaluation of medical technologies [21 Technologies are selected for assessment on the basis of the degree of interest to internists, potential for widespread application, and anticipation of significant benefit or risk. From 1 0 to 1 2 topics are selected each year; the time for completion ranges from 9 to 1 5 months. The reports are reviewed extensively by expert advisers and ultimately become official positions of the college. A task force on assessment of diagnostic and therapeutic cardiovascular procedures develops guidelines on diagnosis and management of cardiovascular disease under the aegis ofthe American College of Cardiology and the American Heart Association [22]. Topics are selected by the task force from suggestions of members of both organizations, and reports ,
].
Downloaded from www.ajronline.org by 23.233.1.133 on 11/09/15 from IP address 23.233.1.133. Copyright ARRS. For personal use only; all rights reserved
AJR:154,
TECHNOLOGY
March 1990
examine several features of a technology, including sensitivity and specificity, indications and contraindications, and costeffectiveness. Reports also consider the training, credentialing, and facilities recommended for use of the technology. The average time from selection of a topic to release of a report is 1 8 months. The AMA has several activities related to technology assessment. The text Drug Evaluations [23], now nearing its seventh edition, is the most comprehensive sourcebook available worldwide on the comparative use of pharmaceuticals in medical practice. The AMA’s Council on Scientific Affairs publishes several reports on medical technologies each year in JAMA. In 1982, the AMA formed the DAUA program described earlier. Ten to 1 2 assessments are performed each year, and the reports are published in JAMA. The average time from selection of a DATTA topic to submission of a report to JAMA for publication is about 3 months. Recently the American College of Radiology started a multiinstitutional study with certain features of technology assessment. This effort, termed the Radiologic Diagnostic Oncology Group (RDOG) study and supported with funds from the National Cancer Institute, is directed first at comparison of the effectiveness of CT with that of MR imaging for staging lung cancer. A parallel study focuses on a comparison of MR imaging and transrectal sonography for staging prostatic cancer. Additional studies are being designed for identification of preferred methods for staging colorectal and pancreatic cancer. These studies are coordinated by the Department of Health Policy Analysis at Harvard University.
Conclusions The importance of technology assessments by professional organizations such as the American College of Radiology, American College of Cardiology, American College of Physicians, and AMA cannot be overemphasized. Organizations such as these have access to the practice environment and the clinical judgment required to evaluate the effectiveness and appropriateness of particular technologies in a clinical setting. This judgment is crucial to continued development and use of medical technologies in an efficacious and costconscious manner for improvement of care and well-being of patients. As Kent and Larson [24] have emphasized, the effectiveness oftechnology assessment efforts often can be enhanced through multidisciplinary efforts involving physicians from several specialties. If an effective counterbalance to efforts to control costs through restrictions on coverage and reimbursement of medical technologies is not developed, further limitations on the access of patients to quality health care can be anticipated. In providing this counterbalance, medical organizations can help physicians advocate the rights of patients to continued access to the quality of health care that they need and deserve. No purpose is closer than that of
The reader’s
attenon
is directed
to thecommentaryon
ASSESSMENT
651
advocacy of patients organizations.
to the true role of professional
medical
REFERENCES 1 . Aelman AS. Assessment and accountability: the third revolution in medical care. N EngIJ Med 1989;319:1220-1222 2. Schwartz WB. The inevitable failure of current cost-containment strategies. JAMA 1987:257:220-224 3. Fineberg HV. Effects of clinical evaluation on the diffusion of medical technology. In: Institute of Medicine. Assessing medical technologies.
Washington,
DC: National Academy Press, 1985:176-210
4. Ellwood PM. Outcomes management: a technology of patient experience. N EnglJ Med 1980:318:1549-1556 5. Roper WG, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care: an initiative to evaluate and improve medical practice. N EngI J Med 1988;319:1197-1202 6. Institute of Medicine. Assessing
medical technologies.
Washington.
DC:
National Academy Press, 1985:1 7. Melamed MA, Flehinger BJ, Zaman MB, Heelan AT, Hallerman ET, Martin N. Detection of true pathologic stage 1 lung cancer in the screening program and the effect on survival. cancer 1981;47(suppl.j:1182-1187 8. Wooner LB, Fontana AS, Sanderson DA, et al. Mayo lung project: evaluation of lung cancer screening through December 1979. Mayo Olin Proc 1981;56:544-555 9. Shapiro S. Evidence on screening for breast cancer from a randomized trial. Cancer 1977:39:2722-2782 10. Eddy DM, Hasseiblad V, McGivney WT. Hendee WA. The value of mammography screening in women under 50 years. JAMA 1988:259:15121519 1 1 . Metz CE. Basic principles of ROC analysis. Semin NucI Med 1978;7: 283-298 12. Wolf GL. Poor quality of clinical research in radiology: another indictment. Radiology 1989;170:31 1-312 13. Cooper LS, Chalmers TC, McCarty M, Berrier J, Sacks HS. The poor quality of early evaluations of magnetic resonance imaging. JAMA 1988;259:3277-3280 14. OffIce of Technology Assessment. Strategies for medical technology assessment. Washington. DC: U.S. Government Printing Office, 1982 15. Louis TA, Fineberg H, Mosteller F. Findings for public health from metan-
alysis. In: Annual review: Reviewslnc., 1986 16. Goodman Assessing
public
health,
1985.
C. Quantitative synthesis methods. medical technologies. Washington.
Palo
Alto,
In: Institute DC:
National
CA:
Annual
of Medicine. Academy
Press, 1985:125-126 1 7. Dalkey NC. The DELPHI method: An experimental study of group opinion. Santa Monica, CA: Rand Corp., 1969 18. Glenn KJ. Assessing medical technology. Medicine and Health Perspectives. Washington. DC: Medicine and Health. June 5, 1989 19. Blue Cross/Blue Shield. Technology evaluation criteria. Chicago: Blue Cross and Blue Shield Association, 1987 20. McGivney WT, Hendee WR. Regulation, coverage and reimbursement of medical technologies. Int J Radiat Once! Biol Phys (in press) 21 . American College of Physicians. clinical Efficacy Assessment Project (CEAPj: Fact sheet, procedural manual and final report. Philadelphia: American College of Physicians, 1983 22. American College of Cardiology. Statements of charge: Joint ACC/AHA Task Force on Assessment of Cardiovascular Procedures (and Subcommittees), Cardiovascular Procedures committee, and Cardiovascular Norms Committee. Bethesda, MD: American College of Cardiology, 1985 23. American Medical Association. Drug evaluations, 6th ad. Chicago: AMA,
1988 24.
Kent DL, Larson
EP. Magnetic
Is clinical efficacy
established
resonance
imaging
of the brain and spine. Ann Intern Med
after the first decade?
1988:108:402-424
thisartkIe,whichappears
onthefoflowuigpages.
]
