•
•
Diagnostic Radiology
Use of Excretory Urogram Information in the Solution of the Renal Cyst/Tumor/Cortical Nodule Problem 1 John R. Thornbury, M.D., and Dennis G. Fryback, Ph.D. A decision analysis was made of the choice of angiography or needle aspiration to solve the cyst/tumor/cortical nodule diagnostic problem. With a test panel of 43 proved cases, the urogram diagnoses and decisions made at the time of actual patient management were compared with those made by a radiologist viewing the test panel as unknowns. The test panel radiologist based his recommendations on his subjective probability estimates of the three diagnoses in each case. Results indicate that the use of subjective probabilities increases the frequency of specific diagnoses and decreases inappropriate choice of angiography or aspiration. INDEX TERMS: Angiography, indications. Biopsies. Decision Theory. Kidney Neoplasms, diagnosis. Kidneys, cysts
Radiology 118:575-578, March 1976
• Table I: Raw Data Final Diagnosis Distribution (Proved and Unproved: 113 Consecutive Urograms)
the day-to-day practice of medicine, the physician makes a sequence of decisions to manage his patients. At each point in this process, he can select from a number of alternative actions that may produce different results, depending upon the state of the patient. The true state of the patient is directly related to the physician's certainty about the diagnosis. Diagnosis is further complicated by the fact that each symptom or sign cluster has several possibilities. To reduce their number and increase the certainty about one diagnosis, differential tests are selected. The value of this information consists in changing the probabilities that:
I
N
(a) (b)
(c)
Benign Simple Cyst New case Previously diagnosed cyst Malignant Tumor Hypernephroma Lymphoma Adrenal primary neoplasm Germ cell neoplasm Normal Kidney Benign cortical nodule Normal variation Miscellaneous Renal sinus lipomatosis Mass, type unknown Localized dysplasia Hydronephrosis Compensatory hypertrophy Postoperative hematoma Renal artery aneurysm
One of the possible diagnoses is the true diagnosis One or more of his possible diagnoses is very unlikely to be the true diagnosis ("rule out" approach) A diagnosis he has not even considered is the true diagnosis.
61 (54%) 49 12 22 (20%) 13 7
1 1 15 (13%)
5 10 15 (13%)
5 3 2 2 1
1 1
Total cases
113 (100%)
Table II: Final Diagnosis Distribution Test Panel (Proved Cases: 43 Urograms) Benign simple cyst Hyperneph roma Cortical nodule Normal variation
There is a growing body of literature on the use of decision theory as a formal tool in medicine (1-6). A good example is the use of excretory urography to decide whether to do needle aspiration or selective renal angiography to solve the cyst/tumor/cortical nodule problem. We have studied this problem from the standpoint of the. radiologist interpreting the urogram and making diagnostic assessments, which are the basis for recommending renal angiography or needle aspiration as the next step in the diagnostic process. In this study we shall compare the decisions made by radiologists familiar with decision theory concepts and those who are not.
Total cases
3 3
(67%) (19%) (7%) (7%)
43
(100%)
29
8
MATERIALS AND METHODS
In the period 1 September 1970 through 31 December 1971, 5,168 excretory urograms were obtained at University Hospital. In 113 of these, the question of a mass lesion in the kidney was raised. TABLE I shows the raw data (proved and unproved cases) diagnosis distribution over the 113 cases. Of the 113, 48 could be
1 From the Department of Radiology, University Hospital (J. R. T.), and the Department of Psychology (D. G. F.), University of Michigan, Ann Arbor, Mich. Presented at the Sixtieth Scientific Assembly and Annual Meeting of the Radiological Society of North America, Chicago, 111., Dec. 1-6, 1974. shan
575
JOHN R. THORNBURY AND DENNIS G. FRYBACK
576
Table III:
Diagnostic Variables
Table IV:
Caliceal configuration *Shape of lesion * Radiodensity /Iucency of lesion "Beaking" present? *Does lesion bulge beyond renal margin? Maximum diameter of lesion Calcification associated with lesion? Discreteness of lesion/parenchyma interface Homogeneity of lesion Thickness of lesion wall Caliceal appearance time (involved kidney) Presence of collateral vessels? Is lesion para pelvic in location? Effect of prior urogram information Effect of clinical information
Pr (TuPr (Cyst) mor)
* Three most discriminating variables as indicated by forward stepwise regression analysis.
identified as "proved cases" upon chart review. These 48 cases also were in the category of unilateral radiologic renal abnormalities in which cyst/tumor/cortical nodule was the diagnostic problem. Criteria for the designation of "proved cases" was as follows: Benign Cyst:
Needle aspiration (including radiography) and cytology resulting in clear yellow fluid, smooth-walled cyst lumen, and negative cytology for malignancy (2 cysts were explored surgically). Tumor: Autopsy or surgical gross pathologic and microscopic examination indicating renal malignancy. Normal: (Including cortical nodule)-normal bilateral selective renal angiogram with normal clinical follow-up observation (regarding possibility of cyst or tumor). From the above proved cases, 39 were selected for the decision analysis study. Since this 39-case sample did not cover a wide enough spectrum of the radiologic findings for the three diagnoses, 4 proved cases (one hypernephroma and three benign simple cysts) were added from the University Hospital experience outside the 1970-1971 study period. Thus, the test film panel totalled 43 cases with diagnosis distribution as shown in TABLE II. To evaluate the effect of use and nonuse of decision theory the actual procedure decisions indicated from
Table V: Urogram Diagnosis (J.R.T.)
25
Total cases
29
3
3 5
8
Examples of Subjective Probability Estimates by J.R.T.
Pr (Normal)
Recommended Next Procedure
True Diagnosis
.98 .90 .60 .495 .30
.02 .10 .40 .495 .70
.00 .00 .00 .01 .00
Needle Aspiration (NA) NA NA Arteriogram (ART) ART
Cyst Cyst Cyst Cyst Cyst
.00 .15 .30 .97
.999 .85 .70 .03
.00 .00 .00 .00
ART ART ART NA
Tumor Tumor Tumor Tumor
.00
.002
.998
ART
Cortical nodule .38 .02 ART Normal .60 variation (Note: Formally, no probabil ity is ever 1.00 or 0.00)
chart review were compared with those made by the radiologist (J. R. T.) on the study team viewing the test panel cases as unknowns. The cases and films were selected by the decision theory expert (D. G. F.) from the study team so that the radiologist had no recent prior knowledge of the clinical or radiologic information in the test cases. He had seen about half the cases at the time the urograms were obtained 3-4 years before. The decision analysis approach was carried out in the following manner. The radiologist was instructed to interpret the urograms as if they occurred among the usual daily gamut of case material in the uroradiology suite. He was told that the test panel included cysts, tumors, cortical nodules, and normal kidneys. He interpreted the urograms evaluating the renal abnormality using 15 diagnostic variables (TABLE III). Some variables had multiple subcategories (e.g., range of lucency to density compared to the nephrogram density) while others had yes/no answers (e.g., Is the lesion parapelvic?). A protocol sheet was filled out for each case. This sheet provided the radiologist with a summary of his urogram findings in each case regarding the variables as he then proceeded to estimate his subjective probability distribution over the three diagnoses of cyst, tumor, and cortical nodule (TABLE IV). The numbers he assigned in each case are the answers to the question "If you saw 100 cases exactly like this one, how many would turn out to be cyst, how
Comparison of Urographic Diagnosis and Outcome
With Decision Analysis Proved Outcome ( \ Cortical Cyst Tumor Nodule Normal
Cyst Tumor Cortical nodule Cyst = Tumor
March 1976
1 2
3
1 1 1
3
Urogram Diagnosis (Chart)
Without Decision Analysis Proved Outcome ( \ Cortical Tumor Nodule Normal Cyst
Cyst Tumor Cortical nodule Mass ?Mass Total cases
20 2
1 5
1
3 4
2
1 1
29
8
3
2
3
Vol. 118
USE OF THE UROGRAM IN THE RENAL CYST /TUMOR/CORTICAL NODULE PROBLEM
many tumor, and how many cortical nodule?" Finally, taking into account his probability distribution over the three diagnoses, the radiologist recommended either needle aspiration or angiography as the appropriate next step in diagnostic evaluation. The decision rule followed by the radiologist was: "If the subjective probability of cyst equals or exceeds 0.50, recommend needle aspiration; otherwise recommend arteriography" (7). For tumor or cortical nodule, the next procedure should be angiography. If needle aspiration is inappropriately chosen, angiography should still be performed for proper preoperative evaluation of hypernephroma. For benign simple cyst, renal needle aspiration is the appropriate next procedure. If angiography is inappropriately performed first and indicates an avascular lesion, needle aspiration is then required to determine whether this is a benign simple cyst or a necrotic hypernephroma. It should be noted that the radiologist was actively involved for several years in research on the use of subjective probability estimates as a measure of diagnostic information and is familiar with decision theory general concepts. The decision theory expert recorded all of the data and placed it in the computer data bank for subsequent analysis. Examples of probability estimates are shown in TABLE IV. Data for the decisions from real clinical situations made when the urograms were originally obtained was extracted from chart review by the decision theory expert. The same chart review was also done later by the radiologist to validate the decision theorist's data. This second review was done after the radiologist had completed the decision analysis approach to the test panel of 43 urograms. Data from chart review for each case included: 1. 2. 3. 4. 5. 6. 7. 8.
Clinical setting Indication for urography Results of urography Radiologist's recommendation of renal angiography or aspiration Whether angiography was performed and results Whether aspiration was performed and results Clinical follow-up including urography Final diagnosis and basis for proof.
The costs of angiography and needle aspiration were calculated for University Hospital: Renal angiography (Performed on an inpatient basis requiring two days' hospitalization) Radiology charge 2 days hospitalization @$110 Total cost Renal needle aspiration (Performed on an outpatient basis) Procedure: total cost
$225 $220 $445
$ 89
Table V I:
577
Diagnostic Radiology
Procedures Required for Definitive Diagnosis
,
With Decision Analysis
Without Decision Analysis
A. Cyst (29) Aspiration @ $89 Arteriogram @ $445
(29 $258 (6) $2670
(27*) $2403 (13) $5785
Total cost
$5251
$8188
Arteriogram
(3) $267 (8) $3560
(0) $0 (8) $3560
Total cost
$3827
$3560
C. Normal or Cortical Nodule (6) Aspiration (0) $0 Arteriogram (6) $2670
(0) $0 (6) $2670
True Diagnosis
B. Tumor (8) Aspiration
Total cost Grand total cost
1
$2670
$2670
$11,748
$15,418
*2 Additional cysts were removed at surgery.
RESULTS Comparison of urogram diagnoses and the outcome with and without decision analysis are shown in TABLE V. Using subjective probabilities, (VA) the radiologist was more prone to make a specific most likely diagnosis more frequently than radiologists who did not use this approach (VB). In the latter group, the nondiscriminatory diagnosis of "mass" or "suspect mass" contained 13 cases, 12 of which the radiologist using subjective probabilities assigned to specific diagnosis of cyst, tumor, or cortical nodule. This resulted in more frequent correct specific diagnoses of cyst, but it was achieved at a cost of misdiagnosing one more cyst as a tumor and two more tumors as cysts. Comparing the practical outcome of the urographic diagnosis (i.e., which procedure to perform next for definitive diagnosis), TABLE VI shows that the more discriminating approach (using subjective probabilities) would permit more frequent use of the appropriate procedure to make the definitive diagnosis. Slightly over twice as many angiograms (13 to 6) were used inappropriately to diagnose cyst using the nonprobability approach. Opposed to this is the fact that three needle aspirations were recommended inappropriately to diagnose tumors using the probability approach. This did not occur at all with the nonprobability approach. DISCUSSION
The difference in total dollar cost to diagnose cyst and tumor in this series is $3,670 or about a third less to reach the correct diagnosis using the decision theory approach as opposed to working without it (TABLE VI).
578
JOHN R. THORNBURY AND DENNIS G. FRYBACK
In addition to the diagnostic discrimination capability of the individual radiologist, this outcome is influenced by two other basic facts. There are many more patients encountered with benign cysts than with renal tumors (TABLE I). In addition, the dollar cost of renal angiography is considerably greater than that of renal needle aspiration, since angiography requires hospitalization while cyst aspiration does not. One might argue that one risks puncturing a renal tumor (misdiagnosed on the urogram as a cyst) using the probability approach. The potential risk of spreading tumor as a consequence of needle aspiration is not a real danger in renal tumors. The best refutation of this supposed risk of needle aspiration is the paper by von Schreeb et at. (8). They compared 77 patients with hypernephroma who had prenephrectorny needle aspiration with 73 patients who were not needled prior to nephrectomy. No cases of seeding were reported, and there was no difference in five-year survival between the two groups. Since 1966 we have aspirated six tumors, mistaking them for cystic lesions on urogram evidence. Our experience indicates no seeding or greater risk to these patients as a result of needling. Two other diagnostic tests, nephrotomography and ultrasound, have been advocated following excretory urography to aid in the choice of aspiration versus angiography. As a rule, formal nephrotomography is done following discovery of a renal space-occupying lesion on excretory urography. Nephrotomography has its share of false diagnoses and ambiguous criteria fulfillment (as do all diagnostic tests). It has been our experience that tailored urography can provide equally useful information and avoids the time, expense, and risk of a second separate contrast procedure. Tailored urography means that the dose of contrast agent used takes into account the patient's renal function and the problem under study. A 30-second nephrogram is always obtained and laminography is always used when the early radiographs raise the question of a cyst, tumor, or cortical nodule. When necessary, contrast material is reinjected before laminography. The diagnostic criteria we use are adapted from the formal nephrotomography literature (TABLE III). Ultrasound as another test to aid in the decision of aspiration versus angiography was not considered in our experiment. Our probability estimate data suggest that there would be no way to predict cases in which ultrasound would avoid inappropriate choice of aspiration or angiography. Thus ultrasound would have had to be done on all 43 patients to avoid the inappropriate
March 1976
choice of angiography for 6 cysts and aspiration of 3 tumors. At our institution, not enough experience has accumulated in the use of ultrasound in the cyst/tumor/ cortical nodule problem to assess its accuracy in differentiating cystic from solid lesions in this specific setting. A more complete decision analysis of the choice between needle aspiration and angiography considers not only the differential dollar cost of the two procedures, but other, nonmonetary costs as well such as relative risks of complications, discomfort experienced by the patient, and the amount of time lost from normal activities. We have completed a decision analysis for this problem including nonmonetary factors and have studied the use of this analysis in conjunction with probability estimates supplied by radiologists having varying degrees of experience with the cyst/tumor/cortical nodule diagnosis. The results are similar to those discussed here and will be reported in a forthcoming paper. From the results of this experiment, we recommend that radiologists explore the usefulness of making recommendations for needle aspiration or renal angiography based on subjective probability estimates. However, it must be constantly kept in mind that the probability estimates are only as valuable as the radiologist's accuracy. The old adage of "garbage in, garbage out" is as appropriate for clinical applications of decision theory as it is for computer applications in real life. Department of Radiology University Hospital Ann Arbor, Michigan 48104 REFERENCES 1. Bunch WH, Andrew GM: Use of decision theory in treatment selection. Clin Orthop 80:39-52, Oct 1971 2. Ginsberg AS, Offensend F: An application of decision theory to a medical diagnOsis-treatment problem. IEEE Trans SSC-4:355362, Sep 1968 3. Gorry GA, Kassirer JP, Essig A, et al: Decision analysis as a basis for computer-aided management of acute renal failure. Am J Med 55:473-484, Oct 1973 4. Gustafson DH, Kestly JJ, Greist JH, et al: Initial evaluation of a subjective Bayesian diagnostic system. Health Serv Res 6:204213, Fall 1971 5. Lusted LB: Introduction to Medical Decision Making. Springfield, 111., Thomas, 1968 6. Schwartz WB, Gorry GA, Kassirer JP,et al: Decision analysisand clinical judgment. Am J Med 55:459-472, Oct 1973 7 Thornbury JR: Needle aspiration of avascular renal lesions. Radiology 105:299-302, Nov 1972 8. Schreeb T von, Arner 0, Skovsted G, et al: Renal adenocarcinoma. Isthere a risk of spreading tumour cells in diagnostic puncture? Scand J Urol Nephrol 1:270-276, 1967
•
Diagnostic Radiology
Use of Excretory Urogram Information in the Solution of the Renal Cyst/Tumor/Cortical Nodule Problem 1 John R. Thornbury, M.D., and Dennis G. Fryback, Ph.D. A decision analysis was made of the choice of angiography or needle aspiration to solve the cyst/tumor/cortical nodule diagnostic problem. With a test panel of 43 proved cases, the urogram diagnoses and decisions made at the time of actual patient management were compared with those made by a radiologist viewing the test panel as unknowns. The test panel radiologist based his recommendations on his subjective probability estimates of the three diagnoses in each case. Results indicate that the use of subjective probabilities increases the frequency of specific diagnoses and decreases inappropriate choice of angiography or aspiration. INDEX TERMS: Angiography, indications. Biopsies. Decision Theory. Kidney Neoplasms, diagnosis. Kidneys, cysts
Radiology 118:575-578, March 1976
• Table I: Raw Data Final Diagnosis Distribution (Proved and Unproved: 113 Consecutive Urograms)
the day-to-day practice of medicine, the physician makes a sequence of decisions to manage his patients. At each point in this process, he can select from a number of alternative actions that may produce different results, depending upon the state of the patient. The true state of the patient is directly related to the physician's certainty about the diagnosis. Diagnosis is further complicated by the fact that each symptom or sign cluster has several possibilities. To reduce their number and increase the certainty about one diagnosis, differential tests are selected. The value of this information consists in changing the probabilities that:
I
N
(a) (b)
(c)
Benign Simple Cyst New case Previously diagnosed cyst Malignant Tumor Hypernephroma Lymphoma Adrenal primary neoplasm Germ cell neoplasm Normal Kidney Benign cortical nodule Normal variation Miscellaneous Renal sinus lipomatosis Mass, type unknown Localized dysplasia Hydronephrosis Compensatory hypertrophy Postoperative hematoma Renal artery aneurysm
One of the possible diagnoses is the true diagnosis One or more of his possible diagnoses is very unlikely to be the true diagnosis ("rule out" approach) A diagnosis he has not even considered is the true diagnosis.
61 (54%) 49 12 22 (20%) 13 7
1 1 15 (13%)
5 10 15 (13%)
5 3 2 2 1
1 1
Total cases
113 (100%)
Table II: Final Diagnosis Distribution Test Panel (Proved Cases: 43 Urograms) Benign simple cyst Hyperneph roma Cortical nodule Normal variation
There is a growing body of literature on the use of decision theory as a formal tool in medicine (1-6). A good example is the use of excretory urography to decide whether to do needle aspiration or selective renal angiography to solve the cyst/tumor/cortical nodule problem. We have studied this problem from the standpoint of the. radiologist interpreting the urogram and making diagnostic assessments, which are the basis for recommending renal angiography or needle aspiration as the next step in the diagnostic process. In this study we shall compare the decisions made by radiologists familiar with decision theory concepts and those who are not.
Total cases
3 3
(67%) (19%) (7%) (7%)
43
(100%)
29
8
MATERIALS AND METHODS
In the period 1 September 1970 through 31 December 1971, 5,168 excretory urograms were obtained at University Hospital. In 113 of these, the question of a mass lesion in the kidney was raised. TABLE I shows the raw data (proved and unproved cases) diagnosis distribution over the 113 cases. Of the 113, 48 could be
1 From the Department of Radiology, University Hospital (J. R. T.), and the Department of Psychology (D. G. F.), University of Michigan, Ann Arbor, Mich. Presented at the Sixtieth Scientific Assembly and Annual Meeting of the Radiological Society of North America, Chicago, 111., Dec. 1-6, 1974. shan
575
JOHN R. THORNBURY AND DENNIS G. FRYBACK
576
Table III:
Diagnostic Variables
Table IV:
Caliceal configuration *Shape of lesion * Radiodensity /Iucency of lesion "Beaking" present? *Does lesion bulge beyond renal margin? Maximum diameter of lesion Calcification associated with lesion? Discreteness of lesion/parenchyma interface Homogeneity of lesion Thickness of lesion wall Caliceal appearance time (involved kidney) Presence of collateral vessels? Is lesion para pelvic in location? Effect of prior urogram information Effect of clinical information
Pr (TuPr (Cyst) mor)
* Three most discriminating variables as indicated by forward stepwise regression analysis.
identified as "proved cases" upon chart review. These 48 cases also were in the category of unilateral radiologic renal abnormalities in which cyst/tumor/cortical nodule was the diagnostic problem. Criteria for the designation of "proved cases" was as follows: Benign Cyst:
Needle aspiration (including radiography) and cytology resulting in clear yellow fluid, smooth-walled cyst lumen, and negative cytology for malignancy (2 cysts were explored surgically). Tumor: Autopsy or surgical gross pathologic and microscopic examination indicating renal malignancy. Normal: (Including cortical nodule)-normal bilateral selective renal angiogram with normal clinical follow-up observation (regarding possibility of cyst or tumor). From the above proved cases, 39 were selected for the decision analysis study. Since this 39-case sample did not cover a wide enough spectrum of the radiologic findings for the three diagnoses, 4 proved cases (one hypernephroma and three benign simple cysts) were added from the University Hospital experience outside the 1970-1971 study period. Thus, the test film panel totalled 43 cases with diagnosis distribution as shown in TABLE II. To evaluate the effect of use and nonuse of decision theory the actual procedure decisions indicated from
Table V: Urogram Diagnosis (J.R.T.)
25
Total cases
29
3
3 5
8
Examples of Subjective Probability Estimates by J.R.T.
Pr (Normal)
Recommended Next Procedure
True Diagnosis
.98 .90 .60 .495 .30
.02 .10 .40 .495 .70
.00 .00 .00 .01 .00
Needle Aspiration (NA) NA NA Arteriogram (ART) ART
Cyst Cyst Cyst Cyst Cyst
.00 .15 .30 .97
.999 .85 .70 .03
.00 .00 .00 .00
ART ART ART NA
Tumor Tumor Tumor Tumor
.00
.002
.998
ART
Cortical nodule .38 .02 ART Normal .60 variation (Note: Formally, no probabil ity is ever 1.00 or 0.00)
chart review were compared with those made by the radiologist (J. R. T.) on the study team viewing the test panel cases as unknowns. The cases and films were selected by the decision theory expert (D. G. F.) from the study team so that the radiologist had no recent prior knowledge of the clinical or radiologic information in the test cases. He had seen about half the cases at the time the urograms were obtained 3-4 years before. The decision analysis approach was carried out in the following manner. The radiologist was instructed to interpret the urograms as if they occurred among the usual daily gamut of case material in the uroradiology suite. He was told that the test panel included cysts, tumors, cortical nodules, and normal kidneys. He interpreted the urograms evaluating the renal abnormality using 15 diagnostic variables (TABLE III). Some variables had multiple subcategories (e.g., range of lucency to density compared to the nephrogram density) while others had yes/no answers (e.g., Is the lesion parapelvic?). A protocol sheet was filled out for each case. This sheet provided the radiologist with a summary of his urogram findings in each case regarding the variables as he then proceeded to estimate his subjective probability distribution over the three diagnoses of cyst, tumor, and cortical nodule (TABLE IV). The numbers he assigned in each case are the answers to the question "If you saw 100 cases exactly like this one, how many would turn out to be cyst, how
Comparison of Urographic Diagnosis and Outcome
With Decision Analysis Proved Outcome ( \ Cortical Cyst Tumor Nodule Normal
Cyst Tumor Cortical nodule Cyst = Tumor
March 1976
1 2
3
1 1 1
3
Urogram Diagnosis (Chart)
Without Decision Analysis Proved Outcome ( \ Cortical Tumor Nodule Normal Cyst
Cyst Tumor Cortical nodule Mass ?Mass Total cases
20 2
1 5
1
3 4
2
1 1
29
8
3
2
3
Vol. 118
USE OF THE UROGRAM IN THE RENAL CYST /TUMOR/CORTICAL NODULE PROBLEM
many tumor, and how many cortical nodule?" Finally, taking into account his probability distribution over the three diagnoses, the radiologist recommended either needle aspiration or angiography as the appropriate next step in diagnostic evaluation. The decision rule followed by the radiologist was: "If the subjective probability of cyst equals or exceeds 0.50, recommend needle aspiration; otherwise recommend arteriography" (7). For tumor or cortical nodule, the next procedure should be angiography. If needle aspiration is inappropriately chosen, angiography should still be performed for proper preoperative evaluation of hypernephroma. For benign simple cyst, renal needle aspiration is the appropriate next procedure. If angiography is inappropriately performed first and indicates an avascular lesion, needle aspiration is then required to determine whether this is a benign simple cyst or a necrotic hypernephroma. It should be noted that the radiologist was actively involved for several years in research on the use of subjective probability estimates as a measure of diagnostic information and is familiar with decision theory general concepts. The decision theory expert recorded all of the data and placed it in the computer data bank for subsequent analysis. Examples of probability estimates are shown in TABLE IV. Data for the decisions from real clinical situations made when the urograms were originally obtained was extracted from chart review by the decision theory expert. The same chart review was also done later by the radiologist to validate the decision theorist's data. This second review was done after the radiologist had completed the decision analysis approach to the test panel of 43 urograms. Data from chart review for each case included: 1. 2. 3. 4. 5. 6. 7. 8.
Clinical setting Indication for urography Results of urography Radiologist's recommendation of renal angiography or aspiration Whether angiography was performed and results Whether aspiration was performed and results Clinical follow-up including urography Final diagnosis and basis for proof.
The costs of angiography and needle aspiration were calculated for University Hospital: Renal angiography (Performed on an inpatient basis requiring two days' hospitalization) Radiology charge 2 days hospitalization @$110 Total cost Renal needle aspiration (Performed on an outpatient basis) Procedure: total cost
$225 $220 $445
$ 89
Table V I:
577
Diagnostic Radiology
Procedures Required for Definitive Diagnosis
,
With Decision Analysis
Without Decision Analysis
A. Cyst (29) Aspiration @ $89 Arteriogram @ $445
(29 $258 (6) $2670
(27*) $2403 (13) $5785
Total cost
$5251
$8188
Arteriogram
(3) $267 (8) $3560
(0) $0 (8) $3560
Total cost
$3827
$3560
C. Normal or Cortical Nodule (6) Aspiration (0) $0 Arteriogram (6) $2670
(0) $0 (6) $2670
True Diagnosis
B. Tumor (8) Aspiration
Total cost Grand total cost
1
$2670
$2670
$11,748
$15,418
*2 Additional cysts were removed at surgery.
RESULTS Comparison of urogram diagnoses and the outcome with and without decision analysis are shown in TABLE V. Using subjective probabilities, (VA) the radiologist was more prone to make a specific most likely diagnosis more frequently than radiologists who did not use this approach (VB). In the latter group, the nondiscriminatory diagnosis of "mass" or "suspect mass" contained 13 cases, 12 of which the radiologist using subjective probabilities assigned to specific diagnosis of cyst, tumor, or cortical nodule. This resulted in more frequent correct specific diagnoses of cyst, but it was achieved at a cost of misdiagnosing one more cyst as a tumor and two more tumors as cysts. Comparing the practical outcome of the urographic diagnosis (i.e., which procedure to perform next for definitive diagnosis), TABLE VI shows that the more discriminating approach (using subjective probabilities) would permit more frequent use of the appropriate procedure to make the definitive diagnosis. Slightly over twice as many angiograms (13 to 6) were used inappropriately to diagnose cyst using the nonprobability approach. Opposed to this is the fact that three needle aspirations were recommended inappropriately to diagnose tumors using the probability approach. This did not occur at all with the nonprobability approach. DISCUSSION
The difference in total dollar cost to diagnose cyst and tumor in this series is $3,670 or about a third less to reach the correct diagnosis using the decision theory approach as opposed to working without it (TABLE VI).
578
JOHN R. THORNBURY AND DENNIS G. FRYBACK
In addition to the diagnostic discrimination capability of the individual radiologist, this outcome is influenced by two other basic facts. There are many more patients encountered with benign cysts than with renal tumors (TABLE I). In addition, the dollar cost of renal angiography is considerably greater than that of renal needle aspiration, since angiography requires hospitalization while cyst aspiration does not. One might argue that one risks puncturing a renal tumor (misdiagnosed on the urogram as a cyst) using the probability approach. The potential risk of spreading tumor as a consequence of needle aspiration is not a real danger in renal tumors. The best refutation of this supposed risk of needle aspiration is the paper by von Schreeb et at. (8). They compared 77 patients with hypernephroma who had prenephrectorny needle aspiration with 73 patients who were not needled prior to nephrectomy. No cases of seeding were reported, and there was no difference in five-year survival between the two groups. Since 1966 we have aspirated six tumors, mistaking them for cystic lesions on urogram evidence. Our experience indicates no seeding or greater risk to these patients as a result of needling. Two other diagnostic tests, nephrotomography and ultrasound, have been advocated following excretory urography to aid in the choice of aspiration versus angiography. As a rule, formal nephrotomography is done following discovery of a renal space-occupying lesion on excretory urography. Nephrotomography has its share of false diagnoses and ambiguous criteria fulfillment (as do all diagnostic tests). It has been our experience that tailored urography can provide equally useful information and avoids the time, expense, and risk of a second separate contrast procedure. Tailored urography means that the dose of contrast agent used takes into account the patient's renal function and the problem under study. A 30-second nephrogram is always obtained and laminography is always used when the early radiographs raise the question of a cyst, tumor, or cortical nodule. When necessary, contrast material is reinjected before laminography. The diagnostic criteria we use are adapted from the formal nephrotomography literature (TABLE III). Ultrasound as another test to aid in the decision of aspiration versus angiography was not considered in our experiment. Our probability estimate data suggest that there would be no way to predict cases in which ultrasound would avoid inappropriate choice of aspiration or angiography. Thus ultrasound would have had to be done on all 43 patients to avoid the inappropriate
March 1976
choice of angiography for 6 cysts and aspiration of 3 tumors. At our institution, not enough experience has accumulated in the use of ultrasound in the cyst/tumor/ cortical nodule problem to assess its accuracy in differentiating cystic from solid lesions in this specific setting. A more complete decision analysis of the choice between needle aspiration and angiography considers not only the differential dollar cost of the two procedures, but other, nonmonetary costs as well such as relative risks of complications, discomfort experienced by the patient, and the amount of time lost from normal activities. We have completed a decision analysis for this problem including nonmonetary factors and have studied the use of this analysis in conjunction with probability estimates supplied by radiologists having varying degrees of experience with the cyst/tumor/cortical nodule diagnosis. The results are similar to those discussed here and will be reported in a forthcoming paper. From the results of this experiment, we recommend that radiologists explore the usefulness of making recommendations for needle aspiration or renal angiography based on subjective probability estimates. However, it must be constantly kept in mind that the probability estimates are only as valuable as the radiologist's accuracy. The old adage of "garbage in, garbage out" is as appropriate for clinical applications of decision theory as it is for computer applications in real life. Department of Radiology University Hospital Ann Arbor, Michigan 48104 REFERENCES 1. Bunch WH, Andrew GM: Use of decision theory in treatment selection. Clin Orthop 80:39-52, Oct 1971 2. Ginsberg AS, Offensend F: An application of decision theory to a medical diagnOsis-treatment problem. IEEE Trans SSC-4:355362, Sep 1968 3. Gorry GA, Kassirer JP, Essig A, et al: Decision analysis as a basis for computer-aided management of acute renal failure. Am J Med 55:473-484, Oct 1973 4. Gustafson DH, Kestly JJ, Greist JH, et al: Initial evaluation of a subjective Bayesian diagnostic system. Health Serv Res 6:204213, Fall 1971 5. Lusted LB: Introduction to Medical Decision Making. Springfield, 111., Thomas, 1968 6. Schwartz WB, Gorry GA, Kassirer JP,et al: Decision analysisand clinical judgment. Am J Med 55:459-472, Oct 1973 7 Thornbury JR: Needle aspiration of avascular renal lesions. Radiology 105:299-302, Nov 1972 8. Schreeb T von, Arner 0, Skovsted G, et al: Renal adenocarcinoma. Isthere a risk of spreading tumour cells in diagnostic puncture? Scand J Urol Nephrol 1:270-276, 1967
