ORIGINAL C O N T R I B U T I O N
clinical variables neural network
Analysis of the Clinical Variables Driving Decision in an Artificial Neural Network Trained to Identify the Presence of Myocardial Infarction
From the Departments of Emergency Medicine and Medicine, University of California, San Diego Medical Center.
William G Baxt, MD
Received for publication March 30, 1992. Revision received July 14, 1992. Accepted for publication July 20, 1992.
Study objective: To determine which clinical variables drive the output of an artificial neural network trained to identify the presence of myocardial infarction. Design: Partial output analysis. Setting:
Tertiary university teaching center.
Participants: Seven hundred six patients more than 18 years old presenting with anterior chest pain. Measurements: Differential network output analysis.
Main results: A methodology was developed as the first step in measuring the impact input clinical variables have on the output (diagnosis) of an artificial neural network trained to identify the presence of acute myocardial infarction. The methodology revealed that the network used the presence of ECG findings, as well as the presence of rales, syncope, jugular venous distension, response to trinitroglycerin, and nausea and vomiting, as major predictive sources. Although this first-step analysis studied individual variables, it must be stated that the network comes to clinical closure based on the settings of all variables in a pattern and that the impact of a single variable cannot be taken out of the context of a pattern. Conclusion: An artificial neural network trained to recognize the presence of myocardial infarction appears to place diagnostic importance on clinical variables that have not been shown previously to be highly predictive for infarction. [Baxt WG: Analysis of the clinical variables driving decision in an artificial neural network trained to identify the presence of myocardial infarction. Ann EmergMed December1992;21:1439-1444.]
DECEMBER1992 21:12 ANNALS 0F EMERGENCYMEDICINE
1439/35
NEURAL NETWORK Baxt
INTRODUCTION The artificial neural network has become a powerful paradigm for the recognition of complex patterns.i-3 Such networks have been shown capable of identifying relationships in input data that are not a p p a r e n t to human analysis, a This technology has recently been applied to a b r o a d n u m b e r of chnical settings, s-15 One such apphcation has been to the identification of the presence of acute myocardial infarction in patients presenting to the emergency d e p a r t m e n t with anterior chest pain. 16-19 In this instance, the network has been shown to be more accurate than either physicians or other computer-based clinical aids. 16,17,19 Because such networks are able to identify relationships between input variables that are not a p p a r e n t to human analysis, the artificial neural network could be used potentially as a technique to identify heretofore u n a p p r e c i a t e d relationships between such inputs. One of the m a j o r drawbacks of artificial neural network technologies has been the inability to identify easily how they derive their output. This emanates from the nonhnear manner in which the network draws relationships between inputted information. This function takes place in the computation within the hidden units of the network. Unraveling the mathematical relationships that the hidden units establish between input data, especially when there is more than one layer of hidden units in the network, has proven difficult to accomplish. One indirect way that this can be a p p r o a c h e d is by the stepwise p e r t u r b a t i o n of isolated individual input variables coupled with an analysis of the effect this has on network output. In this manner one can determine which input variables are having the greatest effect on network decision. The following reports on the application of such a technique to a network trained to recognize the presence of acute myocardial infarction in patients presenting to the ED with anterior chest pain.
MATERIALS
AND
METHODS
An artificial neural network is a group of interconnected mathematical equations that accept input data and calculate an output based on this input. This structure is a predesigned method for the application of least means nonhnear statistical techniques. The mathematics of this process have been published extensively.2, 3 The network used in this study is illustrated (Figure 1). It consisted of 20 input units, two layers of ten internal or hidden units, and an output unit. In the application of an artificial neural network described here, the inputs are selected from the presenting complaints, history, physical examination, and ECG findings (Table 1) of adult patients presenting to the ED with anterior chest pain. The network is trained so that the output represents the presence or absence of acute myocardial infarction. The strategy used in this study can be appreciated by referring to (left side of Figure 2) the feed-forward process of a highly simplified artificial neural network. The input p a t t e r n in this instance consists of two variables, A and B. Each of these variables is entered into the respective input unit of the network. These values are muhiphed by a number called a weight. The value of the weight is determined by the process used to train the network. 1-3 The products of these multiplications are summed and become the net input of the hidden layer trait. This process is represented by: netpi = ~ w i j a p j + bias i j=0
where netvi equals net input of the unit for p a t t e r n p , w is a weight, a is the input value applied to the unit (ie, A or B), j represents the input or presynaptic units, i represents the first layer hidden unit or postsynaptic unit, and bias is a modifiable weight that is multiplied by an input that is always equal to 1. The net activation of the hidden unit is calculated by placing the net input into a logistic function: 1 a
Figure 1. Diagnostic network: 20 x 10 x 10 x ] network with 20 input units, two layers often units, and one output unit
Input units . . . . .
Hi
Hi
V U L ~ U L UlIIL
36/1440
@@@@@
.~
pt
] + e-netpl
The activation of the hidden unit is multiphed by a third weight. This p r o d u c t becomes the net input of the output unit or generically that of other hidden units in multilayered networks. The activation of the output unit or other units is calculated by use of the above equation in a manner analogous to that of the first layer. Changing the value of input A to A' will change network output (right side of Figure 2). If input A is set to a specified value, the effect that A has on network output can be appreciated. Ideally, the easiest way to study how individual input variables affect network output would be to set all inputs to zero and then individually set each input to a positive value and study the numeric impact this has on network output. Unfortunately, the nonlinear manner in which the network operates makes this impossible. The network is highly context-sensitive. That is, if the value of the same input variable in two different patterns were reversed, the net effect on network output would be different in each pattern. This is illustrated (Table 2).
ANNALS 0FEMERGENCYMEBICINE 21:12 DECEMBER 1992
NEURAL NETWORK Baxt
Two p a t t e r n s , A and B, d e r i v e d f r o m two patients who differ by the value of only one i n p u t are depicted. P a t t e r n s A and B differ only in t h a t the p a t i e n t in A has shortness of b r e a t h and the patient in B has syncope. T h e o u t p u t for each initial p a t t e r n is shown, 4.2 x 10 -5 and 2.6 x 10 -5, respectively. T h e effect on n e t w o r k o u t p u t of m a k i n g the same v a r i a b l e , the p r e s e n c e of nausea a n d vomiting, initially 0, 1 is illustrated in p a t t e r n s A' and B'. In the first case, the difference between the initial n e t w o r k o u t p u t and the o u t p u t generated after the change in the v a r i a b l e of n a u s e a and vomiting t e r m e d the delta o u t p u t (A' - A) is 2.6 x 10 -3. In the second case, the delta o u t p u t (B' - B) is 2.0 x 10 -4, a fivefold difference. The effect of i n d i v i d u a l variables is d e p e n d e n t on the state of all the o t h e r v a r i a b l e s in the p a t t e r n . As a result, individual i n p u t variables must be studied in the context of the whole p a t t e r n , and only t h e i r general overall effect can be a p p r e c i a t e d . In view of this, the effect of changing individual v a r i a b l e s within existing p a t t e r n s was studied. The n e t w o r k was t r a i n e d as r e p o r t e d p r e v i o u s l y 17 on 706 patients 18 years or older who p r e s e n t e d to the E D with a n t e r i o r chest pain. T h e weights d e r i v e d f r o m the t r a i n i n g were used for the d e r i v a t i o n of the p a r t i a l outputs. The patterns d e r i v e d f r o m these 706 patients were used for the analysis. E a c h of the 20 i n p u t v a r i a b l e s was v a r i e d for e a c h pattern. I n each case, only one v a r i a b l e was c h a n g e d at a time so t h a t the effect in the midst of the i n t e r p l a y of all existing variables could be m e a s u r e d . P a t i e n t age was v a r i e d by inputting an a r b i t r a r y age f r o m 0 to 90 years at t e n - y e a r intervals, and sex was set f r o m either female or male to 0; response to t r i n i t r o g l y c e r i n was set f r o m 0 (not used) to 2 (incomplete relief) or f r o m 1 (complete relief) or 2 to 0. Binary variables were changed f r o m 0 (not present) to 1 (present) or 1 to 0. F o u r possible settings can exist: Patients either will h a v e or h a v e not sustained a m y o c a r d i a l i n f a r c tion a n d aside f r o m age, the i n p u t v a r i a b l e will h a v e been
Table 1.
changed f r o m either a positive value to 0 or f r o m 0 to a positive value. As e a c h of the 706 p a t t e r n s was individually i n p u t t e d to the n e t w o r k , the p r o g r a m calculated an o u t p u t based on the intact p a t t e r n ( o u t z ) . One of the 20 clinical i n p u t variables was t h e n c h a n g e d as n o t e d above (ie, A to A'). A second o u t p u t was t h e n calculated b a s e d on the modified p a t t e r n ( o u t 2 ) . The first o u t p u t was t h e n s u b t r a c t e d f r o m the second o u t p u t to yield a delta output3: A out = out 2-
out]
This process was t h e n separately r e p e a t e d for each of the 20 variables with the changes m a d e as n o t e d above. As e a c h p a t t e r n was i n p u t t e d in this m a n n e r , the delta o u t p u t was s u m m e d for each of the f o u r settings (noted above) a n d a delta o u t p u t m e a n was calculated for each variable. The artificial n e u r a l n e t w o r k s i m u l a t o r as well as the code for p a r t i a l o u t p u t analysis was written specifically for this study in C and r u n on a U N I X workstation.
Figure 2. Simplified operation o f artificial neural network with two input units, one hidden unit, and one output unit (2 x 1 x 1). The input pattern consists o f the three variables A, A', and B, where A' is a modified A. Each o f these variables is entered into the respective input unit o f the network. These values are multiplied by a number called a weight (Wt[1], Wt[2]). The products o f these multiplications are summed and become the net input o f the hidden layer unit. A" yields a modified input. The input values are placed in a logistic function that calculates the net activations o f the hidden units. The net activations o f the hidden units are multiplied by a third weight (Wt[3]). These products then become the net input o f the output units. These sums are entered into the same logistic function that calculates the net activation o f the output unit (termed network output) and activation', the output modified by the change in A to A'.
A
B
A'
Input variables
Presenting Data
History
Age *t Past acute Sex* myocardial Left anterior infarction location of pain* Angina* Diabetes* Hypertension* Intensity of pain Radiation of pain Nausea and vomiting* Diaphoresis* Syncope* Shortness of breath* Palpitations* Response to nitroglycerin*
Examination Jugular venous distension* Roles*
ECG Findings 2-ram ST elevation* l-ram ST elevation* ST depression* T-wave inversion* Significant ischemic change*
* Variables actually utilized by network. * Analog ceded.
DECEMBER1992
21:12 ANNALS 0F EMERGENCY MEDICINE
A x Wt(1) = X
B x Wt(2) = X
X + Y = Output Activation
X' + Y = Output' Activation'
1
1
Activation x wt(3) = output Activation
A' x Wt(1) = X'
Activation'x Wt(3) = Output' I
Activation'
Activation - Activation' = Delta Activation
1441/37
NEURAL N E T W O R K Baxt
RESULTS
The overall sensitivity and specificity of the trained network on the unchanged patterns was 97%. The results of the process outlined above arc presented (Tables 3 and 4). The effects appreciated by varying age between 0 to 90 years, ablating the effect of sex, and changing the response to trinitroglycerin are summarized (Table 3). Data are divided into the effects appreciated in either patients who had not and had sustained myocardial infarction and by what manner the variable was changed. The effects of reversing the 17 binary variables are summarized (Table 4). Data are also divided into the effects appreciated in either patients who had not or had sustained a myocardial infarction. The results are further broken down into whether the variable was changed from 0 to 1 or from 1 to 0. The data are also arranged in descending order of the raw mean of each variable. The latter was calculated by taking the mean of the sum of each of the two means infarction and noninfarction calculated from changing the variable from 0 to 1 plus the negative sum of the two means calculated from changing the same variable from 1 to 0. When a variable change from 1 to 0 has a negative effect on network output, this implies that the initial state of that variable favors the diagnosis of myocardial infarction, whereas a positive effect would disfavor the diagnosis. The opposite holds when such a variable is changed from 0 to 1. The raw mean, thus, gives a generalized impression of the overall effect each variable is having on the network output. Table 2. Comparison o f the effect o f changing one variable in two patterns varying in one variable Pattern A
Ablating age has a weak positive effect on output (Table 3), actually favoring the diagnosis of myocardial infarction up to age 60 years and then a weak negative effect disfavoring the diagnosis. Ablating the effect of sex also had a positive effect on network output, disfavoring the diagnosis of myocardial infarction in all cases. Changing the response to trinitroglycerin from 1 (complete relief of pain) or 2 (incomplete relief of pain) to 0 (not used) had a negative effect on output, whereas changing it from 0 to 2 had a positive effect on network output. The effect of changing most variables from 0 to 1 had a positive effect on network output (favoring the diagnosis of myocardial infarction) (Table 4). This impression is reinforced when the same variables are changed from 1 to 0. In this case they have the opposite effect and lower network output (disfavoring the diagnosis of myocardial infarction). Six variables have effects opposite to this trend, and most of these are small, hi general, when variables were changed from 0 to 1, the more marked effects were in patients ~-ithout myocardial infarction because the patients with myocardial infarction already had high outputs resultant from the settings of their other variables. The antithesis was true when variables were changed from I to 0. In this case, the greatest decrement in output was in patients with infarction who had high outputs at the onset. The generalized effect of all the variables is summarized (Tables 3 and 4) by the raw mean of each variable. The variables (Table 4) are arranged in the descending order of the size of the raw mean. Table 3. Impact o f varying input on age, sex, and use o f trinitroglycerin in an artificial neural network trained to detect myocardial infarction AGE (yr)
Variable
A
A'
B
B'
Age
Age Sex Location of pain Response Nausea and vomiting Diaphoresis Syncope Shortness of breath Palpitations History of myocardial infarction History of diabetes mellitus History of hypertension History of angina Jugular venous distension Rales 2-mm ST elevation 1-mm ST elevation ST depression T-wave inversion Ischemic change
0,50 1 1 0 0 1 0 1 0 0
0.50 1 1 0 1 1 0 1 0 0
0.05 1 1 0 0 1 1 0 0 0
0.50 1 1 0 1 1 1 0 0 O
0 10 20 30 40 50 60 70 80 90
0 0 0 0 O 0 0 I 1 0
0 0 0 0 0 0 0 1 1 0
0 0 O 0 0 0 0 1 1 0
0 0 0 0 0 0 0 1 1 0
SEX
Output
38/1442
4.2x10 -~
Patients Without Myocardial Infarction
Pattern B
2.6×1'0 5 2.6×10 5
2.3x10 4
0.013343 0.010191 0.007624 0.005386 0.002594 0.001307 0.000561 -0.000580 -0.001897 -0.002190
Female to 0 Noninfarct Mean Delta 0.0419
Infarct Mean Delta 0.0120
Patients With Myocardial Infarction 0.001492 0.001330 0.000885 0.000720 0.000719 0.000471 -0.000506 -0.001877 -0.005295 -0.009134
Infarct Mean Delta -0.0412
0.007417 0.005761 0.004254 0.003053 0.001656 0.000889 0.000027 -0.001229 -0.003596 -0.005662
Male to 0 Noninfarct Mean Delta 0.0534
TRINITROGLYCERIN 1 or2to 0 Noninfarct Mean Delta -0.0084
Total
Infarct Mean Delta 0.0021
Raw Mean -0.0300
Oto2 Noninfarct Mean Delta 0.5862
Infarct Mean Delta 0.0176
Raw Mean 0.0339
ANNALS 0F EMERGENCY MEDICINE 2I:12
DECEMBER1992
NEURAL N E T W O R K Baxt
The raw means of 19 network variables are summarized (Table 5). The presence of the five ECG findings, rales, syncope, jugular venous distension, response to trinitroglycerin, and nausea and vomiting, all have significant positive effects on the network output, favoring the diagnosis of myocardial infarction (Table 5). The remaining variables have smaller positive or negative effects. DISCUSSION
One of the hopes that has arisen from the clinical application of the artificial neural network is that it could be used to afford greater insights into the relationships between clinical variables that characterize various disease states. This has emanated from the findings that in other applications such networks are able to appreciate relationships between i n p u t data that clearly were not apparent to prior h u m a n or computational analysis. Because the initial studies of the application of the artificial neural network to the diagnosis of myocardial infarction have indicated that it performs substantially better than either physicians or other computerassisted aids, 2°-22 there was some justification to the belief that it could be used to identify such relationships. It was on this basis that this study was undertaken. The results suggest that this may be the caste. This approach has been able to develop a generalized impression as to the implicit effects different variables are having on network output (diagnosis) and can be viewed as a possible first step in the unraveling of,t~e methods by which Table 4. Impact of varying binary input variables of an artificial neural network trained to recognize the presence of myocardial infarction 0-1
1-0
Noninfarct Mean
Infarct Mean
Noninfarct Mean
Infarct Mean
Raw Mean
Delta -0.5719 Rales -0.2022 2-mm ST elevation -0.2005 ST depression -0.0666 1-mm ST elevation -0.0368" Syncope -0.0722 " Jugular venous -0,1193 distension Nausea and -0.0226 vomiting T-wave inversion -0.0214 Diaphoresis -0.0152 Shortness of breath -0.0091 History of 0.0051 diabetes mellitus History of 0.0517 myocardial infarction Palpitations 0.0252 History of 0.0464 hypertension History of angina 0.0528 Location of pain 0.1088
-0.5550 -0.2953 -0.0819 -0.2274 -0.1565 -0.0450 -0.0341
0.8428 0.3271 0.2131 0.1687 0.0560 0.0480 0.0454
0.1372 0.0309 0.0631 0,0404 0.0082 0.0083 0,0083
0.7566 0.2706 0.1809 0.1437 0.0489 0.0418 0.0391
-0.0184
0.0333
0.0106
0.0273
-0.1118 -0.0353 -0.0135 0.0004
0.0225 0.0170 0.0040 -0.0043
0.0004 0.0219 0.0044 -0.0234
0,0265 0.0182 0.0066 -0.0078
0.0196
-0.0134
-0.0365
-0.0250
Variable
0.0002 0.0008
-0.0151 -0.0233
-0.0863 -0.0828
-0.0317 -0.0352
0.0176 0.0128
-0.0311 -0.0665
-0,1282 -0.1320
-0.0491 0.0833
DECEMBER 1992 21:12 ANNALS OF EMERGENCYMEDICINE
artificial networks come to clinical closure. These data must be interpreted with a great deal of caution, and these findings cannot be taken literally. The individual effects of the clinical variables must be kept in the context of the patient and not used as isolated independent predictors of outcome. These results do not suggest the use of any of these variables as part of a superstructure for the development of a differential diagnosis. These data only give insight into this interplay and do not suggest the independent use of these results as inert predictors of events. Furthermore, it must be understood that the use of the term "general" has greater implications than may be apparent. Because of the unique m a n n e r by which the network functions, variables that appear to have major impacts based on this study, in certain variable cluster settings, may have a major impact only in that setting. Finally, it must be stated that the netwoi~k is a reflection of the patient data base on which it is trained. As such, its "knowledge" is only that available from the information that can be accrued from that population. Although 706 patients represent a substantial n u m b e r of such data points, this is a minuscule representation of all such patients. This work must be viewed only as a model, and not until this process has been repeated on a very large n u m b e r of such patients can the significance of variable weighting be comfortably taken to represent any reality. The m a n n e r by which the network uses some input data was unexpected. The most striking finding is that the variable that was found to have the greatest influence on network output, significant ischemic ECG change, had a mean impact of only 0.7566. Because the outputs generated by patterns extracted from patients with acute myocardial infarction have outputs of more than 0.9500, it is clear that no single variable in and of itself has the capability of imparting Table 5. Impact according to raw mean of varying each of the 20 input variables of an artificial neural network trained to recognize the presence of myocardial infarction
Variable Delta Rales 2-ram ST elevation ST depression l-ram ST elevation Syncope Jugular venous distension Trinitroglycerin Nausea and vomiting T-wave inversion Diaphoresis Shortness of breath History of diabetes mellitus History of myocardial infarction Palpitations History of hypertension History of angina Location of pain Sex
Raw Mean 0.7566 0.2706 0.1809 0.1437 0,0489 0.0418 0,0391 0,0339 0.0273 0.0265 0.0182 0.0066 -0.0078 -0.0250 -0.0317 -0,0352 -0.0491 -0.0833 -0.0300
1443/ 39
NEURAL NETWORK Baxt
enough positive impact on network output to cause it to rise to this level. The network must recognize the presence of multiple variables to generate an output that is diagnostic of myocardial infarction (more than 0.9500). The second striking feature of this analysis was that although the effects of the ECG findings were expected, the importance of the findings of the presence of rales, nausea and vomiting, and syncope was surprising. In fact, the presence of tales appears to be more important toward supporting the diagnosis of myocardial infarctions than all the ECG variables, save that of significant iachemic change. Although these nonECG variables have been known to be potential indicators of the presence of infarction, that they are given such strong weighting by the network indicates that these may be far more predictive than heretofore appreciated. F u r t h e r , the impact of age was uniformly weak, and the impact of sex was negative in all cases. It would have been expected that the former would have h a d a positive effect at older ages, whereas the latter would have h a d either a null influence or at least a uniformly weak positive influence. Finally, it was also surprising that the variables of history of hypertension, diabetes melhtus, and angina, which in the past have been thought to be predictive of myocardial infarction, had negative impacts on network output and actually disfavored the diagnosis of myocardial infarction. CONCLUSION
This type of analysis of network processing may be able to elucidate new relationships among clinical variables that characterize disease states. These techniques, as well as others,23, 2a may enable the development of logistic rules from trained networks, which m a y b e used prospectively to both expand the knowledge derived from network processing and improve clinical diagnostic accuracy. REFERENCES 1. Widraw G, Heft ME: 1960Adaptiva Switching Circuits Institute of Radio Engineering Western Electronic Show and Convention, Convention Record, Part 4, p 96-104. 2. Rumelhart DE, Hinton GE, Williams RJ: Learning internal representations by error propagation, in Rumelhart BE, McClelland JL (ads): Parallel Distributed Processing: Explorations in the Microstrueture of Cognition. Cambridge, Massachusetts, MIT Press, 1986, p 318-364. 3. McClelland JL, Rumelhart BE: Training hidden units, in McOlelland JL, Rumelhart BE (ads): Explorations in Paraflel Distributed Processing. Cambridge, Massachusetts, MIT Press, 1988, p 121-160. 4. Weigend AS, Huberman BA, Rumelhart DE: Predicting the Future:A Conneetionist Approach. Stanford PDP Research Group, April 1990. 5. Hudson DL, Cohen ME, Anderson MF: Determination of testing efficacy in carcinoma of the lung using a neural network model. Symposium on CemputerAppficatiens in Medical Care 1988Proceedings: 12thAnnual Symposium, Washington,DC1988;t2:251-255. 6. Smith JW, E~zerhartJE, Dickson WC, et al: Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. Symposium on ComputerAppficatiens in Medical Care 1988Proceedings," 12thAnnual Symposium, Washington, DC 1988;12:261-265.
8. Kaufman J J, Chiabera A, Hatem M, et al: A neural network approach for bone fracture healing assessment. IEEEEng Med Bie11990;9:23-30. 9. Hiraiwa A, Shimohara K, Tekunaga Y: EEG topography recognition by neural networks. IEEEEng Mad Bie11990;9:39-42. 10. Cios KJ, Chen K, Langenderfer RA: Use of neural networks in detecting cardiac diseases from echocardiographic images. IEEEEng Met Bie11990;9:58-60. 11. Marconi L, Scalia F, Ridella S, et al: An application of back propagation to medical diagnosis. Proceedings of the International Joint Conference on Neural Networks, Washington, DC 1989;2:577. 12. Eberhart RC, Dobbins BW, Hutton LV: Neural network paradigm comparisons for appendicitis diagnosis. Proceedings of the Fourth Annual IEEESymposium on Computer-Based Medical Systems 1991;298-304. 13, Mulsant 6H, Servan-Schreiber E: A connectionist approach to the diagnosis of dementia. Symposium on ComputerAppficatiens in Medical Care 1988Proceedings: 12thAnnual Symposium, Washington, DC 1988;12:245-250. 14. Bounds DG, Lloyd PJ, Mathew BG: A comparison of neural network and other pattern recognition approaches to the diagnosis of low back disorders. Neural Networks 1990;3:583-591. 15. Yaon YO, Brobst RW, Bergstresser PR, et al: A desktop neural network for dermatology diagnosis. J Neural Network ComputationSummer 1989;43-52. 16. Baxt WG: Use of an artificial neural network for data analysis in clinical decisionmaking: The diagnosis of acute coronary occlusion. Neural Computation 1991;2:480-489. 17. Baxt WG: Use of an artificial neural network for the diagnosis of myocardial infarction. Ann Intern Med 1991;115:843-848. 18. Baxt WG: Improving the accuracy of an artificial neural network using multiple differently trained networks. Neural Computation 1992;4:772-780. 19. Harrison RF, Marshall SJ, Kennedy RL: The early diagnosis of heart attacks: A neurocamputational approach. Proceedings of the International Joint Conference on Neural Networks, Seattle 1991;1:1-5. 20. Pozen MW, D'Agostino RB, Mitchell JB, et al: The usefulness of a predictive instrument to reduce inappropriate admissions to the coronary care unit. Ann Intern Met 1980;92:238-242. 21. Goldman L, Weinberg M, Waisberg M, et al: A computer-derived protocol to aid in the diagnosis of emergency room patients with acute chest pain. N Engl J Med 1982;307:588-596. 22. Goldman L, Cook EF, Brand BA, et al: A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med 1988;318:797-803. 23. Maclin R, Shavlik JW: Refining algorithms with knowledge-based neural networks: Improving the Chau-Fasman algorithm for protein folding, in Hanson S, Drastal G, Rivest R (eds): Computational Learning Theory and Natural Learning Systems. Cambridge, Massachusetts, MIT Press (in press). 24. Towell GG, Shavlik JW: The extraction of refined rules from knowledge-based neural networks. Machine Learning (in press). The author thanks Drs Halbert White and David Zipser for their help with the technical aspects of this study and Kathleen James for her help in the preparation of this manuscript. Address for reprints: William G Baxt, MD Department of Emergency Medicine UCSD Medical Center 200 West Arbor Drive, #8676 San Diego, California 92013-8676
7. Saito K, Nakano R: Medical diagnostic expert system based on PDP model. Proceedings of the International Joint Conferenc~ on Neural Networks, San Diego 1988;2:255-262.
40/1444
ANNALS 0F EMERGENCY MEDICINE
21:12
DECEMBER1992
clinical variables neural network
Analysis of the Clinical Variables Driving Decision in an Artificial Neural Network Trained to Identify the Presence of Myocardial Infarction
From the Departments of Emergency Medicine and Medicine, University of California, San Diego Medical Center.
William G Baxt, MD
Received for publication March 30, 1992. Revision received July 14, 1992. Accepted for publication July 20, 1992.
Study objective: To determine which clinical variables drive the output of an artificial neural network trained to identify the presence of myocardial infarction. Design: Partial output analysis. Setting:
Tertiary university teaching center.
Participants: Seven hundred six patients more than 18 years old presenting with anterior chest pain. Measurements: Differential network output analysis.
Main results: A methodology was developed as the first step in measuring the impact input clinical variables have on the output (diagnosis) of an artificial neural network trained to identify the presence of acute myocardial infarction. The methodology revealed that the network used the presence of ECG findings, as well as the presence of rales, syncope, jugular venous distension, response to trinitroglycerin, and nausea and vomiting, as major predictive sources. Although this first-step analysis studied individual variables, it must be stated that the network comes to clinical closure based on the settings of all variables in a pattern and that the impact of a single variable cannot be taken out of the context of a pattern. Conclusion: An artificial neural network trained to recognize the presence of myocardial infarction appears to place diagnostic importance on clinical variables that have not been shown previously to be highly predictive for infarction. [Baxt WG: Analysis of the clinical variables driving decision in an artificial neural network trained to identify the presence of myocardial infarction. Ann EmergMed December1992;21:1439-1444.]
DECEMBER1992 21:12 ANNALS 0F EMERGENCYMEDICINE
1439/35
NEURAL NETWORK Baxt
INTRODUCTION The artificial neural network has become a powerful paradigm for the recognition of complex patterns.i-3 Such networks have been shown capable of identifying relationships in input data that are not a p p a r e n t to human analysis, a This technology has recently been applied to a b r o a d n u m b e r of chnical settings, s-15 One such apphcation has been to the identification of the presence of acute myocardial infarction in patients presenting to the emergency d e p a r t m e n t with anterior chest pain. 16-19 In this instance, the network has been shown to be more accurate than either physicians or other computer-based clinical aids. 16,17,19 Because such networks are able to identify relationships between input variables that are not a p p a r e n t to human analysis, the artificial neural network could be used potentially as a technique to identify heretofore u n a p p r e c i a t e d relationships between such inputs. One of the m a j o r drawbacks of artificial neural network technologies has been the inability to identify easily how they derive their output. This emanates from the nonhnear manner in which the network draws relationships between inputted information. This function takes place in the computation within the hidden units of the network. Unraveling the mathematical relationships that the hidden units establish between input data, especially when there is more than one layer of hidden units in the network, has proven difficult to accomplish. One indirect way that this can be a p p r o a c h e d is by the stepwise p e r t u r b a t i o n of isolated individual input variables coupled with an analysis of the effect this has on network output. In this manner one can determine which input variables are having the greatest effect on network decision. The following reports on the application of such a technique to a network trained to recognize the presence of acute myocardial infarction in patients presenting to the ED with anterior chest pain.
MATERIALS
AND
METHODS
An artificial neural network is a group of interconnected mathematical equations that accept input data and calculate an output based on this input. This structure is a predesigned method for the application of least means nonhnear statistical techniques. The mathematics of this process have been published extensively.2, 3 The network used in this study is illustrated (Figure 1). It consisted of 20 input units, two layers of ten internal or hidden units, and an output unit. In the application of an artificial neural network described here, the inputs are selected from the presenting complaints, history, physical examination, and ECG findings (Table 1) of adult patients presenting to the ED with anterior chest pain. The network is trained so that the output represents the presence or absence of acute myocardial infarction. The strategy used in this study can be appreciated by referring to (left side of Figure 2) the feed-forward process of a highly simplified artificial neural network. The input p a t t e r n in this instance consists of two variables, A and B. Each of these variables is entered into the respective input unit of the network. These values are muhiphed by a number called a weight. The value of the weight is determined by the process used to train the network. 1-3 The products of these multiplications are summed and become the net input of the hidden layer trait. This process is represented by: netpi = ~ w i j a p j + bias i j=0
where netvi equals net input of the unit for p a t t e r n p , w is a weight, a is the input value applied to the unit (ie, A or B), j represents the input or presynaptic units, i represents the first layer hidden unit or postsynaptic unit, and bias is a modifiable weight that is multiplied by an input that is always equal to 1. The net activation of the hidden unit is calculated by placing the net input into a logistic function: 1 a
Figure 1. Diagnostic network: 20 x 10 x 10 x ] network with 20 input units, two layers often units, and one output unit
Input units . . . . .
Hi
Hi
V U L ~ U L UlIIL
36/1440
@@@@@
.~
pt
] + e-netpl
The activation of the hidden unit is multiphed by a third weight. This p r o d u c t becomes the net input of the output unit or generically that of other hidden units in multilayered networks. The activation of the output unit or other units is calculated by use of the above equation in a manner analogous to that of the first layer. Changing the value of input A to A' will change network output (right side of Figure 2). If input A is set to a specified value, the effect that A has on network output can be appreciated. Ideally, the easiest way to study how individual input variables affect network output would be to set all inputs to zero and then individually set each input to a positive value and study the numeric impact this has on network output. Unfortunately, the nonlinear manner in which the network operates makes this impossible. The network is highly context-sensitive. That is, if the value of the same input variable in two different patterns were reversed, the net effect on network output would be different in each pattern. This is illustrated (Table 2).
ANNALS 0FEMERGENCYMEBICINE 21:12 DECEMBER 1992
NEURAL NETWORK Baxt
Two p a t t e r n s , A and B, d e r i v e d f r o m two patients who differ by the value of only one i n p u t are depicted. P a t t e r n s A and B differ only in t h a t the p a t i e n t in A has shortness of b r e a t h and the patient in B has syncope. T h e o u t p u t for each initial p a t t e r n is shown, 4.2 x 10 -5 and 2.6 x 10 -5, respectively. T h e effect on n e t w o r k o u t p u t of m a k i n g the same v a r i a b l e , the p r e s e n c e of nausea a n d vomiting, initially 0, 1 is illustrated in p a t t e r n s A' and B'. In the first case, the difference between the initial n e t w o r k o u t p u t and the o u t p u t generated after the change in the v a r i a b l e of n a u s e a and vomiting t e r m e d the delta o u t p u t (A' - A) is 2.6 x 10 -3. In the second case, the delta o u t p u t (B' - B) is 2.0 x 10 -4, a fivefold difference. The effect of i n d i v i d u a l variables is d e p e n d e n t on the state of all the o t h e r v a r i a b l e s in the p a t t e r n . As a result, individual i n p u t variables must be studied in the context of the whole p a t t e r n , and only t h e i r general overall effect can be a p p r e c i a t e d . In view of this, the effect of changing individual v a r i a b l e s within existing p a t t e r n s was studied. The n e t w o r k was t r a i n e d as r e p o r t e d p r e v i o u s l y 17 on 706 patients 18 years or older who p r e s e n t e d to the E D with a n t e r i o r chest pain. T h e weights d e r i v e d f r o m the t r a i n i n g were used for the d e r i v a t i o n of the p a r t i a l outputs. The patterns d e r i v e d f r o m these 706 patients were used for the analysis. E a c h of the 20 i n p u t v a r i a b l e s was v a r i e d for e a c h pattern. I n each case, only one v a r i a b l e was c h a n g e d at a time so t h a t the effect in the midst of the i n t e r p l a y of all existing variables could be m e a s u r e d . P a t i e n t age was v a r i e d by inputting an a r b i t r a r y age f r o m 0 to 90 years at t e n - y e a r intervals, and sex was set f r o m either female or male to 0; response to t r i n i t r o g l y c e r i n was set f r o m 0 (not used) to 2 (incomplete relief) or f r o m 1 (complete relief) or 2 to 0. Binary variables were changed f r o m 0 (not present) to 1 (present) or 1 to 0. F o u r possible settings can exist: Patients either will h a v e or h a v e not sustained a m y o c a r d i a l i n f a r c tion a n d aside f r o m age, the i n p u t v a r i a b l e will h a v e been
Table 1.
changed f r o m either a positive value to 0 or f r o m 0 to a positive value. As e a c h of the 706 p a t t e r n s was individually i n p u t t e d to the n e t w o r k , the p r o g r a m calculated an o u t p u t based on the intact p a t t e r n ( o u t z ) . One of the 20 clinical i n p u t variables was t h e n c h a n g e d as n o t e d above (ie, A to A'). A second o u t p u t was t h e n calculated b a s e d on the modified p a t t e r n ( o u t 2 ) . The first o u t p u t was t h e n s u b t r a c t e d f r o m the second o u t p u t to yield a delta output3: A out = out 2-
out]
This process was t h e n separately r e p e a t e d for each of the 20 variables with the changes m a d e as n o t e d above. As e a c h p a t t e r n was i n p u t t e d in this m a n n e r , the delta o u t p u t was s u m m e d for each of the f o u r settings (noted above) a n d a delta o u t p u t m e a n was calculated for each variable. The artificial n e u r a l n e t w o r k s i m u l a t o r as well as the code for p a r t i a l o u t p u t analysis was written specifically for this study in C and r u n on a U N I X workstation.
Figure 2. Simplified operation o f artificial neural network with two input units, one hidden unit, and one output unit (2 x 1 x 1). The input pattern consists o f the three variables A, A', and B, where A' is a modified A. Each o f these variables is entered into the respective input unit o f the network. These values are multiplied by a number called a weight (Wt[1], Wt[2]). The products o f these multiplications are summed and become the net input o f the hidden layer unit. A" yields a modified input. The input values are placed in a logistic function that calculates the net activations o f the hidden units. The net activations o f the hidden units are multiplied by a third weight (Wt[3]). These products then become the net input o f the output units. These sums are entered into the same logistic function that calculates the net activation o f the output unit (termed network output) and activation', the output modified by the change in A to A'.
A
B
A'
Input variables
Presenting Data
History
Age *t Past acute Sex* myocardial Left anterior infarction location of pain* Angina* Diabetes* Hypertension* Intensity of pain Radiation of pain Nausea and vomiting* Diaphoresis* Syncope* Shortness of breath* Palpitations* Response to nitroglycerin*
Examination Jugular venous distension* Roles*
ECG Findings 2-ram ST elevation* l-ram ST elevation* ST depression* T-wave inversion* Significant ischemic change*
* Variables actually utilized by network. * Analog ceded.
DECEMBER1992
21:12 ANNALS 0F EMERGENCY MEDICINE
A x Wt(1) = X
B x Wt(2) = X
X + Y = Output Activation
X' + Y = Output' Activation'
1
1
Activation x wt(3) = output Activation
A' x Wt(1) = X'
Activation'x Wt(3) = Output' I
Activation'
Activation - Activation' = Delta Activation
1441/37
NEURAL N E T W O R K Baxt
RESULTS
The overall sensitivity and specificity of the trained network on the unchanged patterns was 97%. The results of the process outlined above arc presented (Tables 3 and 4). The effects appreciated by varying age between 0 to 90 years, ablating the effect of sex, and changing the response to trinitroglycerin are summarized (Table 3). Data are divided into the effects appreciated in either patients who had not and had sustained myocardial infarction and by what manner the variable was changed. The effects of reversing the 17 binary variables are summarized (Table 4). Data are also divided into the effects appreciated in either patients who had not or had sustained a myocardial infarction. The results are further broken down into whether the variable was changed from 0 to 1 or from 1 to 0. The data are also arranged in descending order of the raw mean of each variable. The latter was calculated by taking the mean of the sum of each of the two means infarction and noninfarction calculated from changing the variable from 0 to 1 plus the negative sum of the two means calculated from changing the same variable from 1 to 0. When a variable change from 1 to 0 has a negative effect on network output, this implies that the initial state of that variable favors the diagnosis of myocardial infarction, whereas a positive effect would disfavor the diagnosis. The opposite holds when such a variable is changed from 0 to 1. The raw mean, thus, gives a generalized impression of the overall effect each variable is having on the network output. Table 2. Comparison o f the effect o f changing one variable in two patterns varying in one variable Pattern A
Ablating age has a weak positive effect on output (Table 3), actually favoring the diagnosis of myocardial infarction up to age 60 years and then a weak negative effect disfavoring the diagnosis. Ablating the effect of sex also had a positive effect on network output, disfavoring the diagnosis of myocardial infarction in all cases. Changing the response to trinitroglycerin from 1 (complete relief of pain) or 2 (incomplete relief of pain) to 0 (not used) had a negative effect on output, whereas changing it from 0 to 2 had a positive effect on network output. The effect of changing most variables from 0 to 1 had a positive effect on network output (favoring the diagnosis of myocardial infarction) (Table 4). This impression is reinforced when the same variables are changed from 1 to 0. In this case they have the opposite effect and lower network output (disfavoring the diagnosis of myocardial infarction). Six variables have effects opposite to this trend, and most of these are small, hi general, when variables were changed from 0 to 1, the more marked effects were in patients ~-ithout myocardial infarction because the patients with myocardial infarction already had high outputs resultant from the settings of their other variables. The antithesis was true when variables were changed from I to 0. In this case, the greatest decrement in output was in patients with infarction who had high outputs at the onset. The generalized effect of all the variables is summarized (Tables 3 and 4) by the raw mean of each variable. The variables (Table 4) are arranged in the descending order of the size of the raw mean. Table 3. Impact o f varying input on age, sex, and use o f trinitroglycerin in an artificial neural network trained to detect myocardial infarction AGE (yr)
Variable
A
A'
B
B'
Age
Age Sex Location of pain Response Nausea and vomiting Diaphoresis Syncope Shortness of breath Palpitations History of myocardial infarction History of diabetes mellitus History of hypertension History of angina Jugular venous distension Rales 2-mm ST elevation 1-mm ST elevation ST depression T-wave inversion Ischemic change
0,50 1 1 0 0 1 0 1 0 0
0.50 1 1 0 1 1 0 1 0 0
0.05 1 1 0 0 1 1 0 0 0
0.50 1 1 0 1 1 1 0 0 O
0 10 20 30 40 50 60 70 80 90
0 0 0 0 O 0 0 I 1 0
0 0 0 0 0 0 0 1 1 0
0 0 O 0 0 0 0 1 1 0
0 0 0 0 0 0 0 1 1 0
SEX
Output
38/1442
4.2x10 -~
Patients Without Myocardial Infarction
Pattern B
2.6×1'0 5 2.6×10 5
2.3x10 4
0.013343 0.010191 0.007624 0.005386 0.002594 0.001307 0.000561 -0.000580 -0.001897 -0.002190
Female to 0 Noninfarct Mean Delta 0.0419
Infarct Mean Delta 0.0120
Patients With Myocardial Infarction 0.001492 0.001330 0.000885 0.000720 0.000719 0.000471 -0.000506 -0.001877 -0.005295 -0.009134
Infarct Mean Delta -0.0412
0.007417 0.005761 0.004254 0.003053 0.001656 0.000889 0.000027 -0.001229 -0.003596 -0.005662
Male to 0 Noninfarct Mean Delta 0.0534
TRINITROGLYCERIN 1 or2to 0 Noninfarct Mean Delta -0.0084
Total
Infarct Mean Delta 0.0021
Raw Mean -0.0300
Oto2 Noninfarct Mean Delta 0.5862
Infarct Mean Delta 0.0176
Raw Mean 0.0339
ANNALS 0F EMERGENCY MEDICINE 2I:12
DECEMBER1992
NEURAL N E T W O R K Baxt
The raw means of 19 network variables are summarized (Table 5). The presence of the five ECG findings, rales, syncope, jugular venous distension, response to trinitroglycerin, and nausea and vomiting, all have significant positive effects on the network output, favoring the diagnosis of myocardial infarction (Table 5). The remaining variables have smaller positive or negative effects. DISCUSSION
One of the hopes that has arisen from the clinical application of the artificial neural network is that it could be used to afford greater insights into the relationships between clinical variables that characterize various disease states. This has emanated from the findings that in other applications such networks are able to appreciate relationships between i n p u t data that clearly were not apparent to prior h u m a n or computational analysis. Because the initial studies of the application of the artificial neural network to the diagnosis of myocardial infarction have indicated that it performs substantially better than either physicians or other computerassisted aids, 2°-22 there was some justification to the belief that it could be used to identify such relationships. It was on this basis that this study was undertaken. The results suggest that this may be the caste. This approach has been able to develop a generalized impression as to the implicit effects different variables are having on network output (diagnosis) and can be viewed as a possible first step in the unraveling of,t~e methods by which Table 4. Impact of varying binary input variables of an artificial neural network trained to recognize the presence of myocardial infarction 0-1
1-0
Noninfarct Mean
Infarct Mean
Noninfarct Mean
Infarct Mean
Raw Mean
Delta -0.5719 Rales -0.2022 2-mm ST elevation -0.2005 ST depression -0.0666 1-mm ST elevation -0.0368" Syncope -0.0722 " Jugular venous -0,1193 distension Nausea and -0.0226 vomiting T-wave inversion -0.0214 Diaphoresis -0.0152 Shortness of breath -0.0091 History of 0.0051 diabetes mellitus History of 0.0517 myocardial infarction Palpitations 0.0252 History of 0.0464 hypertension History of angina 0.0528 Location of pain 0.1088
-0.5550 -0.2953 -0.0819 -0.2274 -0.1565 -0.0450 -0.0341
0.8428 0.3271 0.2131 0.1687 0.0560 0.0480 0.0454
0.1372 0.0309 0.0631 0,0404 0.0082 0.0083 0,0083
0.7566 0.2706 0.1809 0.1437 0.0489 0.0418 0.0391
-0.0184
0.0333
0.0106
0.0273
-0.1118 -0.0353 -0.0135 0.0004
0.0225 0.0170 0.0040 -0.0043
0.0004 0.0219 0.0044 -0.0234
0,0265 0.0182 0.0066 -0.0078
0.0196
-0.0134
-0.0365
-0.0250
Variable
0.0002 0.0008
-0.0151 -0.0233
-0.0863 -0.0828
-0.0317 -0.0352
0.0176 0.0128
-0.0311 -0.0665
-0,1282 -0.1320
-0.0491 0.0833
DECEMBER 1992 21:12 ANNALS OF EMERGENCYMEDICINE
artificial networks come to clinical closure. These data must be interpreted with a great deal of caution, and these findings cannot be taken literally. The individual effects of the clinical variables must be kept in the context of the patient and not used as isolated independent predictors of outcome. These results do not suggest the use of any of these variables as part of a superstructure for the development of a differential diagnosis. These data only give insight into this interplay and do not suggest the independent use of these results as inert predictors of events. Furthermore, it must be understood that the use of the term "general" has greater implications than may be apparent. Because of the unique m a n n e r by which the network functions, variables that appear to have major impacts based on this study, in certain variable cluster settings, may have a major impact only in that setting. Finally, it must be stated that the netwoi~k is a reflection of the patient data base on which it is trained. As such, its "knowledge" is only that available from the information that can be accrued from that population. Although 706 patients represent a substantial n u m b e r of such data points, this is a minuscule representation of all such patients. This work must be viewed only as a model, and not until this process has been repeated on a very large n u m b e r of such patients can the significance of variable weighting be comfortably taken to represent any reality. The m a n n e r by which the network uses some input data was unexpected. The most striking finding is that the variable that was found to have the greatest influence on network output, significant ischemic ECG change, had a mean impact of only 0.7566. Because the outputs generated by patterns extracted from patients with acute myocardial infarction have outputs of more than 0.9500, it is clear that no single variable in and of itself has the capability of imparting Table 5. Impact according to raw mean of varying each of the 20 input variables of an artificial neural network trained to recognize the presence of myocardial infarction
Variable Delta Rales 2-ram ST elevation ST depression l-ram ST elevation Syncope Jugular venous distension Trinitroglycerin Nausea and vomiting T-wave inversion Diaphoresis Shortness of breath History of diabetes mellitus History of myocardial infarction Palpitations History of hypertension History of angina Location of pain Sex
Raw Mean 0.7566 0.2706 0.1809 0.1437 0,0489 0.0418 0,0391 0,0339 0.0273 0.0265 0.0182 0.0066 -0.0078 -0.0250 -0.0317 -0,0352 -0.0491 -0.0833 -0.0300
1443/ 39
NEURAL NETWORK Baxt
enough positive impact on network output to cause it to rise to this level. The network must recognize the presence of multiple variables to generate an output that is diagnostic of myocardial infarction (more than 0.9500). The second striking feature of this analysis was that although the effects of the ECG findings were expected, the importance of the findings of the presence of rales, nausea and vomiting, and syncope was surprising. In fact, the presence of tales appears to be more important toward supporting the diagnosis of myocardial infarctions than all the ECG variables, save that of significant iachemic change. Although these nonECG variables have been known to be potential indicators of the presence of infarction, that they are given such strong weighting by the network indicates that these may be far more predictive than heretofore appreciated. F u r t h e r , the impact of age was uniformly weak, and the impact of sex was negative in all cases. It would have been expected that the former would have h a d a positive effect at older ages, whereas the latter would have h a d either a null influence or at least a uniformly weak positive influence. Finally, it was also surprising that the variables of history of hypertension, diabetes melhtus, and angina, which in the past have been thought to be predictive of myocardial infarction, had negative impacts on network output and actually disfavored the diagnosis of myocardial infarction. CONCLUSION
This type of analysis of network processing may be able to elucidate new relationships among clinical variables that characterize disease states. These techniques, as well as others,23, 2a may enable the development of logistic rules from trained networks, which m a y b e used prospectively to both expand the knowledge derived from network processing and improve clinical diagnostic accuracy. REFERENCES 1. Widraw G, Heft ME: 1960Adaptiva Switching Circuits Institute of Radio Engineering Western Electronic Show and Convention, Convention Record, Part 4, p 96-104. 2. Rumelhart DE, Hinton GE, Williams RJ: Learning internal representations by error propagation, in Rumelhart BE, McClelland JL (ads): Parallel Distributed Processing: Explorations in the Microstrueture of Cognition. Cambridge, Massachusetts, MIT Press, 1986, p 318-364. 3. McClelland JL, Rumelhart BE: Training hidden units, in McOlelland JL, Rumelhart BE (ads): Explorations in Paraflel Distributed Processing. Cambridge, Massachusetts, MIT Press, 1988, p 121-160. 4. Weigend AS, Huberman BA, Rumelhart DE: Predicting the Future:A Conneetionist Approach. Stanford PDP Research Group, April 1990. 5. Hudson DL, Cohen ME, Anderson MF: Determination of testing efficacy in carcinoma of the lung using a neural network model. Symposium on CemputerAppficatiens in Medical Care 1988Proceedings: 12thAnnual Symposium, Washington,DC1988;t2:251-255. 6. Smith JW, E~zerhartJE, Dickson WC, et al: Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. Symposium on ComputerAppficatiens in Medical Care 1988Proceedings," 12thAnnual Symposium, Washington, DC 1988;12:261-265.
8. Kaufman J J, Chiabera A, Hatem M, et al: A neural network approach for bone fracture healing assessment. IEEEEng Med Bie11990;9:23-30. 9. Hiraiwa A, Shimohara K, Tekunaga Y: EEG topography recognition by neural networks. IEEEEng Mad Bie11990;9:39-42. 10. Cios KJ, Chen K, Langenderfer RA: Use of neural networks in detecting cardiac diseases from echocardiographic images. IEEEEng Met Bie11990;9:58-60. 11. Marconi L, Scalia F, Ridella S, et al: An application of back propagation to medical diagnosis. Proceedings of the International Joint Conference on Neural Networks, Washington, DC 1989;2:577. 12. Eberhart RC, Dobbins BW, Hutton LV: Neural network paradigm comparisons for appendicitis diagnosis. Proceedings of the Fourth Annual IEEESymposium on Computer-Based Medical Systems 1991;298-304. 13, Mulsant 6H, Servan-Schreiber E: A connectionist approach to the diagnosis of dementia. Symposium on ComputerAppficatiens in Medical Care 1988Proceedings: 12thAnnual Symposium, Washington, DC 1988;12:245-250. 14. Bounds DG, Lloyd PJ, Mathew BG: A comparison of neural network and other pattern recognition approaches to the diagnosis of low back disorders. Neural Networks 1990;3:583-591. 15. Yaon YO, Brobst RW, Bergstresser PR, et al: A desktop neural network for dermatology diagnosis. J Neural Network ComputationSummer 1989;43-52. 16. Baxt WG: Use of an artificial neural network for data analysis in clinical decisionmaking: The diagnosis of acute coronary occlusion. Neural Computation 1991;2:480-489. 17. Baxt WG: Use of an artificial neural network for the diagnosis of myocardial infarction. Ann Intern Med 1991;115:843-848. 18. Baxt WG: Improving the accuracy of an artificial neural network using multiple differently trained networks. Neural Computation 1992;4:772-780. 19. Harrison RF, Marshall SJ, Kennedy RL: The early diagnosis of heart attacks: A neurocamputational approach. Proceedings of the International Joint Conference on Neural Networks, Seattle 1991;1:1-5. 20. Pozen MW, D'Agostino RB, Mitchell JB, et al: The usefulness of a predictive instrument to reduce inappropriate admissions to the coronary care unit. Ann Intern Met 1980;92:238-242. 21. Goldman L, Weinberg M, Waisberg M, et al: A computer-derived protocol to aid in the diagnosis of emergency room patients with acute chest pain. N Engl J Med 1982;307:588-596. 22. Goldman L, Cook EF, Brand BA, et al: A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med 1988;318:797-803. 23. Maclin R, Shavlik JW: Refining algorithms with knowledge-based neural networks: Improving the Chau-Fasman algorithm for protein folding, in Hanson S, Drastal G, Rivest R (eds): Computational Learning Theory and Natural Learning Systems. Cambridge, Massachusetts, MIT Press (in press). 24. Towell GG, Shavlik JW: The extraction of refined rules from knowledge-based neural networks. Machine Learning (in press). The author thanks Drs Halbert White and David Zipser for their help with the technical aspects of this study and Kathleen James for her help in the preparation of this manuscript. Address for reprints: William G Baxt, MD Department of Emergency Medicine UCSD Medical Center 200 West Arbor Drive, #8676 San Diego, California 92013-8676
7. Saito K, Nakano R: Medical diagnostic expert system based on PDP model. Proceedings of the International Joint Conferenc~ on Neural Networks, San Diego 1988;2:255-262.
40/1444
ANNALS 0F EMERGENCY MEDICINE
21:12
DECEMBER1992
