Establishing Criterion Validity of a Computer-based Clinical Simulation Sandra J. Engberg, MSN, RN-C University of Pittsburgh School of Nursing
Joyce E. White, DrPH, CRNP University of Pittsburgh School of Nursing Steven Belle, Ph.D. University of Pittsburgh Graduate School of Public Health
Abstract
life"(p.26,). Taylor, Grace, Taylor, Finchman, and Skakenl
found a low positive correlation between performance on a computerized patient management problem (PMP) and performance on a multiple choice certification examination and no correlation between the PMP and an oral exam. have found physicians' responses to simulations to be totally different from their responses to actual patients, finding a much more rigid approach and lower levels of diagnostic accuracy when caring for the simulated patients.
Clinical simulations, depictions of real-life patient care situationsUL have been used extensively in medical education. This study determined the criterion validity of a computer-based simulation which permits natural language interface and mns interactively with videotape to be fair to good (K = .61).
Otherst8I
Introduction
Computer-assisted instruction in medical education makes extensive use of simulations, representations of reality that correspond to reallife situations that the student is expected to encounter. The simulation strategy is considered especially appropriate for the mastery of higher order cognitive skills: application, analysis and synthesisf2' Unless they possess criterion validity, that is, evidence that the individual's interaction with the computer reflects his/her performance if presented with a similar problem in the actual clinical setting, however, such simulations can be seen to be of limited value.
This study compared chart notes written by physicians after caring for the computer-simulated patient with chart notes written after seeing actual patients presenting in the clinical setting with the same complaint. The Simulation
The patient presented via the simulation was drawn from the caseload of one of the authors.
She was a 33 year-old woman with a common gynecologic complaint. Using Precept an authoring system described previously/9 II] patient data were stored on the hard drive of an IBM compatible microcomputer where it was accessed by the participants using ordinary English. Precept allows for this natural language interface by the system's recognition of multiple key word synonyms. Unlike rule-based artificial intelligence systems which use grammatical rules to process the users' requests for information, Precept looks for a match between the requests and synonyms which
Background
Several studies have supported the credibility of computer-based simulations, that is, users reported that the simulations accuratelv represented clinical encounters with patients.f3'4'5 In one of the few studies which investigated criterion validity of simulations, Friedman[6l found a low to moderate relationship when physicians' performance with Computer-Based Examination (CBX) simulations were compared with "real 0195-4210/91/$5.00 © 1992 AMIA, Inc.
have been entered into the system in association with the correct response.
737
A check list, developed to record patient information from the progress notes, was used after inter-rater reliability had been established at .77. Using the check list, the investigators extracted data from the progress notes of 6 to 10 patients cared for by each resident. The same check list was used to extract information from the SOAP note written for the simulated patient.
Some of the requested data were presented via interactive videotape. The patient described her chief complaint and its associated manifestations via tape. Parts of the physical exam and lab findings were also presented on tape in response to the users' requests. The addition of videotape segments adds to the credibility of the simulation by (a) allowing the user to associate a person with the simulation, (b) allowing the demonstration of nonverbal cues and (c) requiring the user to interpret physical and laboratory findings.The computer was interfaced with a Sony videotape player with a dual audio channel using BCD Corporation's VIPc videotape system which consists of the motherboard, videotape board, videotape breakout box and VTR control cable.
Results and Discussion
First we selected, in accord with expert clinicians' practice, data which would be used in Then we calculated the the comparison. proportion of time each resident documented these data when caring for actual patients in order to determine the variation in data recorded in actual clinical cases. As demonstrated by Table 1, there was little consistency in the recording of data from cast to case.
Methodology Participants
Next we compared each resident's acquisition of subjective and objective data from the simulated and actual patients. We found that the residents consistently documented more data for the simulated patient than for actual patients (Table 2). Of three possible pieces of subjective data, they documented an average of 2.6 pieces on the simulated patient and 1.852 on the actual patients. There were 11 possible pieces of objective information. They documented an average 7.3 pieces on the simulated patient and 4.8 pieces for the real patients.
The participants were 10 first and second year residents in obstetrics and gynecology in a 543-bed community hospital in a Mid-Atlantic state. Instrument
The instruments were progress notes, written in a SOAP format (subjective data, objective data, assessment and plan) after participants cared for the simulated and actual patients with the same complaint.
Then we compared overall documentation of care for the simulated and actual patients, considering care to have been given to actual patients if it was documented in 60% or more of each resident's cases and not given if documented in 50% or fewer cases. Table 3 shows the results of comparing the history, physical exam, lab testing and diagnosis and intervention recorded by each of the 10 participants. Overall comparability ranged from 0% to 100% with recording of historical data and diagnosis accounting for most of the Almost invariably, poor disagreement. comparability between simulated and actual patient care reflected noting of an item on the record of the simulated patient and failure to note that item on 60% or more of the charts of actual patients.
Procedure
The interactive workstation was set up in a room in the ob/gyn clinic where residents were assigned to care for patients. During a three-day period, all 10 of the residents assigned to the clinic were approached by a member of the nursing staff and invited to participate in the study by caring for the simulated patient, writing a SOAP note, and agreeing to have charts of actual patients for whom they had cared reviewed. After working through the simulation, residents wrote SOAP notes as if they had cared for an actual patient and gave them to the investigators. During the next three months, the investigators reviewed charts of patients who had presented with the same gynecologic complaint to the residents who had cared for the simulated
patient.
738
Tablk 1: Piwten
Claucal cam With SNbItiV and
Data
Subiective Chronology of the problem Nature of the problem Associated manifestations PhYsical exam BUS Vulva Vaginal mucosa Presence of discharge Cervix Uterine fundus Adnexa Breast Abdomen Inguinalnodes Rectal
Dka~iveDta Iwnna
1(%) 2(%)
3(%)
20 20 80
30 50 80
50 70 80
50 20 90
30 70 70
50 80 100
30 80 90
50 75
S0
60 100 90
90 50 30 70 80
90 90 60 80 90 80 80 10 10 0 0
60 50 0 100 60 30 30 10 0 0 0
100 80 60 80 80 80 80 40 40 0 10
0 100 30 100 100 100 80 30 20 0 0
0 50 10 40 50 20 30 0 10 0 0
10 100 70 90 100 90 90 10 70 0 10
50 10 10 75 50 40 40 0 0 0 0
0 100 0 90 90 90 90 40 20 20 0
40 60
100 80 30 40 60 80 80 70 10 0 0 0
80 90 20 40
0 10
4(%) 5(%)
6(%) 7(%)
8(%)
9(%)
10(%)
Table 2: Documentation for Simulated and Actual Clinical Cases for Subjective and Objective Data Only Sbjciu Dat M n Shj e Dta Ojac Data Mw O(*jecd Dat Ruident Sianulat it Acal Patent patiw Acad Pa_1tie 1 2 3 4 5 6 7 8 9 10
3 2 2 3
2 3 3 2 3 3
2 1.2 1.6
2 1.6 1.67 2.2 2 2.5 2
739
9 7 7 9 6 8 6 9 S 7
4 5.6 6
3.4 6.5 5.5 2.1 6.4 2.75 5.38
Tabk 3: Cwir i
(Awt NEW
Wrin Simw in Pop
OrUUl
Dx
66 33 66 66 33 66 66 100 66 66
81 81 91 36 64 70 45 82 64 82
100 100 100 100 50 50 100 100 50 100
0 100 100 0 100 100 0 100 0 100
100 100 100 100 100 100 100 100 100 100
77 77 89 50 61 72 62 89 59 83
63
70
85
60
100
72
IMM60uie
Conclusions and Recommendations
The growth of the use of computer-based simulations in the education of health care professionals is associated with a need to establish their criterion validity. It is of limited value to learn to care for simulated patients if the process of care learned is not related to that required when The continued caring for actual patients. use and of development computer-based simulations without concomitant establishment of criterion validity probably reflects the difficulty with which such validity is established. This study demonstrated the ability to establish such validity while illustrating the ongoing measurement issues associated with such studies.
These results are somewhat better than those of Friedman[5] who reported that field trials comparing performance in the CBX project with audits of hospital records resulted in fair to moderate agreement (K = .33 to .42).
Although the use of chart notes for comparison between care of simulated and actual patients seemed reasonable and avoided the difficulties associated with comparing simulations to criteria such as class standing and performance on exams, in the future we will compare direct observations of actual care provided to patients with logs of the simulated patient interaction which Precept-authored simulations provide. We expect that this comparison will result in findings of higher criterion validity because we suspect that
uu t iftl Sia oe i PELUN andlAdaaa as Ez hxbyaara Vahmv
Tablk 4:
1 2 3 4 5 6 7 8 9 10
AcaldsPatm
Lab
Finally, we calculated a Kappa value (Table 4) to determine the percent agreement above chance for each participant and found excellent agreement for 5 of the participants, fair to good for 2, and poor for 3. (For most purposes, a Kappa of greater than .75 reflects excellent agreement beyond chance; a Kappa of .40 to .75 indicates fair to good agreement and a Kappa of less than .40, poor agreement.)11 Overall agreement for all participants for all data was fair to good (K = .61).
Participant
d
PE
Suhepdx 1 2 3 4 5 6 7 8 9 10 All Subjects
we
Kappa .4624 .8095 .9524 .3950 .8091 .7619 .2999 .8000 .3798 .7174
more data were collected than was recorded in the actual clinical setting.
740
References 1.
2.
9.
Abdulla, A., Henke, J., and Watkins, L. (1987). Computer-aided learning: Experiences, perspectives and promises. Journal of the American College of Cardiology, 2 678-683.
10.
Abdulla, A., Watkins, L., and Henke, J. (1984). The use of natural language entry and laser videodisk technology in CAI. Journal of Medical Education, 59, 739745.
11.
Abdulla, A., Watkins, L., Henke, J., Weitz, F. and Frank, M. (1983). Classroom use of personal computers in medical education: A practical approach. Medical Education, 17 229-232.
Dooling, S. (1986). Designing computer simulations for staff nurse education. Journal of Medical Systems, 10(2), 139148.
Billings, D. (1986). Computer asssited instruction for health professionals: A guide to designing and using CAI courseware. Norwalk, Connecticut: Appleton-Century-Crofts.
3.
deDombal, F. (1969). A computer aided system for learning clinical diagnosis. Lancet, 1, 145-148.
4.
Finchman, S., Grace, M., Taylor, W., Skakun, E. & Davis, F. (1976). Pediatric candidates' attitudes to computerized patient management problems in a certifying examination. Medical Education, 10, 404-407.
5.
White, J., Nativio, D., Kobert, S., & Engberg, S. (1991). Clinical decision making by nurse practitioenrs: The roles of content and process. University of Pittsburgh School of Nursing (unpublished)
6.
Friedman, R. (1990). Some results from the computer-based examination (CBX) project. Computer Applications in the Evaluation of Physician Competence (Ed. J.S. Lloyd). Chicago: American Board of Medical Specialties, 21-30.
7.
Taylor, W., Grace, M., Taylor, T., Finchman, S., Skaken, E. (1976). The use of computerized patient management problems in a certifying examination. Medical Education, 10, 179-182.
8.
Leaper, D., Gill, P., Staneland, J., Horrocks, J., deDombal, F. (1973). Clinical diagnostic process: An analysis. British Medical Journal 3 569-574.
741
Joyce E. White, DrPH, CRNP University of Pittsburgh School of Nursing Steven Belle, Ph.D. University of Pittsburgh Graduate School of Public Health
Abstract
life"(p.26,). Taylor, Grace, Taylor, Finchman, and Skakenl
found a low positive correlation between performance on a computerized patient management problem (PMP) and performance on a multiple choice certification examination and no correlation between the PMP and an oral exam. have found physicians' responses to simulations to be totally different from their responses to actual patients, finding a much more rigid approach and lower levels of diagnostic accuracy when caring for the simulated patients.
Clinical simulations, depictions of real-life patient care situationsUL have been used extensively in medical education. This study determined the criterion validity of a computer-based simulation which permits natural language interface and mns interactively with videotape to be fair to good (K = .61).
Otherst8I
Introduction
Computer-assisted instruction in medical education makes extensive use of simulations, representations of reality that correspond to reallife situations that the student is expected to encounter. The simulation strategy is considered especially appropriate for the mastery of higher order cognitive skills: application, analysis and synthesisf2' Unless they possess criterion validity, that is, evidence that the individual's interaction with the computer reflects his/her performance if presented with a similar problem in the actual clinical setting, however, such simulations can be seen to be of limited value.
This study compared chart notes written by physicians after caring for the computer-simulated patient with chart notes written after seeing actual patients presenting in the clinical setting with the same complaint. The Simulation
The patient presented via the simulation was drawn from the caseload of one of the authors.
She was a 33 year-old woman with a common gynecologic complaint. Using Precept an authoring system described previously/9 II] patient data were stored on the hard drive of an IBM compatible microcomputer where it was accessed by the participants using ordinary English. Precept allows for this natural language interface by the system's recognition of multiple key word synonyms. Unlike rule-based artificial intelligence systems which use grammatical rules to process the users' requests for information, Precept looks for a match between the requests and synonyms which
Background
Several studies have supported the credibility of computer-based simulations, that is, users reported that the simulations accuratelv represented clinical encounters with patients.f3'4'5 In one of the few studies which investigated criterion validity of simulations, Friedman[6l found a low to moderate relationship when physicians' performance with Computer-Based Examination (CBX) simulations were compared with "real 0195-4210/91/$5.00 © 1992 AMIA, Inc.
have been entered into the system in association with the correct response.
737
A check list, developed to record patient information from the progress notes, was used after inter-rater reliability had been established at .77. Using the check list, the investigators extracted data from the progress notes of 6 to 10 patients cared for by each resident. The same check list was used to extract information from the SOAP note written for the simulated patient.
Some of the requested data were presented via interactive videotape. The patient described her chief complaint and its associated manifestations via tape. Parts of the physical exam and lab findings were also presented on tape in response to the users' requests. The addition of videotape segments adds to the credibility of the simulation by (a) allowing the user to associate a person with the simulation, (b) allowing the demonstration of nonverbal cues and (c) requiring the user to interpret physical and laboratory findings.The computer was interfaced with a Sony videotape player with a dual audio channel using BCD Corporation's VIPc videotape system which consists of the motherboard, videotape board, videotape breakout box and VTR control cable.
Results and Discussion
First we selected, in accord with expert clinicians' practice, data which would be used in Then we calculated the the comparison. proportion of time each resident documented these data when caring for actual patients in order to determine the variation in data recorded in actual clinical cases. As demonstrated by Table 1, there was little consistency in the recording of data from cast to case.
Methodology Participants
Next we compared each resident's acquisition of subjective and objective data from the simulated and actual patients. We found that the residents consistently documented more data for the simulated patient than for actual patients (Table 2). Of three possible pieces of subjective data, they documented an average of 2.6 pieces on the simulated patient and 1.852 on the actual patients. There were 11 possible pieces of objective information. They documented an average 7.3 pieces on the simulated patient and 4.8 pieces for the real patients.
The participants were 10 first and second year residents in obstetrics and gynecology in a 543-bed community hospital in a Mid-Atlantic state. Instrument
The instruments were progress notes, written in a SOAP format (subjective data, objective data, assessment and plan) after participants cared for the simulated and actual patients with the same complaint.
Then we compared overall documentation of care for the simulated and actual patients, considering care to have been given to actual patients if it was documented in 60% or more of each resident's cases and not given if documented in 50% or fewer cases. Table 3 shows the results of comparing the history, physical exam, lab testing and diagnosis and intervention recorded by each of the 10 participants. Overall comparability ranged from 0% to 100% with recording of historical data and diagnosis accounting for most of the Almost invariably, poor disagreement. comparability between simulated and actual patient care reflected noting of an item on the record of the simulated patient and failure to note that item on 60% or more of the charts of actual patients.
Procedure
The interactive workstation was set up in a room in the ob/gyn clinic where residents were assigned to care for patients. During a three-day period, all 10 of the residents assigned to the clinic were approached by a member of the nursing staff and invited to participate in the study by caring for the simulated patient, writing a SOAP note, and agreeing to have charts of actual patients for whom they had cared reviewed. After working through the simulation, residents wrote SOAP notes as if they had cared for an actual patient and gave them to the investigators. During the next three months, the investigators reviewed charts of patients who had presented with the same gynecologic complaint to the residents who had cared for the simulated
patient.
738
Tablk 1: Piwten
Claucal cam With SNbItiV and
Data
Subiective Chronology of the problem Nature of the problem Associated manifestations PhYsical exam BUS Vulva Vaginal mucosa Presence of discharge Cervix Uterine fundus Adnexa Breast Abdomen Inguinalnodes Rectal
Dka~iveDta Iwnna
1(%) 2(%)
3(%)
20 20 80
30 50 80
50 70 80
50 20 90
30 70 70
50 80 100
30 80 90
50 75
S0
60 100 90
90 50 30 70 80
90 90 60 80 90 80 80 10 10 0 0
60 50 0 100 60 30 30 10 0 0 0
100 80 60 80 80 80 80 40 40 0 10
0 100 30 100 100 100 80 30 20 0 0
0 50 10 40 50 20 30 0 10 0 0
10 100 70 90 100 90 90 10 70 0 10
50 10 10 75 50 40 40 0 0 0 0
0 100 0 90 90 90 90 40 20 20 0
40 60
100 80 30 40 60 80 80 70 10 0 0 0
80 90 20 40
0 10
4(%) 5(%)
6(%) 7(%)
8(%)
9(%)
10(%)
Table 2: Documentation for Simulated and Actual Clinical Cases for Subjective and Objective Data Only Sbjciu Dat M n Shj e Dta Ojac Data Mw O(*jecd Dat Ruident Sianulat it Acal Patent patiw Acad Pa_1tie 1 2 3 4 5 6 7 8 9 10
3 2 2 3
2 3 3 2 3 3
2 1.2 1.6
2 1.6 1.67 2.2 2 2.5 2
739
9 7 7 9 6 8 6 9 S 7
4 5.6 6
3.4 6.5 5.5 2.1 6.4 2.75 5.38
Tabk 3: Cwir i
(Awt NEW
Wrin Simw in Pop
OrUUl
Dx
66 33 66 66 33 66 66 100 66 66
81 81 91 36 64 70 45 82 64 82
100 100 100 100 50 50 100 100 50 100
0 100 100 0 100 100 0 100 0 100
100 100 100 100 100 100 100 100 100 100
77 77 89 50 61 72 62 89 59 83
63
70
85
60
100
72
IMM60uie
Conclusions and Recommendations
The growth of the use of computer-based simulations in the education of health care professionals is associated with a need to establish their criterion validity. It is of limited value to learn to care for simulated patients if the process of care learned is not related to that required when The continued caring for actual patients. use and of development computer-based simulations without concomitant establishment of criterion validity probably reflects the difficulty with which such validity is established. This study demonstrated the ability to establish such validity while illustrating the ongoing measurement issues associated with such studies.
These results are somewhat better than those of Friedman[5] who reported that field trials comparing performance in the CBX project with audits of hospital records resulted in fair to moderate agreement (K = .33 to .42).
Although the use of chart notes for comparison between care of simulated and actual patients seemed reasonable and avoided the difficulties associated with comparing simulations to criteria such as class standing and performance on exams, in the future we will compare direct observations of actual care provided to patients with logs of the simulated patient interaction which Precept-authored simulations provide. We expect that this comparison will result in findings of higher criterion validity because we suspect that
uu t iftl Sia oe i PELUN andlAdaaa as Ez hxbyaara Vahmv
Tablk 4:
1 2 3 4 5 6 7 8 9 10
AcaldsPatm
Lab
Finally, we calculated a Kappa value (Table 4) to determine the percent agreement above chance for each participant and found excellent agreement for 5 of the participants, fair to good for 2, and poor for 3. (For most purposes, a Kappa of greater than .75 reflects excellent agreement beyond chance; a Kappa of .40 to .75 indicates fair to good agreement and a Kappa of less than .40, poor agreement.)11 Overall agreement for all participants for all data was fair to good (K = .61).
Participant
d
PE
Suhepdx 1 2 3 4 5 6 7 8 9 10 All Subjects
we
Kappa .4624 .8095 .9524 .3950 .8091 .7619 .2999 .8000 .3798 .7174
more data were collected than was recorded in the actual clinical setting.
740
References 1.
2.
9.
Abdulla, A., Henke, J., and Watkins, L. (1987). Computer-aided learning: Experiences, perspectives and promises. Journal of the American College of Cardiology, 2 678-683.
10.
Abdulla, A., Watkins, L., and Henke, J. (1984). The use of natural language entry and laser videodisk technology in CAI. Journal of Medical Education, 59, 739745.
11.
Abdulla, A., Watkins, L., Henke, J., Weitz, F. and Frank, M. (1983). Classroom use of personal computers in medical education: A practical approach. Medical Education, 17 229-232.
Dooling, S. (1986). Designing computer simulations for staff nurse education. Journal of Medical Systems, 10(2), 139148.
Billings, D. (1986). Computer asssited instruction for health professionals: A guide to designing and using CAI courseware. Norwalk, Connecticut: Appleton-Century-Crofts.
3.
deDombal, F. (1969). A computer aided system for learning clinical diagnosis. Lancet, 1, 145-148.
4.
Finchman, S., Grace, M., Taylor, W., Skakun, E. & Davis, F. (1976). Pediatric candidates' attitudes to computerized patient management problems in a certifying examination. Medical Education, 10, 404-407.
5.
White, J., Nativio, D., Kobert, S., & Engberg, S. (1991). Clinical decision making by nurse practitioenrs: The roles of content and process. University of Pittsburgh School of Nursing (unpublished)
6.
Friedman, R. (1990). Some results from the computer-based examination (CBX) project. Computer Applications in the Evaluation of Physician Competence (Ed. J.S. Lloyd). Chicago: American Board of Medical Specialties, 21-30.
7.
Taylor, W., Grace, M., Taylor, T., Finchman, S., Skaken, E. (1976). The use of computerized patient management problems in a certifying examination. Medical Education, 10, 179-182.
8.
Leaper, D., Gill, P., Staneland, J., Horrocks, J., deDombal, F. (1973). Clinical diagnostic process: An analysis. British Medical Journal 3 569-574.
741
