Innovations in Radiologic Education
Impact of High-Fidelity Transvaginal Ultrasound Simulation for Radiology on Residents’ Performance and Satisfaction Rani Ahmad, MD, Ghufran Alhashmi, MD, Amr Ajlan, MD, Bassem Eldeek, MD Rationale and Objectives: Because of the intimate and uncomfortable nature of transvaginal ultrasound, training residents to perform this type of examination is a difficult task. As a consequence, residents may receive inadequate training that leads to a lack of the skills and confidence needed to perform this examination. The aim of the study was to assess the effectiveness of using simulation sessions to teach residents how to perform transvaginal ultrasound, enabling them to diagnose obstetric and gynecologic emergencies and helping them survive on-calls alone while keeping their patients safe. Materials and Methods: We used an experimental study design to compare the confidence levels of 20 senior residents who received clinical training only to those of 25 junior residents who were enrolled in a simulation-based teaching session. We also compared the junior residents’ levels of performance and confidence using transvaginal ultrasound before and after the sessions. Results: The performance of transvaginal ultrasound by the junior residents and their confidence levels significantly improved after they attended the simulation sessions. They had higher levels of confidence than the senior residents who did not attend the session. It was also observed that the number of nondiagnostic transvaginal ultrasounds performed by the on-call resident that needed to be repeated the next day had significantly dropped. Conclusions: Simulation-based teaching sessions are an effective method of education, which improve trainees’ skills and confidence levels and improve patient safety. Key Words: Transvaginal ultrasound; simmulation; postgraduate; residency program; high fidelty. ªAUR, 2015
W
ith advances in computer technology, hardware, and software, ultrasound has become the modality of choice for many medical specialties, such as obstetrics and gynecology (1). Transvaginal ultrasound has been found to be the most extensively used imaging examination in this specialty (2). However, users face many difficulties in learning to use the tool as patients cannot tolerate the examination being extended because of the inexperienced operators. Additionally, training experience is limited due to the restrictions of many hospital policies (3) and the shortage of sonographers (4). The development of high-fidelity mannequins provides an educational environment that fosters training in pelvic ultrasound simulation (5).
Acad Radiol 2015; 22:234–239 From the Radiology Department, King Abdulaziz University Hospital, PO Box 3743, Makkah 21955, Saudi Arabia (R.A.); Medical Education Department, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia (B.E.); and King Abdulaziz University Hospital, King Abdulaziz University, Jeddah, Saudi Arabia (A.A., G.A.). Received June 30, 2014; accepted September 4, 2014. Address correspondence to: R.A. e-mail: [email protected] ªAUR, 2015 http://dx.doi.org/10.1016/j.acra.2014.09.006
234
Simulation is now a well-established educational tool that fosters the learning process of health providers. It can be defined as an artificial replication of sufficient components of the real-world situation to achieve certain goals (6). It imitates many physical appearances, medical procedures, and clinical scenarios that allow trainees to obtain the necessary knowledge or skills without the need for real patients. Simulation offers the learner opportunities to become engaged in experiential learning (7). In comparison to learning that is not experiential in nature, simulation is acknowledged to be more effective (8). Furthermore, in correlation with patient safety, simulation provides new methods of teaching error management and promoting a culture of safety. It also offers ethical benefits, increasing the precision and relevance of training as well as competency assessment (9). Transvaginal ultrasound simulation has been shown to increase the confidentiality of trainees and has the potential to replace the initial clinical training on real patients (2). The objectives of this study are to evaluate the skill acquisition and satisfaction of junior Saudi’s Radiological Residents using the newly introduced simulated session on transvaginal ultrasound to obtain anatomic landmarks of the female pelvis
Academic Radiology, Vol 22, No 2, February 2015
TRANSVAGINAL ULTRASOUND SIMULATION
TABLE 1. McGaghie Mastery Learning Conceptual Framework Element Baseline or diagnostic testing Clear learning objectives, sequenced as units in increasing difficulty Engagement in educational activities (eg, skills practice, data interpretation, reading, and focused on reaching the objectives) A set minimum passing standard (eg, test score) for each educational unit Formative testing to gauge unit completion at a preset minimum passing standard for mastery Advancement to the next educational unit gave a measured achievement at or above the mastery standard Continued practice or study on an educational unit until the mastery standard is reached
and to compare its effectiveness to self-evaluations of the senior residents who have never had any simulation training. MATERIALS AND METHODS Context
The study was performed on the radiology residents under the Saudi Residency Program of the western region, which comprises about half of the Saudi’s residents. The program’s residents are divided into two types: junior and senior. Junior residents are candidates who are in the first 2 years of the program, whereas seniors are in the last 2 years. There are no major differences between the programs of different regions, as they all share the same curriculum, major educational events, and examinations. Junior residents receive 3 months of training in using ultrasound before they become seniors and receive an additional 2 months of training. Study Design
The study is an experimental study test to evaluate the effectiveness and satisfaction of transvaginal ultrasound simulation sessions for radiology residents. Two conceptual frameworks were chosen; one for the designing of the sessions and the other for the evaluation of their outcomes. McGaghie mastery learning conceptual framework (10) is well recognized as the best practice framework for designing simulation sessions. It has seven operational elements, which are listed in Table 1. The outcome of the introduction of this educational intervention was evaluated based on the four levels of evaluation described by Kirkpatrick and Kirkpatrick (11). This framework is considered a standard practice for evaluating educational programs. The first two levels (reaction and learning) will be evaluated during the sessions by using immediate feedback from the participants in the form of a happy-face evaluation sheet after each session along with
Action Performed Presession assessment was performed The required psychomotor skill in obtaining the images of the female pelvic landmarks was clearly identified by the expertise Simulation sessions were conducted focusing on the required skills only Each anatomic landmark is clearly identified Assessment within the sessions was performed with immediate feedback The expected common radiologic appearance of common pathology was discussed briefly in the sessions Postsession assessment was performed after 2–3 months
assessments before and after the sessions. The third level (behavior) was evaluated by the number of calls asking supervisors of the residents to come from home to perform transvaginal ultrasound examinations because residents did not have the confidence to do the initial scanning. This was measured during on-call times in which there was no sonographer on call. The result (the fourth level) was evaluated by checking the rate of the cases that had to be repeated due to inappropriate technique. Formulation of the Assessment Tools
The study aims to assess the performance of junior residents’ presessions and postsessions and to compare their self-assessments of satisfaction with the procedure to the self-assessments of junior residents. Three evaluation forms were generated by the authors. The first was for performance rating, the second was for self-assessment (perception of the residents of their quality in performing the procedure), and the third was for their satisfaction with performing the procedure as a daily practice. The assessment forms were developed through collaborative work between the radiology and the medical education departments at King Abdulaziz University, which has a leader in medical simulation. The performance and self-assessment forms contained a checklist with a scale from zero to nine to evaluate how correctly the residents used the transvaginal ultrasound probe to obtain standard views of the normal anatomic female pelvic structures. The satisfaction form is a checklist with a five-point Likert scale evaluating the resident’s level of confidence, perceived safety level while performing the transvaginal ultrasound on real patients, satisfaction with the quality of images obtained, and with the time needed to complete the study. Piloting of the assessment forms was performed to ensure their validity. To ensure the content validity, the checklist was evaluated by four experts; two from the radiology department and two from the gynecology department. To ensure face validity, the tools of assessment 235
AHMAD ET AL
Academic Radiology, Vol 22, No 2, February 2015
TABLE 2. Reliability Analysis of the Results of the Assessment Tools
Scale Performance assessment tool before the simulation session Performance assessment tool after the simulation session Self-assessment tool before the simulation session Self-assessment tool after the simulation session Satisfaction assessment tool before the session Satisfaction assessment tool after the session
Cronbach Alpha 0.940 0.906 0.934 0.950 0.916 0.933
were distributed to the residents to collect their perceptions. The reliability of the assessment tools was calculated using the Cronbach alpha correlation equation (12) (Table 2). Study Population
The sample population began with 55 radiologic Saudi residents (30 junior and 25 senior residents), all from the western region (which includes five different cities). This sample represented about 65% of all residents. The study excluded any residents who had any extracurricular special training on performing transvaginal ultrasound. After applying the exclusion criteria, we found 25 junior and 20 senior residents who met the criteria and were included in the study. Formulating the Simulation Sessions
The sessions were all conducted in the skill laboratory at King Abdulaziz University. All sessions were facilitated by an expert with 20 years of experience. The learning objectives of the sessions were to master the introduction and the manipulation of the transvaginal ultrasound probe and to learn how to obtain the standard views of a normal female pelvis. The transvaginal ultrasound simulation model (blue Phantom@) was accompanied by an ultrasound machine (Philips IU22@ which is the same machine that residents use in real life) that has a high-frequency transvaginal probe. The duration of each session was 3–4 hours, and each resident participated in at least one session. The quality of the sessions was influenced by the theory of conceptual framework by Kneebone for evaluating clinical simulation for learning procedure skills (13). Kneebone stated that the simulation session should: 1) Allow for sustained deliberate practice within a safe environment, ensuring that recently acquired skills have been consolidated within a defined curriculum that assures regular reinforcement. 2) Provide access to expert tutors when appropriate, ensuring that such support fades when no longer needed. 236
3) Map onto real-life clinical experience, ensuring that learning supports the experience gained within communities of actual practice. 4) Provide a supportive, motivational, and learner-centered milieu that is conducive to learning. Data Collection
Multiple sets of data were collected, including: 1) Pretraining and post-training self-assessment and satisfaction levels of junior residents. 2) Self-assessment and satisfaction levels of senior residents. 3) Pretraining and post-training performance study of junior residents, which was performed by an expert different from the facilitator of the simulation session. The postassessment evaluation was performed about 1 month after the session. Both sessions were performed on routine booked patients after informed consents. 4) Number of calls to supervisors to perform transvaginal ultrasound because of the lack of confidence of a resident. 5) Rate of repeated studies due to inappropriate technique before and after the sessions. Statistical Analysis of Data
The data collected from the rating forms were statistically analyzed using the Statistical Package of Social Sciences version 16.0 (SPSS, Inc., Chicago, IL). The ordinal data of the assessment tools were converted to a scale format, which was then studied using a parametric paired t test for the comparison between the junior residents’ performance presession and postsession and an independent t test for comparison between the perceptions of junior and senior residents. Significance was set at the 95% confidence interval. The effect size was calculated using the Cohen d equation: d = X2 X1/Saverage where X2 X1 is the mean difference between the two groups and Saverage is the average of both standard deviations. Ethical Approval
Approval from the Residents’ Training Program director was obtained along with approval from the research ethics committee of the hospital, with the reference number (1238 – 13).
RESULTS Tables 3 and 4 compare the self-assessment and satisfaction results of the junior and senior residents. The mean scores on the self-assessment of each element of the performance and satisfaction of the junior residents who had the simulation sessions are higher than those of the senior residents, showing a large effect size (d is >0.5). However, there were
Academic Radiology, Vol 22, No 2, February 2015
TRANSVAGINAL ULTRASOUND SIMULATION
TABLE 3. Comparison Between the Junior Residents Who Have Attended the Simulation Training Session and the Senior Residents Who Did Not Attend the Simulation Session, Regarding Their Perception of Their Performance of Transvaginal Ultrasound (Selfassessment) Question I am (I become) familiar with TV probe manipulation I can obtain two views of the uterus I can assess the endometrium I can obtain two views of the ovaries/adnexa I can assess the cervix I can look for free fluid
Juniors (n = 25); m s
Seniors (n = 20); m s
Cohen d
P Value
t
7.4 1.353 8 1.214 7.7 1.302 7.25 1.209 7.05 1.504 7.4 1.759
4.72 2.951 4.68 2.996 5.2 3.175 3.28 2.301 3.36 2.580 5.08 3.040
0.74 0.92 0.65 1.37 1.1 0.6
.001 .001 .002 .001 .001 .004
3.751 4.651 3.3 6.975 5.665 3.027
TV, transvaginal.
TABLE 4. Comparison Between the Junior Residents Who Have Attended the Simulation Training Session and the Senior Residents Who Did Not Attend the Simulation Session Regarding Their Satisfaction of Their Performance of Transvaginal Ultrasound Question I am satisfied about time I need to perform TVUS I am confident to perform TVUS on real patients I am a safe resident obtaining standard views of the female pelvic organs by TVUS I am satisfied with the quality of the images I obtain I feel I can perform TVUS as a routine study
Juniors (n = 25); m s
Seniors (n = 20); m s
Cohen d
P value
t
3.7 1.031 3.75 0.910 3.85 0.875
2.48 1.005 2.48 1.229 2.72 1.308
0.8 0.75 0.65
.001 .001 .002
4 3.85 3.313
3.7 0.923 3.95 0.759
2.4 1.118 2.16 1.028
0.82 1.72
.001 .001
4.181 6.493
TVUS, transvaginal ultrasound.
TABLE 5. Comparison Between the Objective Assessment of Junior Residents’ Performance of Transvaginal Ultrasound Before and After the Simulation Training Session
Question The resident handles the TV probe correctly inside the vaginal The resident obtains two good views of the uterus The resident obtains two good views of the ovaries/ adnexa The resident obtains good views of the endometrium The resident obtains good views of the cervix The resident looks for free fluid Overall, the study is completed
Juniors Before Simulation (n = 25); m s
Juniors After Simulation (n = 25); m s
5.5 3.379
7.75 2.0488
Cohen d
P Value
0.6
.001
3.851
t
5.6 3.761 4.3 3.373
8.0500 1.877 7.3 2.904
0.65 0.65
.002 .001
3.611 4.359
5.5 3.954 4.05 4.11 5.85 4.404 4.9 3.354
7.9 2.573 7.7 2.83 8.1 2.77 7.6 2.037
0.52 0.74 0.45 0.3
.005 .001 .021 .001
3.177 3.822 2.517 4.925
TV, transvaginal.
no significant differences between the scores of the senior residents and the junior residents before they had the simulation sessions. Tables 5–7 compare the performance, self-assessment, and satisfaction of the junior residents’ presimulation and postsimulation sessions. There was an increase in the mean scores of the rating tools with a large effect size in all evaluated elements. The satisfaction from the simulation session was high, with a satisfaction index of 0.85. The total number of calls asking supervisors to come to the hospital to perform transvaginal ultrasound decreased from four calls per month to one call per month. Also, the rate of repeated examinations because of the inappropriate technique
was reduced from eight cases per month to one case per month. DISCUSSION The study shows a significant improvement in the performance and confidence levels of junior residents after the simulation sessions. It also shows higher perceived levels of performance and more satisfaction among junior residents at a statistically significantly higher level than among senior residents. This means that the sessions not only improved the performance of the junior residents but also increased their levels of confidence above those of the senior residents. The insignificant differences between the senior residents 237
AHMAD ET AL
Academic Radiology, Vol 22, No 2, February 2015
TABLE 6. Comparison Between the Junior Residents’ Subjective Self-assessment of Their Performance in Transvaginal Ultrasound Before and After the Simulation Training Session
Question
Juniors Before Simulation (n = 25); m s
Juniors After Simulation (n = 25); m s
Cohen d
P Value
5.15 2.925 3.250 7.4 3.7 3.262 4.35 3.117 3.45 2.625 2.7 2.536 3.8 3.302
7.4 1.875 7.4 1.353 8 1.214 7.7 1.302 7.25 1.209 7.05 .336 7.4 1.759
0.7 0.82 1.5 1.2 1.5 2.7 1
.005 .001 .001 .001 .001 .001 .001
I can introduce TV probe in the correct position I am familiar with TV probe manipulation I can obtain two views of the uterus I can assess the endometrium I can obtain two views of the ovaries/adnexa I can assess the cervix I can look for free fluid
t 3.167 7.625 6.015 5.998 6.418 7.257 6.136
TV, transvaginal.
TABLE 7. Comparison Between the Junior Residents Regarding Their Satisfaction About Their Performance of Transvaginal Ultrasound Before and After the Simulation Training Session
Question I am satisfied with the time needed to perform TVUS I am confident to perform TVUS on real patients I am a safe resident obtaining standard views of female pelvic organs I am satisfied about the quality of the images that I obtain I feel I can perform TVUS as a routine study
Juniors Before Simulation (n = 25); m s
Juniors After Simulation (n = 25); m s
1.95 1.190 2.15 1.348 2.3 1.418
3.7 1.031 3.75 .910 3.85 .875
2.2 1.563 2.2 1.152
Cohen d
P Value
t
1 1 1
.001 .001 .001
5.872 5.446 4.610
3.7 .923
0.9
.001
5.090
3.95 1.698
0.8
.001
6.054
TVUS, transvaginal ultrasound.
and the junior residents before the simulation sessions indicate that both groups came from the same pool and decrease the chances that the junior residents had better performance than the senior residents from the beginning. This, accompanied by the improvement in residents’ confidence to perform the examination during their on-call time, showed performance improvement, measured by a decrease in the rate of repeated examinations.
CONCLUSION Transvaginal ultrasound simulation sessions for the residency program improved the perceived performance and confidence levels of the residents, improving patient safety. They also created a solution for the ethical problem of using a training program to teach residents. Integrating simulation sessions will be recommended for different ultrasound examinations, and the effectiveness of this method will require further studies.
LIMITATIONS At the time of the study, we did not objectively assess the senior residents as the residents themselves felt that this would necessitate losing their anonymity. However, the residents could have completed objective preassessments and postassessments and remain anonymous by recruiting an expert from outside the training program and by having a code number associated with them instead of a name. Future studies could be carried out using this approach. Another possible limitation could have been due to previous training of the junior residents. The exclusion criteria eliminated this confounder, though, by asking the residents if they had received any previous training. Also, the costeffectiveness of the sessions was not calculated as they were free of charge, including the simulation machine and the expert, and the estimation of the cost was not easily calculated. 238
ACKNOWLEDGMENTS The authors want to deliver their deepest thanks to Dr Reda Jamjoom and all his team of Skill lab at King Abdulaziz University. Also the authors are very grateful from all the hard work and efforts made by the senior sonographer Mrs. Rosalinda Huertazuela. REFERENCES 1. Akins Ralitsa, Ho Hoi. Clinical simulation in modern teaching and training of sonography in obstetrics and gynecology. Donald School Journal of Ultrasound in Obstetrics and Gynecology October-December 2009; 3(4): 17–21. 2. Williams CJ, Edie JC, Mulloy B, et al. Ultrasound 2013,Transvaginal ultrasound simulation and its effect on trainee confidence levels: a replacement for initial clinical training? Ultrasound 2013;1–7. http://dx.doi.org/10.1177/17 42271X13481215.
Academic Radiology, Vol 22, No 2, February 2015
3. Gould DA, Reekers JA, Kessel DO, et al. Simulation devices in interventional radiology: validation pending. J Vasc Interv Radiol 2006; 17:215–216. 4. The Society and College of Radiographers. Developing and growing the sonographer workforce: education and training needs. London: SCoR, 2009. 5. Sarmiento J, Stewart K, Aguila J, et al. Pelvic ultrasound simulation training models and case scenarios. Donald School J Ultrasound Obstet Gynecol 2014; 8(1):22–30. 6. Gaba DM. Simulators in anesthesiology, vol. 14. St Louis, MO: MosbyYearbook, 1997; 55–91. 7. Pasquale Susan J. The comprehensive textbook of healthcare simulation: education and learning theory. New York: Springer Science+Business Media, 2013; 51–52. 8. Ziv A. Simulators and simulation-based medical education. In: Dent JA, Harden RM, eds. A practical guide for medical teachers. Edinburgh: Elsevier, 2009.
TRANSVAGINAL ULTRASOUND SIMULATION
9. Ziv Amitai, Small Stephen D, Wolpe Paul Root. Patient safety and simulation-based medical education. Medical teacher 2000; 22(5). 10. McGaghie WC, Siddall VJ, Mazmanian PE, et al., American College of Chest Physicians Health and Science Policy Committee. Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009; 135(3 Suppl):62S–68S. 11. Kirkpatrick D, Kirkpatrick J. Transferring learning to behavior: Using the four levels to improve performance. Berrett-Koehler Publishers; 16–31. Available at: http://site.ebrary.com/id/10221992?ppg=16; 2005. Last access 12-02-2014. 12. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951; 16:297–334. 13. Kneebone R. Evaluating clinical simulations for learning procedural skills: a theory-based approach. Acad Med 2005; 80:549–553.
239
Impact of High-Fidelity Transvaginal Ultrasound Simulation for Radiology on Residents’ Performance and Satisfaction Rani Ahmad, MD, Ghufran Alhashmi, MD, Amr Ajlan, MD, Bassem Eldeek, MD Rationale and Objectives: Because of the intimate and uncomfortable nature of transvaginal ultrasound, training residents to perform this type of examination is a difficult task. As a consequence, residents may receive inadequate training that leads to a lack of the skills and confidence needed to perform this examination. The aim of the study was to assess the effectiveness of using simulation sessions to teach residents how to perform transvaginal ultrasound, enabling them to diagnose obstetric and gynecologic emergencies and helping them survive on-calls alone while keeping their patients safe. Materials and Methods: We used an experimental study design to compare the confidence levels of 20 senior residents who received clinical training only to those of 25 junior residents who were enrolled in a simulation-based teaching session. We also compared the junior residents’ levels of performance and confidence using transvaginal ultrasound before and after the sessions. Results: The performance of transvaginal ultrasound by the junior residents and their confidence levels significantly improved after they attended the simulation sessions. They had higher levels of confidence than the senior residents who did not attend the session. It was also observed that the number of nondiagnostic transvaginal ultrasounds performed by the on-call resident that needed to be repeated the next day had significantly dropped. Conclusions: Simulation-based teaching sessions are an effective method of education, which improve trainees’ skills and confidence levels and improve patient safety. Key Words: Transvaginal ultrasound; simmulation; postgraduate; residency program; high fidelty. ªAUR, 2015
W
ith advances in computer technology, hardware, and software, ultrasound has become the modality of choice for many medical specialties, such as obstetrics and gynecology (1). Transvaginal ultrasound has been found to be the most extensively used imaging examination in this specialty (2). However, users face many difficulties in learning to use the tool as patients cannot tolerate the examination being extended because of the inexperienced operators. Additionally, training experience is limited due to the restrictions of many hospital policies (3) and the shortage of sonographers (4). The development of high-fidelity mannequins provides an educational environment that fosters training in pelvic ultrasound simulation (5).
Acad Radiol 2015; 22:234–239 From the Radiology Department, King Abdulaziz University Hospital, PO Box 3743, Makkah 21955, Saudi Arabia (R.A.); Medical Education Department, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia (B.E.); and King Abdulaziz University Hospital, King Abdulaziz University, Jeddah, Saudi Arabia (A.A., G.A.). Received June 30, 2014; accepted September 4, 2014. Address correspondence to: R.A. e-mail: [email protected] ªAUR, 2015 http://dx.doi.org/10.1016/j.acra.2014.09.006
234
Simulation is now a well-established educational tool that fosters the learning process of health providers. It can be defined as an artificial replication of sufficient components of the real-world situation to achieve certain goals (6). It imitates many physical appearances, medical procedures, and clinical scenarios that allow trainees to obtain the necessary knowledge or skills without the need for real patients. Simulation offers the learner opportunities to become engaged in experiential learning (7). In comparison to learning that is not experiential in nature, simulation is acknowledged to be more effective (8). Furthermore, in correlation with patient safety, simulation provides new methods of teaching error management and promoting a culture of safety. It also offers ethical benefits, increasing the precision and relevance of training as well as competency assessment (9). Transvaginal ultrasound simulation has been shown to increase the confidentiality of trainees and has the potential to replace the initial clinical training on real patients (2). The objectives of this study are to evaluate the skill acquisition and satisfaction of junior Saudi’s Radiological Residents using the newly introduced simulated session on transvaginal ultrasound to obtain anatomic landmarks of the female pelvis
Academic Radiology, Vol 22, No 2, February 2015
TRANSVAGINAL ULTRASOUND SIMULATION
TABLE 1. McGaghie Mastery Learning Conceptual Framework Element Baseline or diagnostic testing Clear learning objectives, sequenced as units in increasing difficulty Engagement in educational activities (eg, skills practice, data interpretation, reading, and focused on reaching the objectives) A set minimum passing standard (eg, test score) for each educational unit Formative testing to gauge unit completion at a preset minimum passing standard for mastery Advancement to the next educational unit gave a measured achievement at or above the mastery standard Continued practice or study on an educational unit until the mastery standard is reached
and to compare its effectiveness to self-evaluations of the senior residents who have never had any simulation training. MATERIALS AND METHODS Context
The study was performed on the radiology residents under the Saudi Residency Program of the western region, which comprises about half of the Saudi’s residents. The program’s residents are divided into two types: junior and senior. Junior residents are candidates who are in the first 2 years of the program, whereas seniors are in the last 2 years. There are no major differences between the programs of different regions, as they all share the same curriculum, major educational events, and examinations. Junior residents receive 3 months of training in using ultrasound before they become seniors and receive an additional 2 months of training. Study Design
The study is an experimental study test to evaluate the effectiveness and satisfaction of transvaginal ultrasound simulation sessions for radiology residents. Two conceptual frameworks were chosen; one for the designing of the sessions and the other for the evaluation of their outcomes. McGaghie mastery learning conceptual framework (10) is well recognized as the best practice framework for designing simulation sessions. It has seven operational elements, which are listed in Table 1. The outcome of the introduction of this educational intervention was evaluated based on the four levels of evaluation described by Kirkpatrick and Kirkpatrick (11). This framework is considered a standard practice for evaluating educational programs. The first two levels (reaction and learning) will be evaluated during the sessions by using immediate feedback from the participants in the form of a happy-face evaluation sheet after each session along with
Action Performed Presession assessment was performed The required psychomotor skill in obtaining the images of the female pelvic landmarks was clearly identified by the expertise Simulation sessions were conducted focusing on the required skills only Each anatomic landmark is clearly identified Assessment within the sessions was performed with immediate feedback The expected common radiologic appearance of common pathology was discussed briefly in the sessions Postsession assessment was performed after 2–3 months
assessments before and after the sessions. The third level (behavior) was evaluated by the number of calls asking supervisors of the residents to come from home to perform transvaginal ultrasound examinations because residents did not have the confidence to do the initial scanning. This was measured during on-call times in which there was no sonographer on call. The result (the fourth level) was evaluated by checking the rate of the cases that had to be repeated due to inappropriate technique. Formulation of the Assessment Tools
The study aims to assess the performance of junior residents’ presessions and postsessions and to compare their self-assessments of satisfaction with the procedure to the self-assessments of junior residents. Three evaluation forms were generated by the authors. The first was for performance rating, the second was for self-assessment (perception of the residents of their quality in performing the procedure), and the third was for their satisfaction with performing the procedure as a daily practice. The assessment forms were developed through collaborative work between the radiology and the medical education departments at King Abdulaziz University, which has a leader in medical simulation. The performance and self-assessment forms contained a checklist with a scale from zero to nine to evaluate how correctly the residents used the transvaginal ultrasound probe to obtain standard views of the normal anatomic female pelvic structures. The satisfaction form is a checklist with a five-point Likert scale evaluating the resident’s level of confidence, perceived safety level while performing the transvaginal ultrasound on real patients, satisfaction with the quality of images obtained, and with the time needed to complete the study. Piloting of the assessment forms was performed to ensure their validity. To ensure the content validity, the checklist was evaluated by four experts; two from the radiology department and two from the gynecology department. To ensure face validity, the tools of assessment 235
AHMAD ET AL
Academic Radiology, Vol 22, No 2, February 2015
TABLE 2. Reliability Analysis of the Results of the Assessment Tools
Scale Performance assessment tool before the simulation session Performance assessment tool after the simulation session Self-assessment tool before the simulation session Self-assessment tool after the simulation session Satisfaction assessment tool before the session Satisfaction assessment tool after the session
Cronbach Alpha 0.940 0.906 0.934 0.950 0.916 0.933
were distributed to the residents to collect their perceptions. The reliability of the assessment tools was calculated using the Cronbach alpha correlation equation (12) (Table 2). Study Population
The sample population began with 55 radiologic Saudi residents (30 junior and 25 senior residents), all from the western region (which includes five different cities). This sample represented about 65% of all residents. The study excluded any residents who had any extracurricular special training on performing transvaginal ultrasound. After applying the exclusion criteria, we found 25 junior and 20 senior residents who met the criteria and were included in the study. Formulating the Simulation Sessions
The sessions were all conducted in the skill laboratory at King Abdulaziz University. All sessions were facilitated by an expert with 20 years of experience. The learning objectives of the sessions were to master the introduction and the manipulation of the transvaginal ultrasound probe and to learn how to obtain the standard views of a normal female pelvis. The transvaginal ultrasound simulation model (blue Phantom@) was accompanied by an ultrasound machine (Philips IU22@ which is the same machine that residents use in real life) that has a high-frequency transvaginal probe. The duration of each session was 3–4 hours, and each resident participated in at least one session. The quality of the sessions was influenced by the theory of conceptual framework by Kneebone for evaluating clinical simulation for learning procedure skills (13). Kneebone stated that the simulation session should: 1) Allow for sustained deliberate practice within a safe environment, ensuring that recently acquired skills have been consolidated within a defined curriculum that assures regular reinforcement. 2) Provide access to expert tutors when appropriate, ensuring that such support fades when no longer needed. 236
3) Map onto real-life clinical experience, ensuring that learning supports the experience gained within communities of actual practice. 4) Provide a supportive, motivational, and learner-centered milieu that is conducive to learning. Data Collection
Multiple sets of data were collected, including: 1) Pretraining and post-training self-assessment and satisfaction levels of junior residents. 2) Self-assessment and satisfaction levels of senior residents. 3) Pretraining and post-training performance study of junior residents, which was performed by an expert different from the facilitator of the simulation session. The postassessment evaluation was performed about 1 month after the session. Both sessions were performed on routine booked patients after informed consents. 4) Number of calls to supervisors to perform transvaginal ultrasound because of the lack of confidence of a resident. 5) Rate of repeated studies due to inappropriate technique before and after the sessions. Statistical Analysis of Data
The data collected from the rating forms were statistically analyzed using the Statistical Package of Social Sciences version 16.0 (SPSS, Inc., Chicago, IL). The ordinal data of the assessment tools were converted to a scale format, which was then studied using a parametric paired t test for the comparison between the junior residents’ performance presession and postsession and an independent t test for comparison between the perceptions of junior and senior residents. Significance was set at the 95% confidence interval. The effect size was calculated using the Cohen d equation: d = X2 X1/Saverage where X2 X1 is the mean difference between the two groups and Saverage is the average of both standard deviations. Ethical Approval
Approval from the Residents’ Training Program director was obtained along with approval from the research ethics committee of the hospital, with the reference number (1238 – 13).
RESULTS Tables 3 and 4 compare the self-assessment and satisfaction results of the junior and senior residents. The mean scores on the self-assessment of each element of the performance and satisfaction of the junior residents who had the simulation sessions are higher than those of the senior residents, showing a large effect size (d is >0.5). However, there were
Academic Radiology, Vol 22, No 2, February 2015
TRANSVAGINAL ULTRASOUND SIMULATION
TABLE 3. Comparison Between the Junior Residents Who Have Attended the Simulation Training Session and the Senior Residents Who Did Not Attend the Simulation Session, Regarding Their Perception of Their Performance of Transvaginal Ultrasound (Selfassessment) Question I am (I become) familiar with TV probe manipulation I can obtain two views of the uterus I can assess the endometrium I can obtain two views of the ovaries/adnexa I can assess the cervix I can look for free fluid
Juniors (n = 25); m s
Seniors (n = 20); m s
Cohen d
P Value
t
7.4 1.353 8 1.214 7.7 1.302 7.25 1.209 7.05 1.504 7.4 1.759
4.72 2.951 4.68 2.996 5.2 3.175 3.28 2.301 3.36 2.580 5.08 3.040
0.74 0.92 0.65 1.37 1.1 0.6
.001 .001 .002 .001 .001 .004
3.751 4.651 3.3 6.975 5.665 3.027
TV, transvaginal.
TABLE 4. Comparison Between the Junior Residents Who Have Attended the Simulation Training Session and the Senior Residents Who Did Not Attend the Simulation Session Regarding Their Satisfaction of Their Performance of Transvaginal Ultrasound Question I am satisfied about time I need to perform TVUS I am confident to perform TVUS on real patients I am a safe resident obtaining standard views of the female pelvic organs by TVUS I am satisfied with the quality of the images I obtain I feel I can perform TVUS as a routine study
Juniors (n = 25); m s
Seniors (n = 20); m s
Cohen d
P value
t
3.7 1.031 3.75 0.910 3.85 0.875
2.48 1.005 2.48 1.229 2.72 1.308
0.8 0.75 0.65
.001 .001 .002
4 3.85 3.313
3.7 0.923 3.95 0.759
2.4 1.118 2.16 1.028
0.82 1.72
.001 .001
4.181 6.493
TVUS, transvaginal ultrasound.
TABLE 5. Comparison Between the Objective Assessment of Junior Residents’ Performance of Transvaginal Ultrasound Before and After the Simulation Training Session
Question The resident handles the TV probe correctly inside the vaginal The resident obtains two good views of the uterus The resident obtains two good views of the ovaries/ adnexa The resident obtains good views of the endometrium The resident obtains good views of the cervix The resident looks for free fluid Overall, the study is completed
Juniors Before Simulation (n = 25); m s
Juniors After Simulation (n = 25); m s
5.5 3.379
7.75 2.0488
Cohen d
P Value
0.6
.001
3.851
t
5.6 3.761 4.3 3.373
8.0500 1.877 7.3 2.904
0.65 0.65
.002 .001
3.611 4.359
5.5 3.954 4.05 4.11 5.85 4.404 4.9 3.354
7.9 2.573 7.7 2.83 8.1 2.77 7.6 2.037
0.52 0.74 0.45 0.3
.005 .001 .021 .001
3.177 3.822 2.517 4.925
TV, transvaginal.
no significant differences between the scores of the senior residents and the junior residents before they had the simulation sessions. Tables 5–7 compare the performance, self-assessment, and satisfaction of the junior residents’ presimulation and postsimulation sessions. There was an increase in the mean scores of the rating tools with a large effect size in all evaluated elements. The satisfaction from the simulation session was high, with a satisfaction index of 0.85. The total number of calls asking supervisors to come to the hospital to perform transvaginal ultrasound decreased from four calls per month to one call per month. Also, the rate of repeated examinations because of the inappropriate technique
was reduced from eight cases per month to one case per month. DISCUSSION The study shows a significant improvement in the performance and confidence levels of junior residents after the simulation sessions. It also shows higher perceived levels of performance and more satisfaction among junior residents at a statistically significantly higher level than among senior residents. This means that the sessions not only improved the performance of the junior residents but also increased their levels of confidence above those of the senior residents. The insignificant differences between the senior residents 237
AHMAD ET AL
Academic Radiology, Vol 22, No 2, February 2015
TABLE 6. Comparison Between the Junior Residents’ Subjective Self-assessment of Their Performance in Transvaginal Ultrasound Before and After the Simulation Training Session
Question
Juniors Before Simulation (n = 25); m s
Juniors After Simulation (n = 25); m s
Cohen d
P Value
5.15 2.925 3.250 7.4 3.7 3.262 4.35 3.117 3.45 2.625 2.7 2.536 3.8 3.302
7.4 1.875 7.4 1.353 8 1.214 7.7 1.302 7.25 1.209 7.05 .336 7.4 1.759
0.7 0.82 1.5 1.2 1.5 2.7 1
.005 .001 .001 .001 .001 .001 .001
I can introduce TV probe in the correct position I am familiar with TV probe manipulation I can obtain two views of the uterus I can assess the endometrium I can obtain two views of the ovaries/adnexa I can assess the cervix I can look for free fluid
t 3.167 7.625 6.015 5.998 6.418 7.257 6.136
TV, transvaginal.
TABLE 7. Comparison Between the Junior Residents Regarding Their Satisfaction About Their Performance of Transvaginal Ultrasound Before and After the Simulation Training Session
Question I am satisfied with the time needed to perform TVUS I am confident to perform TVUS on real patients I am a safe resident obtaining standard views of female pelvic organs I am satisfied about the quality of the images that I obtain I feel I can perform TVUS as a routine study
Juniors Before Simulation (n = 25); m s
Juniors After Simulation (n = 25); m s
1.95 1.190 2.15 1.348 2.3 1.418
3.7 1.031 3.75 .910 3.85 .875
2.2 1.563 2.2 1.152
Cohen d
P Value
t
1 1 1
.001 .001 .001
5.872 5.446 4.610
3.7 .923
0.9
.001
5.090
3.95 1.698
0.8
.001
6.054
TVUS, transvaginal ultrasound.
and the junior residents before the simulation sessions indicate that both groups came from the same pool and decrease the chances that the junior residents had better performance than the senior residents from the beginning. This, accompanied by the improvement in residents’ confidence to perform the examination during their on-call time, showed performance improvement, measured by a decrease in the rate of repeated examinations.
CONCLUSION Transvaginal ultrasound simulation sessions for the residency program improved the perceived performance and confidence levels of the residents, improving patient safety. They also created a solution for the ethical problem of using a training program to teach residents. Integrating simulation sessions will be recommended for different ultrasound examinations, and the effectiveness of this method will require further studies.
LIMITATIONS At the time of the study, we did not objectively assess the senior residents as the residents themselves felt that this would necessitate losing their anonymity. However, the residents could have completed objective preassessments and postassessments and remain anonymous by recruiting an expert from outside the training program and by having a code number associated with them instead of a name. Future studies could be carried out using this approach. Another possible limitation could have been due to previous training of the junior residents. The exclusion criteria eliminated this confounder, though, by asking the residents if they had received any previous training. Also, the costeffectiveness of the sessions was not calculated as they were free of charge, including the simulation machine and the expert, and the estimation of the cost was not easily calculated. 238
ACKNOWLEDGMENTS The authors want to deliver their deepest thanks to Dr Reda Jamjoom and all his team of Skill lab at King Abdulaziz University. Also the authors are very grateful from all the hard work and efforts made by the senior sonographer Mrs. Rosalinda Huertazuela. REFERENCES 1. Akins Ralitsa, Ho Hoi. Clinical simulation in modern teaching and training of sonography in obstetrics and gynecology. Donald School Journal of Ultrasound in Obstetrics and Gynecology October-December 2009; 3(4): 17–21. 2. Williams CJ, Edie JC, Mulloy B, et al. Ultrasound 2013,Transvaginal ultrasound simulation and its effect on trainee confidence levels: a replacement for initial clinical training? Ultrasound 2013;1–7. http://dx.doi.org/10.1177/17 42271X13481215.
Academic Radiology, Vol 22, No 2, February 2015
3. Gould DA, Reekers JA, Kessel DO, et al. Simulation devices in interventional radiology: validation pending. J Vasc Interv Radiol 2006; 17:215–216. 4. The Society and College of Radiographers. Developing and growing the sonographer workforce: education and training needs. London: SCoR, 2009. 5. Sarmiento J, Stewart K, Aguila J, et al. Pelvic ultrasound simulation training models and case scenarios. Donald School J Ultrasound Obstet Gynecol 2014; 8(1):22–30. 6. Gaba DM. Simulators in anesthesiology, vol. 14. St Louis, MO: MosbyYearbook, 1997; 55–91. 7. Pasquale Susan J. The comprehensive textbook of healthcare simulation: education and learning theory. New York: Springer Science+Business Media, 2013; 51–52. 8. Ziv A. Simulators and simulation-based medical education. In: Dent JA, Harden RM, eds. A practical guide for medical teachers. Edinburgh: Elsevier, 2009.
TRANSVAGINAL ULTRASOUND SIMULATION
9. Ziv Amitai, Small Stephen D, Wolpe Paul Root. Patient safety and simulation-based medical education. Medical teacher 2000; 22(5). 10. McGaghie WC, Siddall VJ, Mazmanian PE, et al., American College of Chest Physicians Health and Science Policy Committee. Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009; 135(3 Suppl):62S–68S. 11. Kirkpatrick D, Kirkpatrick J. Transferring learning to behavior: Using the four levels to improve performance. Berrett-Koehler Publishers; 16–31. Available at: http://site.ebrary.com/id/10221992?ppg=16; 2005. Last access 12-02-2014. 12. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951; 16:297–334. 13. Kneebone R. Evaluating clinical simulations for learning procedural skills: a theory-based approach. Acad Med 2005; 80:549–553.
239
