Insights
Significance in medical education research Ronald Markert Department of Internal Medicine, Wright State University, Boonshoft School of Medicine, Dayton, Ohio, USA If you performed well on your second-year OSCE, you were more likely to score satisfactorily on USMLE Step 2
‘When I use a word’, Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean – neither more nor less’.1
literature, clinical teachers should be cautious. Statistical significance is reserved for a relationship between two variables that is unlikely to result from chance (i.e. typically less than a 5% chance: p < 0.05).
Medical educators should take care not to follow the practice of Lewis Carroll’s famous anthropomorphic egg. When reading the word significance (or significant or significantly) in the medical education
For example, correlation is a frequently reported statistic in medical education research. When two variables move in tandem, their relationship is described as positive (between
0.00 and +1.00). Simon et al. found a significant positive correlation (0.395, p < 0.001) between second-year medical student Objective Structured Clinical Examination (OSCE) scores and United States Medical Licensing Examination (USMLE) Step–2 scores.2 If you performed well on your second-year OSCE, you were more likely to score satisfactorily on USMLE Step 2. In contrast, when one variable increases as the other
208 © 2015 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2015; 12: 208–209
tct_12268.indd 208
5/4/2015 4:27:46 PM
decreases, the correlation is negative (between –1.00 and 0.00). Hegazi and Wilson found that for medical students there is a significant negative correlation between age and moral judgement competence (–0.362, p < 0.001).3
real-world meaning? In pragmatic terms, is faculty-member guided training really superior to self-guidance for teaching ultrasonography to new residents when the considerably greater cost of the faculty membermediated approach is considered?
Unhappily, older medical students, who usually have more clinical experience, were found to have less moral reasoning than their younger peers.
Conversely, sometimes practically significant differences are not statistically significant. Recently, our orthopaedics faculty members posited that the mandatory 80–hour workweek for US residents, which began in 2003, would lead to higher scores on the annual Orthopaedics InTraining Examination (OITE). The thought process being that with more ‘free’ time, residents had more time to study. We found that the pre-2003 residents (no 80–hour workweek restriction) averaged the 64th percentile on the OITE, whereas the post-2003 residents averaged the 71st percentile. From our experience this seven-percentile increase was a meaningful improvement, but the difference was not statistically significant. In this case we were hindered by the limitation of having a small number of available graduates in the two time periods (20 and 16, respectively).
In addition to the modifier statistically, readers should understand the concept of practical significance, which indicates that a relationship or difference between groups is meaningful in a real-world education setting. For an ultrasound curriculum, Alba et al. randomly assigned 72 internal medicine interns to faculty member-guided training or self-guidance in a 4–hour simulation lab involving humans and manikins.4 Using a 19–item OSCE checklist (yes/no), both groups scored close to the maximum of 19: with a faculty memberguided mean of 18.33 versus self-guidance mean of 17.43. As p = 0.008, the authors concluded that faculty member-guided training ‘was superior…in…skill acquisition’; however, does a difference between groups of one checklist item have practical
Readers of The Clinical Teacher and other health sciences education journals should understand the difference
between statistical significance and practical significance. A more in-depth account of this topic is available elsewhere.5 In their training, medical educators are advised to adopt teaching and curricular innovations that are supported by evidence from high-quality research. Determining when a statistically significant difference between instructional methods may be too small to result in a notable improvement among learners is an important aspect of practising evidence-based medical education.
Practical significance ... indicates that a relationship or difference between groups is meaningful in a real-world education setting
REFERENCES 1.
Carroll L. Through the LookingGlass, and What Alice Found There. Raleigh, NC: Hayes Barton Press, 1872; Chapter 6.
2.
Simon S, Bui A, Day S, Berti D, Volkan K. The relationship between second-year medical students’ scores and USMLE Step 2 scores. J Eval Clin Pract 2007;13: 901–905.
3.
Hegazi I, Wilson I. Medical education and moral segmentation in medical students. Med Educ 2013;47:1022–1028.
4.
Alba GA, Kelmenson DA, Noble VE, Murray AF, Currier PF. Faculty staff-guided versus self-guided ultrasound training for internal medicine residents. Med Educ 2013;47:1099–1108.
5.
Rosen B, DeMaria A. Statistical significance vs. practical significance: an exploration through health education. Am J Health Educ 2012;43:235–241.
Corresponding author’s contact details: Ronald Markert, Department of Internal Medicine, Wright State University, Boonshoft School of Medicine, Weber Building, 2nd Floor, 128 East Apple Street, Dayton, Ohio 45409, USA. E-mail: [email protected]
Funding: None. Conflict of interest: None. Acknowledgements: None. Ethical approval: Not required. doi: 10.1111/tct.12268
© 2015 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2015; 12: 208–209 209
tct_12268.indd 209
5/4/2015 4:27:48 PM
Significance in medical education research Ronald Markert Department of Internal Medicine, Wright State University, Boonshoft School of Medicine, Dayton, Ohio, USA If you performed well on your second-year OSCE, you were more likely to score satisfactorily on USMLE Step 2
‘When I use a word’, Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean – neither more nor less’.1
literature, clinical teachers should be cautious. Statistical significance is reserved for a relationship between two variables that is unlikely to result from chance (i.e. typically less than a 5% chance: p < 0.05).
Medical educators should take care not to follow the practice of Lewis Carroll’s famous anthropomorphic egg. When reading the word significance (or significant or significantly) in the medical education
For example, correlation is a frequently reported statistic in medical education research. When two variables move in tandem, their relationship is described as positive (between
0.00 and +1.00). Simon et al. found a significant positive correlation (0.395, p < 0.001) between second-year medical student Objective Structured Clinical Examination (OSCE) scores and United States Medical Licensing Examination (USMLE) Step–2 scores.2 If you performed well on your second-year OSCE, you were more likely to score satisfactorily on USMLE Step 2. In contrast, when one variable increases as the other
208 © 2015 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2015; 12: 208–209
tct_12268.indd 208
5/4/2015 4:27:46 PM
decreases, the correlation is negative (between –1.00 and 0.00). Hegazi and Wilson found that for medical students there is a significant negative correlation between age and moral judgement competence (–0.362, p < 0.001).3
real-world meaning? In pragmatic terms, is faculty-member guided training really superior to self-guidance for teaching ultrasonography to new residents when the considerably greater cost of the faculty membermediated approach is considered?
Unhappily, older medical students, who usually have more clinical experience, were found to have less moral reasoning than their younger peers.
Conversely, sometimes practically significant differences are not statistically significant. Recently, our orthopaedics faculty members posited that the mandatory 80–hour workweek for US residents, which began in 2003, would lead to higher scores on the annual Orthopaedics InTraining Examination (OITE). The thought process being that with more ‘free’ time, residents had more time to study. We found that the pre-2003 residents (no 80–hour workweek restriction) averaged the 64th percentile on the OITE, whereas the post-2003 residents averaged the 71st percentile. From our experience this seven-percentile increase was a meaningful improvement, but the difference was not statistically significant. In this case we were hindered by the limitation of having a small number of available graduates in the two time periods (20 and 16, respectively).
In addition to the modifier statistically, readers should understand the concept of practical significance, which indicates that a relationship or difference between groups is meaningful in a real-world education setting. For an ultrasound curriculum, Alba et al. randomly assigned 72 internal medicine interns to faculty member-guided training or self-guidance in a 4–hour simulation lab involving humans and manikins.4 Using a 19–item OSCE checklist (yes/no), both groups scored close to the maximum of 19: with a faculty memberguided mean of 18.33 versus self-guidance mean of 17.43. As p = 0.008, the authors concluded that faculty member-guided training ‘was superior…in…skill acquisition’; however, does a difference between groups of one checklist item have practical
Readers of The Clinical Teacher and other health sciences education journals should understand the difference
between statistical significance and practical significance. A more in-depth account of this topic is available elsewhere.5 In their training, medical educators are advised to adopt teaching and curricular innovations that are supported by evidence from high-quality research. Determining when a statistically significant difference between instructional methods may be too small to result in a notable improvement among learners is an important aspect of practising evidence-based medical education.
Practical significance ... indicates that a relationship or difference between groups is meaningful in a real-world education setting
REFERENCES 1.
Carroll L. Through the LookingGlass, and What Alice Found There. Raleigh, NC: Hayes Barton Press, 1872; Chapter 6.
2.
Simon S, Bui A, Day S, Berti D, Volkan K. The relationship between second-year medical students’ scores and USMLE Step 2 scores. J Eval Clin Pract 2007;13: 901–905.
3.
Hegazi I, Wilson I. Medical education and moral segmentation in medical students. Med Educ 2013;47:1022–1028.
4.
Alba GA, Kelmenson DA, Noble VE, Murray AF, Currier PF. Faculty staff-guided versus self-guided ultrasound training for internal medicine residents. Med Educ 2013;47:1099–1108.
5.
Rosen B, DeMaria A. Statistical significance vs. practical significance: an exploration through health education. Am J Health Educ 2012;43:235–241.
Corresponding author’s contact details: Ronald Markert, Department of Internal Medicine, Wright State University, Boonshoft School of Medicine, Weber Building, 2nd Floor, 128 East Apple Street, Dayton, Ohio 45409, USA. E-mail: [email protected]
Funding: None. Conflict of interest: None. Acknowledgements: None. Ethical approval: Not required. doi: 10.1111/tct.12268
© 2015 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2015; 12: 208–209 209
tct_12268.indd 209
5/4/2015 4:27:48 PM
