M ILITARY M EDICINE, 180, 4:31, 2015
Assessing Task Importance and Anxiety in Medical School: An Instrument Development and Initial Validation Study CDR Henry L. Phillips IV, MSC USN*; Ting Dong, PhDf; Steven J. Durning, MD, PhDf; CDR Anthony R. Artino Jr., MSC USNf
ABSTRACT Recent research in medical education suggests that students’ motivational beliefs, such as their beliefs about the importance of a task, and their emotions are meaningful predictors of learning and performance. The primary purpose of this study was to develop a self-report measure of “task importance” and “anxiety” in relation to several medical education competencies and to collect validity evidence for the new measures. The secondary purpose was to evaluate differences in these measures by year of medical school. Exploratory factor analysis of scores from 368 medical school students suggested two task importance factors and three anxiety factors. The task importance and anxiety subscales were weakly related to each other and exhibited consistently negative and positive correlations, respectively, with three self-efficacy subscales. The task importance subscales were positively related to “metacognition,” whereas “interpersonal skills anxiety” and “health knowledge anxiety” were positively related to “procrastination.” All three anxiety factors were positively related to “avoidance of help seeking,” whereas “inter personal skills and professionalism importance” was negatively related to help avoidance behaviors. Finally, com parisons across the 4 years of medical school indicated that some aspects of task importance and anxiety varied significantly. Overall, findings from this study provide validity evidence for the psychometric quality of these scales, which capture task importance and anxiety in medical students. Limitations and implications for medical education research are discussed.
INTRODUCTION In recent years, more attention has been given to the influence of affective factors, such as motivational beliefs and emo tions, on academic outcomes in educational research.1'2 Find ings from a host of contemporary education studies suggest that student affect explains considerable variance in aca demic outcomes, above and beyond the effects of cognitive factors such as aptitude and prior achievement.3-5 Based, in part, on his own empirical work, Pekrun6 proposed a theoreti cal model of affect to help explain these findings. Pekrun’s control-value theory of achievement em otions4 depicts a reciprocal relationship between the learning environment, personal factors (such as cognition, motivation, and em o tion), and academic outcomes. Pekrun proposed that the learning environment influences students’ appraisals of the “control” they have over their learning environment, as well as the “value” they attach to tasks and activities in that environment. This value appraisal dictates the subjec tive importance attached to the achievement of academic goals. A perceived lack of control, combined with a low
*Naval Air Warfare Center Training Systems Division, 12350 Research Parkway, Orlando, FL 32826-3275. tDepartment of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. This study was approved by the Institutional Review Board of the Uniformed Services University of the Health Sciences, Bethesda, Maryland. The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, Department of the Navy, Department of Defense, nor the U.S. Government. doi: 10.7205/MILMED-D-14-00557
MILITARY MEDICINE, Vol. 180, April Supplement 2015
degree of task importance, can result in maladaptive emo tions, avoidance of help seeking, and procrastination, among other negative outcomes, all of which can contribute to low academic achievement. Conversely, a high degree of control combined with high levels of task importance can lead to adaptive emotions, higher levels of achievement motivation, and greater self-efficacy. Other researchers have examined motivational beliefs, such as task value and self-efficacy, as predictors of dis crete achievement emotions, such as enjoyment and anxi ety.7 These achievement emotions then influence various academic outcomes, such as course grades, continuing motivation to learn.x' 1 and the use of adaptive learning strategies (such as m etacognition10) or maladaptive learn ing strategies (such as procrastination" and avoidance of help seeking12). Within medical education, several scholars have recently called for more research into the role of motivational beliefs and emotions in medical training.5'9' 1-’ To date, however, empirical studies to directly examine theoretical models of motivation and emotion, such as Pekrun’s control-value theory, have been scarce in medical education. In one recent study, Artino et al2 explored the complex interplay between the learning environment, personal factors, and academic outcomes using a sample of second-year medical students in a year-long clinical reasoning course. The authors investi gated two types of motivational beliefs. The first was task value, which has been defined as the degree to which indi viduals find a task interesting, important, and useful.14 The second motivational belief was self-efficacy, which has been defined as students’ judgm ents of their capabilities
31
Assessing Task Importance and Anxiety in Medical School
to successfully perform specific academic tasks.15 Results from this work revealed that task value and self-efficacy beliefs accounted for considerable variance in students’ achievement outcomes. The primary purpose of this study was to develop a quan titative self-report measure of task value beliefs (in par ticular, “task importance,” a specific type of task value) and “anxiety,” an essential negative emotion, in relation to the six core competencies of the Accreditation Council for Graduate Medical Education (ACGME). Because we also sought to compare task importance and anxiety at dif ferent phases of medical education, we chose to develop initial survey items based on these six core competencies, which students are expected to achieve by the end of residency training. Our target population was medical stu dents at various stages of undergraduate medical education at one institution. Once developed, we gathered two types of validity evi dence for the survey results: (1) evidence based on internal structure and (2) evidence based on relations to other vari ables relevant to the control-value theory framework.4 A priori to any evaluations of dimensionality, we posited the following hypotheses based on the theoretical underpinnings of control-value theory: (HI) Task importance factors will be positively cor related with two adaptive factors (self-efficacy and metacognition), and negatively correlated with two maladaptive factors (procrastination and avoidance of help seeking). (H2) Anxiety factors will be negatively correlated with two adaptive factors (self-efficacy and metacog nition), and positively correlated with two mal adaptive factors (procrastination and avoidance of help seeking). The secondary purpose of the current study was to explore potential differences in task importance and anxiety from year 1 of medical school to year 4. Based on the findings of Artino et allfi in their development of several self-efficacy scales, we had the following hypotheses: (H3) Task importance will be higher among more senior students. (H4) Anxiety will be lower among more senior students. As medical students progress through the curriculum, they develop greater efficacy (e.g., Artino et a l16 found seniority related to higher scores on “patient care selfefficacy” and “evidence-based medicine self-efficacy”), and in turn these students are expected to increase the degree of importance they attach to the ACGME compe tencies, upon which scale items were based. By the same token, progress through the medical school curriculum and development as a practitioner are expected to result in less anxiety as students’ confidence in attaining the ACGME competencies increases.
32
Similarly, as students develop greater skill and a deeper understanding of their roles as caregivers, it is expected that they will attach greater importance to, and perceive greater value in, the goals of their academic programs. More senior students are expected to better understand why their assigned academic and practitioner goals are structured as they are, and are more likely to agree with those goals as experience broadens their perspectives.
METHODS Study Context This study was conducted at the F. Edward Hebert School of Medicine, Uniformed Services University of the Health Sciences (USU) under the auspices of the Long-Term Career Outcome Study. At the time of the study, USU matricu lated roughly 170 medical students per year and offered a traditional 4-year curriculum, structured as 2 years of basic sciences coursework followed by 2 years of clinical rota tions (clerkships).
Item Development and Content Validation Initial survey items were written to align with the content of the six core competencies defined by the ACGME17: (1) patient care, (2) medical and population health knowl edge, (3) interpersonal and communication skills, (4) practicebased learning and improvement (i.e., understanding practice and ways to improve practice, such as through quality assur ance), (5) professionalism, and (6) systems-based practice (i.e., understanding issues of access and use of resources in health care17). Since one of our goals was to evaluate medical stu dents’ motivational beliefs and emotions as they related to their medical knowledge and skills at different time points in their medical school education, we developed items targeting the ACGME competencies, which students are expected to achieve by the end of residency training. It is worth noting that at the time of this study, USU’s learning objectives for all medical students were structured around the six ACGME core competencies. After developing a library of draft items, we recruited five senior physician-educators, each with more than 10 years of experience in medical education, to provide content validity evaluations. Using a standardized content validation form,18 we individually asked each physicianeducator to indicate the degree to which the items addressed the targeted constructs. In addition, each physician-educator was asked to provide editorial recommendations for item improvement or elimination. The result of this effort was a list of 19 behavioral items related to the six ACGME core competencies. Next, the 19 items were organized into a survey instrument, which asked respondents to rate their perceptions of self-efficacy, task importance, and anxiety as they related to each of the 19 ACGME-framed items (for a total of 57 items). Details of validation of the selfefficacy subscale results, using a smaller subsample, are
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Task Importance and Anxiety in Medical School TA B LE I.
Results for the Task Importance Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; X = 0; N = 368) Factors Item
C
1
2
3
4
5
6
IM P-1 Apply knowledge of normal function to each of the major organ systems? IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits? IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence? IMP-4 Use effective listening skills when interacting with a patient? IMP-5 Demonstrate caring when counseling a patient? IMP-6 Accurately gather essential information from a patient? IMP-7 Perform a thorough physical exam? IMP-8 Develop an appropriate differential diagnosis? IMP-9 Generate a patient-specific treatment plan? IMP-10 Use information technology to support patient-care decisions? IMP-11 Work effectively with other health care professionals to provide high-quality patient care? IMP-12 Improve clinical practice using a systematic approach? IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems? IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals? IMP-15 Demonstrate sensitivity to patients’ cultural differences? IMP-16 Balance professional responsibilities with personal responsibilities? IMP-17 Discuss methods of controlling health care costs? IM P-18 Practice cost-effective health care delivery that does not compromise quality of care? IMP-19 Apply high-quality health care in deployed military environments?
0.55
0.31 (0.54)
-0.04 (-0.03)
0.46(0.66)
-0.10 (-0.12)
-0.12 (-0.15)
-0.17 (-0.28)
0.77
0.79 (0.84) -0.04 (0.03)
0.06 (0.47)
-0.03 (-0.04)
-0.03 (-0.02)
-0.23 (-0.33)
0.59
0.62 (0.73)
0.03 (0.07)
0.21 (0.52)
-0.04 (-0.05)
-0.02 (-0.05)
-0.07 (-0.18)
0.77
-0.08 (0.35)
0.01 (-0.02)
0.89 (0.87)
0.00 (-0.10)
-0.03 (-0.19)
-0.09 (-0.21)
0.7
-0.10(0.32)
0.03 (-0.01)
0.89 (0.83)
0.05 (-0.08)
0.03 (-0.14)
0.05 (-0.07)
0.77
0.11 (0.48)
-0.02 (-0.04)
0.7 4 (0.83) -0.01 (-0.11)
0.00 (-0.14)
-0.27 (-0.39)
0.54 0.77
0.18 (0.47) 0.64 (0.75)
0.00 (-0.05) -0.09 (-0.03)
0.5 7 (0.69) -0.07 (-0.16) 0.14(0.50) 0.03 (-0.03)
0.03 (-0.05) 0.02 (-0.02)
-0.20 (-0.29) -0.43 (-0.52)
0.84
0.87 (0.86) -0.10 (-0.01)
0.00 (-0.03)
0.00 (0.03)
-0.30 (-0.39)
0.65
0.76 (0.80) -0.06 (0.03)
0.05(0.40)
0.11 (0.05)
0.07 (0.04)
-0.10 (-0.19)
0.6
0.50 (0.69)
0.04 (0.06)
0.37 (0.62)
-0.04 (-0.07)
-0.01 (-0.07)
-0.10 (-0.21)
0.66
0.74 (0.80)
0.07 (0.12)
0.10(0.46)
-0.01(0.00)
0.01 (-0.02)
-0.06 (-0.16)
0.72
0.81 (0.85)
0.01 (0.08)
0.09 (0.45)
0.03 (0.00)
0.02 (0.02)
0.06 (-0.05)
0.67
0.79 (0.80)
0.06 (0.11)
0.06(0.40)
0.01 (-0.01)
0.05 (0.05)
0.15 (0.05)
0.5
0.09 (0.37)
0.02 (0.00)
0.66 (0.66)
0.02 (-0.07)
-0.01 (-0.10)
0.24 (0.14)
0.35
0.05 (0.29)
-0.05 (-0.08)
0.56(0.57)
-0.05 (-0.14)
-0.03 (-0.06)
0.14(0.05)
0.82
0.90 (0.83)
0.06 (0.16)
-0.08 (0.29)
0.05 (0.07)
-0.01 (0.02)
0.34 (0.24)
0.79
0.92 (0.86)
0.02 (0.12)
-0.07 (0.33)
0.02 (0.04)
-0.03 (0.01)
0.22 (0.11)
0.59
0.78 (0.76)
0.04(0.11)
-0.03 (0.35)
-0.06 (-0.01)
-0.07 (-0.03)
0.01 (-0.09)
-0.07 (0.38)
C, Communality; Imp, Task Importance. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) >0.30 on only one factor, and communalities >0.40. Results presented above reflect loadings and communalities for Task Importance items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously.
provided in the work of Artino et al.16 Lists of the task importance and anxiety items are presented in Tables I and II. P a rtic ip a n ts a n d P ro c e d u re s
Participants were recruited from the population of USU medical school students during the 2010-2011 and 2011-
MILITARY MEDICINE, Vol. 180, April Supplement 2015
2012 academic years ( N = 864). In May 2011 and May 2012, all students were contacted by e-mail and invited to complete the online survey of their per ceptions of self-efficacy, task importance, and anxiety. Two follow-up e-mail reminders were sent out over the next 4 weeks, at which point the response period was closed. Participation in the survey was voluntary, and
33
Assessing Task Importance and Anxiety in Medical School TA B LE II.
Results for the Anxiety Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; delta = 0; N = 368) Factors Item
C
1
2
5
6
ANX-1 Applying knowledge of normal function to each of the major organ systems? ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits? ANX-3 Applying knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence? ANX-4 Using effective listening skills when interacting with a patient? ANX-5 Demonstrating caring when counseling a patient? ANX-6 Accurately gathering essential information from a patient? ANX-7 Performing a thorough physical exam? ANX-8 Developing an appropriate differential diagnosis? ANX-9 Generating a patient-specific treatment plan? ANX-10 Using information technology to support patient-care decisions? ANX-l 1 Working effectively with other health care professionals to provide high-quality patient care? ANX-12 Improving clinical practice using a systematic approach? ANX-13 Evaluating evidence from scientific studies relevant to your patients’ health problems? ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals? ANX-15 Demonstrating sensitivity to patients’ cultural differences? ANX-16 Balancing professional responsibilities with personal responsibilities? ANX-17 Discussing methods of controlling health care costs? ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care? ANX-19 Applying high-quality health care in deployed military environments?
0.51
-0.02 (-0.05)
0.06 (0.37)
-0.03 (0.02)
0.18 (0.46)
- 0.59 ( - 0.69)
0.00 (-0.08)
0.55
0.01 (0.02)
0.08 (0.35)
0.05 (0.16)
0.01 (0.34)
- 0.69 ( - 0.73)
0.02 (-0.08)
0.48
0.01 (0.02)
v0.31 (0.53)
0.00 (0.04)
0.09 (0.41)
-0.46 (-0.62)
0.02 (-0.05)
0.81
0.01 (0.01)
0.10(0.44)
-0.01 (-0.15)
0.88 (0.89)
0.07 (-0.35)
0.03 (0.01)
0.79
0.04 (0.04)
0.11 (0.42)
-0.01 (-0.15)
0.90 ( 0.88)
0.15 (-0.28)
0.01 (-0.01)
0.70
0.01 (-0.02)
-0.11 (0.30)
-0.01 (-0.06)
0.71 (0.79)
-0.31 (-0.56)
0.07 (0.01)
0.50 0.76
-0.01 (-0.03) 0.02 (-0.01)
-0.15 (0.22) 0.00 (0.35)
0.00 (0.00) 0.00 (0.14)
0.51 (0.62) -0.01 (0.37)
-0.39 (-0.55) - 0.88 ( - 0.87)
0.01 (-0.05) 0.04 (-0.07)
0.67
0.02 (0.06)
0.18 (0.44)
0.06 (0.21)
-0.11 (0.28)
- 0.76 ( - 0.80)
-0.04 (-0.15)
0.52
-0.09 (-0.03)
0.59 (0.68)
0.02 (-0.01)
0.02 (0.36)
-0.23 (-0.48)
0.00 (-0.03)
0.50
-0.06 (-0.02)
0.39 (0.58)
-0.01 (-0.07)
0.37 (0.58)
-0.09 (-0.42)
-0.14 (-0.16)
0.64
-0.07 (0.01)
0.62 (0.75)
0.03 (-0.02)
0.19 (0.50)
-0.14 (-0.49)
-0.08 (-0.11)
0.66
0.03 (0.10)
0.63 (0.72)
-0.05 (-0.01)
0.07 (0.41)
-0.13 (-0.45)
-0.33 (-0.36)
0.61
0.08 (0.16)
0.62 (0.69) -0.03 (0.05)
0.01 (0.34)
-0.15 (-0.43)
-0.30 (-0.33)
0.61
0.10(0.08)
0.28 (0.53)
-0.10 (-0.16)
0.63 (0.73)
0.07 (-0.30)
-0.05 (-0.07)
0.31
-0.09 (-0.06)
-0.02 (0.23)
0.07 (0.04)
0.34 (0.45)
-0.29 (-0.46)
-0.11 (-0.16)
0.70
0.05 (0.10)
0.86 (0.81) -0.02 (-0.07)
0.00 (0.30)
0.13 (-0.19)
0.13 (0.13)
0.70
0.03 (0.08)
0.79 (0.81)
0.00 (-0.05)
0.05 (0.37)
0.01 (-0.31)
0.18 (0.16)
0.37
0.04 (0.08)
0.48 (0.57)
0.02 (0.03)
0.04 (0.32)
-0.19 (-0.40)
0.06 (0.02)
3
4
Anx, Anxiety; C, Communality. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) >0.30 on only one factor, and communalities >0.40. Results presented above reflect loadings and communalities for Anxiety items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously. Results continued from Table I.
ethical approval was obtained from the USU Institutional Review Board. A n a ly s e s
Before analysis, we screened the data for accuracy and missing values and evaluated all item scores for distribu tion normality. To evaluate the representativeness of our sample, we compared the respondents and the nonrespon
34
dents on undergraduate grade point average, Medical College Admissions Test scores, and age using multivariate ana lyses of variance (MANOVAs), as well as on gender using a x2 test of independence. Next, we conducted an exploratory factor analyses on the task importance and anxiety items to evaluate their dimensional structure. These results were used to define subscales representing dimensions of task importance and
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Task Importance and Anxiety in Medical School anxiety related to the ACGME competencies. Internal con sistency estimates and descriptive statistics were computed for all measures, and correlations were computed to assess relations to other variables. Finally, we conducted MANOVA tests to evaluate differences by year group for the task impor tance and anxiety subscales. We followed the MANOVAs with Bonferroni multiple comparisons of differences across year groups for each subscale. All analyses were completed using SPSS 21.0 (IBM Corporation, 2012, New York). Measures Avoidance of Help Seeking Avoidance of help seeking was assessed using a five-item scale developed by Pajares et al.19 These items assessed the extent to which students avoided asking for help even when they need it. Higher scores indicate greater help avoidance, and all items employed a 5-point, Likert-type response scale. Metacognition Metacognition was assessed using an eight-item scale from the Motivated Strategies for Learning Questionnaire.20 These items assessed the frequency with which students used metacognitive control strategies, including planning, goal setting, comprehension monitoring, and performance regulation to evaluate their progress as learners. Higher scores on this scale generally indicate a greater awareness of one’s own thinking. All items employed a 5-point, Likert-type response scale. Procrastination Procrastination was captured using a four-item scale derived by Wolters." These items assessed the extent to which stu dents disengage academically or tend to delay getting started on academic work. Higher scores indicate a greater degree of procrastination. All items employed a 5-point, Likert-type response scale. Self-Efficacy Self-efficacy was assessed using three subscales derived by Artino et al,16 based on variations of the 19-item ACGME competency items used in the present study. For all three scales, higher scores indicated greater self-efficacy, and all items employed 5-point Likert-type response scales. (a) Patient Care Self-Efficacy: This eight-item scale cap tured the extent to which medical students believe they are capable of providing adequate patient care. (b) Interpersonal Skills Self-Efficacy: This three-item scale captured the extent to which medical students believe they are capable of interacting and communicating effectively with patients. (c) Evidence-Based Medicine Self-Efficacy: This threeitem scale assessed the degree to which students believe they are capable of practicing evidence-based medicine.
MILITARY MEDICINE, Vol. 180, April Supplement 2015
RESULTS
Survey responses were obtained from 368 students (43%) out of a total pool of 864 students enrolled during the 2010-2011 and 2011-2012 academic years. Of these, 131 (36%) were enrolled in medical school year 1 (MS-1), 97 (26.6%) in MS-2, 73 (19.8%) in MS-3, and 67 (18.2%) in MS-4. The sample included 102 (28%) female respondents, which is representative of the overall USU student population. Results of the comparisons of respondents versus non respondents revealed the following statistically significant differences: For MS-1 students, the respondents were, on aver age, slightly older than the nonrespondents [F (1) = 15.10, p < 0.001]. For MS-2 students, the respondents were, on average, slightly older than the nonrespondents [F (1) = 7.69, p < 0.05], had lower undergraduate grade point aver ages [F (1) = 11.18, p < 0.05], and had a higher proportion of males [ / (2) = 16.61, p < 0.05], For MS-3 and MS-4 students, there were no differences between the two groups on any of the four measures. We interpreted these results as evidence that the sample of 368 students was sufficiently similar to the overall student population to justify proceed ing with the study. Evidence Based on Internal Structure: Exploratory Factor Analysis Following data screening, we performed a principal axis factor analysis with oblique rotation (delta = 0), on the set of 38 task importance and anxiety items. The correlation matrix was found to be factorable; the Kaiser-Meyer-Olkin Measure of Sampling Adequacy was 0.90, which is con sidered extremely good.21 Bartlett’s test of sphericity ( j 2 = 11136.15, d f = 703, p < 0.01) indicated that the correlation matrix was not an identity matrix, and all measures of sam pling adequacy were deemed sufficient (i.e., >0.60). The scree plot and Eigenvalue criteria indicated a sixfactor solution, with all factors accounting for 63.2% of the total variance in the items. Evaluation of the communalities extracted revealed that two anxiety items (ANX-16 and ANX-19) and one task importance item (IMP-16) had communalities
Assessing Task Importance and Anxiety in Medical School: An Instrument Development and Initial Validation Study CDR Henry L. Phillips IV, MSC USN*; Ting Dong, PhDf; Steven J. Durning, MD, PhDf; CDR Anthony R. Artino Jr., MSC USNf
ABSTRACT Recent research in medical education suggests that students’ motivational beliefs, such as their beliefs about the importance of a task, and their emotions are meaningful predictors of learning and performance. The primary purpose of this study was to develop a self-report measure of “task importance” and “anxiety” in relation to several medical education competencies and to collect validity evidence for the new measures. The secondary purpose was to evaluate differences in these measures by year of medical school. Exploratory factor analysis of scores from 368 medical school students suggested two task importance factors and three anxiety factors. The task importance and anxiety subscales were weakly related to each other and exhibited consistently negative and positive correlations, respectively, with three self-efficacy subscales. The task importance subscales were positively related to “metacognition,” whereas “interpersonal skills anxiety” and “health knowledge anxiety” were positively related to “procrastination.” All three anxiety factors were positively related to “avoidance of help seeking,” whereas “inter personal skills and professionalism importance” was negatively related to help avoidance behaviors. Finally, com parisons across the 4 years of medical school indicated that some aspects of task importance and anxiety varied significantly. Overall, findings from this study provide validity evidence for the psychometric quality of these scales, which capture task importance and anxiety in medical students. Limitations and implications for medical education research are discussed.
INTRODUCTION In recent years, more attention has been given to the influence of affective factors, such as motivational beliefs and emo tions, on academic outcomes in educational research.1'2 Find ings from a host of contemporary education studies suggest that student affect explains considerable variance in aca demic outcomes, above and beyond the effects of cognitive factors such as aptitude and prior achievement.3-5 Based, in part, on his own empirical work, Pekrun6 proposed a theoreti cal model of affect to help explain these findings. Pekrun’s control-value theory of achievement em otions4 depicts a reciprocal relationship between the learning environment, personal factors (such as cognition, motivation, and em o tion), and academic outcomes. Pekrun proposed that the learning environment influences students’ appraisals of the “control” they have over their learning environment, as well as the “value” they attach to tasks and activities in that environment. This value appraisal dictates the subjec tive importance attached to the achievement of academic goals. A perceived lack of control, combined with a low
*Naval Air Warfare Center Training Systems Division, 12350 Research Parkway, Orlando, FL 32826-3275. tDepartment of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814. This study was approved by the Institutional Review Board of the Uniformed Services University of the Health Sciences, Bethesda, Maryland. The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, Department of the Navy, Department of Defense, nor the U.S. Government. doi: 10.7205/MILMED-D-14-00557
MILITARY MEDICINE, Vol. 180, April Supplement 2015
degree of task importance, can result in maladaptive emo tions, avoidance of help seeking, and procrastination, among other negative outcomes, all of which can contribute to low academic achievement. Conversely, a high degree of control combined with high levels of task importance can lead to adaptive emotions, higher levels of achievement motivation, and greater self-efficacy. Other researchers have examined motivational beliefs, such as task value and self-efficacy, as predictors of dis crete achievement emotions, such as enjoyment and anxi ety.7 These achievement emotions then influence various academic outcomes, such as course grades, continuing motivation to learn.x' 1 and the use of adaptive learning strategies (such as m etacognition10) or maladaptive learn ing strategies (such as procrastination" and avoidance of help seeking12). Within medical education, several scholars have recently called for more research into the role of motivational beliefs and emotions in medical training.5'9' 1-’ To date, however, empirical studies to directly examine theoretical models of motivation and emotion, such as Pekrun’s control-value theory, have been scarce in medical education. In one recent study, Artino et al2 explored the complex interplay between the learning environment, personal factors, and academic outcomes using a sample of second-year medical students in a year-long clinical reasoning course. The authors investi gated two types of motivational beliefs. The first was task value, which has been defined as the degree to which indi viduals find a task interesting, important, and useful.14 The second motivational belief was self-efficacy, which has been defined as students’ judgm ents of their capabilities
31
Assessing Task Importance and Anxiety in Medical School
to successfully perform specific academic tasks.15 Results from this work revealed that task value and self-efficacy beliefs accounted for considerable variance in students’ achievement outcomes. The primary purpose of this study was to develop a quan titative self-report measure of task value beliefs (in par ticular, “task importance,” a specific type of task value) and “anxiety,” an essential negative emotion, in relation to the six core competencies of the Accreditation Council for Graduate Medical Education (ACGME). Because we also sought to compare task importance and anxiety at dif ferent phases of medical education, we chose to develop initial survey items based on these six core competencies, which students are expected to achieve by the end of residency training. Our target population was medical stu dents at various stages of undergraduate medical education at one institution. Once developed, we gathered two types of validity evi dence for the survey results: (1) evidence based on internal structure and (2) evidence based on relations to other vari ables relevant to the control-value theory framework.4 A priori to any evaluations of dimensionality, we posited the following hypotheses based on the theoretical underpinnings of control-value theory: (HI) Task importance factors will be positively cor related with two adaptive factors (self-efficacy and metacognition), and negatively correlated with two maladaptive factors (procrastination and avoidance of help seeking). (H2) Anxiety factors will be negatively correlated with two adaptive factors (self-efficacy and metacog nition), and positively correlated with two mal adaptive factors (procrastination and avoidance of help seeking). The secondary purpose of the current study was to explore potential differences in task importance and anxiety from year 1 of medical school to year 4. Based on the findings of Artino et allfi in their development of several self-efficacy scales, we had the following hypotheses: (H3) Task importance will be higher among more senior students. (H4) Anxiety will be lower among more senior students. As medical students progress through the curriculum, they develop greater efficacy (e.g., Artino et a l16 found seniority related to higher scores on “patient care selfefficacy” and “evidence-based medicine self-efficacy”), and in turn these students are expected to increase the degree of importance they attach to the ACGME compe tencies, upon which scale items were based. By the same token, progress through the medical school curriculum and development as a practitioner are expected to result in less anxiety as students’ confidence in attaining the ACGME competencies increases.
32
Similarly, as students develop greater skill and a deeper understanding of their roles as caregivers, it is expected that they will attach greater importance to, and perceive greater value in, the goals of their academic programs. More senior students are expected to better understand why their assigned academic and practitioner goals are structured as they are, and are more likely to agree with those goals as experience broadens their perspectives.
METHODS Study Context This study was conducted at the F. Edward Hebert School of Medicine, Uniformed Services University of the Health Sciences (USU) under the auspices of the Long-Term Career Outcome Study. At the time of the study, USU matricu lated roughly 170 medical students per year and offered a traditional 4-year curriculum, structured as 2 years of basic sciences coursework followed by 2 years of clinical rota tions (clerkships).
Item Development and Content Validation Initial survey items were written to align with the content of the six core competencies defined by the ACGME17: (1) patient care, (2) medical and population health knowl edge, (3) interpersonal and communication skills, (4) practicebased learning and improvement (i.e., understanding practice and ways to improve practice, such as through quality assur ance), (5) professionalism, and (6) systems-based practice (i.e., understanding issues of access and use of resources in health care17). Since one of our goals was to evaluate medical stu dents’ motivational beliefs and emotions as they related to their medical knowledge and skills at different time points in their medical school education, we developed items targeting the ACGME competencies, which students are expected to achieve by the end of residency training. It is worth noting that at the time of this study, USU’s learning objectives for all medical students were structured around the six ACGME core competencies. After developing a library of draft items, we recruited five senior physician-educators, each with more than 10 years of experience in medical education, to provide content validity evaluations. Using a standardized content validation form,18 we individually asked each physicianeducator to indicate the degree to which the items addressed the targeted constructs. In addition, each physician-educator was asked to provide editorial recommendations for item improvement or elimination. The result of this effort was a list of 19 behavioral items related to the six ACGME core competencies. Next, the 19 items were organized into a survey instrument, which asked respondents to rate their perceptions of self-efficacy, task importance, and anxiety as they related to each of the 19 ACGME-framed items (for a total of 57 items). Details of validation of the selfefficacy subscale results, using a smaller subsample, are
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Task Importance and Anxiety in Medical School TA B LE I.
Results for the Task Importance Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; X = 0; N = 368) Factors Item
C
1
2
3
4
5
6
IM P-1 Apply knowledge of normal function to each of the major organ systems? IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits? IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence? IMP-4 Use effective listening skills when interacting with a patient? IMP-5 Demonstrate caring when counseling a patient? IMP-6 Accurately gather essential information from a patient? IMP-7 Perform a thorough physical exam? IMP-8 Develop an appropriate differential diagnosis? IMP-9 Generate a patient-specific treatment plan? IMP-10 Use information technology to support patient-care decisions? IMP-11 Work effectively with other health care professionals to provide high-quality patient care? IMP-12 Improve clinical practice using a systematic approach? IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems? IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals? IMP-15 Demonstrate sensitivity to patients’ cultural differences? IMP-16 Balance professional responsibilities with personal responsibilities? IMP-17 Discuss methods of controlling health care costs? IM P-18 Practice cost-effective health care delivery that does not compromise quality of care? IMP-19 Apply high-quality health care in deployed military environments?
0.55
0.31 (0.54)
-0.04 (-0.03)
0.46(0.66)
-0.10 (-0.12)
-0.12 (-0.15)
-0.17 (-0.28)
0.77
0.79 (0.84) -0.04 (0.03)
0.06 (0.47)
-0.03 (-0.04)
-0.03 (-0.02)
-0.23 (-0.33)
0.59
0.62 (0.73)
0.03 (0.07)
0.21 (0.52)
-0.04 (-0.05)
-0.02 (-0.05)
-0.07 (-0.18)
0.77
-0.08 (0.35)
0.01 (-0.02)
0.89 (0.87)
0.00 (-0.10)
-0.03 (-0.19)
-0.09 (-0.21)
0.7
-0.10(0.32)
0.03 (-0.01)
0.89 (0.83)
0.05 (-0.08)
0.03 (-0.14)
0.05 (-0.07)
0.77
0.11 (0.48)
-0.02 (-0.04)
0.7 4 (0.83) -0.01 (-0.11)
0.00 (-0.14)
-0.27 (-0.39)
0.54 0.77
0.18 (0.47) 0.64 (0.75)
0.00 (-0.05) -0.09 (-0.03)
0.5 7 (0.69) -0.07 (-0.16) 0.14(0.50) 0.03 (-0.03)
0.03 (-0.05) 0.02 (-0.02)
-0.20 (-0.29) -0.43 (-0.52)
0.84
0.87 (0.86) -0.10 (-0.01)
0.00 (-0.03)
0.00 (0.03)
-0.30 (-0.39)
0.65
0.76 (0.80) -0.06 (0.03)
0.05(0.40)
0.11 (0.05)
0.07 (0.04)
-0.10 (-0.19)
0.6
0.50 (0.69)
0.04 (0.06)
0.37 (0.62)
-0.04 (-0.07)
-0.01 (-0.07)
-0.10 (-0.21)
0.66
0.74 (0.80)
0.07 (0.12)
0.10(0.46)
-0.01(0.00)
0.01 (-0.02)
-0.06 (-0.16)
0.72
0.81 (0.85)
0.01 (0.08)
0.09 (0.45)
0.03 (0.00)
0.02 (0.02)
0.06 (-0.05)
0.67
0.79 (0.80)
0.06 (0.11)
0.06(0.40)
0.01 (-0.01)
0.05 (0.05)
0.15 (0.05)
0.5
0.09 (0.37)
0.02 (0.00)
0.66 (0.66)
0.02 (-0.07)
-0.01 (-0.10)
0.24 (0.14)
0.35
0.05 (0.29)
-0.05 (-0.08)
0.56(0.57)
-0.05 (-0.14)
-0.03 (-0.06)
0.14(0.05)
0.82
0.90 (0.83)
0.06 (0.16)
-0.08 (0.29)
0.05 (0.07)
-0.01 (0.02)
0.34 (0.24)
0.79
0.92 (0.86)
0.02 (0.12)
-0.07 (0.33)
0.02 (0.04)
-0.03 (0.01)
0.22 (0.11)
0.59
0.78 (0.76)
0.04(0.11)
-0.03 (0.35)
-0.06 (-0.01)
-0.07 (-0.03)
0.01 (-0.09)
-0.07 (0.38)
C, Communality; Imp, Task Importance. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) >0.30 on only one factor, and communalities >0.40. Results presented above reflect loadings and communalities for Task Importance items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously.
provided in the work of Artino et al.16 Lists of the task importance and anxiety items are presented in Tables I and II. P a rtic ip a n ts a n d P ro c e d u re s
Participants were recruited from the population of USU medical school students during the 2010-2011 and 2011-
MILITARY MEDICINE, Vol. 180, April Supplement 2015
2012 academic years ( N = 864). In May 2011 and May 2012, all students were contacted by e-mail and invited to complete the online survey of their per ceptions of self-efficacy, task importance, and anxiety. Two follow-up e-mail reminders were sent out over the next 4 weeks, at which point the response period was closed. Participation in the survey was voluntary, and
33
Assessing Task Importance and Anxiety in Medical School TA B LE II.
Results for the Anxiety Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; delta = 0; N = 368) Factors Item
C
1
2
5
6
ANX-1 Applying knowledge of normal function to each of the major organ systems? ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits? ANX-3 Applying knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence? ANX-4 Using effective listening skills when interacting with a patient? ANX-5 Demonstrating caring when counseling a patient? ANX-6 Accurately gathering essential information from a patient? ANX-7 Performing a thorough physical exam? ANX-8 Developing an appropriate differential diagnosis? ANX-9 Generating a patient-specific treatment plan? ANX-10 Using information technology to support patient-care decisions? ANX-l 1 Working effectively with other health care professionals to provide high-quality patient care? ANX-12 Improving clinical practice using a systematic approach? ANX-13 Evaluating evidence from scientific studies relevant to your patients’ health problems? ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals? ANX-15 Demonstrating sensitivity to patients’ cultural differences? ANX-16 Balancing professional responsibilities with personal responsibilities? ANX-17 Discussing methods of controlling health care costs? ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care? ANX-19 Applying high-quality health care in deployed military environments?
0.51
-0.02 (-0.05)
0.06 (0.37)
-0.03 (0.02)
0.18 (0.46)
- 0.59 ( - 0.69)
0.00 (-0.08)
0.55
0.01 (0.02)
0.08 (0.35)
0.05 (0.16)
0.01 (0.34)
- 0.69 ( - 0.73)
0.02 (-0.08)
0.48
0.01 (0.02)
v0.31 (0.53)
0.00 (0.04)
0.09 (0.41)
-0.46 (-0.62)
0.02 (-0.05)
0.81
0.01 (0.01)
0.10(0.44)
-0.01 (-0.15)
0.88 (0.89)
0.07 (-0.35)
0.03 (0.01)
0.79
0.04 (0.04)
0.11 (0.42)
-0.01 (-0.15)
0.90 ( 0.88)
0.15 (-0.28)
0.01 (-0.01)
0.70
0.01 (-0.02)
-0.11 (0.30)
-0.01 (-0.06)
0.71 (0.79)
-0.31 (-0.56)
0.07 (0.01)
0.50 0.76
-0.01 (-0.03) 0.02 (-0.01)
-0.15 (0.22) 0.00 (0.35)
0.00 (0.00) 0.00 (0.14)
0.51 (0.62) -0.01 (0.37)
-0.39 (-0.55) - 0.88 ( - 0.87)
0.01 (-0.05) 0.04 (-0.07)
0.67
0.02 (0.06)
0.18 (0.44)
0.06 (0.21)
-0.11 (0.28)
- 0.76 ( - 0.80)
-0.04 (-0.15)
0.52
-0.09 (-0.03)
0.59 (0.68)
0.02 (-0.01)
0.02 (0.36)
-0.23 (-0.48)
0.00 (-0.03)
0.50
-0.06 (-0.02)
0.39 (0.58)
-0.01 (-0.07)
0.37 (0.58)
-0.09 (-0.42)
-0.14 (-0.16)
0.64
-0.07 (0.01)
0.62 (0.75)
0.03 (-0.02)
0.19 (0.50)
-0.14 (-0.49)
-0.08 (-0.11)
0.66
0.03 (0.10)
0.63 (0.72)
-0.05 (-0.01)
0.07 (0.41)
-0.13 (-0.45)
-0.33 (-0.36)
0.61
0.08 (0.16)
0.62 (0.69) -0.03 (0.05)
0.01 (0.34)
-0.15 (-0.43)
-0.30 (-0.33)
0.61
0.10(0.08)
0.28 (0.53)
-0.10 (-0.16)
0.63 (0.73)
0.07 (-0.30)
-0.05 (-0.07)
0.31
-0.09 (-0.06)
-0.02 (0.23)
0.07 (0.04)
0.34 (0.45)
-0.29 (-0.46)
-0.11 (-0.16)
0.70
0.05 (0.10)
0.86 (0.81) -0.02 (-0.07)
0.00 (0.30)
0.13 (-0.19)
0.13 (0.13)
0.70
0.03 (0.08)
0.79 (0.81)
0.00 (-0.05)
0.05 (0.37)
0.01 (-0.31)
0.18 (0.16)
0.37
0.04 (0.08)
0.48 (0.57)
0.02 (0.03)
0.04 (0.32)
-0.19 (-0.40)
0.06 (0.02)
3
4
Anx, Anxiety; C, Communality. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) >0.30 on only one factor, and communalities >0.40. Results presented above reflect loadings and communalities for Anxiety items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously. Results continued from Table I.
ethical approval was obtained from the USU Institutional Review Board. A n a ly s e s
Before analysis, we screened the data for accuracy and missing values and evaluated all item scores for distribu tion normality. To evaluate the representativeness of our sample, we compared the respondents and the nonrespon
34
dents on undergraduate grade point average, Medical College Admissions Test scores, and age using multivariate ana lyses of variance (MANOVAs), as well as on gender using a x2 test of independence. Next, we conducted an exploratory factor analyses on the task importance and anxiety items to evaluate their dimensional structure. These results were used to define subscales representing dimensions of task importance and
MILITARY MEDICINE, Vol. 180, April Supplement 2015
Assessing Task Importance and Anxiety in Medical School anxiety related to the ACGME competencies. Internal con sistency estimates and descriptive statistics were computed for all measures, and correlations were computed to assess relations to other variables. Finally, we conducted MANOVA tests to evaluate differences by year group for the task impor tance and anxiety subscales. We followed the MANOVAs with Bonferroni multiple comparisons of differences across year groups for each subscale. All analyses were completed using SPSS 21.0 (IBM Corporation, 2012, New York). Measures Avoidance of Help Seeking Avoidance of help seeking was assessed using a five-item scale developed by Pajares et al.19 These items assessed the extent to which students avoided asking for help even when they need it. Higher scores indicate greater help avoidance, and all items employed a 5-point, Likert-type response scale. Metacognition Metacognition was assessed using an eight-item scale from the Motivated Strategies for Learning Questionnaire.20 These items assessed the frequency with which students used metacognitive control strategies, including planning, goal setting, comprehension monitoring, and performance regulation to evaluate their progress as learners. Higher scores on this scale generally indicate a greater awareness of one’s own thinking. All items employed a 5-point, Likert-type response scale. Procrastination Procrastination was captured using a four-item scale derived by Wolters." These items assessed the extent to which stu dents disengage academically or tend to delay getting started on academic work. Higher scores indicate a greater degree of procrastination. All items employed a 5-point, Likert-type response scale. Self-Efficacy Self-efficacy was assessed using three subscales derived by Artino et al,16 based on variations of the 19-item ACGME competency items used in the present study. For all three scales, higher scores indicated greater self-efficacy, and all items employed 5-point Likert-type response scales. (a) Patient Care Self-Efficacy: This eight-item scale cap tured the extent to which medical students believe they are capable of providing adequate patient care. (b) Interpersonal Skills Self-Efficacy: This three-item scale captured the extent to which medical students believe they are capable of interacting and communicating effectively with patients. (c) Evidence-Based Medicine Self-Efficacy: This threeitem scale assessed the degree to which students believe they are capable of practicing evidence-based medicine.
MILITARY MEDICINE, Vol. 180, April Supplement 2015
RESULTS
Survey responses were obtained from 368 students (43%) out of a total pool of 864 students enrolled during the 2010-2011 and 2011-2012 academic years. Of these, 131 (36%) were enrolled in medical school year 1 (MS-1), 97 (26.6%) in MS-2, 73 (19.8%) in MS-3, and 67 (18.2%) in MS-4. The sample included 102 (28%) female respondents, which is representative of the overall USU student population. Results of the comparisons of respondents versus non respondents revealed the following statistically significant differences: For MS-1 students, the respondents were, on aver age, slightly older than the nonrespondents [F (1) = 15.10, p < 0.001]. For MS-2 students, the respondents were, on average, slightly older than the nonrespondents [F (1) = 7.69, p < 0.05], had lower undergraduate grade point aver ages [F (1) = 11.18, p < 0.05], and had a higher proportion of males [ / (2) = 16.61, p < 0.05], For MS-3 and MS-4 students, there were no differences between the two groups on any of the four measures. We interpreted these results as evidence that the sample of 368 students was sufficiently similar to the overall student population to justify proceed ing with the study. Evidence Based on Internal Structure: Exploratory Factor Analysis Following data screening, we performed a principal axis factor analysis with oblique rotation (delta = 0), on the set of 38 task importance and anxiety items. The correlation matrix was found to be factorable; the Kaiser-Meyer-Olkin Measure of Sampling Adequacy was 0.90, which is con sidered extremely good.21 Bartlett’s test of sphericity ( j 2 = 11136.15, d f = 703, p < 0.01) indicated that the correlation matrix was not an identity matrix, and all measures of sam pling adequacy were deemed sufficient (i.e., >0.60). The scree plot and Eigenvalue criteria indicated a sixfactor solution, with all factors accounting for 63.2% of the total variance in the items. Evaluation of the communalities extracted revealed that two anxiety items (ANX-16 and ANX-19) and one task importance item (IMP-16) had communalities
