Research Report
Assessing Medical Students’ Perceptions of Patient Safety: The Medical Student Safety Attitudes and Professionalism Survey Joshua M. Liao, MD, Jason M. Etchegaray, PhD, S. Tyler Williams, David H. Berger, MD, MHCM, Sigall K. Bell, MD, and Eric J. Thomas, MD, MPH
Abstract Purpose To develop and test the psychometric properties of a survey to measure students’ perceptions about patient safety as observed on clinical rotations. Method In 2012, the authors surveyed 367 graduating fourth-year medical students at three U.S. MD-granting medical schools. They assessed the survey’s reliability and construct and concurrent validity. They examined correlations between students’ perceptions of organizational cultural factors, organizational patient safety measures, and students’ intended safety behaviors. They also calculated percent positive scores for cultural factors.
S
ince the Institute of Medicine reports To Err Is Human1 and Crossing the Quality Chasm2 demonstrated the prevalence and impact of medical errors, patient safety and quality improvement have become areas of focus in U.S. health care. Attention from policy makers, payers, and groups such as the Agency for Healthcare Research and Quality (AHRQ), National Patient Safety Agency, and World Health Organization has resulted in numerous safety initiatives at the national and institutional levels. Most of these initiatives have targeted physicians, administrators, and other health care professionals. Less attention has been paid to medical students, a population that plays a crucial role in carrying the patient safety agenda forward. Please see the end of this article for information about the authors. Correspondence should be addressed to Dr. Etchegaray, 6410 Fannin, UPB 1100, Houston, TX 77030-3006; telephone: (713) 500-6782; e-mail: [email protected]. Acad Med. 2014;89:343–351. First published online December 19, 2013 doi: 10.1097/ACM.0000000000000124
Academic Medicine, Vol. 89, No. 2 / February 2014
Results Two hundred twenty-eight students (62%) responded. Analyses identified five cultural factors (teamwork culture, safety culture, error disclosure culture, experiences with professionalism, and comfort expressing professional concerns) that had construct validity, concurrent validity, and good reliability (Cronbach alphas > 0.70). Across schools, percent positive scores for safety culture ranged from 28% (95% confidence interval [CI], 13%–43%) to 64% (30%–98%), while those for teamwork culture ranged from 47% (32%–62%) to 74% (66%–81%). They were low for error disclosure culture (range: 10% [0%–20%] to
27% [20%–35%]), experiences with professionalism (range: 7% [0%–15%] to 23% [16%–30%]), and comfort expressing professional concerns (range: 17% [5%–29%] to 38% [8%–69%]). Each cultural factor correlated positively with perceptions of overall patient safety as observed in clinical rotations (r = 0.37–0.69, P < .05) and at least one safety behavioral intent item.
Despite calls to include longitudinal patient safety and quality improvement instruction in undergraduate medical education,3–5 not all medical schools teach these topics to students. Among those that do, curricula vary in format, content, rigor, and evaluation methods.6 Further, many formal educational efforts focus on the acquisition of safety knowledge. Few tools have been developed to help researchers and educators understand how students’ actual clinical experiences may affect their attitudes about patient safety and their intentions regarding their future safety-related behavior.
explicitly taught and what they actually learn.7–9 The resulting culture in the clinical learning environment may deliver messages that oppose the content of the formal curriculum, professional values, and even practices for safe patient care.
There is growing awareness that learners are influenced not only by what is taught in the formal curriculum but also by unspoken, powerful messages and beliefs conveyed through interactions with superiors and educators during their clinical rotations. Work on this “hidden curriculum” has shown that there are frequently stark differences between what medical educators say and what they do (knowingly or unknowingly) when acting as role models for students and, in turn, between what students are
Conclusions This study provided initial evidence for the survey’s reliability and validity and illustrated its applicability for determining whether students’ clinical experiences exemplify positive patient safety environments.
Organizations’ cultural factors—safety culture, teamwork culture, error disclosure culture, and professionalism— are distinct from safety knowledge and are important for understanding aspects of the health care workplace environment that have been linked with patient safety and quality of care. For example, staff members’ perceptions of safety culture and teamwork culture are associated with their perceptions of how well their units are providing safe patient care.10,11 Physicians’ perceptions of error disclosure culture are associated with their intent to disclose a hypothetical error to a patient.12 Students who experience unprofessional behavior may be willing to do things they believe are wrong in order to avoid poor evaluations and to fit in with their clinical teams.13 Taken together, these four cultural factors can predict important patient safety outcomes.
343
Research Report
We sought to understand how these cultural factors affect medical students. To do this, we developed the Medical Student Safety Attitudes and Professionalism Survey (MSSAPS) to measure students’ perceptions of these cultural factors in their clinical rotations. In this study, we examined the survey’s reliability and its construct and concurrent validity. We also tested the following two hypotheses: 1. Medical students’ perceptions of safety culture, teamwork culture, error disclosure culture, and professionalism will be positively correlated with their perceptions of patient safety on their clinical rotations. 2. Medical students’ perceptions of these four cultural factors will be positively correlated with their intended behavior concerning patient safety issues. Our overall goal was to demonstrate that medical students could provide valuable information about their experiences with respect to the cultural factors we identified, and that this information could have important educational and clinical implications. Method
Developing the MSSAPS We examined the literature to identify published survey items that we could use to measure medical students’ perceptions of their clinical rotation experiences with respect to four cultural factors—safety culture, teamwork culture, error disclosure culture, and professionalism—and overall patient safety, as well as students’ intended safetyrelated behaviors. For safety culture and teamwork culture, we adapted items from the Safety Attitudes Questionnaire (SAQ).11 For error disclosure culture, we adapted items from our recently published survey.12 For professionalism, we adapted items from the SAQ11 and a survey on medical student moral distress in clinical learning environments.14 For patient safety, we adapted the overall patient safety grade from the AHRQ’s Hospital Survey on Patient Safety Culture15 and included five safety behavioral intent items—one adapted from a survey about student safety perceptions16 and four that we created. In creating the safety behavioral intent items, members of our team—two medical
344
students (J.L., S.W.), two patient safety experts (E.T., D.B.), and one psychometrician (J.E.)—chose commonly accepted safety behaviors, such as participation in quality improvement activities and reporting and encouraging others to report errors and patient safety issues.
questionnaire during patient safety sessions included in the course. Students were invited via e-mail to participate in the Web-based survey. The survey took most respondents 10 to 15 minutes to complete, according to SurveyMonkey time stamps (when the survey was first accessed and last modified).
We pilot tested the first draft of the survey by e-mailing a link to the survey, which was hosted by SurveyMonkey (www.SurveyMonkey.com), to 38 medical students at two medical schools. We asked these students to (1) complete the draft survey and (2) indicate which items they found to be “unclear, confusing, or difficult to answer.” We received feedback from 14 students and made substantive changes to the instructions and item wording based on their input.
Statistical analysis
In the MSAPPS as administered in this study, we asked students to think about “all of their rotations” when responding. Thirty-four cultural factor items and five safety behavioral intent items were rated using a five-point Likert scale (ranging from 1 = disagree strongly to 5 = agree strongly, with NA = not applicable). Overall patient safety was measured on a five-point scale (ranging from A = excellent to E = failing) in response to the following item: “What overall grade would you give all of your rotations on patient safety?” We also included items to collect demographic information (age, gender, and anticipated residency specialty) as well as eight additional items not relevant to this study. We reverse-coded negatively worded items during data analysis. Participants and survey administration Graduating fourth-year medical students at three U.S. MD-granting medical schools were eligible to participate in this study. All three schools were located in urban areas; one school was state funded and the other two were private. The number of faculty at the schools ranged from approximately 1,000 to more than 9,000. This study was approved by the institutional review board of the University of Texas Health Science Center at Houston. In February to April 2012, we administered the MSSAPS to consenting students as part of their fourth-year capstone courses focusing on “transition to residency.” The capstone course was mandatory at two sites and voluntary at the third. At each study site, students were introduced to the
Our hypotheses allowed us to test concurrent validity—that is, whether a predictor is correlated with an outcome at the same point in time. Concurrent validity is important to establish because it shows that a link exists between a predictor (i.e., a cultural factor) and a criterion (i.e., overall patient safety grade or a future behavioral intent item). Additionally, we collected data to determine whether the MSSAPS also had evidence for reliability and construct validity. Because the survey items we adapted from other sources were developed predominantly for physicians and staff, we first conducted confirmatory factor analysis (CFA) to ensure that these items functioned in a similar psychometric manner for medical students. CFA provides evidence for construct validity by indicating whether survey items measured the intended factors (in this study, safety culture, teamwork culture, error disclosure culture, and professionalism). We chose the CFA approach rather than exploratory factor analysis because we had a priori beliefs, based on data with other populations,11,12 about how items would behave statistically (i.e., which items belonged with certain factors). We determined the adequacy of CFA results on the basis of statistical and practical aspects of model fit.17 We examined the χ2/df ratio, Tucker–Lewis Index (TLI),18 comparative fit index (CFI),19 and root mean square error of approximation (RMSEA).20 The χ2/df ratio should be < 3, TLI and CFI values should be ≥ 0.90, and RMSEA values should be ≤ 0.05.21,22 We then computed an initial CFA model for each cultural factor. On the basis of parameter estimates and examination of survey item content, we modified the initial model into a revised model and then tested the revised model. We also assessed the reliability of each factor with Cronbach’s alpha (should be > 0.70). By conducting these analyses, we were able to have confidence in which survey items would be needed to effectively measure the study’s cultural factors.
Academic Medicine, Vol. 89, No. 2 / February 2014
Research Report
We then tested our two hypotheses by examining the associations via Spearman correlations between the cultural factors and the patient safety measures, which were (1) safety behavioral intent (consisting of five separate items about the participant’s intent regarding future behavior) and (2) overall patient safety grade (one item representing the participant’s overall assessment of patient safety observed during clinical rotations). We also examined percent positive scores for each cultural factor because this is a typical way to report culture scores.11,12 Percent positive scores allow medical educators to see variability within and among schools, allowing them to identify areas of concern. We computed percent positive by adding the numbers of participants who agreed slightly or strongly with the items comprising each factor and then dividing the sum by the total number of participants who answered the items for that factor. Results
responded. Response rates varied by school, as follows: 164/200 (82%) at School A; 49/145 (34%) at School B; and 15/19 (79%) at School C. Of the 188 students who responded to the demographic questions, 96 (51%) were male, and 92 (49%) were female. The majority (94%; n = 177) indicated their age was in the range of 25 to 34 years. MSSAPS performance characteristics: Evidence of construct validity and reliability Safety culture. The initial CFA model for safety culture had an adequate CFI value but was lacking on the other indices. The regression weights for the last 2 of the factor’s 10 items were lower than the other items, so we excluded items 9 and 10. We ran a revised model with items 1 through 8 only and obtained much better model fit (Δ χ2 was statistically significant at P < .001) and improved values for χ2/df ratio, RMSEA, CFI, and TLI (Table 1; see Table 2 for items). Cronbach’s alpha for the 8-item factor was 0.89. Teamwork culture. The initial CFA model for teamwork culture had poor values for the indices. The regression
Of the 367 fourth-year students invited to participate in the survey, 228 (62%)
weights for the last 2 of the factor’s 8 items were lower than for the other items, so we excluded items 7 and 8. The revised model, which included items 1 through 6, had a significantly improved fit: Δ χ2 was statistically significant at P 0.05, it improved from the initial model. Cronbach’s alpha values for experiences with professionalism and comfort expressing professional concerns were 0.87 and 0.84, respectively. CFA results overview. In summary, CFA identified five cultural factors comprising 28 items: safety culture (8 items), teamwork culture (6 items), error disclosure culture (4 items), experiences with professionalism (7 items), and
347
Research Report
comfort expressing professional concerns (3 items). On the basis of the CFA results, we accepted the revised model as depicting each cultural factor (see Table 2). Hypothesis testing: Medical students’ perceptions of cultural factors and safety We report the response frequencies for the cultural factor items in Table 2 and for the safety behavioral intent items in Table 3. As described above, we calculated Spearman correlations between the five cultural factors, the five safety behavioral intent items, and overall patient safety grade as observed during clinical rotations. We found that each of the cultural factors was correlated with overall patient safety grade (r = 0.37–0.69, P < .05) and at least one of the safety behavioral intent items (Table 4). Safety culture was the strongest predictor of the four patient-safetyrelated behavioral intent items. Percent positive scores for cultural factors There was substantial variation in percent positive scores across participating institutions, with statistically significant differences in safety and teamwork cultures (Table 5). Safety culture positive perception scores ranged from 28% (95% confidence interval [CI], 13%–43%) to 64% (95% CI, 30%–98%), while teamwork culture percent positive scores ranged from 47% (95% CI, 32%–62%) to 74% (95% CI, 66%–81%). Error disclosure culture percent positive scores were low, ranging from 10% (95% CI, 0%–20%) to 27% (95% CI, 20%–35%). Percent positive scores were also low for experiences with professionalism, ranging from 7% (95% CI, 0%–15%) to 23%
experiences are associated with their safety attitudes and intended behaviors. On the basis of our findings, we believe the MSSAPS can serve as an important tool for measuring these aspects of clinical experiences and targeting areas that need improvement in student/trainee education and in faculty development to provide trainees with effective role models.22
(95% CI, 16%–30%). Of note, individual items revealed that among responding students, 159/213 (75%) agreed that a team member had made disparaging remarks about a patient, and 140/213 (66%) agreed that a team member had been disrespectful to someone below himself/herself in the team hierarchy (see Table 2). Percent positive scores were low for comfort expressing professional concerns, with scores ranging from 17% (95% CI, 5%–29%) to 38% (95% CI, 8%–69%). Individual items revealed that at least one-quarter of responding students did not feel comfortable expressing concerns to superiors about patient safety (52/213; 25%) or their own mistreatment (91/213; 43%). Finally, percent positive for overall patient safety grade ranged from 38% (95% CI, 23%– 53%) to 79% (95% CI, 54%–100%).
Our study shows that the MSSAPS is psychometrically sound and provides descriptive data that can be used to promote curricular and cultural change. For example, less than onethird of students from any of the three participating schools had positive perceptions of the error disclosure culture or professional behaviors they witnessed during their clinical rotations. Students reported they did not receive education about error disclosure or see errors routinely disclosed to patients and families; such experiences could lead them to internalize the message that disclosure is not an institutional priority. The low percentage of students with positive perceptions of experiences with professionalism suggests that students witnessed disrespectful behaviors among their team members (residents and attending physicians) and between their teams and their patients. The positive perception scores for comfort expressing professional concerns were also low, suggesting that many students had difficulty speaking up about unprofessional behaviors (e.g., mistreatment of themselves and others) or concerns about patient safety. In comparison, students viewed safety and teamwork cultures more
Discussion
Medical students’ experiences during clinical rotations have an important influence on their attitudes toward patient safety and their future behaviors. Negative experiences with respect to safety culture, teamwork culture, error disclosure culture, and professionalism—including both experiences with professionalism and comfort expressing personal concerns— have the potential to contradict lessons learned during formal patient safety curricula. In this study, we have provided initial evidence for the reliability and validity of the MSSAPS in measuring medical students’ perceptions of the abovementioned cultural factors as experienced during clinical rotations. Our study demonstrates that medical students’ clinical
Table 3 Responses of Graduating Fourth-Year Medical Students on the Safety Behavioral Intent Items of the Medical Student Safety Attitudes and Professionalism Survey, 2012 No. (%) responding* Disagree strongly 1
Disagree slightly 2
Neutral 3
Agree slightly 4
Agree strongly 5
1. I expect to participate in quality improvement initiatives 2. I intend to encourage my colleagues to tell patients and their families about medical errors that impacted their care
1 (1) 3 (2)
6 (3) 5 (3)
27 (14) 13 (7)
95 (50) 77 (42)
62 (32) 87 (47)
3. I intend to encourage my colleagues to report any patient safety issues they encounter
4 (2)
2 (1)
6 (3)
75 (40)
100 (53)
Safety behavioral intent item
4. I intend to report any patient safety issues I encounter
3 (1)
3 (1)
9 (4)
65 (32)
121 (60)
5. If I saw a medical error committed by one of my team members, I would keep it to myself
6 (3)
15 (7)
23 (11)
95 (47)
63 (31)
*A total of 228 medical students at three U.S. MD-granting medical schools responded to the survey. Denominators differ by item because of missing data. Row percentages may not add to 100% because of rounding.
348
Academic Medicine, Vol. 89, No. 2 / February 2014
Research Report
Table 4 Spearman Correlations Between Types of Cultural Factors, Overall Patient Safety Grade, and Safety Behavioral Intent Items, Medical Student Safety Attitudes and Professionalism Survey* Cultural factor: r (95% confidence interval) Error Experiences Comfort expressing disclosure with professional culture professionalism concerns
Patient safety measure Overall patient safety grade
Safety culture
Teamwork culture
0.37† (0.19, 0.52)
0.41† (0.24, 0.58)
0.49† (0.34, 0.62)
0.69† (0.59, 0.78)
0.53† (0.39, 0.66)
I expect to participate in quality improvement initiatives
0.04 (–0.14, 0.22)
–0.08 (–0.27, 0.11)
0.05 (–0.15, 0.25)
0.17 (–0.03, 0.36)
0.16 (–0.02, 0.33)
I intend to encourage my colleagues to tell patients and their families about medical errors that impacted their care
0.20† (0.001, 0.38)
0.23† (0.04, 0.40)
0.22† (0.04, 0.40)
0.34† (0.16, 0.51)
0.35† (0.17, 0.50)
I intend to encourage my colleagues to report any patient safety issues they encounter
0.17 (–0.03, 0.35)
0.25† (0.08, 0.43)
0.35† (0.16, 0.52)
0.42† (0.26, 0.57)
0.36† (0.19, 0.52)
I intend to report any patient safety issues I encounter
0.13 (–0.05, 0.30)
0.19† (0.01, 0.36)
0.18 (–0.002, 0.36)
0.31† (0.13, 0.47)
0.34† (0.16, 0.51)
I saw a medical error committed by one of my If team members, I would keep it to myself
0.18 (–0.02, 0.39)
(0.13, 0.49)
0.09 (–0.12, 0.32)
0.20† (–0.01, 0.40)
0.22† (0.02, 0.42)
Safety behavioral intent item
0.32†
*This analysis used data from 228 graduating fourth-year medical students at three U.S. MD-granting medical schools who completed the survey in 2012. † P < .05.
perception scores on the five cultural factors. Overall, these findings provide initial evidence for the validity of the MSSAPS and indicate that perceptions of patient safety may be shaped not only by safety and teamwork cultures but also by experiences with professionalism.32
positively, perhaps because of relatively long-standing efforts by hospitals to measure and make improvements in these areas.23–31 Importantly, each of the five cultural factors was correlated with students’ perceptions of overall patient safety for their clinical rotations (overall patient safety grade), and all of the factors except error disclosure culture were correlated with at least three of the five safety behavioral intent items. Further, the school with the lowest overall patient safety grade also had the lowest positive
Our study is the first to simultaneously measure medical student attitudes about teamwork culture, safety culture, error disclosure culture, and professionalism. It builds on prior studies of student
Table 5 Percent Positive Scores by Cultural Factor or Patient Safety Measure, Medical Student Safety Attitudes and Professionalism Survey, Three U.S. MD-Granting Medical Schools, 2012* Respondents with positive perceptions, % (95% confidence interval) Factor
School A
School B
School C
Safety culture† Teamwork culture†
57 (48–66) 74 (66–81)
28 (13–43) 47 (32–62)
64 (30–98) 71 (44–99)
Error disclosure culture
27 (20–35)
10 (0–20)
15 (0–38)
Experiences with professionalism
23 (16–30)
7 (0–15)
20 (0–50)
Comfort expressing professional concerns
34 (26–43)
17 (5–29)
38 (8–69)
Overall patient safety grade†
70 (62–77)
38 (23–53)
79 (54–100)
*A total of 228 graduating fourth-year medical students responded to the survey: 164 from School A; 49 from School B; and 15 from School C. Denominators differ for each factor at each institution because of missing data. † Significant difference via chi-square test at P < .05.
Academic Medicine, Vol. 89, No. 2 / February 2014
experiences with unprofessional behavior and the hidden curriculum’s impact on patient safety33–35 by demonstrating quantitative correlations between students’ lived experiences in the hidden curriculum (represented by the cultural factors), their perceptions of patient safety, and their intentions regarding their future safetyrelated behaviors. Students frequently report experiencing and assimilating into a culture of unprofessional behaviors36 that likely undermine the explicit teachings of medical school curricula. Despite national attention and focused efforts to tackle unprofessional behavior, a recent study documented a sobering persistence of medical student mistreatment.37 Given the importance of patient safety improvements, our results linking measures of professionalism (i.e., experiences with professionalism and comfort expressing professional concerns) with patient safety measures may provide additional motivation to change medical culture in light of the emphasis that health care professionals, regulators, payers, and the public have placed on improving patient safety. The MSSAPS can be used for internal benchmarking purposes. Given the concurrent validity evidence for the instrument, students’ patient safety perceptions should improve as clinical
349
Research Report
rotation sites make improvements in the areas mentioned above. Educators could also administer the MSSAPS after individual rotations to identify problematic clinical experiences. Organizations could use percent positive scores to determine where to prioritize improvement efforts. For example, on the basis of the scores in this study, all three participating institutions could focus on improvements related to error disclosure culture and professionalism (both experiences with professionalism and comfort expressing concerns; see Table 5). School B could also target safety culture and teamwork culture as areas for improvement. Tracking percent positive scores longitudinally would allow schools to conduct year-to-year comparisons of changes in perceptions. In this way, the MSSAPS could be used at the institution level to identify aspects of clinical rotations and overall culture that may deviate from the mission of training safe, professional physicians. Additionally, the MSSAPS could be useful for external benchmarking and to help medical schools and teaching hospitals answer the call to create cultures of respect that promote ethical, safe behavior.38 By targeting medical students—who are often sensitive to behavioral modeling—this survey could serve as a much-needed tool for measuring the effectiveness of institutional efforts to address disrespectful, unprofessional behavior. This study has several limitations. First, it includes only the experiences of graduating fourth-year medical students from three institutions. Additional work across other medical schools and years of study is required to better understand the generalizability of our results. Second, further validity testing should be done, specifically to probe whether student responses (i.e., perceptions) relate to actual clinical behavior, because some students may not have had enough experience to distinguish between professional and unprofessional behaviors or between safe and unsafe acts. Third, the voluntary participation in School C’s capstone course (and therefore in the MSSAPS) may introduce bias. Fourth, there has traditionally been a lack of student exposure and experience with topics like medical error disclosure, which may limit the validity of student responses to corresponding survey items. Finally, additional research is needed to
350
determine to what degree incremental changes in clinical curricula based on MSSAPS results would be reflected in subsequent administrations of the survey. In conclusion, this study provides initial evidence for the reliability and validity of the MSSAPS, a survey instrument that can be used to determine whether medical students are having clinical experiences in positive patient safety learning environments. Our data suggest that a large proportion of students are exposed during their clinical rotations to behaviors that do not exemplify best practices for safe and professional patient care. However, we believe that by building on our findings that unprofessional behaviors in clinical learning environments are linked to students’ perceptions of patient safety, educators can help create a sense of urgency and encourage more leaders and clinical educators to address both explicit and hidden patient safety curricula—and the institutional cultures that affect them. Acknowledgments: Dr. Bell’s work in humanism was supported through an Arnold P. Gold Foundation Professorship. The authors acknowledge course directors Katharine K. Treadway, MD, and Margaret Uthman, MD, for facilitating access to their students. Other disclosures: None reported. Ethical approval: This study was approved by the institutional review board of the University of Texas Health Science Center at Houston. Dr. Liao is a resident physician, Department of Internal Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, Massachusetts. At the time of writing, he was a fourth-year medical student, Baylor College of Medicine, Houston, Texas. Dr. Etchegaray is assistant professor, Department of Internal Medicine, University of Texas Medical School at Houston and UT-Memorial Hermann Center for Healthcare Quality and Safety, Houston, Texas. Mr. Williams is a fourth-year medical student, University of Texas Health Science Center at Houston, Houston, Texas. Dr. Berger is professor and vice chair, Michael E. DeBakey Department of Surgery, Baylor College of Medicine, and vice president and chief medical officer, Baylor College of Medicine Medical Center, Houston, Texas. Dr. Bell is assistant professor of medicine, division of general medicine and primary care, Harvard Medical School, Boston, Massachusetts. Dr. Thomas is professor of medicine, associate dean for health care quality, and director, UTMemorial Hermann Center for Healthcare Quality and Safety and at University of Texas Medical School at Houston, Houston, Texas.
References 1 Institute of Medicine. Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000. 2 Institute of Medicine. Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001. 3 DesHarnais SI, Nash DB. Reforming way medical students and physicians are taught about quality and safety. Mt Sinai J Med. 2011;78:834–841. 4 Berwick DM, Finkelstein JA. Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new “public interest.” Acad Med. 2010;85:S56–S65. 5 Lucian Leape Institute Roundtable on Reforming Medical Education. Unmet Needs: Teaching Physicians to Provide Safe Patient Care. Boston, Mass: Lucian Leape Institute at the National Patient Safety Foundation; 2010. 6 Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: A systematic review. Acad Med. 2010;85:1425–1439. 7 Chuang AW, Nuthalapaty FS, Casey PM, et al. To the point: Reviews in medical education—taking control of the hidden curriculum. Am J Obstet Gynecol. 2010;203:316.e1–e6. 8 Inui TS. A Flag in the Wind: Educating for Professionalism in Medicine. Washington, DC: Association of American Medical Colleges; 2003. 9 Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69:861–871. 10 Etchegaray JM, Thomas EJ. Comparing two safety culture surveys: Safety Attitudes Questionnaire and Hospital Survey on Patient Safety. BMJ Qual Saf. 2012;21:490– 498. 11 Sexton JB, Helmreich RL, Neilands TB, et al. The Safety Attitudes Questionnaire: Psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6:44. 12 Etchegaray JM, Gallagher TH, Bell SK, Dunlap B, Thomas EJ. Error disclosure: A new domain for safety culture assessment. BMJ Qual Saf. 2012;21:594–599. 13 Feudtner C, Christakis DA, Christakis NA. Do clinical clerks suffer ethical erosion? Students’ perceptions of their ethical environment and personal development. Acad Med. 1994;69:670–679. 14 Wiggleton C, Petrusa E, Loomis K, et al. Medical students’ experiences of moral distress: Development of a Web-based survey. Acad Med. 2010;85:111–117. 15 Agency for Healthcare Research and Quality. Hospital Survey on Patient Safety Culture. http://www.ahrq.gov/qual/ patientsafetyculture/hospsurvindex.htm. Accessed October 17, 2013. 16 Leung GK, Patil NG. Patient safety in the undergraduate curriculum: Medical students’ perception. Hong Kong Med J. 2010;16: 101–105. 17 Reise SP, Widaman KF, Pugh RH. Confirmatory factor analysis and item
Academic Medicine, Vol. 89, No. 2 / February 2014
Research Report
18 19 20
21 22 23
24
25
response theory: Two approaches for exploring measurement invariance. Psychol Bull. 1993;114:552–566. Bentler PM, Bonnet DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull. 1980;88:588–606. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107:238–246. Steiger JH, Lind J. Statistically based tests for the number of common factors. Paper presented at: Annual Meeting of the Psychometric Society; May 1980; Iowa City, Iowa. Bollen KA. Structural Equations With Latent Variables. New York, NY: Wiley; 1989. Kline RB. Principles and Practice of Structural Equation Modeling. New York, NY: The Guilford Press; 2005. Bell SK, Moorman DW, Delbanco T. Improving the patient, family, and clinician experience after harmful events: The “when things go wrong” curriculum. Acad Med. 2010;85:1010–1017. Vigorito MC, McNicoll L, Adams L, Sexton B. Improving safety culture results in Rhode Island ICUs: Lessons learned from the development of action-oriented plans. Jt Comm J Qual Patient Saf. 2011;37:509–514. Carney BT, West P, Neily JB, Mills PD, Bagian JP. Improving perceptions of
Academic Medicine, Vol. 89, No. 2 / February 2014
26
27
28
29
30
31
teamwork climate with the Veterans Health Administration medical team training program. Am J Med Qual. 2011;26: 480–484. Sexton JB, Berenholtz SM, Goeschel CA, et al. Assessing and improving safety climate in a large cohort of intensive care units. Crit Care Med. 2011;39:934–939. Paine LA, Rosenstein BJ, Sexton JB, Kent P, Holzmueller CG, Pronovost PJ. Assessing and improving safety culture throughout an academic medical centre: A prospective cohort study. Qual Saf Health Care. 2010;19:547–554. O’Leary KJ, Wayne DB, Haviley C, Slade ME, Lee J, Williams MV. Improving teamwork: Impact of structured interdisciplinary rounds on a medical teaching unit. J Gen Intern Med. 2010;25:826–832. Huang DT, Clermont G, Kong L, et al. Intensive care unit safety culture and outcomes: A U.S. multicenter study. Int J Qual Health Care. 2010;22:151–161. Pronovost PJ, Berenholtz SM, Goeschel C, et al. Improving patient safety in intensive care units in Michigan. J Crit Care. 2008;23:207–221. Thomas EJ, Sexton JB, Neilands TB, Frankel A, Helmreich RL. The effect of executive walk rounds on nurse safety climate
32
33
34
35
36
37
38
attitudes. A randomized trial of clinical units. BMC Health Serv Res. 2005;5:28. Leape LL, Shore MF, Dienstag JL, et al. A culture of respect, part 1: The nature and causes of disrespectful behavior by physicians. Acad Med. 2012;87:1–8. Walton MM. Hierarchies: The Berlin Wall of patient safety. Qual Saf Health Care. 2006;15:229–230. Brainard AH, Brislen HC. Viewpoint: Learning professionalism: A view from the trenches. Acad Med. 2007;82:1010–1014. Lucey C, Souba W. Perspective: The problem with the problem of professionalism. Acad Med. 2010;85:1018–1024. Reddy ST, Farnan JM, Yoon JD, et al. Third-year medical students’ participation in and perceptions of unprofessional behaviors. Acad Med. 2007;82(10 suppl): S35–S39. Fried JM, Vermillion M, Parker NH, Uijtdehaage S. Eradicating medical student mistreatment: A longitudinal study of one institution’s efforts. Acad Med. 2012;87: 1191–1198. Leape LL, Shore MF, Dienstag JL, et al. A culture of respect, part 2: Creating a culture of respect. Acad Med. 2012;87:1–6
351
Assessing Medical Students’ Perceptions of Patient Safety: The Medical Student Safety Attitudes and Professionalism Survey Joshua M. Liao, MD, Jason M. Etchegaray, PhD, S. Tyler Williams, David H. Berger, MD, MHCM, Sigall K. Bell, MD, and Eric J. Thomas, MD, MPH
Abstract Purpose To develop and test the psychometric properties of a survey to measure students’ perceptions about patient safety as observed on clinical rotations. Method In 2012, the authors surveyed 367 graduating fourth-year medical students at three U.S. MD-granting medical schools. They assessed the survey’s reliability and construct and concurrent validity. They examined correlations between students’ perceptions of organizational cultural factors, organizational patient safety measures, and students’ intended safety behaviors. They also calculated percent positive scores for cultural factors.
S
ince the Institute of Medicine reports To Err Is Human1 and Crossing the Quality Chasm2 demonstrated the prevalence and impact of medical errors, patient safety and quality improvement have become areas of focus in U.S. health care. Attention from policy makers, payers, and groups such as the Agency for Healthcare Research and Quality (AHRQ), National Patient Safety Agency, and World Health Organization has resulted in numerous safety initiatives at the national and institutional levels. Most of these initiatives have targeted physicians, administrators, and other health care professionals. Less attention has been paid to medical students, a population that plays a crucial role in carrying the patient safety agenda forward. Please see the end of this article for information about the authors. Correspondence should be addressed to Dr. Etchegaray, 6410 Fannin, UPB 1100, Houston, TX 77030-3006; telephone: (713) 500-6782; e-mail: [email protected]. Acad Med. 2014;89:343–351. First published online December 19, 2013 doi: 10.1097/ACM.0000000000000124
Academic Medicine, Vol. 89, No. 2 / February 2014
Results Two hundred twenty-eight students (62%) responded. Analyses identified five cultural factors (teamwork culture, safety culture, error disclosure culture, experiences with professionalism, and comfort expressing professional concerns) that had construct validity, concurrent validity, and good reliability (Cronbach alphas > 0.70). Across schools, percent positive scores for safety culture ranged from 28% (95% confidence interval [CI], 13%–43%) to 64% (30%–98%), while those for teamwork culture ranged from 47% (32%–62%) to 74% (66%–81%). They were low for error disclosure culture (range: 10% [0%–20%] to
27% [20%–35%]), experiences with professionalism (range: 7% [0%–15%] to 23% [16%–30%]), and comfort expressing professional concerns (range: 17% [5%–29%] to 38% [8%–69%]). Each cultural factor correlated positively with perceptions of overall patient safety as observed in clinical rotations (r = 0.37–0.69, P < .05) and at least one safety behavioral intent item.
Despite calls to include longitudinal patient safety and quality improvement instruction in undergraduate medical education,3–5 not all medical schools teach these topics to students. Among those that do, curricula vary in format, content, rigor, and evaluation methods.6 Further, many formal educational efforts focus on the acquisition of safety knowledge. Few tools have been developed to help researchers and educators understand how students’ actual clinical experiences may affect their attitudes about patient safety and their intentions regarding their future safety-related behavior.
explicitly taught and what they actually learn.7–9 The resulting culture in the clinical learning environment may deliver messages that oppose the content of the formal curriculum, professional values, and even practices for safe patient care.
There is growing awareness that learners are influenced not only by what is taught in the formal curriculum but also by unspoken, powerful messages and beliefs conveyed through interactions with superiors and educators during their clinical rotations. Work on this “hidden curriculum” has shown that there are frequently stark differences between what medical educators say and what they do (knowingly or unknowingly) when acting as role models for students and, in turn, between what students are
Conclusions This study provided initial evidence for the survey’s reliability and validity and illustrated its applicability for determining whether students’ clinical experiences exemplify positive patient safety environments.
Organizations’ cultural factors—safety culture, teamwork culture, error disclosure culture, and professionalism— are distinct from safety knowledge and are important for understanding aspects of the health care workplace environment that have been linked with patient safety and quality of care. For example, staff members’ perceptions of safety culture and teamwork culture are associated with their perceptions of how well their units are providing safe patient care.10,11 Physicians’ perceptions of error disclosure culture are associated with their intent to disclose a hypothetical error to a patient.12 Students who experience unprofessional behavior may be willing to do things they believe are wrong in order to avoid poor evaluations and to fit in with their clinical teams.13 Taken together, these four cultural factors can predict important patient safety outcomes.
343
Research Report
We sought to understand how these cultural factors affect medical students. To do this, we developed the Medical Student Safety Attitudes and Professionalism Survey (MSSAPS) to measure students’ perceptions of these cultural factors in their clinical rotations. In this study, we examined the survey’s reliability and its construct and concurrent validity. We also tested the following two hypotheses: 1. Medical students’ perceptions of safety culture, teamwork culture, error disclosure culture, and professionalism will be positively correlated with their perceptions of patient safety on their clinical rotations. 2. Medical students’ perceptions of these four cultural factors will be positively correlated with their intended behavior concerning patient safety issues. Our overall goal was to demonstrate that medical students could provide valuable information about their experiences with respect to the cultural factors we identified, and that this information could have important educational and clinical implications. Method
Developing the MSSAPS We examined the literature to identify published survey items that we could use to measure medical students’ perceptions of their clinical rotation experiences with respect to four cultural factors—safety culture, teamwork culture, error disclosure culture, and professionalism—and overall patient safety, as well as students’ intended safetyrelated behaviors. For safety culture and teamwork culture, we adapted items from the Safety Attitudes Questionnaire (SAQ).11 For error disclosure culture, we adapted items from our recently published survey.12 For professionalism, we adapted items from the SAQ11 and a survey on medical student moral distress in clinical learning environments.14 For patient safety, we adapted the overall patient safety grade from the AHRQ’s Hospital Survey on Patient Safety Culture15 and included five safety behavioral intent items—one adapted from a survey about student safety perceptions16 and four that we created. In creating the safety behavioral intent items, members of our team—two medical
344
students (J.L., S.W.), two patient safety experts (E.T., D.B.), and one psychometrician (J.E.)—chose commonly accepted safety behaviors, such as participation in quality improvement activities and reporting and encouraging others to report errors and patient safety issues.
questionnaire during patient safety sessions included in the course. Students were invited via e-mail to participate in the Web-based survey. The survey took most respondents 10 to 15 minutes to complete, according to SurveyMonkey time stamps (when the survey was first accessed and last modified).
We pilot tested the first draft of the survey by e-mailing a link to the survey, which was hosted by SurveyMonkey (www.SurveyMonkey.com), to 38 medical students at two medical schools. We asked these students to (1) complete the draft survey and (2) indicate which items they found to be “unclear, confusing, or difficult to answer.” We received feedback from 14 students and made substantive changes to the instructions and item wording based on their input.
Statistical analysis
In the MSAPPS as administered in this study, we asked students to think about “all of their rotations” when responding. Thirty-four cultural factor items and five safety behavioral intent items were rated using a five-point Likert scale (ranging from 1 = disagree strongly to 5 = agree strongly, with NA = not applicable). Overall patient safety was measured on a five-point scale (ranging from A = excellent to E = failing) in response to the following item: “What overall grade would you give all of your rotations on patient safety?” We also included items to collect demographic information (age, gender, and anticipated residency specialty) as well as eight additional items not relevant to this study. We reverse-coded negatively worded items during data analysis. Participants and survey administration Graduating fourth-year medical students at three U.S. MD-granting medical schools were eligible to participate in this study. All three schools were located in urban areas; one school was state funded and the other two were private. The number of faculty at the schools ranged from approximately 1,000 to more than 9,000. This study was approved by the institutional review board of the University of Texas Health Science Center at Houston. In February to April 2012, we administered the MSSAPS to consenting students as part of their fourth-year capstone courses focusing on “transition to residency.” The capstone course was mandatory at two sites and voluntary at the third. At each study site, students were introduced to the
Our hypotheses allowed us to test concurrent validity—that is, whether a predictor is correlated with an outcome at the same point in time. Concurrent validity is important to establish because it shows that a link exists between a predictor (i.e., a cultural factor) and a criterion (i.e., overall patient safety grade or a future behavioral intent item). Additionally, we collected data to determine whether the MSSAPS also had evidence for reliability and construct validity. Because the survey items we adapted from other sources were developed predominantly for physicians and staff, we first conducted confirmatory factor analysis (CFA) to ensure that these items functioned in a similar psychometric manner for medical students. CFA provides evidence for construct validity by indicating whether survey items measured the intended factors (in this study, safety culture, teamwork culture, error disclosure culture, and professionalism). We chose the CFA approach rather than exploratory factor analysis because we had a priori beliefs, based on data with other populations,11,12 about how items would behave statistically (i.e., which items belonged with certain factors). We determined the adequacy of CFA results on the basis of statistical and practical aspects of model fit.17 We examined the χ2/df ratio, Tucker–Lewis Index (TLI),18 comparative fit index (CFI),19 and root mean square error of approximation (RMSEA).20 The χ2/df ratio should be < 3, TLI and CFI values should be ≥ 0.90, and RMSEA values should be ≤ 0.05.21,22 We then computed an initial CFA model for each cultural factor. On the basis of parameter estimates and examination of survey item content, we modified the initial model into a revised model and then tested the revised model. We also assessed the reliability of each factor with Cronbach’s alpha (should be > 0.70). By conducting these analyses, we were able to have confidence in which survey items would be needed to effectively measure the study’s cultural factors.
Academic Medicine, Vol. 89, No. 2 / February 2014
Research Report
We then tested our two hypotheses by examining the associations via Spearman correlations between the cultural factors and the patient safety measures, which were (1) safety behavioral intent (consisting of five separate items about the participant’s intent regarding future behavior) and (2) overall patient safety grade (one item representing the participant’s overall assessment of patient safety observed during clinical rotations). We also examined percent positive scores for each cultural factor because this is a typical way to report culture scores.11,12 Percent positive scores allow medical educators to see variability within and among schools, allowing them to identify areas of concern. We computed percent positive by adding the numbers of participants who agreed slightly or strongly with the items comprising each factor and then dividing the sum by the total number of participants who answered the items for that factor. Results
responded. Response rates varied by school, as follows: 164/200 (82%) at School A; 49/145 (34%) at School B; and 15/19 (79%) at School C. Of the 188 students who responded to the demographic questions, 96 (51%) were male, and 92 (49%) were female. The majority (94%; n = 177) indicated their age was in the range of 25 to 34 years. MSSAPS performance characteristics: Evidence of construct validity and reliability Safety culture. The initial CFA model for safety culture had an adequate CFI value but was lacking on the other indices. The regression weights for the last 2 of the factor’s 10 items were lower than the other items, so we excluded items 9 and 10. We ran a revised model with items 1 through 8 only and obtained much better model fit (Δ χ2 was statistically significant at P < .001) and improved values for χ2/df ratio, RMSEA, CFI, and TLI (Table 1; see Table 2 for items). Cronbach’s alpha for the 8-item factor was 0.89. Teamwork culture. The initial CFA model for teamwork culture had poor values for the indices. The regression
Of the 367 fourth-year students invited to participate in the survey, 228 (62%)
weights for the last 2 of the factor’s 8 items were lower than for the other items, so we excluded items 7 and 8. The revised model, which included items 1 through 6, had a significantly improved fit: Δ χ2 was statistically significant at P 0.05, it improved from the initial model. Cronbach’s alpha values for experiences with professionalism and comfort expressing professional concerns were 0.87 and 0.84, respectively. CFA results overview. In summary, CFA identified five cultural factors comprising 28 items: safety culture (8 items), teamwork culture (6 items), error disclosure culture (4 items), experiences with professionalism (7 items), and
347
Research Report
comfort expressing professional concerns (3 items). On the basis of the CFA results, we accepted the revised model as depicting each cultural factor (see Table 2). Hypothesis testing: Medical students’ perceptions of cultural factors and safety We report the response frequencies for the cultural factor items in Table 2 and for the safety behavioral intent items in Table 3. As described above, we calculated Spearman correlations between the five cultural factors, the five safety behavioral intent items, and overall patient safety grade as observed during clinical rotations. We found that each of the cultural factors was correlated with overall patient safety grade (r = 0.37–0.69, P < .05) and at least one of the safety behavioral intent items (Table 4). Safety culture was the strongest predictor of the four patient-safetyrelated behavioral intent items. Percent positive scores for cultural factors There was substantial variation in percent positive scores across participating institutions, with statistically significant differences in safety and teamwork cultures (Table 5). Safety culture positive perception scores ranged from 28% (95% confidence interval [CI], 13%–43%) to 64% (95% CI, 30%–98%), while teamwork culture percent positive scores ranged from 47% (95% CI, 32%–62%) to 74% (95% CI, 66%–81%). Error disclosure culture percent positive scores were low, ranging from 10% (95% CI, 0%–20%) to 27% (95% CI, 20%–35%). Percent positive scores were also low for experiences with professionalism, ranging from 7% (95% CI, 0%–15%) to 23%
experiences are associated with their safety attitudes and intended behaviors. On the basis of our findings, we believe the MSSAPS can serve as an important tool for measuring these aspects of clinical experiences and targeting areas that need improvement in student/trainee education and in faculty development to provide trainees with effective role models.22
(95% CI, 16%–30%). Of note, individual items revealed that among responding students, 159/213 (75%) agreed that a team member had made disparaging remarks about a patient, and 140/213 (66%) agreed that a team member had been disrespectful to someone below himself/herself in the team hierarchy (see Table 2). Percent positive scores were low for comfort expressing professional concerns, with scores ranging from 17% (95% CI, 5%–29%) to 38% (95% CI, 8%–69%). Individual items revealed that at least one-quarter of responding students did not feel comfortable expressing concerns to superiors about patient safety (52/213; 25%) or their own mistreatment (91/213; 43%). Finally, percent positive for overall patient safety grade ranged from 38% (95% CI, 23%– 53%) to 79% (95% CI, 54%–100%).
Our study shows that the MSSAPS is psychometrically sound and provides descriptive data that can be used to promote curricular and cultural change. For example, less than onethird of students from any of the three participating schools had positive perceptions of the error disclosure culture or professional behaviors they witnessed during their clinical rotations. Students reported they did not receive education about error disclosure or see errors routinely disclosed to patients and families; such experiences could lead them to internalize the message that disclosure is not an institutional priority. The low percentage of students with positive perceptions of experiences with professionalism suggests that students witnessed disrespectful behaviors among their team members (residents and attending physicians) and between their teams and their patients. The positive perception scores for comfort expressing professional concerns were also low, suggesting that many students had difficulty speaking up about unprofessional behaviors (e.g., mistreatment of themselves and others) or concerns about patient safety. In comparison, students viewed safety and teamwork cultures more
Discussion
Medical students’ experiences during clinical rotations have an important influence on their attitudes toward patient safety and their future behaviors. Negative experiences with respect to safety culture, teamwork culture, error disclosure culture, and professionalism—including both experiences with professionalism and comfort expressing personal concerns— have the potential to contradict lessons learned during formal patient safety curricula. In this study, we have provided initial evidence for the reliability and validity of the MSSAPS in measuring medical students’ perceptions of the abovementioned cultural factors as experienced during clinical rotations. Our study demonstrates that medical students’ clinical
Table 3 Responses of Graduating Fourth-Year Medical Students on the Safety Behavioral Intent Items of the Medical Student Safety Attitudes and Professionalism Survey, 2012 No. (%) responding* Disagree strongly 1
Disagree slightly 2
Neutral 3
Agree slightly 4
Agree strongly 5
1. I expect to participate in quality improvement initiatives 2. I intend to encourage my colleagues to tell patients and their families about medical errors that impacted their care
1 (1) 3 (2)
6 (3) 5 (3)
27 (14) 13 (7)
95 (50) 77 (42)
62 (32) 87 (47)
3. I intend to encourage my colleagues to report any patient safety issues they encounter
4 (2)
2 (1)
6 (3)
75 (40)
100 (53)
Safety behavioral intent item
4. I intend to report any patient safety issues I encounter
3 (1)
3 (1)
9 (4)
65 (32)
121 (60)
5. If I saw a medical error committed by one of my team members, I would keep it to myself
6 (3)
15 (7)
23 (11)
95 (47)
63 (31)
*A total of 228 medical students at three U.S. MD-granting medical schools responded to the survey. Denominators differ by item because of missing data. Row percentages may not add to 100% because of rounding.
348
Academic Medicine, Vol. 89, No. 2 / February 2014
Research Report
Table 4 Spearman Correlations Between Types of Cultural Factors, Overall Patient Safety Grade, and Safety Behavioral Intent Items, Medical Student Safety Attitudes and Professionalism Survey* Cultural factor: r (95% confidence interval) Error Experiences Comfort expressing disclosure with professional culture professionalism concerns
Patient safety measure Overall patient safety grade
Safety culture
Teamwork culture
0.37† (0.19, 0.52)
0.41† (0.24, 0.58)
0.49† (0.34, 0.62)
0.69† (0.59, 0.78)
0.53† (0.39, 0.66)
I expect to participate in quality improvement initiatives
0.04 (–0.14, 0.22)
–0.08 (–0.27, 0.11)
0.05 (–0.15, 0.25)
0.17 (–0.03, 0.36)
0.16 (–0.02, 0.33)
I intend to encourage my colleagues to tell patients and their families about medical errors that impacted their care
0.20† (0.001, 0.38)
0.23† (0.04, 0.40)
0.22† (0.04, 0.40)
0.34† (0.16, 0.51)
0.35† (0.17, 0.50)
I intend to encourage my colleagues to report any patient safety issues they encounter
0.17 (–0.03, 0.35)
0.25† (0.08, 0.43)
0.35† (0.16, 0.52)
0.42† (0.26, 0.57)
0.36† (0.19, 0.52)
I intend to report any patient safety issues I encounter
0.13 (–0.05, 0.30)
0.19† (0.01, 0.36)
0.18 (–0.002, 0.36)
0.31† (0.13, 0.47)
0.34† (0.16, 0.51)
I saw a medical error committed by one of my If team members, I would keep it to myself
0.18 (–0.02, 0.39)
(0.13, 0.49)
0.09 (–0.12, 0.32)
0.20† (–0.01, 0.40)
0.22† (0.02, 0.42)
Safety behavioral intent item
0.32†
*This analysis used data from 228 graduating fourth-year medical students at three U.S. MD-granting medical schools who completed the survey in 2012. † P < .05.
perception scores on the five cultural factors. Overall, these findings provide initial evidence for the validity of the MSSAPS and indicate that perceptions of patient safety may be shaped not only by safety and teamwork cultures but also by experiences with professionalism.32
positively, perhaps because of relatively long-standing efforts by hospitals to measure and make improvements in these areas.23–31 Importantly, each of the five cultural factors was correlated with students’ perceptions of overall patient safety for their clinical rotations (overall patient safety grade), and all of the factors except error disclosure culture were correlated with at least three of the five safety behavioral intent items. Further, the school with the lowest overall patient safety grade also had the lowest positive
Our study is the first to simultaneously measure medical student attitudes about teamwork culture, safety culture, error disclosure culture, and professionalism. It builds on prior studies of student
Table 5 Percent Positive Scores by Cultural Factor or Patient Safety Measure, Medical Student Safety Attitudes and Professionalism Survey, Three U.S. MD-Granting Medical Schools, 2012* Respondents with positive perceptions, % (95% confidence interval) Factor
School A
School B
School C
Safety culture† Teamwork culture†
57 (48–66) 74 (66–81)
28 (13–43) 47 (32–62)
64 (30–98) 71 (44–99)
Error disclosure culture
27 (20–35)
10 (0–20)
15 (0–38)
Experiences with professionalism
23 (16–30)
7 (0–15)
20 (0–50)
Comfort expressing professional concerns
34 (26–43)
17 (5–29)
38 (8–69)
Overall patient safety grade†
70 (62–77)
38 (23–53)
79 (54–100)
*A total of 228 graduating fourth-year medical students responded to the survey: 164 from School A; 49 from School B; and 15 from School C. Denominators differ for each factor at each institution because of missing data. † Significant difference via chi-square test at P < .05.
Academic Medicine, Vol. 89, No. 2 / February 2014
experiences with unprofessional behavior and the hidden curriculum’s impact on patient safety33–35 by demonstrating quantitative correlations between students’ lived experiences in the hidden curriculum (represented by the cultural factors), their perceptions of patient safety, and their intentions regarding their future safetyrelated behaviors. Students frequently report experiencing and assimilating into a culture of unprofessional behaviors36 that likely undermine the explicit teachings of medical school curricula. Despite national attention and focused efforts to tackle unprofessional behavior, a recent study documented a sobering persistence of medical student mistreatment.37 Given the importance of patient safety improvements, our results linking measures of professionalism (i.e., experiences with professionalism and comfort expressing professional concerns) with patient safety measures may provide additional motivation to change medical culture in light of the emphasis that health care professionals, regulators, payers, and the public have placed on improving patient safety. The MSSAPS can be used for internal benchmarking purposes. Given the concurrent validity evidence for the instrument, students’ patient safety perceptions should improve as clinical
349
Research Report
rotation sites make improvements in the areas mentioned above. Educators could also administer the MSSAPS after individual rotations to identify problematic clinical experiences. Organizations could use percent positive scores to determine where to prioritize improvement efforts. For example, on the basis of the scores in this study, all three participating institutions could focus on improvements related to error disclosure culture and professionalism (both experiences with professionalism and comfort expressing concerns; see Table 5). School B could also target safety culture and teamwork culture as areas for improvement. Tracking percent positive scores longitudinally would allow schools to conduct year-to-year comparisons of changes in perceptions. In this way, the MSSAPS could be used at the institution level to identify aspects of clinical rotations and overall culture that may deviate from the mission of training safe, professional physicians. Additionally, the MSSAPS could be useful for external benchmarking and to help medical schools and teaching hospitals answer the call to create cultures of respect that promote ethical, safe behavior.38 By targeting medical students—who are often sensitive to behavioral modeling—this survey could serve as a much-needed tool for measuring the effectiveness of institutional efforts to address disrespectful, unprofessional behavior. This study has several limitations. First, it includes only the experiences of graduating fourth-year medical students from three institutions. Additional work across other medical schools and years of study is required to better understand the generalizability of our results. Second, further validity testing should be done, specifically to probe whether student responses (i.e., perceptions) relate to actual clinical behavior, because some students may not have had enough experience to distinguish between professional and unprofessional behaviors or between safe and unsafe acts. Third, the voluntary participation in School C’s capstone course (and therefore in the MSSAPS) may introduce bias. Fourth, there has traditionally been a lack of student exposure and experience with topics like medical error disclosure, which may limit the validity of student responses to corresponding survey items. Finally, additional research is needed to
350
determine to what degree incremental changes in clinical curricula based on MSSAPS results would be reflected in subsequent administrations of the survey. In conclusion, this study provides initial evidence for the reliability and validity of the MSSAPS, a survey instrument that can be used to determine whether medical students are having clinical experiences in positive patient safety learning environments. Our data suggest that a large proportion of students are exposed during their clinical rotations to behaviors that do not exemplify best practices for safe and professional patient care. However, we believe that by building on our findings that unprofessional behaviors in clinical learning environments are linked to students’ perceptions of patient safety, educators can help create a sense of urgency and encourage more leaders and clinical educators to address both explicit and hidden patient safety curricula—and the institutional cultures that affect them. Acknowledgments: Dr. Bell’s work in humanism was supported through an Arnold P. Gold Foundation Professorship. The authors acknowledge course directors Katharine K. Treadway, MD, and Margaret Uthman, MD, for facilitating access to their students. Other disclosures: None reported. Ethical approval: This study was approved by the institutional review board of the University of Texas Health Science Center at Houston. Dr. Liao is a resident physician, Department of Internal Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, Massachusetts. At the time of writing, he was a fourth-year medical student, Baylor College of Medicine, Houston, Texas. Dr. Etchegaray is assistant professor, Department of Internal Medicine, University of Texas Medical School at Houston and UT-Memorial Hermann Center for Healthcare Quality and Safety, Houston, Texas. Mr. Williams is a fourth-year medical student, University of Texas Health Science Center at Houston, Houston, Texas. Dr. Berger is professor and vice chair, Michael E. DeBakey Department of Surgery, Baylor College of Medicine, and vice president and chief medical officer, Baylor College of Medicine Medical Center, Houston, Texas. Dr. Bell is assistant professor of medicine, division of general medicine and primary care, Harvard Medical School, Boston, Massachusetts. Dr. Thomas is professor of medicine, associate dean for health care quality, and director, UTMemorial Hermann Center for Healthcare Quality and Safety and at University of Texas Medical School at Houston, Houston, Texas.
References 1 Institute of Medicine. Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000. 2 Institute of Medicine. Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001. 3 DesHarnais SI, Nash DB. Reforming way medical students and physicians are taught about quality and safety. Mt Sinai J Med. 2011;78:834–841. 4 Berwick DM, Finkelstein JA. Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new “public interest.” Acad Med. 2010;85:S56–S65. 5 Lucian Leape Institute Roundtable on Reforming Medical Education. Unmet Needs: Teaching Physicians to Provide Safe Patient Care. Boston, Mass: Lucian Leape Institute at the National Patient Safety Foundation; 2010. 6 Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: A systematic review. Acad Med. 2010;85:1425–1439. 7 Chuang AW, Nuthalapaty FS, Casey PM, et al. To the point: Reviews in medical education—taking control of the hidden curriculum. Am J Obstet Gynecol. 2010;203:316.e1–e6. 8 Inui TS. A Flag in the Wind: Educating for Professionalism in Medicine. Washington, DC: Association of American Medical Colleges; 2003. 9 Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69:861–871. 10 Etchegaray JM, Thomas EJ. Comparing two safety culture surveys: Safety Attitudes Questionnaire and Hospital Survey on Patient Safety. BMJ Qual Saf. 2012;21:490– 498. 11 Sexton JB, Helmreich RL, Neilands TB, et al. The Safety Attitudes Questionnaire: Psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6:44. 12 Etchegaray JM, Gallagher TH, Bell SK, Dunlap B, Thomas EJ. Error disclosure: A new domain for safety culture assessment. BMJ Qual Saf. 2012;21:594–599. 13 Feudtner C, Christakis DA, Christakis NA. Do clinical clerks suffer ethical erosion? Students’ perceptions of their ethical environment and personal development. Acad Med. 1994;69:670–679. 14 Wiggleton C, Petrusa E, Loomis K, et al. Medical students’ experiences of moral distress: Development of a Web-based survey. Acad Med. 2010;85:111–117. 15 Agency for Healthcare Research and Quality. Hospital Survey on Patient Safety Culture. http://www.ahrq.gov/qual/ patientsafetyculture/hospsurvindex.htm. Accessed October 17, 2013. 16 Leung GK, Patil NG. Patient safety in the undergraduate curriculum: Medical students’ perception. Hong Kong Med J. 2010;16: 101–105. 17 Reise SP, Widaman KF, Pugh RH. Confirmatory factor analysis and item
Academic Medicine, Vol. 89, No. 2 / February 2014
Research Report
18 19 20
21 22 23
24
25
response theory: Two approaches for exploring measurement invariance. Psychol Bull. 1993;114:552–566. Bentler PM, Bonnet DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull. 1980;88:588–606. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107:238–246. Steiger JH, Lind J. Statistically based tests for the number of common factors. Paper presented at: Annual Meeting of the Psychometric Society; May 1980; Iowa City, Iowa. Bollen KA. Structural Equations With Latent Variables. New York, NY: Wiley; 1989. Kline RB. Principles and Practice of Structural Equation Modeling. New York, NY: The Guilford Press; 2005. Bell SK, Moorman DW, Delbanco T. Improving the patient, family, and clinician experience after harmful events: The “when things go wrong” curriculum. Acad Med. 2010;85:1010–1017. Vigorito MC, McNicoll L, Adams L, Sexton B. Improving safety culture results in Rhode Island ICUs: Lessons learned from the development of action-oriented plans. Jt Comm J Qual Patient Saf. 2011;37:509–514. Carney BT, West P, Neily JB, Mills PD, Bagian JP. Improving perceptions of
Academic Medicine, Vol. 89, No. 2 / February 2014
26
27
28
29
30
31
teamwork climate with the Veterans Health Administration medical team training program. Am J Med Qual. 2011;26: 480–484. Sexton JB, Berenholtz SM, Goeschel CA, et al. Assessing and improving safety climate in a large cohort of intensive care units. Crit Care Med. 2011;39:934–939. Paine LA, Rosenstein BJ, Sexton JB, Kent P, Holzmueller CG, Pronovost PJ. Assessing and improving safety culture throughout an academic medical centre: A prospective cohort study. Qual Saf Health Care. 2010;19:547–554. O’Leary KJ, Wayne DB, Haviley C, Slade ME, Lee J, Williams MV. Improving teamwork: Impact of structured interdisciplinary rounds on a medical teaching unit. J Gen Intern Med. 2010;25:826–832. Huang DT, Clermont G, Kong L, et al. Intensive care unit safety culture and outcomes: A U.S. multicenter study. Int J Qual Health Care. 2010;22:151–161. Pronovost PJ, Berenholtz SM, Goeschel C, et al. Improving patient safety in intensive care units in Michigan. J Crit Care. 2008;23:207–221. Thomas EJ, Sexton JB, Neilands TB, Frankel A, Helmreich RL. The effect of executive walk rounds on nurse safety climate
32
33
34
35
36
37
38
attitudes. A randomized trial of clinical units. BMC Health Serv Res. 2005;5:28. Leape LL, Shore MF, Dienstag JL, et al. A culture of respect, part 1: The nature and causes of disrespectful behavior by physicians. Acad Med. 2012;87:1–8. Walton MM. Hierarchies: The Berlin Wall of patient safety. Qual Saf Health Care. 2006;15:229–230. Brainard AH, Brislen HC. Viewpoint: Learning professionalism: A view from the trenches. Acad Med. 2007;82:1010–1014. Lucey C, Souba W. Perspective: The problem with the problem of professionalism. Acad Med. 2010;85:1018–1024. Reddy ST, Farnan JM, Yoon JD, et al. Third-year medical students’ participation in and perceptions of unprofessional behaviors. Acad Med. 2007;82(10 suppl): S35–S39. Fried JM, Vermillion M, Parker NH, Uijtdehaage S. Eradicating medical student mistreatment: A longitudinal study of one institution’s efforts. Acad Med. 2012;87: 1191–1198. Leape LL, Shore MF, Dienstag JL, et al. A culture of respect, part 2: Creating a culture of respect. Acad Med. 2012;87:1–6
351
