Letters to the Editor Theresa K. Lester, MA Accreditation and program analyst, Office of Institutional Assessment and Planning, Ohio University Heritage College of Osteopathic Medicine, Athens, Ohio.
Shigeru Okada, PhD Assistant professor of pediatrics, Department of Pediatrics, and instructor of record for the secondyear Endocrine and Metabolism course, Ohio University Heritage College of Osteopathic Medicine, Athens, Ohio.
References 1 Prober C, Khan S. Medical education reimagined: A call to action. Acad Med. 2013;88:1407–1410. 2 Chandran L, Fleit H, Shroyer L. Academic medicine change management: The power of the liaison committee on medical education accreditation process. Acad Med. 2013;88:1225–1231. 3 Kotter JP. Leading Change. Boston, Mass: Harvard Business School Press; 1996.
In Reply to Goldberg and to Hurtubise et al: I appreciate the specific issues raised by Dr. Goldberg and by Dr. Hurtubise et al. Dr. Goldberg correctly implies that there is a lack of definitive data supporting an optimal teaching strategy. There is, however, increasing literature regarding the effectiveness of hybrid learning models. A meta-analysis published by the U.S. Department of Education concluded that “on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction” with larger effects if the online learning was blended with faceto-face instruction.1 Former Princeton President William Bowen and the nonprofit organization ITHAKA have published studies demonstrating the effectiveness of a combination of online and in-class instruction in learning outcomes with fewer hours of study than traditional in-class instruction.2 Dr. Goldberg expresses concern that “reducing the opportunity … to establish a framework, integrate course content, and then apply this content to problem solving may be counter to effective education.” I agree. Our proposed model is specifically designed to integrate course content, delivered in digestible doses (e.g., as short videos), before engaging the students in the critical, interactive application exercises. This is aimed to ensure that the relevance of the facts to medical practice is evident and that the knowledge is more sustained.
Academic Medicine, Vol. 89, No. 5 / May 2014
Dr. Goldberg questions several of our general statements about medical education, including our assertion that it is “not compelling,” especially during the preclinical years. This statement is based upon innumerable conversations with our medical students and education deans from across the country. Dr. Goldberg also questions our belief that “much of what will be taught … will prove to be wrong.” One need only consider the recent changes in long-standing recommendations regarding cholesterol control, prostatespecific antigen monitoring, frequency of mammograms, and the value of hormone replacement therapy to recognize that what we “know” continues to evolve. In 2005, Dr. John Ioannidis3 published a manuscript describing the reasons why many published research claims turn out to be wrong. Ioannidis cites literature supporting the concern that “in modern research, false findings may be the majority or even the vast majority of published research claims.” Hurtubise et al present some positive data regarding the use of the flippedclassroom approach at their school and caution that a “change management approach” will be required to implement this model of teaching more broadly in medical education. I could not agree more. Long-standing practices in medicine and education are difficult to change individually, let alone together! Disclosures: None reported. Charles G. Prober, MD Senior associate dean for medical education and professor of pediatrics, microbiology and immunology, Stanford School of Medicine, Stanford, California; [email protected].
References 1 Means B, Toyama Y, Murphy R, Bakia M, Jones K. Evaluation of Evidence-Based Practices in Online Learning. A MetaAnalysis and Review of Online Learning Studies. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development; 2010. 2 Bowen WG, Chingos MM, Lack KL, Nygren TI. Interactive Learning Online at Public Universities: Evidence from Randomized Trials. New York, NY: ITHAKA. 2012. http:// www.sr.ithaka.org/research-publications/ interactive-learning-online-publicuniversities-evidence-randomized-trials. Accessed March 6, 2014. 3 Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2:e124.
Cost Must Be a Theme in Our Measurement of Accountability To the Editor: Baron has presented a fascinating account of the current state of public accountability for graduate medical education (GME) outcomes.1 He is wisely reluctant to recommend too many extra measures and thereby to risk unnecessarily adding to an already heavy measurement burden. Instead he concentrates on themes such as competence, the learning environment, and workforce outcomes. However, there is one theme that I feel he has missed out on—that of cost. It is difficult to think of a context other than GME where authorities discuss accountability without explicit mention of cost. Shouldn’t we have better public accountability because public money is being spent to fund GME programs? Doesn’t the public have the right to ask whether it gets value for the money spent on these programs? Could institutions run lower-cost programs and achieve the same outcomes? Or could a higher quantity and/or quality of trained specialists be produced with the same expenditure? The short answer to all of these questions is that we don’t know because the research has not yet been done.2 Fortunately, there is a growing interest in cost and value in medical education. Measuring the cost of programs, however, is not completely straightforward, as it is often difficult to delineate and separate out the cost of the provision of medical education and the cost of provision of health care. This is particularly true of GME where trainees learn and work at the same time.3 However, some components of education (and therefore their costs) can be separated from health care. These include curriculum design, direct provision of medical education (e.g., through lectures, small-group sessions, or simulation), assessment of trainees, and evaluation of training programs. Even more difficult is the development of measures that touch on cost as well as outcomes. These might include outcomes of good comparative cost-effectiveness, or favorable cost-benefit or cost-utility ratios, or even simple cost feasibility studies.4(p10) Baron touches on these concepts when discussing workforce outcomes, but only tangentially. When discussing workforce outcomes, we are considering our return on investment.
697
Letters to the Editor
In a country that needs more rural physicians, the 26.1% of sponsoring institutions that are producing no rural physicians are providing a low return on investment.1 Themes of accountability within medical education have been around a long time, but we seem to have made fitful progress. Could the overt measurement of cost and value make us realize that we now need to make more rapid progress? Disclosures: None reported. Kieran Walsh, FRCPI Clinical director of BMJ Learning, BMJ Publishing Group, London, UK; [email protected].
References 1 Baron RB. Can we achieve public accountability for graduate medical education outcomes? Acad Med. 2013;88:1199–1201. 2 Walsh K. Cost Effectiveness in Medical Education. Abingdon, UK: Radcliffe; 2010. 3 Williams C. Understanding the essential elements of work-based learning and its relevance to everyday clinical practice. J Nurs Manag. 2010;18:624–632. 4 Levin HM, McEwan PJ. Cost Effectiveness Analysis: Methods and Applications. 2nd ed. Thousand Oaks, Calif: Sage Publications; 2001.
In Reply to Walsh: I fully agree with Dr. Walsh’s comments that cost is an essential element of public accountability for graduate medical education (GME). This is especially true since public money is used to fund GME programs. In fact, most discussions of GME accountability have been driven by proposals to decrease public funding of GME. The creation of accurate and reliable accountability measures as discussed in my commentary would in large part be used to drive a portion of public funding to programs and institutions that meet desired training outcomes. As Dr. Walsh underscores, measuring the costs of GME is not completely straightforward. Some costs, such as the “direct” teaching costs reported on annual Medicare cost reports, are easier to define. These include a portion of trainee and faculty salaries and benefits and a portion of teaching-related overhead costs. Much more challenging is the calculation of “indirect” costs, the additional costs of teaching institutions ascribed to the involvement of residents and fellows in patient care. Most challenging, however, is the measurement of the increased (or decreased) revenue received by health care
698
institutions and providers due to the patient care provided by resident and fellows. Despite the challenges, I agree that more careful cost analyses of GME are both feasible and necessary. The key, however, will be to ensure that all costs and all revenues are captured. In the meantime, GME measures focusing on competence, the learning environment, and workforce outcomes can be initiated immediately to incentivize better GME outcomes and provide public accountability. Disclosures: None reported. Robert B. Baron, MD, MS Professor of medicine and associate dean for graduate and continuing medical education, Division of General Internal Medicine, Department of Medicine, University of California, San Francisco, School of Medicine, San Francisco, California; [email protected].
Population Health Management: An Approach to Improve the Integration of the Health Care and Public Health Systems To the Editor: The Institute of Medicine released a report titled Primary Care and Public Health: Exploring Integration to Improve Population Health1 that called for the creative collaboration of health care and public health systems for the purpose of improving population health. The report stated, “The traditional separation between primary health care providers and public health professionals is impeding greater success in meeting their shared goal of ensuring the health of populations.”1 This call to action is timely as we deliver care in our transformed health care system. Hence, I maintain that one approach to answer this call involves actively intervening with the health of populations via population health management (PHM). PHM is a tool “used to describe a variety of approaches developed to foster health and quality of care improvements while managing costs.”2 PHM utilizes various management approaches that address the prominent disease, contributing lifestyle factors, and resultant disability issues, for instance, via integrating interventions that require input from systems that consider the determinants which most significantly affect the
health of the target population, such as employees or diabetes patients within a health system’s service area.3 PHM makes ethical sense on paper but I argue that its implementation in any health care or public health system is challenging and requires a culture change and a development of skills not necessarily taught in medical education, including engaging community-based participation, or collaboration with nonmedical professionals. Similarly, public health education which addresses community health issues via a population lens needs to teach professionals to expand the practice of their skills to a setting other than the community, but to include an environment that could be a large employer corporation, health system, or hospital. Both stakeholders need to learn to integrate their philosophies and operations since their desired outcome is the same—a healthy population. Since the health care and public health systems are currently unable to implement this approach alone, an integrative method offered by PHM is required and possibly a reinvigoration of the call to reunite medical and public health education. Furthermore, PHM has the potential to contribute to the evaluation of the effectiveness of our reformed health care system since it allows for assessing the efficiency of health care delivery, while striving to improve quality of care and reduce costs. The overarching goal of PHM is to keep populations healthy via an integrative, preventive approach so it is a model that should be embraced by the health care and public health systems, as well as their respective educational systems that produce these practicing professionals. Disclosures: None reported. Rosemary M. Caron, PhD, MPH Associate professor and former MPH program director, University of New Hampshire, Durham, New Hampshire; [email protected].
References 1 Institute of Medicine. Primary Care and Public Health: Exploring Integration to Improve Population Health. Washington, DC: National Academies Press; 2012. 2 McAlearney AS. Population Health Management: Strategies to Improve Outcomes. Chicago, Ill: Health Administration Press; 2003. 3 Birk S. Population health: Strategies that deliver value and results. Healthc Exec. 2013;28(4):10–12.
Academic Medicine, Vol. 89, No. 5 / May 2014
Shigeru Okada, PhD Assistant professor of pediatrics, Department of Pediatrics, and instructor of record for the secondyear Endocrine and Metabolism course, Ohio University Heritage College of Osteopathic Medicine, Athens, Ohio.
References 1 Prober C, Khan S. Medical education reimagined: A call to action. Acad Med. 2013;88:1407–1410. 2 Chandran L, Fleit H, Shroyer L. Academic medicine change management: The power of the liaison committee on medical education accreditation process. Acad Med. 2013;88:1225–1231. 3 Kotter JP. Leading Change. Boston, Mass: Harvard Business School Press; 1996.
In Reply to Goldberg and to Hurtubise et al: I appreciate the specific issues raised by Dr. Goldberg and by Dr. Hurtubise et al. Dr. Goldberg correctly implies that there is a lack of definitive data supporting an optimal teaching strategy. There is, however, increasing literature regarding the effectiveness of hybrid learning models. A meta-analysis published by the U.S. Department of Education concluded that “on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction” with larger effects if the online learning was blended with faceto-face instruction.1 Former Princeton President William Bowen and the nonprofit organization ITHAKA have published studies demonstrating the effectiveness of a combination of online and in-class instruction in learning outcomes with fewer hours of study than traditional in-class instruction.2 Dr. Goldberg expresses concern that “reducing the opportunity … to establish a framework, integrate course content, and then apply this content to problem solving may be counter to effective education.” I agree. Our proposed model is specifically designed to integrate course content, delivered in digestible doses (e.g., as short videos), before engaging the students in the critical, interactive application exercises. This is aimed to ensure that the relevance of the facts to medical practice is evident and that the knowledge is more sustained.
Academic Medicine, Vol. 89, No. 5 / May 2014
Dr. Goldberg questions several of our general statements about medical education, including our assertion that it is “not compelling,” especially during the preclinical years. This statement is based upon innumerable conversations with our medical students and education deans from across the country. Dr. Goldberg also questions our belief that “much of what will be taught … will prove to be wrong.” One need only consider the recent changes in long-standing recommendations regarding cholesterol control, prostatespecific antigen monitoring, frequency of mammograms, and the value of hormone replacement therapy to recognize that what we “know” continues to evolve. In 2005, Dr. John Ioannidis3 published a manuscript describing the reasons why many published research claims turn out to be wrong. Ioannidis cites literature supporting the concern that “in modern research, false findings may be the majority or even the vast majority of published research claims.” Hurtubise et al present some positive data regarding the use of the flippedclassroom approach at their school and caution that a “change management approach” will be required to implement this model of teaching more broadly in medical education. I could not agree more. Long-standing practices in medicine and education are difficult to change individually, let alone together! Disclosures: None reported. Charles G. Prober, MD Senior associate dean for medical education and professor of pediatrics, microbiology and immunology, Stanford School of Medicine, Stanford, California; [email protected].
References 1 Means B, Toyama Y, Murphy R, Bakia M, Jones K. Evaluation of Evidence-Based Practices in Online Learning. A MetaAnalysis and Review of Online Learning Studies. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development; 2010. 2 Bowen WG, Chingos MM, Lack KL, Nygren TI. Interactive Learning Online at Public Universities: Evidence from Randomized Trials. New York, NY: ITHAKA. 2012. http:// www.sr.ithaka.org/research-publications/ interactive-learning-online-publicuniversities-evidence-randomized-trials. Accessed March 6, 2014. 3 Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2:e124.
Cost Must Be a Theme in Our Measurement of Accountability To the Editor: Baron has presented a fascinating account of the current state of public accountability for graduate medical education (GME) outcomes.1 He is wisely reluctant to recommend too many extra measures and thereby to risk unnecessarily adding to an already heavy measurement burden. Instead he concentrates on themes such as competence, the learning environment, and workforce outcomes. However, there is one theme that I feel he has missed out on—that of cost. It is difficult to think of a context other than GME where authorities discuss accountability without explicit mention of cost. Shouldn’t we have better public accountability because public money is being spent to fund GME programs? Doesn’t the public have the right to ask whether it gets value for the money spent on these programs? Could institutions run lower-cost programs and achieve the same outcomes? Or could a higher quantity and/or quality of trained specialists be produced with the same expenditure? The short answer to all of these questions is that we don’t know because the research has not yet been done.2 Fortunately, there is a growing interest in cost and value in medical education. Measuring the cost of programs, however, is not completely straightforward, as it is often difficult to delineate and separate out the cost of the provision of medical education and the cost of provision of health care. This is particularly true of GME where trainees learn and work at the same time.3 However, some components of education (and therefore their costs) can be separated from health care. These include curriculum design, direct provision of medical education (e.g., through lectures, small-group sessions, or simulation), assessment of trainees, and evaluation of training programs. Even more difficult is the development of measures that touch on cost as well as outcomes. These might include outcomes of good comparative cost-effectiveness, or favorable cost-benefit or cost-utility ratios, or even simple cost feasibility studies.4(p10) Baron touches on these concepts when discussing workforce outcomes, but only tangentially. When discussing workforce outcomes, we are considering our return on investment.
697
Letters to the Editor
In a country that needs more rural physicians, the 26.1% of sponsoring institutions that are producing no rural physicians are providing a low return on investment.1 Themes of accountability within medical education have been around a long time, but we seem to have made fitful progress. Could the overt measurement of cost and value make us realize that we now need to make more rapid progress? Disclosures: None reported. Kieran Walsh, FRCPI Clinical director of BMJ Learning, BMJ Publishing Group, London, UK; [email protected].
References 1 Baron RB. Can we achieve public accountability for graduate medical education outcomes? Acad Med. 2013;88:1199–1201. 2 Walsh K. Cost Effectiveness in Medical Education. Abingdon, UK: Radcliffe; 2010. 3 Williams C. Understanding the essential elements of work-based learning and its relevance to everyday clinical practice. J Nurs Manag. 2010;18:624–632. 4 Levin HM, McEwan PJ. Cost Effectiveness Analysis: Methods and Applications. 2nd ed. Thousand Oaks, Calif: Sage Publications; 2001.
In Reply to Walsh: I fully agree with Dr. Walsh’s comments that cost is an essential element of public accountability for graduate medical education (GME). This is especially true since public money is used to fund GME programs. In fact, most discussions of GME accountability have been driven by proposals to decrease public funding of GME. The creation of accurate and reliable accountability measures as discussed in my commentary would in large part be used to drive a portion of public funding to programs and institutions that meet desired training outcomes. As Dr. Walsh underscores, measuring the costs of GME is not completely straightforward. Some costs, such as the “direct” teaching costs reported on annual Medicare cost reports, are easier to define. These include a portion of trainee and faculty salaries and benefits and a portion of teaching-related overhead costs. Much more challenging is the calculation of “indirect” costs, the additional costs of teaching institutions ascribed to the involvement of residents and fellows in patient care. Most challenging, however, is the measurement of the increased (or decreased) revenue received by health care
698
institutions and providers due to the patient care provided by resident and fellows. Despite the challenges, I agree that more careful cost analyses of GME are both feasible and necessary. The key, however, will be to ensure that all costs and all revenues are captured. In the meantime, GME measures focusing on competence, the learning environment, and workforce outcomes can be initiated immediately to incentivize better GME outcomes and provide public accountability. Disclosures: None reported. Robert B. Baron, MD, MS Professor of medicine and associate dean for graduate and continuing medical education, Division of General Internal Medicine, Department of Medicine, University of California, San Francisco, School of Medicine, San Francisco, California; [email protected].
Population Health Management: An Approach to Improve the Integration of the Health Care and Public Health Systems To the Editor: The Institute of Medicine released a report titled Primary Care and Public Health: Exploring Integration to Improve Population Health1 that called for the creative collaboration of health care and public health systems for the purpose of improving population health. The report stated, “The traditional separation between primary health care providers and public health professionals is impeding greater success in meeting their shared goal of ensuring the health of populations.”1 This call to action is timely as we deliver care in our transformed health care system. Hence, I maintain that one approach to answer this call involves actively intervening with the health of populations via population health management (PHM). PHM is a tool “used to describe a variety of approaches developed to foster health and quality of care improvements while managing costs.”2 PHM utilizes various management approaches that address the prominent disease, contributing lifestyle factors, and resultant disability issues, for instance, via integrating interventions that require input from systems that consider the determinants which most significantly affect the
health of the target population, such as employees or diabetes patients within a health system’s service area.3 PHM makes ethical sense on paper but I argue that its implementation in any health care or public health system is challenging and requires a culture change and a development of skills not necessarily taught in medical education, including engaging community-based participation, or collaboration with nonmedical professionals. Similarly, public health education which addresses community health issues via a population lens needs to teach professionals to expand the practice of their skills to a setting other than the community, but to include an environment that could be a large employer corporation, health system, or hospital. Both stakeholders need to learn to integrate their philosophies and operations since their desired outcome is the same—a healthy population. Since the health care and public health systems are currently unable to implement this approach alone, an integrative method offered by PHM is required and possibly a reinvigoration of the call to reunite medical and public health education. Furthermore, PHM has the potential to contribute to the evaluation of the effectiveness of our reformed health care system since it allows for assessing the efficiency of health care delivery, while striving to improve quality of care and reduce costs. The overarching goal of PHM is to keep populations healthy via an integrative, preventive approach so it is a model that should be embraced by the health care and public health systems, as well as their respective educational systems that produce these practicing professionals. Disclosures: None reported. Rosemary M. Caron, PhD, MPH Associate professor and former MPH program director, University of New Hampshire, Durham, New Hampshire; [email protected].
References 1 Institute of Medicine. Primary Care and Public Health: Exploring Integration to Improve Population Health. Washington, DC: National Academies Press; 2012. 2 McAlearney AS. Population Health Management: Strategies to Improve Outcomes. Chicago, Ill: Health Administration Press; 2003. 3 Birk S. Population health: Strategies that deliver value and results. Healthc Exec. 2013;28(4):10–12.
Academic Medicine, Vol. 89, No. 5 / May 2014
