COMMENTARY
Methodological Innovations in Public Health Education: Transdisciplinary Problem Solving In 2008, the faculty of the Brown School at Washington University in St. Louis designed a Master of Public Health program centered on transdisciplinary problem solving in public health. We have described the rationale for our approach, guiding principles and pedagogy for the program, and specific transdisciplinary competencies students acquire. We have explained how transdisciplinary content has been organized and delivered, how the program is being evaluated, and how we have demonstrated the feasibility of this approach for a Master of Public Health degree. (Am J Public Health. 2015;105: S99–S103. doi:10.2105/AJPH. 2014.302462)
Edward F. Lawlor, PhD, Matthew W. Kreuter, PhD, MPH, Anne K. Sebert-Kuhlmann, PhD, MPH, and Timothy D. McBride, PhD
THE ARGUMENT FOR improving public health education through case studies and blending disciplines has been made for the past decade,1,2 setting the stage for interdisciplinary and transdisciplinary education that will build workforce capacity in science and practice to solve complex public health problems. In 2008 the faculty of the Brown School at Washington University in St. Louis embarked on the design of a Master of Public Health (MPH) degree program that would operationalize this mandate for public health education.3 The charge to the faculty was to begin with a blank slate and be innovative yet be sure to integrate all the requirements of the Council for Public Health Education for an accredited program. The Council for Public Health Education was highly supportive of this approach to program and curriculum development from the outset. Working across disciplines to address a common challenge can take different forms. Multidisciplinary work is typically characterized by a sequential or additive combination of ideas or methods. Interdisciplinary approaches involve sharing and coordination across fields but with participants still anchored in the models and methods of their own discipline. By contrast, transdisciplinary approaches involve developing shared new frameworks that integrate and extend concepts and methods from among different disciplines, thereby transcending disciplinary boundaries.4---6 Our
Supplement 1, 2015, Vol 105, No. S1 | American Journal of Public Health
focus on transdisciplinary problem solving had its roots in a broader institutional philosophy of public health. In 2008, Washington University launched a university-wide public health initiative dedicated to involving all seven schools— architecture, arts and sciences, engineering, business, law, medicine, and social work—in collaborative public health research and teaching. The integrating structure, the Institute for Public Health, appointed 165 public health scholars from across these seven schools. The Brown School, a longstanding and premier school of social work with a long history of contributions in mental health research, also committed itself to the ideal of transdisciplinary professional education in public health focused on breaking down academic silos and integrating nonacademic and nonpublic health-focused community partners into the educational model. The initial concept of the MPH program was explicitly transdisciplinary. Student recruitment materials, curricular design, community partnerships, and faculty recruitment all reflected this priority. The uptake of this idea was immediate and enthusiastic: the school recruited faculty and students who were looking for this approach to public health education and research. The public health faculty of the Brown School reviewed the latest thinking and key concepts from the literature on transdisciplinary science,7---15 consulted with leaders
of the transdisciplinary movement, and invited numerous faculty from other public health schools and programs to learn about models and challenges in providing transdisciplinary education. Especially influential in this process was the work of Daniel Stokols on transdisciplinarity, team science, and evaluation, which is captured in this recent definition of transdisciplinarity: An integrative process whereby scholars and practitioners from both academic disciplines and non-academic fields work jointly to develop and use novel conceptual and methodological approaches that synthesize and extend discipline-specific perspectives, theories, methods, and translational strategies to yield innovative solutions to particular scientific and societal problems.16(p6)
Several central ideas emerged from the process of bringing transdisciplinary approaches into the curriculum planning process: d
d
d
Teaching and student work should engage the relevant communities, leaders, and organizations necessary for problem understanding and solutions. Public health education needs to encompass the entire arc from underlying science to the organizational, social, and policy challenges of interventions. Public health education should be intensely applied and problem driven, with students pushed to design and implement solutions to important real-world public health problems that draw on evidence when it is available but also are innovative and feasible.
Lawlor et al. | Peer Reviewed | Commentary | S99
COMMENTARY
d
The courses should integrate team-based learning.
The core innovation that resulted from this process was the creation of required transdisciplinary problem-solving courses, or, as they came to be known, TPS courses.
TRANSDISCIPLINARY COMPETENCIES We followed guidance from the Council for Public Health Education to develop competencies that span the five core disciplines of public health and the seven crosscutting thematic areas.17 However, neither Council for Public Health Education guidance nor the Council on Linkages Between Academia and Public Health Practice’s Core Competencies for Public Health Professionals18 includes competencies focused on transdisciplinary education. So we developed a set of competencies for transdisciplinary problem solving in public health. In an iterative process among faculty members, we established seven such competencies that conveyed the base of knowledge and skills we expected students to master.19 These are as follows: 1. Explaining why the complex, multifactorial nature of problems in public health and health disparities requires a transdisciplinary approach 2. Describing how social, economic, behavioral, environmental, and biological conditions contribute to health outcomes using theoretical approaches drawn from diverse disciplines 3. Distinguishing the features of transdisciplinary collaboration 4. Defining problems in a transdisciplinary way and developing shared conceptual
frameworks from disciplinespecific theories and models 5. Developing and applying processes that integrate and promote transdisciplinary perspectives, contributions, and collaborations 6. Applying transdisciplinary solutions to public health problems using appropriate analytical tools drawn from public health or other disciplines 7. Demonstrating the ability to communicate transdisciplinary research evidence to key stakeholders to influence policy and practice As is typical in any MPH curriculum, the competencies to be developed in each course were spelled out in the syllabus. In addition, we mapped TPS competencies by course in a matrix that indicates which courses are responsible for the primary development of each competency and which reinforce or extend each competency. This process flagged course---competency gaps and helped us adapt courses accordingly. The goal was to have each competency developed in more than one TPS course.
CONCEPTUAL FRAMEWORK The conceptual framework for this educational program, drawing from Stokols,20 looks to evaluate the quality, appropriateness, and implementation of problemsolving approaches; the competencies acquired by students; and the long-term capacity of graduates to advance innovative public health solutions. Because this approach is embedded in professional education, our framing and criteria for evaluation are somewhat different from those applied
S100 | Commentary | Peer Reviewed | Lawlor et al.
to transdisciplinary science or transdisciplinary collaboration.15 These distinctions are important because much of the literature on transdisciplinarity focuses on the behavior of scientific teams, in which intellectual products are the outcome of interest, or collaboration, in which the outcomes of interests are the dynamics and behavior of groups or teams. In public health graduate education, we are interested in both the competencies and skills of students to perform this style of public health practice as well as the innovation and utility of solutions that are produced for improving public health. Several evaluative domains guide this curriculum and approach to public health education. First, does the intermixing of disciplines, scientists, practitioners, and community produce original and productive reframing of public health problems and generate effective solutions? Second, does the transdisciplinary approach help develop skills, competencies, habits of mind, and the propensity to work effectively with others to solve public health problems? Third, does this educational approach produce distinctive lifelong career choices, transdisciplinary work styles, and an innovative impact in public health? The first domain is assessed by the work products themselves, including judging performed by clients and external organizations. Ultimately, the outcome test of these courses is whether the proposed solutions are implemented in policy or practice. The second domain is assessed by evaluating competencies for transdisciplinary work, most directly by reviewing students’ culminating experiences. The formal and rigorous assessment of transdisciplinary skills and competencies is still evolving in
our program. Deeper questions about the capacity of students to engage in effective group work, thoroughly integrate disciplines, and incorporate the arc of cells to society in formulating solutions will need more careful conceptualization, measurement, and evaluation. Although we have plans to longitudinally assess the lifetime application of this transdisciplinary approach in graduates’ careers, it will take time for this measurement to occur.
PEDAGOGY Every TPS course begins with the identification of the multiple factors that influence a specific public health problem. Students are coached to recognize how the problem is defined and addressed by different science and practice disciplines as well as how it may be seen from the perspective of those with different political ideologies. Students must demonstrate the ability to integrate these ideas into a more comprehensive understanding of problem and solution. Non---public-health-focused community partners play an important role in many TPS courses and can be engaged in a variety of ways. For example, a course may be structured to have students apply transdisciplinary thinking to priority needs of a community partner. In a TPS course on criminal justice, students work on policies addressing the occupational experiences of corrections officers—a need identified by the Department of Public Safety. At semester’s end, the Department of Public Safety invited students to present their solutions to the Division of Corrections personnel department and the mayor’s office at the city hall. Community partners also can provide insight into problems
American Journal of Public Health | Supplement 1, 2015, Vol 105, No. S1
COMMENTARY
from a nontraditional public health perspective. For example, horticulturalists from the Missouri Botanical Garden have instructed students in a TPS course on the built environment on sustainability practices such as rain gardens and the use of plants in and around homes. Neighborhood stabilization team members from the city of St. Louis have described their role in improving housing and safety for residents in urban neighborhoods. Finally, external partners can be engaged in evaluating the transdisciplinary solutions that students generate. This often occurs through participation on review panels at the end of a TPS course. In one such course on health disparities, agency leaders from Social Services and Tobacco Control in Missouri selected and actually implemented a student proposal to connect food stamp recipients in the state with the Missouri Tobacco Quitline.21
TRANSDISCIPLINARY PROBLEM-SOLVING COURSES In the first five years of our MPH program, we developed 16 TPS courses that collectively were offered 34 times. Three additional courses codeveloped with or taught by faculty from the School of Law, the School of Design, and the Department of Anthropology also were approved as TPS courses. Our courses tackled public health challenges such as child obesity, global undernutrition, pregnancy outcomes, sexual health, tobacco, and child maltreatment. Although all TPS courses adhered to the requirements, goals, and competencies spelled out previously, faculty were encouraged to innovate how their TPS course was structured and
delivered. As a result, a variety of approaches emerged, with courses varying in (1) the selection and organization of transdisciplinary content, (2) the relative emphasis placed on transdisciplinary understanding versus solutions, and (3) the instructional model for delivering transdisciplinary learning. An example of this framework and outcome of a TPS approach is the Healthy Families Initiative Implementation student team project in remote rural India.22 This project was part of an eight-week TPS course on implementing public health interventions in developing countries that was jointly sponsored by the Brown School and the IKP Centre for Technologies in Public Health.23 Student teams collaborated with practitioners, scientists in India, nongovernmental organizations, and community residents to design and evaluate novel interventions for stubbornly challenging public health problems. Projects included improving cardiovascular health, reducing oropharyngeal cancer, reducing diarrheal disease, addressing dental disease, and reducing high levels of anemia among women. In Healthy Families, the reduction of maternal anemia was conceptualized not as the delivery of appropriate micronutrients but rather as a complex sequence of understanding the epidemiology of maternal anemia (which affects 59.3% of pregnant women in the Thanjavur district), diagnostics; cultural and social determinants of nutrition, the evidence about effectiveness of educational and provider strategies, compliance behavior, appropriate and innovative evaluation criteria, and systems design for implementation.
Organizing Trandisciplinary Content Our TPS courses have used three approaches. The first, cells to
Supplement 1, 2015, Vol 105, No. S1 | American Journal of Public Health
society, organizes disciplinary perspectives by levels or pathways of influence on a public health problem. This approach was popularized in biomedical and health science research communities and embraced by the National Institutes of Health and other institutions.21,24 It considers how biological, behavioral, environmental, policy, economic, and other processes and factors—as well as the interplay among them—contribute to a public health problem. As an example, a TPS course on obesity examined how bacteria in the gut microbiome, biological mechanisms that regulate metabolism, lifestyle behaviors such as diet and activity, agricultural and economic policies, and food marketing and labeling regulations can influence obesity. Considering these perspectives together brings a richer understanding of obesity causes and challenges students to generate a more integrative model of obesity prevention. A variant of this approach focuses on spheres of influence that are not inherently hierarchical as are the levels of an ecological model but rather that operate within a single level, such as the social environment. For example, a TPS course on child maltreatment explores how youths affected by violence or neglect are likely to be involved in many health and social service systems including child welfare, law enforcement, courts, medical care, mental health care, and housing shelters. Understanding these touch points and the goals of each system prepares students to think more broadly about potential opportunities for a coordinated and effective response. Other courses have used a strategic blending of selected disciplinary perspectives. By contrast to TPS courses that examine
a problem from many different angles, this approach focuses on a few handpicked disciplines that may be ripe for integration, with promising but untapped synergy. It reflects a purposeful exploration of the intersection between disciplines to solve a public health problem. For instance, one TPS course sought to integrate health and social services systems to reduce the prevalence and impact of smoking in low-income populations. The course focused on practice disciplines in public health (tobacco control) and social services (food security) as well as theory and research from the field of organizational behavior, specifically within public agencies.25 Students in this course worked in teams to develop integration strategies and presented these as proposals to agency leaders.
Understanding vs Solutions Each TPS course covers the continuum from understanding to solving a public health problem, with corresponding goals of helping students think and act differently on the basis of a transdisciplinary perspective. But the relative emphasis on understanding versus solving varies from course to course. For example, a TPS course on pregnancy outcomes strives for a deeper understanding of the myriad exposures that contribute to adverse maternal and child outcomes. Others, such as a health disparities course in which students learn to communicate effectively about social determinants of health to policy makers and the public, emphasize generating promising solutions. These differences reflect the background, interests, and orientation of the instructor but also the maturity of the knowledge base on a particular public health
Lawlor et al. | Peer Reviewed | Commentary | S101
COMMENTARY
challenge. They are often reflected in the kinds of assignments or products students complete in a course. For example, TPS courses focused on understanding might culminate in the development of integrative causal models, whereas solution-focused courses could generate specific strategies for prevention.
Instructional Models Most instructors have delivered their TPS courses using one or more of four instructional approaches, each with its strengths and limitations. Team teaching pairs instructors from different disciplines central to the course content. Besides the obvious advantages of broader expertise, this model also tends to draw students from different disciplines across campus (usually those of the participating instructors), which can create valuable interactions among students with different training and perspectives. But we have found it can also create administrative challenges assigning or dividing teaching credit for instructors as well as allocating limited seats in the class across schools. Some TPS courses consider so many disciplines that no two instructors could reasonably be expected to have deep knowledge of all content to be covered. In these cases, some TPS courses have opted for an expert-of-the-week model. Like team teaching, its main advantage is the breadth of expertise students are exposed to. However, we have found this guest speaker--dependent model hard to sustain because leading experts may not be consistently available to contribute every semester or year, especially to a course outside their school or department. Recording their live contributions for future use seems a reasonable solution to this challenge. Several TPS courses use a design studio approach modeled
after architectural training. Design studio work focuses on time-limited projects (usually one semester) addressing “complex and openended problems” through “rapid iteration of design solutions.”26(p349) It is informed by frequent formal and informal critique from peers and instructors as well as by precedent—examining past solutions to problems deemed relevant to the current project. In some TPS courses, students spend up to half of each class session (60---90 min) working on transdisciplinary solutions to a public health problem. During these periods, instructors provide students with feedback and suggestions. Students typically spend much more time working on their solutions outside class than in class, but the design studio approach ensures that they receive regular feedback throughout the development process. In our experience, this type of learning is initially unfamiliar to many public health students and requires coaching (and coaxing) to get meaningful peer critiques early on. It is a better fit for solutionfocused TPS courses. Finally, some TPS courses have been taught in situ, in environments as diverse as rural India, Haiti, and Washington, DC, and have relied much more on experiential learning, faculty supervision, and field applications. These courses are so popular among students we have had to adopt a competitive application process for enrollment because demand exceeds course capacity. On the down side, they are extremely timeconsuming to plan, logistically complicated, and expensive to execute.
FACULTY, CULTURE, AND INSTITUTIONAL SUPPORT FOR TPS A critical element in creating the environment for transdisciplinary public health education,
S102 | Commentary | Peer Reviewed | Lawlor et al.
reinforcing its importance, and promoting a distinctive style of education is recruiting and retaining faculty who exemplify this approach. The school’s faculty recruitment process purposely selects talented faculty with the training and aspirations to engage in team science and transdisciplinary learning. A faculty committee investigated research environments—such as the Bell Labs—that leveraged transdisciplinary scientific approaches. An internal network analysis of faculty research and teaching investigated cross-disciplinary faculty research, teaching, and service activities. The Promotion and Tenure Committee took up the issue of how transdisciplinary research and education could be credited and recognized in promotion and tenure reviews. School administrators recognized that many TPS courses would require resources beyond what is needed in a typical course and created a specific budget to support them. In short, institutional support and engagement at all levels of the school have been critical to building this approach.
CONCLUSIONS Transdisciplinary public health is a concept that is compelling in principle but difficult in practice. As schools and programs of public health continue to innovate with new approaches to address complex public health problems, the experiences we have described show that it is feasible to provide transdisciplinary graduate education in public health, with significant innovation in teaching methods, partnerships, and even the recruitment and promotion of faculty. We do not presume that our approach is the best or only way to organize and deliver
transdisciplinary public health education, but we hope that sharing it will contribute to growing discussion and action on this important topic and help public health educators think about ways to integrate transdisciplinary learning activities. Although we have not yet conducted a formal evaluation of our TPS model, we have gathered evaluative data in several forms. We adapted our standard course evaluation to include special items for TPS courses (e.g., assessing how well the course fostered a transdisciplinary approach, included perspectives from diverse disciplines, or used a cells-to-society approach). We also have substantial anecdotal evidence of impact from the perspective of faculty members, students, and prospective students. Faculty members’ interest in developing and teaching TPS courses quickly grew to the point that we had to implement a multistage committee process to review, approve, and track TPS-designated courses. Student demand for the courses also grew, and many wanted to take TPS courses beyond the program requirements and use them to fulfill elective credits. As a result, we have had to control registration to ensure that students needing to meet their TPS requirements have enrollment priorities over those who have already completed the requirement. In addition, every year a significant portion of applicants to our MPH program indicates (without solicitation) that their interest in our program is closely tied to our transdisciplinary focus. Part of this almost certainly reflects prospective students customizing their applications to match our program’s identity, but it is equally clear that there is a sizable niche of
American Journal of Public Health | Supplement 1, 2015, Vol 105, No. S1
COMMENTARY
students with not only genuine interest in transdisciplinary thinking but also undergraduate or professional experiences learning and applying this approach. Finally, it is remarkable how quickly, thoroughly, and palpably the TPS perspective has permeated our school’s culture and identity. “Transdisciplinary” has become as commonplace as “population” or “prevention” in any discussion of public health and the Brown School. And it is not spoken just by those in the school but also by top university leaders, board members, and community partners. Although influential and important public health leaders have emphasized the importance of training transdisciplinary public health professionals, we are still in a transitional period in which careers, faculty incentives, and professional reinforcements have not caught up. It is important for the public health field writ large to embrace this approach to create a cadre of transdisciplinary scientists and practitioners who will promote systemic change. Students and faculty are motivated to apply this approach to public health education and research. j
About the Authors Edward F. Lawlor, Matthew W. Kreuter, Anne K. Sebert-Kuhlmann, and Timothy D. McBride are with the Brown School, Washington University, St. Louis, MO. Correspondence should be sent to Matthew W. Kreuter, PhD, MPH, Kahn Family Professor and Associate Dean for Public Health, CB-1009, 700 Rosedale Avenue, St. Louis, MO 63112 (e-mail: mkreuter@wustl. edu). Reprints can be ordered at http://www. ajph.org by clicking the “Reprints” link. This article was accepted November 7, 2014.
Contributors E. F. Lawlor and M. W. Kreuter outlined the article. All authors contributed to writing and editing the article.
References 1. National Research Council. Facilitating Interdisciplinary Research. Washington, DC: National Academies Press; 2004. 2. Gebbie K, Rosenstock L, Hernandez LM. Future role of schools of public health in educating public health professionals for the 21st century. In: Gebbie K, Rosenstock L, Hernandez LM, eds. Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century. Washington, DC: National Academies Press; 2003:108---128. 3. Luke D, Moreland-Russell S, Herbers S. Measuring success: an evaluation framework for transdisciplinary public health. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Methods, and Practice. San Francisco, CA: Jossey-Bass; 2013:89---90. 4. Stokols D. Toward a science of transdisciplinary action research. Am J Community Psychol. 2006;38(1---2):63---77. 5. Graybill JK, Dooling S, Shandas V, Withey J, Greve A, Simon GL. A rough guide to interdisciplinarity: graduate student perspectives. Bioscience. 2006;56 (9):757---763. 6. Mâsse LC, Moser RP, Stokols D, et al. Measuring collaboration and transdisciplinary integration in team science. Am J Prev Med. 2008;35(2 suppl):S151--S160. 7. Abrams DB. Applying transdisciplinary research strategies to understanding and eliminating health disparities. Health Educ Behav. 2006;33(4):515---531. 8. Nash JM, Collins BN, Loughlin SE, et al. Training the transdisciplinary scientist: a general framework applied to tobacco use behavior. Nicotine Tob Res. 2003;5(suppl 1):S41---S53. 9. Stokols D, Harvey R, Gress J, Fuqua J, Phillips K. In vivo studies of transdisciplinary scientific collaboration: lessons learned and implications for active living research. Am J Prev Med. 2005;28 (2 suppl 2):202---213. 10. Stokols D, Hall KL, Taylor BK, Moser RP. The science of team science: overview of the field and introduction to the supplement. Am J Prev Med. 2008;35 (2 suppl):S77---S89. 11. Rashid JR, Spengler RF, Wagner RM, et al. Eliminating health disparities through transdisciplinary research, crossagency collaboration, and public participation. Am J Public Health. 2009;99 (11):1955---1961. 12. Emmons KM, Viswanath K, Colditz GA. The role of transdisciplinary collaboration in translating and disseminating health research: lessons learned and exemplars of success. Am J Prev Med. 2008;35(2 suppl):S204---S210.
Supplement 1, 2015, Vol 105, No. S1 | American Journal of Public Health
13. Ruddy G, Rhee K. Transdisciplinary teams in primary care for the underserved: a literature review. J Health Care Poor Underserved. 2005;16(2):248---256. 14. Israel BA, Schulz AJ, Parker EA, Becker AB. Review of communitybased research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19: 173---202. 15. Stokols D, Fuqua J, Gress J, et al. Evaluating transdisciplinary science. Nicotine Tob Res. 2003;5(suppl 1):S21---S39. 16. Stokols D, Hall KL, Vogel AL. Transdisciplinary public health: definitions, core characteristics, and strategies for success. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: Jossey-Bass; 2013:3---30.
24. Patterson RE, Colditz GA, Hu FB, et al. The 2011---2016 Transdisciplinary Research on Energetics and Cancer (TREC) initiative: rationale and design. Cancer Causes Control. 2013;24(4): 695---704. 25. Kreuter MW, Pfeiffer D. Transdisciplinary problem solving for integrating public health and social service systems to address health disparities. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: Jossey-Bass; 2013:249---263. 26. Kuhn S. Learning from the architecture studio: implications for projectbased pedagogy. Int J Eng Educ. 2001; 17(4---5):349---352.
17. Calhoun JG, Ramiah K, Weist EM, Shortell SM. Development of a core competency model for the master of public health degree. Am J Public Health. 2008;98(9):1598---1607. 18. Public Health Foundation. Core competencies for public health professionals. Available at: http://www.phf.org/ programs/corecompetencies/Pages/ About_the_Core_Competencies_for_ Public_Health_Professionals.aspx. Accessed on October 24, 2014. 19. Arnold LD, Hipp JA, Kuhlmann AS, Budd E. Competencies in transdisciplinary public health education. In: HaireJoshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: JosseyBass; 2013:53---77. 20. Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The ecology of team science: understanding contextual influences on transdisciplinary collaboration. Am J Prev Med. 2008;35(2 suppl):S96--S115. 21. Morgan GD, Kobus K, Gerlach KK, et al. Facilitating transdisciplinary research: the experience of the transdisciplinary tobacco use research centers. Nicotine Tob Res. 2003;5(suppl 1):S11---S19. 22. Raghavan R, Chockalingam R, Johar Z. Implementing public health interventions in developing countries: a transdisciplinary solution for safe drinking water in rural India. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: Jossey-Bass; 2013:297--317. 23. Armstrong E, Shuckerow DS, Tafoya A, Raghavan R, Johar Z. Maternal and child health intervention: healthy families initiative. 2012. Available at: http:// www.ictph.org.in/tps-2011/paper5introduction.html. Accessed on January 8, 2015.
Lawlor et al. | Peer Reviewed | Commentary | S103
Methodological Innovations in Public Health Education: Transdisciplinary Problem Solving In 2008, the faculty of the Brown School at Washington University in St. Louis designed a Master of Public Health program centered on transdisciplinary problem solving in public health. We have described the rationale for our approach, guiding principles and pedagogy for the program, and specific transdisciplinary competencies students acquire. We have explained how transdisciplinary content has been organized and delivered, how the program is being evaluated, and how we have demonstrated the feasibility of this approach for a Master of Public Health degree. (Am J Public Health. 2015;105: S99–S103. doi:10.2105/AJPH. 2014.302462)
Edward F. Lawlor, PhD, Matthew W. Kreuter, PhD, MPH, Anne K. Sebert-Kuhlmann, PhD, MPH, and Timothy D. McBride, PhD
THE ARGUMENT FOR improving public health education through case studies and blending disciplines has been made for the past decade,1,2 setting the stage for interdisciplinary and transdisciplinary education that will build workforce capacity in science and practice to solve complex public health problems. In 2008 the faculty of the Brown School at Washington University in St. Louis embarked on the design of a Master of Public Health (MPH) degree program that would operationalize this mandate for public health education.3 The charge to the faculty was to begin with a blank slate and be innovative yet be sure to integrate all the requirements of the Council for Public Health Education for an accredited program. The Council for Public Health Education was highly supportive of this approach to program and curriculum development from the outset. Working across disciplines to address a common challenge can take different forms. Multidisciplinary work is typically characterized by a sequential or additive combination of ideas or methods. Interdisciplinary approaches involve sharing and coordination across fields but with participants still anchored in the models and methods of their own discipline. By contrast, transdisciplinary approaches involve developing shared new frameworks that integrate and extend concepts and methods from among different disciplines, thereby transcending disciplinary boundaries.4---6 Our
Supplement 1, 2015, Vol 105, No. S1 | American Journal of Public Health
focus on transdisciplinary problem solving had its roots in a broader institutional philosophy of public health. In 2008, Washington University launched a university-wide public health initiative dedicated to involving all seven schools— architecture, arts and sciences, engineering, business, law, medicine, and social work—in collaborative public health research and teaching. The integrating structure, the Institute for Public Health, appointed 165 public health scholars from across these seven schools. The Brown School, a longstanding and premier school of social work with a long history of contributions in mental health research, also committed itself to the ideal of transdisciplinary professional education in public health focused on breaking down academic silos and integrating nonacademic and nonpublic health-focused community partners into the educational model. The initial concept of the MPH program was explicitly transdisciplinary. Student recruitment materials, curricular design, community partnerships, and faculty recruitment all reflected this priority. The uptake of this idea was immediate and enthusiastic: the school recruited faculty and students who were looking for this approach to public health education and research. The public health faculty of the Brown School reviewed the latest thinking and key concepts from the literature on transdisciplinary science,7---15 consulted with leaders
of the transdisciplinary movement, and invited numerous faculty from other public health schools and programs to learn about models and challenges in providing transdisciplinary education. Especially influential in this process was the work of Daniel Stokols on transdisciplinarity, team science, and evaluation, which is captured in this recent definition of transdisciplinarity: An integrative process whereby scholars and practitioners from both academic disciplines and non-academic fields work jointly to develop and use novel conceptual and methodological approaches that synthesize and extend discipline-specific perspectives, theories, methods, and translational strategies to yield innovative solutions to particular scientific and societal problems.16(p6)
Several central ideas emerged from the process of bringing transdisciplinary approaches into the curriculum planning process: d
d
d
Teaching and student work should engage the relevant communities, leaders, and organizations necessary for problem understanding and solutions. Public health education needs to encompass the entire arc from underlying science to the organizational, social, and policy challenges of interventions. Public health education should be intensely applied and problem driven, with students pushed to design and implement solutions to important real-world public health problems that draw on evidence when it is available but also are innovative and feasible.
Lawlor et al. | Peer Reviewed | Commentary | S99
COMMENTARY
d
The courses should integrate team-based learning.
The core innovation that resulted from this process was the creation of required transdisciplinary problem-solving courses, or, as they came to be known, TPS courses.
TRANSDISCIPLINARY COMPETENCIES We followed guidance from the Council for Public Health Education to develop competencies that span the five core disciplines of public health and the seven crosscutting thematic areas.17 However, neither Council for Public Health Education guidance nor the Council on Linkages Between Academia and Public Health Practice’s Core Competencies for Public Health Professionals18 includes competencies focused on transdisciplinary education. So we developed a set of competencies for transdisciplinary problem solving in public health. In an iterative process among faculty members, we established seven such competencies that conveyed the base of knowledge and skills we expected students to master.19 These are as follows: 1. Explaining why the complex, multifactorial nature of problems in public health and health disparities requires a transdisciplinary approach 2. Describing how social, economic, behavioral, environmental, and biological conditions contribute to health outcomes using theoretical approaches drawn from diverse disciplines 3. Distinguishing the features of transdisciplinary collaboration 4. Defining problems in a transdisciplinary way and developing shared conceptual
frameworks from disciplinespecific theories and models 5. Developing and applying processes that integrate and promote transdisciplinary perspectives, contributions, and collaborations 6. Applying transdisciplinary solutions to public health problems using appropriate analytical tools drawn from public health or other disciplines 7. Demonstrating the ability to communicate transdisciplinary research evidence to key stakeholders to influence policy and practice As is typical in any MPH curriculum, the competencies to be developed in each course were spelled out in the syllabus. In addition, we mapped TPS competencies by course in a matrix that indicates which courses are responsible for the primary development of each competency and which reinforce or extend each competency. This process flagged course---competency gaps and helped us adapt courses accordingly. The goal was to have each competency developed in more than one TPS course.
CONCEPTUAL FRAMEWORK The conceptual framework for this educational program, drawing from Stokols,20 looks to evaluate the quality, appropriateness, and implementation of problemsolving approaches; the competencies acquired by students; and the long-term capacity of graduates to advance innovative public health solutions. Because this approach is embedded in professional education, our framing and criteria for evaluation are somewhat different from those applied
S100 | Commentary | Peer Reviewed | Lawlor et al.
to transdisciplinary science or transdisciplinary collaboration.15 These distinctions are important because much of the literature on transdisciplinarity focuses on the behavior of scientific teams, in which intellectual products are the outcome of interest, or collaboration, in which the outcomes of interests are the dynamics and behavior of groups or teams. In public health graduate education, we are interested in both the competencies and skills of students to perform this style of public health practice as well as the innovation and utility of solutions that are produced for improving public health. Several evaluative domains guide this curriculum and approach to public health education. First, does the intermixing of disciplines, scientists, practitioners, and community produce original and productive reframing of public health problems and generate effective solutions? Second, does the transdisciplinary approach help develop skills, competencies, habits of mind, and the propensity to work effectively with others to solve public health problems? Third, does this educational approach produce distinctive lifelong career choices, transdisciplinary work styles, and an innovative impact in public health? The first domain is assessed by the work products themselves, including judging performed by clients and external organizations. Ultimately, the outcome test of these courses is whether the proposed solutions are implemented in policy or practice. The second domain is assessed by evaluating competencies for transdisciplinary work, most directly by reviewing students’ culminating experiences. The formal and rigorous assessment of transdisciplinary skills and competencies is still evolving in
our program. Deeper questions about the capacity of students to engage in effective group work, thoroughly integrate disciplines, and incorporate the arc of cells to society in formulating solutions will need more careful conceptualization, measurement, and evaluation. Although we have plans to longitudinally assess the lifetime application of this transdisciplinary approach in graduates’ careers, it will take time for this measurement to occur.
PEDAGOGY Every TPS course begins with the identification of the multiple factors that influence a specific public health problem. Students are coached to recognize how the problem is defined and addressed by different science and practice disciplines as well as how it may be seen from the perspective of those with different political ideologies. Students must demonstrate the ability to integrate these ideas into a more comprehensive understanding of problem and solution. Non---public-health-focused community partners play an important role in many TPS courses and can be engaged in a variety of ways. For example, a course may be structured to have students apply transdisciplinary thinking to priority needs of a community partner. In a TPS course on criminal justice, students work on policies addressing the occupational experiences of corrections officers—a need identified by the Department of Public Safety. At semester’s end, the Department of Public Safety invited students to present their solutions to the Division of Corrections personnel department and the mayor’s office at the city hall. Community partners also can provide insight into problems
American Journal of Public Health | Supplement 1, 2015, Vol 105, No. S1
COMMENTARY
from a nontraditional public health perspective. For example, horticulturalists from the Missouri Botanical Garden have instructed students in a TPS course on the built environment on sustainability practices such as rain gardens and the use of plants in and around homes. Neighborhood stabilization team members from the city of St. Louis have described their role in improving housing and safety for residents in urban neighborhoods. Finally, external partners can be engaged in evaluating the transdisciplinary solutions that students generate. This often occurs through participation on review panels at the end of a TPS course. In one such course on health disparities, agency leaders from Social Services and Tobacco Control in Missouri selected and actually implemented a student proposal to connect food stamp recipients in the state with the Missouri Tobacco Quitline.21
TRANSDISCIPLINARY PROBLEM-SOLVING COURSES In the first five years of our MPH program, we developed 16 TPS courses that collectively were offered 34 times. Three additional courses codeveloped with or taught by faculty from the School of Law, the School of Design, and the Department of Anthropology also were approved as TPS courses. Our courses tackled public health challenges such as child obesity, global undernutrition, pregnancy outcomes, sexual health, tobacco, and child maltreatment. Although all TPS courses adhered to the requirements, goals, and competencies spelled out previously, faculty were encouraged to innovate how their TPS course was structured and
delivered. As a result, a variety of approaches emerged, with courses varying in (1) the selection and organization of transdisciplinary content, (2) the relative emphasis placed on transdisciplinary understanding versus solutions, and (3) the instructional model for delivering transdisciplinary learning. An example of this framework and outcome of a TPS approach is the Healthy Families Initiative Implementation student team project in remote rural India.22 This project was part of an eight-week TPS course on implementing public health interventions in developing countries that was jointly sponsored by the Brown School and the IKP Centre for Technologies in Public Health.23 Student teams collaborated with practitioners, scientists in India, nongovernmental organizations, and community residents to design and evaluate novel interventions for stubbornly challenging public health problems. Projects included improving cardiovascular health, reducing oropharyngeal cancer, reducing diarrheal disease, addressing dental disease, and reducing high levels of anemia among women. In Healthy Families, the reduction of maternal anemia was conceptualized not as the delivery of appropriate micronutrients but rather as a complex sequence of understanding the epidemiology of maternal anemia (which affects 59.3% of pregnant women in the Thanjavur district), diagnostics; cultural and social determinants of nutrition, the evidence about effectiveness of educational and provider strategies, compliance behavior, appropriate and innovative evaluation criteria, and systems design for implementation.
Organizing Trandisciplinary Content Our TPS courses have used three approaches. The first, cells to
Supplement 1, 2015, Vol 105, No. S1 | American Journal of Public Health
society, organizes disciplinary perspectives by levels or pathways of influence on a public health problem. This approach was popularized in biomedical and health science research communities and embraced by the National Institutes of Health and other institutions.21,24 It considers how biological, behavioral, environmental, policy, economic, and other processes and factors—as well as the interplay among them—contribute to a public health problem. As an example, a TPS course on obesity examined how bacteria in the gut microbiome, biological mechanisms that regulate metabolism, lifestyle behaviors such as diet and activity, agricultural and economic policies, and food marketing and labeling regulations can influence obesity. Considering these perspectives together brings a richer understanding of obesity causes and challenges students to generate a more integrative model of obesity prevention. A variant of this approach focuses on spheres of influence that are not inherently hierarchical as are the levels of an ecological model but rather that operate within a single level, such as the social environment. For example, a TPS course on child maltreatment explores how youths affected by violence or neglect are likely to be involved in many health and social service systems including child welfare, law enforcement, courts, medical care, mental health care, and housing shelters. Understanding these touch points and the goals of each system prepares students to think more broadly about potential opportunities for a coordinated and effective response. Other courses have used a strategic blending of selected disciplinary perspectives. By contrast to TPS courses that examine
a problem from many different angles, this approach focuses on a few handpicked disciplines that may be ripe for integration, with promising but untapped synergy. It reflects a purposeful exploration of the intersection between disciplines to solve a public health problem. For instance, one TPS course sought to integrate health and social services systems to reduce the prevalence and impact of smoking in low-income populations. The course focused on practice disciplines in public health (tobacco control) and social services (food security) as well as theory and research from the field of organizational behavior, specifically within public agencies.25 Students in this course worked in teams to develop integration strategies and presented these as proposals to agency leaders.
Understanding vs Solutions Each TPS course covers the continuum from understanding to solving a public health problem, with corresponding goals of helping students think and act differently on the basis of a transdisciplinary perspective. But the relative emphasis on understanding versus solving varies from course to course. For example, a TPS course on pregnancy outcomes strives for a deeper understanding of the myriad exposures that contribute to adverse maternal and child outcomes. Others, such as a health disparities course in which students learn to communicate effectively about social determinants of health to policy makers and the public, emphasize generating promising solutions. These differences reflect the background, interests, and orientation of the instructor but also the maturity of the knowledge base on a particular public health
Lawlor et al. | Peer Reviewed | Commentary | S101
COMMENTARY
challenge. They are often reflected in the kinds of assignments or products students complete in a course. For example, TPS courses focused on understanding might culminate in the development of integrative causal models, whereas solution-focused courses could generate specific strategies for prevention.
Instructional Models Most instructors have delivered their TPS courses using one or more of four instructional approaches, each with its strengths and limitations. Team teaching pairs instructors from different disciplines central to the course content. Besides the obvious advantages of broader expertise, this model also tends to draw students from different disciplines across campus (usually those of the participating instructors), which can create valuable interactions among students with different training and perspectives. But we have found it can also create administrative challenges assigning or dividing teaching credit for instructors as well as allocating limited seats in the class across schools. Some TPS courses consider so many disciplines that no two instructors could reasonably be expected to have deep knowledge of all content to be covered. In these cases, some TPS courses have opted for an expert-of-the-week model. Like team teaching, its main advantage is the breadth of expertise students are exposed to. However, we have found this guest speaker--dependent model hard to sustain because leading experts may not be consistently available to contribute every semester or year, especially to a course outside their school or department. Recording their live contributions for future use seems a reasonable solution to this challenge. Several TPS courses use a design studio approach modeled
after architectural training. Design studio work focuses on time-limited projects (usually one semester) addressing “complex and openended problems” through “rapid iteration of design solutions.”26(p349) It is informed by frequent formal and informal critique from peers and instructors as well as by precedent—examining past solutions to problems deemed relevant to the current project. In some TPS courses, students spend up to half of each class session (60---90 min) working on transdisciplinary solutions to a public health problem. During these periods, instructors provide students with feedback and suggestions. Students typically spend much more time working on their solutions outside class than in class, but the design studio approach ensures that they receive regular feedback throughout the development process. In our experience, this type of learning is initially unfamiliar to many public health students and requires coaching (and coaxing) to get meaningful peer critiques early on. It is a better fit for solutionfocused TPS courses. Finally, some TPS courses have been taught in situ, in environments as diverse as rural India, Haiti, and Washington, DC, and have relied much more on experiential learning, faculty supervision, and field applications. These courses are so popular among students we have had to adopt a competitive application process for enrollment because demand exceeds course capacity. On the down side, they are extremely timeconsuming to plan, logistically complicated, and expensive to execute.
FACULTY, CULTURE, AND INSTITUTIONAL SUPPORT FOR TPS A critical element in creating the environment for transdisciplinary public health education,
S102 | Commentary | Peer Reviewed | Lawlor et al.
reinforcing its importance, and promoting a distinctive style of education is recruiting and retaining faculty who exemplify this approach. The school’s faculty recruitment process purposely selects talented faculty with the training and aspirations to engage in team science and transdisciplinary learning. A faculty committee investigated research environments—such as the Bell Labs—that leveraged transdisciplinary scientific approaches. An internal network analysis of faculty research and teaching investigated cross-disciplinary faculty research, teaching, and service activities. The Promotion and Tenure Committee took up the issue of how transdisciplinary research and education could be credited and recognized in promotion and tenure reviews. School administrators recognized that many TPS courses would require resources beyond what is needed in a typical course and created a specific budget to support them. In short, institutional support and engagement at all levels of the school have been critical to building this approach.
CONCLUSIONS Transdisciplinary public health is a concept that is compelling in principle but difficult in practice. As schools and programs of public health continue to innovate with new approaches to address complex public health problems, the experiences we have described show that it is feasible to provide transdisciplinary graduate education in public health, with significant innovation in teaching methods, partnerships, and even the recruitment and promotion of faculty. We do not presume that our approach is the best or only way to organize and deliver
transdisciplinary public health education, but we hope that sharing it will contribute to growing discussion and action on this important topic and help public health educators think about ways to integrate transdisciplinary learning activities. Although we have not yet conducted a formal evaluation of our TPS model, we have gathered evaluative data in several forms. We adapted our standard course evaluation to include special items for TPS courses (e.g., assessing how well the course fostered a transdisciplinary approach, included perspectives from diverse disciplines, or used a cells-to-society approach). We also have substantial anecdotal evidence of impact from the perspective of faculty members, students, and prospective students. Faculty members’ interest in developing and teaching TPS courses quickly grew to the point that we had to implement a multistage committee process to review, approve, and track TPS-designated courses. Student demand for the courses also grew, and many wanted to take TPS courses beyond the program requirements and use them to fulfill elective credits. As a result, we have had to control registration to ensure that students needing to meet their TPS requirements have enrollment priorities over those who have already completed the requirement. In addition, every year a significant portion of applicants to our MPH program indicates (without solicitation) that their interest in our program is closely tied to our transdisciplinary focus. Part of this almost certainly reflects prospective students customizing their applications to match our program’s identity, but it is equally clear that there is a sizable niche of
American Journal of Public Health | Supplement 1, 2015, Vol 105, No. S1
COMMENTARY
students with not only genuine interest in transdisciplinary thinking but also undergraduate or professional experiences learning and applying this approach. Finally, it is remarkable how quickly, thoroughly, and palpably the TPS perspective has permeated our school’s culture and identity. “Transdisciplinary” has become as commonplace as “population” or “prevention” in any discussion of public health and the Brown School. And it is not spoken just by those in the school but also by top university leaders, board members, and community partners. Although influential and important public health leaders have emphasized the importance of training transdisciplinary public health professionals, we are still in a transitional period in which careers, faculty incentives, and professional reinforcements have not caught up. It is important for the public health field writ large to embrace this approach to create a cadre of transdisciplinary scientists and practitioners who will promote systemic change. Students and faculty are motivated to apply this approach to public health education and research. j
About the Authors Edward F. Lawlor, Matthew W. Kreuter, Anne K. Sebert-Kuhlmann, and Timothy D. McBride are with the Brown School, Washington University, St. Louis, MO. Correspondence should be sent to Matthew W. Kreuter, PhD, MPH, Kahn Family Professor and Associate Dean for Public Health, CB-1009, 700 Rosedale Avenue, St. Louis, MO 63112 (e-mail: mkreuter@wustl. edu). Reprints can be ordered at http://www. ajph.org by clicking the “Reprints” link. This article was accepted November 7, 2014.
Contributors E. F. Lawlor and M. W. Kreuter outlined the article. All authors contributed to writing and editing the article.
References 1. National Research Council. Facilitating Interdisciplinary Research. Washington, DC: National Academies Press; 2004. 2. Gebbie K, Rosenstock L, Hernandez LM. Future role of schools of public health in educating public health professionals for the 21st century. In: Gebbie K, Rosenstock L, Hernandez LM, eds. Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century. Washington, DC: National Academies Press; 2003:108---128. 3. Luke D, Moreland-Russell S, Herbers S. Measuring success: an evaluation framework for transdisciplinary public health. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Methods, and Practice. San Francisco, CA: Jossey-Bass; 2013:89---90. 4. Stokols D. Toward a science of transdisciplinary action research. Am J Community Psychol. 2006;38(1---2):63---77. 5. Graybill JK, Dooling S, Shandas V, Withey J, Greve A, Simon GL. A rough guide to interdisciplinarity: graduate student perspectives. Bioscience. 2006;56 (9):757---763. 6. Mâsse LC, Moser RP, Stokols D, et al. Measuring collaboration and transdisciplinary integration in team science. Am J Prev Med. 2008;35(2 suppl):S151--S160. 7. Abrams DB. Applying transdisciplinary research strategies to understanding and eliminating health disparities. Health Educ Behav. 2006;33(4):515---531. 8. Nash JM, Collins BN, Loughlin SE, et al. Training the transdisciplinary scientist: a general framework applied to tobacco use behavior. Nicotine Tob Res. 2003;5(suppl 1):S41---S53. 9. Stokols D, Harvey R, Gress J, Fuqua J, Phillips K. In vivo studies of transdisciplinary scientific collaboration: lessons learned and implications for active living research. Am J Prev Med. 2005;28 (2 suppl 2):202---213. 10. Stokols D, Hall KL, Taylor BK, Moser RP. The science of team science: overview of the field and introduction to the supplement. Am J Prev Med. 2008;35 (2 suppl):S77---S89. 11. Rashid JR, Spengler RF, Wagner RM, et al. Eliminating health disparities through transdisciplinary research, crossagency collaboration, and public participation. Am J Public Health. 2009;99 (11):1955---1961. 12. Emmons KM, Viswanath K, Colditz GA. The role of transdisciplinary collaboration in translating and disseminating health research: lessons learned and exemplars of success. Am J Prev Med. 2008;35(2 suppl):S204---S210.
Supplement 1, 2015, Vol 105, No. S1 | American Journal of Public Health
13. Ruddy G, Rhee K. Transdisciplinary teams in primary care for the underserved: a literature review. J Health Care Poor Underserved. 2005;16(2):248---256. 14. Israel BA, Schulz AJ, Parker EA, Becker AB. Review of communitybased research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19: 173---202. 15. Stokols D, Fuqua J, Gress J, et al. Evaluating transdisciplinary science. Nicotine Tob Res. 2003;5(suppl 1):S21---S39. 16. Stokols D, Hall KL, Vogel AL. Transdisciplinary public health: definitions, core characteristics, and strategies for success. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: Jossey-Bass; 2013:3---30.
24. Patterson RE, Colditz GA, Hu FB, et al. The 2011---2016 Transdisciplinary Research on Energetics and Cancer (TREC) initiative: rationale and design. Cancer Causes Control. 2013;24(4): 695---704. 25. Kreuter MW, Pfeiffer D. Transdisciplinary problem solving for integrating public health and social service systems to address health disparities. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: Jossey-Bass; 2013:249---263. 26. Kuhn S. Learning from the architecture studio: implications for projectbased pedagogy. Int J Eng Educ. 2001; 17(4---5):349---352.
17. Calhoun JG, Ramiah K, Weist EM, Shortell SM. Development of a core competency model for the master of public health degree. Am J Public Health. 2008;98(9):1598---1607. 18. Public Health Foundation. Core competencies for public health professionals. Available at: http://www.phf.org/ programs/corecompetencies/Pages/ About_the_Core_Competencies_for_ Public_Health_Professionals.aspx. Accessed on October 24, 2014. 19. Arnold LD, Hipp JA, Kuhlmann AS, Budd E. Competencies in transdisciplinary public health education. In: HaireJoshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: JosseyBass; 2013:53---77. 20. Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The ecology of team science: understanding contextual influences on transdisciplinary collaboration. Am J Prev Med. 2008;35(2 suppl):S96--S115. 21. Morgan GD, Kobus K, Gerlach KK, et al. Facilitating transdisciplinary research: the experience of the transdisciplinary tobacco use research centers. Nicotine Tob Res. 2003;5(suppl 1):S11---S19. 22. Raghavan R, Chockalingam R, Johar Z. Implementing public health interventions in developing countries: a transdisciplinary solution for safe drinking water in rural India. In: Haire-Joshu D, McBride TD, eds. Transdisciplinary Public Health: Research, Education, and Practice. San Francisco, CA: Jossey-Bass; 2013:297--317. 23. Armstrong E, Shuckerow DS, Tafoya A, Raghavan R, Johar Z. Maternal and child health intervention: healthy families initiative. 2012. Available at: http:// www.ictph.org.in/tps-2011/paper5introduction.html. Accessed on January 8, 2015.
Lawlor et al. | Peer Reviewed | Commentary | S103
