Ann. N.Y. Acad. Sci. ISSN 0077-8923

A N N A L S O F T H E N E W Y O R K A C A D E M Y O F SC I E N C E S Issue: A Global Research Agenda for Nutrition Science

Strengthening implementation and utilization of nutrition interventions through research: a framework and research agenda Purnima Menon,1 Namukolo M. Covic,2 Paige B. Harrigan,3 Susan E. Horton,4 Nabeeha M. Kazi,5 Sascha Lamstein,6 Lynnette Neufeld,7 Erica Oakley,8 and David Pelletier9 1 International Food Policy Research Institute, New Delhi, India. 2 Centre of Excellence for Nutrition, North-West University, South Africa. 3 Nutrition, Health and Food Security, Save the Children, Washington, DC. 4 Balsillie School of International Affairs, University of Waterloo, Ontario, Canada. 5 Humanitas Global Development, Washington, DC. 6 John Snow, Inc, Boston, Massachusetts. 7 Micronutrient Initiative, Ottawa, Ontario, Canada. 8 Humanitas Global Development, Washington, DC. 9 Division of Nutritional Sciences, Cornell University, Ithaca, New York

Address for correspondence: Purnima Menon, Senior Research Fellow, International Food Policy Research Institute, NASC Complex, New Delhi 110012, India. [email protected]

Undernutrition among women and children contributes to almost half the global burden of child mortality in developing countries. The impact of nutrition on economic development has highlighted the need for evidence-based solutions and yielded substantial global momentum. However, it is now recognized that the impact of evidence-based interventions is limited by the lack of evidence on the best operational strategies for scaling up nutrition interventions. With the goal of encouraging greater engagement in implementation research in nutrition and generating evidence on implementation and utilization of nutrition interventions, this paper brings together a framework and a broad analysis of literature to frame and highlight the crucial importance of research on the delivery and utilization of nutrition interventions. The paper draws on the deliberations of a high-level working group, an e-consultation, a conference, and the published literature. It proposes a framework and areas of research that have been quite neglected, and yet are critical to better understanding through careful research to enable better translation of global and national political momentum for nutrition into public health impact. Keywords: nutrition; delivery; utilization; framework; implementation research; agenda

Introduction Undernutrition, overweight, and obesity together account for an astounding proportion of the global burden of disease.1 Undernutrition among women and children contributes to almost one-half of the global burden of child mortality in developing countries, in particular. In recent years, the recognition of this problem has led to much global and national advocacy to raise the profile of undernutrition as a development issue and to highlight the need for evidence-based solutions. The availability of solutions to address undernutrition has emerged from evidence reviews such as the series of articles published in the Lancet,2,3 a recent set of articles on maternal undernutrition,4 and evidence databases

such as the World Health Organization (WHO) electronic Library of Evidence-informed Nutrition Actions (e-LENA) (http://www.who.int/elena/en/). These reviews and global recommendations that have drawn on them, such as the Scaling Up Nutrition: A Framework for Action (SUN),5 have proposed the types of interventions that need to be delivered at scale to achieve rapid reductions in different manifestations of undernutrition: stunting, wasting, childhood anemia, maternal anemia, and pregnancy outcomes, among others. The SUN movement, in particular, has also emphasized the role of direct or nutrition-specific interventions, and of enhancing the potential for indirect or nutritionsensitive interventions, to together address the multifaceted etiology of undernutrition in most

doi: 10.1111/nyas.12447 C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

39

Nutrition implementation framework and research agenda

developing countries. Although several questions remain about how best to address some of the emerging and underlying drivers of undernutrition, such as poor sanitation, food insecurity, or women’s empowerment, overall, there is a base of knowledge to be able to begin to scale-up several direct interventions to ensure better diet quality for women and children. The 2013 Lancet series on Maternal and Child Undernutrition recommends that 10 types of interventions be scaled-up to improve nutrition outcomes globally and goes further than the 2008 Lancet series by highlighting that delivery strategies, especially community-based platforms, show promise and potential for scaling up coverage of evidencebased interventions.2 The Copenhagen Consensus 2012 also showed that bundled nutrition interventions to reduce undernutrition in preschoolers continue to be high-impact investments (see http://www.copenhagenconsensus.com/projects/co penhagen-consensus-2012). It is, however, well acknowledged that low intervention coverage and quality of implementation can continue to contribute to poor outcomes for children in developing countries unless delivery itself is improved.6,7 Many of the direct nutrition interventions that are currently recommended aim to address the quality of diets, especially the diets of infants and young children. These interventions fall in the realm of behavioral interventions to address infant and young child feeding (IYCF) practices, such as promotion and counseling for exclusive breastfeeding and behavior change communication (BCC) interventions to improve complementary feeding and hygiene behaviors. Other interventions focus on enhancing the availability of, and the access to, micronutrient supplements, fortified complementary foods, and complementary food supplements. In spite of the global consensus that scaling up these interventions is critical to ensure that women and children have access to critical nutritional inputs, there is much more known about the efficacy and effectiveness of several of these interventions,8 than about how best to deliver these interventions. Although there is recognition that these interventions might be delivered through multiple platforms, such as community-based nutrition programs, health system platforms that integrate nutrition interventions, and/or market-based interventions, there is evidence to suggest that, in fact, 40

Menon et al.

little is known about how effective some of these delivery platforms are at achieving either process or impact outcomes.9 Strengthening implementation and translating political and financial commitments to affect the ground require several elements to be in place,10 but a critical component that can accelerate effective solutions is research on how to ensure high-quality at-scale implementation of these interventions.11 As the types of delivery platforms used for delivering nutrition interventions increase and change, it is amply clear that the nutrition community can anticipate the need for more research on the comparative advantages of the multitude of delivery strategies for implementing direct interventions. In recent years, two separate calls to action were made to boost the availability of evidence to scale-up direct interventions.11,12 In addition, there have been other attempts to highlight the types of issues inherent in delivering for impact.13,14 Overall, however, there has been little of the type of research needed to inform the delivery of key interventions at scale and to support the achievement of population-level impacts. The funding bias in research is in itself a limitation; one estimate suggests that implementation-focused research gets only 3% of funding compared to studies on basic mechanisms or developing and improving health technologies.15 Issues pertaining to the design and delivery of interventions have also not received adequate attention by researchers, thus compounding the effects of low funding and publication bias. The primary objective of this paper, therefore, is to describe a nutrition implementation-focused framework and research agenda, developed through a multistage, multistakeholder process, for studying issues of intervention implementation and utilization in nutrition. The paper also attempts to showcase examples of implementation research in nutrition. By doing this, the paper aims to inspire researchers from nutrition and beyond to engage with this critical topic and to encourage research funders to demand and support more high-quality research on delivering critical nutrition interventions for impact. Methods In response to the growing concern about the need for research to bring more knowledge about delivering impact, the New York Academy of Sciences

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

convened a working group to deliberate on this topic. This was followed by a survey of various respondents who identified key topics around which more research is perceived to be needed. A rapid literature review and several working group deliberations led to the development of this paper, which aims to set a research agenda, in broad terms, to address knowledge gaps that are potentially hampering the scaling up of impact on nutrition. The paper is not based on a typical systematic review of the current literature related to implementation research in nutrition. An original set of literature was identified via a “snowball” literaturegathering effort that started with literature contributed by the initial high-level working group, with additional literature search and gathering by the coordinating organization (Humanitas Global Development) for the working group meetings. For the paper writing and revision, we conducted further limited searches of the literature to identify papers related to implementation research frameworks and methods in nutrition, using terms such as “nutrition delivery science” and “nutrition implementation research.” The literature on examples within framework domains is limited to those identified in the literature-compilation process and further short literature searches conducted by individual section writers. Very recent papers in nutrition16 have also highlighted the gaps in the literature and called for development of an overarching framework and common terminology for implementation research in nutrition, thus providing additional confidence that the literature is indeed lean in this area. Furthermore, the literature search for keywords such as “nutrition implementation research,” “nutrition delivery science,” and “nutrition implementation science” on PubMed and Google Scholar yielded no papers in nutrition that, in fact, offer such a framework. Thus, this paper aims to fill what is identified as a major gap in the literature; it also aims to explicitly link key concerns in delivering for nutrition impact with similar thinking in other fields, notably those of implementation research in the areas of health systems17 and mental health.18 This paper uses the example of IYCF BCC and the supply of micronutrients and food to highlight key research needs in the area of implementation research in undernutrition, and to demonstrate the urgent need for a global nutrition implementation research agenda to scale-up impact on undernutrition.

Nutrition implementation framework and research agenda

Table 1. Research topics initially identified by the delivery science working group

(1) Research to identify innovative methods for measuring and validating behavior and behavior changea (2) Research to identify required staff competencies and their impact on program effectiveness and sustainability (3) Research on adequate indicators for nutrition delivery capacity, performance, and actual costs (4) Research to understand demand creation for interventions at civil society levels (5) Research on implementation and impact pathways to improve agriculture–nutrition programsb (6) Research on why delivery systems do not reach sufficient numbers and lack the desired impacta (7) Research on the safe administration of micronutrients (8) Research on optimal IYCF delivery systems and processesb a b

Next two highly ranked topics. Most highly ranked topics in the e-consultation process.

E-consultation A web-based consultation was implemented from June to August 2012, with approximately 54 respondents. The goals of the consultation were to secure feedback on major areas of nutrition science research that require more investigation and evaluation, to prioritize key nutrition science research topics, and to understand specific areas of interest by sector. For this process, input was received from a multitude of stakeholders, including academia, intergovernmental organizations, nongovernmental organizations, the private sector, and the public sector. The respondents prioritized and provided feedback on the eight priority research areas initially identified by the delivery science working group (Table 1). The results of the e-consultation confirmed that the proposed research areas did indeed reflect current research gaps. The analysis of the results of the e-consultation also brought a few of the research areas to the top of the initial list: (1) research on optimal IYCF delivery systems and processes and (2) research on implementation and impact pathways to improve agriculture–nutrition programs. The other topics, namely research to identify innovative methods for measuring and validating behavior and behavior change and to understand

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

41

Nutrition implementation framework and research agenda

why delivery systems do not reach sufficient numbers and lack the desired impact, were also ranked highly. The delivery science working group reviewed the results of the consultation process carefully, and concluded that, in effect, the broader arena of implementation research in nutrition is too nascent to focus only on the few priority areas highlighted by the e-consultation results. After substantial deliberation, it was concluded that there was a need for a unifying approach to thinking about issues related to implementation of nutrition interventions, and that a framework for implementation research in nutrition could help guide such processes. Over several working group meetings, the group then developed and refined the Nutrition Implementation Framework (described later). After development of the framework, group members were assigned the responsibility to describe the core domains of the framework based on a brief literature review and to highlight the importance of the domain in relation to achieving coverage and impact of nutrition interventions. Each section leader also provided examples of the types of research questions related to that domain, and finally, where possible, provided examples of studies in nutrition that had begun to provide insights into the issues within that domain. The next sections of the paper describe the framework itself and describe the domains in detail. The paper concludes with discussion of some of the essential considerations and challenges in performing implementation research in the context of nutrition interventions and makes recommendations for researchers, research funders, and program implementers. Framework and scope The need to strengthen implementation and to undertake research to improve implementation is not limited to nutrition. Indeed, the challenges and importance of implementation have been recognized for decades in the policy literature,19–21 and a substantial body of empirical research is available in relation to health care, global and public health, mental health, education, social welfare, and other fields, from industrialized countries as well as from low- and middle-income countries. In addition to empirical research, this literature includes several theories and frameworks for conceptualizing the implementation process.18,20–25 42

Menon et al.

This proliferation of theories and frameworks has led, in recent years, to the creation of a number of integrative or meta-frameworks that outline the range of factors and dynamics that shape implementation processes and outcomes.26–28 Such meta-frameworks provide stable frames of reference for designing, conducting, interpreting, and synthesizing implementation research and for guiding implementation practice. They are a sign of a maturing field of inquiry, are essential for ensuring coherence in the accumulating body of empirical work, and can help avoid the perception that such work is purely descriptive and relevant only in particular contexts. Such frameworks do not currently exist in the realm of nutrition science. The working group on delivery science undertook an iterative process to develop a middle-range framework that strikes a balance among comprehensiveness, comprehensibility, and utility. This iterative process included consideration of some meta-frameworks in the implementation science literature, some tools or frameworks commonly used in development practice (e.g., log frames and program impact pathways (PIPs)), and attention to the issues that the consultation considered most salient to BCC, micronutrients, and food supplements (see Table 1 consultation topics). The resulting framework is shown in Figure 1. The Nutrition Implementation Framework includes several distinct components, each with several elements: (1) the logic-of-results frameworks as found in log-frames (inputs, processes, outcomes, and impacts); (2) PIPs that detail the underlying program theory relating inputs to the desired results; (3) the critically important implementation core elements (domains 2–4) that are central to implementation meta-frameworks and typically are not included in results frameworks; (4) explicit recognition of the importance of contextual factors at national, organizational, community, and household levels; (5) recognition of the importance of the enabling policy environment and governance; and (6) the strategic and management capacities for commitment building, adaptive management, and sustainability. It is important to note that each of these components is expandable (meaning each has a large number of elements embodied within it, see Box 1 and Fig. 2) and the overall framework is adaptable (to a wide variety of nutrition interventions and settings) while still retaining conceptual coherence

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

Nutrition implementation framework and research agenda

Figure 1. The Nutrition Implementation Framework.

and generalizability. As with other such conceptual frameworks, it is intended to facilitate an assessment and identification of the factors that may most need attention in a given situation, without implying a need to address all factors in all situations.18 As noted, one objective of the working group was to develop a robust framework of this type, and a second objective was to illustrate its applicability to nutrition programs. For the latter purpose, the group chose BCC, micronutrient interventions, and food supplements in the context of IYCF programs, because these are important and common components of large-scale nutrition programs. These three categories of interventions together also cover the spectrum of interventions currently recommended in global guidance for addressing undernutrition through direct nutrition-specific interventions. The following sections illustrate the expandability and applicability of each of the framework components and domains in the case of IYCF programs and some of the empirical evidence, and evidence needs, for each. We do this by describing some of the critical insights into the different domains of the Nutrition Implementation Framework, drawing on literature in nutrition and health systems. The literature is used to illustrate the framework domains and to provide examples of studies that

have attempted to study these domains. Each section below describes the major domains/areas of the framework, highlights why research on that domain is important for nutrition, provides examples of studies where possible, and tries to conclude with critical research questions for nutrition that emerge from that domain. The working group also discussed the significant role that costing studies can play in generating evidence on implementation choices and modalities; however, since costing studies must cross-cut several implementation domains, their role within nutrition implementation research is described separately, in Box 2. Before proceeding to describe the framework domains, it is worth noting that the deliberation processes and literature-review process that led to this paper are focused on issues of undernutrition rather than issues of overweight, obesity, and noncommunicable diseases. The framework described earlier, however, is more broadly applicable to interventions that aim to address those issues, and examples from intervention areas targeting those outcomes or interventions could also be examined. There are, indeed, important parallels between the framework and the discourse in this paper and issues discussed in the area of obesity prevention in the United States.29

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

43

Nutrition implementation framework and research agenda

Menon et al.

Box 1. Expandability of the Nutrition Implementation Framework components The Nutrition Implementation Framework is a pragmatic and high-level summary of the implementation factors and processes that affect the quality, coverage, equity, utilization, demand, outcomes, impacts, and sustainability of nutrition programs. As with all high-level summaries (such as the UNICEF Conceptual Framework) there is a great deal of detail and complexity subsumed within each component, and it often is necessary to consider the components in greater detail. Some examples are as follows: 1. Results frameworks: These are one of the most common tools used in development to guide the design, monitoring, and evaluation of policies and programs, and their applicability in nutrition programs has been described by Marsh.30 They help rationalize and make explicit the relations among inputs and the anticipated outputs, outcomes, and impacts, but they do not by themselves contain or reveal the underlying biological, behavioral, or social theory by which inputs are expected to produce the desired results. 2. Program Implementation Plans (PIPs) or program theory: These tools complement the results frameworks by drawing upon bio/behavioral/social theory to make explicit the mechanisms by which the inputs produce the desired results.31,32 Creating a PIP ex ante usually reveals faulty logic and assumptions and thereby improves program planning, and they also identify more nuanced and sensitive Monitoring and Evaluation (M&E) indicators than results frameworks. 3. Implementation core (domains 2–4): These represent the primary activities and processes by which program inputs are to be translated into desired results. They include such things as staff selection, training, deployment, coaching, supervision, and evaluation,22 in addition to supply chains and logistics related to commodities. These often are the primary levers used for operationalizing the PIP or program theory. 4. Contextual factors: The success with which the levers in the implementation core produce the desired results is shaped by many contextual factors at the organizational level. These include characteristics of the organization (e.g., size, structure, history, culture, leadership, readiness, openness, turnover, physical facilities), and characteristics of the staff (e.g., knowledge, beliefs, competence, self-efficacy, motivation, values, identification with the program).28 Often, these factors are not addressed when planning and implementing a program or adding interventions to an existing program, thereby contributing to unsatisfactory results. Similarly, relevant contextual factors at community and household levels include such things as education, beliefs, distance, resource constraints, and gender dynamics. These tend to be better understood in nutrition programs but often not well addressed, further contributing to unsatisfactory results. 5. Enabling environment and governance: These represent contextual factors even more remote from the core implementation levers, but no less important. They include the level, predictability, and sustainability of financing; the competing demands of multiple donors on implementers; coordinating mechanisms; and investments in capacity, hiring, compensation, and other human resource policies and procurement regulations, among10,21,33 others. 6. Strategic and management capacities: Given the persistent and dynamic interplay of contextual factors with the implementation core, and the fact that the core implementation inputs themselves may be poorly implemented, strategic and (adaptive) management capacities are ultimately the most important determinants of program effectiveness and sustainability.10,21,34

Domain 1: Planning and inputs to implement and strengthen nutrition programs Implementation design and plans, choice of micronutrient and food interventions, workforce training across the implementation process, and adequate production and procurement mecha-

44

nisms are important to consider. For these, an enabling policy and governance environment are also essential ingredients. Although research related to intervention choices and the financing and resourcing of nutrition programs falls more deeply in the realm of nutrition policy processes than

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

Nutrition implementation framework and research agenda

Figure 2. An illustration of the expandability of the implementation core of the Nutrition Implementation Framework. Adapted from Ref. 28.

more typical implementation research, elements in this first domain of the Nutrition Implementation Framework set the tone and pace for all that occurs as part of the implementation process itself. Thus, strengthening research on intervention choices, governance, and policy processes as an input to implementation is much needed. Insights from the literature on program planning and intervention choices. Nutrition planners are faced with difficult choices in the contexts they work in, often with fewer resources than needed. Understanding these choice-making processes better, with a view to equipping nutrition planners with the strategic skills and capacities needed to enable more relevant choices, is important. The literature on this area in nutrition, however, is limited,21,35 even though it is now clearly recognized that most global guidelines and recommendations must often endure difficult processes of translation into policy and program choices at national and subnational levels to translate into impact.36 What type of research is needed? Research on delivering nutrition interventions should focus on shedding light on the processes through which intervention choices are made in different contexts, and the roles of information, evidence, political po-

sitions, and funding resources. For example, it is accepted that appropriate intervention choices are best made by balancing elements of biological and epidemiological needs, such as the role of background diets and how they influence response to interventions,7 along with information on existing delivery platforms and financing mechanisms in specific contexts. However, there are few studies on how choices are made in nutrition.35 Political will, a supportive national/regional intersectoral and political environment, and strategic leadership capacity are recognized as being important to generate effective decisions, but research on such decision making is limited. Several questions of interest emerged from the consultation process.

1. What strategic capacities are critical bottlenecks in nutrition policy and program-design processes? 2. What are the core elements of formative research that are needed to appropriately design behavior change/micronutrient programs that will have impact? What contextual factors are critically important to consider? 3. What models for developing BCC interventions are most generalizable and scalable?

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

45

Nutrition implementation framework and research agenda

Menon et al.

Box 2. Costing studies as a critical component of implementation research in nutrition The cost of interventions is another important dimension for analysis of program delivery. Ultimately, the cost effectiveness of interventions is a key tool for policy and prioritization. One supporting piece of evidence in terms of directing increased attention to nutrition as a component of the international child health agenda has been the favorable cost-effectiveness (and cost–benefit) ratios. These have highlighted nutrition’s importance in all three Copenhagen Consensus exercises (2004, 2008, and 2012). It is now fairly standard that new nutrition interventions provide estimates of cost-effectiveness, in order to ascertain whether they should be included in national (and international) strategies. Recent studies have been conducted for biofortification, weekly iron and folate supplementation, and micronutrient sprinkles, for example. Horton et al.85 rank various nutrition interventions and suggest that micronutrient interventions (zinc and vitamin A) are in the range of $5–20 per disability-adjusted life year (DALY) saved, treatment of severely acutely malnourished children (SAM) is in the range of $40/DALY saved, and behavior change is $50–150/DALY saved, whereas interventions involving complementary food are in the range of $500–$1000 per DALY saved. Comparative cost analyses can also be very informative for program delivery. In the same way that effectiveness studies are often combined in meta-analyses, comparative cost analyses provide valuable information as to how costs vary across contexts, the factors affecting costs, and (potentially) highlighting efficiencies and problems in delivery. Just as for effectiveness studies, using similar assumptions and methodologies is crucial to the value of these comparisons. One very useful study86 undertook a comparison of costs of 43 supplementation and 55 fortification studies in several world regions. They noted that there are large variations in estimated costs of the same intervention. For vitamin A supplementation, the range is a factor of 1000 (in part related to another finding, that the cost of the supplement is a minor fraction of the delivery cost, and hence cost estimates which ignore personnel costs wildly underestimate the program cost). For fortification, the range is a factor of 2 for iodine, 6 for iron, and 15 for vitamin A. They note that harmonizing cost methodology is key to advancing studies in this area. Another example of a comparative study—this time of the same intervention in different regions of one country87 —compared the cost of delivery of school-based treatment of helminth infection at the district level in Uganda. Cost per child was shown to vary by a factor of two across districts, and cost per case of anemia prevented by a factor of four. This type of analysis can identify factors which affect variation in delivery cost (scale economies), as well potentially as inefficiencies. (This is somewhat similar to techniques used in health sector management, such as hospital scorecards). Another approach that may complement the meta-analyses of costing studies is to generate costing templates to use for comparative work. The WHO CHOICE (CHoosing Interventions which are Cost-Effective) template (http://www.who.int/choice/en) was a previous example, and the newer interagency OneHealth tool (http://www.futuresinstitute.org/onehealth.aspx) is a newer incarnation. These tools are obviously only as good as the assumptions underlying them, and in applying these tools it is important to link them to actual data to test their veracity. For example, the WHO CHOICE tool tended to provide unrealistically low estimates of the cost of fortification programs, and some applications underestimated the indirect costs of interventions. It is easier to calculate the costs of personnel and consumables required for health and nutrition programs, and much harder to include management costs, space, and gasoline for transport—all of which may be shared with other programs and are hard to apportion. Nevertheless, these are useful tools with which to structure thinking and planning for interventions, and to analyze the efficiency of existing ones. Overall, the costing of nutrition interventions is, and will continue to be, a critical area of research on enhancing program effectiveness and supporting nutrition advocacy.88

Core implementation domains 2 and 3: Upstream and midstream processes (management, training, supervision, and workforce motivation) There is a long-standing and wide recognition of the importance of upstream processes such as manage46

ment capacities and leadership, training approaches, supervision, and motivations in determining the success or failure of nutrition programs.37–43 The success of interventions is “critically dependent on supportive supervision and worker motivation, with service quality, efficiency, and equity, all directly

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

mediated by workers’ willingness to apply themselves to their tasks.”44 To improve the quality of nutrition services, programs need to provide a motivating environment and encourage retention of health and nutrition workers through better training, supportive supervision, referrals and connections to formal health systems, and incentives,45,46 but little information is available from the nutrition literature on how best to set up and achieve these, especially in a cost-effective manner. Insights from the literature on program management. Aspects of program management and context identified as success factors in large-scale programs were community involvement; inclusion of social groups; collaboration with other existing complementary programs; remuneration, incentives, capacity-building, and professional development for program staff; dynamic leaders; cost consciousness, transparency, and accountability of fund allocation; adaptability to new and changing situations; and action-oriented monitoring.40 A common theme that emerges from the literature on health service delivery is the importance of leadership skills (charismatic, committed, visionary).46,47 A review of community-based strategies specifically for breastfeeding promotion found that “change requires vision and managerial and leadership skills.”39 In a study of 22 rural clinics in South Africa, the factors that were identified as contributors to dramatic differences in the quality of care of severely malnourished children included leadership, teamwork, induction of incoming staff, and managerial supervision and support; clinical guidelines and training alone were not sufficient to ensure quality of care.41 A review of breastfeeding promotion programs concluded that training of large numbers of health workers and community members was key to success, especially training that emphasized practice and used a variety of training techniques.48 Others have noted that nutrition training alone does not, unfortunately, adequately prepare frontline workers for counseling and negotiation for behavior change.49 A key gap in the area of research on training itself, though, is that training depth, content, and organization are often not carefully considered, documented, reported, or evaluated.37 Although findings are often not consistent, what is emphasized is the need for rigorous country-adapted training of

Nutrition implementation framework and research agenda

various cadres at all levels and multiple sectors, and follow-up through mentoring, follow-up training, and/or supervision.49–51 Both poor program management and supervision are frequently blamed for program failures.37,43 Indeed, investing in the tools for and capacities of managers and supervisors has great potential, but is recognized as one of the weakest links in programming for nutrition.52,53 Evidence of the cost and impact of alternative approaches to supervision is needed to guide implementation at scale. Although the importance of supervision is widely acknowledged, it has not been very well studied in nutrition; only a few examples exist.54–56 Despite the importance of worker motivation to improving quality of care and reducing attrition, little is understood about what motivates nutrition workers in different contexts. With regard to health workers, Willis-Shattuck et al. found that career development, recognition for services rendered, availability of supplies and appropriate infrastructure, and management issues can affect motivation. They also found that even though financial rewards are often motivational, they appear to be insufficient as incentives for health workers.53 Indeed, a study in the United States55,56 demonstrates the nuances of the domains of frontline worker motivation. It highlights that perceived value of work, supportive supervision, and actions to enhance worker autonomy play critical roles. Factors influencing motivation can certainly vary from country to country, from one type of nutrition service provider to another or even from one type of intervention to another, and much more data are needed on what works in different contexts or across contexts. Gaps in research on program management processes. Despite the critical role of management, training, and motivations, limited evidence seems to exist on optimal management capacities, training processes, and approaches to motivating a nutrition workforce. More evidence-based guidance is needed to effectively address these aspects of nutrition programs and services. Questions that should be answered through careful cross-country research include the following: 1. Management (i) What management capacities and training processes are required for the production,

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

47

Nutrition implementation framework and research agenda

distribution, delivery, and consumption of fortified foods and supplements? (ii) What metrics of management capacities, training processes, and motivational approaches could be systematically integrated into existing monitoring and evaluation systems of nutrition programs? (iii) What are the priority management processes (human resource management, planning, budgeting, logistics, etc.) and what are their relationships to intervention success or failure, particularly when implemented at scale? 2. Training (i) How do alternative approaches to training a range of nutrition service providers from multiple contexts compare in terms of their effect on knowledge and practices of providers and the ultimate target population (caregivers, children, etc.)? (ii) How do training needs differ for programs focused on behavioral change versus those focused on delivering micronutrients/fortified foods? 3. Supervision (i) What are the key elements of supervision and what are their relationships to intervention success or failure, particularly when implemented at scale? (ii) What metrics of management and supervision processes could be systematically integrated into existing monitoring and evaluation systems of nutrition programs? (iii) How do alternative approaches to supervision compare in terms of their effect on knowledge and practices of providers and the ultimate target population (caregivers, pregnant women, etc.) in a range of cultural contexts? (iv) What are the skills or competencies required for effective supervision of nutrition services and programs? 4. Motivation (i) What factors affect motivation of nutrition service providers from various sectors, levels, and cultural contexts? (ii) What is the effect of motivation of the nutrition workforce on nutritional outcomes?

48

Menon et al.

Core implementation domain 4: Downstream processes of frontline capacities and supply chain issues Downstream processes refers to the inputs and activities occurring at the frontline clinic and/or community level related to implementing IYCN interventions and the factors that might affect the quality and degree of implementation at this level. Depending on the package of services, this may include counseling for behavior change, taking anthropometric measurements, classification of an individual’s nutrition status based on those measurements, and distributing supplements or foods. For programs that distribute micronutrient supplements or complementary food products, forecasting, inventory control and management, and local level distribution are also significant frontline tasks. A fundamental underlying factor that will influence the completeness and fidelity of implementation of all of these activities is frontline worker capacities and workload and ability to coordinate and manage the many activities and inputs that might be required to do so. The success of IYCN programs is dependent on the extent to which behaviors related to child feeding and the quality of complementary foods are improved; in turn, the success of a program to change those behaviors is dependent on the quality of the behavior-change activities as executed at the frontline. Certainly, careful design and implementation of such strategies, based on understanding barriers to behavior change, have been shown to be successful at small scales.57 Central-level supply chain issues are central to ensure stocks and quality of supplies in programs, but frontline workers usually play an important role as well, for example, in management and quality control of stock, ensuring appropriate storage conditions, and stock rotation. In some contexts, they may also be responsible for forecasting and ordering new supplies. Insights from the literature on frontline implementation. The performance of even the most highly motivated frontline workers program may still be affected by their individual capacity, their real and perceived workload, and the extent to which any new intervention adds to that. Ensuring adequate access to, coverage, and utilization of nutrition interventions by the intended beneficiaries have long been recognized as key determinants of success,

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

but lack of effective implementation by frontline workers is equally important. Evidence of incomplete implementation and poor fidelity by staff has been documented as an important reason for lack of compliance with nutrition interventions in different populations and areas of nutrition, for example, in the institutionalized elderly in Denmark.58 As part of an evaluation of the integrated management of childhood illness (IMCI) program in South Africa, poor implementation of nutrition assessment was identified as a key barrier to appropriate classification of children, further complicated by lack of or limited skills of the workers for some aspects of their responsibilities.59 Few studies in infant and young child nutrition have examined these issues systematically, and even fewer have done so at the scale that might be necessary when national programs are implemented across diverse regions. In Scotland, one study60 used a hierarchy of health service attributes model to explore factors that led to successful improvements in communities. They found that in communities where breastfeeding practices did not improve or worsened, physical and resource constraints and low motivation dominated, whereas in communities where breastfeeding increased, these issues were less of a concern and there was evidence of leadership and multidisciplinary collaboration. A study in Peru assessed the dose (to what extent did the intervention take place) and fidelity (quality or adherence to protocol) of an intervention and the extent to which this influenced a series of outcomes, as part of a process evaluation of an IYCN program.61,62 Although it was an excellent example of the use of impact pathway–based process evaluation to inform program improvement, the study did not directly address downstream processes. A systematic approach to addressing downstream issues must also take into consideration how processes may affect implementation not only in the immediate but in the longer term. For example, in Bangladesh, one study found that the addition of activities to address community-based management of SAM did not affect quality of care or other community health worker (CHW) activities, but frontline workers expressed concerns that the additional responsibilities affected their domestic life.63 Loechl et al.64 also noted that the addition of implementation of micronutrient powders in a child health program in Haiti led to moderate

Nutrition implementation framework and research agenda

increase in their workload, but was compensated for by the renewed motivation from seeing children recover (in Bangladesh) or mothers’ appreciation of the product (in Haiti). Overall, ensuring adequate attention to documenting the added workload of new interventions in both the short term and long term would help improve the evidence base for program design and implementation effectiveness. The importance of supply chains, including management at frontline facility and community levels, for the success of programs that distribute nutritional products (e.g., supplements, fortified foods) are essential to the success of commodity-based programs.13 We found no published literature, however, that addressed issues of capacity, consistency, or potential impact on program implementation of frontline workers’ capacity to forecast and manage stock of nutritional products. One market-based intervention in Kenya found that purchases of micronutrient powders by community vendors as part of a social-marketing scheme were higher among those who lived closer to the wholesale office,65 highlighting the potential importance of supply chain and convenience at the frontline level. Research areas that need further attention. Implementation research must help address questions related to challenges to program implementation in frontline facilities and communities by including a focus on issues such as (1) long-term management of workload, taking into consideration the multiple tasks assigned to frontline and community workers across multiple sectors (where applicable) with the goal of streamlining time-consuming processes that may not be critical (e.g., excessive record keeping or reporting requirements); (2) effective ways to assess skill attainment and maintenance and mechanisms to manage factors that might influence these (e.g., worker literacy/education) and incorporate skills assessment into routine monitoring systems; and (3) exploring methods to limit the extent to which variation in frontline worker capacity would influence program delivery, for example, by minimizing dependence of supply chain management on frontline workers. Ultimately, the success of an intervention will depend on the extent to which it is actually implemented and the fidelity of its implementation, both critically dependent on the capacity of frontline workers and the quality of frontline facilities.

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

49

Nutrition implementation framework and research agenda

Core implementation domain 5: Utilization and outcomes This section covers the outcomes of nutrition interventions, such as the quality of implementation, service delivery, and coverage of nutrition interventions. For IYCF BCC, these nutrition interventions are largely behavioral interventions, which are delivered in many ways, including community-based and formal health service systems and providers (including government and nongovernment), or population-oriented strategies such as social mobilization and communications through mass media campaigns. Micronutrient interventions to improve the quality of foods fed to young children may include delivery of micronutrients through public systems and/or market-based approaches for nutrition products such as micronutrient powders and fortified foods. For each of these, diversity in service/delivery-related issues can influence the quality and impact of interventions. Insights from the nutrition literature on issues related to service delivery, quality, and coverage of interventions. Five reviews, all relevant to service delivery in nutrition programs,9,13,66,67 together provide insights into critical issues to be examined in relation to implementation research, as well as gaps in this literature. The 2013 Lancet series on Maternal and Child Undernutrition highlighted the crucial nature of delivery strategies and platforms for nutrition-specific interventions to achieve greater coverage of recommended nutrition interventions. The series organized the strategies into distinct channels: fortification of staple foods and specific foods; cash transfer programs; communitybased platforms for nutrition education and promotion; IMCIs; school-based delivery platforms; child health days; and delivery of nutrition interventions in humanitarian emergency settings for scaling up and coverage. Community-based platforms, which work with community health workers (CHWs) to improve access, engage stakeholders, and promote behavior change (both preventive and curative) in service delivery, are well established and have great potential to improve the adoption of desired practices in hard-to-reach populations, but the evidence is of weak quality. An older review of intervention effectiveness, by delivery strategy, for specific nutrition and health interventions, done for the WHO9 found that the evidence var50

Menon et al.

ied depending on the type of delivery strategy. The main delivery categories were community, facility, population (e.g., mass media, social mobilization), and other (e.g., market based, pharmacy, fortification at central level). For process outcomes such as coverage, more evidence was available about the efficacy/effectiveness of delivery through community and facility platforms. Among community platforms, more evidence was available about the effectiveness of home visits than for the effectiveness of community groups, followed by community assemblies and mobile clinics. Another recent paper assessed the potential of four broad delivery platforms—health, agriculture, market-based, and social protection programs—in the delivery of micronutrient powders (MNPs).13 Within each platform, areas of performance examined included targeting, efficacy, implementation quality, utilization, impact, coverage, and sustainability. Although room for improvement was noted, all four platforms were found to have the potential to effectively deliver micronutrient interventions if conditions like strong behavior-change communication, adequate supply-side inputs, and rigorous evaluations of effectiveness, quality, and impact pathways were met. Findings in a recent systematic review for scaleup of breastfeeding programs67 summarized 22 enabling factors and 15 barriers, indicating the need for multiple program “gears” to work in coordination to achieve scale. These included evidence-based advocacy, resources, workforce development, program delivery, and promotion and research/evaluation. A recent Lancet series on equity in the coverage of child survival health and nutrition strategies66 identified intervention delivery channels—clinical or curative, outreach, community-based preventive or promotional, and legislative or mass media—and examined current evidence of strategies to overcome bottlenecks toward effective coverage of interventions. The authors identified a number of knowledge gaps specific to equity, some based on the difficulty in accurate measurement of use and on the complexity and expense of field trials for implementationsystem interventions. Insights from specific studies in nutrition indicate that nutrition evaluations and studies have documented the impact of nutrition programs on improved IYCF BCC and/or micronutrient practices or status in the context of IYCF (e.g., MNPs).68

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

Few also explicitly addressed the intervening implementation factors that influence why certain outcomes may have been achieved.61,64 A common feature of such studies is the reliance on program theory to organize concepts and study elements. Some quality outcomes studied in the context of nutrition-impact evaluations have included measures of frontline performance, client knowledge, and/or general awareness/attitudes of specific interventions. The study of the market-based micronutrient powder–distribution model in Kenya, for example, highlighted the successful use of monitoring data and timely action to identify and address implementation challenges as they occurred across the program.65 The study in Peru61 found that, although implementation of the program at health centers in Peru was “less than optimal” overall, aspects of the implementation were nevertheless significant variables in the impact pathway to contribute to success. Overall, few studies tend to look systematically at delivery-related factors, although they are recognized as being important to the adoption of practices and impact in nutrition programs.30 Research needs related to implementation, service delivery, and coverage. Our review reinforces that the current literature on IYCF BCC and MNP interventions does not yet include sufficient information on issues related to service delivery, quality, and coverage. Studies are available, but most are focused on impact for the client/beneficiary and do not routinely examine the implementation components of the programs. In addition, most studies on these topics are on smaller scale interventions, and are thus limited in capturing experiences at scale. Furthermore, the limited literature is stronger on the use of supplements and breastfeeding practices, and more research is needed on interventions to improve complementary feeding. Some critical areas for further research in this area include (1) how to address human resource issues that constrain program implementation (by state and nonstate providers); (2) what influences quality, equity, and coverage for interventions at scale; and (3) how to cost-effectively improve the coverage of existing nutrition interventions. The types of research issues highlighted in this section link directly with the following questions from the online e-consultation for this focus area: (1) What are some indicators for nutrition implementation capacity, performance, and actual costs?

Nutrition implementation framework and research agenda

and (2) Why do implementation systems not reach sufficient numbers and lack the desired impact? These questions, highlighted by consultation members, resonate with the limited but emerging literature in nutrition, and a stronger literature in the field of health systems. There is recent literature on complex adaptive systems in health services,69,70 new guidance for how to conduct relevant research in health systems,17,71 and a rising movement in the health systems to link researchers studying these issues (http://www.who.int/alliance-hpsr/en/). Core implementation domain 6: Outcome level aspects While a priority, research on what creates successful behavior change at scale is inconsistent. Evidence-based low-cost behaviors are universally identified as among the most potentially effective interventions needed to reach priority nutrition and health goals, such as the Millennium Development Goals. Yet the use of existing services and recommended nutrition behaviors often remains low. Better knowledge of how to research, design, and implement scalable behavior-change strategies is critical to improving the demand, coverage, and adherence for priority nutrition interventions. This section covers the outcome level of nutrition interventions, specific to the use of nutrition services and practices. These behaviors may need to be supported, created, or changed, and this is recognized as complex. Some behavior categories that were looked at in the studies reviewed included attitudes/awareness, use of service by program clients, whether there was a trial and/or initial adoption of the recommended practice (usually by the caregiver), and/or whether the practices were sustained over a period of time. Insights from the nutrition literature. Although it has been shown that strategies to improve behaviors can have positive effects on feeding practices,8 a number of the specific steps or core components (such as use of formative research and the need to adapt to local conditions) that define adequate or well-designed strategies remain elusive, especially within discussions of demand creation for priority behaviors at scale. In fact, recent reviews7,72 found that there were many gaps in how behavior-oriented research is conducted, and recommended clearer application of theories (e.g., frameworks, programimpact pathways) of behavior-change research and

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

51

Nutrition implementation framework and research agenda

the use of common metrics on how programs designed, implemented, and tracked behavior strategies. The behavior-focused outcomes from IYCF and use of micronutrient powders studies that have been examined in the nutrition literature include: attitudes/awareness, use of service by program clients, whether there was a trial and/or initial adoption of the recommended practice, and/or whether the practices were sustained over a period of time.50,57,61,73–76 These studies generally found successful achievement of outcomes across multiple categories. Some of the consistency in the findings of these studies may point to the fact that these studies were all well guided by program theory, the overall quality of the studies conducted was high, and program implementation itself was generally solid. At the same time, for the field of nutrition, a major challenge is that most of these studies have been conducted in the context of smaller scale research projects or programs, and most were also embedded in programs focused on a limited set of delivery strategies (e.g., community-based nutrition programs or health facilities). As noted earlier, the impact of alternative delivery strategies, including the use of mass media or market-based delivery strategies, has not been evaluated in depth. A major challenge for assessing the behavioral outcomes of IYCF BCC and micronutrient interventions is validity and bias in reported behaviors assessed using dietary recall.77 In addition, in the context of interventions, it is indeed possible that systematic recall bias is introduced in intervened groups owing to social desirability or social approval bias when impact assessments are conducted. Although measurement issues and the issue of recall bias is acknowledged in relation to measurement of breastfeeding,78,79 and that of intervention-induced bias is reasonably well recognized in behavioral intervention research in developed countries,80 these issues are not as recognized in the context of nutrition behavioral research in developing countries. Last but not least, nutrition-intervention research may help to unpack and identify maternal and household capabilities that might contribute to the variability that is often seen in intervention effectiveness, even in efficacy trials of interventions. This is a truly neglected area, and most studies only mention such factors in passing when reflecting on intervention effectiveness. Robust frameworks exist to pro52

Menon et al.

vide guidance on how to think about microlevel enabling environments for nutrition,81 but they have not been adapted or well tested in the framework of interventions. Research needs related to intervention outcomes. The types of research issues highlighted in this section link directly with a question from the Focus Area 3 (FA#3) online consultation (How to generate demand for interventions at civil society levels? What creates demand and successful behavior change at-scale?) Some areas for further investigation include: 1. What factors contribute to impact of nutrition interventions implemented at large scale? 2. What research models are most appropriate for impact-focused research in large-scale programs? 3. What measures of behavioral impact are the most valid and reliable indicators for BCC programs? 4. How do maternal and household factors influence intervention effectiveness? Summary Using a diverse and inclusive set of processes, this paper has developed a framework and research agenda to inform investments in high-priority research to enable scaling up the impact of known and new nutrition interventions. The framework highlights the following core implementation domains around which to generate more evidence: (1) program planning and implementation choices; (2) program management (including training, supervision, motivation); (3) frontline provider and facility capacities; (4) client demand, uptake, and utilization of interventions; and (5) developing indicators to assess impact and unpack client factors influencing program impact. We posit in this paper that, overall, the study of implementation and utilization of nutrition interventions is too nascent to sharply prioritize only a few key questions. Rather, we offer a framework that enables researchers to link across the major domains of necessary research in these areas and highlight the role of the identified factors in enhancing, or dampening, the impact of nutrition interventions. In doing so, we provide an umbrella research agenda for implementation research in nutrition, linked to specific examples of research questions.

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

Nutrition implementation framework and research agenda

What is clear in our development of this paper is that, even in the limited area of BCC interventions focused on IYCF and on micronutrient interventions, we identified several aspects of implementation around which there is limited highquality literature—this includes, but is not limited to, formative research on prioritizing interventions, modalities to support effective supply chains for micronutrients, best strategies for training frontline workers in different nutrition delivery systems, strategies for motivating and eliciting high-quality performance from frontline workers and for ensuring high demand of both BCC and micronutrient interventions, and research to identify critical maternal and household-level factors that can dampen or enhance intervention effects. The planning and conduct of implementation research face several challenges common to other emerging frontiers in population nutrition research.82 These issues require attention from nutrition researchers as well as funders, editors, and research administrators with an interest in nutrition. Some practical considerations that arise when designing and conducting research on delivery-related issues are useful to consider and are described below.

Doing research within programs. Implementation science research is best done within programs being implemented to deliver nutrition interventions. However, research within the context of programs carries its own set of challenges, including (1) reconciling sometimes conflicting research objectives and values with program implementation objectives and values; (2) identifying feasible, yet rigorous, study designs for implementation research within large-scale programs; (3) balancing research timelines with implementation timelines; and (4) pressures to disseminate findings immediately versus waiting to publish in academic journals. Several studies in nutrition have been done in smaller scale settings,61,64,65,83 and examples of long-term and large-scale research initiatives are emerging,32 but such examples are limited, for some of the very reasons mentioned here. Funding and capacity for implementation research in nutrition. Funding for research on implementation of nutrition interventions is essential to support and foster high-quality research and to build the capacity to demand and conduct such research, but the funding for

Table 2. Domains of implementation, utilization, and outcomes and related areas of research

Implementation processes

Implementation outcomes

r r r r r

r Acceptability r Fit r Feasibility r Costs r Application r Adoption r Adaption r Fidelity r Penetration

r

r r r r r r

Formative research Planning Implementation strategy Change theory Engaging r Opinion leaders r Formal leaders r Champions r Facilitators Execution r Components r Sequence r Intensity r Duration Quality Feedback Evaluation Reflection Decisions Adjustments

Service outcomes r r r r r r

Efficiency Effectiveness Equity Safety Patient-centeredness Timeliness

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Client outcomes r Health/function r Symptoms r Satisfaction

53

Nutrition implementation framework and research agenda

Menon et al.

Table 3. Areas of research investigation and types of methods that can be used in nutrition implementation research

Areas of investigation

Methods

Implementation r Quality of training, facilities, and service delivery r Incentives for performance r Role of supervision r Facility quality r Counseling quality r CHW workload r CHW job satisfaction Uptake and utilization of interventions r Access and exposure to services r Satisfaction with service r Ability to pay for services Costing studies Contextual factors r Role of other programs/interventions r Role of family and society

nutrition-implementation research lags behind funding investments in discovery of new technologies.15 In addition, the contrast between the field of health systems and nutrition delivery systems is quite stark, for instance, in the extent and quality of the implementation research and the capacity to undertake research on such issues. Multiple potential end points and the complexity of implementation systems. The end points related to implementation quality, coverage, scale, targeting, and program utilization are numerous (Table 2), and developing research on these end points can seem daunting to nutrition researchers. At the same time, several of these outcomes are often more within the control of program implementers than end points with longer causal chains between the program inputs and impacts, and are amenable to study through diverse methods (Table 3). Another challenge, where key areas for research are not clear, is that the complexity and wide range of factors that affect implementation creates challenges for researchers who are keen to conduct studies to identify key bottlenecks throughout the implementation system. Approaches for doing so and for prioritizing key areas for research and program improvement include the Program Assessment Guide or other learning collaborative approaches, which offer practical tools to facilitate analyses, develop program quality improvement plans, and identify areas for

54

Qualitative research methods r Review of program documents (manuals, job aids) r Structured direct observations r Home-based interviews with CHWs, other program staff, and mothers (qualitative or quantitative) r Focus group discussions r “Shadowing” of program staff Quantitative research methods r Pre- and posttraining assessments (CHWs) r CHW surveys r Household surveys r Exit interviews at facilities r Market surveys Data from routine MIS/program documentation

additional research, drawing on contextual knowledge and perspectives of diverse stakeholders.14 Interdisciplinary research. Given the diverse nature of the possible study outcomes in nutrition implementation, research on these topics is best done in collaboration with researchers in fields outside of the core of nutrition science. Such research demands disciplinary orientations from management, organizational behavior, health systems, economics, sociology, political science, and anthropology, to name a few. Rather than resist, the nutrition community must reach out to these disciplines and build familiarity with their methods and approaches. Incentives for researchers. Researchers need to be able to publish their research in high-quality journals. However, traditional publication venues in the field of nutrition could potentially lock researchers studying issues related to implementation out of the mainstream of what are considered nutrition science journals. In addition, there are significant challenges to develop research publications based on what can often be negative or simply very contextspecific findings about program implementation or utilization. This, combined with a low-demand environment or a low-funding environment, can lead to a negative publication bias. Traditional versus newer paradigms for research. There are two possible dominant approaches for

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

research on program/intervention implementation and utilization. First, research that uses a more traditional research orientation is researcher driven and uses more traditional methods including the use of randomized controlled trials. This approach will likely yield collections of individual studies that together build up a generalizable knowledge base on key implementation factors/barriers and evaluate possible interventions to address them. At the same time, given the larger desire for research on issues related to intervention implementation to inform program and policy actions, this approach can be limiting in that it might not address user needs in given contexts and might entail the use of research methods and operate in time frames that are potentially incompatible with decision-making time frames in real policy and program contexts. A second approach to building knowledge on program implementation is a more integrated, user-driven approach. This can be achieved by creating learning platforms at national or even subnational levels, where researchers and implementers routinely come together to discuss implementation issues (with a view of the whole system in mind), resolve those that are clear from the outset, do focused research on others in order to understand them better and resolve them, and maintain an evolving learning, implementation, and improvement agenda over time. This approach has many attractions, including a higher likelihood that the research itself will lead to changes and improvements over time. However, it has the disadvantage that not all such context-specific research enters the research literature, because of low research capacity, short time frames, and low funding for such research at national or subnational levels. Although there are substantial challenges in this area, it is useful to remember that they are certainly not insurmountable. Recent reviews and thought pieces in mainstream nutrition journals6,11,16,19,82,84 and the several nutrition studies cited in this paper are already emblematic of important efforts to further the field. At the same time, the stated challenges of building demand for such research, financing the research, offering guidance on how best to link programs and research, and ensuring adequate incentives to publish are key areas to develop solutions to and generate resources. In closing, therefore, we offer the following recommendations to strengthen the base of evidence

Nutrition implementation framework and research agenda

on implementation of nutrition interventions and programs. First, the implementation literature on descriptions of interventions itself needs to be built up systematically; for nutrition as well as other domains of implementation research, intervention trials and program implementation descriptions need to include better and more detailed descriptions of what inputs and processes go into interventions.20 This will ensure that adequate information is available on issues pertaining to domains 1 and 2 in this paper. An important component of descriptions of program implementation are costing studies, which describe the actual costs of implementing certain interventions (see Box 2), and we encourage a stronger emphasis on costing as part of program documentation. Second, a consistent approach to evaluation/intervention research is essential to build a coherent body of research. This requires a program theory–driven approach and linked impact and process evaluations of intervention that measure several intermediate outcomes adequately in addition to impact indicators. The consistent use of the Nutrition Implementation Framework, linked with program- or intervention-specific impact pathways can help to identify indicators of interim processes and outcomes that are most relevant for the diverse types of nutrition interventions and implementation platforms for these interventions. Third, we recommend the creation of a database of completed and ongoing research on delivering critical nutrition interventions. This could be linked to existing databases maintained by the WHO such as the e-LENA or the Global Database on Implementation of Nutrition Actions (GINA). Additional existing resources that nutrition researchers can link with and contribute to include the Grid-Enabled Measures (GEM), which is a database of behavioral and social science measures organized by theoretical constructs (http://cancercontrol.cancer.gov/brp/gem.html) and the Consolidated Framework for Implementation Research (CFIR) Wiki (http://www.wiki.cfir.net/index.php?title=Main_Page). Fourth, we note that a variety of specific efforts are likely needed within the next year or two to facilitate comparable research across countries/regions to generate cross-cutting generalizable information to foster better links between nutrition researchers

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

55

Nutrition implementation framework and research agenda

with other disciplines. The areas of health systems and the social sciences (economics, sociology, and anthropology) are two essential places to start. Finally, efforts to raise resources for implementation research in nutrition and to advocate for implementation research are crucial to move this area of research forward. A good starting point for this could be prioritizing critical research needs within country-led SUN activities and developing a cross-country research-needs portfolio that is funded bilaterally or through multidonor cross-country trust funds. Conclusions In the current context, where implementation systems for nutrition interventions are currently suboptimal and the use of existing services and recommended behaviors is low, especially among the poor, a more systematic and complete understanding of program delivery, including quality of implementation, service delivery, and coverage, as identified in this paper, can greatly strengthen the ability to successfully scale-up and sustain prioritized nutrition interventions. The global and national momentum for nutrition cannot be realized without the necessary investments in building an evidence base to support the implementation of nutrition interventions. Acknowledgments Contributions from the initial delivery science working group members who developed some of the core implementation research-related ideas and generated the questions for the e-consultation are deeply acknowledged. These include contributions from Jane Badham, JB Consultancy, South Africa; Shawn Baker, Helen Keller International; Priya Bapat, Humanitas Global; Robert E. Black, Johns Hopkins Bloomberg School of Public Health; Francesco Branca, WHO; Kathryn G. Dewey, University of California-Davis; Stuart R. Gillespie, International Food Policy Research Institute; Klaus Kraemer, Sight and Life; Chizuru Nishida, WHO; Amanda Pomeroy, John Snow, Inc.; Andrew C. Serazin, Matatu; Patrick Webb, Tufts University; Keith P. West, Johns Hopkins University; Edelweiss WentzelViljoen, North-West University, South Africa. We are grateful for the support from Mandana Arabi and Mireille McLean, of the Sackler Institute for Nutrition Science, to the process of developing 56

Menon et al.

this paper, and to supporting the consultations. We also gratefully acknowledge Gretel Pelto, Cornell University, for her thoughtful remarks as a discussant for this paper, which was presented December 17-18, 2012, at the conference “A Global Research Agenda for Nutrition Science: Building Consensus on Knowledge Gaps to Stimulate High Impact Research.” The conference was held at the New York Academy of Sciences, New York, NY; more information is available at: http://www.nyas. org/Publications/Ebriefings/Detail.aspx?cid=38eb8 a81-ed18-49f4-b9d1-0a5d887e712c. We thank Kavita Singh, Research Analyst, IFPRI, for assistance with citations. Individual author contributions were as follows. Purnima Menon developed the paper outline; wrote the introduction and summary sections; co-wrote the section on domain 5; and revised, completed, and submitted the full draft of the paper. Namukolo M. Covic co-wrote the introduction, wrote domain 2 and 3, and contributed to the overall literature review. Paige B. Harrigan co-wrote the section on domain 5, wrote the section on domain 6, and supported the overall literature review. Susan E. Horton wrote sections related to costing. Nabeeha M. Kazi coordinated and led the e-consultation and wrote sections on the outcomes of nutrition programs for domain 6. Sascha Lamstein led the writing for domains 2 and 3. Lynnette Neufeld led the writing on domain 4. Erica Oakley performed analysis and writing on sections related to e-consultation and supported the final paper preparation and literature review. David Pelletier developed and wrote sections describing the framework and reviewed and contributed to the summary section. All authors were core participants in the paper development process, including several teleconference calls, an in-person meeting in December 2012, and review and comments on paper outlines and drafts. Conflicts of interest The authors declare no conflicts of interest. References 1. Black, R.E., C.G. Victora, S.P. Walker, et al. 2013. Maternal and child undernutrition and overweight in low-income and middle-income countries. Lancet 382: 427–451. 2. Bhutta, Z.A., J.K. Das, A. Rizvi, et al. 2013. Evidence-based interventions for improvement of maternal and child nutrition: what can be done and at what cost? Lancet 382: 452–477.

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

3. Bhutta, Z., T. Ahmed, R.E. Black, et al. 2008. What works? Interventions for maternal and child undernutrition and survival. Lancet 371: 417–440. 4. Ramakrishnan, U., F. Grant, T. Goldenberg, et al. 2012. Effect of women’s nutrition before and during early pregnancy on maternal and infant outcomes: a systematic review. Paediatr. Perinat. Epidemiol. 26(Suppl. 1): 285–301. 5. SUN. Scaling. 2010. Up Nutrition. A framework for action. http://siteresources.worldbank.org/NUTRITION/Resources/ 281846-1131636806329/PolicyBriefNutrition.pdf. 6. Habicht, J.P. & G.H. Pelto. 2014. From biological to program efficacy: promoting dialogue among the research, policy, and program communities. Adv. Nutr. 5: 27–34. 7. Habicht, J.P. & G.H. Pelto. 2011. Multiple micronutrient interventions are efficacious, but research on adequacy, plausibility, and implementation needs attention. J. Nutr. 142: 205S–209S. 8. Dewey, K.G. & S. Adu-Afarwuah. 2008. Systematic review of the efficacy and effectiveness of complementary feeding interventions in developing countries. Matern. Child. Nutr. 4(Suppl. 1): 24–85. 9. Frongillo, E., S. Gillies, C. Horan, et al. 2007. Delivering for impact: a systematic review of strategies used for delivering critical public health interventions. Unpublished Work. 10. Gillespie, S., L. Haddad, V. Mannar, et al. 2013. The politics of reducing malnutrition: building commitment and accelerating progress. Lancet 382: 552–569. 11. Leroy, J.L. & P. Menon. 2008. From efficacy to public health impact: a call for research on program delivery and utilization in nutrition. J. Nutr. 138: 628–629. 12. Shekar, M. 2008. Delivery sciences in nutrition. Lancet 371: 1751. 13. Olney, D.K., R. Rawat & M.T. Ruel. 2012. Identifying potential programs and platforms to deliver multiple micronutrient interventions. J. Nutr. 142: 178S–185S. 14. Pelletier, D., A. Corsi, L. Hoey, et al. 2011. The Program Assessment Guide: an approach for structuring contextual knowledge and experience to improve the design, delivery, and effectiveness of nutrition interventions. J. Nutr. 141: 2084–2091. 15. Leroy, J.L., J.P. Habicht, G. Pelto & S.M. Bertozzi. 2007. Current priorities in health research funding and lack of impact on the number of child deaths per year. Am. J. Public Health 97: 219–223. 16. Stoltzfus, R.J. 2014. How can the scientific community support the generation of the evidence needed to improve the quality of guidelines for micronutrient interventions? Adv. Nutr. 5: 40–45. 17. Peters, D.H., T. Adam, O. Alonge, et al. 2013. Implementation research: what it is and how to do it. BMJ 347: f6753. 18. Mendel, P., L.S. Meredith, M. Schoenbaum, et al. 2008. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm. Policy Ment. Health 35: 21–37. 19. Garrett, J.L. 2008. Improving results for nutrition: a commentary on an agenda and the need for implementation research. J. Nutr. 138: 646–650.

Nutrition implementation framework and research agenda

20. Michie, S., D. Fixsen, J. Grimshaw & M. Eccles. 2009. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci. 4: 40. 21. Pelletier, D.L., E.A. Frongillo, S. Gervais, et al. 2011. Nutrition agenda setting, policy formulation and implementation: lessons from the mainstreaming nutrition initiative. Health Policy Plan 27: 19–21. 22. Fixsen, D.J., S.F. Naoom, K.A. Blase, et al. 2005. Implementation Research: A Synthesis of the Literature. (FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. 23. Saunders, R.P., M.H. Evans & P. Joshi. 2005. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot. Pract. 6: 134–147. 24. Shiell, A., P. Hawe & L. Gold. 2008. Complex interventions or complex systems? Implications for health economic evaluation. BMJ 336: 1281–1283. 25. Stead, M., G. Hastings & D. Eadie. 2002. The challenge of evaluating complex interventions: a framework for evaluating media advocacy. Health Educ. Res. 17: 351–364. 26. Meyers, D.C., J.A. Durlak & A. Wandersman. 2012. The quality implementation framework: a synthesis of critical steps in the implementation process. Am. J. Commun. Psychol. 50: 462–480. 27. Hendriks, A.M., M.W. Jansen, J.S. Gubbels, et al. 2013. Proposing a conceptual framework for integrated local public health policy, applied to childhood obesity-the behavior change ball. Implement. Sci. 8: 46. 28. Damschroder, L.J., D.C. Aron, R.E. Keith, et al. 2009. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement. Sci. 4: 50. 29. Kumanyika, S.K., L. Parker & L.J. Sim. 2010. Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making. Washington, DC: National Academies Press. 30. Marsh, D.R., J.C. Alegre & K.Z. Waltensperger. 2008. A results framework serves both program design and delivery science. J. Nutr. 138: 630–633. 31. Kim, S.S., J.P. Habicht, P. Menon & R.J. Stoltzfus. 2011. How Do Programs Work to Improve Child Nutrition? IFPRI Discussion Paper 01105. Washington DC: IFPRI. 32. Rawat, R., P.H. Nguyen, D. Ali, et al. 2013. Learning how programs achieve their impact: embedding theory-driven process evaluation and other program learning mechanisms in alive & thrive. Food Nutr. Bull. 34(3 Suppl.): S212–S225. 33. Pelletier, D. 2008. The Mainstreaming Initiative: Commitment, Consensus and Strategic Capacity: An Evidence-Based Agenda. SCN News No. 36. 34. Pelletier, D.L., P. Menon, T. Ngo, et al. 2011. The nutrition policy process: the role of strategic capacity in advancing national nutrition agendas. Food Nutr. Bull. 32: S59–S69. 35. Hoey, L. & D.L. Pelletier. 2011. The management of conflict in nutrition policy formulation: choosing growth-monitoring indicators in the context of dual burden. Food Nutr. Bull. 32: S82–S91.

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

57

Nutrition implementation framework and research agenda

36. Rudan, I., L. Kapiriri, M. Tomlinson, et al. 2010. Evidencebased priority setting for health care and research: tools to support policy in maternal, neonatal, and child health in Africa. PLoS Med. 7: e1000308. 37. Lehmann, U. & D. Sanders. 2007. Community Health Workers: What Do We Know About Them? Geneva: World Health Organization. 38. WHO. 2003. Community-Based Strategies for Breastfeeding Promotion and Support in Developing Countries. Geneva: World Health Organization. 39. Martin, L.H., M. Hossain, C. Casanovas & A.B. Guyon. 2008. Learning from Large-Scale Community-Based Programmes to Improve Breastfeeding Practices. Geneva: World Health Organization. 40. Iannotti, L. & S. Gillespie. 2002. Successful Community Nutrition Programming: Lessons from Kenya, Tanzania, and Uganda. Washington DC: USAID. 41. Puoane, T., K. Cuming, D. Sanders & A. Ashworth. 2008. Why do some hospitals achieve better care of severely malnourished children than others? Five-year follow-up of rural hospitals in Eastern Cape, South Africa. Health Policy Plan. 23: 428–437. 42. Giugliani, C., B.B. Duncan, E. Harzheim, et al. 2010. The impact of a short-term intervention using the WHO guidelines for the management of severe malnutrition at a rural facility in Angola. Arch. Dis. Child. 95: 198–202. 43. ACC/SCN. 2013. Managing Successful Nutrition Programmes—Nutrition Policy Discussion Paper No. 8. ACC/SCN. 44. Franco, L.M., S. Bennett & R. Kanfer. 2002. Health sector reform and public sector health worker motivation: a conceptual framework. Soc. Sci. Med. 54: 1255–1266. 45. Healthy Child Uganda. 2011. Community Health Workers and Community Case Management: Policies and Progress in Uganda within a Global Context. 46. Kennedy, E. 1991. Successful Nutrition Programs in Africa: What Makes Them Work? Washington DC: World Bank. 47. Peters, D.H., S. El-Saharty, B. Siadat, et al. 2009. Improving health service delivery in developing countries. Washington, DC: World Bank. 48. Quinn, V.J., A.B. Guyon, J.W. Schubert, et al. 2005. Improving breastfeeding practices on a broad scale at the community level: success stories from Africa and Latin America. J. Hum. Lact. 21: 345–354. 49. Mucha, N. & M. Tharaney. 2013. Strengthening Human Capacity to Scale Up Nutrition. Bread for the World Institute, Hellen Keller International. Available at http://www.bread.org/event/gathering-2013/internationalmeeting/pdf/strengthening-human-capacity.pdf. Accessed April 23, 2014. 50. Pelto, G.H., I. Santos, H. Goncalves, et al. 2004. Nutrition counseling training changes physician behavior and improves caregiver knowledge acquisition. J. Nutr. 134: 357– 362. 51. Abosede, O. & J. McGuire. 1991. Improving Women’s and Children’s Nutrition in Sub-Saharan Africa. Washington DC: World Bank Policy Research and External Affairs Working Paper WPS 723.

58

Menon et al.

52. UNICEF. 2012. Supportive supervision/mentoring and monitoring for community IYCF. Available at http://www. unicef.org/nutrition/files/Supervision_monitoring_module _Nov_2012.pdf. Accessed April 23, 2014. 53. Willis-Shattuck, M., P. Bidwell, S. Thomas, et al. 2008. Motivation and retention of health workers in developing countries: a systematic review. BMC Health Serv. Res. 8: 247. 54. Daniels, K., B. Nor, D. Jackson, et al. 2010. Supervision of community peer counsellors for infant feeding in South Africa: an exploratory qualitative study. Hum. Resour. Health 8: 6. 55. Dickin, K.L., J.S. Dollahite & J.P. Habicht. 2011. Enhancing the intrinsic work motivation of community nutrition educators: how supportive supervision and job design foster autonomy. J. Ambul. Care Manage 34: 260–273. 56. Dickin, K.L., J.S. Dollahite & J.P. Habicht. 2010. Job satisfaction and retention of community nutrition educators: the importance of perceived value of the program, consultative supervision, and work relationships. J. Nutr. Educ. Behav. 42: 337–344. 57. Bonvecchio, A., G.H. Pelto, E. Escalante, et al. 2007. Maternal knowledge and use of a micronutrient supplement was improved with a programmatically feasible intervention in Mexico. J. Nutr. 137: 440–446. 58. Beck, A.M., K. Damkjaer & I. Tetens. 2009. Lack of compliance of staff in an intervention study with focus on nutrition, exercise and oral care among old (65 +yrs) Danish nursing home residents. Aging Clin. Exp. Res. 21: 143–149. 59. Horwood, C., K. Vermaak, N. Rollins, et al. 2009. An evaluation of the quality of IMCI assessments among IMCI trained health workers in South Africa. PLoS One 4: e5937. 60. Hoddinott, P., J. Britten & R. Pill. 2010. Why do interventions work in some places and not others: a breastfeeding support group trial. Soc. Sci. Med. 70: 769–778. 61. Robert, R.C., J. Gittelsohn, H.M. Creed-Kanashiro, et al. 2006. Process evaluation determines the pathway of success for a health center-delivered, nutrition education intervention for infants in Trujillo. Peru. J. Nutr. 136: 634–641. 62. Robert, R.C., J. Gittelsohn, H.M. Creed-Kanashiro, et al. 2007. Implementation examined in a health centerdelivered, educational intervention that improved infant growth in Trujillo, Peru: successes and challenges. Health Educ. Res. 22: 318–331. 63. Puett, C., J. Coates, H. Alderman, et al. 2012. Does greater workload lead to reduced quality of preventive and curative care among community health workers in Bangladesh? Food Nutr. Bull. 33: 273–287. 64. Loechl, C.U., P. Menon, M. Arimond, et al. 2009. Using programme theory to assess the feasibility of delivering micronutrient Sprinkles through a food-assisted maternal and child health and nutrition programme in rural Haiti. Matern. Child Nutr. 5: 33–48. 65. Suchdev, P.S., L. Ruth, A. Obure, et al. 2010. Monitoring the marketing, distribution, and use of Sprinkles micronutrient powders in rural western Kenya. Food Nutr. Bull. 31(2 Suppl.): S168–S178. 66. Chopra, M., A. Sharkey, N. Dalmiya, et al. 2012. Strategies to improve health coverage and narrow the equity gap in child survival, health, and nutrition. Lancet 380: 1331–1340.

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

Menon et al.

67. Perez-Escamilla, R., L. Curry, D. Minhas, et al. 2012. Scaling up of breastfeeding promotion programs in low- and middle-income countries: the “breastfeeding gear” model. Adv. Nutr. 3: 790–800. 68. World Bank. 2010. What Can We Learn from Nutrition Impact Evaluations? Lessons from a Review of Interventions to Reduce Malnutrition in Developing Countries. Washington DC: World Bank, Independent Evaluation Group. 69. Hill, P.S. 2011. Understanding global health governance as a complex adaptive system. Glob. Public Health 6: 593– 605. 70. Paina, L. & D.H. Peters. 2012. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan. 27: 365–373. 71. WHO. 2012. Health Policy and Systems Research: A Methodology Reader. Geneva, Switzerland: World Health Organization. 72. Aboud, F.E. & D.R. Singla. 2012. Challenges to changing health behaviours in developing countries: a critical overview. Soc. Sci. Med. 75: 589–594. 73. Affleck, W. & G. Pelto. 2012. Caregivers’ responses to an intervention to improve young child feeding behaviors in rural Bangladesh: a mixed method study of the facilitators and barriers to change. Soc. Sci. Med. 75: 651–658. 74. Penny, M.E., H.M. Creed-Kanashiro, R.C. Robert, et al. 2005. Effectiveness of an educational intervention delivered through the health services to improve nutrition in young children: a cluster-randomised controlled trial. Lancet 365: 1863–1872. 75. Zaman, S., R.N. Ashraf & J. Martines. 2008. Training in complementary feeding counselling of healthcare workers and its influence on maternal behaviours and child growth: a cluster-randomized controlled trial in Lahore, Pakistan. J. Health Popul. Nutr. 26: 210–222. 76. Santos, I., C.G. Victora, J. Martines, et al. 2001. Nutrition counseling increases weight gain among Brazilian children. J. Nutr. 131: 2866–2873. 77. Contento, I.R., J.S. Randell & C.E. Basch. 2002. Review and analysis of evaluation measures used in nutrition education intervention research. J. Nutr. Educ. Behav. 34: 2–25.

Nutrition implementation framework and research agenda

78. Moore, S.E., A.M. Prentice, W.A. Coward, et al. 2007. Use of stable-isotope techniques to validate infant feeding practices reported by Bangladeshi women receiving breastfeeding counseling. Am. J. Clin. Nutr. 85: 1075–1082. 79. Li, R., K.S. Scanlon & M.K. Serdula. 2005. The validity and reliability of maternal recall of breastfeeding practice. Nutr. Rev. 63: 103–110. 80. Hebert, J.R., L. Clemow, L. Pbert, et al. 1995. Social desirability bias in dietary self-report may compromise the validity of dietary intake measures. Int. J. Epidemiol. 24: 389–398. 81. Engle, P.L., P. Menon & L. Haddad. 1999. Care and nutrition: concepts and measurement. World Dev. 27: 1309–1337. 82. Pelletier, D.L., C.M. Porter, G.A. Aarons, et al. 2013. Expanding the frontiers of population nutrition research: new questions, new methods, and new approaches. Adv. Nutr. 4: 92–114. 83. Jefferds, M.E., L. Ogange, M. Owuor, et al. 2010. Formative research exploring acceptability, utilization, and promotion in order to develop a micronutrient powder (Sprinkles) intervention among Luo families in western Kenya. Food Nutr. Bull. 31(2 Suppl.): S179–S185. 84. Stoltzfus, R.J. 2008. Research needed to strengthen science and programs for the control of iron deficiency and its consequences in young children. J. Nutr. 138: 2542–2546. 85. Horton, S., H. Alderman & J. Rivera. 2009. Hunger and malnutrition. In Global Crises, Global Solutions: Costs and Benefits. B. Lomborg, Ed.: 305–346. Cambridge, UK: Cambridge University Press. 86. Fiedler, J.L., T.G. Sanghvi & M.K. Saunders. 2008. A review of the micronutrient intervention cost literature: program design and policy lessons. Int. J. Health Plan. Manage 23: 373–397. 87. Brooker, S., N.B. Kabatereine, F. Fleming & N. Devlin. 2008. Cost and cost-effectiveness of a nationwide school-based helminth control in Uganda: intra-country variation and effects of scaling-up. Health Policy Plan 23: 24–35. 88. Horton, S., M. Shekar, C. McDonald, et al. 2010. Scaling Up Nutrition: What Will It Cost? Washington DC: The International Bank for Reconstruction and Development/The World Bank.

C 2014 New York Academy of Sciences. Ann. N.Y. Acad. Sci. 1332 (2014) 39–59 

59

Copyright of Annals of the New York Academy of Sciences is the property of WileyBlackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.

Strengthening implementation and utilization of nutrition interventions through research: a framework and research agenda.

Undernutrition among women and children contributes to almost half the global burden of child mortality in developing countries. The impact of nutriti...
508KB Sizes 0 Downloads 3 Views