Scientific economic Richard
W.
consensus, planning Longhurst,’
MS.
nutrition
and
David
L. Call,2
The concept of nutrition as a crucial determinant of national development is a subject under considerable discussion at the moment. The conference at Massachusetts Institute of Technology on Nutrition, National Development and Planning in October 1971 (1) and the proposed action of the World Bank to designate
nutrition
programs
as
areas
for
investment are indications of this growing realization (2). In a larger context, those concerned with allocating resources for national development indicate that rather than concentrate on maximizing economic growth, more attention should be directed to the characteristics of the individuals affected by economic development. The following cornments about the role of nutrition in national development should be viewed from the perspective that if we are to be concerned with redirecting development efforts more accurately to the poorer elements in society, nutrition programs are one means by which this could be achieved. The question with respect to the development of a nation is two-layered: What is the trade-off between the magnitude of development benefits and their pattern of distribution? What is the impact of nutrition as a component in this predicament? In seeking answers to specific questions about nutrition programs, the planner faces a dilemma. Although prima facie nutrition must have some impact on disease resistance, labor productivity, reproductive capacity, intellectual development and receptivity to education, the exact relationship is unknown. In other words, the production, which relates inputs (nutrition in this case) to outputs (say, labor productivity), has not been quantified. The
American
Journal
of Clinical
Nutrition
programs
28: OCTOBER
article
and
Ph.D.
Some planner
How
of
the would
would
improved
productivity,
What
important like to e.g.,
sort
questions which a have answered are:
nutrition in
of impact
affect
agricultural
would
labor
activities?
changes
in nutri-
tion have on birth weight and on infant mortality? How does nutrition affect intellectual development in preschool years? How much would the provision of a school lunch affect learning ability? These are all questions that represent one link in assessing the.role of nutrition in the development process. Questions concerning the concept of adequacy of diet and the interrelationships of nutrition and parasitic diseases need to be answered to provide subsidiary information in the planning process. Still another step in the progression brings us to questions concerning the use of certain indicators to measure change in factors on which nutrition has an impact, e.g.,
IQ for intellectual for
pregnancy
diseases
and
development,
birth
weight
outcome,
incidence of certain status of health of a population of work output for labor produc-
for
volume
tivity.
Blame strength tion and 1
Graduate
for
lack
of the various School
of understanding
relationships parameters of Arts
and
of the
between of human Social
Studies,
nutriperSussex
University, this author Agricultural
England. This research was carried out while was a graduate student in the Department of Economics and Graduate School of Nutrition at Cornell University, Ithaca, New York. 2 Director of Cooperative Extension, New York State College of Agriculture and Life Sciences and New York State College of Human Ecology, Cornell University, Ithaca, New York. This research was carried out while this author was the H. E. Babcock Professor of Food Economics in the Graduate School of Nutrition at Cornell University.
1975, pp. 1 177-1 182. Printed
in U.S.A.
I 177
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
special
LONGHURST
1178
formance nutritionists important not
should not be accorded to the engaged in research. Although work still needs to be done, society
organized
that
so
the impact
of nutri-
tion on humans can be isolated as a single variable in controlled longitudinal studies. Efforts to translate the results of nutrition research into operational programs that can
be
subsequently
appraised
and
evaluated
alongside other programs have been trated not only by the failure ofeconomists planners and nutritionists to understand others’ work, but also by a disappointing
of agreement
between
nutritionists
frusor each lack
as to pri-
orities for investment of very limited resources. This paper documents an investigation that tried to derive a consensus of opinion among nutritionists, pediatricians
and
members
possible
of
changes
associated
disciplines
on
in
parameters
of
certain
human performance in response to an investment program that could be fitted into a planning model. It is reasonable to state that no published research gives a definite indication as to how much a change in nutrient intake weight, ductivity. Use
will affect educability,
such parameters morbidity or
of subjective
as work
birth pro-
estimation
In these circumstances, the authors decided to experiment with the Delphi technique (3). This method calls for interaction of a group of experts under controlled conditions in an attempt to obtain a consensus on certain quantitative estimations. The Delphi technique is carried out by a succession of mailed questionnaires. This is done to eliminate the unfavorable “personality” effects inherent in
a consensus reached by roundtable discussion where certain individuals may impose their views
by virtue
or vocal have
of greater
ability.
shown
that
that could later more accurate discussion. A typical
in the
Delphi
experts
is
and
(4, 5)
estimation
of values
the Delphi was a than roundtable
invites
a group
to estimate subject
standing
investigators
be verified, mechanism
knowledged uncertain
academic
Several
to many
a value
of
ac-
which
interpreta-
tions. In cases where there is no means of obtaining a “true” value, the use of subjectiv-
CALL
ity in the form of expert opinion is a definite improvement over no value at all. The ranges of the estimates obtained from the experts in the first round of the questionnaire, together
with
the
median
representative
experts
with
naire. They estimations is especially the extremes is preserved
to five sive
the second
(the
rounds
most
is sent
round
to the
of the question-
of questionnaires. ambiguous
and
important
Delphi
On
succes-
questions
are
information
as necessary. Of course, derive a consensus.
The
statistic
opinion)
are asked to reconsider their in the light ofgroup opinion. This important for those who are at of the distribution. Anonymity at all times and there may be up
rounds
phrased
value
of group
it is not
technique
is not
re-
inserted necessary
without
to
diffi-
culties but these can be minimized with efficient administration. Some of the criticisms mentioned by Helmer (6) are instability of panel membership, time lapse between questionnaires, ambiguous questions, respondents’ competence, substantive breadth versus specialization and consensus by undue averaging. This latter criticism is unjustified
on the basis
that
the
respondent
who
main-
tains an extreme position should be able to convince the others ofthe rationale behind his estimate. It is not yet known how much of the convergence that takes place is induced by the process itself rather than by elimination of the basic causes of disagreement, although some evidence suggests that most of the change in the median is caused by those unsure of their answers (3).
The computerization removed
many
for the
large
of the Delphi
of these
scale
criticisms,
experiment;
has now especially
it can
be an
efficient technique retaining its important characteristics of anonymity, controlled feedback and the important end product of a consensus response.
The
important
questionnaire
that
the former
difference and
between
a survey
attempts
a Delphi
questionnaire
to draw
out from
is
the
respondents the underlying premises for their opinions. Hence, given the initial assumption that in a particular situation benefits can be derived from the utilization of subjective estimations from experts, then the Delphi can be considered the “best” means for obtaining
such
estimates.
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
is
AND
NUTRITION
The
Delphi
PROGRAMS
AND
TABLE Occupations
experiment
have to be directed were.
to whom
Some description or specification of a target group is necessary when a group of experts is asked to estimate the impact of nutrition programs on birth weight and intellectual development. A survey carried out by the Graduate School of Nutrition at Cornell University (7) to assess the impact of the USDA Food Certificate Program produced a considerable amount of background material
on the
socioeconomic
status
of a group
of
low-income mothers in a depressed urban area. Hence it was proposed that the panel should be asked to predict the effect of certain programs in the areas of nutrition, medical care and preschool education on this target group within a specified time period. In the area of pregnancy outcome (birth weights), 24 experts in the field of pregnancy, nutrition and medical care were invited to participate in the experiment. They were told that only 15
mm of their
time needed
to be devoted
to each
questionnaire. Eighteen subsequently agreed to take part. In the area of intellectual development, 22 experts with interest in child psychology, development and nutrition were invited to participate and 12 agreed to do so.
A breakdown
of the subject-matter
interests
of these experts is given in Table 1. The first round of the Delphi mailed early in 1971 attempted to sort out the relative importance of environmental components that might be maniptilated by the introduction of a government program. The respondents were asked to rate (with respect to the importance of the target group) the effect such factors as age of the mother at delivery, number of previous pregnancies, extent of
PLANNING
I of respondents
Pregnancy
.4)
outcome
Epidemiologist Obstetrician Neonatalogist Pathologist Pediatrician Four
of the in nutrition.
development
Occupation
2 2 5 1 I 7
Nutritionist
a
B) Intellectual No.
Occupation”
interest
I179
No.
Pediatrician Nutritionist Sociologist Psychologist
nonnutritionists
4 5 2 I
expressed
a
strong
mother’s education, and contact with prenatal services would have on birth weight. The other group of respondents were asked to rate the effect of factors such as presence of the father, family income level and quality of housing conditions on performance of 5-yearolds on an IQ test. (We do not intend to enter the controversy over the usefulness of IQ scores as a measure of intelligence. Although we recognize its limitations, it represents the best quantitative measure developed so far.)
On round,
the basis of the results the second questionnaire
of this first proposed
possible intervention programs based on their administrative feasibility and on the incorporation of these components thought to be important from responses to the first-round questionnaire. The respondents were supplied with such information on the target group as income level, medical care utilization, and the like. Specific programs were described and the respondents in one group asked to estimate the impact which 6 years of each program would have on the incidence of low birth weight (the survey having shown that this was 15%). The respondents in the second group were asked to estimate the impact of 6 years of each program on the average IQ score of 5-year-old children (having assumed for the purposes of this study that the average
IQ score
of 5-year-olds
Descriptions available on
are
available
was then
of the proposed request for those
For
the
Delphi
the
follow-
fect
design
“program,”
(8).
are and
assessing changes in birth weight, ing programs were proposed: 1) income maintenance 2) supplemental feeding either
program
elsewhere
at 85 points). programs interested,
or by a pure i.e.,
nutrition
by a given dietary efwould
be
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
Two factors which the authors believed could be influenced by nutrition and would in turn have considerable influence on national development were birth weight and intellectual development of preschool children. Given some quantification of the linkages between nutrition, birth weight and intellectual development, if only by subjective estimation, the authors could project some magnitude of the national income generated by increased earnings. Income is only one part of a multiple objective planning function; atten-
tion would also the beneficiaries
ECONOMIC
1180
removed
as
a
limiting
LONGHURST
AND
CALL
by giving
final
estimates,
factor
scores
of 5-year-olds
were
the
following
submitted
2)
preschool
3)
The Tables
ond
rized
in the form
central
value
tions
of degree
along was
with
in
sent
the
following
Results
from
panel
their
were
was and
and
third
round
round
rounds.
in total to assess
There
in values between
the of the imof incidence IQ scores of were
upper
was
a slight
range. The respondents the degree of confidence
summa-
estimate.
regard
on
It
to
is
the
sources
from
dietary to consider and how
government
Their
on pregnancy
would level.
in
effect”
where
the elements efficiently the
provide There
reis obvi-
some feel that maternal and
outcome” Incidence of low birth weight in target group after 6 years on program (given a present rate of
Intervention
program
I 5%) Median value %
Income
maintenance
Supplemental
12.5
range
% II -15
13.5 10
12.5-14 9-10
10-15 7-13
12 14
10- 13 1 2- 14 12.5-14
8- 14 10- 15 12-15
14
12.5-14
10-15
care:
Comprehensive
health
Prenatal
only
clinics
care
12.5
Visitingnurse
Abortionfacilities a
12-14
Total
feeding:
Viagivenprogramdesign Puredietaryeffect Prenatal
lnterquartile range %
No.
=
15 (3 of the
respondents
did
not
complete
the
third
round.
See
Table
I.)
to
disappointing
ously clear disagreement when 6 years of a comprehensive
could
Delphi
consensus is disappoint range within which
“pure
mechanism
particular
third
development.
especially
not have design
members
the
intellectual
experts did of program
summary
so they
values on
The lack of scientific ing, given the small
This
estimates.
two
questionnaire
panel
members
individual
questionnaire
3. for sec-
2 of third
the
2 and
are not presented
virtually no change interquartile range
third
of
in Tables
percent of the respondents rated their predictions as “pretty good” (value of 4) or “very sure” on the third Delphi questionnaire on birth weights. Thirty-one percent registered
of the median
of consensus).
to all
reconsider TABLE
population
comments
the
presented
for the second
reduction were asked
value (as the representative of the group), the range and total range (as indica-
interquartile
cases,
questionnaire.
they had in their answers (on a scale of I for “sheer guess” to 5 for “very sure”). Fifty-one
education
the
of
are
as there medians
programs
forms: a) local day care centers b) education centers c) parental education The second-round estimations pact of these programs in terms of low birth weight and average in
results
experiment
three
5-year-olds
in some
third-round
Results
proposed: income maintenance supplemental feeding
1)
revised
on the
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
all pregnant women in the group the optimum diet for the most successful pregnancy outcome 3) prenatal care at the following levels of Intensity of care and expenditure: a) comprehensive health care b) prenatal clinics only c) visiting nurse system 4) an abortion program For the Delphi assessing changes in IQ
NUTRITION
TABLE
PROGRAMS
AND
ECONOMIC
1181
PLANNING
3
Results
of third
questionnaire
Income
intellectual
development” Average IQ score of 5-year-olds in target group after 6 ars on program (given a present score of85 points)
program
maintenance
Supplemental
feeding
Median
Interquartile
value
range
Total
range
90
89-90
88-100
90
90-92
87-.
93
90-95 9395
87-95 9(J95
86-95
85-97
1 10
Preschool
programs: Local day care centers Education centers Parental education a
No.
=
10 (2 of the
93_95b
91 respondents
did
not
complete
the
child health program will have no effect on the incidence of low birth weight of babies born to low-income mothers and others feel that it will totally remove the differential in low birth weight that exists between poor and nonpoor groups in the United States. In the case of the respondents who predicted no effect for this program and the one who predicted substantial effect, both cited personal experience with identical implemented programs as the basis for their predictions. Use of the estimates and Delphi method in nutrition
the planning
third
round.
research, definite making that the developed ment in
In
value
of the
Delphi
in this
situation
rests on the judgment that it is the best possible technique in these circumstances. We would very much like to be able to refer to studies that tell us rationally the relationship between nutrition and say, birth weight. This information could then be applied in the context of planning better programs in the area of pregnancy care. But we also have to recognize that choices have to be made between alternative approaches whatever the state of knowledge because resources are scarce. Since choices have to be made, it is therefore vital that we understand as far as possible the premises under which they are made. If a Delphi consisting of experts does
not
provide
information
for
making
invest-
ment choices, then someone else, probably far less well informed, will make the allocative decision. Although the technique may seem unsatisfactory vis-a-vis conventional nutrition
Table
I.)
b
Bimodal.
from this point of view improvement on the present process. Experience to date Delphi is the best technique for using expert opinion the most controlled manner.
this
experiment
we
believe
Delphi was a success as all relevant and con the impact of nutrition Serious arguments and convincing were produced by respondents to positions they took. All took the very seriously and it provided means for knowledgeable people
The The
See
authors
makes present
The
would
contend
it is a decisionindicates currently and judg-
that
this
method
for a considerable means of collective
advance over decision-making.
study
the
showed
that
the
points pro were raised. evidence support the experiment a valuable to interact.
Delphi
the
tech-
nique could be an effective means of combining practical programs, political views and scientific reasoning in reaching a consensus in an area involving the application of scientific research to public policy. The confidence ratings form a useful means, particularly in a large group, of assessing the state of the art.
If no
clear
consensus
is derived,
then
this
highlights the situation for the policymaker and scientist, calling for either more research
that
is directly
realization constraints
How
policy
related
and
a greater
on the part of the researcher faced by the policymaker.
can
these
results
be used
of the
to improve
the allocation of resources for nutrition planning? The methods of economic project analysis, although extremely useful and successfully applied in many areas, cannot be used
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
Intervention
on
LONGHURST
1182
AND
with confidence in an area such as nutrition planning (as a means of selecting between programs), where it is very difficult to quanbenefits.
However,
the
technique
still
remains useful as a means of sharpening the judgment of the policymaker. Assessment of the magnitude of benefits is necessary in order to compare them with the costs, all in the same unit. The programs with the highest ratio
of benefits
to costs
naturally
should
be
those that are funded. The economic benefits of nutrition programs in this case would result from more people entering the earning stream as a consequence of decreased mortality
caused
babies,
by a decrease and, in the case
IQ
scores,
by
a greater
by greater educability.
in low birth-weight of improvements
earning
caused
If somewhat
tenu-
factors.
2.
3.
ously and unrealistically, it is possible to quantify linkages between IQ scores, years of school and later earning capacity. Human capital valuations of this nature are not acknowledged to be a particularly useful
4.
basis
6.
for decisions
on government
health
and
education programs such as those proposed in the Delphi questionnaires. They generally provide the conclusion that less should be spent on the poor than is spent on the rich, since the poor earn less money. However, the income generation is only one facet of program evaluation; only one part of the value of
a
References I.
in
power
a program. Attention to factors such as equity, regional economic growth and environmental quality is also important. Unfortunately, the economists and planners have yet to develop a methodology which objectively assesses the tradeoffs between these
5.
7.
8.
BERG, A., N. S. SCRIMSHAW ANt) D. L. CALL (editors). Nutrition, National Development and Planning. Cambridge: Mass. Inst. Tech. Press, 1973. MCNAMARA, R. S. Address to the Board of Governors. International Bank for Reconstruction and Development. Washington, D.C., 1971. PILL, J. The Delphi method: substance, context, a critique and an annotated bibliography. Socio-Economic Planning Sciences 5: 55-71, 1971. CAMPBELL, R. M. A Methodological Study of the Utilization of Experts in Business Forecasting. Ph.D. Dissertation. Los Angeles: Univ. ofCalifornia, 1966. DALKEY, N. C. The Delphi Method: An Experimental Study of Group Opinion. Rand Corporation. RM-5888-PR, 1969. HELMER, 0. Analysis of the Future: The Delphi Method. Rand Corporation, paper P-3558, Santa Monica, March 1967. WUNDERLE, R. E. Evaluation of the Pilot Food Certificate Program. Ph.D. Dissertation. Ithaca, N.Y.: Cornell Univ., 1971. LONGHURST, R. W. An Economic Evaluation of Human Resource Programs with Respect to Pregnancy Outcome and Intellectual Development. MS. Thesis. Ithaca, N.Y.: Cornell Univ., 1971.
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
tify
CALL
W.
consensus, planning Longhurst,’
MS.
nutrition
and
David
L. Call,2
The concept of nutrition as a crucial determinant of national development is a subject under considerable discussion at the moment. The conference at Massachusetts Institute of Technology on Nutrition, National Development and Planning in October 1971 (1) and the proposed action of the World Bank to designate
nutrition
programs
as
areas
for
investment are indications of this growing realization (2). In a larger context, those concerned with allocating resources for national development indicate that rather than concentrate on maximizing economic growth, more attention should be directed to the characteristics of the individuals affected by economic development. The following cornments about the role of nutrition in national development should be viewed from the perspective that if we are to be concerned with redirecting development efforts more accurately to the poorer elements in society, nutrition programs are one means by which this could be achieved. The question with respect to the development of a nation is two-layered: What is the trade-off between the magnitude of development benefits and their pattern of distribution? What is the impact of nutrition as a component in this predicament? In seeking answers to specific questions about nutrition programs, the planner faces a dilemma. Although prima facie nutrition must have some impact on disease resistance, labor productivity, reproductive capacity, intellectual development and receptivity to education, the exact relationship is unknown. In other words, the production, which relates inputs (nutrition in this case) to outputs (say, labor productivity), has not been quantified. The
American
Journal
of Clinical
Nutrition
programs
28: OCTOBER
article
and
Ph.D.
Some planner
How
of
the would
would
improved
productivity,
What
important like to e.g.,
sort
questions which a have answered are:
nutrition in
of impact
affect
agricultural
would
labor
activities?
changes
in nutri-
tion have on birth weight and on infant mortality? How does nutrition affect intellectual development in preschool years? How much would the provision of a school lunch affect learning ability? These are all questions that represent one link in assessing the.role of nutrition in the development process. Questions concerning the concept of adequacy of diet and the interrelationships of nutrition and parasitic diseases need to be answered to provide subsidiary information in the planning process. Still another step in the progression brings us to questions concerning the use of certain indicators to measure change in factors on which nutrition has an impact, e.g.,
IQ for intellectual for
pregnancy
diseases
and
development,
birth
weight
outcome,
incidence of certain status of health of a population of work output for labor produc-
for
volume
tivity.
Blame strength tion and 1
Graduate
for
lack
of the various School
of understanding
relationships parameters of Arts
and
of the
between of human Social
Studies,
nutriperSussex
University, this author Agricultural
England. This research was carried out while was a graduate student in the Department of Economics and Graduate School of Nutrition at Cornell University, Ithaca, New York. 2 Director of Cooperative Extension, New York State College of Agriculture and Life Sciences and New York State College of Human Ecology, Cornell University, Ithaca, New York. This research was carried out while this author was the H. E. Babcock Professor of Food Economics in the Graduate School of Nutrition at Cornell University.
1975, pp. 1 177-1 182. Printed
in U.S.A.
I 177
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
special
LONGHURST
1178
formance nutritionists important not
should not be accorded to the engaged in research. Although work still needs to be done, society
organized
that
so
the impact
of nutri-
tion on humans can be isolated as a single variable in controlled longitudinal studies. Efforts to translate the results of nutrition research into operational programs that can
be
subsequently
appraised
and
evaluated
alongside other programs have been trated not only by the failure ofeconomists planners and nutritionists to understand others’ work, but also by a disappointing
of agreement
between
nutritionists
frusor each lack
as to pri-
orities for investment of very limited resources. This paper documents an investigation that tried to derive a consensus of opinion among nutritionists, pediatricians
and
members
possible
of
changes
associated
disciplines
on
in
parameters
of
certain
human performance in response to an investment program that could be fitted into a planning model. It is reasonable to state that no published research gives a definite indication as to how much a change in nutrient intake weight, ductivity. Use
will affect educability,
such parameters morbidity or
of subjective
as work
birth pro-
estimation
In these circumstances, the authors decided to experiment with the Delphi technique (3). This method calls for interaction of a group of experts under controlled conditions in an attempt to obtain a consensus on certain quantitative estimations. The Delphi technique is carried out by a succession of mailed questionnaires. This is done to eliminate the unfavorable “personality” effects inherent in
a consensus reached by roundtable discussion where certain individuals may impose their views
by virtue
or vocal have
of greater
ability.
shown
that
that could later more accurate discussion. A typical
in the
Delphi
experts
is
and
(4, 5)
estimation
of values
the Delphi was a than roundtable
invites
a group
to estimate subject
standing
investigators
be verified, mechanism
knowledged uncertain
academic
Several
to many
a value
of
ac-
which
interpreta-
tions. In cases where there is no means of obtaining a “true” value, the use of subjectiv-
CALL
ity in the form of expert opinion is a definite improvement over no value at all. The ranges of the estimates obtained from the experts in the first round of the questionnaire, together
with
the
median
representative
experts
with
naire. They estimations is especially the extremes is preserved
to five sive
the second
(the
rounds
most
is sent
round
to the
of the question-
of questionnaires. ambiguous
and
important
Delphi
On
succes-
questions
are
information
as necessary. Of course, derive a consensus.
The
statistic
opinion)
are asked to reconsider their in the light ofgroup opinion. This important for those who are at of the distribution. Anonymity at all times and there may be up
rounds
phrased
value
of group
it is not
technique
is not
re-
inserted necessary
without
to
diffi-
culties but these can be minimized with efficient administration. Some of the criticisms mentioned by Helmer (6) are instability of panel membership, time lapse between questionnaires, ambiguous questions, respondents’ competence, substantive breadth versus specialization and consensus by undue averaging. This latter criticism is unjustified
on the basis
that
the
respondent
who
main-
tains an extreme position should be able to convince the others ofthe rationale behind his estimate. It is not yet known how much of the convergence that takes place is induced by the process itself rather than by elimination of the basic causes of disagreement, although some evidence suggests that most of the change in the median is caused by those unsure of their answers (3).
The computerization removed
many
for the
large
of the Delphi
of these
scale
criticisms,
experiment;
has now especially
it can
be an
efficient technique retaining its important characteristics of anonymity, controlled feedback and the important end product of a consensus response.
The
important
questionnaire
that
the former
difference and
between
a survey
attempts
a Delphi
questionnaire
to draw
out from
is
the
respondents the underlying premises for their opinions. Hence, given the initial assumption that in a particular situation benefits can be derived from the utilization of subjective estimations from experts, then the Delphi can be considered the “best” means for obtaining
such
estimates.
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
is
AND
NUTRITION
The
Delphi
PROGRAMS
AND
TABLE Occupations
experiment
have to be directed were.
to whom
Some description or specification of a target group is necessary when a group of experts is asked to estimate the impact of nutrition programs on birth weight and intellectual development. A survey carried out by the Graduate School of Nutrition at Cornell University (7) to assess the impact of the USDA Food Certificate Program produced a considerable amount of background material
on the
socioeconomic
status
of a group
of
low-income mothers in a depressed urban area. Hence it was proposed that the panel should be asked to predict the effect of certain programs in the areas of nutrition, medical care and preschool education on this target group within a specified time period. In the area of pregnancy outcome (birth weights), 24 experts in the field of pregnancy, nutrition and medical care were invited to participate in the experiment. They were told that only 15
mm of their
time needed
to be devoted
to each
questionnaire. Eighteen subsequently agreed to take part. In the area of intellectual development, 22 experts with interest in child psychology, development and nutrition were invited to participate and 12 agreed to do so.
A breakdown
of the subject-matter
interests
of these experts is given in Table 1. The first round of the Delphi mailed early in 1971 attempted to sort out the relative importance of environmental components that might be maniptilated by the introduction of a government program. The respondents were asked to rate (with respect to the importance of the target group) the effect such factors as age of the mother at delivery, number of previous pregnancies, extent of
PLANNING
I of respondents
Pregnancy
.4)
outcome
Epidemiologist Obstetrician Neonatalogist Pathologist Pediatrician Four
of the in nutrition.
development
Occupation
2 2 5 1 I 7
Nutritionist
a
B) Intellectual No.
Occupation”
interest
I179
No.
Pediatrician Nutritionist Sociologist Psychologist
nonnutritionists
4 5 2 I
expressed
a
strong
mother’s education, and contact with prenatal services would have on birth weight. The other group of respondents were asked to rate the effect of factors such as presence of the father, family income level and quality of housing conditions on performance of 5-yearolds on an IQ test. (We do not intend to enter the controversy over the usefulness of IQ scores as a measure of intelligence. Although we recognize its limitations, it represents the best quantitative measure developed so far.)
On round,
the basis of the results the second questionnaire
of this first proposed
possible intervention programs based on their administrative feasibility and on the incorporation of these components thought to be important from responses to the first-round questionnaire. The respondents were supplied with such information on the target group as income level, medical care utilization, and the like. Specific programs were described and the respondents in one group asked to estimate the impact which 6 years of each program would have on the incidence of low birth weight (the survey having shown that this was 15%). The respondents in the second group were asked to estimate the impact of 6 years of each program on the average IQ score of 5-year-old children (having assumed for the purposes of this study that the average
IQ score
of 5-year-olds
Descriptions available on
are
available
was then
of the proposed request for those
For
the
Delphi
the
follow-
fect
design
“program,”
(8).
are and
assessing changes in birth weight, ing programs were proposed: 1) income maintenance 2) supplemental feeding either
program
elsewhere
at 85 points). programs interested,
or by a pure i.e.,
nutrition
by a given dietary efwould
be
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
Two factors which the authors believed could be influenced by nutrition and would in turn have considerable influence on national development were birth weight and intellectual development of preschool children. Given some quantification of the linkages between nutrition, birth weight and intellectual development, if only by subjective estimation, the authors could project some magnitude of the national income generated by increased earnings. Income is only one part of a multiple objective planning function; atten-
tion would also the beneficiaries
ECONOMIC
1180
removed
as
a
limiting
LONGHURST
AND
CALL
by giving
final
estimates,
factor
scores
of 5-year-olds
were
the
following
submitted
2)
preschool
3)
The Tables
ond
rized
in the form
central
value
tions
of degree
along was
with
in
sent
the
following
Results
from
panel
their
were
was and
and
third
round
round
rounds.
in total to assess
There
in values between
the of the imof incidence IQ scores of were
upper
was
a slight
range. The respondents the degree of confidence
summa-
estimate.
regard
on
It
to
is
the
sources
from
dietary to consider and how
government
Their
on pregnancy
would level.
in
effect”
where
the elements efficiently the
provide There
reis obvi-
some feel that maternal and
outcome” Incidence of low birth weight in target group after 6 years on program (given a present rate of
Intervention
program
I 5%) Median value %
Income
maintenance
Supplemental
12.5
range
% II -15
13.5 10
12.5-14 9-10
10-15 7-13
12 14
10- 13 1 2- 14 12.5-14
8- 14 10- 15 12-15
14
12.5-14
10-15
care:
Comprehensive
health
Prenatal
only
clinics
care
12.5
Visitingnurse
Abortionfacilities a
12-14
Total
feeding:
Viagivenprogramdesign Puredietaryeffect Prenatal
lnterquartile range %
No.
=
15 (3 of the
respondents
did
not
complete
the
third
round.
See
Table
I.)
to
disappointing
ously clear disagreement when 6 years of a comprehensive
could
Delphi
consensus is disappoint range within which
“pure
mechanism
particular
third
development.
especially
not have design
members
the
intellectual
experts did of program
summary
so they
values on
The lack of scientific ing, given the small
This
estimates.
two
questionnaire
panel
members
individual
questionnaire
3. for sec-
2 of third
the
2 and
are not presented
virtually no change interquartile range
third
of
in Tables
percent of the respondents rated their predictions as “pretty good” (value of 4) or “very sure” on the third Delphi questionnaire on birth weights. Thirty-one percent registered
of the median
of consensus).
to all
reconsider TABLE
population
comments
the
presented
for the second
reduction were asked
value (as the representative of the group), the range and total range (as indica-
interquartile
cases,
questionnaire.
they had in their answers (on a scale of I for “sheer guess” to 5 for “very sure”). Fifty-one
education
the
of
are
as there medians
programs
forms: a) local day care centers b) education centers c) parental education The second-round estimations pact of these programs in terms of low birth weight and average in
results
experiment
three
5-year-olds
in some
third-round
Results
proposed: income maintenance supplemental feeding
1)
revised
on the
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
all pregnant women in the group the optimum diet for the most successful pregnancy outcome 3) prenatal care at the following levels of Intensity of care and expenditure: a) comprehensive health care b) prenatal clinics only c) visiting nurse system 4) an abortion program For the Delphi assessing changes in IQ
NUTRITION
TABLE
PROGRAMS
AND
ECONOMIC
1181
PLANNING
3
Results
of third
questionnaire
Income
intellectual
development” Average IQ score of 5-year-olds in target group after 6 ars on program (given a present score of85 points)
program
maintenance
Supplemental
feeding
Median
Interquartile
value
range
Total
range
90
89-90
88-100
90
90-92
87-.
93
90-95 9395
87-95 9(J95
86-95
85-97
1 10
Preschool
programs: Local day care centers Education centers Parental education a
No.
=
10 (2 of the
93_95b
91 respondents
did
not
complete
the
child health program will have no effect on the incidence of low birth weight of babies born to low-income mothers and others feel that it will totally remove the differential in low birth weight that exists between poor and nonpoor groups in the United States. In the case of the respondents who predicted no effect for this program and the one who predicted substantial effect, both cited personal experience with identical implemented programs as the basis for their predictions. Use of the estimates and Delphi method in nutrition
the planning
third
round.
research, definite making that the developed ment in
In
value
of the
Delphi
in this
situation
rests on the judgment that it is the best possible technique in these circumstances. We would very much like to be able to refer to studies that tell us rationally the relationship between nutrition and say, birth weight. This information could then be applied in the context of planning better programs in the area of pregnancy care. But we also have to recognize that choices have to be made between alternative approaches whatever the state of knowledge because resources are scarce. Since choices have to be made, it is therefore vital that we understand as far as possible the premises under which they are made. If a Delphi consisting of experts does
not
provide
information
for
making
invest-
ment choices, then someone else, probably far less well informed, will make the allocative decision. Although the technique may seem unsatisfactory vis-a-vis conventional nutrition
Table
I.)
b
Bimodal.
from this point of view improvement on the present process. Experience to date Delphi is the best technique for using expert opinion the most controlled manner.
this
experiment
we
believe
Delphi was a success as all relevant and con the impact of nutrition Serious arguments and convincing were produced by respondents to positions they took. All took the very seriously and it provided means for knowledgeable people
The The
See
authors
makes present
The
would
contend
it is a decisionindicates currently and judg-
that
this
method
for a considerable means of collective
advance over decision-making.
study
the
showed
that
the
points pro were raised. evidence support the experiment a valuable to interact.
Delphi
the
tech-
nique could be an effective means of combining practical programs, political views and scientific reasoning in reaching a consensus in an area involving the application of scientific research to public policy. The confidence ratings form a useful means, particularly in a large group, of assessing the state of the art.
If no
clear
consensus
is derived,
then
this
highlights the situation for the policymaker and scientist, calling for either more research
that
is directly
realization constraints
How
policy
related
and
a greater
on the part of the researcher faced by the policymaker.
can
these
results
be used
of the
to improve
the allocation of resources for nutrition planning? The methods of economic project analysis, although extremely useful and successfully applied in many areas, cannot be used
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
Intervention
on
LONGHURST
1182
AND
with confidence in an area such as nutrition planning (as a means of selecting between programs), where it is very difficult to quanbenefits.
However,
the
technique
still
remains useful as a means of sharpening the judgment of the policymaker. Assessment of the magnitude of benefits is necessary in order to compare them with the costs, all in the same unit. The programs with the highest ratio
of benefits
to costs
naturally
should
be
those that are funded. The economic benefits of nutrition programs in this case would result from more people entering the earning stream as a consequence of decreased mortality
caused
babies,
by a decrease and, in the case
IQ
scores,
by
a greater
by greater educability.
in low birth-weight of improvements
earning
caused
If somewhat
tenu-
factors.
2.
3.
ously and unrealistically, it is possible to quantify linkages between IQ scores, years of school and later earning capacity. Human capital valuations of this nature are not acknowledged to be a particularly useful
4.
basis
6.
for decisions
on government
health
and
education programs such as those proposed in the Delphi questionnaires. They generally provide the conclusion that less should be spent on the poor than is spent on the rich, since the poor earn less money. However, the income generation is only one facet of program evaluation; only one part of the value of
a
References I.
in
power
a program. Attention to factors such as equity, regional economic growth and environmental quality is also important. Unfortunately, the economists and planners have yet to develop a methodology which objectively assesses the tradeoffs between these
5.
7.
8.
BERG, A., N. S. SCRIMSHAW ANt) D. L. CALL (editors). Nutrition, National Development and Planning. Cambridge: Mass. Inst. Tech. Press, 1973. MCNAMARA, R. S. Address to the Board of Governors. International Bank for Reconstruction and Development. Washington, D.C., 1971. PILL, J. The Delphi method: substance, context, a critique and an annotated bibliography. Socio-Economic Planning Sciences 5: 55-71, 1971. CAMPBELL, R. M. A Methodological Study of the Utilization of Experts in Business Forecasting. Ph.D. Dissertation. Los Angeles: Univ. ofCalifornia, 1966. DALKEY, N. C. The Delphi Method: An Experimental Study of Group Opinion. Rand Corporation. RM-5888-PR, 1969. HELMER, 0. Analysis of the Future: The Delphi Method. Rand Corporation, paper P-3558, Santa Monica, March 1967. WUNDERLE, R. E. Evaluation of the Pilot Food Certificate Program. Ph.D. Dissertation. Ithaca, N.Y.: Cornell Univ., 1971. LONGHURST, R. W. An Economic Evaluation of Human Resource Programs with Respect to Pregnancy Outcome and Intellectual Development. MS. Thesis. Ithaca, N.Y.: Cornell Univ., 1971.
Downloaded from https://academic.oup.com/ajcn/article-abstract/28/10/1177/4716644 by University of Michigan- Flint user on 11 January 2019
tify
CALL
