A Bayesian Approach To Health Project Estimation JOHN DENNIS CHASSE, PHD
Abstract: This paper illustrates how a Bayesian statistical approach was used to estimate the outcomes of the National Tuberculosis Program in India. Such an estimate, it is argued, is necessary for a properjudgment about a project's social usefulness. The process of medical care delivery is reduced to a set of conditioned probabilities. The numbers are estimated using as source material medical records, the results of medical research, and the opinion of experts.
Bayesian methods of estimation are used and their value is discussed. The final discussion contains a brief treatment of the role of project analysis in public decision making. The place of Bayesian methods in project analysis is briefly illustrated, demonstrating their operational value in the field of public health decision making. (Am. J. Public Health 66:747-754, 1976)
A rational decision to spend the taxpayers' money on a public health project demands a prior estimate of its social desirability. A perfect estimate would include an accounting of the resources used, a model linking these resources to all socially relevant outcomes. and a ranking of these outcomes in an order of social desirability. The model linking resources and outcomes deserves attention for a number of reasons. First, as this paper hopes to show. there is more of a problem here than meets the eye. Second. the problem is conceptually soluble. It does not suffer from the value dilemmas which plague the construction of a social preference function. Finally, the problem is important. If nothing else is possible. a rational decision should at least rest on a good estimate of the program's ability to do what it claims to do. A Bayesian approach to developing such an estimate is advocated here. This approach will be used to develop a "best" estimate of the output of the Indian District Tuberculosis Program. This estimate will be accompanied by a measure of uncertainty.* The importance of in-
cluding a measure of uncertainty is gaining recognition in the field of public project appraisal.' Part I of the following discussion tries to clarify the ,,output problem" and to show how Bayesian methods can help solve it. Part II describes the Indian National Tuberculosis Program and uses a decision tree to organize the problem. Part III contains an explanation of the way in which the tree was set up as well as an estimation of probabilities. Part IV discusses policy implications.
1. The Output Problem in Health Program Analysis Arrow defined the output of a medical service from the
point of view of one person as the reduction in the likelihood of sickness and death and its concomitant suffering.2 Output
fessor of Economics. Savannah State College. and Public Health Consultant to the U.S. DHEW Office of International Health. This paper, submitted October 28. 1975. was revised and accepted for publication in the Journal February 12, 1976.
from the point of view of the public decision maker. however, would be in aggregate terms-the number cured or saved from sickness and death. The problem is that there is no conceptually adequate way to directly measure output in this sense. A record of the number of people cured would not do. as it would omit those cured in the program's absence, and thus overestimate the program's output. But most health systems. even in affluent countries do not even go this far. They record what econo-
*Consider two projects. The estimated outcome for project A is exactly the same as the estimated outcome for project B. But the estimate for project A has only a 10 per cent chance of being wrong while that for project B has a 50 per cent chance of being wrong. Obviously project A should be preferred over project B if the ex-
pected outcome is socially desirable. For this reason, such classical uncertainty measures as variances and confidence intervals are considered important in a rational decision process.
Address reprint requests to Dr. John D. Chasse. 502 Mt. Vernon Place. Apt. 4. Rockville. MD 20850. Dr. Chasse is Assistant Pro-
AJPH August, 1976, Vol. 66, No. 8
747
CHASSE
mists call inputs or intermediate goods-hospital days, patients treated, admissions, and releases-and somehow the analyst must move from this information to inferences about the reduction in sickness and death. In a poor country there is an additional problem. Few such countries can afford a health intelligence system adequate even by medical standards. In fact, anyone trying to study anything in a poor country hears time and again that practically any statistic he finds is useless. Admitting this, he must decide what to do next. There are three possible approaches. The first rejects statistical analysis and depends on the opinion of individuals with long experience in the field. This may provide the best present estimate, but it does not create statements that can be refuted by later investigation. If the forecast proves wrong, it is impossible to find out what assumption was wrong or which causal link was incorrect. The second approach ignores the problem and proceeds to apply the whole gamut of classical statistical methods to the existing data. There are two problems with this procedure. First, an unjustified air of objectivity is maintained; second, a certain dishonesty creeps in. Somehow, the final report fails to mention that the mean was developed from three observations, or that the regression analysis had six degrees of freedom. Finally, whatever evidence exists in the opinions of people with long experience is ignored. There is a third way-not totally satisfactory, but flexible enough to avoid the problems mentioned above. Bayesian statistical techniques can create a measure of output in a way that lays all the assumptions open to refutation, but also permits the use of expert opinion where recorded numbers are nonexistent or hopelessly inadequate.
II. The Tuberculosis Program and the Shape of the Analysis Every district in India has a network of hospitals and health centers. In addition, the residents of the main town usually have access to a TB clinic. The quality of service and facilities varies at these clinics. The average operating cost is about Rs. 20,000.3 There is no coordination of TB clinic activities with those of the hospitals and health centers. This situation will be referred to as the status quo (SQ). The District Tuberculosis Program (DTP) turns the clinic into a district TB center with the task of coordinating all TB activities in the district. Establishing the DTP involves equipping the tuberculosis center with an X-ray machine and staffing it with a treatment organizer, a laboratory technician, an X-ray technician, a medical officer, and a BCG team.3 Capital costs are estimated at Rs. 210,000, recurring costs at Rs. 150,000.' The present analysis is limited to the case-finding and treatment activities of the DTP. To go further would make the discussion too long and complex. 748
Organizing the Problem Decision trees have been used to analyze medical decisions, and they have been used to analyze public expenditure decisions.6-'0 The use here is not very sophisticated, as the decision tree is merely used to organize the problem. The decision to expand or contract the DTP can be reduced to one of expanding it into one more district. This is close to the economist's notion of marginal product if the whole district unit is treated as a variable, the national superstructure as a fixed input, and costs and outputs are assumed to be the same for each district unit. These assumptions simplify the present treatment, and their denial would not particularly invalidate the method. Figure 1 illustrates this process. The box on the left side of the figure represents a decision fork. Here the decision maker has control. If he decides not to expand the program then he chooses the lower path and the present system continues unaided by the DTP. If he does decide to expand, then he chooses the upper path. Following this path, one comes to a circle which represents the probability that the victim will be found assuming the existence of the DTP. Here chance reigns. The symbol P(cflDTP) means the probability that the individual will be found given that the DTP exists.* Finding is not curing, however. The treatment must continue, and it must work. So there are two other probability forks. By multiplying the probabilities one can calculate the joint probability that all events will occur-that the victim will be found, will take his medicine, and will be cured. This is written at the end of the line P(cf,T,CIDTP). By adding the joint probabilities at the end of the lines, it is possible to calculate the total likelihood of a cure. The same process is blocked out for the mirror image at the bottom of the diagram which will give the probability for a cure under the status quo. In a similar manner, the likelihood of sickness and death under the two different assumptions can be calculated. and once this is done it will be possible to give a decision maker an estimate of uncertainty involved for the whole population covered.
III. Justification and Estimation of the Probabilities This section explains why the decision tree was set up in this way and it estimates the probabilities. It will be easier to *Although the symbols are explained in the diagram. it might help to have them all listed. DTP = The District Tuberculosis Program SQ = The Status Quo P(cf) = The probability that the individual will be found P(BP) = The probability of being bacteriologically positive (sick) P(T) = The probability of continuing treatment P(C) = The probability of a cure P(D) = The probability of death P(ncf) = The probability of not being found by the system P(NT) = The probability of not continuing treatment AJPH August, 1976, Vol. 66, No. 8
BAYESIAN OUTCOME ESTIMATION
follow if the three steps are kept in mind: (1) The individual is found; (2) He continues treatment; (3) He is cured. The different combinations of these probabilities produce the decision tree. Probabilities must be calculated both for the DTP and for the SQ.
Likelihood of Finidinig the Individual
TABLE 1-Estimate of the Prevalence of Tuberculosis (Sputum Positive Cases per 1,000) Using Three Studies. 10-12 Studies
Measure
Madanappalle and NSS
Madanappalle, NSS and Tumkur
Mean Variance
2.99 0.0824
2.99 0.06197
Meaning and Importance of the Likelihood Many forces keep people in poor countries from approaching health facilities-distance, folk beliefs. ignorance of the fact that doctors and nurses are able and willing to help them, etc. Getting a patient to approach a health center or hospital is. however, only half the problem. The staff of such a facility is so loaded with work that. in the few minutes they can spare, they are likely to substitute a cursory guess for a diagnosis. hand the individual some pills. and send him on his way.
The Estimation of the Probabilities Let N = the number of sputum positives discovered in a distnct V = the number of actual TB victims N/V = the fraction of TB victims found The problem is reduced to estimating the mean and variance of N/V. The number of victims (V) can be found by multiplying the prevalence rate (R) by the population (P). Estimatindg the Mleatt and Variance of tle Prevalence: Three studies were used. The earliest (Madanappalle) was supervised by J. Frimodt-Moller in 1955 in South India.' The second (NSS) was the National Sample Survey.1I The last (Tumkur) was the Indian Council of Medical Research Survey in Tumkur District in South India.1 Since Moller's study is the earliest. his estimates create an expectation which is either confirmed or corrected by the later studies. These three studies are combined by continuous revision.* Table I presents the results. Estimatintg the Meant(iad Variance of lthe Number of Ctases in a District: Goodman has rigorously derived a formula for estimating the mean and variance of a product.'3 The National Sample Survey found that the prevalence of TB was statistically independent of urban or rural residence. of geographical location, and of district size. This permits the use of formulas assuming statistical independence. Where E(X) = the
*Let Mr MO Vo Mn Vn
= the revised mean, Vr = the revised variance = the mean of the first or prior distribution and = the variance of the first or prior distribution = the mean of the second or posterior distribution = the variance of the second or posterior distribu-
Then:
tion MR = (Mo/Vo) + (Mn/Vn)
(l/Vo) + (l/Vn) l/Vr = (l/Vo) + (l/Vn) This is rigorously proved in Thomas H. Wonnacott and Ronald J. Wonnacott, Introductory Statistics (New York: Wiley and Sons. 1969). pages 324-327. AJPH August, 1976, Vol. 66, No. 8
mean or expected value of X and V(X) equals the variance of X. the formulas are: (1) E(XY) = E(X)E(Y) (2) V(XY) = X2V(Y) + Y2V(X) + V(X)V(Y) Using average population figures from the 1970 census. the mean number of cases per district was estimated in this manner as 4861.74. the variance at 1207369. and the standard deviation at 1098.8. Es.ti,ma te of tlie Na tu,ber of Cases Foi)nid: Detailed records of the number of cases actually discovered in a district are unavailable. but summary statistics exist. Thus, it is possible to obtain means and ranges. but not to obtain the original data from which these were calculated. A range cannot be used in further- calculation. but a variance can be used. A formula devised for this type of situation approximates the standard deviation by dividing the range by six.14 Nair. citing quarterly reports of the DTP and the SQ. gives the mean number of cases diagnosed per quarter in districts where the DTP exists as 158.15 The range is between 143 and 190. Adjusting the figures to reflect an annual casefinding yield produces the following estimate for the DTP: Mean = 632 Range = 572-760 Standard Deviation = 42 Variance = 1764 The status quo estimate presented problems. The existing data lump together cases and suspects. Since both the DTP and the SQ draw from the same statistical population. it was assumed that the ratio of cases to suspects would be the same in both. After some testing for consistency. the following case-finding estimates were made for the SQ: Mean = 241 Range = 177-362 Standard Deviation = 40.3 Variance = 1624.09 Estitiniate of tlie Fraction N/V: The objective here is an estimate of the fraction of TB cases (N/V) that are found in a typical district. The number of cases existing in a district (V) has been estimated. The number found (N) has also been estimated. These are estimates or random v ariables. Both italicized words must be stressed. A simple ratio of the means would assume that the numbers were fixed, and would not produce a variance from which an estimate of uncertainty could be derived. There is no rigorously derived method for estimating this. Two methods of approximation were used (described in Appendix A). Both methods produced. for all practical purposes. the same estimate. The DTP mean is put at about 14 per cent and the SQ mean at about 5 per cent. 749
CHASSE
Probability that the Individual, Once Diagnosed, Will Continue Treatment for the Required Length of Time
TABLE 3-Patients in Rural Areas Completing Eighty Per cent or More of Their Drug Collection
Meaning of the Probability To be cured, the individual must return to the health center once a month to collect his pills. Even patients in sanatoria and hospitals often hide pills under their pillows; when quinine was prescribed for malaria, Indian soldiers had to be forced to take it in the presence of superior officer.'6 A peasant sometimes has a five or ten mile walk ahead of him as the private cost of taking the medicine.
Study
Estimation of the Probabilities Probability for the DTP: An apparent conflict in the data can be removed by assuming a higher probability of pill collection for urban than for rural areas. However, this makes it necessary to estimate additional probabilities.* Estimation of P(TIU): The consensus among medical men seemed to be that if an individual made 80 per cent of his collections, the drugs should have their effect. The probability that an individual would do so was calculated by pooling small sample studies. The estimates for three studies dealing with Urban patients are summarized in Table 2. A beta distribution was assumed here.** The resulting estimates w,ere as follows: Mean (expected value) = .6803 Mode (most likely value) = .6813 Variance = .000593 Two studies were combined to estimate the fraction of rural residents that continued to collect their pills. They are shown in Table 3. The mean and mode are both .34, and the variance is .22. TABLE 2-Drug Collection of Patients in Urban T.B. Clinics No. of Patients
Study
Total
Bangalorel 7 Dehli18 Dehli19
118
No. completing 80% or more collections
84
78 54
162
116
Patients collecting 80% or more of their pills
No. of Patients
Rural20
Hospital Tumkur21 Study
283
96
16189
5504
Estimation for the Status-Quo: D. R. Nagpaul, the director of the National Tuberculosis Programme, on the basis of drug usage, estimated that, for the SQ, 50 to 100 cases would complete treatment in a district in one year.2 Using the method discussed abovet, a midpoint of 75 was assumed as the mean, and the range between the most optimistic and pessimistic estimates was divided by six to yield a rough estimate of the standard deviation. Using the formula for the mean and variance of a quotient, and dividing by the number diagnosed in the status quo situation gave a probability of .3199. Since the variance was small calculated in this manner, the simple binomial variance of .220 was used. The method used here is admittedly arbitrary, but the arbitrariness is not hidden, and unlike other equally arbitrary methods, it makes a precise prediction which can be refuted, and points to the type of data needed to improve the accuracy of the final estimate. Estimation of Per cent Rural and Per cent Urban: After assessing evidence on the rural-urban mix of patients in a city health center, the relationship between drug collections and distance from the health center, the distance of the average rural resident from a health center, and the proportion of total cases accounted for by peripheral centers, the per cent rural was estimated at 28.1 and the per cent urban at 71.9 with a variance of 1 1.9 per cent.2 Probability of a Cure with and without Treatment: The effectiveness of treatment was estimated by combining a number of small studies using the same method as that used to find the probability of continuing treatment. The result was: P(CIT) = 0.9526 P(DIT) = 0.0474 Variance = 0.000213
The Probabilities Together *Urban assuming found, P(Ulcf); Continuing treatment assuming urban, P(TIU); Rural assuming found P(RIcf); Continuing treatment assuming rural P(TIR). **The beta distribution is used because of its flexibility. This is discussed in John D. Freund, Mathematical Statistics. Second Edition (Englewood Cliffs: Prentice-Hall, 1971). Chapters 4, 5 and 9. The method of estimation used here can be found in Samuel A. Schmitt, Measuring Uncertainty: An Elementarv Introduction to Bayesian Statistics (Reading, Mass.: Addison-Wesley, 1969), pp. 115-120. Let s = the number of successes, f = the number of fail-
Referring to Figure 1 again, the decision maker's position is represented by the box on the left side of the figure. The "decision fork" symbolizes his absolute control over whether the DTP will, in fact, be established in a particular district. The outcome from then on is subject to chance. For any particular TB victim, a number of events may or may not occur. The victim may or may not come into contact with the system. He may or may not continue to journey to the health
ures.
Mean = (s+l)/(s+f+2) Mode = s/(s+f) Variance = (s+ l)(f+ l)/(s+f+2)2(s+f+3) 750
tSee subhead "Estimate of the Number of Cases Found" AJPH August, 1976, Vol. 66, No. 8
BAYESIAN OUTCOME ESTIMATION
center faithfully every month. Finally, even if he takes the medicine, he has a slight chance of dying. By multiplying the probabilities previously calculated. and adding at the tips of the trees. an estimate was obtained of the likelihood of sickness and death for the random individual in an average district. The variance was calculated by applying the formula for the variance of a product over and over again. The calculations were done using an APL* program on a computer. and the results are shown in Tables 4. 5, and 6. Since the DTP is an on-going program the relevant policy questions concern expanding. contracting. or continuing the DTP. This could logically be reduced to the decision of adding one more unit. The public decision maker would be interested in the lives saved and sickness years averted by such a one district expansion. This can now be calculated. To keep the illustration simple. assume that those sick are sick for one year. The DTP would save about (.297 - .281)X4862 = 78 lives per year. The variance would be calculated as (.158 + .190)X4862 = 1692. This implies a standard deviation of 41. The number saved from sickness would be (.414 - .387)X4862= 131. Calculating the variance yields (.220 + .183)X4862 1959. and the standard deviation, again. would be 44. Both of these measures would imply a likelihood of more than 90 per cent that the prevention of death and sickness would exceed zero. The death figure with a z value of 1.9 just misses the 95 per cent significance level. TABLE 4-Estimates of Death, Sickness, and Cure on the Assumption that the DTP Exists in a District Probability Estimate
Fate of the Individual
Mean
Variance
Death Continued Illness Cure
.281 .387 .332
.158 .183 .210
IV. Discussion and Policy Conclusions Inadequate data and expert opinion have been combined to produce a quantitive estimate of the effect of a TB program on the health of the target population. The process of arriving at this estimate can be thought of as a set of refutable hypotheses. open to comment and correction. It would be hoped that in the actual decision-making process such corrections would be made. The technique produces an estimate of the effect of the program on lives and sickness. together with an estimate of uncertainty. This is not the only set of effects that should receive consideration, but other ef*APL-A Programming Language is a time-sharing language used with the IBM 360 computer. devised by Kenneth E. Iverson.23
AJPH August, 1976, Vol. 66, No. 8
TABLE 5-Estimates of Death, Sickness, and Cure Assuming that the Present Network of Health Facilities (SQ) Tries to Cope Without the Assistance of the DTP Probability Estimate Fate of the Individual
Mean
Variance
Death Continued Illness Cure
.297 .414 .289
.190 .220 .251
fects could be estimated in the same way or. alternatively. these data could be used as a basis for their estimation. The weaknesses and the broad policy implications of this technique are considered further in the sections which follow.
Aretas o lncomnpleteness i1n tile Pr-esenit Analysis Several changes could have added more realism to the decision tree. There is evidence that the cure rate is positively related to the degree of drug collection.24 A different death. cure and sickness probability could have been constructed for 40 per cent collections. 60 per cent collections. 80 per cent collections. 100 per cent collections. In place of a one year time span. a two or three year time span could have been substituted. The case-finding probability included an assumption of correct diagnosis. Another decision tree treatment actually has taken into account the likelihood of incorrect diagnosis." Finally. there is the likelihood of contributing to the spread of drug resistant strains of bacteria. There is always a trade-off between simplicity and realism in model building.2 5. 26 The degree of realism should be determined in part by the type of data available, in part by relevance, and in part by sensitivity analysis. Some complications may not affect the final result. In the present example. it was desired to keep the model simple enough to permit presentation. Time can be added by making the decision tree more complicated. or by using the decision tree results as input for an epidemiological model. This was the way time was included in a full analysis of the DTP.5 Some of the techniques used may disturb the rigorous scientist. Particularly open to objection is Wagle's method for estimating the standard deviation.'4 However, one must point out that the alternatives are to make no estimate and TABLE
5-Estimates of Death, Sickness, and Cure
on the Assumption that the Individual is Left to his Own Resources22 Probability Estimate
Fate of the Individual
Mean
Variance
Death Continued Illness Cure
.301 .421 .278
.210 .244 .201
751
CHASSE
ignore expert opinion or to use such opinion in a way that ignores uncertainty, or assumptions. The knowledge aimed at is not the scientific conclusion of a researcher. It is more analogous to clinical diagnosis where the best available techniques may still produce only partial understanding.
The Policy Implications ofProject Analysis Project analysis cannot remove all ambiguity, but it can reduce it. The DTP has not lived up to the hopes of its designers-reaching 50 per cent of the sputum positives.27 An editorial in the Indian Journal of Tuberculosis complained:28 It is common knowledge that for various reasons the National Tuberculosis Programme is not making satisfactory headway. even though it is a scientifically sound programme.
Project analysis can play two roles here. The first is to force explicit recognition of true criteria. Living up to expectations or not living up to expectations is not a rational criterion of project selection. Lives saved is one, sickness prevented another. Project analysis can also improve knowledge of a project's ability to fulfill rational criteria. In the present example, it has been shown that there is a good likelihood that the DTP "warts and all" is probably saving lives and reducing sickness. Whether it should be expanded depends on alternative uses of the funds to be devoted to it.
The Present Effort and Project Analysis. Significant progress has been made in developing models which will calculate the correct mix of activities given a welfare function and data on coefficients.29 3( The methods illustrated here would seem complementary to these efforts in the following ways: * Bayesian methods can be used to estimate coefficients for those models and to give some notion of the uncertainty inherent in poor data and necessary assumptions. * Project design often uses "idealized" coefficients. Bayesian methods can use actual program statistics after a program has been in operation to take account of less than ideal performance and to point out areas for further investigation and step by step improvement. * A Bayesian approach seems more adapted to the fact that public expenditure decisions, like medical diagnoses. cannot rest on perfect knowledge. Estimates are needed when the decision is made. The approach does not offer a total solution to the problem. but it would seem capable of contributing to the solution.
REFERENCES 1. Reutlinger, S. Techniques for Project Appraisal Under Uncertainty. Baltimore: Johns Hopkins Press, 1970. 2. Arrow, K. J. Uncertainty and the Welfare Economics of Medical Care. Essays in the Theory of Risk-Bearing. Kenneth J. Arrow Ed. Amsterdam: North-Holland Press. 177-219, 1971. 3. Nagpaul, D. R. District Tuberculosis Programme in Concept and Outline. Indian Journal of Tuberculosis 15: 186-198. 1967. 4. Sen. A. S. and Basu, R. N. Economics of health-cost of tu-
berculosis. Indian Journal of Tuberculosis. 18:146, 1972. 752
5. Chasse, J. D. The Cost-Benefit Analysis of Disease Control Programs in Developing Countries with Applications to the Indian National Tuberculosis Programme. Ph.D. Dissertation. Syracuse. N.Y.: Syracuse University. 1974. 6. Ginsberg. A. S. and Offenbach, F. L. An Application of Decision Theory to a Medical Diagnosis Treatment Problem. IEEE Transactions on Systems Science and Cybernetics. SSC-4:335362, 1968. 7. Simon, J. S. Toward formal evaluation of scientific research projects: a case study of population control. Policy Sciences. 3: 177-181, 1971. 8. Howard, R. A.. Matheson. J. E., and North, D. W. The decision to seed hurricanes. Science. 176:1191-1202. 1971. 9. Schweitzer, S. 0. The Economics of the Early Diagnosis of Disease. Evaluation in Health Services Delivery. Richard Yaffe and David Alkind, Eds. New York: Engineering Foundation. 184-196, 1973. 10. Moller. J. F. A community-wide tuberculosis study in a South Indian rural population. 1950-1955. WHO Bulletin. 22:61-170. 1960. 11. Indian Council of Medical Research. Tuberculosis in India: a sample survey. Special Report Series No. 34. Delhi: Indian Council of Medical Research. 1959. 12. Narain, R., Geser, A., Jambunathan. M. V. and Subramanian. M. Some aspects of a tuberculosis prevalence survey in a South Indian district. WHO Bulletin. 29:641-644. 1963. 13. Goodman, L. A. On the Exact Variance of Products. Amer. Stat. Assoc. J. 55:708-713. 1960. 14. Wagle, B. Statistical analysis of risk in capital investment projects. Operational Research Quarterly. 18:13-33, 1967. 15. Nair, S. S. Assessment and monitoring of the national tuberculosis programme. Indian Journal of Tuberculosis. 14:131-134. 1967. 16. Narain, R., Mayurnath. S., Subba Rao, M. S. and Murthy. S. S. F Feasibility of a chemoprophylaxis trial in India against tuberculosis. Indian Journal of Medical Research. 59:339-353. 1971. 17. Banerji, D. Behavior of tuberculosis patients towards treatment organization offering limited supervision. Indian Journal of Tuberculosis. 14:161. 1967. 18. Singh, M. M. and Banerji, D. Follow-up study of patients of pulmonary tuberculosis treated in an urban TB clinic. Indian Journal of Tuberculosis. 15: 160. 1968. 19. Mathur, K. C.. Singh, K. D.. Chakravarty. A.. Basu. N. and Kandelwal. M. L. Results of domiciliary treatment in an urban TB clinic, Indian Journal of Tuberculosis. 9:62. 1963. 20. Gothi G. D. and Baily. G. V. J. Problems of treatment of tuberculosis patients in rural areas. Indian Journal of Tuberculosis. 12:64, 1964. 21. Narain, R., Mayurnath. S.. Subba Rao M. S. and Murthy, S. S. Feasibility of a chemoprophylaxis trial in India. Indian Journal of Medical Research. 59:351-352, 1971. 22. Olakowski, T. Paper Presented in a Symposium on Chemotherapy of Tuberculosis. Madras: Tuberculosis Chemotherapy Centre, February, 1972. 23. Iverson. K. E. A Programming Language. New York: Wiley, 1962. 24. Pamra, S. P. and Mathur, G. P. Drug default in an urban community. Indian Journal of Tuberculosis. 14:199-203. 1967. 25. Raiffa. H. Decision Analysis: Introductory Lectures on Choices under Uncertainty. Reading, Mass.: Addison-Wesley, 1968. 26. Theil, H. Economic Forecasts and Policy. Amsterdam: NorthHolland Press. 1961. 27. Banerji, D. India's National Tuberculosis Programme in Relation to the Proposed Social and Economic Development Plans. Proceedings of the Twentieth Tuberculosis and Chest Disease Workers' Conference. New Delhi: National Tuberculosis Association. 1965. 28. A scheme for community programme. Indian Journal of Tuberculosis. 18:39. 1972. 29. Correa. Hector. Population, Health, Nutrition, and Development. Toronto and London: D. C. Heath and Co.. 1975.
AJPH August, 1976, Vol. 66, No. 8
BAYESIAN OUTCOME ESTIMATION FIGURE 1-A Decision Tree for the National Tuberculosis PrograElm.
He is found, continues treatment, dies. /P(cf,T,D DTP) He dies P(D I T)
j He is Cured P(C T) He continues \ He is found, continues treatment. P(T I cf) treatment, is cured.
P(cf,T,C DTP) He is found, does not conHe does not tinue treatment, dies. continue / P(cf,NT,D DTP) He dies The victim treatment. (P(NT DTP) P(D NT) is found. He is found, does not con\,/H P(cf DTP) He is sick - tinue treatment, is sick
P(BP NT)
P(cf,NT,BP DTP)
He :s cured ~~~~P((C NT) \ He is found, does not ccn\ tinue treatment, is cured. P(cf,NT,C DTP)
\
The victim is not found P(ncf DTP) He is not found and / dies. P(ncf, D DTP) He dies P(D ncf) / He is He is not found and is I( nc) sick. P(ncf, BP DTP)
,sBk
He is cured P(C ncf) He is not found and is cured. P(C. ncf SO) He is not found and / dies, P(D, ncf SQ) He dies
P(D ncf) He is
sickP(BP I ncf)
xpand.
He is not found and
is sick,
P(DP,
ncf
SO)
He is cured He is not found.
P(ncf SO)
P(C ncf) \ He is not found and is cured. P(C, ncf SQ)
He is found, does not tinue treatment, dies. / P(D,cf,NT SQ) He dies
He is found.
P(cf
O)
con-
P(DINT)
\
He/ is sick-
He does not continue
facf, SO) P(NT SO) P\NT
\BPHe
< P(BP, NT)
He is found, does not continue treatment, is sick. P(BP,cf,NT SQ)
He is cured P(C, NT) He is found, does not con\ tinue treatment, is cured.
P(C,cf,NT SQ) He continues treatment. P(T I cf,SQ)
He is found, continues treatment and dies. / P(cf,T,D SQ) He dies P(D I T)
He is cured P(C T) \ He is found, continues treatment, and is cured. P(cf,T,C SO)
AJPH August, 1976, Vol. 66, No. 8
753
CHASSE 30. Feldstein, M. S.. Piot. M. A. and Sundaresan. T. K. Resource allocation model for public health planning: a case study of Tuberculosis control. WHO Bulletin. 48: Supplement, 1973.
SELECTED REFERENCES Chatterji, J. C.. Sen. H. K.. Chakraborty, A. K.. Subba Rao. K. B.. Kapoor. M. L. and Chatterji. A. K. Comparison trial of three regimens in a treatment of pulmonary tuberculosis. Indian Journal of Tuberculosis. 17:8-17, 1970. Gothi, G. D. and Baily. G. V. J. Problems of treatment of tuberculosis patients in rural areas. Indian Journal of Tuberculosis. 12: 62-68. Nagpaul. D. R., Viswanath, M. K., Dwarakanath, G. A socioepidemiological study of outpatients attending a city tuberculosis clinic in India to judge the place of specialized centres in a tuberculosis control programme. WHO Bulletin. 43:17-34, 1970.
Narang.
R. K. Ethionamide and isoniazid in middle-aged and elderly patients. Indian Journal of Tuberculosis. 17:32-35, 1970. Pamra. S. P.. Narasimhan, R.. Hem, Raj. Mathur, G. P. A trial of relatively inexpensive regimens in the domiciliary treatment of tuberculosis. Indian Journal of Tuberculosis. 17:54-60, 1970. Republic of India. Program Evaluation Organization. Regional Variations in Social Development and Levels of Living. PEO Publication No. 59. Delhi: Planning Commission. 1969. 11:104-105.
ACKNOWLEDGMENTS The materials in this article had their origin in the author's PhD dissertation. The ideas owe something to the dissertation advisors, John Henning, Jerry Miner, and Jesse Burkhead, all of Syracuse University. The author also benefited from comments of John Daly of the U.S. Office of International Health and from a referee of this Journal.
APPENDIX A Estimation of the Fraction N/V Although there is no formula for measuring precisely the mean and variance of two random variables, there is an approximation formula.* Define Z = X/Y. The mean and variance of X are known and the mean and variance of Y are known. A normal distribution is assumed for all variables. Assuming (Y -E(Y))/E(Y) is small for all observations. the following equations will give an approximate value.
(I)
E(Z)
=
E(Y) [1
E(Y)2-
I
size was set at 300. The computer produced a vector of 300 numbers drawn randomly from the distribution described by the mean and the standard deviation of the numerator and a similar set of numbers drawn randomly from the mean and standard deviation of the denominator. Each element in the numerator vector was divided by a corresponding element in the denominator vector to produce 300 estimated decimals. The mean and standard deviation of the sample of 300 decimals was then calculated. The results for the two methods are given in table A.1. TABLE A.1-Estimate of the Fraction of T.B. Cases Found and Diagnosed in the Average District in India.
(2) V(Z) - [E(X)]2 F V(X) + V(Y) + 3V(X)V(Y) [E(Y)]2 [ {E(X)}2 {E(Y)}' {E(X)}' {E(Y)} J
The fraction, number of cases found over number of cases existing in a district, was calculated using this formula. It was also calculated using a simulation technique. In the simulation. a normal distribution was hypothesized for both the numerator and the denominator, and the sample H Tctlc8 Statistics: An Interl)crived in N. 1 Johnson InL. mecdiatc cxthook onidon: (an'ihrldgc lUnikcrsiti Pi.ss. 1964. pp.
178. 1T9
754
Method of Estimation Measure
Formula
Simulation
DTP Mean DTP Variance SQ Mean SQ Variance
0.1366 0.0009 0.0521 0.0002
0.1354 0.0012 0.0526 0.0004
The difference was small enough so that changing from one to the other made no difference on the final decision tree estimate within 4 significant digits.
AJPH August, 1976, Vol. 66, No. 8
Abstract: This paper illustrates how a Bayesian statistical approach was used to estimate the outcomes of the National Tuberculosis Program in India. Such an estimate, it is argued, is necessary for a properjudgment about a project's social usefulness. The process of medical care delivery is reduced to a set of conditioned probabilities. The numbers are estimated using as source material medical records, the results of medical research, and the opinion of experts.
Bayesian methods of estimation are used and their value is discussed. The final discussion contains a brief treatment of the role of project analysis in public decision making. The place of Bayesian methods in project analysis is briefly illustrated, demonstrating their operational value in the field of public health decision making. (Am. J. Public Health 66:747-754, 1976)
A rational decision to spend the taxpayers' money on a public health project demands a prior estimate of its social desirability. A perfect estimate would include an accounting of the resources used, a model linking these resources to all socially relevant outcomes. and a ranking of these outcomes in an order of social desirability. The model linking resources and outcomes deserves attention for a number of reasons. First, as this paper hopes to show. there is more of a problem here than meets the eye. Second. the problem is conceptually soluble. It does not suffer from the value dilemmas which plague the construction of a social preference function. Finally, the problem is important. If nothing else is possible. a rational decision should at least rest on a good estimate of the program's ability to do what it claims to do. A Bayesian approach to developing such an estimate is advocated here. This approach will be used to develop a "best" estimate of the output of the Indian District Tuberculosis Program. This estimate will be accompanied by a measure of uncertainty.* The importance of in-
cluding a measure of uncertainty is gaining recognition in the field of public project appraisal.' Part I of the following discussion tries to clarify the ,,output problem" and to show how Bayesian methods can help solve it. Part II describes the Indian National Tuberculosis Program and uses a decision tree to organize the problem. Part III contains an explanation of the way in which the tree was set up as well as an estimation of probabilities. Part IV discusses policy implications.
1. The Output Problem in Health Program Analysis Arrow defined the output of a medical service from the
point of view of one person as the reduction in the likelihood of sickness and death and its concomitant suffering.2 Output
fessor of Economics. Savannah State College. and Public Health Consultant to the U.S. DHEW Office of International Health. This paper, submitted October 28. 1975. was revised and accepted for publication in the Journal February 12, 1976.
from the point of view of the public decision maker. however, would be in aggregate terms-the number cured or saved from sickness and death. The problem is that there is no conceptually adequate way to directly measure output in this sense. A record of the number of people cured would not do. as it would omit those cured in the program's absence, and thus overestimate the program's output. But most health systems. even in affluent countries do not even go this far. They record what econo-
*Consider two projects. The estimated outcome for project A is exactly the same as the estimated outcome for project B. But the estimate for project A has only a 10 per cent chance of being wrong while that for project B has a 50 per cent chance of being wrong. Obviously project A should be preferred over project B if the ex-
pected outcome is socially desirable. For this reason, such classical uncertainty measures as variances and confidence intervals are considered important in a rational decision process.
Address reprint requests to Dr. John D. Chasse. 502 Mt. Vernon Place. Apt. 4. Rockville. MD 20850. Dr. Chasse is Assistant Pro-
AJPH August, 1976, Vol. 66, No. 8
747
CHASSE
mists call inputs or intermediate goods-hospital days, patients treated, admissions, and releases-and somehow the analyst must move from this information to inferences about the reduction in sickness and death. In a poor country there is an additional problem. Few such countries can afford a health intelligence system adequate even by medical standards. In fact, anyone trying to study anything in a poor country hears time and again that practically any statistic he finds is useless. Admitting this, he must decide what to do next. There are three possible approaches. The first rejects statistical analysis and depends on the opinion of individuals with long experience in the field. This may provide the best present estimate, but it does not create statements that can be refuted by later investigation. If the forecast proves wrong, it is impossible to find out what assumption was wrong or which causal link was incorrect. The second approach ignores the problem and proceeds to apply the whole gamut of classical statistical methods to the existing data. There are two problems with this procedure. First, an unjustified air of objectivity is maintained; second, a certain dishonesty creeps in. Somehow, the final report fails to mention that the mean was developed from three observations, or that the regression analysis had six degrees of freedom. Finally, whatever evidence exists in the opinions of people with long experience is ignored. There is a third way-not totally satisfactory, but flexible enough to avoid the problems mentioned above. Bayesian statistical techniques can create a measure of output in a way that lays all the assumptions open to refutation, but also permits the use of expert opinion where recorded numbers are nonexistent or hopelessly inadequate.
II. The Tuberculosis Program and the Shape of the Analysis Every district in India has a network of hospitals and health centers. In addition, the residents of the main town usually have access to a TB clinic. The quality of service and facilities varies at these clinics. The average operating cost is about Rs. 20,000.3 There is no coordination of TB clinic activities with those of the hospitals and health centers. This situation will be referred to as the status quo (SQ). The District Tuberculosis Program (DTP) turns the clinic into a district TB center with the task of coordinating all TB activities in the district. Establishing the DTP involves equipping the tuberculosis center with an X-ray machine and staffing it with a treatment organizer, a laboratory technician, an X-ray technician, a medical officer, and a BCG team.3 Capital costs are estimated at Rs. 210,000, recurring costs at Rs. 150,000.' The present analysis is limited to the case-finding and treatment activities of the DTP. To go further would make the discussion too long and complex. 748
Organizing the Problem Decision trees have been used to analyze medical decisions, and they have been used to analyze public expenditure decisions.6-'0 The use here is not very sophisticated, as the decision tree is merely used to organize the problem. The decision to expand or contract the DTP can be reduced to one of expanding it into one more district. This is close to the economist's notion of marginal product if the whole district unit is treated as a variable, the national superstructure as a fixed input, and costs and outputs are assumed to be the same for each district unit. These assumptions simplify the present treatment, and their denial would not particularly invalidate the method. Figure 1 illustrates this process. The box on the left side of the figure represents a decision fork. Here the decision maker has control. If he decides not to expand the program then he chooses the lower path and the present system continues unaided by the DTP. If he does decide to expand, then he chooses the upper path. Following this path, one comes to a circle which represents the probability that the victim will be found assuming the existence of the DTP. Here chance reigns. The symbol P(cflDTP) means the probability that the individual will be found given that the DTP exists.* Finding is not curing, however. The treatment must continue, and it must work. So there are two other probability forks. By multiplying the probabilities one can calculate the joint probability that all events will occur-that the victim will be found, will take his medicine, and will be cured. This is written at the end of the line P(cf,T,CIDTP). By adding the joint probabilities at the end of the lines, it is possible to calculate the total likelihood of a cure. The same process is blocked out for the mirror image at the bottom of the diagram which will give the probability for a cure under the status quo. In a similar manner, the likelihood of sickness and death under the two different assumptions can be calculated. and once this is done it will be possible to give a decision maker an estimate of uncertainty involved for the whole population covered.
III. Justification and Estimation of the Probabilities This section explains why the decision tree was set up in this way and it estimates the probabilities. It will be easier to *Although the symbols are explained in the diagram. it might help to have them all listed. DTP = The District Tuberculosis Program SQ = The Status Quo P(cf) = The probability that the individual will be found P(BP) = The probability of being bacteriologically positive (sick) P(T) = The probability of continuing treatment P(C) = The probability of a cure P(D) = The probability of death P(ncf) = The probability of not being found by the system P(NT) = The probability of not continuing treatment AJPH August, 1976, Vol. 66, No. 8
BAYESIAN OUTCOME ESTIMATION
follow if the three steps are kept in mind: (1) The individual is found; (2) He continues treatment; (3) He is cured. The different combinations of these probabilities produce the decision tree. Probabilities must be calculated both for the DTP and for the SQ.
Likelihood of Finidinig the Individual
TABLE 1-Estimate of the Prevalence of Tuberculosis (Sputum Positive Cases per 1,000) Using Three Studies. 10-12 Studies
Measure
Madanappalle and NSS
Madanappalle, NSS and Tumkur
Mean Variance
2.99 0.0824
2.99 0.06197
Meaning and Importance of the Likelihood Many forces keep people in poor countries from approaching health facilities-distance, folk beliefs. ignorance of the fact that doctors and nurses are able and willing to help them, etc. Getting a patient to approach a health center or hospital is. however, only half the problem. The staff of such a facility is so loaded with work that. in the few minutes they can spare, they are likely to substitute a cursory guess for a diagnosis. hand the individual some pills. and send him on his way.
The Estimation of the Probabilities Let N = the number of sputum positives discovered in a distnct V = the number of actual TB victims N/V = the fraction of TB victims found The problem is reduced to estimating the mean and variance of N/V. The number of victims (V) can be found by multiplying the prevalence rate (R) by the population (P). Estimatindg the Mleatt and Variance of tle Prevalence: Three studies were used. The earliest (Madanappalle) was supervised by J. Frimodt-Moller in 1955 in South India.' The second (NSS) was the National Sample Survey.1I The last (Tumkur) was the Indian Council of Medical Research Survey in Tumkur District in South India.1 Since Moller's study is the earliest. his estimates create an expectation which is either confirmed or corrected by the later studies. These three studies are combined by continuous revision.* Table I presents the results. Estimatintg the Meant(iad Variance of lthe Number of Ctases in a District: Goodman has rigorously derived a formula for estimating the mean and variance of a product.'3 The National Sample Survey found that the prevalence of TB was statistically independent of urban or rural residence. of geographical location, and of district size. This permits the use of formulas assuming statistical independence. Where E(X) = the
*Let Mr MO Vo Mn Vn
= the revised mean, Vr = the revised variance = the mean of the first or prior distribution and = the variance of the first or prior distribution = the mean of the second or posterior distribution = the variance of the second or posterior distribu-
Then:
tion MR = (Mo/Vo) + (Mn/Vn)
(l/Vo) + (l/Vn) l/Vr = (l/Vo) + (l/Vn) This is rigorously proved in Thomas H. Wonnacott and Ronald J. Wonnacott, Introductory Statistics (New York: Wiley and Sons. 1969). pages 324-327. AJPH August, 1976, Vol. 66, No. 8
mean or expected value of X and V(X) equals the variance of X. the formulas are: (1) E(XY) = E(X)E(Y) (2) V(XY) = X2V(Y) + Y2V(X) + V(X)V(Y) Using average population figures from the 1970 census. the mean number of cases per district was estimated in this manner as 4861.74. the variance at 1207369. and the standard deviation at 1098.8. Es.ti,ma te of tlie Na tu,ber of Cases Foi)nid: Detailed records of the number of cases actually discovered in a district are unavailable. but summary statistics exist. Thus, it is possible to obtain means and ranges. but not to obtain the original data from which these were calculated. A range cannot be used in further- calculation. but a variance can be used. A formula devised for this type of situation approximates the standard deviation by dividing the range by six.14 Nair. citing quarterly reports of the DTP and the SQ. gives the mean number of cases diagnosed per quarter in districts where the DTP exists as 158.15 The range is between 143 and 190. Adjusting the figures to reflect an annual casefinding yield produces the following estimate for the DTP: Mean = 632 Range = 572-760 Standard Deviation = 42 Variance = 1764 The status quo estimate presented problems. The existing data lump together cases and suspects. Since both the DTP and the SQ draw from the same statistical population. it was assumed that the ratio of cases to suspects would be the same in both. After some testing for consistency. the following case-finding estimates were made for the SQ: Mean = 241 Range = 177-362 Standard Deviation = 40.3 Variance = 1624.09 Estitiniate of tlie Fraction N/V: The objective here is an estimate of the fraction of TB cases (N/V) that are found in a typical district. The number of cases existing in a district (V) has been estimated. The number found (N) has also been estimated. These are estimates or random v ariables. Both italicized words must be stressed. A simple ratio of the means would assume that the numbers were fixed, and would not produce a variance from which an estimate of uncertainty could be derived. There is no rigorously derived method for estimating this. Two methods of approximation were used (described in Appendix A). Both methods produced. for all practical purposes. the same estimate. The DTP mean is put at about 14 per cent and the SQ mean at about 5 per cent. 749
CHASSE
Probability that the Individual, Once Diagnosed, Will Continue Treatment for the Required Length of Time
TABLE 3-Patients in Rural Areas Completing Eighty Per cent or More of Their Drug Collection
Meaning of the Probability To be cured, the individual must return to the health center once a month to collect his pills. Even patients in sanatoria and hospitals often hide pills under their pillows; when quinine was prescribed for malaria, Indian soldiers had to be forced to take it in the presence of superior officer.'6 A peasant sometimes has a five or ten mile walk ahead of him as the private cost of taking the medicine.
Study
Estimation of the Probabilities Probability for the DTP: An apparent conflict in the data can be removed by assuming a higher probability of pill collection for urban than for rural areas. However, this makes it necessary to estimate additional probabilities.* Estimation of P(TIU): The consensus among medical men seemed to be that if an individual made 80 per cent of his collections, the drugs should have their effect. The probability that an individual would do so was calculated by pooling small sample studies. The estimates for three studies dealing with Urban patients are summarized in Table 2. A beta distribution was assumed here.** The resulting estimates w,ere as follows: Mean (expected value) = .6803 Mode (most likely value) = .6813 Variance = .000593 Two studies were combined to estimate the fraction of rural residents that continued to collect their pills. They are shown in Table 3. The mean and mode are both .34, and the variance is .22. TABLE 2-Drug Collection of Patients in Urban T.B. Clinics No. of Patients
Study
Total
Bangalorel 7 Dehli18 Dehli19
118
No. completing 80% or more collections
84
78 54
162
116
Patients collecting 80% or more of their pills
No. of Patients
Rural20
Hospital Tumkur21 Study
283
96
16189
5504
Estimation for the Status-Quo: D. R. Nagpaul, the director of the National Tuberculosis Programme, on the basis of drug usage, estimated that, for the SQ, 50 to 100 cases would complete treatment in a district in one year.2 Using the method discussed abovet, a midpoint of 75 was assumed as the mean, and the range between the most optimistic and pessimistic estimates was divided by six to yield a rough estimate of the standard deviation. Using the formula for the mean and variance of a quotient, and dividing by the number diagnosed in the status quo situation gave a probability of .3199. Since the variance was small calculated in this manner, the simple binomial variance of .220 was used. The method used here is admittedly arbitrary, but the arbitrariness is not hidden, and unlike other equally arbitrary methods, it makes a precise prediction which can be refuted, and points to the type of data needed to improve the accuracy of the final estimate. Estimation of Per cent Rural and Per cent Urban: After assessing evidence on the rural-urban mix of patients in a city health center, the relationship between drug collections and distance from the health center, the distance of the average rural resident from a health center, and the proportion of total cases accounted for by peripheral centers, the per cent rural was estimated at 28.1 and the per cent urban at 71.9 with a variance of 1 1.9 per cent.2 Probability of a Cure with and without Treatment: The effectiveness of treatment was estimated by combining a number of small studies using the same method as that used to find the probability of continuing treatment. The result was: P(CIT) = 0.9526 P(DIT) = 0.0474 Variance = 0.000213
The Probabilities Together *Urban assuming found, P(Ulcf); Continuing treatment assuming urban, P(TIU); Rural assuming found P(RIcf); Continuing treatment assuming rural P(TIR). **The beta distribution is used because of its flexibility. This is discussed in John D. Freund, Mathematical Statistics. Second Edition (Englewood Cliffs: Prentice-Hall, 1971). Chapters 4, 5 and 9. The method of estimation used here can be found in Samuel A. Schmitt, Measuring Uncertainty: An Elementarv Introduction to Bayesian Statistics (Reading, Mass.: Addison-Wesley, 1969), pp. 115-120. Let s = the number of successes, f = the number of fail-
Referring to Figure 1 again, the decision maker's position is represented by the box on the left side of the figure. The "decision fork" symbolizes his absolute control over whether the DTP will, in fact, be established in a particular district. The outcome from then on is subject to chance. For any particular TB victim, a number of events may or may not occur. The victim may or may not come into contact with the system. He may or may not continue to journey to the health
ures.
Mean = (s+l)/(s+f+2) Mode = s/(s+f) Variance = (s+ l)(f+ l)/(s+f+2)2(s+f+3) 750
tSee subhead "Estimate of the Number of Cases Found" AJPH August, 1976, Vol. 66, No. 8
BAYESIAN OUTCOME ESTIMATION
center faithfully every month. Finally, even if he takes the medicine, he has a slight chance of dying. By multiplying the probabilities previously calculated. and adding at the tips of the trees. an estimate was obtained of the likelihood of sickness and death for the random individual in an average district. The variance was calculated by applying the formula for the variance of a product over and over again. The calculations were done using an APL* program on a computer. and the results are shown in Tables 4. 5, and 6. Since the DTP is an on-going program the relevant policy questions concern expanding. contracting. or continuing the DTP. This could logically be reduced to the decision of adding one more unit. The public decision maker would be interested in the lives saved and sickness years averted by such a one district expansion. This can now be calculated. To keep the illustration simple. assume that those sick are sick for one year. The DTP would save about (.297 - .281)X4862 = 78 lives per year. The variance would be calculated as (.158 + .190)X4862 = 1692. This implies a standard deviation of 41. The number saved from sickness would be (.414 - .387)X4862= 131. Calculating the variance yields (.220 + .183)X4862 1959. and the standard deviation, again. would be 44. Both of these measures would imply a likelihood of more than 90 per cent that the prevention of death and sickness would exceed zero. The death figure with a z value of 1.9 just misses the 95 per cent significance level. TABLE 4-Estimates of Death, Sickness, and Cure on the Assumption that the DTP Exists in a District Probability Estimate
Fate of the Individual
Mean
Variance
Death Continued Illness Cure
.281 .387 .332
.158 .183 .210
IV. Discussion and Policy Conclusions Inadequate data and expert opinion have been combined to produce a quantitive estimate of the effect of a TB program on the health of the target population. The process of arriving at this estimate can be thought of as a set of refutable hypotheses. open to comment and correction. It would be hoped that in the actual decision-making process such corrections would be made. The technique produces an estimate of the effect of the program on lives and sickness. together with an estimate of uncertainty. This is not the only set of effects that should receive consideration, but other ef*APL-A Programming Language is a time-sharing language used with the IBM 360 computer. devised by Kenneth E. Iverson.23
AJPH August, 1976, Vol. 66, No. 8
TABLE 5-Estimates of Death, Sickness, and Cure Assuming that the Present Network of Health Facilities (SQ) Tries to Cope Without the Assistance of the DTP Probability Estimate Fate of the Individual
Mean
Variance
Death Continued Illness Cure
.297 .414 .289
.190 .220 .251
fects could be estimated in the same way or. alternatively. these data could be used as a basis for their estimation. The weaknesses and the broad policy implications of this technique are considered further in the sections which follow.
Aretas o lncomnpleteness i1n tile Pr-esenit Analysis Several changes could have added more realism to the decision tree. There is evidence that the cure rate is positively related to the degree of drug collection.24 A different death. cure and sickness probability could have been constructed for 40 per cent collections. 60 per cent collections. 80 per cent collections. 100 per cent collections. In place of a one year time span. a two or three year time span could have been substituted. The case-finding probability included an assumption of correct diagnosis. Another decision tree treatment actually has taken into account the likelihood of incorrect diagnosis." Finally. there is the likelihood of contributing to the spread of drug resistant strains of bacteria. There is always a trade-off between simplicity and realism in model building.2 5. 26 The degree of realism should be determined in part by the type of data available, in part by relevance, and in part by sensitivity analysis. Some complications may not affect the final result. In the present example. it was desired to keep the model simple enough to permit presentation. Time can be added by making the decision tree more complicated. or by using the decision tree results as input for an epidemiological model. This was the way time was included in a full analysis of the DTP.5 Some of the techniques used may disturb the rigorous scientist. Particularly open to objection is Wagle's method for estimating the standard deviation.'4 However, one must point out that the alternatives are to make no estimate and TABLE
5-Estimates of Death, Sickness, and Cure
on the Assumption that the Individual is Left to his Own Resources22 Probability Estimate
Fate of the Individual
Mean
Variance
Death Continued Illness Cure
.301 .421 .278
.210 .244 .201
751
CHASSE
ignore expert opinion or to use such opinion in a way that ignores uncertainty, or assumptions. The knowledge aimed at is not the scientific conclusion of a researcher. It is more analogous to clinical diagnosis where the best available techniques may still produce only partial understanding.
The Policy Implications ofProject Analysis Project analysis cannot remove all ambiguity, but it can reduce it. The DTP has not lived up to the hopes of its designers-reaching 50 per cent of the sputum positives.27 An editorial in the Indian Journal of Tuberculosis complained:28 It is common knowledge that for various reasons the National Tuberculosis Programme is not making satisfactory headway. even though it is a scientifically sound programme.
Project analysis can play two roles here. The first is to force explicit recognition of true criteria. Living up to expectations or not living up to expectations is not a rational criterion of project selection. Lives saved is one, sickness prevented another. Project analysis can also improve knowledge of a project's ability to fulfill rational criteria. In the present example, it has been shown that there is a good likelihood that the DTP "warts and all" is probably saving lives and reducing sickness. Whether it should be expanded depends on alternative uses of the funds to be devoted to it.
The Present Effort and Project Analysis. Significant progress has been made in developing models which will calculate the correct mix of activities given a welfare function and data on coefficients.29 3( The methods illustrated here would seem complementary to these efforts in the following ways: * Bayesian methods can be used to estimate coefficients for those models and to give some notion of the uncertainty inherent in poor data and necessary assumptions. * Project design often uses "idealized" coefficients. Bayesian methods can use actual program statistics after a program has been in operation to take account of less than ideal performance and to point out areas for further investigation and step by step improvement. * A Bayesian approach seems more adapted to the fact that public expenditure decisions, like medical diagnoses. cannot rest on perfect knowledge. Estimates are needed when the decision is made. The approach does not offer a total solution to the problem. but it would seem capable of contributing to the solution.
REFERENCES 1. Reutlinger, S. Techniques for Project Appraisal Under Uncertainty. Baltimore: Johns Hopkins Press, 1970. 2. Arrow, K. J. Uncertainty and the Welfare Economics of Medical Care. Essays in the Theory of Risk-Bearing. Kenneth J. Arrow Ed. Amsterdam: North-Holland Press. 177-219, 1971. 3. Nagpaul, D. R. District Tuberculosis Programme in Concept and Outline. Indian Journal of Tuberculosis 15: 186-198. 1967. 4. Sen. A. S. and Basu, R. N. Economics of health-cost of tu-
berculosis. Indian Journal of Tuberculosis. 18:146, 1972. 752
5. Chasse, J. D. The Cost-Benefit Analysis of Disease Control Programs in Developing Countries with Applications to the Indian National Tuberculosis Programme. Ph.D. Dissertation. Syracuse. N.Y.: Syracuse University. 1974. 6. Ginsberg. A. S. and Offenbach, F. L. An Application of Decision Theory to a Medical Diagnosis Treatment Problem. IEEE Transactions on Systems Science and Cybernetics. SSC-4:335362, 1968. 7. Simon, J. S. Toward formal evaluation of scientific research projects: a case study of population control. Policy Sciences. 3: 177-181, 1971. 8. Howard, R. A.. Matheson. J. E., and North, D. W. The decision to seed hurricanes. Science. 176:1191-1202. 1971. 9. Schweitzer, S. 0. The Economics of the Early Diagnosis of Disease. Evaluation in Health Services Delivery. Richard Yaffe and David Alkind, Eds. New York: Engineering Foundation. 184-196, 1973. 10. Moller. J. F. A community-wide tuberculosis study in a South Indian rural population. 1950-1955. WHO Bulletin. 22:61-170. 1960. 11. Indian Council of Medical Research. Tuberculosis in India: a sample survey. Special Report Series No. 34. Delhi: Indian Council of Medical Research. 1959. 12. Narain, R., Geser, A., Jambunathan. M. V. and Subramanian. M. Some aspects of a tuberculosis prevalence survey in a South Indian district. WHO Bulletin. 29:641-644. 1963. 13. Goodman, L. A. On the Exact Variance of Products. Amer. Stat. Assoc. J. 55:708-713. 1960. 14. Wagle, B. Statistical analysis of risk in capital investment projects. Operational Research Quarterly. 18:13-33, 1967. 15. Nair, S. S. Assessment and monitoring of the national tuberculosis programme. Indian Journal of Tuberculosis. 14:131-134. 1967. 16. Narain, R., Mayurnath. S., Subba Rao, M. S. and Murthy. S. S. F Feasibility of a chemoprophylaxis trial in India against tuberculosis. Indian Journal of Medical Research. 59:339-353. 1971. 17. Banerji, D. Behavior of tuberculosis patients towards treatment organization offering limited supervision. Indian Journal of Tuberculosis. 14:161. 1967. 18. Singh, M. M. and Banerji, D. Follow-up study of patients of pulmonary tuberculosis treated in an urban TB clinic. Indian Journal of Tuberculosis. 15: 160. 1968. 19. Mathur, K. C.. Singh, K. D.. Chakravarty. A.. Basu. N. and Kandelwal. M. L. Results of domiciliary treatment in an urban TB clinic, Indian Journal of Tuberculosis. 9:62. 1963. 20. Gothi G. D. and Baily. G. V. J. Problems of treatment of tuberculosis patients in rural areas. Indian Journal of Tuberculosis. 12:64, 1964. 21. Narain, R., Mayurnath. S.. Subba Rao M. S. and Murthy, S. S. Feasibility of a chemoprophylaxis trial in India. Indian Journal of Medical Research. 59:351-352, 1971. 22. Olakowski, T. Paper Presented in a Symposium on Chemotherapy of Tuberculosis. Madras: Tuberculosis Chemotherapy Centre, February, 1972. 23. Iverson. K. E. A Programming Language. New York: Wiley, 1962. 24. Pamra, S. P. and Mathur, G. P. Drug default in an urban community. Indian Journal of Tuberculosis. 14:199-203. 1967. 25. Raiffa. H. Decision Analysis: Introductory Lectures on Choices under Uncertainty. Reading, Mass.: Addison-Wesley, 1968. 26. Theil, H. Economic Forecasts and Policy. Amsterdam: NorthHolland Press. 1961. 27. Banerji, D. India's National Tuberculosis Programme in Relation to the Proposed Social and Economic Development Plans. Proceedings of the Twentieth Tuberculosis and Chest Disease Workers' Conference. New Delhi: National Tuberculosis Association. 1965. 28. A scheme for community programme. Indian Journal of Tuberculosis. 18:39. 1972. 29. Correa. Hector. Population, Health, Nutrition, and Development. Toronto and London: D. C. Heath and Co.. 1975.
AJPH August, 1976, Vol. 66, No. 8
BAYESIAN OUTCOME ESTIMATION FIGURE 1-A Decision Tree for the National Tuberculosis PrograElm.
He is found, continues treatment, dies. /P(cf,T,D DTP) He dies P(D I T)
j He is Cured P(C T) He continues \ He is found, continues treatment. P(T I cf) treatment, is cured.
P(cf,T,C DTP) He is found, does not conHe does not tinue treatment, dies. continue / P(cf,NT,D DTP) He dies The victim treatment. (P(NT DTP) P(D NT) is found. He is found, does not con\,/H P(cf DTP) He is sick - tinue treatment, is sick
P(BP NT)
P(cf,NT,BP DTP)
He :s cured ~~~~P((C NT) \ He is found, does not ccn\ tinue treatment, is cured. P(cf,NT,C DTP)
\
The victim is not found P(ncf DTP) He is not found and / dies. P(ncf, D DTP) He dies P(D ncf) / He is He is not found and is I( nc) sick. P(ncf, BP DTP)
,sBk
He is cured P(C ncf) He is not found and is cured. P(C. ncf SO) He is not found and / dies, P(D, ncf SQ) He dies
P(D ncf) He is
sickP(BP I ncf)
xpand.
He is not found and
is sick,
P(DP,
ncf
SO)
He is cured He is not found.
P(ncf SO)
P(C ncf) \ He is not found and is cured. P(C, ncf SQ)
He is found, does not tinue treatment, dies. / P(D,cf,NT SQ) He dies
He is found.
P(cf
O)
con-
P(DINT)
\
He/ is sick-
He does not continue
facf, SO) P(NT SO) P\NT
\BPHe
< P(BP, NT)
He is found, does not continue treatment, is sick. P(BP,cf,NT SQ)
He is cured P(C, NT) He is found, does not con\ tinue treatment, is cured.
P(C,cf,NT SQ) He continues treatment. P(T I cf,SQ)
He is found, continues treatment and dies. / P(cf,T,D SQ) He dies P(D I T)
He is cured P(C T) \ He is found, continues treatment, and is cured. P(cf,T,C SO)
AJPH August, 1976, Vol. 66, No. 8
753
CHASSE 30. Feldstein, M. S.. Piot. M. A. and Sundaresan. T. K. Resource allocation model for public health planning: a case study of Tuberculosis control. WHO Bulletin. 48: Supplement, 1973.
SELECTED REFERENCES Chatterji, J. C.. Sen. H. K.. Chakraborty, A. K.. Subba Rao. K. B.. Kapoor. M. L. and Chatterji. A. K. Comparison trial of three regimens in a treatment of pulmonary tuberculosis. Indian Journal of Tuberculosis. 17:8-17, 1970. Gothi, G. D. and Baily. G. V. J. Problems of treatment of tuberculosis patients in rural areas. Indian Journal of Tuberculosis. 12: 62-68. Nagpaul. D. R., Viswanath, M. K., Dwarakanath, G. A socioepidemiological study of outpatients attending a city tuberculosis clinic in India to judge the place of specialized centres in a tuberculosis control programme. WHO Bulletin. 43:17-34, 1970.
Narang.
R. K. Ethionamide and isoniazid in middle-aged and elderly patients. Indian Journal of Tuberculosis. 17:32-35, 1970. Pamra. S. P.. Narasimhan, R.. Hem, Raj. Mathur, G. P. A trial of relatively inexpensive regimens in the domiciliary treatment of tuberculosis. Indian Journal of Tuberculosis. 17:54-60, 1970. Republic of India. Program Evaluation Organization. Regional Variations in Social Development and Levels of Living. PEO Publication No. 59. Delhi: Planning Commission. 1969. 11:104-105.
ACKNOWLEDGMENTS The materials in this article had their origin in the author's PhD dissertation. The ideas owe something to the dissertation advisors, John Henning, Jerry Miner, and Jesse Burkhead, all of Syracuse University. The author also benefited from comments of John Daly of the U.S. Office of International Health and from a referee of this Journal.
APPENDIX A Estimation of the Fraction N/V Although there is no formula for measuring precisely the mean and variance of two random variables, there is an approximation formula.* Define Z = X/Y. The mean and variance of X are known and the mean and variance of Y are known. A normal distribution is assumed for all variables. Assuming (Y -E(Y))/E(Y) is small for all observations. the following equations will give an approximate value.
(I)
E(Z)
=
E(Y) [1
E(Y)2-
I
size was set at 300. The computer produced a vector of 300 numbers drawn randomly from the distribution described by the mean and the standard deviation of the numerator and a similar set of numbers drawn randomly from the mean and standard deviation of the denominator. Each element in the numerator vector was divided by a corresponding element in the denominator vector to produce 300 estimated decimals. The mean and standard deviation of the sample of 300 decimals was then calculated. The results for the two methods are given in table A.1. TABLE A.1-Estimate of the Fraction of T.B. Cases Found and Diagnosed in the Average District in India.
(2) V(Z) - [E(X)]2 F V(X) + V(Y) + 3V(X)V(Y) [E(Y)]2 [ {E(X)}2 {E(Y)}' {E(X)}' {E(Y)} J
The fraction, number of cases found over number of cases existing in a district, was calculated using this formula. It was also calculated using a simulation technique. In the simulation. a normal distribution was hypothesized for both the numerator and the denominator, and the sample H Tctlc8 Statistics: An Interl)crived in N. 1 Johnson InL. mecdiatc cxthook onidon: (an'ihrldgc lUnikcrsiti Pi.ss. 1964. pp.
178. 1T9
754
Method of Estimation Measure
Formula
Simulation
DTP Mean DTP Variance SQ Mean SQ Variance
0.1366 0.0009 0.0521 0.0002
0.1354 0.0012 0.0526 0.0004
The difference was small enough so that changing from one to the other made no difference on the final decision tree estimate within 4 significant digits.
AJPH August, 1976, Vol. 66, No. 8
