Article pubs.acs.org/est
The Value of Information for Managing Contaminated Sediments Matthew E. Bates,*,† Magnus Sparrevik,‡,§,∥ Nicolas de Lichy,⊥ and Igor Linkov† †
Environmental Laboratory, Engineer Research and Development Center, US Army Corps of Engineers, 696 Virginia Road, Concord, Massachusetts 01742, United States ‡ The Norwegian Defence Estates Agency, Forsvarsbygg, P.O. Box 405 Sentrum, Oslo, NO-0103, Norway § Norwegian Geotechnical Institute, P.O. Box 3930 Ullevål Stadion, Oslo, NO-0806, Norway ∥ Department of Industrial Economics and Technology Management, Norwegian University of Technology, Trondheim 7491, Norway ⊥ London School of Economics, Houghton Street, London WC2A 2AE, United Kingdom S Supporting Information *
ABSTRACT: Effective management of contaminated sediments is important for long-term human and environmental health, but site-management decisions are often made under high uncertainty and without the help of structured decision support tools. Potential trade-offs between remedial costs, environmental effects, human health risks, and societal benefits, as well as fundamental differences in stakeholder priorities, complicate decision making. Formal decision-analytic tools such as multicriteria decision analysis (MCDA) move beyond ad hoc decision support to quantitatively and holistically rank management alternatives and add transparency and replicability to the evaluation process. However, even the best decisions made under uncertainty may be found suboptimal in hindsight, once additional scientific, social, economic, or other details become known. Value of information (VoI) analysis extends MCDA by systematically evaluating the impact of uncertainty on a decision. VoI prioritizes future research in terms of expected decision relevance by helping decision makers estimate the likelihood that additional information will improve decision confidence or change their selection of a management plan. In this study, VoI analysis evaluates uncertainty, estimates decision confidence, and prioritizes research to inform selection of a sediment capping strategy for the dibenzo-p-dioxin and -furan contaminated Grenland fjord system in southern Norway. The VoI model extends stochastic MCDA to model decisions with and without simulated new information and compares decision confidence across scenarios with different degrees of remaining uncertainty. Results highlight opportunities for decision makers to benefit from additional information by anticipating the improved decision confidence (or lack thereof) expected from reducing uncertainties for each criterion or combination of criteria. This case study demonstrates the usefulness of VoI analysis for environmental decisions by predicting when decisions can be made confidently, for prioritizing areas of research to pursue to improve decision confidence, and for differentiating between decision-relevant and decision-irrelevant differences in evaluation perspectives, all of which help guide meaningful deliberation toward effective consensus solutions.
■
INTRODUCTION Over the past centuries, harmful contaminants from urban and industrial effluent, agricultural runoff, and other point and nonpoint sources have accumulated in sediments around the world, causing significant health risks to humans and ecological receptors. Managing contaminated sites for the protection of human and environmental health has become a common issue with significant implications for a variety of stakeholders including local residents, regulators, and commercial and industrial land users. The multitude of stakeholder perspectives, combined with considerable uncertainties about the impacts of contamination and the effects of various remedial alternatives, make managing contaminated sediments highly complex.1−6 Uncertain information can result in significant cost associated with management.7 Current methods for management include various forms and extents of capping, removal, and active treatment.8 These can be evaluated against each other according to their expected costs, benefits, and risks.9,10 As any sediment management alternative is unlikely to simulta© 2014 American Chemical Society
neously have the largest benefit, lowest cost, and least risk, decision makers must conduct trade-off analyses to determine which alternative will provide the greatest holistic value. Traditional decision support methods often fail to effectively evaluate all important aspects of complex problems that may be better suited for analyses leveraging systematic and quantitative decision-analytic methods.11−13 Methods such as multicriteria decision analysis (MCDA) allow decision makers to separately capture each important aspect of the problem, describe tradeoffs and priorities between objectives, incorporate uncertainties, rank alternatives, and transparently share results with the stakeholder community.14−16 These tools are valuable because they decompose complex problems into more tractable parts, allow one to ask the right stakeholders or experts for each Received: Revised: Accepted: Published: 9478
February 11, 2014 June 19, 2014 June 24, 2014 June 24, 2014 dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
■
Article
METHODS Case Study MCDA Problem Formulation. Sediments in the 66 km2 Grenland fjord system in southern Norway (Figure 1) are contaminated with PCDD/Fs from industrial activity by
different piece of information, and provide transparency and replicability to the process. These methods can more thoroughly evaluate disparate remedial alternatives since they allow decision makers to simultaneously consider multiple decision criteria and clearly communicate why alternatives are ranked as they are. In recent years, this type of analysis has been increasingly applied to complex environmental problems from contaminated sediment management to ecosystem restoration and spatial planning.10−11,17−20 Even properly structured remediation decisions are often based on uncertain data, which adds further complexity. Factors such as imprecise knowledge about how management alternatives will ultimately perform in terms of each criterion, a need for trade-offs between competing criteria, and differences in priorities between evaluation perspectives, make it difficult to evaluate alternatives with full and transparent consideration of all relevant detail. Stochastic MCDA methods can account for uncertain inputs by using Monte Carlo simulations to produce a probability distribution of results,21,22 but even these distributions of alternative rankings may change as new information is gained. To determine how different reductions in uncertainty may affect decision confidence and alternative rankings, analysts can perform a value of information (VoI) analysis.23−25 This estimates changes in decision outcomes that could result from gaining specific information related to MCDA inputs prior to making a decision. Similar to a decision-focused sensitivity analysis, the results show which data is most important to the decision, help direct research efforts toward decision-relevant information, and help decision makers identify the minimum research needed to confidently choose a management alternative.26−28 The importance of quantitatively prioritizing research and estimating decision confidence has been explicitly emphasized by recent US and international initiatives,11,17,29 and lack of robust decision confidence indicators has been a criticism of traditional MCDA modeling.30−32 In comparison to the resources required for physical research, the resources required to structure a remedial decision through MCDA/VoI analysis are minor, but the potential benefits of MCDA/VoI modeling can be vast. In this paper, VoI is applied to a contaminated-sediment management case study in the dibenzo-p-dioxin and -furan (PCDD/F) contaminated Grenland fjord system of southern Norway. This VoI builds on a stochastic MCDA performed for the Grenland fjord site by Sparrevik, Barton, Bates, and Linkov33 (SBBL), a previous paper by the authors. That paper uses stochastic MCDA to rank remedial alternatives based on estimated performance across human health, societal benefit, remedial cost, and environmental impact criteria. This is done from cost effectiveness, cost benefit, and value plural evaluation perspectives and incorporates uncertainty related to alternative performance. The resulting distributions of alternative rankings indicate that no alternative is clearly dominant given the currently available information and that further action may be needed to reduce uncertainty prior to making a selection. This paper extends the previous stochastic MCDA for Grenland fjord with a decision-directed VoI analysis that estimates decision confidence under different information-gathering scenarios, prioritizes research strategies, and projects the degree to which different types of information are expected to change which alternatives are perceived as best.
Figure 1. Grenland fjord system in southern Norway, showing the extent of six remedial alternatives under consideration in the MCDA & VoI analyses. The inner and outer fjord systems are separated by the shallow Brevik Sill. Remedial alternatives include natural recovery (NR; green, yellow, red, and orange), capping only the highest contaminated areas in the inner fjord (HIFC; orange) or outer fjord (HOFC; yellow), capping the entire area of the inner fjord (IFC; red and orange), or a larger extent of the outer fjord (OFC; green and yellow), and capping the whole inner fjord plus a large portion of the outer fjord (WFC; green, yellow, red and orange). Figure reprinted from ref 33. Copyright 2012 American Chemical Society.
the Hydro Porsgrunn magnesium plant that has been discharging wastewater to the area since 1951, and with polychlorinated biphenyls (PCBs) from other local sources. Reductions in toxic discharges have measurably decreased concentrations in marine organisms but levels are still high, leading to fishing restrictions and health advisories against consuming local seafood.34 Mitigating measures, including isolating the contaminated sediments by covering them with a thin layer cap of clean material, are currently being proposed and studied.35,36 SBBL formulate the sediment capping decision for the Grenland fjord as an MCDA problem consisting of the following steps: (1) formulating general objectives, (2) enumerating alternatives management plans, (3) developing specific criteria and metrics that decompose the general objectives into measurable attributes, (4) measuring the performance of each alternative remediation plan against those criteria, and (5) weighting the relative importance of each criterion with respect to the others before calculating results. SBBL use MCDA to rank six remedial alternatives for the fjord. These include a low-cost, minimally effective option of natural recovery (NR), in which natural resedimentation covers contaminated sediments over time, and also a high-cost, highly effective option for whole fjord capping (WFC) that includes contaminated sediments in the entire inner fjord and 9479
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
denotes the alternative under consideration, A represents the set of all alternatives, and v represents the number of alternatives to be evaluated (valid for v > 1):
most of the outer fjord. Intermediate alternatives include options for inner fjord capping (IFC), outer fjord capping (OFC), or capping of only the highly contaminated hot spots in the inner (HIFC) or outer (HOFC) fjord. Four criteria are used to compare and evaluate the six alternative capping approaches: human health risk reduction, socio-economic benefits expected from removing seafood consumption advisories, remediation cost, and estimated life cycle environmental impacts. Because the performance of the six alternatives on each of these four criteria is uncertain, skewed normal distributions of performance scores are used based on input data for median, 5th percentile, and 95th percentile values for each alternative for each criterion. These distributions represent the best a priori expert interpretation of literature estimates of potential remediation outcomes in the fjord, as shown graphically in Supporting Information, Table S1 and as explained by SBBL. Trade-offs between the four criteria are applied differently in each of three weighting schemes meant to replicate cost effectiveness (CE), cost benefit (CB), and value plural (VP) methods of evaluation. The cost effectiveness weighting scheme gives equal weight to health risk reductions and costs but no weight to the other criteria. The cost benefit weighting scheme gives equal weight to socioeconomic benefits and costs but no weight to the others. The value plural weighting scheme gives equal weight to all four criteria. (Further detail regarding the case study alternatives, criteria, weighting schemes, and performance data are described by SBBL; see also Supporting Information.) PROMETHEE II MCDA. Once the decision problem is formulated, different MCDA methods can be applied with slightly different assumptions and operations to normalize and aggregate raw performance score data across criteria to derive a total relative preference score for each alternative. Available MCDA methods can generally be categories as either multiattribute utility theory (MAUT) or outranking methods. MAUT methods use specific knowledge of utility or value functions to infer preference for different levels of alternative performance across a fixed scale, whereas outranking methods avoid assumptions about the scales used to normalize higher or lower performance scores into preference values and instead use pairwise comparisons to identify the relative dominance among each pair of alternatives and aggregate those scores into total preference scores.16 For cases where the input data is uncertain, stochastic MCDA can use Monte Carlo simulations to repeatedly apply the chosen MCDA formula to each set of alternative performance data sampled from the input distributions. SBBL use the PROMETHEE II outranking MCDA method37 in 10 000 Monte Carlo simulations to estimate the distribution of total preference scores (“net flows”) for each management alternative (i.e., natural recovery or some extent of capping) under each weighting scheme. In each simulation, a preference function, P(k,a), compares the performance score of alternative k directly with the performance scores of other alternatives, for example, a, on some criterion to establish relative dominance on a 0−1 scale where a score of 0 represents the strict domination of k by a and a score of 1 represents the strict dominance of k over a. An aggregate dominance score, φj′(k), is calculated for each alternative on each criterion based on preference functions evaluating each alternative k with all other alternatives, where j represents the decision criterion (i.e., health risk, socioeconomic benefit, life cycle environmental impact, or cost), k
φj′(k) =
(∑a ≠ k ∈ A Pj(k , a) − Pj(a , k)) v−1
The value of φj′(k) is constant across weighting schemes and ranges from −1 to 1 for each alternative on each criterion, where a score of +1 implies that alternative k is the strictly most preferred among all alternatives on criterion j, and a score of −1 implies that alternative k is the strictly least preferred among all alternatives on criterion j. Here, φj′(k) uses a linear comparison function with a threshold of strict preference, p, set at 10% of the range (maximum to minimum) of values for each criterion (see Supporting Information and Figure S1 for additional details on the threshold of strict preference and its use). As the dominance scores for each alternative are independent of the weighting scheme (i.e., CE, CB, or VP), i, weighing is applied to aggregate flows as a weighted sum across criteria to calculate the net flow, φj(k), which defines the final rank order and total relative dominance of each alternative for each weighting scheme: φi(k) =
∑ wij·φj′(k) j
where the weight, wij, indicates the relative importance of criterion j among all other criteria (out of 100% total weight) under weighting scheme i. Since net flow is an expression of the relative degree to which each alternative outperforms the others, it can be interpreted as a measure of decision conf idence, where higher values on a 0−1 scale represent greater confidence in the dominant performance of an alternative on the criteria that matter most. PROMETHEE II Expected Value of Perfect and PartialPerfect Information. When summarizing the results of a stochastic MCDA, the expected alternative ranking remains uncertain because the knowledge of underlying input performance scores are uncertain. However, not all uncertainties in the input data are equally relevant to the decision because uncertainties in performance scores contribute differently to resulting net flow and rank order. (For example, it is possible for uncertainty on the least important criterion to hold the greatest sway over the decision if there is great overlap among alternatives on that criterion but no overlap among other criteria and otherwise similar net flow scores.) Through VoI analysis, it is possible to determine which data uncertainties are expected to have the greatest average effect on the total score of each alternative, potentially enabling selection of a better alternative than would be selected in the base case. Decisiondirected VoI analysis is seen as an integral part of environmental decision making, where alternatives are evaluated, decision confidence is assessed, the minimum necessary research is defined and prioritized, science is undertaken until prioritized uncertainties are reduced, and site management action is ultimately taken (Figure 2). Howard23 and Raiffa24 define the value of additional information as the difference between the expected value of the outcome that would be achieved under current knowledge and the expected value that could be achieved under posterior knowledge. In classical approaches, this value is represented economically or in terms of utility. Consistent with the SBBL PROMETHEE II model formulation, this study adapts classical 9480
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
alternative in those simulations than in the base case. On the basis of the outcome of the VoI analysis, research can be targeted to reduce uncertainty on input values where the payoffin this case improved total net flow scores from selecting the best actual management alternativeis largest. In the base case where no additional information is generated, the expected net flow of the optimal decision (i.e., to choose the alternative with the maximum average net flow) is calculated for each weighting scheme as N
φi _no = max k(∑ φi , n(k))/N
Figure 2. Schematic expression of the integral role of stochastic MCDA and VoI analyses in environmental decision-making under uncertainty.
n=1
where N is the number of simulations, here implemented as N = 10 000. This represents selection of the alternative (k) from the stochastic MCDA with the highest average score based on current uncertain estimates of future performance. It is often possible to improve decision confidence by collecting additional information prior to making a selection. An upper bound for the value of engaging in research is assessed by simulating perfect knowledge of all uncertainties, selecting the highest scoring alternative in each trial, and then taking the average of those various highest scores. While we do not know which scores will ultimately emerge in the real world, this simulation shows how high a score we could expect on average if we knew all relevant information prior to needing to choose an alternative. Because the decision maker has perfect information prior to each decision in each trial, the expected net flow with perfect information is equivalent to the average of the net flows of the highest scoring alternative in each of N = 10 000 simulations. Average net flow with perfect information, φperfect, is calculated for each weighting scheme as
approaches following Linkov et al.26 to represent this value in terms of increased net flow, the holistic total score that results from evaluating each alternative against the weighted criteria of the MCDA model. Since net flow is a weighted measure of the extent to which one alternative outperforms all others in the areas that are most important, VoI in this context is not interpreted as a monetary value but rather as an improvement in decision confidence (e.g., related to discovering that an alternative outperforms the others even more strongly than initially believed) due to the additional information. The following VoI model extends the SBBL stochastic MCDA into a two-stage nested Monte Carlo simulation of decisions with and without additional information to infer the potential impact of information on decision outcomes, that is, to identify which uncertainties are most decision-relevant. The model first simulates gaining perfect information about one or more uncertain input variables by probabilistically sampling values from the skewed normal input distributions. Each sampled value represents the true performance of some alternative in some trial, corresponding to one potential true future state of the world. While different in each trial, across the trials the observed distribution of sampled points will reflect the input distribution from which it is drawn. Then, within each trial, a second Monte Carlo simulation runs to sample values for the remaining variables for which perfect knowledge is not simulated and calculates a distribution of expected net flows given the perfectly known information presumed in the first trial. While it is impossible to know a priori what the actual future state of the world will be, a comparison of average net flows from simulations with some perfect information to those without information in the base case sets a bound on the expected value of partial perfect information (EVPPI) from that type of research. We interpret this an upper bound because we expect most real world research projects to provide something less than perfect information. To find the expected value of perfect information (EVPI), the upper bound on the value of all research on this topic, information on every uncertainty is simulated as simultaneously known and available before the decision. “Partial perfect” information assumes absolute certainty on one or more input values without addressing the uncertainty of the others, while complete perfect information assumes simultaneous certainty of all inputs is available in each trial. In these VoI simulations, any increase in average net flow over the base case can be attributed to simulations where an alternative perceived as inferior on average in the base case is actually seen to outperform all other alternatives once specific information is simulated, enabling selection of a higher scoring
N
φi _perfect =
∑ (max k(φi ,n(k)))/N n=1
The difference between the simulated net flow scores with and without information (the EVPI) can reveal the extent to which sufficient information is already present to make a reasonable decision: EVPIi = φi_perfect − φi_no. If this difference between net flows with and without new information is minor, there may be little to be gained by waiting (or paying) for additional information to be revealed prior to selecting an alternative. It is also possible to quantify relative contributions to increased net flow from information about only a subset of criteria, which is useful for prioritizing research between criteria. For example, given two research efforts of otherwise similar scope and cost but investigating alternative performance on different criteria, is one expected to change our base case decision more than the other? For criteria in each subset C for which information is simulated as known, the aggregate dominance score, F, for each alternative is calculated in each of N = 10 000 trials as Fj(k) = φ′j (k), as defined previously. The known scores of criteria in C for which we are simulating perfect information are sampled once in each of the N trials and then presumed known for each of M = 10 000 additional, second-stage inner simulations of other criteria not being researched in C. For criteria not in subset C, an alternative means of calculating the aggregate dominance score takes an average over M inner-stage Monte Carlo simulations: Fj(k) = ΣmM = 1(φj,m ′ (k))/M. On the basis of these two means of calculating Fj(k) depending on whether or not each j is in C, the total net flow for each alternative for each weighting scheme 9481
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
in each of the N simulations is calculated as the weighted sum: φi(k) = Σjwij·Fj(k). Finally, these values are averaged over all N trials to obtain the expected net flow of the optimal decision with perfect information about only criteria in C and while retaining the original uncertainty of all other criteria not in C:
of the simulations where perfect information is present. Consequently, the alternatives expected to be selected on the basis of the stochastic MCDA analysis have a high chance of being found suboptimal once further information is revealed. These results could be an indicator for many decision makers that further data collection to reduce uncertainties prior to making a decision is worth pursuing. Value of Information Results. Funding, equipment, and staffing limitations often mean that it is only feasible to pursue limited research prior to action. VoI results can help prioritize further Grenland fjord research in terms of expected decision impact. The net flow values with no additional information relate to the level of decision confidence that a decision maker might have regarding a chosen alternative performing better than the other alternatives in the base case. Similarly, the average flow net value with perfect information indicates the maximum decision confidence that can be expected if all possible research were performed prior to the decision. The absolute and percent differences in these values help decision makers decide if additional research may be warranted. Results show the average net flow of the highest scoring alternatives under different simulated information scenarios (Figure 4). From a cost effectiveness perspective, perfect information on both health risks and costs (the weighted criteria in that scheme) increases the average net flow by 0.16 points (179% increase), and partial perfect information on costs alone increases net flow by 0.10 points (a 115% increase) or on health risks alone increases net flow by 0.09 points (a 98% increase). From a cost benefit perspective, perfect information on both societal benefits and costs (the weighted criteria in that scheme) increases the average net flow by 0.17 points (a 52% increase), and partial perfect information on societal benefits alone increases net flow by 0.13 points (a 41% increase) or on costs alone increases net flow by 0.04 points (an 11% increase). From a value plural perspective, research into environmental risks (omitted from Figure 4) is expected to be of negligible relevance, increasing net flow by less than 0.01 points over any given research portfolio without it. Research into societal benefits is expect to be most useful, increasing net flow by 0.06 points (a 30% increase), followed by research into health risks alone and then costs alone, in that each are expected to increase net flow by about 0.10 points (a 5% increase). Research into multiple criteria simultaneously can further increase net flow over the gains from researching single criteria, but with smaller additional percent gains. Perfect information on all four criteria is simulated as increasing average net flow by 0.08 points (a 43% increase). Note that this evaluation perspective weights all criteria equally but shows that even perfect information on environmental effect is anticipated to have negligible effect on the decision. This shows that while environmental factors are still equally important to the decision, they are already sufficiently well quantified to enable robust decision making and further research in to this topic should not be prioritized over research in more decision-relevant areas. Without information, the decision maker is simulated as making one choice, selecting the alternative with the highest average net flow (as identified in Figure 3 and the leftmost chart of each panel in Figure 5). Knowledge of partial-perfect information both increases the average net flow of the simulated decision (Figure 4) and changes the ratios in which the alternatives are chosen in the simulations (remaining charts of each panel in Figure 5). The charts in Figure 5 show the ratios of how often each alternative is chosen in each of the
N
φi _C =
∑ (max k(Fi(k)))/N n=1
The expected increase in net flow resulting from perfect information about only the criteria in C is calculated as EVPPIi = φi_c − φi_no. This is done for all possible combinations of criteria C to compare improvements in decision confidence expected from different research strategies. This is helpful for prioritizing research strategies based on their decision relevance. The value of information simulated under various research portfolios can be compared with the monetary and temporal costs required to gather reasonably complete information in those portfolios, to prioritize research given limited resources.
■
RESULTS AND DISCUSSION Stochastic MCDA Results. Results of the SBBL stochastic MCDA (Supporting Information, Figures S2 and S3) indicated that partial remediation of the inner fjord (HIFC) was the preferred alternative under the value plural weighting scheme. HIFC was also most preferred under the cost effectiveness weighting scheme, though the differences in net flow under this scheme were small and all alternatives scored close to zero (indicating near equivalence). Under the cost benefit weighting scheme, partial remediation of the outer fjord (HOFC) was identified as the most preferred alternative. Similar results are reproduced in this paper, using a threshold of strict preference of 10%, and extended into VoI results. Rank order results (Figure 3) give decision makers an indicator of decision confidence by showing how consistently
Figure 3. Percent of trials in which each alternative is ranked first over the 10 000 Monte Carlo simulations of the MCDA/VoI model simulating perfect information under each weighting scheme.
each alternative can be expected to be ranked in a certain order relative to the other alternatives, under existing uncertainty regarding decision model inputs. Although the stochastic MCDA indicates the average preferred alternative from a cost effectiveness perspective to be HIFC, when perfect information is available, HIFC is seen to rank first only about 40% of the time, with other alternatives outranking it nearly 60% of the time. Similarly, the average preferred alternatives from the cost benefit and value plural evaluation perspectives, HOFC and HIFC respectively, are expected to rank first in only about half 9482
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
Figure 4. Average total net flows of the top ranking alternatives in each trial for each weighting scheme: with no additional research, with partial perfect information on each individual criterion, with partial perfect information on combinations of criteria, and with perfect information on all criteria.
Figure 5. Proportion of time each alternative is chosen in each information scenario from each evaluation perspective. Ratios of alternatives that draw closest to ratios under perfect information are desired. (Each pie chart decomposes a net flow bar from Figure 4 to show how each alternative is represented in the average net flow score in each scenario.)
and often conflicting criteria. MCDA has advantages over lower-dimensional decision methods because it can transparently incorporate stakeholder and management priorities between the different criteria, simulate multiple evaluation perspectives, and integrate data measured on different scales to holistically evaluate and rank all management alternatives under consideration. Stochastic MCDA extends traditional MCDA by acknowledging that alternative rankings cannot be established with absolute certainty due to a frequent lack of knowledge about the eventual performance of the alternatives on the various criteria. Stochastic MCDA describes alternative performance through probabilistic distributions that reflect these uncertainties, which helps decision makers make the best use of current information to identify alternatives expected to have the most preferable average outcomes. However, when performance scores are highly uncertain, as is common in many environmental applications, resulting
information scenarios summarized by the bars in Figure 4. Using this to compare perceived preference between the alternatives with different information provides decision makers another visual tool to reference when deciding how much information to pursue (e.g., for identifying which partial perfect information scenario seems sufficiently close to the perfect information scenario that additional refinement may not be necessary). For example, from the cost benefit perspective (top right panel of Figure 5), the spread of perceived preference among the alternatives in the “Societal info” chart looks sufficiently similar to that of the “Societal & Cost Info” chart that a decision maker might forego pursuing additional cost research even though Figure 4 shows that it could still increase average net flow. Application to Environmental Site Management. Complex environmental decisions are often made with uncertain data and while facing trade-offs between multiple 9483
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
Notes
decision confidence can remain low and the expected rank order from a stochastic MCDA may be subject to change as new information affecting the input distributions is introduced. When faced with low decision confidence, decision makers and stakeholders may be tempted to try to reduce uncertainties in the analysis through unguided collection of additional data. As demonstrated in this paper, VoI analysis can extend stochastic MCDA by simulating various potential information gathering strategies for reducing the underlying uncertainties. By showing how rankings and net flows are expected to change under different information scenarios, decision makers and researchers can differentiate between decision-relevant and decisionirrelevant uncertainties, determine if existing information is sufficient for decision making, and design research plans reasonably expected to most improve decision making with the least additional investment (Figure 2). This choice of which research strategy to pursue should reflect consideration of the time and resources available for use, the decision confidence expected in the base case (Figure 3), the expected absolute net flows with and without new information (Figure 4), the expected percent increase in net flow available through new information (Figure 4), and the relative agreement in perceived rank order simulated in cases with partial perfect vs full perfect information (Figure 5). When considered together, these results are informative for moving beyond qualitative or intuitive treatments of uncertainty to quantify information value. Environmental site managers provided with these types of transparent and quantitative results can better choose which research areas to invest in, can quickly run sensitive and scenario analyses regarding distribution shape, modeled criteria, weighting and evaluation perspectives, etc., and can more clearly communicate the potential impact of further research to partners and stakeholders. The Grenland Fjord results both illustrate the importance of selecting an appropriate evaluation perspective (and of comparing results across perspectives) and highlight the potential differences in decision relevance of different research. This emphasizes the importance of prioritizing research based on expected outcome in the specific problem context rather than based on researcher familiarity or some other qualitative measure. VoI results can provide a transparent and objective basis for furthering discussions between stakeholders and decision makers, quantifying the potential impact of research (or lack thereof), prioritizing research based on expected outcomes, and clearly showing how uncertainties in input data are expected to affect decision confidence and the ranking of alternative site management plans. As an analytical effort, the time and resources needed for a VoI analysis are often insignificant in comparison to the resources they can save by prioritizing and bounding the need for further physical research.
■
The authors declare no competing financial interest.
■
ACKNOWLEDGMENTS The authors would like to thank previous studies for the work performed on the Norwegian Grenland fjord system, especially the Opticap project (www.opticap.no), the SEDFLEX project, and the Norwegian Research Council that financed those studies. Thanks are also due to Kelsie Baker and Cate Fox-Lent for help with graphics, editing, and data curation, and to Professor Jeff Keisler who was instrumental in teaching and critiquing the VoI methods. The authors additionally thank Dr. Todd Bridges, Dr. Martin Schultz, and are grateful for financial support from the Dredging Operation Environmental Research (DOER) program of the US Army Corps of Engineers (USACE). Permission was granted by the USACE Chief of Engineers to publish this material.
■
ASSOCIATED CONTENT
* Supporting Information S
More detailed information is available about the preference function used in the PROMETHEE II Outranking MCDA method, background on the Grenland fjord region, and the input data and results from the SBLL model. This material is available free of charge via the Internet at http://pubs.acs.org.
■
REFERENCES
(1) Bridges, T. S.; Apitz, S. E.; Evison, L.; Keckler, K.; Logan, M.; Nadeau, S.; Wenning, R. J. Risk-based decision making to manage contaminated sediments. Integr. Environ. Assess. Manage. 2006, 2 (1), 51−58. (2) Förstner, U.; Salomons, W. Trends and challenges in sediment research 2008: The role of sediments in river basin management. J. Soils Sediments. 2008, 8 (5), 281−283. (3) Weber, R.; Gaus, C.; Tysklind, M.; Johnston, P.; Forter, M.; Hollert, H.; Heinisch, E.; Holoubek, I.; Lloyd-Smith, M.; Masunaga, S.; Moccarelli, P.; Santillo, D.; Seike, N.; Symons, R.; Torres, J. P. M.; Verta, M.; Varbelow, G.; Vijgen, J.; Watson, A.; Costner, P.; Woelz, J.; Wycisk, P.; Zennegg, M. Dioxin-and POP-contaminated sites Contemporary and future relevance and challenges. Environ. Sci. Pollut. Res. 2008, 15 (5), 363−393. (4) Sparrevik, M.; Breedveld, G. D. From ecological risk assessments to risk governance. evaluation of the Norwegian management system for contaminated sediments. Integr. Environ. Assess. Manage. 2010, 6 (2), 240−248. (5) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Seager, T. P.; Bridges, T.; Gardner, K. H.; Rogers, S. H.; Belluck, D. A.; Meyer, A. Multicriteria decision analysis: a comprehensive decision approach for management of contaminated sediments. Risk Anal. 2006, 26 (1), 61−78. (6) National Research Council. Contaminated Sediments in Ports and Waterways: Cleanup Strategies and Technologies; National Academies Press: Washington, DC, 1997. (7) Linkov, I.; Ames, M. R.; Crouch, E. A.; Satterstrom, F. K. Uncertainty in octanol−water partition coefficient: Implications for risk assessment and remedial costs. Environ. Sci. Technol. 2005, 39 (18), 6917−6922. (8) Ghosh, U.; Luthy, R. G.; Cornelissen, G.; Werner, D.; Menzie, C. A. In-situ sorbent amendments: A new direction in contaminated sediment management. Environ. Sci. Technol. 2011, 45 (4), 1163− 1168. (9) Linkov, I.; Seager, T. P. Coupling multi-criteria decision analysis, life-cycle assessment, and risk assessment for emerging threats. Environ. Sci. Technol. 2011, 45 (12), 5068−74. (10) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Seager, T. P.; Bridges, T.; Gardner, K. H.; Rogers, S. H.; Belluck, D. A.; Meyer, A. Multicriteria decision analysis: A comprehensive decision approach for management of contaminated sediments. Risk Anal. 2006, 26 (1), 61− 78. (11) Gamper, C. D.; Turcanu, C. On the governmental use of multicriteria analysis. Ecol. Econ. 2007, 62 (2), 298−307. (12) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Batchelor, C.; Bridges, T.; Ferguson, E. From comparative risk assessment to multi-criteria decision analysis and adaptive management: Recent developments and applications. Environ. Int. 2006, 32 (8), 1072−1093.
AUTHOR INFORMATION
Corresponding Author
*E-mail: [email protected]. 9484
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
(13) Linkov, I.; Moberg, E. Multi-Criteria Decision Analysis: Environmental Applications and Case Studies; CRC Press: Boca Raton, FL, 2012. (14) Howard, R. A. The foundations of decision analysis. IEEE Trans. Syst. Sci. Cyber., 1968, 4 (3), 211−219. (15) Keeney, R. L.; Raiffa, H. Decisions with Multiple Objectives: Preferences and Value Tradeoffs; Wiley: New York, 1976. (16) Belton, V.; Stewart, T. J. Multiple Criteria Decision Analysis an Integrated Approach; Kluwer Academic: Boston MA, 2002. (17) National Research Council. Environmental Decisions in the Face of Uncertainty; The National Academies Press: Washington, DC, 2013. (18) Mendoza, G. A.; Martins, H. Multi-criteria decision analysis in natural resource management: A critical review of methods and new modelling paradigms. Forest Ecol. Manage. 2006, 230 (1−3), 1−22. (19) Alvarez-Guerra, M.; Canis, L.; Voulvoulis, N.; Viguri, J. R.; Linkov, I. Prioritization of sediment management alternatives using stochastic multicriteria acceptability analysis. Sci. Total Environ. 2010, 408 (20), 4354−4367. (20) Linkov, I.; Sahay, S.; Kiker, G.; Bridges, T.; Seager, T. P. Multicriteria decision analysis: A framework for managing contaminated sediments. In Strategic Management of Marine Ecosystems; Springer: Amsterdam, The Netherlands, 2005; pp 271−297. (21) Hyde, K.; Maier, H. R.; Colby, C. Incorporating uncertainty in the PROMETHEE MCDA method. J. Multi-Criteria Decis. Anal. 2003, 12 (4−5), 245−259. (22) Tervonen, T.; Figueira, J. R.; Lahdelma, R.; Dias, J. A.; Salminen, P. A stochastic method for robustness analysis in sorting problems. Eur. J. Operat. Res. 2009, 192 (1), 236−242. (23) Howard, R. A. Information value theory. IEEE Trans. Syst. Sci. Cybernetics, 1966, 2 (1), 22−26. (24) Raiffa, H. Decision Analysis: Introductory Lectures on Choices under Uncertainty; Addison-Wesley: Reading MA, 1968. (25) Keisler, J. M.; Collier, Z. A.; Chu, E.; Sinatra, N.; Linkov, I. Value of information analysis: The state of application. Environ. Syst. Decis. 2013, DOI: 10.1007/s10669-013-9439-4. (26) Linkov, I.; Bates, M. E.; Canis, L. J.; Seager, T. P.; Keisler, J. M. A decision-directed approach for prioritizing research into the impact of nanomaterials on the environment and human health. Nat. Nanotechnol. 2011, 6 (12), 784−787. (27) Brennan, A.; Kharroubi, S.; O’Hagan, A.; Chilcott, J. Calculating partial expected value of perfect information via Monte Carlo sampling algorithms. Med. Decis. Making 2007, 27 (4), 448−470. (28) Keilser, J. M. Value of information: Facilitating targeted information acquisition in decision processes. Environ. Systems and Decisions 2014, DOI: 10.1007/s10669-014-9493-6. (29) The Presidential/Congressional Commission on Risk Assessment and Risk Management. Framework for Environmental Health Risk Management, Final Report, Volume 1. Washington, DC, 1997; http:// oaspub.epa.gov/eims/eimscomm.getfile?p_download_id=36372. (30) Hubbard, D.; Samuelson, D. Modeling without Measurements: How the decision analysis culture’s lack of empiricism reduces its effectiveness. OR/MS Today 2009, 36 (5), 26−31, http://www.ormstoday.org/orms-10-09/frrisk.html. (31) Morgan, M. G.; Henrion, M. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis; Cambridge University Press: New York, 1990. (32) Felli, J. C.; Hazen, G. B. Do sensitivity analyses really capture problem sensitivity? An empirical analysis based on information value. Risk, Decis. Policy 1999, 4 (2), 79−98. (33) Sparrevik, M.; Barton, D. N.; Bates, M. E.; Linkov, I. Use of stochastic multi-criteria decision analysis to support sustainable management of contaminated sediments. Environ. Sci. Technol. 2012, 46 (3), 1326−1334. (34) Knutzen, J.; Bjerkeng, B.; Næs, K.; Schlabach, M. Polychlorinated dibenzofurans/dibenzo-p-dioxins (PCDF/PCDDs) and other dioxin-like substances in marine organisms from the Grenland fjords, S. Norway, 1975−2001: Present contamination levels, trends and species specific accumulation of PCDF/PCDD congeners. Chemosphere 2003, 52 (4), 745−760.
(35) Barton, D. N.; Navrud, S.; Bjørkeslett, H.; Lilleby, I. Economic benefits of large-scale remediation of contaminated marine sedimentsA literature review and an application to the Grenland fjords in Norway. J. Soils Sediments 2010, 10 (2), 186−201. (36) Saloranta, T. M.; Armitage, J. M.; Haario, H.; Næs, K.; Cousins, I. T.; Barton, D. N. Modeling the effects and uncertainties of contaminated sediment remediation scenarios in a Norwegian Fjord by Markov chain Monte Carlo simulation. Environ. Sci. Technol. 2007, 42 (1), 200−206. (37) Brans, J. P.; Vincke, P.; Mareschal, B. How to select and how to rank projects: The PROMETHEE method. Eur. J. Operat. Res. 1986, 24, 228−238.
9485
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
The Value of Information for Managing Contaminated Sediments Matthew E. Bates,*,† Magnus Sparrevik,‡,§,∥ Nicolas de Lichy,⊥ and Igor Linkov† †
Environmental Laboratory, Engineer Research and Development Center, US Army Corps of Engineers, 696 Virginia Road, Concord, Massachusetts 01742, United States ‡ The Norwegian Defence Estates Agency, Forsvarsbygg, P.O. Box 405 Sentrum, Oslo, NO-0103, Norway § Norwegian Geotechnical Institute, P.O. Box 3930 Ullevål Stadion, Oslo, NO-0806, Norway ∥ Department of Industrial Economics and Technology Management, Norwegian University of Technology, Trondheim 7491, Norway ⊥ London School of Economics, Houghton Street, London WC2A 2AE, United Kingdom S Supporting Information *
ABSTRACT: Effective management of contaminated sediments is important for long-term human and environmental health, but site-management decisions are often made under high uncertainty and without the help of structured decision support tools. Potential trade-offs between remedial costs, environmental effects, human health risks, and societal benefits, as well as fundamental differences in stakeholder priorities, complicate decision making. Formal decision-analytic tools such as multicriteria decision analysis (MCDA) move beyond ad hoc decision support to quantitatively and holistically rank management alternatives and add transparency and replicability to the evaluation process. However, even the best decisions made under uncertainty may be found suboptimal in hindsight, once additional scientific, social, economic, or other details become known. Value of information (VoI) analysis extends MCDA by systematically evaluating the impact of uncertainty on a decision. VoI prioritizes future research in terms of expected decision relevance by helping decision makers estimate the likelihood that additional information will improve decision confidence or change their selection of a management plan. In this study, VoI analysis evaluates uncertainty, estimates decision confidence, and prioritizes research to inform selection of a sediment capping strategy for the dibenzo-p-dioxin and -furan contaminated Grenland fjord system in southern Norway. The VoI model extends stochastic MCDA to model decisions with and without simulated new information and compares decision confidence across scenarios with different degrees of remaining uncertainty. Results highlight opportunities for decision makers to benefit from additional information by anticipating the improved decision confidence (or lack thereof) expected from reducing uncertainties for each criterion or combination of criteria. This case study demonstrates the usefulness of VoI analysis for environmental decisions by predicting when decisions can be made confidently, for prioritizing areas of research to pursue to improve decision confidence, and for differentiating between decision-relevant and decision-irrelevant differences in evaluation perspectives, all of which help guide meaningful deliberation toward effective consensus solutions.
■
INTRODUCTION Over the past centuries, harmful contaminants from urban and industrial effluent, agricultural runoff, and other point and nonpoint sources have accumulated in sediments around the world, causing significant health risks to humans and ecological receptors. Managing contaminated sites for the protection of human and environmental health has become a common issue with significant implications for a variety of stakeholders including local residents, regulators, and commercial and industrial land users. The multitude of stakeholder perspectives, combined with considerable uncertainties about the impacts of contamination and the effects of various remedial alternatives, make managing contaminated sediments highly complex.1−6 Uncertain information can result in significant cost associated with management.7 Current methods for management include various forms and extents of capping, removal, and active treatment.8 These can be evaluated against each other according to their expected costs, benefits, and risks.9,10 As any sediment management alternative is unlikely to simulta© 2014 American Chemical Society
neously have the largest benefit, lowest cost, and least risk, decision makers must conduct trade-off analyses to determine which alternative will provide the greatest holistic value. Traditional decision support methods often fail to effectively evaluate all important aspects of complex problems that may be better suited for analyses leveraging systematic and quantitative decision-analytic methods.11−13 Methods such as multicriteria decision analysis (MCDA) allow decision makers to separately capture each important aspect of the problem, describe tradeoffs and priorities between objectives, incorporate uncertainties, rank alternatives, and transparently share results with the stakeholder community.14−16 These tools are valuable because they decompose complex problems into more tractable parts, allow one to ask the right stakeholders or experts for each Received: Revised: Accepted: Published: 9478
February 11, 2014 June 19, 2014 June 24, 2014 June 24, 2014 dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
■
Article
METHODS Case Study MCDA Problem Formulation. Sediments in the 66 km2 Grenland fjord system in southern Norway (Figure 1) are contaminated with PCDD/Fs from industrial activity by
different piece of information, and provide transparency and replicability to the process. These methods can more thoroughly evaluate disparate remedial alternatives since they allow decision makers to simultaneously consider multiple decision criteria and clearly communicate why alternatives are ranked as they are. In recent years, this type of analysis has been increasingly applied to complex environmental problems from contaminated sediment management to ecosystem restoration and spatial planning.10−11,17−20 Even properly structured remediation decisions are often based on uncertain data, which adds further complexity. Factors such as imprecise knowledge about how management alternatives will ultimately perform in terms of each criterion, a need for trade-offs between competing criteria, and differences in priorities between evaluation perspectives, make it difficult to evaluate alternatives with full and transparent consideration of all relevant detail. Stochastic MCDA methods can account for uncertain inputs by using Monte Carlo simulations to produce a probability distribution of results,21,22 but even these distributions of alternative rankings may change as new information is gained. To determine how different reductions in uncertainty may affect decision confidence and alternative rankings, analysts can perform a value of information (VoI) analysis.23−25 This estimates changes in decision outcomes that could result from gaining specific information related to MCDA inputs prior to making a decision. Similar to a decision-focused sensitivity analysis, the results show which data is most important to the decision, help direct research efforts toward decision-relevant information, and help decision makers identify the minimum research needed to confidently choose a management alternative.26−28 The importance of quantitatively prioritizing research and estimating decision confidence has been explicitly emphasized by recent US and international initiatives,11,17,29 and lack of robust decision confidence indicators has been a criticism of traditional MCDA modeling.30−32 In comparison to the resources required for physical research, the resources required to structure a remedial decision through MCDA/VoI analysis are minor, but the potential benefits of MCDA/VoI modeling can be vast. In this paper, VoI is applied to a contaminated-sediment management case study in the dibenzo-p-dioxin and -furan (PCDD/F) contaminated Grenland fjord system of southern Norway. This VoI builds on a stochastic MCDA performed for the Grenland fjord site by Sparrevik, Barton, Bates, and Linkov33 (SBBL), a previous paper by the authors. That paper uses stochastic MCDA to rank remedial alternatives based on estimated performance across human health, societal benefit, remedial cost, and environmental impact criteria. This is done from cost effectiveness, cost benefit, and value plural evaluation perspectives and incorporates uncertainty related to alternative performance. The resulting distributions of alternative rankings indicate that no alternative is clearly dominant given the currently available information and that further action may be needed to reduce uncertainty prior to making a selection. This paper extends the previous stochastic MCDA for Grenland fjord with a decision-directed VoI analysis that estimates decision confidence under different information-gathering scenarios, prioritizes research strategies, and projects the degree to which different types of information are expected to change which alternatives are perceived as best.
Figure 1. Grenland fjord system in southern Norway, showing the extent of six remedial alternatives under consideration in the MCDA & VoI analyses. The inner and outer fjord systems are separated by the shallow Brevik Sill. Remedial alternatives include natural recovery (NR; green, yellow, red, and orange), capping only the highest contaminated areas in the inner fjord (HIFC; orange) or outer fjord (HOFC; yellow), capping the entire area of the inner fjord (IFC; red and orange), or a larger extent of the outer fjord (OFC; green and yellow), and capping the whole inner fjord plus a large portion of the outer fjord (WFC; green, yellow, red and orange). Figure reprinted from ref 33. Copyright 2012 American Chemical Society.
the Hydro Porsgrunn magnesium plant that has been discharging wastewater to the area since 1951, and with polychlorinated biphenyls (PCBs) from other local sources. Reductions in toxic discharges have measurably decreased concentrations in marine organisms but levels are still high, leading to fishing restrictions and health advisories against consuming local seafood.34 Mitigating measures, including isolating the contaminated sediments by covering them with a thin layer cap of clean material, are currently being proposed and studied.35,36 SBBL formulate the sediment capping decision for the Grenland fjord as an MCDA problem consisting of the following steps: (1) formulating general objectives, (2) enumerating alternatives management plans, (3) developing specific criteria and metrics that decompose the general objectives into measurable attributes, (4) measuring the performance of each alternative remediation plan against those criteria, and (5) weighting the relative importance of each criterion with respect to the others before calculating results. SBBL use MCDA to rank six remedial alternatives for the fjord. These include a low-cost, minimally effective option of natural recovery (NR), in which natural resedimentation covers contaminated sediments over time, and also a high-cost, highly effective option for whole fjord capping (WFC) that includes contaminated sediments in the entire inner fjord and 9479
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
denotes the alternative under consideration, A represents the set of all alternatives, and v represents the number of alternatives to be evaluated (valid for v > 1):
most of the outer fjord. Intermediate alternatives include options for inner fjord capping (IFC), outer fjord capping (OFC), or capping of only the highly contaminated hot spots in the inner (HIFC) or outer (HOFC) fjord. Four criteria are used to compare and evaluate the six alternative capping approaches: human health risk reduction, socio-economic benefits expected from removing seafood consumption advisories, remediation cost, and estimated life cycle environmental impacts. Because the performance of the six alternatives on each of these four criteria is uncertain, skewed normal distributions of performance scores are used based on input data for median, 5th percentile, and 95th percentile values for each alternative for each criterion. These distributions represent the best a priori expert interpretation of literature estimates of potential remediation outcomes in the fjord, as shown graphically in Supporting Information, Table S1 and as explained by SBBL. Trade-offs between the four criteria are applied differently in each of three weighting schemes meant to replicate cost effectiveness (CE), cost benefit (CB), and value plural (VP) methods of evaluation. The cost effectiveness weighting scheme gives equal weight to health risk reductions and costs but no weight to the other criteria. The cost benefit weighting scheme gives equal weight to socioeconomic benefits and costs but no weight to the others. The value plural weighting scheme gives equal weight to all four criteria. (Further detail regarding the case study alternatives, criteria, weighting schemes, and performance data are described by SBBL; see also Supporting Information.) PROMETHEE II MCDA. Once the decision problem is formulated, different MCDA methods can be applied with slightly different assumptions and operations to normalize and aggregate raw performance score data across criteria to derive a total relative preference score for each alternative. Available MCDA methods can generally be categories as either multiattribute utility theory (MAUT) or outranking methods. MAUT methods use specific knowledge of utility or value functions to infer preference for different levels of alternative performance across a fixed scale, whereas outranking methods avoid assumptions about the scales used to normalize higher or lower performance scores into preference values and instead use pairwise comparisons to identify the relative dominance among each pair of alternatives and aggregate those scores into total preference scores.16 For cases where the input data is uncertain, stochastic MCDA can use Monte Carlo simulations to repeatedly apply the chosen MCDA formula to each set of alternative performance data sampled from the input distributions. SBBL use the PROMETHEE II outranking MCDA method37 in 10 000 Monte Carlo simulations to estimate the distribution of total preference scores (“net flows”) for each management alternative (i.e., natural recovery or some extent of capping) under each weighting scheme. In each simulation, a preference function, P(k,a), compares the performance score of alternative k directly with the performance scores of other alternatives, for example, a, on some criterion to establish relative dominance on a 0−1 scale where a score of 0 represents the strict domination of k by a and a score of 1 represents the strict dominance of k over a. An aggregate dominance score, φj′(k), is calculated for each alternative on each criterion based on preference functions evaluating each alternative k with all other alternatives, where j represents the decision criterion (i.e., health risk, socioeconomic benefit, life cycle environmental impact, or cost), k
φj′(k) =
(∑a ≠ k ∈ A Pj(k , a) − Pj(a , k)) v−1
The value of φj′(k) is constant across weighting schemes and ranges from −1 to 1 for each alternative on each criterion, where a score of +1 implies that alternative k is the strictly most preferred among all alternatives on criterion j, and a score of −1 implies that alternative k is the strictly least preferred among all alternatives on criterion j. Here, φj′(k) uses a linear comparison function with a threshold of strict preference, p, set at 10% of the range (maximum to minimum) of values for each criterion (see Supporting Information and Figure S1 for additional details on the threshold of strict preference and its use). As the dominance scores for each alternative are independent of the weighting scheme (i.e., CE, CB, or VP), i, weighing is applied to aggregate flows as a weighted sum across criteria to calculate the net flow, φj(k), which defines the final rank order and total relative dominance of each alternative for each weighting scheme: φi(k) =
∑ wij·φj′(k) j
where the weight, wij, indicates the relative importance of criterion j among all other criteria (out of 100% total weight) under weighting scheme i. Since net flow is an expression of the relative degree to which each alternative outperforms the others, it can be interpreted as a measure of decision conf idence, where higher values on a 0−1 scale represent greater confidence in the dominant performance of an alternative on the criteria that matter most. PROMETHEE II Expected Value of Perfect and PartialPerfect Information. When summarizing the results of a stochastic MCDA, the expected alternative ranking remains uncertain because the knowledge of underlying input performance scores are uncertain. However, not all uncertainties in the input data are equally relevant to the decision because uncertainties in performance scores contribute differently to resulting net flow and rank order. (For example, it is possible for uncertainty on the least important criterion to hold the greatest sway over the decision if there is great overlap among alternatives on that criterion but no overlap among other criteria and otherwise similar net flow scores.) Through VoI analysis, it is possible to determine which data uncertainties are expected to have the greatest average effect on the total score of each alternative, potentially enabling selection of a better alternative than would be selected in the base case. Decisiondirected VoI analysis is seen as an integral part of environmental decision making, where alternatives are evaluated, decision confidence is assessed, the minimum necessary research is defined and prioritized, science is undertaken until prioritized uncertainties are reduced, and site management action is ultimately taken (Figure 2). Howard23 and Raiffa24 define the value of additional information as the difference between the expected value of the outcome that would be achieved under current knowledge and the expected value that could be achieved under posterior knowledge. In classical approaches, this value is represented economically or in terms of utility. Consistent with the SBBL PROMETHEE II model formulation, this study adapts classical 9480
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
alternative in those simulations than in the base case. On the basis of the outcome of the VoI analysis, research can be targeted to reduce uncertainty on input values where the payoffin this case improved total net flow scores from selecting the best actual management alternativeis largest. In the base case where no additional information is generated, the expected net flow of the optimal decision (i.e., to choose the alternative with the maximum average net flow) is calculated for each weighting scheme as N
φi _no = max k(∑ φi , n(k))/N
Figure 2. Schematic expression of the integral role of stochastic MCDA and VoI analyses in environmental decision-making under uncertainty.
n=1
where N is the number of simulations, here implemented as N = 10 000. This represents selection of the alternative (k) from the stochastic MCDA with the highest average score based on current uncertain estimates of future performance. It is often possible to improve decision confidence by collecting additional information prior to making a selection. An upper bound for the value of engaging in research is assessed by simulating perfect knowledge of all uncertainties, selecting the highest scoring alternative in each trial, and then taking the average of those various highest scores. While we do not know which scores will ultimately emerge in the real world, this simulation shows how high a score we could expect on average if we knew all relevant information prior to needing to choose an alternative. Because the decision maker has perfect information prior to each decision in each trial, the expected net flow with perfect information is equivalent to the average of the net flows of the highest scoring alternative in each of N = 10 000 simulations. Average net flow with perfect information, φperfect, is calculated for each weighting scheme as
approaches following Linkov et al.26 to represent this value in terms of increased net flow, the holistic total score that results from evaluating each alternative against the weighted criteria of the MCDA model. Since net flow is a weighted measure of the extent to which one alternative outperforms all others in the areas that are most important, VoI in this context is not interpreted as a monetary value but rather as an improvement in decision confidence (e.g., related to discovering that an alternative outperforms the others even more strongly than initially believed) due to the additional information. The following VoI model extends the SBBL stochastic MCDA into a two-stage nested Monte Carlo simulation of decisions with and without additional information to infer the potential impact of information on decision outcomes, that is, to identify which uncertainties are most decision-relevant. The model first simulates gaining perfect information about one or more uncertain input variables by probabilistically sampling values from the skewed normal input distributions. Each sampled value represents the true performance of some alternative in some trial, corresponding to one potential true future state of the world. While different in each trial, across the trials the observed distribution of sampled points will reflect the input distribution from which it is drawn. Then, within each trial, a second Monte Carlo simulation runs to sample values for the remaining variables for which perfect knowledge is not simulated and calculates a distribution of expected net flows given the perfectly known information presumed in the first trial. While it is impossible to know a priori what the actual future state of the world will be, a comparison of average net flows from simulations with some perfect information to those without information in the base case sets a bound on the expected value of partial perfect information (EVPPI) from that type of research. We interpret this an upper bound because we expect most real world research projects to provide something less than perfect information. To find the expected value of perfect information (EVPI), the upper bound on the value of all research on this topic, information on every uncertainty is simulated as simultaneously known and available before the decision. “Partial perfect” information assumes absolute certainty on one or more input values without addressing the uncertainty of the others, while complete perfect information assumes simultaneous certainty of all inputs is available in each trial. In these VoI simulations, any increase in average net flow over the base case can be attributed to simulations where an alternative perceived as inferior on average in the base case is actually seen to outperform all other alternatives once specific information is simulated, enabling selection of a higher scoring
N
φi _perfect =
∑ (max k(φi ,n(k)))/N n=1
The difference between the simulated net flow scores with and without information (the EVPI) can reveal the extent to which sufficient information is already present to make a reasonable decision: EVPIi = φi_perfect − φi_no. If this difference between net flows with and without new information is minor, there may be little to be gained by waiting (or paying) for additional information to be revealed prior to selecting an alternative. It is also possible to quantify relative contributions to increased net flow from information about only a subset of criteria, which is useful for prioritizing research between criteria. For example, given two research efforts of otherwise similar scope and cost but investigating alternative performance on different criteria, is one expected to change our base case decision more than the other? For criteria in each subset C for which information is simulated as known, the aggregate dominance score, F, for each alternative is calculated in each of N = 10 000 trials as Fj(k) = φ′j (k), as defined previously. The known scores of criteria in C for which we are simulating perfect information are sampled once in each of the N trials and then presumed known for each of M = 10 000 additional, second-stage inner simulations of other criteria not being researched in C. For criteria not in subset C, an alternative means of calculating the aggregate dominance score takes an average over M inner-stage Monte Carlo simulations: Fj(k) = ΣmM = 1(φj,m ′ (k))/M. On the basis of these two means of calculating Fj(k) depending on whether or not each j is in C, the total net flow for each alternative for each weighting scheme 9481
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
in each of the N simulations is calculated as the weighted sum: φi(k) = Σjwij·Fj(k). Finally, these values are averaged over all N trials to obtain the expected net flow of the optimal decision with perfect information about only criteria in C and while retaining the original uncertainty of all other criteria not in C:
of the simulations where perfect information is present. Consequently, the alternatives expected to be selected on the basis of the stochastic MCDA analysis have a high chance of being found suboptimal once further information is revealed. These results could be an indicator for many decision makers that further data collection to reduce uncertainties prior to making a decision is worth pursuing. Value of Information Results. Funding, equipment, and staffing limitations often mean that it is only feasible to pursue limited research prior to action. VoI results can help prioritize further Grenland fjord research in terms of expected decision impact. The net flow values with no additional information relate to the level of decision confidence that a decision maker might have regarding a chosen alternative performing better than the other alternatives in the base case. Similarly, the average flow net value with perfect information indicates the maximum decision confidence that can be expected if all possible research were performed prior to the decision. The absolute and percent differences in these values help decision makers decide if additional research may be warranted. Results show the average net flow of the highest scoring alternatives under different simulated information scenarios (Figure 4). From a cost effectiveness perspective, perfect information on both health risks and costs (the weighted criteria in that scheme) increases the average net flow by 0.16 points (179% increase), and partial perfect information on costs alone increases net flow by 0.10 points (a 115% increase) or on health risks alone increases net flow by 0.09 points (a 98% increase). From a cost benefit perspective, perfect information on both societal benefits and costs (the weighted criteria in that scheme) increases the average net flow by 0.17 points (a 52% increase), and partial perfect information on societal benefits alone increases net flow by 0.13 points (a 41% increase) or on costs alone increases net flow by 0.04 points (an 11% increase). From a value plural perspective, research into environmental risks (omitted from Figure 4) is expected to be of negligible relevance, increasing net flow by less than 0.01 points over any given research portfolio without it. Research into societal benefits is expect to be most useful, increasing net flow by 0.06 points (a 30% increase), followed by research into health risks alone and then costs alone, in that each are expected to increase net flow by about 0.10 points (a 5% increase). Research into multiple criteria simultaneously can further increase net flow over the gains from researching single criteria, but with smaller additional percent gains. Perfect information on all four criteria is simulated as increasing average net flow by 0.08 points (a 43% increase). Note that this evaluation perspective weights all criteria equally but shows that even perfect information on environmental effect is anticipated to have negligible effect on the decision. This shows that while environmental factors are still equally important to the decision, they are already sufficiently well quantified to enable robust decision making and further research in to this topic should not be prioritized over research in more decision-relevant areas. Without information, the decision maker is simulated as making one choice, selecting the alternative with the highest average net flow (as identified in Figure 3 and the leftmost chart of each panel in Figure 5). Knowledge of partial-perfect information both increases the average net flow of the simulated decision (Figure 4) and changes the ratios in which the alternatives are chosen in the simulations (remaining charts of each panel in Figure 5). The charts in Figure 5 show the ratios of how often each alternative is chosen in each of the
N
φi _C =
∑ (max k(Fi(k)))/N n=1
The expected increase in net flow resulting from perfect information about only the criteria in C is calculated as EVPPIi = φi_c − φi_no. This is done for all possible combinations of criteria C to compare improvements in decision confidence expected from different research strategies. This is helpful for prioritizing research strategies based on their decision relevance. The value of information simulated under various research portfolios can be compared with the monetary and temporal costs required to gather reasonably complete information in those portfolios, to prioritize research given limited resources.
■
RESULTS AND DISCUSSION Stochastic MCDA Results. Results of the SBBL stochastic MCDA (Supporting Information, Figures S2 and S3) indicated that partial remediation of the inner fjord (HIFC) was the preferred alternative under the value plural weighting scheme. HIFC was also most preferred under the cost effectiveness weighting scheme, though the differences in net flow under this scheme were small and all alternatives scored close to zero (indicating near equivalence). Under the cost benefit weighting scheme, partial remediation of the outer fjord (HOFC) was identified as the most preferred alternative. Similar results are reproduced in this paper, using a threshold of strict preference of 10%, and extended into VoI results. Rank order results (Figure 3) give decision makers an indicator of decision confidence by showing how consistently
Figure 3. Percent of trials in which each alternative is ranked first over the 10 000 Monte Carlo simulations of the MCDA/VoI model simulating perfect information under each weighting scheme.
each alternative can be expected to be ranked in a certain order relative to the other alternatives, under existing uncertainty regarding decision model inputs. Although the stochastic MCDA indicates the average preferred alternative from a cost effectiveness perspective to be HIFC, when perfect information is available, HIFC is seen to rank first only about 40% of the time, with other alternatives outranking it nearly 60% of the time. Similarly, the average preferred alternatives from the cost benefit and value plural evaluation perspectives, HOFC and HIFC respectively, are expected to rank first in only about half 9482
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
Figure 4. Average total net flows of the top ranking alternatives in each trial for each weighting scheme: with no additional research, with partial perfect information on each individual criterion, with partial perfect information on combinations of criteria, and with perfect information on all criteria.
Figure 5. Proportion of time each alternative is chosen in each information scenario from each evaluation perspective. Ratios of alternatives that draw closest to ratios under perfect information are desired. (Each pie chart decomposes a net flow bar from Figure 4 to show how each alternative is represented in the average net flow score in each scenario.)
and often conflicting criteria. MCDA has advantages over lower-dimensional decision methods because it can transparently incorporate stakeholder and management priorities between the different criteria, simulate multiple evaluation perspectives, and integrate data measured on different scales to holistically evaluate and rank all management alternatives under consideration. Stochastic MCDA extends traditional MCDA by acknowledging that alternative rankings cannot be established with absolute certainty due to a frequent lack of knowledge about the eventual performance of the alternatives on the various criteria. Stochastic MCDA describes alternative performance through probabilistic distributions that reflect these uncertainties, which helps decision makers make the best use of current information to identify alternatives expected to have the most preferable average outcomes. However, when performance scores are highly uncertain, as is common in many environmental applications, resulting
information scenarios summarized by the bars in Figure 4. Using this to compare perceived preference between the alternatives with different information provides decision makers another visual tool to reference when deciding how much information to pursue (e.g., for identifying which partial perfect information scenario seems sufficiently close to the perfect information scenario that additional refinement may not be necessary). For example, from the cost benefit perspective (top right panel of Figure 5), the spread of perceived preference among the alternatives in the “Societal info” chart looks sufficiently similar to that of the “Societal & Cost Info” chart that a decision maker might forego pursuing additional cost research even though Figure 4 shows that it could still increase average net flow. Application to Environmental Site Management. Complex environmental decisions are often made with uncertain data and while facing trade-offs between multiple 9483
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
Notes
decision confidence can remain low and the expected rank order from a stochastic MCDA may be subject to change as new information affecting the input distributions is introduced. When faced with low decision confidence, decision makers and stakeholders may be tempted to try to reduce uncertainties in the analysis through unguided collection of additional data. As demonstrated in this paper, VoI analysis can extend stochastic MCDA by simulating various potential information gathering strategies for reducing the underlying uncertainties. By showing how rankings and net flows are expected to change under different information scenarios, decision makers and researchers can differentiate between decision-relevant and decisionirrelevant uncertainties, determine if existing information is sufficient for decision making, and design research plans reasonably expected to most improve decision making with the least additional investment (Figure 2). This choice of which research strategy to pursue should reflect consideration of the time and resources available for use, the decision confidence expected in the base case (Figure 3), the expected absolute net flows with and without new information (Figure 4), the expected percent increase in net flow available through new information (Figure 4), and the relative agreement in perceived rank order simulated in cases with partial perfect vs full perfect information (Figure 5). When considered together, these results are informative for moving beyond qualitative or intuitive treatments of uncertainty to quantify information value. Environmental site managers provided with these types of transparent and quantitative results can better choose which research areas to invest in, can quickly run sensitive and scenario analyses regarding distribution shape, modeled criteria, weighting and evaluation perspectives, etc., and can more clearly communicate the potential impact of further research to partners and stakeholders. The Grenland Fjord results both illustrate the importance of selecting an appropriate evaluation perspective (and of comparing results across perspectives) and highlight the potential differences in decision relevance of different research. This emphasizes the importance of prioritizing research based on expected outcome in the specific problem context rather than based on researcher familiarity or some other qualitative measure. VoI results can provide a transparent and objective basis for furthering discussions between stakeholders and decision makers, quantifying the potential impact of research (or lack thereof), prioritizing research based on expected outcomes, and clearly showing how uncertainties in input data are expected to affect decision confidence and the ranking of alternative site management plans. As an analytical effort, the time and resources needed for a VoI analysis are often insignificant in comparison to the resources they can save by prioritizing and bounding the need for further physical research.
■
The authors declare no competing financial interest.
■
ACKNOWLEDGMENTS The authors would like to thank previous studies for the work performed on the Norwegian Grenland fjord system, especially the Opticap project (www.opticap.no), the SEDFLEX project, and the Norwegian Research Council that financed those studies. Thanks are also due to Kelsie Baker and Cate Fox-Lent for help with graphics, editing, and data curation, and to Professor Jeff Keisler who was instrumental in teaching and critiquing the VoI methods. The authors additionally thank Dr. Todd Bridges, Dr. Martin Schultz, and are grateful for financial support from the Dredging Operation Environmental Research (DOER) program of the US Army Corps of Engineers (USACE). Permission was granted by the USACE Chief of Engineers to publish this material.
■
ASSOCIATED CONTENT
* Supporting Information S
More detailed information is available about the preference function used in the PROMETHEE II Outranking MCDA method, background on the Grenland fjord region, and the input data and results from the SBLL model. This material is available free of charge via the Internet at http://pubs.acs.org.
■
REFERENCES
(1) Bridges, T. S.; Apitz, S. E.; Evison, L.; Keckler, K.; Logan, M.; Nadeau, S.; Wenning, R. J. Risk-based decision making to manage contaminated sediments. Integr. Environ. Assess. Manage. 2006, 2 (1), 51−58. (2) Förstner, U.; Salomons, W. Trends and challenges in sediment research 2008: The role of sediments in river basin management. J. Soils Sediments. 2008, 8 (5), 281−283. (3) Weber, R.; Gaus, C.; Tysklind, M.; Johnston, P.; Forter, M.; Hollert, H.; Heinisch, E.; Holoubek, I.; Lloyd-Smith, M.; Masunaga, S.; Moccarelli, P.; Santillo, D.; Seike, N.; Symons, R.; Torres, J. P. M.; Verta, M.; Varbelow, G.; Vijgen, J.; Watson, A.; Costner, P.; Woelz, J.; Wycisk, P.; Zennegg, M. Dioxin-and POP-contaminated sites Contemporary and future relevance and challenges. Environ. Sci. Pollut. Res. 2008, 15 (5), 363−393. (4) Sparrevik, M.; Breedveld, G. D. From ecological risk assessments to risk governance. evaluation of the Norwegian management system for contaminated sediments. Integr. Environ. Assess. Manage. 2010, 6 (2), 240−248. (5) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Seager, T. P.; Bridges, T.; Gardner, K. H.; Rogers, S. H.; Belluck, D. A.; Meyer, A. Multicriteria decision analysis: a comprehensive decision approach for management of contaminated sediments. Risk Anal. 2006, 26 (1), 61−78. (6) National Research Council. Contaminated Sediments in Ports and Waterways: Cleanup Strategies and Technologies; National Academies Press: Washington, DC, 1997. (7) Linkov, I.; Ames, M. R.; Crouch, E. A.; Satterstrom, F. K. Uncertainty in octanol−water partition coefficient: Implications for risk assessment and remedial costs. Environ. Sci. Technol. 2005, 39 (18), 6917−6922. (8) Ghosh, U.; Luthy, R. G.; Cornelissen, G.; Werner, D.; Menzie, C. A. In-situ sorbent amendments: A new direction in contaminated sediment management. Environ. Sci. Technol. 2011, 45 (4), 1163− 1168. (9) Linkov, I.; Seager, T. P. Coupling multi-criteria decision analysis, life-cycle assessment, and risk assessment for emerging threats. Environ. Sci. Technol. 2011, 45 (12), 5068−74. (10) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Seager, T. P.; Bridges, T.; Gardner, K. H.; Rogers, S. H.; Belluck, D. A.; Meyer, A. Multicriteria decision analysis: A comprehensive decision approach for management of contaminated sediments. Risk Anal. 2006, 26 (1), 61− 78. (11) Gamper, C. D.; Turcanu, C. On the governmental use of multicriteria analysis. Ecol. Econ. 2007, 62 (2), 298−307. (12) Linkov, I.; Satterstrom, F. K.; Kiker, G.; Batchelor, C.; Bridges, T.; Ferguson, E. From comparative risk assessment to multi-criteria decision analysis and adaptive management: Recent developments and applications. Environ. Int. 2006, 32 (8), 1072−1093.
AUTHOR INFORMATION
Corresponding Author
*E-mail: [email protected]. 9484
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
Environmental Science & Technology
Article
(13) Linkov, I.; Moberg, E. Multi-Criteria Decision Analysis: Environmental Applications and Case Studies; CRC Press: Boca Raton, FL, 2012. (14) Howard, R. A. The foundations of decision analysis. IEEE Trans. Syst. Sci. Cyber., 1968, 4 (3), 211−219. (15) Keeney, R. L.; Raiffa, H. Decisions with Multiple Objectives: Preferences and Value Tradeoffs; Wiley: New York, 1976. (16) Belton, V.; Stewart, T. J. Multiple Criteria Decision Analysis an Integrated Approach; Kluwer Academic: Boston MA, 2002. (17) National Research Council. Environmental Decisions in the Face of Uncertainty; The National Academies Press: Washington, DC, 2013. (18) Mendoza, G. A.; Martins, H. Multi-criteria decision analysis in natural resource management: A critical review of methods and new modelling paradigms. Forest Ecol. Manage. 2006, 230 (1−3), 1−22. (19) Alvarez-Guerra, M.; Canis, L.; Voulvoulis, N.; Viguri, J. R.; Linkov, I. Prioritization of sediment management alternatives using stochastic multicriteria acceptability analysis. Sci. Total Environ. 2010, 408 (20), 4354−4367. (20) Linkov, I.; Sahay, S.; Kiker, G.; Bridges, T.; Seager, T. P. Multicriteria decision analysis: A framework for managing contaminated sediments. In Strategic Management of Marine Ecosystems; Springer: Amsterdam, The Netherlands, 2005; pp 271−297. (21) Hyde, K.; Maier, H. R.; Colby, C. Incorporating uncertainty in the PROMETHEE MCDA method. J. Multi-Criteria Decis. Anal. 2003, 12 (4−5), 245−259. (22) Tervonen, T.; Figueira, J. R.; Lahdelma, R.; Dias, J. A.; Salminen, P. A stochastic method for robustness analysis in sorting problems. Eur. J. Operat. Res. 2009, 192 (1), 236−242. (23) Howard, R. A. Information value theory. IEEE Trans. Syst. Sci. Cybernetics, 1966, 2 (1), 22−26. (24) Raiffa, H. Decision Analysis: Introductory Lectures on Choices under Uncertainty; Addison-Wesley: Reading MA, 1968. (25) Keisler, J. M.; Collier, Z. A.; Chu, E.; Sinatra, N.; Linkov, I. Value of information analysis: The state of application. Environ. Syst. Decis. 2013, DOI: 10.1007/s10669-013-9439-4. (26) Linkov, I.; Bates, M. E.; Canis, L. J.; Seager, T. P.; Keisler, J. M. A decision-directed approach for prioritizing research into the impact of nanomaterials on the environment and human health. Nat. Nanotechnol. 2011, 6 (12), 784−787. (27) Brennan, A.; Kharroubi, S.; O’Hagan, A.; Chilcott, J. Calculating partial expected value of perfect information via Monte Carlo sampling algorithms. Med. Decis. Making 2007, 27 (4), 448−470. (28) Keilser, J. M. Value of information: Facilitating targeted information acquisition in decision processes. Environ. Systems and Decisions 2014, DOI: 10.1007/s10669-014-9493-6. (29) The Presidential/Congressional Commission on Risk Assessment and Risk Management. Framework for Environmental Health Risk Management, Final Report, Volume 1. Washington, DC, 1997; http:// oaspub.epa.gov/eims/eimscomm.getfile?p_download_id=36372. (30) Hubbard, D.; Samuelson, D. Modeling without Measurements: How the decision analysis culture’s lack of empiricism reduces its effectiveness. OR/MS Today 2009, 36 (5), 26−31, http://www.ormstoday.org/orms-10-09/frrisk.html. (31) Morgan, M. G.; Henrion, M. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis; Cambridge University Press: New York, 1990. (32) Felli, J. C.; Hazen, G. B. Do sensitivity analyses really capture problem sensitivity? An empirical analysis based on information value. Risk, Decis. Policy 1999, 4 (2), 79−98. (33) Sparrevik, M.; Barton, D. N.; Bates, M. E.; Linkov, I. Use of stochastic multi-criteria decision analysis to support sustainable management of contaminated sediments. Environ. Sci. Technol. 2012, 46 (3), 1326−1334. (34) Knutzen, J.; Bjerkeng, B.; Næs, K.; Schlabach, M. Polychlorinated dibenzofurans/dibenzo-p-dioxins (PCDF/PCDDs) and other dioxin-like substances in marine organisms from the Grenland fjords, S. Norway, 1975−2001: Present contamination levels, trends and species specific accumulation of PCDF/PCDD congeners. Chemosphere 2003, 52 (4), 745−760.
(35) Barton, D. N.; Navrud, S.; Bjørkeslett, H.; Lilleby, I. Economic benefits of large-scale remediation of contaminated marine sedimentsA literature review and an application to the Grenland fjords in Norway. J. Soils Sediments 2010, 10 (2), 186−201. (36) Saloranta, T. M.; Armitage, J. M.; Haario, H.; Næs, K.; Cousins, I. T.; Barton, D. N. Modeling the effects and uncertainties of contaminated sediment remediation scenarios in a Norwegian Fjord by Markov chain Monte Carlo simulation. Environ. Sci. Technol. 2007, 42 (1), 200−206. (37) Brans, J. P.; Vincke, P.; Mareschal, B. How to select and how to rank projects: The PROMETHEE method. Eur. J. Operat. Res. 1986, 24, 228−238.
9485
dx.doi.org/10.1021/es500717t | Environ. Sci. Technol. 2014, 48, 9478−9485
