HHS Public Access Author manuscript Author Manuscript
Value Health. Author manuscript; available in PMC 2017 August 02. Published in final edited form as: Value Health. 2017 February ; 20(2): 185–192. doi:10.1016/j.jval.2016.11.013.
Evaluating Frameworks that Provide Value Measures for Health Care Interventions Jeanne S. Mandelblatt, MD, MPH*, Scott D. Ramsey, MD, PhD**, Tracy A. Lieu, MD, MPH***, and Charles E. Phelps, PhD**** *Georgetown
University Medical Center, Lombardi Comprehensive Cancer Center, Department of Oncology, Washington, DC
Author Manuscript
**Fred
Hutchinson Cancer Research Center, Seattle, Washington
***Division
of Research, Kaiser Permanente Northern California, Oakland, California
****University
of Rochester, Rochester, NY
Abstract
Author Manuscript Author Manuscript
The recent acceleration of scientific discovery has led to greater choices in health care. These new technologies, diagnostic tests, and pharmaceuticals have widely varying impact on patients and populations in terms of benefits, toxicities, and costs, stimulating a resurgence of interest in the creation of frameworks intended to measure value in health. Many of these are offered by providers and/or advocate organizations with expertise and interest in specific diseases (e.g., cancer, heart disease). To help assess the utility of and potential biases embedded in these frameworks, we created an evaluation taxonomy with seven basic components: (1) define the purpose; (2) detail the conceptual approach, including perspectives, methods for obtaining preferences of decision makers (e.g., patients), and ability to incorporate multiple dimensions of value; (3) discuss inclusions and exclusions of elements included in the framework, and whether the framework assumes clinical intervention or offers alternatives such as palliative care or watchful waiting; (4) evaluate data sources and their scientific validity; (5) assess the intervention’s effect on total costs of treating a defined population; (6) analyze how uncertainty is incorporated; and (7) illuminates possible conflicts of interest among those creating the framework. We apply the taxonomy to four representative value frameworks recently published by professional organizations focused on treatment of cancer and heart disease and vaccine use. We conclude that each of these efforts has strengths and weaknesses when evaluated using our taxonomy, and suggest pathways to enhance the utility of value-assessing frameworks for policy and clinical decision-making.
Keywords Value frameworks; Cost-effectiveness; Multi-attribute decision analysis
Address correspondence to: Charles E. Phelps, 30250 S Highway 1, Gualala, CA 95445, [email protected].
Mandelblatt et al.
Page 2
Author Manuscript
INTRODUCTION The explosion of scientific knowledge, discovery, and technology began a new era of health care, wherein deployment of health care interventions depends upon personal, lifestyle, genetic, and molecular disease factors. While these advances have the potential to improve health, they increase the complexity of analyses to determine value at reasonable and affordable costs.(1–8)
Author Manuscript
Faced with new health care interventions and escalating costs of care, many professional organizations have developed frameworks to measure and/or rank the value of novel interventions. (3;7;9–22). Some of these new frameworks use components of costeffectiveness analysis, while others create de novo value measures. It remains unclear how such frameworks will guide clinical practice policy, and whether they offer particular advantages over traditional approaches such as cost-effectiveness analysis. We summarize a taxonomy of principles -- expanding upon cost-effectiveness theory -- that provide an approach to assess different value frameworks. Next, we apply this taxonomy to evaluate exemplar value frameworks to illustrate strengths, gaps, and potential concerns. We intend that this synthesis stimulate discussion about best practices and use of value metrics in policy and clinical decisions. We derived these principles in our synthesis of standard practice for the reporting of cost-effectiveness analyses, (23–25), recent ISPOR summaries of multiple criteria decision analysis (MCDA) reporting guidelines, (26) standard medical journal requirements for conflict-of-interest disclosures (27), standard Cochrane review methods (28) and our own analysis. While we draw many of these ideas from others, assembling them to evaluate models that estimate the value of health care is novel.
Author Manuscript
VALUE TAXONOMY Value frameworks can assist in public health prioritization, practice guidelines, formulary or related resource expenditure decisions (e.g., purchase of equipment or coverage of a specific drug), and patient-physician decision-making about a specific care choice. Table 1 describes our taxonomy to evaluate these value frameworks. We intend this to apply to these different purposes, but specific elements may be more or less salient for different uses of the value framework. The following narrative highlights the implications of these varying purposes when using this taxonomy to evaluate value frameworks. Conceptual Approach, Perspective, and Preferences in Measuring Value
Author Manuscript
Value frameworks have evolved, in part, to provide greater standardization and transparency in health care decision-making. (29) Further, formal frameworks can minimize biases of intuitive decision making from issues of framing of the problem, risk analysis, intertemporal tradeoffs, and others. (30) We began our taxonomy in the formal framework of cost-effectiveness analysis (CEA). CEA analyses most commonly use the societal perspective, but this perspective usually does not correspond to that held by individual participants in health care decisions – patients, providers, payers, public policy decision makers, or owners of resources used to create
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 3
Author Manuscript
health care interventions, nor does it consider community well-being or other important policy issues such as the distribution of benefits and costs across various subsets of the population (among other omissions). No single perspective can represent the interests of all participants in value-based decisions. For example, physicians generally advise patients using a patient perspective, but may introduce the societal perspective in their role as steward of societal resources (31) or, in some cases, their own professional organization’s or personal financial situation. Thus, we believe that multiple perspectives should be considered, varying by individual or organization assessing the intervention, consistent with the most recent recommendations for reporting cost-effective analyses. (24;25)
Author Manuscript
CEA can also be useful in developing value frameworks since it provides data on intermediate events of importance to judging the value of an intervention, such as toxicity rates and overall survival, measured in life-years saved, quality-adjusted life years, or disability adjusted life years. (23–25; 32; 33) Compared with the existing care-standard, the ratio of added costs to added gains in health benefits-- the incremental cost-effectiveness ratio (ICER) – stands as one value criterion. However, no consensus exists on cut-points to determine “acceptability” of a given ICER, and both illness severity and the number of people affected can influence “cut point” choices.
Author Manuscript
CEA also does not typically capture less tangible factors such as fear associated with diseases like HIV, cancer, degenerative neurological conditions, or mental illness, since these do not represent distinct health outcome states for which utility measurements exist. In concept, CEA could incorporate such issues, but in many cases, the data requirements would be impossibly complex. Full CEA evaluation requires population-based utility level estimates for every state of nature in the model. Capturing such things as fear of the diseases, contagion effects, restrictions on freedom, and modification of religious rites (all arising in the response to Ebola virus) provide examples of issues that CEA cannot meaningfully capture. While grounded in CEA, our taxonomy extends beyond this foundation. CEA does not generally consider many other attributes of decision-making that affect use of health care interventions. For example, CEA cannot address the distribution of health burdens that might fall particularly heavily on disadvantaged populations (e.g., sickle cell anemia in AfricanAmericans), or whether some aspects of a medical intervention have important moral or ethical components, (e.g., abortion after detection of a fetal genetic mutation with profound consequences.
Author Manuscript
Applying CEA for decision making brings another issue to the foreground: a new technology may pass standard “cutoff” requirements but then overwhelm the available budget – the issue of “affordability.” CEA methods currently are not equipped to deal with this issue. Multi-criteria decision analysis tools offer one approach to formally include these “other issues” into the value framework. These tools include such approaches as the Analytic
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 4
Author Manuscript
Hierarchy Process (34) and the Multi-Attribute Utility Theory, (35–36), both used in recent health care decision processes. (37–40) These models bring to the foreground an issue that does not arise in CEA: how does one balance among competing goals in a value model? Multi-criteria decision analysis methods use various methods to elicit weights attached to each (potentially) competing goal. With a single decision maker, these methods work well in general (34;36), but best methods to combine individual into group preferences remain elusive. If group estimates are not derived from relevant population samples, our framework suggests that analysts should provide good justification and clear logic for using alternative sources.
Author Manuscript
Our taxonomy proposes that value frameworks incorporate preferences, examine multiple perspectives, and are transparent. In addition, we suggest that value frameworks should consider distribution and ethical issues and other pertinent factors by using multi-criteria decision analysis methods. Accounting for all Components of the Health Condition and Intervention
Author Manuscript
Value frameworks should consider many aspects of each disease process and all short-term and downstream benefits and harms of interventions. For example, in cancer, including only one cost component, such as drug acquisition costs for a new treatment, would bias the overall assessment by ignoring other costs, such as those of drug administration, supportive care, and other aspects of therapy. In treatment of HIV, cancer, and other conditions, drugs may have significant toxicity that can affect their value. Screening tests like mammography can save lives, but also carry with them meaningful rates of false positive results and overdiagnosis. These issues should enter a full value assessment, not only in health status outcomes and costs but also in emotional burdens created. (41;42) Some value frameworks treat all harms as equal in consequence to patients or other decision-makers. For example, when the intervention is a drug treatment, rates of all side effects might be used in comparing the value of two therapies. This creates biases when the intensities and/or durations of the side effects differ, or when some effects have greater consequence than others to individual patients (e.g., nausea/vomiting vs. fatigue vs. increased risk of iatrogenic death). In best practice, our taxonomy recommends that individual preferences on these issues inform the overall assessments of alternative therapies, and that when infeasible, population-based estimates serve as an alternative.
Author Manuscript
Current value frameworks tend to focus on active therapy; e.g., drugs, surgery, and/or invasive tests. More recently, for certain diseases, such as early, non-aggressive prostate cancers, with invasive and potentially toxic standard-care therapies, alternatives now include no therapy (active surveillance). In the advanced disease setting, palliative interventions are becoming increasingly considered. If value frameworks do not explicitly consider these less aggressive approaches (i.e., only comparing novel therapies with standard therapy), the resulting decision may be biased towards therapeutic action, possibly conflicting with patient preferences. Therefore, we suggest that value frameworks compare all feasible options for patients, not just those using therapeutic intervention.
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 5
Author Manuscript
What are the sources of and process for use of data in the framework? Does this process conform to accepted methods for evidence evaluation? Several accepted methods and hierarchies evaluate the quality of evidence about health care intervention effects. Randomized controlled trials generally provide stronger evidence than other methods (e.g., historical comparisons, case-control studies, and others), and sample sizes and research design in all studies affect precision and accuracy of parameter estimates. (28)
Author Manuscript
In cases where good quality, empirically based evidence on key issues in the value framework is not available, expert judgment may provide the only realistic option. Using expert-based estimates creates biases when the experts have a professional or financial stake in choices being evaluated (e.g., profit from sales of a device or drug, or enhanced reimbursement for delivery of a new agent -- see below for further discussion). When possible, valuation models should use experts not in conflict-of-interest situations and should clearly identify when any such conflicts exist. Does the framework include the full economic consequences of all intervention components? As in standard CEA, our taxonomy recommends that value frameworks include the full cost of all the components and consequences of an intervention. For example, in frameworks to evaluate drug treatments, economic impacts would include drug costs, provider time and delivery costs, staff time, facilities and equipment overhead and costs, costs of treatmentrelated side-effects and supportive care medications required, patient time and travel costs, and costs of all downstream events until death from the disease or other causes. To the extent that frameworks exclude portions of these costs, they result in biased recommendations.
Author Manuscript
Many value frameworks have different perspectives and include different costs. Patientperspective frameworks, for example, might normally include only copayments made by patients. Payer-centric frameworks may focus only on the costs and savings to the health care provider. These approaches can lead to sub-optimal value-based decisions from a societal perspective. Costs also need to be considered in the context of extant budgets. For example, hepatitis C medications have cost-effectiveness ratios within the range of other accepted interventions, but still have high absolute costs that may overwhelm pre-established budgets.
Author Manuscript
Projected budget impact has played a key role in some health care system decisions about these medications, mostly appearing as an independent discussion of “budget impact” in addition to standard cost-effectiveness analysis. (17) We believe that this approach misses a key point: budgets and acceptable levels for cost-effectiveness ratios cannot be considered independently. Introducing a new (and expensive) treatment adds budget pressure, which invariably will raise the value of incremental resources (what economists call the “shadow price”). The proper solution will likely involve reassessment of the existing budget and reexamination of interventions that cannot compete (in a cost-effectiveness paradigm) with the newly tightened budget situation.
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 6
How does the framework account for uncertainty in effect or cost data?
Author Manuscript
Many analytic methods incorporate effects of uncertainty on the conclusions of a value recommendation. A key aspect of characterizing uncertainty relates to identification of gaps in knowledge or evidence about aspects of care that could change the value conclusions. Our taxonomy does not specify a particular method to identify the impact of uncertainty, but rather that uncertainty and gaps in data be explicitly evaluated. Conflicts of Interest
Author Manuscript
Professional specialty organizations often have the best knowledge about a particular set of health care interventions, and thus are well-placed to develop value frameworks, but may also have conflicts of interest in these issues. Value assessments may be subject to bias if financial incentives favor more (and costlier) interventions in certain delivery settings (e.g., fee for service vs. managed care or accountable care organizations). These issues extend also to the use of facilities owned in part or whole, directly or indirectly, by decision-making providers. Beyond financial incentives, the purpose of specialist organizations is to advocate for professional authority and action, which may further bias these sponsors toward interventions based in their specialties (such as chemotherapy, which is rooted in oncology, compared with palliative care, with separate specialty affiliations). Thus, we should consider if value framework’s sponsoring organization has a professional stake in the process, and if so, whether the framework potentially biases the results in favor of the sponsor’s position. Examples of Recent Frameworks
Author Manuscript
Value frameworks have recently emerged from different medical specialty groups representing oncology, (9;16–22) cancer screening, (10) radiation oncology, (11;12) cardiology,(7) surgery, (13) pediatrics, (3) and infectious disease and vaccines. (14) To demonstrate the application of our taxonomy, we selected four exemplar value frameworks spanning several specialties (Table 2). Oncology—The oncology frameworks largely focus on the value of new drugs, driven by the more rapid escalation of drug costs relative to other components of cancer care; and the impact of drug costs on patient co-payments. (31;43;44) In other disease specialties, new treatment paradigms, and expensive drugs and high priced technology have also stimulated the development of value frameworks.
Author Manuscript
The American Society of Clinical Oncology’s (ASCO) Task Force on Value in Cancer Care (18;19) and the National Comprehensive Cancer Center Network (16) both developed value assessments for systemic therapy drugs. Comments about this work recognize that the intended use centers on patient-based treatment decisions rather than a societal perspective. These frameworks followed many elements recommended in our taxonomy, but omit several key value components. First, they only focus on clinical outcomes related to survival and drug costs. Second, by limiting the framework to clinical trial evidence about episodes of drug treatment, they also do not permit assessment of the value of a given therapy in the general population. (45) Third, the framework excludes other important costs of care, such
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 7
Author Manuscript
as future need for radiation, repeat surgery, or palliative care. Fourth, many relevant outcomes needed to determine value from a societal or patient perspective are excluded, including quality of life impact of therapy, (46) perhaps as a result of omission of such data in the original clinical trials of efficacy. (47) The revised ASCO framework (19) endeavors to capture all types of toxicity, but can only succeed to the extent lower-grade adverse events are reported. Next, the method used to assign value points is both arbitrary and relies primarily on opinions of providers rather than samples of patients or their families. Finally, there is no accounting for uncertainty in the underlying data or the result, giving a false sense of precision when comparing among treatments.
Author Manuscript
Other oncology frameworks include Drug Abacus, (48) the European Society for Medical Oncology’s Magnitude of Clinical Benefit Scale, (22) and the pan-Canadian Oncology Drug Review (pCODR), (20;21). These frameworks seek to inform policy rather than clinical decision-making. They too focus primarily on drugs, with Drug Abacus evaluating cost of drug development and production vs. actual retail pricing. The pCODR is the only value framework in oncology that includes a societal perspective, patient stakeholders, components of a CEA framework including QALYs, considers evidence beyond trials, and determines how the short-term budget impact of new interventions affect the Canadian health care budget.
Author Manuscript
These oncology frameworks were developed by oncologists (with other researchers) who have professional and financial stakes in the drugs they prescribe, so that the possibility of conflict of interest arises. A recent review of oncology drug value frameworks was supported by a pharmaceutical company. The authors declared that they owned stock or consulted for this company. (2)
Author Manuscript
Cardiology—The American College of Cardiology and the American Heart Association (ACC-AHA) developed another recent value framework (7) to inform their practice guidelines. This framework rates both the quality of evidence and the value of care, with the latter based on a CEA framework. Based on CEA results, they developed a score for value from low to high using the World Health Organization WHO-CHOICE project, including the use of cutoff values of 1X to 3X per capita Gross Domestic Product. (49) Notably, the writing committee had fewer than 50% of members with industry relationships, and all of these relationships were published. Although the primary data resources for this evaluation come from independent entities, we note that when committee members are the providers of therapies, they may have inherent conflicts of interest that can cause deliberate or subconscious bias. In general, the ACC-AHA framework robustly addresses almost all of the key components of our taxonomy but could be improved by more explicit definitions of intervention components, and approaches to uncertainty. Vaccines—Frameworks for evaluation of vaccines against infectious diseases present many complexities that do not arise with other medical interventions, including the need to capture spread of disease in the population, changes in population demographics due to aging and migration, the natural history of immunity, herd protection, community well-being, the need
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 8
Author Manuscript
to model the population equilibrium, the lack of post-market surveillance for adverse events, and the ability of the vaccine to prevent disease or disease related effects (e.g., post-herpetic neuralgia). (6,50–51). These factors present unique challenges in defining the perspective, individual preferences versus effects of treatment of one person on unaffected persons (i.e., herd effects), and time horizon for a value framework. (39) Despite this complexity, the European guidelines for vaccine value assessment considers virtually all of the components in our taxonomy. (52) These guidelines explicitly address conflicts of interest: the developers excluded employees of drug or vaccine companies and required members to disclose potential conflicts.
CONCLUSION
Author Manuscript
Although the movement to examine the value of health care is rapidly gaining prominence, no consensus exists on value’s structure and components. Wider implementation of value frameworks may be possible by further use of data from clinical trials and information made available through electronic records.(46;53)) It will also be useful to include education about value frameworks in medical and public health training.(454;55) Our taxonomy could support these efforts and extends prior overviews (2;25) by including a comprehensive set of metrics grounded in cost-effectiveness analysis.
Acknowledgments This work was funded by grant #U01CA183081 from the National Cancer Institute at the National Institutes of Health. This work was also supported in part by National Cancer Institute grants R35 CA197289 and U01 CA152958.
Author Manuscript
References
Author Manuscript
1. Lawler M, French D, Henderson R, et al. Shooting for the Moon or Flying Too Near the Sun? Crossing the Value Rubicon in Precision Cancer Care. Public Health Genomics. 2016; 19(3):132–6. [PubMed: 27238737] 2. Chandra A, Shafrin J, Dhawan R. Utility of Cancer Value Frameworks for Patients, Payers, and Physicians. JAMA. 2016 May 17; 315(19):2069–70. [PubMed: 27187295] 3. Swank MF, Brennan LK, Gentry D, Kemner AL. Using frameworks to diagram value in complex policy and environmental interventions to prevent childhood obesity. J Public Health Manag Pract. 2015 May; 21(Suppl 3):S116–S120. [PubMed: 25828214] 4. Rogowski W, Payne K, Schnell-Inderst P, et al. Concepts of ‘personalization’ in personalized medicine: implications for economic evaluation. Pharmacoeconomics. 2015 Jan; 33(1):49–59. [PubMed: 25249200] 5. Committe on valuing community-based, non-clinical prevention programs; board on population health and public health practice; Institute of Medicine. Washington (DC): National Academies Press (US); 2012 Oct. An integrated framework for assessing the value of community-based prevention. 10-29-2012 6. Pronk NP, Hernandez LM, Lawrence RS. An integrated framework for assessing the value of community-based prevention: a report of the Institute of Medicine. Prev Chronic Dis. 2013; 10:120323. [PubMed: 23411033] 7. Anderson JL, Heidenreich PA, Barnett PG, et al. ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines. Circulation. 2014 Jun 3; 129(22):2329–45. [PubMed: 24677315]
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 9
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
8. Porter ME. What is value in health care? N Engl J Med. 2010 Dec 23; 363(26):2477–81. [PubMed: 21142528] 9. de Souza JA, Seiwert TY. A value framework in head and neck cancer care. Am Soc Clin Oncol Educ Book. 2014:e304–e309. [PubMed: 24857117] 10. Harris RP, Wilt TJ, Qaseem A. A value framework for cancer screening: advice for high-value care from the American College of Physicians. Ann Intern Med. 2015 May 19; 162(10):712–7. [PubMed: 25984846] 11. Thaker NG, Pugh TJ, Mahmood U, et al. Defining the value framework for prostate brachytherapy using patient-centered outcome metrics and time-driven activity-based costing. Brachytherapy. 2016 May; 15(3):274–82. [PubMed: 26916105] 12. Teckie S, McCloskey SA, Steinberg ML. Value: a framework for radiation oncology. J Clin Oncol. 2014 Sep 10; 32(26):2864–70. [PubMed: 25113759] 13. Yount KW, Turrentine FE, Lau CL, Jones RS. Putting the value framework to work in surgery. J Am Coll Surg. 2015 Apr; 220(4):596–604. [PubMed: 25728143] 14. Barnighausen T, Bloom DE, Cafiero-Fonseca ET, O’Brien JC. Valuing vaccination. Proc Natl Acad Sci U S A. 2014 Aug 26; 111(34):12313–9. [PubMed: 25136129] 15. Mitera G, Earle C, Latosinsky S, et al. Choosing Wisely Canada cancer list: ten low-value or harmful practices that should be avoided in cancer care. J Oncol Pract. 2015 May; 11(3):e296– e303. [PubMed: 25980018] 16. National Comprehensive Cancer Network. [Accessed August 1, 2016] Available at: http:// www.nccn.org/evidenceblocks/pdf 17. Institute for Clinical and Economic Review. [Accessed August 1, 2016] Methods. The ICER approach. Available at: http//:icer-review.org/methodology/icers-methods/ 18. Schnipper LE, Davidson NE, Wollins DS, et al. American Society of Clinical Oncology Statement: A Conceptual Framework to Assess the Value of Cancer Treatment Options. J Clin Oncol. 2015 Aug 10; 33(23):2563–77. [PubMed: 26101248] 19. Schnipper LE, Davidson NE, Wollins DS, et al. Updating the American Society of Clinical Oncology Value Framework: Revisions and Reflections in Response to Comments Received. J Clin Oncol. 2016 May 31. 20. Cheung MC, Sabharwal M, Chambers A, et al. Multiple Dimensions of Value: Evaluative Frameworks for New Cancer Therapies. J Clin Oncol. 2016 Apr 20; 34(12):1428–9. [PubMed: 26884567] 21. Hoch JS, Sabharwal M. Informing Canada’s cancer drug funding decisions with scientific evidence and patient perspectives: the Pan-Canadian Oncology Drug Review. Curr Oncol. 2013 Apr; 20(2): 121–4. [PubMed: 23559876] 22. Cherny NI, Sullivan R, Dafni U, et al. A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Ann Oncol. 2015 Aug; 26(8):1547–73. [PubMed: 26026162] 23. Gold, MR.Siegel, JE.Russell, LB., Weinstein, MC., editors. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996. 24. Neumann, PJ.Sanders, GD.Russell, LD.Siegel, JD., Ganiats, TG., editors. Cost-Effectiveness in Health and Medicine. 2. New York: Oxford University Press; 2017. 25. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA. 2016; 316(10):1093–1103. DOI: 10.1001/jama.2016.12195 [PubMed: 27623463] 26. Marsh K, IJzerman M, Thokala P, et al. Multiple Criteria Decision Analysis for Health Care Decision Making—Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016; 19:125–137. [PubMed: 27021745] 27. [last accessed 10 October, 2016] http://www.icmje.org/recommendations/browse/roles-andresponsibilities/author-responsibilities--conflicts-of-interest.html), standard statistical evaluation tools for medical research 28. [last accessed 10 October, 2016] http://training.cochrane.org/handbook Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 10
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
29. Madhavan G, Phelps CE, Colwell RR, Rappuoli R, Fineberg HV. In Plain View: A Transparent Systems Approach for Enhancing Health Policy Decisions. The Bridge: Linking Engineering and Society. 2016 Summer;46(2):5–10. 30. Kahneman, D. Thinking, Fast and Slow. New York: Farrar, Strass and Giroux; 2011. 31. Schnipper LE, Bastian A. New Frameworks to Assess Value of Cancer Care: Strengths and Limitations. Oncologist. 2016 Jun; 21(6):654–8. [PubMed: 27245568] 32. [Accessed August 1, 2016] http://www.who.int/healthinfo/global_burden_disease/metrics_daly/en/ 33. Malone DC, Berg NS, Claxton K, et al. International Society for Pharmacoeconomics and Outcomes Research Comments on the American Society of Clinical Oncology Value Framework. J Clin Oncol. 2016 Jun 13. 34. Saaty TL. Decision making with the analytic hierarchy process. Int J Serv Sciences. 2008; 1(1):83– 98. 35. Edwards W, Barron FH. SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement. Organizational Behavior and Human Decision Processes. 1994; 60:306–325. 36. Barron FH, Barrett BE. Decision Quality Using Ranked Attribute Weights. Management Science. Nov; 1996 42(11):1515–152. 37. Schmidt K, Aumann I, Hollander I, et al. Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting. BMC Med Inform Decis Mak. 2015; 15:112. [PubMed: 26703458] 38. Madhavan, G.Sangha, K.Phelps, C., et al., editors. Ranking Vaccines: a prioritization framework: phase I: demonstration of concept and a software blueprint/. Washington, DC: National Academies Press (US); 2012. 39. Madhavan, G.Sangha, K.Phelps, C., et al., editors. Ranking vaccines: a prioritization softward tool: phase II: prototype of a decision-support system. Washington, DC: National Academies Press (US); 2013. 40. Madhavana, G.Phelps, C.Rappuoli, R., et al., editors. Ranking Vaccines: Applications of a Prioritization Software Tool: Phase III: Use Case Scenarios and Data Framework. Washington, DC: National Academy Press; Dec 15. 2014 41. Nelson HD, Fu R, Cantor A, et al. Effectiveness of Breast Cancer Screening: Systematic Review and Meta-analysis to Update the 2009 U.S. Preventive Services Task Force Recommendation. Ann Intern Med. 2016 Feb 16; 164(4):244–55. [PubMed: 26756588] 42. Nelson HD, Pappas M, Cantor A, et al. Harms of Breast Cancer Screening: Systematic Review to Update the 2009 U.S. Preventive Services Task Force Recommendation. Ann Intern Med. 2016 Feb 16; 164(4):256–67. [PubMed: 26756737] 43. Henry J. Kaiser Family Foundation. [Accessed May 18, 2016] Employer health benefits survey. 2015. Available at http://kff.org/report-section/ehbs-2015-summary-of-findings/ 44. Shankaran V, Jolly S, Blough D, Ramsey SD. Risk factors for financial hardship in patients receiving adjuvant chemotherapy for colon cancer: a population-based exploratory analysis. J Clin Oncol. 2012 May 10; 30(14):1608–14. [PubMed: 22412136] 45. Angelis A, Kanavos P. Critique of the American Society of Clinical Oncology Value Assessment Framework for Cancer Treatments: Putting Methodologic Robustness First. J Clin Oncol. 2016 Jun 13. 46. Basch E. Toward a Patient-Centered Value Framework in Oncology. JAMA. 2016 May 17; 315(19):2073–4. [PubMed: 27187297] 47. Li J, Asabere A, Kelly S, et al. Application Of Asco Value Framework Evaluations Of Net Health Benefit For Oncology Drugs Launched In The United States Between 2013 And 2015. Value Health. 2015 Nov.18(7):A489. 48. [Accessed August 1, 2016] Evidence driven drug pricing project. Accessed at: http:// www.drugabacus.org/ 49. [Accessed August 1, 2016] Choosing interventions that are cost-effective project. http:// www.who.int/choice/cost-effectiveness/en/ 50. Ultsch B, Damm O, Beutels P, et al. Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a European Vaccine Economics Community. Pharmacoeconomics. 2016 Mar; 34(3):227–44. [PubMed: 26477039] Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 11
Author Manuscript
51. Ultsch B, Damm O, Beutels P, et al. Methods for Health Economic Evaluations of Vaccines Results from an International Expert-Workshop. Value Health. 2014 Nov.17(7):A552. 52. Ultsch B, Damm O, Perleth M, Wichmann O. Health Economics In Immunization DecisionMaking - Results From A Systematic Literature Research and A Stakeholder Symposium In Germany. Value Health. 2015 Nov.18(7):A569. 53. Fayanju OM, Mayo TL, Spinks TE, et al. Value-Based Breast Cancer Care: A Multidisciplinary Approach for Defining Patient-Centered Outcomes. Ann Surg Oncol. 2016 Aug; 23(8):2385–90. [PubMed: 26979306] 54. Patel MS, Davis MM, Lypson ML. The VALUE Framework: training residents to provide valuebased care for their patients. J Gen Intern Med. 2012 Sep; 27(9):1210–4. [PubMed: 22573146] 55. Phelps C, Madhavan G, Rappuoli R, et al. Strategic Planning in Population Health and Public Health Practice: A Call to Action for Higher Education. Milbank Q. 2016 Mar; 94(1):109–25. [PubMed: 26994711]
Author Manuscript Author Manuscript Author Manuscript Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 12
Author Manuscript
Summary Summarizes principles that could be used to evaluate value frameworks and applies these to analyze strengths, gaps, and potential concerns about some recently published frameworks.
Author Manuscript Author Manuscript Author Manuscript Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 13
Table 1
Author Manuscript
Proposed Taxonomy for Evaluation of Frameworks Assessing the Value of Health Care Interventions Category
Component detail
Purpose Is the purpose defined and are the elements of the framework consistent with the purpose? Conceptual approach, perspective, and preferences Is the value structure of the framework clear and transparent? If it requires assumptions, are do they have general and clinical face validity? What perspective does the framework use (e.g., societal, patient, provider, payer)? Is the perspective made explicit or embedded in the framework? Does the framework include standard methods of measuring value, such as cost-effectiveness analysis?
Author Manuscript
Does the framework consider multiple attributes of medical interventions such as individual preferences, equity and the distribution of health care benefits and costs, and issues involving externalities (such as contagious diseases), and the value of scientific breakthrough? If the framework includes multiple dimensions of value, are the weights for each component population-based or expert-provided, and are the methods for eliciting such weights both clearly stated and methodologically sound? Does the framework allow individual patient preferences? If so, are they elicited with methods known to be free of bias and produce reliable results? Intervention components and comparators Does the framework include all components and consequences of the intervention, or merely a portion of those (e.g., drug acquisition costs)? Does the framework aggregate or disaggregate such things as toxicity or other side effects of intervention? Does the framework assume as a baseline that some intervention will be provided, or does it allow for “watchful waiting” or “palliative care” as an option? Data sources
Author Manuscript
Are clinical and other data derived from expert opinion, population surveys, or other sources; are the sources made clear, and is the process replicable? Economics/costs What is the effect of the intervention on the total cost of treating a defined population, including whether inclusion of the intervention will increase, decrease or leave unchanged the total costs of care for that population? Uncertainty and identification of important gaps
Is uncertainty in data about costs or effects considered in conclusions about value? Are there gaps in knowledge about aspects of care that could change the value rating?
Conflicts of interest
Does the sponsoring organization have a financial stake in the process, and if so, is this declared? Does the framework bias the results in favor of the sponsor’s financial position?
Author Manuscript Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 14
Table 2
Author Manuscript
Examples of Application of Proposed Taxonomy for Evaluation of Frameworks Assessing the Value of Health Care Interventions Protocol Element
American Society of Clinical Oncology (19)
National Comprehensive Cancer Network (16)
American Heart AssociationAmerican College of Cardiology (7)
European Vaccine Working Group (33)
Stated purpose
Patient-physician drug treatment decisions; separate framework for adjuvant vs. metastatic Rx
Patient-physician drug treatment decisions for surgery, radiation, and drugs
Inform practice guidelines and performance measures
Public health policy decisions
Framework consistent with purpose
Yes; will need to develop format for use in practice
Yes; included with online treatment guidelines
Yes
Yes
Clear and transparent conceptual framework
No
No
Yes
Yes
Do assumptions have empirically supported clinical basis?
Bonus points subjectively assigned
Scoring system subjective/expert opinion
N/A
Yes
Is the perspective made explicit
Implies clinical decision making perspective
Implies clinical decision making perspective
Societal, with separate analyses with patient perspectives for clinical decisions; consider subgroups
Societal with long term population outcomes
References existing methods of assessing value, allowing comparison across diseases
De novo method: net health benefit score and costs per drug treatment course. “Bonus points” and toxicity included in net benefits score for a new regimen vs. control therapy Points for benefit are assigned based on categories of median survival Does not allow comparisons across multiple available regimens
De novo method: evidence block score from 1–5 based on subjective assessment of benefits Does not allow comparisons across multiple available regimens
Uses costs per QALY to grade value low to high value using WHOCHOICE framework; includes uncertain value category. Value categories can be based on societal cut-points or a percent of GDP
Uses dynamic agent based models and costs per QALY May consider single or multiple vaccines
Considers multi-decision attributes (preferences, equity, distribution, externalities)
Preferences included in metastatic framework only, but based on expert designation of “bonus points” for symptom palliation, and QOL, longer life with metastases
unknown
Considers preferences, distribution, and budget impacts
Explicitly considers use of multi-attribute theory to assist with decisions
Only clinical benefit as gain in survival, major toxicity, and costs of drugs and supportive care for drug delivery; patient copay for drugs included. Other effects and costs excluded. Update to framework includes lower grade toxicities. Only examines one drug and one control drug
Effectiveness, safety (i.e., major toxicity), quality of evidence, consistency of evidence, and affordability No direct comparisons among treatment modalities
Implicit in CEA framework, but not specified; framework used as a broad template
Very broad horizon and effects considered in a CEA framework
Purpose
Approach
Author Manuscript Author Manuscript
Intervention components and comparators All components and comparators considered
Author Manuscript
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 15
Author Manuscript
Protocol Element
American Society of Clinical Oncology (19)
National Comprehensive Cancer Network (16)
American Heart AssociationAmerican College of Cardiology (7)
European Vaccine Working Group (33)
Toxicity/side effects considered
Yes, but not in costs for hospitalization, etc.
For some regimens
Implicit but not specified
Yes
Considers no treatment as a comparator (e.g., “watchful waiting” or “palliative care” )
No
For some treatment options (surgery, radiation, drugs)
Unknown
N/A
RCT and expert opinion
RCT and expert opinion
Specifies US data. Data rated from A–C based on a formal review of the evidence quality; includes randomized trials and observational data
Vaccine trials using intention to treat results
All costs stemming from the intervention included
Only drug costs; drug delivery and supportive care for drug delivery Includes patient co-pays for drugs but not time costs
Drug cost, supportive care, and drug administration, and toxicity costs; costs ranked subjectively on an “affordability scale” from 1–5 (inexpensive to very expensive)
Includes all costs in CEA framework
Includes all costs in CEA framework
budget impact assessed
Budget impact not considered
Budget impact not considered
??
??
Uncertainty and information gaps considered
Not considered
Not considered
Not explicitly considered; does include an unknown category for value
Yes
Conflicts of interest
Not explicitly addressed. Disclosures included in journal article. Potential conflicts since value primarily determined by oncologists who deliver (and bill) care
Not explicitly addressed. Potential conflicts since value primarily determined by oncologists who deliver (and bill) care
Explicitly addressed;
Value Health. Author manuscript; available in PMC 2017 August 02. Published in final edited form as: Value Health. 2017 February ; 20(2): 185–192. doi:10.1016/j.jval.2016.11.013.
Evaluating Frameworks that Provide Value Measures for Health Care Interventions Jeanne S. Mandelblatt, MD, MPH*, Scott D. Ramsey, MD, PhD**, Tracy A. Lieu, MD, MPH***, and Charles E. Phelps, PhD**** *Georgetown
University Medical Center, Lombardi Comprehensive Cancer Center, Department of Oncology, Washington, DC
Author Manuscript
**Fred
Hutchinson Cancer Research Center, Seattle, Washington
***Division
of Research, Kaiser Permanente Northern California, Oakland, California
****University
of Rochester, Rochester, NY
Abstract
Author Manuscript Author Manuscript
The recent acceleration of scientific discovery has led to greater choices in health care. These new technologies, diagnostic tests, and pharmaceuticals have widely varying impact on patients and populations in terms of benefits, toxicities, and costs, stimulating a resurgence of interest in the creation of frameworks intended to measure value in health. Many of these are offered by providers and/or advocate organizations with expertise and interest in specific diseases (e.g., cancer, heart disease). To help assess the utility of and potential biases embedded in these frameworks, we created an evaluation taxonomy with seven basic components: (1) define the purpose; (2) detail the conceptual approach, including perspectives, methods for obtaining preferences of decision makers (e.g., patients), and ability to incorporate multiple dimensions of value; (3) discuss inclusions and exclusions of elements included in the framework, and whether the framework assumes clinical intervention or offers alternatives such as palliative care or watchful waiting; (4) evaluate data sources and their scientific validity; (5) assess the intervention’s effect on total costs of treating a defined population; (6) analyze how uncertainty is incorporated; and (7) illuminates possible conflicts of interest among those creating the framework. We apply the taxonomy to four representative value frameworks recently published by professional organizations focused on treatment of cancer and heart disease and vaccine use. We conclude that each of these efforts has strengths and weaknesses when evaluated using our taxonomy, and suggest pathways to enhance the utility of value-assessing frameworks for policy and clinical decision-making.
Keywords Value frameworks; Cost-effectiveness; Multi-attribute decision analysis
Address correspondence to: Charles E. Phelps, 30250 S Highway 1, Gualala, CA 95445, [email protected].
Mandelblatt et al.
Page 2
Author Manuscript
INTRODUCTION The explosion of scientific knowledge, discovery, and technology began a new era of health care, wherein deployment of health care interventions depends upon personal, lifestyle, genetic, and molecular disease factors. While these advances have the potential to improve health, they increase the complexity of analyses to determine value at reasonable and affordable costs.(1–8)
Author Manuscript
Faced with new health care interventions and escalating costs of care, many professional organizations have developed frameworks to measure and/or rank the value of novel interventions. (3;7;9–22). Some of these new frameworks use components of costeffectiveness analysis, while others create de novo value measures. It remains unclear how such frameworks will guide clinical practice policy, and whether they offer particular advantages over traditional approaches such as cost-effectiveness analysis. We summarize a taxonomy of principles -- expanding upon cost-effectiveness theory -- that provide an approach to assess different value frameworks. Next, we apply this taxonomy to evaluate exemplar value frameworks to illustrate strengths, gaps, and potential concerns. We intend that this synthesis stimulate discussion about best practices and use of value metrics in policy and clinical decisions. We derived these principles in our synthesis of standard practice for the reporting of cost-effectiveness analyses, (23–25), recent ISPOR summaries of multiple criteria decision analysis (MCDA) reporting guidelines, (26) standard medical journal requirements for conflict-of-interest disclosures (27), standard Cochrane review methods (28) and our own analysis. While we draw many of these ideas from others, assembling them to evaluate models that estimate the value of health care is novel.
Author Manuscript
VALUE TAXONOMY Value frameworks can assist in public health prioritization, practice guidelines, formulary or related resource expenditure decisions (e.g., purchase of equipment or coverage of a specific drug), and patient-physician decision-making about a specific care choice. Table 1 describes our taxonomy to evaluate these value frameworks. We intend this to apply to these different purposes, but specific elements may be more or less salient for different uses of the value framework. The following narrative highlights the implications of these varying purposes when using this taxonomy to evaluate value frameworks. Conceptual Approach, Perspective, and Preferences in Measuring Value
Author Manuscript
Value frameworks have evolved, in part, to provide greater standardization and transparency in health care decision-making. (29) Further, formal frameworks can minimize biases of intuitive decision making from issues of framing of the problem, risk analysis, intertemporal tradeoffs, and others. (30) We began our taxonomy in the formal framework of cost-effectiveness analysis (CEA). CEA analyses most commonly use the societal perspective, but this perspective usually does not correspond to that held by individual participants in health care decisions – patients, providers, payers, public policy decision makers, or owners of resources used to create
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 3
Author Manuscript
health care interventions, nor does it consider community well-being or other important policy issues such as the distribution of benefits and costs across various subsets of the population (among other omissions). No single perspective can represent the interests of all participants in value-based decisions. For example, physicians generally advise patients using a patient perspective, but may introduce the societal perspective in their role as steward of societal resources (31) or, in some cases, their own professional organization’s or personal financial situation. Thus, we believe that multiple perspectives should be considered, varying by individual or organization assessing the intervention, consistent with the most recent recommendations for reporting cost-effective analyses. (24;25)
Author Manuscript
CEA can also be useful in developing value frameworks since it provides data on intermediate events of importance to judging the value of an intervention, such as toxicity rates and overall survival, measured in life-years saved, quality-adjusted life years, or disability adjusted life years. (23–25; 32; 33) Compared with the existing care-standard, the ratio of added costs to added gains in health benefits-- the incremental cost-effectiveness ratio (ICER) – stands as one value criterion. However, no consensus exists on cut-points to determine “acceptability” of a given ICER, and both illness severity and the number of people affected can influence “cut point” choices.
Author Manuscript
CEA also does not typically capture less tangible factors such as fear associated with diseases like HIV, cancer, degenerative neurological conditions, or mental illness, since these do not represent distinct health outcome states for which utility measurements exist. In concept, CEA could incorporate such issues, but in many cases, the data requirements would be impossibly complex. Full CEA evaluation requires population-based utility level estimates for every state of nature in the model. Capturing such things as fear of the diseases, contagion effects, restrictions on freedom, and modification of religious rites (all arising in the response to Ebola virus) provide examples of issues that CEA cannot meaningfully capture. While grounded in CEA, our taxonomy extends beyond this foundation. CEA does not generally consider many other attributes of decision-making that affect use of health care interventions. For example, CEA cannot address the distribution of health burdens that might fall particularly heavily on disadvantaged populations (e.g., sickle cell anemia in AfricanAmericans), or whether some aspects of a medical intervention have important moral or ethical components, (e.g., abortion after detection of a fetal genetic mutation with profound consequences.
Author Manuscript
Applying CEA for decision making brings another issue to the foreground: a new technology may pass standard “cutoff” requirements but then overwhelm the available budget – the issue of “affordability.” CEA methods currently are not equipped to deal with this issue. Multi-criteria decision analysis tools offer one approach to formally include these “other issues” into the value framework. These tools include such approaches as the Analytic
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 4
Author Manuscript
Hierarchy Process (34) and the Multi-Attribute Utility Theory, (35–36), both used in recent health care decision processes. (37–40) These models bring to the foreground an issue that does not arise in CEA: how does one balance among competing goals in a value model? Multi-criteria decision analysis methods use various methods to elicit weights attached to each (potentially) competing goal. With a single decision maker, these methods work well in general (34;36), but best methods to combine individual into group preferences remain elusive. If group estimates are not derived from relevant population samples, our framework suggests that analysts should provide good justification and clear logic for using alternative sources.
Author Manuscript
Our taxonomy proposes that value frameworks incorporate preferences, examine multiple perspectives, and are transparent. In addition, we suggest that value frameworks should consider distribution and ethical issues and other pertinent factors by using multi-criteria decision analysis methods. Accounting for all Components of the Health Condition and Intervention
Author Manuscript
Value frameworks should consider many aspects of each disease process and all short-term and downstream benefits and harms of interventions. For example, in cancer, including only one cost component, such as drug acquisition costs for a new treatment, would bias the overall assessment by ignoring other costs, such as those of drug administration, supportive care, and other aspects of therapy. In treatment of HIV, cancer, and other conditions, drugs may have significant toxicity that can affect their value. Screening tests like mammography can save lives, but also carry with them meaningful rates of false positive results and overdiagnosis. These issues should enter a full value assessment, not only in health status outcomes and costs but also in emotional burdens created. (41;42) Some value frameworks treat all harms as equal in consequence to patients or other decision-makers. For example, when the intervention is a drug treatment, rates of all side effects might be used in comparing the value of two therapies. This creates biases when the intensities and/or durations of the side effects differ, or when some effects have greater consequence than others to individual patients (e.g., nausea/vomiting vs. fatigue vs. increased risk of iatrogenic death). In best practice, our taxonomy recommends that individual preferences on these issues inform the overall assessments of alternative therapies, and that when infeasible, population-based estimates serve as an alternative.
Author Manuscript
Current value frameworks tend to focus on active therapy; e.g., drugs, surgery, and/or invasive tests. More recently, for certain diseases, such as early, non-aggressive prostate cancers, with invasive and potentially toxic standard-care therapies, alternatives now include no therapy (active surveillance). In the advanced disease setting, palliative interventions are becoming increasingly considered. If value frameworks do not explicitly consider these less aggressive approaches (i.e., only comparing novel therapies with standard therapy), the resulting decision may be biased towards therapeutic action, possibly conflicting with patient preferences. Therefore, we suggest that value frameworks compare all feasible options for patients, not just those using therapeutic intervention.
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 5
Author Manuscript
What are the sources of and process for use of data in the framework? Does this process conform to accepted methods for evidence evaluation? Several accepted methods and hierarchies evaluate the quality of evidence about health care intervention effects. Randomized controlled trials generally provide stronger evidence than other methods (e.g., historical comparisons, case-control studies, and others), and sample sizes and research design in all studies affect precision and accuracy of parameter estimates. (28)
Author Manuscript
In cases where good quality, empirically based evidence on key issues in the value framework is not available, expert judgment may provide the only realistic option. Using expert-based estimates creates biases when the experts have a professional or financial stake in choices being evaluated (e.g., profit from sales of a device or drug, or enhanced reimbursement for delivery of a new agent -- see below for further discussion). When possible, valuation models should use experts not in conflict-of-interest situations and should clearly identify when any such conflicts exist. Does the framework include the full economic consequences of all intervention components? As in standard CEA, our taxonomy recommends that value frameworks include the full cost of all the components and consequences of an intervention. For example, in frameworks to evaluate drug treatments, economic impacts would include drug costs, provider time and delivery costs, staff time, facilities and equipment overhead and costs, costs of treatmentrelated side-effects and supportive care medications required, patient time and travel costs, and costs of all downstream events until death from the disease or other causes. To the extent that frameworks exclude portions of these costs, they result in biased recommendations.
Author Manuscript
Many value frameworks have different perspectives and include different costs. Patientperspective frameworks, for example, might normally include only copayments made by patients. Payer-centric frameworks may focus only on the costs and savings to the health care provider. These approaches can lead to sub-optimal value-based decisions from a societal perspective. Costs also need to be considered in the context of extant budgets. For example, hepatitis C medications have cost-effectiveness ratios within the range of other accepted interventions, but still have high absolute costs that may overwhelm pre-established budgets.
Author Manuscript
Projected budget impact has played a key role in some health care system decisions about these medications, mostly appearing as an independent discussion of “budget impact” in addition to standard cost-effectiveness analysis. (17) We believe that this approach misses a key point: budgets and acceptable levels for cost-effectiveness ratios cannot be considered independently. Introducing a new (and expensive) treatment adds budget pressure, which invariably will raise the value of incremental resources (what economists call the “shadow price”). The proper solution will likely involve reassessment of the existing budget and reexamination of interventions that cannot compete (in a cost-effectiveness paradigm) with the newly tightened budget situation.
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 6
How does the framework account for uncertainty in effect or cost data?
Author Manuscript
Many analytic methods incorporate effects of uncertainty on the conclusions of a value recommendation. A key aspect of characterizing uncertainty relates to identification of gaps in knowledge or evidence about aspects of care that could change the value conclusions. Our taxonomy does not specify a particular method to identify the impact of uncertainty, but rather that uncertainty and gaps in data be explicitly evaluated. Conflicts of Interest
Author Manuscript
Professional specialty organizations often have the best knowledge about a particular set of health care interventions, and thus are well-placed to develop value frameworks, but may also have conflicts of interest in these issues. Value assessments may be subject to bias if financial incentives favor more (and costlier) interventions in certain delivery settings (e.g., fee for service vs. managed care or accountable care organizations). These issues extend also to the use of facilities owned in part or whole, directly or indirectly, by decision-making providers. Beyond financial incentives, the purpose of specialist organizations is to advocate for professional authority and action, which may further bias these sponsors toward interventions based in their specialties (such as chemotherapy, which is rooted in oncology, compared with palliative care, with separate specialty affiliations). Thus, we should consider if value framework’s sponsoring organization has a professional stake in the process, and if so, whether the framework potentially biases the results in favor of the sponsor’s position. Examples of Recent Frameworks
Author Manuscript
Value frameworks have recently emerged from different medical specialty groups representing oncology, (9;16–22) cancer screening, (10) radiation oncology, (11;12) cardiology,(7) surgery, (13) pediatrics, (3) and infectious disease and vaccines. (14) To demonstrate the application of our taxonomy, we selected four exemplar value frameworks spanning several specialties (Table 2). Oncology—The oncology frameworks largely focus on the value of new drugs, driven by the more rapid escalation of drug costs relative to other components of cancer care; and the impact of drug costs on patient co-payments. (31;43;44) In other disease specialties, new treatment paradigms, and expensive drugs and high priced technology have also stimulated the development of value frameworks.
Author Manuscript
The American Society of Clinical Oncology’s (ASCO) Task Force on Value in Cancer Care (18;19) and the National Comprehensive Cancer Center Network (16) both developed value assessments for systemic therapy drugs. Comments about this work recognize that the intended use centers on patient-based treatment decisions rather than a societal perspective. These frameworks followed many elements recommended in our taxonomy, but omit several key value components. First, they only focus on clinical outcomes related to survival and drug costs. Second, by limiting the framework to clinical trial evidence about episodes of drug treatment, they also do not permit assessment of the value of a given therapy in the general population. (45) Third, the framework excludes other important costs of care, such
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 7
Author Manuscript
as future need for radiation, repeat surgery, or palliative care. Fourth, many relevant outcomes needed to determine value from a societal or patient perspective are excluded, including quality of life impact of therapy, (46) perhaps as a result of omission of such data in the original clinical trials of efficacy. (47) The revised ASCO framework (19) endeavors to capture all types of toxicity, but can only succeed to the extent lower-grade adverse events are reported. Next, the method used to assign value points is both arbitrary and relies primarily on opinions of providers rather than samples of patients or their families. Finally, there is no accounting for uncertainty in the underlying data or the result, giving a false sense of precision when comparing among treatments.
Author Manuscript
Other oncology frameworks include Drug Abacus, (48) the European Society for Medical Oncology’s Magnitude of Clinical Benefit Scale, (22) and the pan-Canadian Oncology Drug Review (pCODR), (20;21). These frameworks seek to inform policy rather than clinical decision-making. They too focus primarily on drugs, with Drug Abacus evaluating cost of drug development and production vs. actual retail pricing. The pCODR is the only value framework in oncology that includes a societal perspective, patient stakeholders, components of a CEA framework including QALYs, considers evidence beyond trials, and determines how the short-term budget impact of new interventions affect the Canadian health care budget.
Author Manuscript
These oncology frameworks were developed by oncologists (with other researchers) who have professional and financial stakes in the drugs they prescribe, so that the possibility of conflict of interest arises. A recent review of oncology drug value frameworks was supported by a pharmaceutical company. The authors declared that they owned stock or consulted for this company. (2)
Author Manuscript
Cardiology—The American College of Cardiology and the American Heart Association (ACC-AHA) developed another recent value framework (7) to inform their practice guidelines. This framework rates both the quality of evidence and the value of care, with the latter based on a CEA framework. Based on CEA results, they developed a score for value from low to high using the World Health Organization WHO-CHOICE project, including the use of cutoff values of 1X to 3X per capita Gross Domestic Product. (49) Notably, the writing committee had fewer than 50% of members with industry relationships, and all of these relationships were published. Although the primary data resources for this evaluation come from independent entities, we note that when committee members are the providers of therapies, they may have inherent conflicts of interest that can cause deliberate or subconscious bias. In general, the ACC-AHA framework robustly addresses almost all of the key components of our taxonomy but could be improved by more explicit definitions of intervention components, and approaches to uncertainty. Vaccines—Frameworks for evaluation of vaccines against infectious diseases present many complexities that do not arise with other medical interventions, including the need to capture spread of disease in the population, changes in population demographics due to aging and migration, the natural history of immunity, herd protection, community well-being, the need
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 8
Author Manuscript
to model the population equilibrium, the lack of post-market surveillance for adverse events, and the ability of the vaccine to prevent disease or disease related effects (e.g., post-herpetic neuralgia). (6,50–51). These factors present unique challenges in defining the perspective, individual preferences versus effects of treatment of one person on unaffected persons (i.e., herd effects), and time horizon for a value framework. (39) Despite this complexity, the European guidelines for vaccine value assessment considers virtually all of the components in our taxonomy. (52) These guidelines explicitly address conflicts of interest: the developers excluded employees of drug or vaccine companies and required members to disclose potential conflicts.
CONCLUSION
Author Manuscript
Although the movement to examine the value of health care is rapidly gaining prominence, no consensus exists on value’s structure and components. Wider implementation of value frameworks may be possible by further use of data from clinical trials and information made available through electronic records.(46;53)) It will also be useful to include education about value frameworks in medical and public health training.(454;55) Our taxonomy could support these efforts and extends prior overviews (2;25) by including a comprehensive set of metrics grounded in cost-effectiveness analysis.
Acknowledgments This work was funded by grant #U01CA183081 from the National Cancer Institute at the National Institutes of Health. This work was also supported in part by National Cancer Institute grants R35 CA197289 and U01 CA152958.
Author Manuscript
References
Author Manuscript
1. Lawler M, French D, Henderson R, et al. Shooting for the Moon or Flying Too Near the Sun? Crossing the Value Rubicon in Precision Cancer Care. Public Health Genomics. 2016; 19(3):132–6. [PubMed: 27238737] 2. Chandra A, Shafrin J, Dhawan R. Utility of Cancer Value Frameworks for Patients, Payers, and Physicians. JAMA. 2016 May 17; 315(19):2069–70. [PubMed: 27187295] 3. Swank MF, Brennan LK, Gentry D, Kemner AL. Using frameworks to diagram value in complex policy and environmental interventions to prevent childhood obesity. J Public Health Manag Pract. 2015 May; 21(Suppl 3):S116–S120. [PubMed: 25828214] 4. Rogowski W, Payne K, Schnell-Inderst P, et al. Concepts of ‘personalization’ in personalized medicine: implications for economic evaluation. Pharmacoeconomics. 2015 Jan; 33(1):49–59. [PubMed: 25249200] 5. Committe on valuing community-based, non-clinical prevention programs; board on population health and public health practice; Institute of Medicine. Washington (DC): National Academies Press (US); 2012 Oct. An integrated framework for assessing the value of community-based prevention. 10-29-2012 6. Pronk NP, Hernandez LM, Lawrence RS. An integrated framework for assessing the value of community-based prevention: a report of the Institute of Medicine. Prev Chronic Dis. 2013; 10:120323. [PubMed: 23411033] 7. Anderson JL, Heidenreich PA, Barnett PG, et al. ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines. Circulation. 2014 Jun 3; 129(22):2329–45. [PubMed: 24677315]
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 9
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
8. Porter ME. What is value in health care? N Engl J Med. 2010 Dec 23; 363(26):2477–81. [PubMed: 21142528] 9. de Souza JA, Seiwert TY. A value framework in head and neck cancer care. Am Soc Clin Oncol Educ Book. 2014:e304–e309. [PubMed: 24857117] 10. Harris RP, Wilt TJ, Qaseem A. A value framework for cancer screening: advice for high-value care from the American College of Physicians. Ann Intern Med. 2015 May 19; 162(10):712–7. [PubMed: 25984846] 11. Thaker NG, Pugh TJ, Mahmood U, et al. Defining the value framework for prostate brachytherapy using patient-centered outcome metrics and time-driven activity-based costing. Brachytherapy. 2016 May; 15(3):274–82. [PubMed: 26916105] 12. Teckie S, McCloskey SA, Steinberg ML. Value: a framework for radiation oncology. J Clin Oncol. 2014 Sep 10; 32(26):2864–70. [PubMed: 25113759] 13. Yount KW, Turrentine FE, Lau CL, Jones RS. Putting the value framework to work in surgery. J Am Coll Surg. 2015 Apr; 220(4):596–604. [PubMed: 25728143] 14. Barnighausen T, Bloom DE, Cafiero-Fonseca ET, O’Brien JC. Valuing vaccination. Proc Natl Acad Sci U S A. 2014 Aug 26; 111(34):12313–9. [PubMed: 25136129] 15. Mitera G, Earle C, Latosinsky S, et al. Choosing Wisely Canada cancer list: ten low-value or harmful practices that should be avoided in cancer care. J Oncol Pract. 2015 May; 11(3):e296– e303. [PubMed: 25980018] 16. National Comprehensive Cancer Network. [Accessed August 1, 2016] Available at: http:// www.nccn.org/evidenceblocks/pdf 17. Institute for Clinical and Economic Review. [Accessed August 1, 2016] Methods. The ICER approach. Available at: http//:icer-review.org/methodology/icers-methods/ 18. Schnipper LE, Davidson NE, Wollins DS, et al. American Society of Clinical Oncology Statement: A Conceptual Framework to Assess the Value of Cancer Treatment Options. J Clin Oncol. 2015 Aug 10; 33(23):2563–77. [PubMed: 26101248] 19. Schnipper LE, Davidson NE, Wollins DS, et al. Updating the American Society of Clinical Oncology Value Framework: Revisions and Reflections in Response to Comments Received. J Clin Oncol. 2016 May 31. 20. Cheung MC, Sabharwal M, Chambers A, et al. Multiple Dimensions of Value: Evaluative Frameworks for New Cancer Therapies. J Clin Oncol. 2016 Apr 20; 34(12):1428–9. [PubMed: 26884567] 21. Hoch JS, Sabharwal M. Informing Canada’s cancer drug funding decisions with scientific evidence and patient perspectives: the Pan-Canadian Oncology Drug Review. Curr Oncol. 2013 Apr; 20(2): 121–4. [PubMed: 23559876] 22. Cherny NI, Sullivan R, Dafni U, et al. A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Ann Oncol. 2015 Aug; 26(8):1547–73. [PubMed: 26026162] 23. Gold, MR.Siegel, JE.Russell, LB., Weinstein, MC., editors. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996. 24. Neumann, PJ.Sanders, GD.Russell, LD.Siegel, JD., Ganiats, TG., editors. Cost-Effectiveness in Health and Medicine. 2. New York: Oxford University Press; 2017. 25. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA. 2016; 316(10):1093–1103. DOI: 10.1001/jama.2016.12195 [PubMed: 27623463] 26. Marsh K, IJzerman M, Thokala P, et al. Multiple Criteria Decision Analysis for Health Care Decision Making—Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016; 19:125–137. [PubMed: 27021745] 27. [last accessed 10 October, 2016] http://www.icmje.org/recommendations/browse/roles-andresponsibilities/author-responsibilities--conflicts-of-interest.html), standard statistical evaluation tools for medical research 28. [last accessed 10 October, 2016] http://training.cochrane.org/handbook Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 10
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
29. Madhavan G, Phelps CE, Colwell RR, Rappuoli R, Fineberg HV. In Plain View: A Transparent Systems Approach for Enhancing Health Policy Decisions. The Bridge: Linking Engineering and Society. 2016 Summer;46(2):5–10. 30. Kahneman, D. Thinking, Fast and Slow. New York: Farrar, Strass and Giroux; 2011. 31. Schnipper LE, Bastian A. New Frameworks to Assess Value of Cancer Care: Strengths and Limitations. Oncologist. 2016 Jun; 21(6):654–8. [PubMed: 27245568] 32. [Accessed August 1, 2016] http://www.who.int/healthinfo/global_burden_disease/metrics_daly/en/ 33. Malone DC, Berg NS, Claxton K, et al. International Society for Pharmacoeconomics and Outcomes Research Comments on the American Society of Clinical Oncology Value Framework. J Clin Oncol. 2016 Jun 13. 34. Saaty TL. Decision making with the analytic hierarchy process. Int J Serv Sciences. 2008; 1(1):83– 98. 35. Edwards W, Barron FH. SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement. Organizational Behavior and Human Decision Processes. 1994; 60:306–325. 36. Barron FH, Barrett BE. Decision Quality Using Ranked Attribute Weights. Management Science. Nov; 1996 42(11):1515–152. 37. Schmidt K, Aumann I, Hollander I, et al. Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting. BMC Med Inform Decis Mak. 2015; 15:112. [PubMed: 26703458] 38. Madhavan, G.Sangha, K.Phelps, C., et al., editors. Ranking Vaccines: a prioritization framework: phase I: demonstration of concept and a software blueprint/. Washington, DC: National Academies Press (US); 2012. 39. Madhavan, G.Sangha, K.Phelps, C., et al., editors. Ranking vaccines: a prioritization softward tool: phase II: prototype of a decision-support system. Washington, DC: National Academies Press (US); 2013. 40. Madhavana, G.Phelps, C.Rappuoli, R., et al., editors. Ranking Vaccines: Applications of a Prioritization Software Tool: Phase III: Use Case Scenarios and Data Framework. Washington, DC: National Academy Press; Dec 15. 2014 41. Nelson HD, Fu R, Cantor A, et al. Effectiveness of Breast Cancer Screening: Systematic Review and Meta-analysis to Update the 2009 U.S. Preventive Services Task Force Recommendation. Ann Intern Med. 2016 Feb 16; 164(4):244–55. [PubMed: 26756588] 42. Nelson HD, Pappas M, Cantor A, et al. Harms of Breast Cancer Screening: Systematic Review to Update the 2009 U.S. Preventive Services Task Force Recommendation. Ann Intern Med. 2016 Feb 16; 164(4):256–67. [PubMed: 26756737] 43. Henry J. Kaiser Family Foundation. [Accessed May 18, 2016] Employer health benefits survey. 2015. Available at http://kff.org/report-section/ehbs-2015-summary-of-findings/ 44. Shankaran V, Jolly S, Blough D, Ramsey SD. Risk factors for financial hardship in patients receiving adjuvant chemotherapy for colon cancer: a population-based exploratory analysis. J Clin Oncol. 2012 May 10; 30(14):1608–14. [PubMed: 22412136] 45. Angelis A, Kanavos P. Critique of the American Society of Clinical Oncology Value Assessment Framework for Cancer Treatments: Putting Methodologic Robustness First. J Clin Oncol. 2016 Jun 13. 46. Basch E. Toward a Patient-Centered Value Framework in Oncology. JAMA. 2016 May 17; 315(19):2073–4. [PubMed: 27187297] 47. Li J, Asabere A, Kelly S, et al. Application Of Asco Value Framework Evaluations Of Net Health Benefit For Oncology Drugs Launched In The United States Between 2013 And 2015. Value Health. 2015 Nov.18(7):A489. 48. [Accessed August 1, 2016] Evidence driven drug pricing project. Accessed at: http:// www.drugabacus.org/ 49. [Accessed August 1, 2016] Choosing interventions that are cost-effective project. http:// www.who.int/choice/cost-effectiveness/en/ 50. Ultsch B, Damm O, Beutels P, et al. Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a European Vaccine Economics Community. Pharmacoeconomics. 2016 Mar; 34(3):227–44. [PubMed: 26477039] Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 11
Author Manuscript
51. Ultsch B, Damm O, Beutels P, et al. Methods for Health Economic Evaluations of Vaccines Results from an International Expert-Workshop. Value Health. 2014 Nov.17(7):A552. 52. Ultsch B, Damm O, Perleth M, Wichmann O. Health Economics In Immunization DecisionMaking - Results From A Systematic Literature Research and A Stakeholder Symposium In Germany. Value Health. 2015 Nov.18(7):A569. 53. Fayanju OM, Mayo TL, Spinks TE, et al. Value-Based Breast Cancer Care: A Multidisciplinary Approach for Defining Patient-Centered Outcomes. Ann Surg Oncol. 2016 Aug; 23(8):2385–90. [PubMed: 26979306] 54. Patel MS, Davis MM, Lypson ML. The VALUE Framework: training residents to provide valuebased care for their patients. J Gen Intern Med. 2012 Sep; 27(9):1210–4. [PubMed: 22573146] 55. Phelps C, Madhavan G, Rappuoli R, et al. Strategic Planning in Population Health and Public Health Practice: A Call to Action for Higher Education. Milbank Q. 2016 Mar; 94(1):109–25. [PubMed: 26994711]
Author Manuscript Author Manuscript Author Manuscript Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 12
Author Manuscript
Summary Summarizes principles that could be used to evaluate value frameworks and applies these to analyze strengths, gaps, and potential concerns about some recently published frameworks.
Author Manuscript Author Manuscript Author Manuscript Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 13
Table 1
Author Manuscript
Proposed Taxonomy for Evaluation of Frameworks Assessing the Value of Health Care Interventions Category
Component detail
Purpose Is the purpose defined and are the elements of the framework consistent with the purpose? Conceptual approach, perspective, and preferences Is the value structure of the framework clear and transparent? If it requires assumptions, are do they have general and clinical face validity? What perspective does the framework use (e.g., societal, patient, provider, payer)? Is the perspective made explicit or embedded in the framework? Does the framework include standard methods of measuring value, such as cost-effectiveness analysis?
Author Manuscript
Does the framework consider multiple attributes of medical interventions such as individual preferences, equity and the distribution of health care benefits and costs, and issues involving externalities (such as contagious diseases), and the value of scientific breakthrough? If the framework includes multiple dimensions of value, are the weights for each component population-based or expert-provided, and are the methods for eliciting such weights both clearly stated and methodologically sound? Does the framework allow individual patient preferences? If so, are they elicited with methods known to be free of bias and produce reliable results? Intervention components and comparators Does the framework include all components and consequences of the intervention, or merely a portion of those (e.g., drug acquisition costs)? Does the framework aggregate or disaggregate such things as toxicity or other side effects of intervention? Does the framework assume as a baseline that some intervention will be provided, or does it allow for “watchful waiting” or “palliative care” as an option? Data sources
Author Manuscript
Are clinical and other data derived from expert opinion, population surveys, or other sources; are the sources made clear, and is the process replicable? Economics/costs What is the effect of the intervention on the total cost of treating a defined population, including whether inclusion of the intervention will increase, decrease or leave unchanged the total costs of care for that population? Uncertainty and identification of important gaps
Is uncertainty in data about costs or effects considered in conclusions about value? Are there gaps in knowledge about aspects of care that could change the value rating?
Conflicts of interest
Does the sponsoring organization have a financial stake in the process, and if so, is this declared? Does the framework bias the results in favor of the sponsor’s financial position?
Author Manuscript Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 14
Table 2
Author Manuscript
Examples of Application of Proposed Taxonomy for Evaluation of Frameworks Assessing the Value of Health Care Interventions Protocol Element
American Society of Clinical Oncology (19)
National Comprehensive Cancer Network (16)
American Heart AssociationAmerican College of Cardiology (7)
European Vaccine Working Group (33)
Stated purpose
Patient-physician drug treatment decisions; separate framework for adjuvant vs. metastatic Rx
Patient-physician drug treatment decisions for surgery, radiation, and drugs
Inform practice guidelines and performance measures
Public health policy decisions
Framework consistent with purpose
Yes; will need to develop format for use in practice
Yes; included with online treatment guidelines
Yes
Yes
Clear and transparent conceptual framework
No
No
Yes
Yes
Do assumptions have empirically supported clinical basis?
Bonus points subjectively assigned
Scoring system subjective/expert opinion
N/A
Yes
Is the perspective made explicit
Implies clinical decision making perspective
Implies clinical decision making perspective
Societal, with separate analyses with patient perspectives for clinical decisions; consider subgroups
Societal with long term population outcomes
References existing methods of assessing value, allowing comparison across diseases
De novo method: net health benefit score and costs per drug treatment course. “Bonus points” and toxicity included in net benefits score for a new regimen vs. control therapy Points for benefit are assigned based on categories of median survival Does not allow comparisons across multiple available regimens
De novo method: evidence block score from 1–5 based on subjective assessment of benefits Does not allow comparisons across multiple available regimens
Uses costs per QALY to grade value low to high value using WHOCHOICE framework; includes uncertain value category. Value categories can be based on societal cut-points or a percent of GDP
Uses dynamic agent based models and costs per QALY May consider single or multiple vaccines
Considers multi-decision attributes (preferences, equity, distribution, externalities)
Preferences included in metastatic framework only, but based on expert designation of “bonus points” for symptom palliation, and QOL, longer life with metastases
unknown
Considers preferences, distribution, and budget impacts
Explicitly considers use of multi-attribute theory to assist with decisions
Only clinical benefit as gain in survival, major toxicity, and costs of drugs and supportive care for drug delivery; patient copay for drugs included. Other effects and costs excluded. Update to framework includes lower grade toxicities. Only examines one drug and one control drug
Effectiveness, safety (i.e., major toxicity), quality of evidence, consistency of evidence, and affordability No direct comparisons among treatment modalities
Implicit in CEA framework, but not specified; framework used as a broad template
Very broad horizon and effects considered in a CEA framework
Purpose
Approach
Author Manuscript Author Manuscript
Intervention components and comparators All components and comparators considered
Author Manuscript
Value Health. Author manuscript; available in PMC 2017 August 02.
Mandelblatt et al.
Page 15
Author Manuscript
Protocol Element
American Society of Clinical Oncology (19)
National Comprehensive Cancer Network (16)
American Heart AssociationAmerican College of Cardiology (7)
European Vaccine Working Group (33)
Toxicity/side effects considered
Yes, but not in costs for hospitalization, etc.
For some regimens
Implicit but not specified
Yes
Considers no treatment as a comparator (e.g., “watchful waiting” or “palliative care” )
No
For some treatment options (surgery, radiation, drugs)
Unknown
N/A
RCT and expert opinion
RCT and expert opinion
Specifies US data. Data rated from A–C based on a formal review of the evidence quality; includes randomized trials and observational data
Vaccine trials using intention to treat results
All costs stemming from the intervention included
Only drug costs; drug delivery and supportive care for drug delivery Includes patient co-pays for drugs but not time costs
Drug cost, supportive care, and drug administration, and toxicity costs; costs ranked subjectively on an “affordability scale” from 1–5 (inexpensive to very expensive)
Includes all costs in CEA framework
Includes all costs in CEA framework
budget impact assessed
Budget impact not considered
Budget impact not considered
??
??
Uncertainty and information gaps considered
Not considered
Not considered
Not explicitly considered; does include an unknown category for value
Yes
Conflicts of interest
Not explicitly addressed. Disclosures included in journal article. Potential conflicts since value primarily determined by oncologists who deliver (and bill) care
Not explicitly addressed. Potential conflicts since value primarily determined by oncologists who deliver (and bill) care
Explicitly addressed;
