Practical Radiation Oncology (2011) 1, 81–82
www.practicalradonc.org
Commentary
The overthrow of the (evidence) hierarchy Michael Steinberg MD ⁎ Department of Radiation Oncology, University of California Los Angeles, Los Angeles, California Received 15 February 2011; accepted 15 February 2011
Comparative effectiveness research (CER) has many definitions and is often misunderstood or understood only at its simplest; comparing treatment A to treatment B. Bekelman et al, in their article “Implications of Comparative Effectiveness Research for Radiation Oncology,” present a sophisticated methods commentary that not only introduces clinicians to the concept of CER, but also clarifies our understanding of what constitutes useful evidence required to demonstrate the value of our treatments in this time of rising health care costs and declining resources.1 This article should be required reading for all practicing radiation oncologists and radiation oncology residents. If the article by Bekelman et al embodies an introductory course in understanding the nature of evidence supporting medical interventions, required reading for an advanced course is Sir Michael Rawlins' Harveian Oration given before the Royal College of Physicians in 2008 entitled, “On the evidence for decisions about the use of therapeutic interventions.”2 In this extraordinary work, Rawlins lays out the epistemological polemic regarding the scientific method as it relates to the development of medical knowledge, especially the debate surrounding the use of inductive versus deductive methods for substantiating medical evidence and the application or misapplication of levels or hierarchies of evidence. Rawlins points out that medical evidence serves as the foundation for providers, payors, and policy makers to understand the appropriate
See Related Article on page 72. Conflicts of interest: None. ⁎ 200 UCLA Medical Center Plaza, B265, Department of Radiation Oncology, Los Angeles, CA 90095. E-mail address: [email protected].
use of medical procedures and treatments in clinical practice. He notes, however, that in recent years too many decision makers have adhered rigidly to the hierarchical evidence format which places randomized clinical trials (RCTs) or their meta-analytic big brothers precariously at the pinnacle of such evidence. Rawlins observes that the customary bounding of the RCTs inquiry creates critical questions of generalizability of their conclusions and that the considerable resources required for their successful completion do not recommend the RCTs to their lofty position on the evidence ladder. In fact, as we enter the era of personalized medicine, the infinite number of RCTs addressing ever decreasing sized cohorts of patients could render evidence gathering an endless process. RCTs do have certain advantages, but also disadvantages. Likewise, observational evidence has noteworthy merits and wellknown limitations. Some of the limitations may be mitigated by the use of Bayesian methods of design and analysis for both experimental and observational data. Rawlins further argues that the strengths and weaknesses of each needs to be understood, and that judgment needs to be applied regarding the fitness for use in particular circumstances for each component of the evidence base.2 Now, consider radiation oncology with its rapid technological advancement in the past decade and associated significant improvements in care, often based on technological enhancements in the accuracy and precision of dose deposition occurring over short development cycles. Randomization of a patient to a known inferior radiation dose distribution with excess exposure of normal tissue (unacceptable to patients and their doctors) in the face of non-coverage by payors due to lack of high level hierarchical evidence and lack of
1879-8500/$ – see front matter © 2011 American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved. doi:10.1016/j.prro.2011.02.002
82
M. Steinberg
capital or structural infrastructure to support the development of such evidence, creates a seemingly insoluble dilemma foreshadowing gridlock unless something is done. Judgment is essential, Rawlins warns, to solve this dilemma because decisions made based on the evidentiary process have consequences for patients and society. Acceptance of new medical treatments based on an illusory hierarchy of evidence may not only lead to decreased access or denial of better care, but could foster a dysfunctional mechanism for incorporation of new treatments, as some have claimed plagues the pharmaceutical industry.3 Rawlins further warns us that debates about the relative importance of the various components of the evidence base are ostensibly a distraction because, in fact, “a diversity of approaches” steeped in judgment are required to understand “what health interventions work, for which patients, under what condition,”1,2 and, I might add, at what cost? The American Society for Radiation Oncology (ASTRO) has struggled for years with this problem of how to address our need for evidentiary support for new radiation oncology technologies and treatments. In 2004, consistent with the approach of the time, ASTRO put in place a guidelines process whose methodology was essentially dependent on the content of our extant literature and on the evidence hierarchy. Recently, by action of the Board of Directors, ASTRO guidelines have evolved in their purpose to become evidence-based recommendations for “when” to use radiation therapy in the treatment of disease. After it is determined if radiation therapy should be done in a particular circumstance, the question then becomes “how” the treatment should be done. To address this crucial question the Board of Directors recently established the ASTRO Best Practices (BPs) Initiative,
Practical Radiation Oncology: April-June 2011
whose methodology establishes the appropriate use of radiation oncology technology and determines the best treatment methods and patient care processes. BPs will serve to inform radiation oncologists, payors, and policy makers of best practices for the application and use of technology. BPs are developed through a rigorous process utilizing the RAND Corporation/University of California Los Angeles Appropriateness Methodology, which includes the critical evaluation of evidence in the Rawlins genre and the use of validated multidisciplinary “expert panels.”4 BPs will prescribe the appropriate use of radiation oncology technology, treatment methods, and patient care processes as well as expose inappropriate processes that should be proscribed. Ultimately, the right treatment with the right technology delivered in the right way and given to the right patient will be the outcome. In the future, the development, analysis, and understanding of medical evidence for radiation oncology and for all other medical specialties will likely reside in novel analytical methods and interpretations of both experimental and observational data. Radiation oncology, by embracing the precepts of Rawlins and the clarifying work of Bekelman et al, is preparing for that future.
References 1. Bekelman JE, Shah A, Hahn SM. Implications of comparative effectiveness research for radiation oncology. Practical Radiat Oncol. 2011;1:72-80. 2. Rawlins M. De Testimonio: on the evidence for decisions about the use of therapeutic interventions. Clin Med. 2008;8:579-588. 3. Deyo RA, Patrick DL. Hope or hype: the obsession with medical advances and the high cost of false promises. New York: AMACOM Press; 2005. 4. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2:53-63.
www.practicalradonc.org
Commentary
The overthrow of the (evidence) hierarchy Michael Steinberg MD ⁎ Department of Radiation Oncology, University of California Los Angeles, Los Angeles, California Received 15 February 2011; accepted 15 February 2011
Comparative effectiveness research (CER) has many definitions and is often misunderstood or understood only at its simplest; comparing treatment A to treatment B. Bekelman et al, in their article “Implications of Comparative Effectiveness Research for Radiation Oncology,” present a sophisticated methods commentary that not only introduces clinicians to the concept of CER, but also clarifies our understanding of what constitutes useful evidence required to demonstrate the value of our treatments in this time of rising health care costs and declining resources.1 This article should be required reading for all practicing radiation oncologists and radiation oncology residents. If the article by Bekelman et al embodies an introductory course in understanding the nature of evidence supporting medical interventions, required reading for an advanced course is Sir Michael Rawlins' Harveian Oration given before the Royal College of Physicians in 2008 entitled, “On the evidence for decisions about the use of therapeutic interventions.”2 In this extraordinary work, Rawlins lays out the epistemological polemic regarding the scientific method as it relates to the development of medical knowledge, especially the debate surrounding the use of inductive versus deductive methods for substantiating medical evidence and the application or misapplication of levels or hierarchies of evidence. Rawlins points out that medical evidence serves as the foundation for providers, payors, and policy makers to understand the appropriate
See Related Article on page 72. Conflicts of interest: None. ⁎ 200 UCLA Medical Center Plaza, B265, Department of Radiation Oncology, Los Angeles, CA 90095. E-mail address: [email protected].
use of medical procedures and treatments in clinical practice. He notes, however, that in recent years too many decision makers have adhered rigidly to the hierarchical evidence format which places randomized clinical trials (RCTs) or their meta-analytic big brothers precariously at the pinnacle of such evidence. Rawlins observes that the customary bounding of the RCTs inquiry creates critical questions of generalizability of their conclusions and that the considerable resources required for their successful completion do not recommend the RCTs to their lofty position on the evidence ladder. In fact, as we enter the era of personalized medicine, the infinite number of RCTs addressing ever decreasing sized cohorts of patients could render evidence gathering an endless process. RCTs do have certain advantages, but also disadvantages. Likewise, observational evidence has noteworthy merits and wellknown limitations. Some of the limitations may be mitigated by the use of Bayesian methods of design and analysis for both experimental and observational data. Rawlins further argues that the strengths and weaknesses of each needs to be understood, and that judgment needs to be applied regarding the fitness for use in particular circumstances for each component of the evidence base.2 Now, consider radiation oncology with its rapid technological advancement in the past decade and associated significant improvements in care, often based on technological enhancements in the accuracy and precision of dose deposition occurring over short development cycles. Randomization of a patient to a known inferior radiation dose distribution with excess exposure of normal tissue (unacceptable to patients and their doctors) in the face of non-coverage by payors due to lack of high level hierarchical evidence and lack of
1879-8500/$ – see front matter © 2011 American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved. doi:10.1016/j.prro.2011.02.002
82
M. Steinberg
capital or structural infrastructure to support the development of such evidence, creates a seemingly insoluble dilemma foreshadowing gridlock unless something is done. Judgment is essential, Rawlins warns, to solve this dilemma because decisions made based on the evidentiary process have consequences for patients and society. Acceptance of new medical treatments based on an illusory hierarchy of evidence may not only lead to decreased access or denial of better care, but could foster a dysfunctional mechanism for incorporation of new treatments, as some have claimed plagues the pharmaceutical industry.3 Rawlins further warns us that debates about the relative importance of the various components of the evidence base are ostensibly a distraction because, in fact, “a diversity of approaches” steeped in judgment are required to understand “what health interventions work, for which patients, under what condition,”1,2 and, I might add, at what cost? The American Society for Radiation Oncology (ASTRO) has struggled for years with this problem of how to address our need for evidentiary support for new radiation oncology technologies and treatments. In 2004, consistent with the approach of the time, ASTRO put in place a guidelines process whose methodology was essentially dependent on the content of our extant literature and on the evidence hierarchy. Recently, by action of the Board of Directors, ASTRO guidelines have evolved in their purpose to become evidence-based recommendations for “when” to use radiation therapy in the treatment of disease. After it is determined if radiation therapy should be done in a particular circumstance, the question then becomes “how” the treatment should be done. To address this crucial question the Board of Directors recently established the ASTRO Best Practices (BPs) Initiative,
Practical Radiation Oncology: April-June 2011
whose methodology establishes the appropriate use of radiation oncology technology and determines the best treatment methods and patient care processes. BPs will serve to inform radiation oncologists, payors, and policy makers of best practices for the application and use of technology. BPs are developed through a rigorous process utilizing the RAND Corporation/University of California Los Angeles Appropriateness Methodology, which includes the critical evaluation of evidence in the Rawlins genre and the use of validated multidisciplinary “expert panels.”4 BPs will prescribe the appropriate use of radiation oncology technology, treatment methods, and patient care processes as well as expose inappropriate processes that should be proscribed. Ultimately, the right treatment with the right technology delivered in the right way and given to the right patient will be the outcome. In the future, the development, analysis, and understanding of medical evidence for radiation oncology and for all other medical specialties will likely reside in novel analytical methods and interpretations of both experimental and observational data. Radiation oncology, by embracing the precepts of Rawlins and the clarifying work of Bekelman et al, is preparing for that future.
References 1. Bekelman JE, Shah A, Hahn SM. Implications of comparative effectiveness research for radiation oncology. Practical Radiat Oncol. 2011;1:72-80. 2. Rawlins M. De Testimonio: on the evidence for decisions about the use of therapeutic interventions. Clin Med. 2008;8:579-588. 3. Deyo RA, Patrick DL. Hope or hype: the obsession with medical advances and the high cost of false promises. New York: AMACOM Press; 2005. 4. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2:53-63.
