Opinion
VIEWPOINT
Benjamin Djulbegovic, MD, PhD Division of Evidence-Based Medicine, Department of Internal Medicine; and Departments of Hematology and Health Outcome Behavior, H. Lee Mofitt Cancer Center and Research Institute, University of South Florida, Tampa. Iztok Hozo, PhD Department of Mathematics, Indiana University, Gary. John P. A. Ioannidis, MD, DSc Stanford Prevention Research Center, Department of Medicine, and Department of Health Research and Policy, Stanford University School of Medicine; Department of Statistics, Stanford University School of Humanities and Sciences; and Meta-Research Innovation Center at Stanford, Stanford, California.
Corresponding Author: John P. A. Ioannidis, MD, DSc, Stanford Prevention Research Center, Stanford University, 1265 Welch Rd, MSOB X306, Stanford, CA 94305 (jioannid @stanford.edu).
Improving the Drug Development Process More Not Less Randomized Trials ments. In fact, industry researchers have taken the lead in raising concerns about the reproducibility of preclinical research and suggesting potential solutions.6 For preclinical research conducted by nonindustry researchers, similarly rigorous practices can also be routinely adopted and requested.7 Funders and journals can specify that they will sponsor and publish animal studies only if they fulfill rigorous randomization criteria.8 Justified exceptions to this rule are likely to be rare. More rigorous adoption of randomized trials at all 3 phases may similarly help improve the clinical part of the drug development process. Insights in pharmacokinetics and pharmacodynamics for diverse drug doses can usually be obtained with very limited sample sizes (eg, 2-3 participants per dose level and approximately 10-20 participants total) in what are increasingly called phase 0 trials. Phase 0 trials may or may not be randomized. However, once a potential dose(s) has been selected, subsequent phase 1 and 2 trials should be routinely randomized with best standard of care being the routine comparator. This idea is not as radical as it may sound. More than 4 decades ago, Thomas Chalmers proposed that most scientific and ethical clinical trial design should be based on the principle “randomize the first patient.”9 Chalmers provided compelling logic for why the first patient should Almost half a century later, a large be randomized and conducted a number of randomized trials in which he did number of phase 1 and phase 2 trials are randomize the first patient invited to parstill nonrandomized. ticipate in the study. However, almost half a century later, a large number of new compounds each year, of which only 1 in 6000 phase 1 and phase 2 trials are still nonrandomized. An move on to testing in phase 1 trials.4 There is no way that evaluation of registered protocols first received by Clinitesting 1 600 000 new compounds in humans would calTrials.gov between August 1, 2013, and August 15, be affordable. However, there is probably substantial 2013, shows that 53 of 105 phase 1 trials (50%) and 42 room for improving the performance of preclinical ani- of 113 phase 2 trials (37%) were nonrandomized. These mal research. The major enhancements would include are likely to be underestimates, because nonregistraadoption of reproducible research practices and more tion is likely to be far more common for nonrandomrigorous randomized designs. Currently, much of the ized studies than for randomized studies. Empirical studpublished preclinical research is not reproducible: many ies show that almost all phase 1 studies in oncology are animal studies or other preclinical evaluations do not nonrandomized.10 Across medicine, probably more than adopt experimental, randomized designs with rigorous 80% of phase 1 studies and more than 50% of phase 2 assessments (eg, blinded measurements of well- studies are currently nonrandomized. This corredefined outcomes), and study results are selectively re- sponds to many thousands of nonrandomized clinical ported in the literature. trials conducted annually. This literature yields an excess of statistically Once a specific dose (or dose range) has been sesignificant findings that cannot be eventually lected for a new treatment, it makes little sense to colreplicated,5,6 let alone translated into clinical successes. lect uncontrolled observational data, instead of comFor preclinical research conducted by the industry, rou- paring this dose or doses against the best standard of tine adoption of rigorous randomized designs should care. The results of single-group, uncontrolled studies be straightforward—no company wants to spend mil- are always difficult to interpret. If the results are ”posilions of dollars for the clinical testing of useless treat- tive,” it cannot be proved that they are not attributable Drug development is often a lengthy and expensive process. Extensive preclinical testing via in vitro and animal experimentation aims to select drugs most likely to work in humans. Under the current system, only about half of the drugs succeed in moving from phase 1 (dosefinding) to phase 2 (safety and efficacy).1 For drugs that enter phase 2, less than 1 in 3 succeed; for those entering phase 3 (pivotal efficacy), that number decreases to less than 1 in 2.1,2 Less than 20% of drugs entering phase 1 testing successfully reach the end of the 3-phase evaluation. The percentage can vary from one specialty area to another, and it can be less than 5% to 10% for oncologic and neurologic diseases.3 Some observers questioning the efficiency of the process have suggested that the role of randomized trials should be deemphasized. Conversely, the current development system can be improved with more routine adoption of rigorous randomized designs, both in the preclinical screening and during all phases of clinical testing. The current preclinical screening evaluation in animals and cell cultures may not be efficient and might not be informative about what will happen in humans. However, there could be a different interpretation. Pharmaceutical companies test about 1 600 000
jama.com
JAMA January 22/29, 2014 Volume 311, Number 4
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Yale University User on 05/16/2015
355
Opinion Viewpoint
to chance or a favorable sample of selected patients; if the results are unfavorable, the reverse arguments can be raised. Moreover, from an ethical perspective, in a randomized phase 1 or 2 trial the patients will have a 50% chance of being assigned to a better treatment, whereas in a single-group phase 1 or 2 trial the patients will be allocated to a treatment that (based on assumed eventual success rates) is far less than 50% likely to be the best currently available. The unethical dimension of nonrandomized clinical studies had already been recognized by Chalmers. In addition to being more ethical, adoption of randomized design throughout the drug development process would most likely improve the efficiency, ie, enable faster development of new, successful treatments. The proposed shift to randomized phase 1 and 2 trials could also help more efficiently identify ineffective or harmful treatments. This would be easier and more accurate to demonstrate in a randomized design than in an uncontrolled setting. Uncontrolled studies may lead to the abandonment of some potentially useful treatments or may fail to demonstrate the problems of many ineffective treatARTICLE INFORMATION Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. Funding/Support: The authors were supported in part by National Institutes of Health (NIH) grants 1R01NS044417-01, 1 R01 NS052956-01, and 1R01CA133594-01 (NIH Office of Research Integrity). The Meta-Research Innovation Center at Stanford is supported by a grant from the Laura and John Arnold Foundation.
356
ments and protract expensive clinical testing. For a development system with substantially high failure rates at all steps, not optimizing the study designs and their accuracy makes no sense. It could be argued that phase 1 or 2 studies may offer guidance for the design of the phase 3 trial, eg, by suggesting what the plausible effect size might be and thus informing sample size calculations for the definitive phase 3 trial. However, plausible effect sizes are far more likely to be accurately informed from a controlled trial than an uncontrolled one. Moreover, most phase 3 trials should focus on what treatment effect sizes are clinically meaningful to detect rather than try to secure a nominally statistically significant but clinically unimportant result based on the “insider” information supposedly provided by phase 1 or 2 data. The drug development process could be substantially improved if rigorous randomized trials become the first rather than the last step in the process of discovery of new, effective drugs and if randomization permeates testing at all stages. Both for animals and for humans, more, not fewer, randomized trials are needed.
2. Djulbegovic B, Kumar A, Glasziou P, Miladinovic B, Chalmers I. Medical research: trial unpredictability yields predictable therapy gains. Nature. 2013;500(7463):395-396. 3. Bhattacharjee Y. Biomedicine. Pharma firms push for sharing of cancer trial data. Science. 2012;338(6103):29. 4. Pisano G. Can science be a business? lessons from biotech. Harvard Business Review. 2006;84(10)1-13.
REFERENCES
5. Tsilidis KK, Panagiotou OA, Sena ES, et al. Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol. 2013;11(7):e1001609.
1. Paul SM, Mytelka DS, Dunwiddie CT, et al. How to improve R&D productivity: the pharmaceutical industry’s grand challenge. Nat Rev Drug Discov. 2010;9(3):203-214.
6. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011;10(9):712-713.
7. Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature. 2012;483(7391):531-533. 8. Landis SC, Amara SG, Asadullah K, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012;490(7419):187-191. 9. Chalmers TC. Randomization of the first patient. Med Clin North Am. 1975;59(4):1035-1038. 10. Horstmann E, McCabe MS, Grochow L, et al. Risks and benefits of phase 1 oncology trials, 1991 through 2002. N Engl J Med. 2005;352(9): 895-904.
JAMA January 22/29, 2014 Volume 311, Number 4
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Yale University User on 05/16/2015
jama.com
VIEWPOINT
Benjamin Djulbegovic, MD, PhD Division of Evidence-Based Medicine, Department of Internal Medicine; and Departments of Hematology and Health Outcome Behavior, H. Lee Mofitt Cancer Center and Research Institute, University of South Florida, Tampa. Iztok Hozo, PhD Department of Mathematics, Indiana University, Gary. John P. A. Ioannidis, MD, DSc Stanford Prevention Research Center, Department of Medicine, and Department of Health Research and Policy, Stanford University School of Medicine; Department of Statistics, Stanford University School of Humanities and Sciences; and Meta-Research Innovation Center at Stanford, Stanford, California.
Corresponding Author: John P. A. Ioannidis, MD, DSc, Stanford Prevention Research Center, Stanford University, 1265 Welch Rd, MSOB X306, Stanford, CA 94305 (jioannid @stanford.edu).
Improving the Drug Development Process More Not Less Randomized Trials ments. In fact, industry researchers have taken the lead in raising concerns about the reproducibility of preclinical research and suggesting potential solutions.6 For preclinical research conducted by nonindustry researchers, similarly rigorous practices can also be routinely adopted and requested.7 Funders and journals can specify that they will sponsor and publish animal studies only if they fulfill rigorous randomization criteria.8 Justified exceptions to this rule are likely to be rare. More rigorous adoption of randomized trials at all 3 phases may similarly help improve the clinical part of the drug development process. Insights in pharmacokinetics and pharmacodynamics for diverse drug doses can usually be obtained with very limited sample sizes (eg, 2-3 participants per dose level and approximately 10-20 participants total) in what are increasingly called phase 0 trials. Phase 0 trials may or may not be randomized. However, once a potential dose(s) has been selected, subsequent phase 1 and 2 trials should be routinely randomized with best standard of care being the routine comparator. This idea is not as radical as it may sound. More than 4 decades ago, Thomas Chalmers proposed that most scientific and ethical clinical trial design should be based on the principle “randomize the first patient.”9 Chalmers provided compelling logic for why the first patient should Almost half a century later, a large be randomized and conducted a number of randomized trials in which he did number of phase 1 and phase 2 trials are randomize the first patient invited to parstill nonrandomized. ticipate in the study. However, almost half a century later, a large number of new compounds each year, of which only 1 in 6000 phase 1 and phase 2 trials are still nonrandomized. An move on to testing in phase 1 trials.4 There is no way that evaluation of registered protocols first received by Clinitesting 1 600 000 new compounds in humans would calTrials.gov between August 1, 2013, and August 15, be affordable. However, there is probably substantial 2013, shows that 53 of 105 phase 1 trials (50%) and 42 room for improving the performance of preclinical ani- of 113 phase 2 trials (37%) were nonrandomized. These mal research. The major enhancements would include are likely to be underestimates, because nonregistraadoption of reproducible research practices and more tion is likely to be far more common for nonrandomrigorous randomized designs. Currently, much of the ized studies than for randomized studies. Empirical studpublished preclinical research is not reproducible: many ies show that almost all phase 1 studies in oncology are animal studies or other preclinical evaluations do not nonrandomized.10 Across medicine, probably more than adopt experimental, randomized designs with rigorous 80% of phase 1 studies and more than 50% of phase 2 assessments (eg, blinded measurements of well- studies are currently nonrandomized. This corredefined outcomes), and study results are selectively re- sponds to many thousands of nonrandomized clinical ported in the literature. trials conducted annually. This literature yields an excess of statistically Once a specific dose (or dose range) has been sesignificant findings that cannot be eventually lected for a new treatment, it makes little sense to colreplicated,5,6 let alone translated into clinical successes. lect uncontrolled observational data, instead of comFor preclinical research conducted by the industry, rou- paring this dose or doses against the best standard of tine adoption of rigorous randomized designs should care. The results of single-group, uncontrolled studies be straightforward—no company wants to spend mil- are always difficult to interpret. If the results are ”posilions of dollars for the clinical testing of useless treat- tive,” it cannot be proved that they are not attributable Drug development is often a lengthy and expensive process. Extensive preclinical testing via in vitro and animal experimentation aims to select drugs most likely to work in humans. Under the current system, only about half of the drugs succeed in moving from phase 1 (dosefinding) to phase 2 (safety and efficacy).1 For drugs that enter phase 2, less than 1 in 3 succeed; for those entering phase 3 (pivotal efficacy), that number decreases to less than 1 in 2.1,2 Less than 20% of drugs entering phase 1 testing successfully reach the end of the 3-phase evaluation. The percentage can vary from one specialty area to another, and it can be less than 5% to 10% for oncologic and neurologic diseases.3 Some observers questioning the efficiency of the process have suggested that the role of randomized trials should be deemphasized. Conversely, the current development system can be improved with more routine adoption of rigorous randomized designs, both in the preclinical screening and during all phases of clinical testing. The current preclinical screening evaluation in animals and cell cultures may not be efficient and might not be informative about what will happen in humans. However, there could be a different interpretation. Pharmaceutical companies test about 1 600 000
jama.com
JAMA January 22/29, 2014 Volume 311, Number 4
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Yale University User on 05/16/2015
355
Opinion Viewpoint
to chance or a favorable sample of selected patients; if the results are unfavorable, the reverse arguments can be raised. Moreover, from an ethical perspective, in a randomized phase 1 or 2 trial the patients will have a 50% chance of being assigned to a better treatment, whereas in a single-group phase 1 or 2 trial the patients will be allocated to a treatment that (based on assumed eventual success rates) is far less than 50% likely to be the best currently available. The unethical dimension of nonrandomized clinical studies had already been recognized by Chalmers. In addition to being more ethical, adoption of randomized design throughout the drug development process would most likely improve the efficiency, ie, enable faster development of new, successful treatments. The proposed shift to randomized phase 1 and 2 trials could also help more efficiently identify ineffective or harmful treatments. This would be easier and more accurate to demonstrate in a randomized design than in an uncontrolled setting. Uncontrolled studies may lead to the abandonment of some potentially useful treatments or may fail to demonstrate the problems of many ineffective treatARTICLE INFORMATION Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. Funding/Support: The authors were supported in part by National Institutes of Health (NIH) grants 1R01NS044417-01, 1 R01 NS052956-01, and 1R01CA133594-01 (NIH Office of Research Integrity). The Meta-Research Innovation Center at Stanford is supported by a grant from the Laura and John Arnold Foundation.
356
ments and protract expensive clinical testing. For a development system with substantially high failure rates at all steps, not optimizing the study designs and their accuracy makes no sense. It could be argued that phase 1 or 2 studies may offer guidance for the design of the phase 3 trial, eg, by suggesting what the plausible effect size might be and thus informing sample size calculations for the definitive phase 3 trial. However, plausible effect sizes are far more likely to be accurately informed from a controlled trial than an uncontrolled one. Moreover, most phase 3 trials should focus on what treatment effect sizes are clinically meaningful to detect rather than try to secure a nominally statistically significant but clinically unimportant result based on the “insider” information supposedly provided by phase 1 or 2 data. The drug development process could be substantially improved if rigorous randomized trials become the first rather than the last step in the process of discovery of new, effective drugs and if randomization permeates testing at all stages. Both for animals and for humans, more, not fewer, randomized trials are needed.
2. Djulbegovic B, Kumar A, Glasziou P, Miladinovic B, Chalmers I. Medical research: trial unpredictability yields predictable therapy gains. Nature. 2013;500(7463):395-396. 3. Bhattacharjee Y. Biomedicine. Pharma firms push for sharing of cancer trial data. Science. 2012;338(6103):29. 4. Pisano G. Can science be a business? lessons from biotech. Harvard Business Review. 2006;84(10)1-13.
REFERENCES
5. Tsilidis KK, Panagiotou OA, Sena ES, et al. Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol. 2013;11(7):e1001609.
1. Paul SM, Mytelka DS, Dunwiddie CT, et al. How to improve R&D productivity: the pharmaceutical industry’s grand challenge. Nat Rev Drug Discov. 2010;9(3):203-214.
6. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011;10(9):712-713.
7. Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature. 2012;483(7391):531-533. 8. Landis SC, Amara SG, Asadullah K, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012;490(7419):187-191. 9. Chalmers TC. Randomization of the first patient. Med Clin North Am. 1975;59(4):1035-1038. 10. Horstmann E, McCabe MS, Grochow L, et al. Risks and benefits of phase 1 oncology trials, 1991 through 2002. N Engl J Med. 2005;352(9): 895-904.
JAMA January 22/29, 2014 Volume 311, Number 4
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a Yale University User on 05/16/2015
jama.com
