News & Analysis
“The cancer focus, I think, is a natural,” he said. “To be able to ramp that up is timely and very appealing.” Even prior to the announcement of the Precision Medicine Initiative, the NIH launched 3 trials that are screening patients’ tumors for genetic abnormalities. Based on the screening results, patients are enrolled into a clinical trial of an investigational drug that might target the specific molecular pathology driving their cancer (http: //1.usa.gov/1EdWhDQ). The NCI MATCH Trial is enrolling adults with advanced solid tumors and lymphomas that have stopped responding to standard therapy. Pediatric MATCH is the counterpart trial for children. The third trial, the Lung Master Protocol, or Lung-MAP, is a public-private collaborative
effort implementing a similar approach for patients with squamous cell lung cancer, which currently has few treatment options beyond surgery (http://bit.ly/1zIq85a). The participants in these trials could potentially become a part of the Precision Medicine Initiative cohort. “Efforts such as Lung-MAP and the NCI MATCH program are within the scope of this initiative,” said Vassiliki Papadimitrakopoulou, MD, professor of medicine at MD Anderson Cancer Center and a co–principal investigator for Lung-MAP. In September 2011, Dana-Farber also launched its Profile program, becoming the first hospital to offer tumor genotyping to all patients with cancer, Rollins said. “We see 16 000 new patients a year. We want to turn them into a giant cohort
study, like the Framingham Study,” he said. “Breast cancer patients with a mutation in gene X: do they live longer? do they live shorter? That’s a very powerful scientific thing to do.” Rollins said Dana-Farber “would definitely consider contributing the Profile patient cohort to the Precision Medicine Initiative if the protections for patient confidentiality are sufficiently stringent.” While improving the treatment of cancer is one of the initiative’s shorter-term aims, it has a more ambitious longer-term goal, Collins said at last month’s NIH meeting. Eventually, he said, the plan is to “generate the knowledge base necessary to move precision medicine into virtually all areas of health and disease.”
The JAMA Forum
An Observational Study Goes Where Randomized Clinical Trials Have Not Austin B. Frakt, PhD
Doug Levy
R
andomized clinical trials (RCTs) are considered the “gold standard” for providing actionable evidence to guide clinical decision making. However, they cannot always address important questions. For instance, statistically significant results for low-frequency outcomes like mortality sometimes require longer follow-up times or larger studies than can be practically undertaken. In such cases, we have a choice: we can either go without evidence or we can turn to observational studies. Such studies often can be much larger and accommodate longer follow-ups. But because participants are not explicitly randomly assigned to treatment and control groups, observational studies can produce biased results. There are, however, advances in methods that can minimize that bias and increase our confidence in findings. Consider a recent observational study comparing 2 classes of drugs that are used to treat type 2 diabetes when initial treatment fails to control blood-sugar level (http: //bit.ly/1E2RSV3). The investigators found that use of sulfonylureas (SUs) is associated with greater mortality and more avoidable hospitalizations than the use of thiazo-
jama.com
lidinediones (TZDs). Both types of medications are frequently prescribed as second-line treatments, but concerns about the cardiovascular safety of TZDs have been a focus of controversy. Given the new findings, should clinicians now favor TZDs? That depends on an assessment of the study’s sample and methods. An advantage of the study, published in Value in Health, is that it was 20 times larger and had a much longer follow-up than any prior comparative-effectiveness RCT of second-line diabetes medications (http://bit.ly/1E2RSV3). This permitted the investigators to obtain outcomes on lowfrequency events like mortality; prior RCTs principally only had power to examine blood glucose control. The study, based on merged Veterans Health Administration (VA) and Medicare data, examined more than 80 000 patients for up to 10 years. It found that, relative to TZDs, SUs cause a 68% increase in risk of avoidable hospitalization and a 50% increase in risk of death. (Full disclosure: the study was led by my colleague Julia Prentice, PhD, and I work closely with another coauthor, Steven Pizer, PhD.)
Austin B. Frakt, PhD
That the study by Prentice and colleagues is not an RCT should be viewed as only a mild weakness, in my view, because the instrumental variable methods it employed were rigorously tested and found to be very strong. Admittedly, some would disagree with the interpretation of this design as a mild weakness. The key threat to any observational study is bias from factors that cannot be observed and controlled. Randomized clinical
(Reprinted) JAMA March 17, 2015 Volume 313, Number 11
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: on 12/20/2017
1091
News & Analysis
trials address this issue with randomization, the proverbial flip of a coin. As its source of random variation, the Prentice et al study used physician prescribing patterns instead. Patients were effectively randomized to receive an SU or TZD according to the how frequently their physician prescribed one over the other in the prior year. Patients cared for by a clinician more likely to prescribe an SU can be thought of as more likely to be randomized to receive an SU, and similarly for TZDs. Supporting this approach, prescribing pattern has been applied as an instrument in prior work (http://1.usa .gov/1sZhvC4, http://1.usa.gov/1yI1vpa, and http://bit.ly/1zx0MZd). Such prescribing patterns are only a valid source of randomization if, like an RCT’s coin flip, they are not correlated with any unobservable factor that also affects outcomes (for example, general quality of care). This is the key assumption and the one many people find hard to swallow for instrumental variable studies. How do we increase our confidence this assumption holds in this case? If prescribing pattern is a good randomizer, there should be a balance of observable factors, like demographics or prevalence of other diagnoses, among patients more commonly prescribed SUs or TDZs (http://bit.ly /15zRYFt). If we observe such balance, that should increase our confidence that there is also balance among unobservable factors too, just like in an RCT. Checking balance on observables is a falsification test, and one that is standard in RCT reporting (http://bit .ly/1EsuBcl). Prentice’s team conducted this falsification test and showed balance in demographic, diagnoses, and provider quality variables. That’s all we expect to see to convince us of an RCT’s validity, but this is an instrumental variable study, so Prentice’s team did more. Imagine if an RCT’s coin flip was found to affect outcomes even in a population that never received the treatment under study
1092
(that is, those assigned to treatment didn’t receive it). This would suggest the coin flip that was thought to be random really was not, invalidating the RCT. This can occur if, for example, there is a breakdown in procedure and people assigned to the treatment group are systematically different than those assigned to the control group. The same logic applies to instrumental variable studies, and Prentice and coauthors looked for evidence of such a problem. The authors examined 2 populations that bracketed the study population in disease severity and did not receive the treatment under study: a healthier population taking metformin (typically the first-line treatment for type 2 diabetes) but not receiving a second-line treatment and a sicker population that had been prescribed metformin and then insulin without any other diabetes drug. The same potential bias related to causal factors that are unobservable to the researcher (if there are any) is as likely to apply to these 2 populations as to the primary sample. So if prescribing patterns were correlated with outcomes in these populations, that would invalidate it as a randomizer. (A JAMA Viewpoint by Vinay Prasad, MD, and Anupam B. Jena, MD, PhD, also describes the application of falsification tests, but to postmarketing surveillance of medications (http://bit.ly/15ufUJM).) For neither of these “bracketing” populations in the study by Prentice et al was prescribing pattern related to outcomes. This provides strong support for prescribing pattern as a randomizer. It’s consistent with the assumption that prescribing pattern only affects outcomes through its effect on treatment with SU or TZD, just as one would desire of an RCT’s coin flip. In summary, there is considerable evidence that, in this study, prescribing pattern randomizes patients to SU or TZD in a manner that one would expect of an RCT. Consequently, there should be some confidence that the findings are reporting
something valid. Should they be replicated with an RCT before they can be trusted? Maybe the more important question is, “Could an RCT be fielded?” It would be unprecedented. Neither a New England Comparative Effectiveness Public Advisory Council literature review (http://bit.ly/1Ch3How) nor one by the Canadian Agency for Drugs and Technologies in Health (http://1.usa.gov/1xzqC9r) found an adequately powered RCT for examination of mortality or development of diabetesrelated complications related to secondline treatments. In principle, one could be undertaken. In practice, amassing enough participants to maintain study protocols for long enough is a big challenge, perhaps an insurmountable one. If that’s true, and clinicians want to make evidence-based decisions based on mortality for second-line treatment of type 2 diabetes (or in similar clinical situations), they have no choice but to rely on observational studies. Is the study by Prentice et al strong enough? I’m interested in your answer to this question. You may submit comments to me at [email protected]. Author Affiliation: Health economist with the Boston VA Healthcare System, and associate professor at Boston University, School of Medicine and School of Public Health, Boston, Massachusetts. Corresponding Author: Austin B. Frakt, PhD ([email protected]). Published online January 15, 2015, at http: //newsatjama.jama.com/category/the-jama-forum/. Disclaimer: Each entry in The JAMA Forum expresses the opinions of the author but does not necessarily reflect the views or opinions of JAMA, the editorial staff, or the American Medical Association. The views expressed in this post are those of the author and do not necessarily reflect the position of the Department of Veterans Affairs or Boston University. Additional Information: Information about The JAMA Forum is available at http://newsatjama.jama .com/about/. Information about disclosures of potential conflicts of interest may be found at http: //newsatjama.jama.com/jama-forum-disclosures/.
JAMA March 17, 2015 Volume 313, Number 11 (Reprinted)
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: on 12/20/2017
jama.com
“The cancer focus, I think, is a natural,” he said. “To be able to ramp that up is timely and very appealing.” Even prior to the announcement of the Precision Medicine Initiative, the NIH launched 3 trials that are screening patients’ tumors for genetic abnormalities. Based on the screening results, patients are enrolled into a clinical trial of an investigational drug that might target the specific molecular pathology driving their cancer (http: //1.usa.gov/1EdWhDQ). The NCI MATCH Trial is enrolling adults with advanced solid tumors and lymphomas that have stopped responding to standard therapy. Pediatric MATCH is the counterpart trial for children. The third trial, the Lung Master Protocol, or Lung-MAP, is a public-private collaborative
effort implementing a similar approach for patients with squamous cell lung cancer, which currently has few treatment options beyond surgery (http://bit.ly/1zIq85a). The participants in these trials could potentially become a part of the Precision Medicine Initiative cohort. “Efforts such as Lung-MAP and the NCI MATCH program are within the scope of this initiative,” said Vassiliki Papadimitrakopoulou, MD, professor of medicine at MD Anderson Cancer Center and a co–principal investigator for Lung-MAP. In September 2011, Dana-Farber also launched its Profile program, becoming the first hospital to offer tumor genotyping to all patients with cancer, Rollins said. “We see 16 000 new patients a year. We want to turn them into a giant cohort
study, like the Framingham Study,” he said. “Breast cancer patients with a mutation in gene X: do they live longer? do they live shorter? That’s a very powerful scientific thing to do.” Rollins said Dana-Farber “would definitely consider contributing the Profile patient cohort to the Precision Medicine Initiative if the protections for patient confidentiality are sufficiently stringent.” While improving the treatment of cancer is one of the initiative’s shorter-term aims, it has a more ambitious longer-term goal, Collins said at last month’s NIH meeting. Eventually, he said, the plan is to “generate the knowledge base necessary to move precision medicine into virtually all areas of health and disease.”
The JAMA Forum
An Observational Study Goes Where Randomized Clinical Trials Have Not Austin B. Frakt, PhD
Doug Levy
R
andomized clinical trials (RCTs) are considered the “gold standard” for providing actionable evidence to guide clinical decision making. However, they cannot always address important questions. For instance, statistically significant results for low-frequency outcomes like mortality sometimes require longer follow-up times or larger studies than can be practically undertaken. In such cases, we have a choice: we can either go without evidence or we can turn to observational studies. Such studies often can be much larger and accommodate longer follow-ups. But because participants are not explicitly randomly assigned to treatment and control groups, observational studies can produce biased results. There are, however, advances in methods that can minimize that bias and increase our confidence in findings. Consider a recent observational study comparing 2 classes of drugs that are used to treat type 2 diabetes when initial treatment fails to control blood-sugar level (http: //bit.ly/1E2RSV3). The investigators found that use of sulfonylureas (SUs) is associated with greater mortality and more avoidable hospitalizations than the use of thiazo-
jama.com
lidinediones (TZDs). Both types of medications are frequently prescribed as second-line treatments, but concerns about the cardiovascular safety of TZDs have been a focus of controversy. Given the new findings, should clinicians now favor TZDs? That depends on an assessment of the study’s sample and methods. An advantage of the study, published in Value in Health, is that it was 20 times larger and had a much longer follow-up than any prior comparative-effectiveness RCT of second-line diabetes medications (http://bit.ly/1E2RSV3). This permitted the investigators to obtain outcomes on lowfrequency events like mortality; prior RCTs principally only had power to examine blood glucose control. The study, based on merged Veterans Health Administration (VA) and Medicare data, examined more than 80 000 patients for up to 10 years. It found that, relative to TZDs, SUs cause a 68% increase in risk of avoidable hospitalization and a 50% increase in risk of death. (Full disclosure: the study was led by my colleague Julia Prentice, PhD, and I work closely with another coauthor, Steven Pizer, PhD.)
Austin B. Frakt, PhD
That the study by Prentice and colleagues is not an RCT should be viewed as only a mild weakness, in my view, because the instrumental variable methods it employed were rigorously tested and found to be very strong. Admittedly, some would disagree with the interpretation of this design as a mild weakness. The key threat to any observational study is bias from factors that cannot be observed and controlled. Randomized clinical
(Reprinted) JAMA March 17, 2015 Volume 313, Number 11
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: on 12/20/2017
1091
News & Analysis
trials address this issue with randomization, the proverbial flip of a coin. As its source of random variation, the Prentice et al study used physician prescribing patterns instead. Patients were effectively randomized to receive an SU or TZD according to the how frequently their physician prescribed one over the other in the prior year. Patients cared for by a clinician more likely to prescribe an SU can be thought of as more likely to be randomized to receive an SU, and similarly for TZDs. Supporting this approach, prescribing pattern has been applied as an instrument in prior work (http://1.usa .gov/1sZhvC4, http://1.usa.gov/1yI1vpa, and http://bit.ly/1zx0MZd). Such prescribing patterns are only a valid source of randomization if, like an RCT’s coin flip, they are not correlated with any unobservable factor that also affects outcomes (for example, general quality of care). This is the key assumption and the one many people find hard to swallow for instrumental variable studies. How do we increase our confidence this assumption holds in this case? If prescribing pattern is a good randomizer, there should be a balance of observable factors, like demographics or prevalence of other diagnoses, among patients more commonly prescribed SUs or TDZs (http://bit.ly /15zRYFt). If we observe such balance, that should increase our confidence that there is also balance among unobservable factors too, just like in an RCT. Checking balance on observables is a falsification test, and one that is standard in RCT reporting (http://bit .ly/1EsuBcl). Prentice’s team conducted this falsification test and showed balance in demographic, diagnoses, and provider quality variables. That’s all we expect to see to convince us of an RCT’s validity, but this is an instrumental variable study, so Prentice’s team did more. Imagine if an RCT’s coin flip was found to affect outcomes even in a population that never received the treatment under study
1092
(that is, those assigned to treatment didn’t receive it). This would suggest the coin flip that was thought to be random really was not, invalidating the RCT. This can occur if, for example, there is a breakdown in procedure and people assigned to the treatment group are systematically different than those assigned to the control group. The same logic applies to instrumental variable studies, and Prentice and coauthors looked for evidence of such a problem. The authors examined 2 populations that bracketed the study population in disease severity and did not receive the treatment under study: a healthier population taking metformin (typically the first-line treatment for type 2 diabetes) but not receiving a second-line treatment and a sicker population that had been prescribed metformin and then insulin without any other diabetes drug. The same potential bias related to causal factors that are unobservable to the researcher (if there are any) is as likely to apply to these 2 populations as to the primary sample. So if prescribing patterns were correlated with outcomes in these populations, that would invalidate it as a randomizer. (A JAMA Viewpoint by Vinay Prasad, MD, and Anupam B. Jena, MD, PhD, also describes the application of falsification tests, but to postmarketing surveillance of medications (http://bit.ly/15ufUJM).) For neither of these “bracketing” populations in the study by Prentice et al was prescribing pattern related to outcomes. This provides strong support for prescribing pattern as a randomizer. It’s consistent with the assumption that prescribing pattern only affects outcomes through its effect on treatment with SU or TZD, just as one would desire of an RCT’s coin flip. In summary, there is considerable evidence that, in this study, prescribing pattern randomizes patients to SU or TZD in a manner that one would expect of an RCT. Consequently, there should be some confidence that the findings are reporting
something valid. Should they be replicated with an RCT before they can be trusted? Maybe the more important question is, “Could an RCT be fielded?” It would be unprecedented. Neither a New England Comparative Effectiveness Public Advisory Council literature review (http://bit.ly/1Ch3How) nor one by the Canadian Agency for Drugs and Technologies in Health (http://1.usa.gov/1xzqC9r) found an adequately powered RCT for examination of mortality or development of diabetesrelated complications related to secondline treatments. In principle, one could be undertaken. In practice, amassing enough participants to maintain study protocols for long enough is a big challenge, perhaps an insurmountable one. If that’s true, and clinicians want to make evidence-based decisions based on mortality for second-line treatment of type 2 diabetes (or in similar clinical situations), they have no choice but to rely on observational studies. Is the study by Prentice et al strong enough? I’m interested in your answer to this question. You may submit comments to me at [email protected]. Author Affiliation: Health economist with the Boston VA Healthcare System, and associate professor at Boston University, School of Medicine and School of Public Health, Boston, Massachusetts. Corresponding Author: Austin B. Frakt, PhD ([email protected]). Published online January 15, 2015, at http: //newsatjama.jama.com/category/the-jama-forum/. Disclaimer: Each entry in The JAMA Forum expresses the opinions of the author but does not necessarily reflect the views or opinions of JAMA, the editorial staff, or the American Medical Association. The views expressed in this post are those of the author and do not necessarily reflect the position of the Department of Veterans Affairs or Boston University. Additional Information: Information about The JAMA Forum is available at http://newsatjama.jama .com/about/. Information about disclosures of potential conflicts of interest may be found at http: //newsatjama.jama.com/jama-forum-disclosures/.
JAMA March 17, 2015 Volume 313, Number 11 (Reprinted)
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: on 12/20/2017
jama.com
