The new era of clinical research using data for multiple purposes Robert M. Califf MD PII: DOI: Reference:
S0002-8703(14)00225-7 doi: 10.1016/j.ahj.2014.04.014 YMHJ 4617
To appear in:
American Heart Journal
Received date: Accepted date:
29 April 2014 29 April 2014
Please cite this article as: Califf Robert M., The new era of clinical research using data for multiple purposes, American Heart Journal (2014), doi: 10.1016/j.ahj.2014.04.014
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT The new era of clinical research using data for multiple purposes
RI P
T
Robert M. Califf, MD
From the Division of Cardiology, Department of Medicine, Duke University School of
SC
Medicine, Durham, NC, and the Duke Translational Medicine Institute, Durham, NC.
MA
NU
Word count: 983
Address for correspondence:
AC
CE
PT
ED
Robert M. Califf, MD, MACC Vice Chancellor for Clinical & Translational Research Director, Duke Translational Medicine Institute Donald F. Fortin Professor of Cardiology Duke University School of Medicine 200 Trent Street, 1117 Davison Bldg Durham, NC 27705 Tel: 919-668-8820; Fax: 919-668-7103 [email protected]
1
ACCEPTED MANUSCRIPT In this issue of the American Heart Journal, Kjøller and colleagues1 make the case that public registry data from patients entered into a randomized clinical trial (RCT) can be used to
T
estimate the effect of an intervention in patients with coronary artery disease, with a degree of
RI P
accuracy comparable to that possible with fully adjudicated, prospectively collected data. In a cleverly designed study, they were able to compare results from the randomized CLARICOR
SC
trial2 with cardiovascular event rates from an ongoing Danish public registry with the results
NU
derived from a clinical events committee (CEC) that reviewed the possible events using a predefined protocol. Interestingly, the absolute event rates were noticeably different: agreement
MA
was found between the 2 methods in only 74% of hospital discharges and 60% for cause of death. However, these major differences were equally distributed in the 2 treatment groups,
ED
leading to a similar estimate of treatment effect.
PT
The most common errors were false-positive attributions from the registry in cases that were ruled non-events by the adjudication committee. This is not surprising, because clinical
CE
diagnoses tend to err on the more serious side of classification, whereas an adjudication
AC
committee applies rigorous criteria that would exclude borderline cases. A similar issue with estimation of death from cardiovascular causes was observed, although this is less concerning, given the known difficulty determining cause of death with even the best data available. Ever since the beginning of organized clinical epidemiology, inter- and intra-observer variability has been observed to be high for clinical phenomena.3,4 There is no “magic number” that constitutes an acceptable rate of agreement between rigorous CEC classification and routine clinical records,5 but classification errors can affect the absolute estimate of the number needed to treat or harm as well as estimates of statistical significance, even when the relative treatment effect is not altered. More studies like the one performed by Kjøller and colleagues are needed, if
2
ACCEPTED MANUSCRIPT we are to develop a better sense of how to interpret data drawn from public sources or collected during routine care. Fortunately, rapidly evolving methods that exploit electronic health records
T
(EHRs) and registries now offer the potential ability to accelerate evidence development by a log
RI P
order or more while simultaneously reducing research costs by the same proportion. The term computable phenotype encompasses methods that employ routinely collected data
SC
to determine whether a diagnosis of a condition exists or a clinical event has occurred. In theory,
NU
such methods can be applied across the large volumes of data contained in the records of multiple health systems to answer research questions. However, this seemingly simple concept is
MA
at present greatly complicated by the fact that each health systems records and codes information differently, necessitating painstaking characterization of the operating characteristics for each
ED
measure in each system. Meanwhile, the proliferation of data standards has given rise to the quip
PT
that “…the wonderful thing about medical informatics standards is that there are so many to choose from.”
CE
However, it is only a matter of time until a tipping point is reached, especially given intense
AC
efforts from the Office of the National Coordinator for Health Information Technology and increasingly coordinated activity by the NIH. Once this point is reached, we will likely see rapid convergence on “universal” standards for recording clinical and laboratory information that will enable aggregate analysis across multiple platforms and systems. The urgency of this issue warrants considerable attention from the clinical care, research, and regulatory communities. The vast majority of clinical recommendations are not supported by high-quality evidence,6 leading to significant variations in practice. In many cases, when welldesigned and adequately powered clinical trials are finally done, they reveal surprising results. Examples of therapies that have unexpectedly proven harmful include high-dose
3
ACCEPTED MANUSCRIPT erythropoietin,7,8 antiarrhythmic drugs,9 and intensive glucose lowering.10 The costs of clinical trials are escalating and most do not enroll on time. Further, a very high proportion of trials are
T
inadequately sized to furnish the needed statistical power to answer critical clinical questions.11
RI P
Amid these challenges, there is growing enthusiasm for using registries and EHRs to identify cohorts, to serve as the core data sources for characterizing populations, and to measure
SC
outcomes. Recent examples include a cardiac catheterization registry in Sweden12 and the U.S.
NU
National Cardiovascular Data Registry (NCDR).13 The further evolution of these concepts is represented by NIH Healthcare Systems Research Collaboratory,14 which has launched a series
MA
of pragmatic trials using EHRs within integrated health systems at a fraction of the cost of usual clinical trials. This experimental program, administered by the NIH Common Fund,15 has been
ED
sufficiently successful that multiple NIH funding opportunities have now been announced for
PT
similar pragmatic trials across multiple institutes. Another recent initiative to explore these possibilities is the newly-formed Patient-Centered
CE
Outcomes Research Network (PCORnet; http://www.pcornet.org/). Among the largest clinical
AC
research efforts in history, this national “network of networks” comprises 11 major clinical data research networks (CDRNs) encompassing up to 100 million Americans and 18 patient-powered research networks (PPRNs) driven by patients and advocates with an intense interest in specific diseases both rare and common. For both the Collaboratory and PCORnet, a major informatics effort is under way to catalog computable phenotypes and make them publicly available. Unfortunately, as noted earlier, variations in EHR data coding immensely complicate this undertaking. Recent evaluation of data coding for the diagnosis of type 2 diabetes mellitus across the Duke University system found
4
ACCEPTED MANUSCRIPT almost an almost twofold difference in the number of people with the diagnosis, depending on the choice among definitions from a series of previously published computable phenotypes.16
T
The “end game” for clinical research is becoming clearer. In many parts of the world, we are
RI P
nearing 100% penetration of EHRs, with integrated health systems creating highly functional enterprise data warehouses. Ultimately, a comprehensive fabric of information will connect
SC
people across previous boundaries, radically accelerating evidence generation and consumption
NU
and leading to global sharing of knowledge among researchers, clinicians, policy makers, and
AC
CE
PT
ED
MA
patients and their families.
5
ACCEPTED MANUSCRIPT References 1.
Kjøller E, Hilden J, Winkel P, et al. Agreement between public register and
T
adjudication committee outcome in a cardiovascular randomized clinical trial. Am
2.
RI P
Heart J 2014.
Jespersen CM1, Als-Nielsen B, Damgaard M, et al. Randomised placebo controlled
SC
multicentre trial to assess short term clarithromycin for patients with stable coronary heart disease: CLARICOR trial. BMJ 2006;332:22-7.
Feinstein AR, Kramer MS. Clinical biostatistics. LIII. The architecture of
NU
3.
observer/method variability and other types of process research. Clin Pharmacol Ther
4.
MA
1980;28:551-63.
Jollis JG, Ancukiewicz M, DeLong ER, et al. Discordance of databases designed for
ED
claims payment versus clinical information systems: Implications for outcomes research. Ann Intern Med 1993,119:844-50 Lopes RD, Dickerson S, Hafley G, et al. Methodology of a reevaluation of
PT
5.
cardiovascular outcomes in the RECORD trial: study design and conduct. Am Heart J
Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence
AC
6.
CE
2013;166:208-216.e28.
underlying the ACC/AHA clinical practice guidelines. JAMA 2009;301:831-41. 7.
Pfeffer MA, Burdmann EA, Chen CY, et al. A trial of darbepoetin alfa in type 2 diabetes and chronic kidney disease. N Engl J Med 2009;361:2019-32.
8.
Singh AK, Szczech L, Tang KL, et al. Correction of anemia with epoetin alfa in chronic kidney disease. N Engl J Med 2006;355:2085-98.
9.
Echt DS, Liebson PR, Mitchell LB, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med 1991;324:781-8.
6
ACCEPTED MANUSCRIPT 10. Gerstein HC, Miller ME, Byington RP, et al. Effects of intensive glucose lowering in type 2 diabetes. N Engl J Med. 2008;358:2545-59.
RI P
ClinicalTrials.gov, 2007-2010. JAMA 2012;307:1838-47.
T
11. Califf RM, Zarin DA, Kramer JM, et al. Characteristics of clinical trials registered in
12. Fröbert O, Lagerqvist B, Olivecrona GK, et al. Thrombus Aspiration during ST-
SC
Segment Elevation Myocardial Infarction. N Engl J Med 2013;369:1587-97. 13. Hess CN, Rao SV, Kong DF, et al. Embedding a randomized clinical trial into an
NU
ongoing registry infrastructure: Unique opportunities for efficiency in design of the Study of Access site For Enhancement of Percutaneous Coronary Intervention for
MA
Women (SAFE-PCI for Women). Am Heart J 2013;166:421-428. 14. NIH Health Systems Research Collaboratory website. Available at:
ED
https://www.nihcollaboratory.org/Pages/default.aspx. Accessed April 1, 2014. 15. National Institutes of Health website. Office of Strategic Coordination - the Common
PT
Fund. Available at: http://commonfund.nih.gov/. Accessed April 1, 2014.
CE
16. Richesson RL, Rusincovitch SA, Wixted D, et al. A comparison of phenotype
AC
definitions for diabetes mellitus. J Am Med Inform Assoc 2013;20:e319-26.
7
S0002-8703(14)00225-7 doi: 10.1016/j.ahj.2014.04.014 YMHJ 4617
To appear in:
American Heart Journal
Received date: Accepted date:
29 April 2014 29 April 2014
Please cite this article as: Califf Robert M., The new era of clinical research using data for multiple purposes, American Heart Journal (2014), doi: 10.1016/j.ahj.2014.04.014
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT The new era of clinical research using data for multiple purposes
RI P
T
Robert M. Califf, MD
From the Division of Cardiology, Department of Medicine, Duke University School of
SC
Medicine, Durham, NC, and the Duke Translational Medicine Institute, Durham, NC.
MA
NU
Word count: 983
Address for correspondence:
AC
CE
PT
ED
Robert M. Califf, MD, MACC Vice Chancellor for Clinical & Translational Research Director, Duke Translational Medicine Institute Donald F. Fortin Professor of Cardiology Duke University School of Medicine 200 Trent Street, 1117 Davison Bldg Durham, NC 27705 Tel: 919-668-8820; Fax: 919-668-7103 [email protected]
1
ACCEPTED MANUSCRIPT In this issue of the American Heart Journal, Kjøller and colleagues1 make the case that public registry data from patients entered into a randomized clinical trial (RCT) can be used to
T
estimate the effect of an intervention in patients with coronary artery disease, with a degree of
RI P
accuracy comparable to that possible with fully adjudicated, prospectively collected data. In a cleverly designed study, they were able to compare results from the randomized CLARICOR
SC
trial2 with cardiovascular event rates from an ongoing Danish public registry with the results
NU
derived from a clinical events committee (CEC) that reviewed the possible events using a predefined protocol. Interestingly, the absolute event rates were noticeably different: agreement
MA
was found between the 2 methods in only 74% of hospital discharges and 60% for cause of death. However, these major differences were equally distributed in the 2 treatment groups,
ED
leading to a similar estimate of treatment effect.
PT
The most common errors were false-positive attributions from the registry in cases that were ruled non-events by the adjudication committee. This is not surprising, because clinical
CE
diagnoses tend to err on the more serious side of classification, whereas an adjudication
AC
committee applies rigorous criteria that would exclude borderline cases. A similar issue with estimation of death from cardiovascular causes was observed, although this is less concerning, given the known difficulty determining cause of death with even the best data available. Ever since the beginning of organized clinical epidemiology, inter- and intra-observer variability has been observed to be high for clinical phenomena.3,4 There is no “magic number” that constitutes an acceptable rate of agreement between rigorous CEC classification and routine clinical records,5 but classification errors can affect the absolute estimate of the number needed to treat or harm as well as estimates of statistical significance, even when the relative treatment effect is not altered. More studies like the one performed by Kjøller and colleagues are needed, if
2
ACCEPTED MANUSCRIPT we are to develop a better sense of how to interpret data drawn from public sources or collected during routine care. Fortunately, rapidly evolving methods that exploit electronic health records
T
(EHRs) and registries now offer the potential ability to accelerate evidence development by a log
RI P
order or more while simultaneously reducing research costs by the same proportion. The term computable phenotype encompasses methods that employ routinely collected data
SC
to determine whether a diagnosis of a condition exists or a clinical event has occurred. In theory,
NU
such methods can be applied across the large volumes of data contained in the records of multiple health systems to answer research questions. However, this seemingly simple concept is
MA
at present greatly complicated by the fact that each health systems records and codes information differently, necessitating painstaking characterization of the operating characteristics for each
ED
measure in each system. Meanwhile, the proliferation of data standards has given rise to the quip
PT
that “…the wonderful thing about medical informatics standards is that there are so many to choose from.”
CE
However, it is only a matter of time until a tipping point is reached, especially given intense
AC
efforts from the Office of the National Coordinator for Health Information Technology and increasingly coordinated activity by the NIH. Once this point is reached, we will likely see rapid convergence on “universal” standards for recording clinical and laboratory information that will enable aggregate analysis across multiple platforms and systems. The urgency of this issue warrants considerable attention from the clinical care, research, and regulatory communities. The vast majority of clinical recommendations are not supported by high-quality evidence,6 leading to significant variations in practice. In many cases, when welldesigned and adequately powered clinical trials are finally done, they reveal surprising results. Examples of therapies that have unexpectedly proven harmful include high-dose
3
ACCEPTED MANUSCRIPT erythropoietin,7,8 antiarrhythmic drugs,9 and intensive glucose lowering.10 The costs of clinical trials are escalating and most do not enroll on time. Further, a very high proportion of trials are
T
inadequately sized to furnish the needed statistical power to answer critical clinical questions.11
RI P
Amid these challenges, there is growing enthusiasm for using registries and EHRs to identify cohorts, to serve as the core data sources for characterizing populations, and to measure
SC
outcomes. Recent examples include a cardiac catheterization registry in Sweden12 and the U.S.
NU
National Cardiovascular Data Registry (NCDR).13 The further evolution of these concepts is represented by NIH Healthcare Systems Research Collaboratory,14 which has launched a series
MA
of pragmatic trials using EHRs within integrated health systems at a fraction of the cost of usual clinical trials. This experimental program, administered by the NIH Common Fund,15 has been
ED
sufficiently successful that multiple NIH funding opportunities have now been announced for
PT
similar pragmatic trials across multiple institutes. Another recent initiative to explore these possibilities is the newly-formed Patient-Centered
CE
Outcomes Research Network (PCORnet; http://www.pcornet.org/). Among the largest clinical
AC
research efforts in history, this national “network of networks” comprises 11 major clinical data research networks (CDRNs) encompassing up to 100 million Americans and 18 patient-powered research networks (PPRNs) driven by patients and advocates with an intense interest in specific diseases both rare and common. For both the Collaboratory and PCORnet, a major informatics effort is under way to catalog computable phenotypes and make them publicly available. Unfortunately, as noted earlier, variations in EHR data coding immensely complicate this undertaking. Recent evaluation of data coding for the diagnosis of type 2 diabetes mellitus across the Duke University system found
4
ACCEPTED MANUSCRIPT almost an almost twofold difference in the number of people with the diagnosis, depending on the choice among definitions from a series of previously published computable phenotypes.16
T
The “end game” for clinical research is becoming clearer. In many parts of the world, we are
RI P
nearing 100% penetration of EHRs, with integrated health systems creating highly functional enterprise data warehouses. Ultimately, a comprehensive fabric of information will connect
SC
people across previous boundaries, radically accelerating evidence generation and consumption
NU
and leading to global sharing of knowledge among researchers, clinicians, policy makers, and
AC
CE
PT
ED
MA
patients and their families.
5
ACCEPTED MANUSCRIPT References 1.
Kjøller E, Hilden J, Winkel P, et al. Agreement between public register and
T
adjudication committee outcome in a cardiovascular randomized clinical trial. Am
2.
RI P
Heart J 2014.
Jespersen CM1, Als-Nielsen B, Damgaard M, et al. Randomised placebo controlled
SC
multicentre trial to assess short term clarithromycin for patients with stable coronary heart disease: CLARICOR trial. BMJ 2006;332:22-7.
Feinstein AR, Kramer MS. Clinical biostatistics. LIII. The architecture of
NU
3.
observer/method variability and other types of process research. Clin Pharmacol Ther
4.
MA
1980;28:551-63.
Jollis JG, Ancukiewicz M, DeLong ER, et al. Discordance of databases designed for
ED
claims payment versus clinical information systems: Implications for outcomes research. Ann Intern Med 1993,119:844-50 Lopes RD, Dickerson S, Hafley G, et al. Methodology of a reevaluation of
PT
5.
cardiovascular outcomes in the RECORD trial: study design and conduct. Am Heart J
Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence
AC
6.
CE
2013;166:208-216.e28.
underlying the ACC/AHA clinical practice guidelines. JAMA 2009;301:831-41. 7.
Pfeffer MA, Burdmann EA, Chen CY, et al. A trial of darbepoetin alfa in type 2 diabetes and chronic kidney disease. N Engl J Med 2009;361:2019-32.
8.
Singh AK, Szczech L, Tang KL, et al. Correction of anemia with epoetin alfa in chronic kidney disease. N Engl J Med 2006;355:2085-98.
9.
Echt DS, Liebson PR, Mitchell LB, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med 1991;324:781-8.
6
ACCEPTED MANUSCRIPT 10. Gerstein HC, Miller ME, Byington RP, et al. Effects of intensive glucose lowering in type 2 diabetes. N Engl J Med. 2008;358:2545-59.
RI P
ClinicalTrials.gov, 2007-2010. JAMA 2012;307:1838-47.
T
11. Califf RM, Zarin DA, Kramer JM, et al. Characteristics of clinical trials registered in
12. Fröbert O, Lagerqvist B, Olivecrona GK, et al. Thrombus Aspiration during ST-
SC
Segment Elevation Myocardial Infarction. N Engl J Med 2013;369:1587-97. 13. Hess CN, Rao SV, Kong DF, et al. Embedding a randomized clinical trial into an
NU
ongoing registry infrastructure: Unique opportunities for efficiency in design of the Study of Access site For Enhancement of Percutaneous Coronary Intervention for
MA
Women (SAFE-PCI for Women). Am Heart J 2013;166:421-428. 14. NIH Health Systems Research Collaboratory website. Available at:
ED
https://www.nihcollaboratory.org/Pages/default.aspx. Accessed April 1, 2014. 15. National Institutes of Health website. Office of Strategic Coordination - the Common
PT
Fund. Available at: http://commonfund.nih.gov/. Accessed April 1, 2014.
CE
16. Richesson RL, Rusincovitch SA, Wixted D, et al. A comparison of phenotype
AC
definitions for diabetes mellitus. J Am Med Inform Assoc 2013;20:e319-26.
7
