SPECIAL FOCUS y The ARRA investment in CER Research Article For reprint orders, please contact: [email protected]
American Recovery and Reinvestment Act investments in data infrastructure
Aim: This article describes American Reinvestment and Recovery Act comparative effectiveness research data infrastructure (DI) investments and identifies facilitators and barriers to implementation. Materials & methods: We reviewed original project proposals, conducted an investigator survey and interviewed project officers and principal investigators. Results: DI projects assembled or enhanced existing clinical datasets, established linkages between public and private data sources and built infrastructure. Facilitators included building on existing relationships across organizations and making collection as seamless as possible for clinicians. Conclusion: To sustain DI, investigators should reduce the burden of comparative effectiveness research data collection on practices, adequately address data privacy and security issues, resolve or lessen the impact of data-linking issues and build research capacity for other investigators and clinicians.
Bonnie O’Day*,1, Tessa Kieffer1,2, Sarah Forrestal1 & Dominick Esposito1 Mathematica Policy Research, 1100 First Street NE, Suite 1200, Washington, DC 20002, USA 2 American Hospital Association, 155 N Wacker Drive, Chicago, IL 60606, USA *Author for correspondence: Tel.: +1 202 264 3455 Fax: +1 202 863 1763 [email protected] 1
Keywords: American Recovery and Reinvestment Act • ARRA • CER • comparative effectiveness research • data infrastructure
The American Recovery and Reinvestment Act (ARRA) of 2009 directed approximately US$300 million to data infrastructure (DI) projects that would develop new or enhance existing data resources for comparative effectiveness research (CER). As the second largest area of investment in the ARRA CER portfolio, DI investments accounted for 97 projects and 28% of the US$1.1 billion in funding. These projects created or enhanced DI that could be used for future research and were also engaged in evidence generation or synthesis, improving the evidence base for decision makers. This article describes these investments and identifies facilitators and barriers to implementation, lessons learned for future DI investments and a discussion of opportunities for future investments. The ARRA CER DI projects build upon already existing examples of CER data infrastructure, such as the HMO Research Network, a consortium of 19 large regional healthcare organizations. These organiza-
10.2217/CER.14.56 © 2014 Future Medicine Ltd
tions collaborate through research networks on cancer, cardiovascular disease, mental health and other diseases and are funded by Administration for Healthcare Research and Quality and the NIH. The Research Network also includes the US FDA’s Mini-Sentinel, a monitoring system for FDA-regulated medical products. Other networks include the Vaccine Safety Datalink, NIH’s Health System Collaboratory Distributed Research Network and other large-scale disease management registries. Most recently, the PatientCentered Outcomes Research Institute is supporting the development of PCORNet, the National Patient-Centered Research Network to create a large, highly representative network to conduct clinical outcomes research. These networks seek to enhance clinical research by bringing patient care and research together through a broad spectrum of population-based research studies. The development of DI for the conduct of CER is one of the core areas of focus of the Federal Coordinating Council for Com-
J. Comp. Eff. Res. (2014) 3(6), 591–600
part of
ISSN 2042-6305
591
Research Article O’Day, Kieffer, Forrestal & Esposito parative Effectiveness Research (FCCCER) strategic framework. At the time of the ARRA CER investment, data sources were often fragmented and limited. For example, sources may not have contained longitudinal data; were limited in scope; or failed to capture the robust, high quality, clinically rich data necessary to conduct CER. In its 2009 report, FCCCER asserted that an inventory of CER DI was necessary to track the capacity of existing data sources and provide a basis for its future development [1] . To address diverse CER priorities, FCCCER recommended that the inventory include observational databases, registries, claims and other administrative databases, pharmacy and laboratory data, adverse events registries, electronic health record (EHR) networks and other health information technology. The Institute of Medicine (IOM) also developed a report on topics for CER as part of the overall ARRA CER initiatives. The IOM Committee on CER Prioritization identified and ranked 100 priority topics based on input from a diverse group of stakeholders. To a large extent, the CER portfolio, including the DI projects, addressed the priorities of the FCCCER and the IOM [2] . In our analysis of ARRA CER DI projects, we examined the following policy questions: • What lessons about future DI investments emerged from the ARRA CER DI projects? • What factors did the funded investigators cite as having facilitated or inhibited the development of CER DI? The scope of this evaluation was a midstream assessment of the CER DI investment as a whole, rather than an evaluation of individual projects. As we conducted much of the evaluation concurrently with the implementation of ARRA-funded CER projects, of necessity we focused on preliminary lessons learned rather than long-term outcomes. Materials & methods To examine the policy questions and accomplish our research objectives, we conducted two phases of research. First, we used a short abstraction tool with all 97 projects to extract data from the proposals redacted by removing the applicant’s name and contact information (The list of projects is available from the authors upon request). We cataloged projects based on 12 attributes relevant to DI investments (e.g., funding amounts, data sources used and health conditions addressed). We then conducted an online survey of principal investigators of the 97 projects (72% of whom completed the survey) and conducted discussions with a purposive sample of 12 principal inves-
592
J. Comp. Eff. Res. (2014) 3(6)
tigators and project directors and 11 project officers. Second, we divided the universe of 97 DI projects into nine subgroups based on some of the above attributes (Table 1) . These groupings are based primarily on the federal Requests for Applications through which the projects were funded, though we also reviewed projects’ specific aims from the original proposals to help determine groupings. We focused the remainder of our data collection on seven of the nine subgroups, which made up nearly 80% of the ARRA CER DI investments. (The other two subgroups included broad activities specific to the time and circumstances of ARRA CER funding, or were examples of how to conduct CER with new or existing data infrastructure. These projects would be limited in their opportunity to detect lessons learned for more general DI investments.) We conducted semi-structured telephone discussions with principal investigators or project directors from 48 projects and 15 project officers within the seven subgroups. After constructing a database of project characteristics (e.g., project name, subgroup, methodological approach of the project and target audience), we identified emerging themes from the interview data, developed draft coding categories and applied them to the first set of discussions within each of the seven categories. The coding categories were the same across all seven categories, but more discrete subcodes differed, enabling comparisons across categories but still allowing for discrete themes in each category to emerge. After conducting interviews, we identified common themes, such as challenges and facilitators and notable differences by types of respondents; asked interviewees to review the findings for accuracy; and revised these based on their feedback. We received responses on the summaries from 25 investigators and seven project officers. Results Characteristics of DI investments A diverse group of DI investments was made that developed &/or enhanced a wide range of data resources for CER
The ARRA CER portfolio included DI projects that typically used existing data sources covering many patients of all ages from across the country (Table 2) . More than 70% of the projects proposed to bring together or enhance existing administrative and clinical datasets for CER, which reflected the need for the ARRA CER projects to get up and running quickly. About 10% of projects proposed to develop new databases for CER. For example, one project proposed to develop a registry to integrate the collection of treat-
future science group
American Recovery and Reinvestment Act investments in data infrastructure
Research Article
Table 1. Number of new principal investigator/project director and project officer discussions by project subgroup. Project subgroups for which we conducted discussions
Projects (n)
Discussions with PIs/PDs (n)
Discussions with POs (n)
Expansion of research capability to study CER
17
8
2
Enhancing capabilities of research networks to conduct CER
14
7
3
Enhancing state data for CER
10
7
2
Patient registry development or enhancement
11
7
2
Information technology for processing healthcare data
7
7
2
Scalable distributed research networks for CER
4
4
2
Enhancing existing federal data for CER
12
8
2
Project subgroups not included: – New or enhanced use of existing data for CER – Other DI investments
14 8
– –
– –
Total
97
48
15
Other DI investments include two projects that enhanced global infrastructure for conducting CER, two technical assistance contracts for DI, two projects that established data coordinating centers and two strategic planning contracts for CER conducted by federal agencies. CER: Comparative effectiveness research; DI: Data infrastructure; PD: Project director; PI: Principal investigator; PO: Federal project officer.
ment response data and biologic samples from patients with rheumatic arthritis, and another proposed to develop a pediatric rheumatology registry using data collected from existing registries, ongoing clinical trials and newly developed treatment protocols. About half of these investments proposed to bring together multiple data sources, and half also sought to leverage EHR data sources for CER. Nearly half of the projects proposed to include patient populations of all ages, and about 10% aimed to focus specifically on children. About two-thirds of the projects used datasets containing 100,000 or more covered lives (patients); more than a third had more than 1 million records. Those with fewer than 100,000 records included, for example, projects that involved prospective survey or biological sample collection, addressed specific disease populations (e.g., patients with pediatric rheumatic disease) or were engaged in proof-of-concept work. The projects covered much geographical ground as well, with 40% using datasets covering multiple states and nearly a quarter using nationally representative datasets such as Medicare or Medicaid Analytic eXtract data. DI investments addressed a number of medical condition & intervention categories
Of the 97 DI projects, 56 proposed to build infrastructure or address research questions specific to one or more medical condition category (Table 3) . The most common areas included cardiovascular diseases (13 projects) – hypertension, stroke and congestive
future science group
heart failure, among others – and various caiomncers (nine projects). For example, one project reported enhancing an existing cancer registry to support CER through data management resources by automating data abstraction and user interfaces, developing standards and methodologies to integrate data and conducting a pilot project to assess the ability of the DI to support clinical trials. About 57% of DI projects addressed one or more FCCCER priority populations and 18% addressed an IOM priority. Among the 39 projects that addressed a healthcare intervention, the most common were pharmaceutical treatments (13 projects) and the management of chronic conditions such as diabetes or chronic pain (nine projects). For example, one project developed both a retrospective and a prospective chronic pain data registry to identify specific pain management interventions that are most effective for specific patient types with chronic pain. Projects addressed DI needs beyond new or expanded data resources
Beyond linking data sources, the majority of DI projects created data solutions and methodologies that brought together data (62 projects) or improved or enhanced users’ access to data (61 projects; Table 4 ). Most of the DI projects (78) involved establishing linkages between various data sources relevant to CER. One project reported enhancing the Centers for Medicare & Medicaid Services’ Chronic Conditions Warehouse by increasing the size of the Medi-
www.futuremedicine.com
593
Research Article O’Day, Kieffer, Forrestal & Esposito care population in the database and adding 10 years of dual-eligible (Medicare and Medicaid) beneficiaries to it. A third of projects attempted to establish links between clinical datasets and administrative data sources, which included commercial health plan claims; Medicare or Medicaid claims; or other relevant administrative data (such as billing data, vital records, economic data and demographic data). Many of these investments (43) also addressed the dissemination of data to users through the infrastructure created. Reported progress toward goals
Most investigators who were leading ARRA CER DI investments reported in the final evaluation phase that their projects had met or were on their way to meeting their goals. For example, staff reported that projects had automated data collection and aggregation of data from multiple data sources, such as clinical data
housed in EHRs, health system utilization or claims data, biologic samples and patient-reported data to conduct CER. Investigators reported developing databases and datasets (including public use files) that incorporated data from various sources and were large enough that projects could address questions specific to IOM priority populations. Projects reported successfully integrating informatics tools and algorithms into EHRs for clinical use and for conducting CER, such as by demonstrating the effectiveness of search algorithms in identifying patients by condition and improving researchers’ and clinicians’ access to data through the development of data extraction systems, direct-query methods and end-user interfaces. Projects expanded existing research networks (e.g., of primary care practices) to include new practices or health centers to conduct prospective CER. This included adding new sources of data as well as expanding existing DI. Investigators linked disparate
Table 2. Project design characteristics for projects in the American Recovery and Reinvestment Act comparative effectiveness research portfolio with primary comparative effectiveness research data infrastructure elements. Project design characteristics
Projects (n)
All projects (%)
Type of DI being developed or enhanced: – Brought existing data sources together for CER – Developed new databases for CER – Enhanced existing databases for CER
60 10 9
61.9 10.3 9.3
Data sources used: – Medicare and/or Medicaid – Private administrative databases – Multiple administrative and/or clinical data – EHR data – Single source administrative data – VA, SAMHSA – Clinical trial data – Enhances or augments data for randomized controlled trials
26 16 46 46 5 4 2 20
26.8 16.5 47.4 47.4 5.2 4.1 2.1 20.6
Populations covered: – Children only – Adults only – All populations
9 36 47
9.3 37.1 48.5
Lives (n): – ≤100,000 – >100,000, ≤1 million – >1 million, ≤5 million – >5 million
35 19 13 21
36.1 19.6 13.4 21.6
Geographic spread: – One locale or state – Multiple states – National
23 39 23
23.7 40.2 23.7
ARRA: American Recovery and Reinvestment Act of 2009; CER: Comparative effectiveness research; DI: Data infrastructure; SAMHSA: Substance Abuse and Mental Health Services Administration; VA: Department of Veterans Affairs. Data taken from redacted grantee and contractor proposals provided by the Office of the Secretary, the Agency for Healthcare Research and Quality and the NIH.
594
J. Comp. Eff. Res. (2014) 3(6)
future science group
American Recovery and Reinvestment Act investments in data infrastructure
Research Article
Table 3. Project design characteristics for projects in the American Recovery and Reinvestment Act comparative effectiveness research portfolio with primary comparative effectiveness research data infrastructure elements. Project design characteristics
Projects (n)
All projects (%)
Projects that addressed one or more medical conditions : – Cardiovascular diseases – Various cancers – Multiple chronic conditions – Diabetes – Kidney disease – Rheumatic disease – Obesity
56 15 9 8 7 5 3 2
57.7 15.5 9.3 8.2 7.2 5.2 3.1 2.1
Projects that addressed one or more interventions: – Pharmaceutical treatments – Strategies to manage chronic conditions – Prevention – Procedures and surgery – Diagnostic testing – Delivery system interventions – Medical devices
39 13 9 5 5 4 3 1
40.2 13.4 9.3 5.2 5.2 4.1 3.1 1.0
Projects (n)
97
100.0
†
15 projects addressed one condition. ARRA: American Recovery and Reinvestment Act of 2009; CER: Comparative effectiveness research; DI: Data infrastructure. Data taken from redacted grantee and contractor proposals provided by the Office of the Secretary, the Agency for Healthcare Research and Quality and the NIH. †
data elements across existing databases and/or medical records systems across hospitals, practices, community-based clinics, integrated delivery systems or government datasets. Multiple projects completed pilot tests to demonstrate the utility of the DI and conduct CER on highpriority topics. For example, one project reported linking claims, point-of-care, patient-reported outcomes, enrollment and utilization data from 11 partner organizations and conducting pilot studies. Another pilot study reported comparing medication treatment choices and patterns for children with attention deficit hyperactivity disorder, and another examined pre- and post-operative bariatric surgical care management and weight loss outcomes among adults with obesity. Facilitators of project success
Investigators reported several facilitators as beneficial to project success, including an existing infrastructure to build upon, a project team with relevant expertise and strong partner collaborations. These attributes are particularly important for CER DI investments because many integrated complex data across multiple organizations, using a large team representing multiple interests and fields of expertise. Moreover, all these facilitators enabled researchers to obtain data used for CER more quickly, which pro-
future science group
vided the opportunity for CER evidence to be disseminated to users (such as policymakers, providers and patients) more rapidly. Building upon an existing DI-enabled project to move ahead more quickly
According to investigator reports, many projects built upon existing infrastructure with a foundation of informatics, network architecture, clinical research and other fields of expertise. A review of applications revealed that 69 projects brought together existing data, developed and enhanced existing databases, or both. Investigators told us that they used the lessons learned from building previous DI (such as data warehouses and registries) to move ARRA CER projects forward quickly, enabling research teams to leverage existing resources to produce datasets for CER more efficiently and generate CER evidence more quickly. For example, one investigator credited previous time spent developing a standalone data warehouse and establishing a robust infrastructure of hardware and software as facilitators of his project’s success. Another investigator said he used the lessons learned from developing a previous retrospective registry to build a prospective registry. Several investigators told us they had existing practice-based research networks, clinical networks and DI in place to support clinical research and quality improve-
www.futuremedicine.com
595
Research Article O’Day, Kieffer, Forrestal & Esposito ment. Investigators reported that the ARRA CER investment enabled them to enhance these resources and broaden their scope to conduct CER. Assembling a project team with expertise & experience appropriate for the project facilitated shorter project ramp-up time
Teams with multiple types of expertise, such as clinical, informatics, programming and CER, enabled projects to address complex problems from different perspectives. Investigators said that, in particular, teams with members who had experience and skills relevant to conducting CER with data that link EHR and administrative data often found it easier to remain on schedule to meet their goals, a concern for many ARRA CER projects given the short award period (no more than 3 years). For example, principal investigators described how they were able to leverage their prior experience and existing team in developing data standardization and integration protocols when linking EHR and administrative data. They said that experience with data sources facilitated shorter ramp-up times and reduced the need for troubleshooting, which enabled researchers to use the data more quickly. Experienced teams said they had baseline methodology already in place at the start of the projects. In one case, an investigator used his previously developed algorithm templates to link maternal and infant hospitalization and other datasets.
Strong collaboration with partners helped to build trust among the team
Project staff that had existing relationships with provider networks, clinical sites, hospitals or government agencies said it was often somewhat easier to build the necessary trust to share data, establish shared governance policies, agree upon processes and procure data sources than it would have been absent in such relationships. Investigators said that they had to demonstrate a return on investment to project partners, such as clinicians and practice administrators, to obtain their trust to participate and to establish a transparent data governance structure to share their patient data. If partners did not perceive that they would gain from the database, or from the resulting research, they viewed project requirements as burdensome and did not meet the requirements in a timely manner. Barriers or challenges to project success
Investigators mentioned lacking an existing project infrastructure to build upon, experienced team members, and partner buy-in as challenges to their project’s success. In addition, they described challenges in linking data from different systems, data completeness and consistency problems and regulatory issues. Although many challenges were anticipated, investigators commonly underestimated the time it would take to resolve them. For example, some teams underestimated how long it would take to clean, format and link
Table 4. Project design characteristics for projects in the American Recovery and Reinvestment Act comparative effectiveness research portfolio with primary comparative effectiveness research data infrastructure elements. Characteristics of DI project
Projects (n)
All projects (%)
Creating data solutions and methodologies
62
63.9
Improving or enhancing access to data
61
62.9
Improving dissemination of data to users
43
44.3
Creating analytic methods and informatics tools
22
22.7
Addressing health data security issues
18
18.6
Identifying or creating data standards
13
13.4
Establishing rules for the governance of data
12
12.4
Linkages between data sources: – Clinical and administrative data – Medicare or Medicaid and/or other sources – Various administrative data sources – Various clinical sources – Clinical trials data
78 32 21 11 9 4
80.4 33.0 21.6 11.3 9.3 4.1
Number of projects
97
100.0
ARRA: American Recovery and Reinvestment Act of 2009; CER: Comparative effectiveness research; DI: Data infrastructure. Data taken from redacted grantee and contractor proposals provided by the Office of the Secretary, the Agency for Healthcare Research and Quality and the NIH.
596
J. Comp. Eff. Res. (2014) 3(6)
future science group
American Recovery and Reinvestment Act investments in data infrastructure
nonstandardized EHR data across varied sources necessary to obtain rich, detailed data to conduct CER. Investigators identified the following challenges. Linking data from different EHRs or data systems was complex
Investigators reported that collecting and linking patient data from health systems that used different EHRs and other databases to manage patient data was a challenging and complex task. The process required extensive and iterative testing to ensure, for example, that all data were transformed into the appropriate format (including, for instance, reconciliation of inconsistent terms, variables, values and time periods) and that the linked dataset included all the elements necessary for the intended research. Investigators said they found that practices often had different technological capabilities, which increased the difficulty of linking EHR data. They told us that even within the same practice or hospital, data were often housed in disparate clinical data repositories (e.g., laboratory reports, pathology results, medical records, imaging results and pharmacy data) that were developed separately to fit highly specific needs; as a result, integrating multiple repositories was difficult. Data completeness & consistency varied
Project staff reported that data completeness and consistency varied by source, a situation that often led to project delays or challenges. Investigators told us that transforming patient and administrative data into useful information for CER was hampered by incomplete, missing or inaccurate data. Across projects, we heard that some datasets had incorrect coding or inaccurate measures for some elements. One investigator reported that project staff had to conduct a supplemental survey because a dataset did not differentiate between two existing codes for a treatment the project team wanted to study. Regulatory & security issues caused delays
Many investigators reported experiencing delays due to regulatory processes that they had planned for but that took longer than anticipated. Several projects reported delays due to the time required to establish data use agreements and meet Health Insurance Portability and Accountability Act (HIPAA) requirements. Some investigators said that their projects suffered unanticipated delays because they needed to obtain approval from multiple institutional review boards (IRBs) at various organizations before a study could proceed. For example, one project worked with statewide hospital partners to integrate existing patient-level datasets and reported having to obtain
future science group
Research Article
IRB approval from each of more than 15 hospital sites. Projects reported that they had to address privacy and data security issues, such as collecting data on laptops, defining and protecting what constitutes personally identifiable information in EHRs and undergoing extensive security reviews. For example, one investigator described an institution-initiated HIPAA audit of his data warehouse project that required significant time and project resources to complete. In some cases, projects reported achieving efficiencies by using common contracting language, common multisite data use agreement language and shared IRB applications. Project sustainability
Investigators pursued creative strategies to sustain over time the DI developed under their ARRA-funded projects, including seeking additional funding sources and strategic partnerships (such as with industry or philanthropic organizations). Some projects reported developing business models or exploring the use of network participation fees to sustain the data resources. Other investigators told us that to continue, they relied upon institutional support and in-kind donations of faculty and organizations associated with the project. They also reported the importance of decreasing burden on physician practices and other institutional partners as a way of increasing institutional support for sustaining the projects. For example, some project investigators suggested that having a dedicated staff member to work with practices facilitated building trust with sites. These investigators reported that it was beneficial to have, at each site, a dedicated coordinator, supported by project funds, who maintained responsibility for day-to-day study operations such as submitting data and handling administrative issues. Investigators secured buy-in from practices by providing valuable resources such as a practice data summary with analysis and benchmarking statistics. Some projects reported that hospitals were more likely to participate and support project sustainability when they were convinced that they would receive a benefit, such as reports to help them identify ways to improve quality. Discussion DI projects assembled or enhanced existing administrative and clinical datasets, developed new databases, brought together multiple data sources and leveraged EHR data sources for CER. A large number of DI investments built infrastructure to address research questions specific to clinical conditions to increase capacity to conduct clinically relevant CER. Most of the DI projects established linkages between data, such as between clinical datasets and administrative data sources. Although claims data alone may be used
www.futuremedicine.com
597
Research Article O’Day, Kieffer, Forrestal & Esposito to identify a defined population, they are often not detailed enough for CER because claims alone may not make clear the specific types of procedures or treatments that were provided or which specific subgroups of patients were treated. Conversely, EHR data without claims data are of little value; EHR linked with claims data can prove useful for research. Some projects developed CER data infrastructure by expanding existing research networks (e.g., of primary care practices) to include new community-based or safety net practices with which to conduct prospective CER relevant to a broader range of priority populations. Although the primary focus of DI projects was to build data infrastructure to enable CER, there were also benefits from these DI investments to other areas of CER investment, such as research, methods development and the dissemination and translation of CER findings. Most of these DI projects included an original research component that tested the potential impact of these investments in their ability to achieve intervention or clinical goals supported by the new data infrastructure. Some of these projects also had a secondary focus on the development of new methods in conjunction with the creation of new or enhanced data sources for CER analyses. In addition, the development of data infrastructure can also enhance dissemination and translation of CER; almost half of these investments included a dissemination component. Based on our findings, the potential for future CER DI projects to be successful and sustainable may be dependent upon several factors: Reducing burden on practices that collect data for CER
To reduce the burden on practices that collect data for CER, investigators should consider developing data infrastructure that is readily understandable to clinicians, meets their needs, fits into their clinical workflow and can be collected as part of routine clinical care and will, in the long run, increase the quality of the medical care they provide; and providing support as needed. Establishing strong relationships with practices facilitated access to clinically rich data and helped ease the burden of collection for research purposes. Addressing data privacy & security issues
Investigators were confronted with unanticipated barriers in working with de-identified data and limited datasets, which affected project timelines as well as the conduct of CER. IRBs often insist that data be de-identified to protect patient confidentiality, but researchers found it very difficult if not impossible to conduct research using these data. Many challenges were related to linking patient data sources and studying subpopulations.
598
J. Comp. Eff. Res. (2014) 3(6)
To study subpopulations, investigators need access to rich, detailed patient data for subgroups of populations. Often, datasets were de-identified in ways that constrained researchers’ ability to conduct comprehensive CER; for example, administrative claims data often did not include dates of service, race and ethnicity identifiers and location data other than the first three digits of zip codes that were useless for determining region or location based on clinical practice patterns. Patient characteristics, socioeconomic attributes and geographic location data points could enable subpopulation comparisons and allow identification of patients for clinical trials, registries and patient-level studies. Regulatory processes, such as IRB protocols and data use agreements that allow researchers to retain identifiers, would facilitate project sustainability. Data owners are often concerned about the security of shared data. A critical first step toward addressing such concerns involves building trust among potential data partners at the outset of a project and establishing data governance strategies that are sufficient to preserve patient privacy and at the same time allow data to be shared between clinicians and researchers. For example, one project established, for all data partners, a master consortium agreement that stipulated the uses and rules for data. Other projects established distributed research networks as a solution to this problem. Addressing data standardization issues
Though investigators anticipated challenges with linking data sources, the time required to resolve the issues was more extensive than planned. The primary purpose of collecting data for most clinics is to ensure that services provided are billed properly. Uses of these data for research are considered secondarily, when considered at all. Claims data, when combined with clinical data, could be a powerful research dataset, if data linkage issues can be resolved. Data linkage issues mentioned by investigators included datasets with different identifier numbers across clinics in the same healthcare system, de-identified data, missing data, fragmented data and disparate data sources. Resolving or lessening the impact of data linkage issues would support project sustainability because it would allow investigators to work with datasets more quickly, to produce CER evidence more quickly and to mitigate potential delays in project timelines. Increasing the amount of data, by increasing the number of patients, the time period included or the number of data domains and linked datasets would provide investigators who conduct CER with larger subgroups of populations that have less common medical conditions or undergo less common treatments, enabling the generation of CER evidence for those subgroups. Multiple data sources
future science group
American Recovery and Reinvestment Act investments in data infrastructure
must include medical charts (obtaining EHR data, abstracting data from paper records or pulling from an existing registry); laboratory reports; pharmaceutical records; and patient-reported data. Building research capacity for other investigators &/or clinicians
Many projects sought to build research capacity within their organization or institution, as well as in practices and health centers, in an effort to make the infrastructure sustainable. Supporting training and capacity building means that practices, health centers and new investigators will have the skills to secure their own grants to take advantage of the infrastructure these projects developed. For example, to sustain the work beyond the ARRA funding period, several projects in the Enhancing State Data subgroup trained staff at various organizations to collect data on race and ethnicity. Research Networks projects provided training to community health center clinicians and staff who went on to seek their own funding for local-level research projects. Several projects included researchers who conducted CER with a newly developed data resource and trained other researchers in the necessary skills to analyze large datasets. Conclusion & future perspective Through our study of Recovery Act–funded projects, we identified several lessons learned that may inform considerations for future investments in CER DI.
Research Article
First, projects with established DI and technology in place to enable them to efficiently continue their data linkages into the future might provide more rapid returns on CER. Second, projects that developed data resources that can be used for CER and other complementary purposes enhanced buy-in and cooperation by providers. For example, some CER DI could be leveraged to provide practices with more data for quality improvement, meaningful use and pay-for-performance initiatives. Such uses have the added advantage of demonstrating the value of CER DI to real-world stakeholders who will be asked to buy-in by sharing their data. Third, because a priority for CER is clarifying the effectiveness of different treatment strategies in specific subgroups of patients, projects that have already successfully linked large data sources to support CER on large samples of often underrepresented subgroups should offer promising areas for continued investment. Projects that combine rich sources of clinical and administrative data across many practices also offer promising resources for future high-impact CER. For example, registries represent a potentially valuable investment because they aggregate data on a wide variety of patients with high-priority conditions from across multiple relevant clinical settings. Financial & competing interests disclosure The project on which this research was based was funded by the US Department of Health and Human Services Assistant
Executive summary • The American Recovery and Reinvestment Act comparative effectiveness research (CER) data infrastructure projects developed and/or enhanced a wide range of data resources for CER, some of which were already being incorporated into grant proposals and future projects at the time of the interviews. • More than half of the projects proposed to build infrastructure or address research questions specific to one or more medical conditions, including cardiovascular diseases and cancer. • At the time of the study, projects were making strong progress toward meeting goals, which included automating data collection and aggregating data from multiple data sources, developing databases and datasets that incorporated data from various sources and were large enough that projects could address questions specific to the Institute of Medicine priority populations, successfully integrating informatics tools and algorithms into electronic health records for clinical use and for conducting CER, expanding existing research networks and conducting pilot tests. • Project investigators reported several facilitators of project success, including an existing infrastructure to build upon, a project team with relevant expertise and strong partner collaborations. • Investigators mentioned lacking an existing project infrastructure to build upon, experienced team members or partner buy-in as challenges to their project’s success. In addition, they described challenges in linking data from different systems, data completeness and consistency problems and regulatory issues. • To sustain their data infrastructure projects, investigators should attempt to reduce burden on practices that collect data for CER, address data security and standardization issues and build research capacity for other investigators and/or clinicians. • Projects with mature data infrastructure and technology in place to enable them to efficiently continue their data linkages into the future might provide more rapid returns on CER. • Prioritizing support for data resources that can be used for CER and other complementary purposes may enhance the potential return on investment to and buy-in by providers.
future science group
www.futuremedicine.com
599
Research Article O’Day, Kieffer, Forrestal & Esposito Secretary for Planning and Evaluation. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
Ethical conduct of research
References
2
1
600
Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. National Academies Press, Washington, DC, USA (2009).
J. Comp. Eff. Res. (2014) 3(6)
The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. US Department of Health and Human Services, Washington, DC, USA (2009).
future science group
American Recovery and Reinvestment Act investments in data infrastructure
Aim: This article describes American Reinvestment and Recovery Act comparative effectiveness research data infrastructure (DI) investments and identifies facilitators and barriers to implementation. Materials & methods: We reviewed original project proposals, conducted an investigator survey and interviewed project officers and principal investigators. Results: DI projects assembled or enhanced existing clinical datasets, established linkages between public and private data sources and built infrastructure. Facilitators included building on existing relationships across organizations and making collection as seamless as possible for clinicians. Conclusion: To sustain DI, investigators should reduce the burden of comparative effectiveness research data collection on practices, adequately address data privacy and security issues, resolve or lessen the impact of data-linking issues and build research capacity for other investigators and clinicians.
Bonnie O’Day*,1, Tessa Kieffer1,2, Sarah Forrestal1 & Dominick Esposito1 Mathematica Policy Research, 1100 First Street NE, Suite 1200, Washington, DC 20002, USA 2 American Hospital Association, 155 N Wacker Drive, Chicago, IL 60606, USA *Author for correspondence: Tel.: +1 202 264 3455 Fax: +1 202 863 1763 [email protected] 1
Keywords: American Recovery and Reinvestment Act • ARRA • CER • comparative effectiveness research • data infrastructure
The American Recovery and Reinvestment Act (ARRA) of 2009 directed approximately US$300 million to data infrastructure (DI) projects that would develop new or enhance existing data resources for comparative effectiveness research (CER). As the second largest area of investment in the ARRA CER portfolio, DI investments accounted for 97 projects and 28% of the US$1.1 billion in funding. These projects created or enhanced DI that could be used for future research and were also engaged in evidence generation or synthesis, improving the evidence base for decision makers. This article describes these investments and identifies facilitators and barriers to implementation, lessons learned for future DI investments and a discussion of opportunities for future investments. The ARRA CER DI projects build upon already existing examples of CER data infrastructure, such as the HMO Research Network, a consortium of 19 large regional healthcare organizations. These organiza-
10.2217/CER.14.56 © 2014 Future Medicine Ltd
tions collaborate through research networks on cancer, cardiovascular disease, mental health and other diseases and are funded by Administration for Healthcare Research and Quality and the NIH. The Research Network also includes the US FDA’s Mini-Sentinel, a monitoring system for FDA-regulated medical products. Other networks include the Vaccine Safety Datalink, NIH’s Health System Collaboratory Distributed Research Network and other large-scale disease management registries. Most recently, the PatientCentered Outcomes Research Institute is supporting the development of PCORNet, the National Patient-Centered Research Network to create a large, highly representative network to conduct clinical outcomes research. These networks seek to enhance clinical research by bringing patient care and research together through a broad spectrum of population-based research studies. The development of DI for the conduct of CER is one of the core areas of focus of the Federal Coordinating Council for Com-
J. Comp. Eff. Res. (2014) 3(6), 591–600
part of
ISSN 2042-6305
591
Research Article O’Day, Kieffer, Forrestal & Esposito parative Effectiveness Research (FCCCER) strategic framework. At the time of the ARRA CER investment, data sources were often fragmented and limited. For example, sources may not have contained longitudinal data; were limited in scope; or failed to capture the robust, high quality, clinically rich data necessary to conduct CER. In its 2009 report, FCCCER asserted that an inventory of CER DI was necessary to track the capacity of existing data sources and provide a basis for its future development [1] . To address diverse CER priorities, FCCCER recommended that the inventory include observational databases, registries, claims and other administrative databases, pharmacy and laboratory data, adverse events registries, electronic health record (EHR) networks and other health information technology. The Institute of Medicine (IOM) also developed a report on topics for CER as part of the overall ARRA CER initiatives. The IOM Committee on CER Prioritization identified and ranked 100 priority topics based on input from a diverse group of stakeholders. To a large extent, the CER portfolio, including the DI projects, addressed the priorities of the FCCCER and the IOM [2] . In our analysis of ARRA CER DI projects, we examined the following policy questions: • What lessons about future DI investments emerged from the ARRA CER DI projects? • What factors did the funded investigators cite as having facilitated or inhibited the development of CER DI? The scope of this evaluation was a midstream assessment of the CER DI investment as a whole, rather than an evaluation of individual projects. As we conducted much of the evaluation concurrently with the implementation of ARRA-funded CER projects, of necessity we focused on preliminary lessons learned rather than long-term outcomes. Materials & methods To examine the policy questions and accomplish our research objectives, we conducted two phases of research. First, we used a short abstraction tool with all 97 projects to extract data from the proposals redacted by removing the applicant’s name and contact information (The list of projects is available from the authors upon request). We cataloged projects based on 12 attributes relevant to DI investments (e.g., funding amounts, data sources used and health conditions addressed). We then conducted an online survey of principal investigators of the 97 projects (72% of whom completed the survey) and conducted discussions with a purposive sample of 12 principal inves-
592
J. Comp. Eff. Res. (2014) 3(6)
tigators and project directors and 11 project officers. Second, we divided the universe of 97 DI projects into nine subgroups based on some of the above attributes (Table 1) . These groupings are based primarily on the federal Requests for Applications through which the projects were funded, though we also reviewed projects’ specific aims from the original proposals to help determine groupings. We focused the remainder of our data collection on seven of the nine subgroups, which made up nearly 80% of the ARRA CER DI investments. (The other two subgroups included broad activities specific to the time and circumstances of ARRA CER funding, or were examples of how to conduct CER with new or existing data infrastructure. These projects would be limited in their opportunity to detect lessons learned for more general DI investments.) We conducted semi-structured telephone discussions with principal investigators or project directors from 48 projects and 15 project officers within the seven subgroups. After constructing a database of project characteristics (e.g., project name, subgroup, methodological approach of the project and target audience), we identified emerging themes from the interview data, developed draft coding categories and applied them to the first set of discussions within each of the seven categories. The coding categories were the same across all seven categories, but more discrete subcodes differed, enabling comparisons across categories but still allowing for discrete themes in each category to emerge. After conducting interviews, we identified common themes, such as challenges and facilitators and notable differences by types of respondents; asked interviewees to review the findings for accuracy; and revised these based on their feedback. We received responses on the summaries from 25 investigators and seven project officers. Results Characteristics of DI investments A diverse group of DI investments was made that developed &/or enhanced a wide range of data resources for CER
The ARRA CER portfolio included DI projects that typically used existing data sources covering many patients of all ages from across the country (Table 2) . More than 70% of the projects proposed to bring together or enhance existing administrative and clinical datasets for CER, which reflected the need for the ARRA CER projects to get up and running quickly. About 10% of projects proposed to develop new databases for CER. For example, one project proposed to develop a registry to integrate the collection of treat-
future science group
American Recovery and Reinvestment Act investments in data infrastructure
Research Article
Table 1. Number of new principal investigator/project director and project officer discussions by project subgroup. Project subgroups for which we conducted discussions
Projects (n)
Discussions with PIs/PDs (n)
Discussions with POs (n)
Expansion of research capability to study CER
17
8
2
Enhancing capabilities of research networks to conduct CER
14
7
3
Enhancing state data for CER
10
7
2
Patient registry development or enhancement
11
7
2
Information technology for processing healthcare data
7
7
2
Scalable distributed research networks for CER
4
4
2
Enhancing existing federal data for CER
12
8
2
Project subgroups not included: – New or enhanced use of existing data for CER – Other DI investments
14 8
– –
– –
Total
97
48
15
Other DI investments include two projects that enhanced global infrastructure for conducting CER, two technical assistance contracts for DI, two projects that established data coordinating centers and two strategic planning contracts for CER conducted by federal agencies. CER: Comparative effectiveness research; DI: Data infrastructure; PD: Project director; PI: Principal investigator; PO: Federal project officer.
ment response data and biologic samples from patients with rheumatic arthritis, and another proposed to develop a pediatric rheumatology registry using data collected from existing registries, ongoing clinical trials and newly developed treatment protocols. About half of these investments proposed to bring together multiple data sources, and half also sought to leverage EHR data sources for CER. Nearly half of the projects proposed to include patient populations of all ages, and about 10% aimed to focus specifically on children. About two-thirds of the projects used datasets containing 100,000 or more covered lives (patients); more than a third had more than 1 million records. Those with fewer than 100,000 records included, for example, projects that involved prospective survey or biological sample collection, addressed specific disease populations (e.g., patients with pediatric rheumatic disease) or were engaged in proof-of-concept work. The projects covered much geographical ground as well, with 40% using datasets covering multiple states and nearly a quarter using nationally representative datasets such as Medicare or Medicaid Analytic eXtract data. DI investments addressed a number of medical condition & intervention categories
Of the 97 DI projects, 56 proposed to build infrastructure or address research questions specific to one or more medical condition category (Table 3) . The most common areas included cardiovascular diseases (13 projects) – hypertension, stroke and congestive
future science group
heart failure, among others – and various caiomncers (nine projects). For example, one project reported enhancing an existing cancer registry to support CER through data management resources by automating data abstraction and user interfaces, developing standards and methodologies to integrate data and conducting a pilot project to assess the ability of the DI to support clinical trials. About 57% of DI projects addressed one or more FCCCER priority populations and 18% addressed an IOM priority. Among the 39 projects that addressed a healthcare intervention, the most common were pharmaceutical treatments (13 projects) and the management of chronic conditions such as diabetes or chronic pain (nine projects). For example, one project developed both a retrospective and a prospective chronic pain data registry to identify specific pain management interventions that are most effective for specific patient types with chronic pain. Projects addressed DI needs beyond new or expanded data resources
Beyond linking data sources, the majority of DI projects created data solutions and methodologies that brought together data (62 projects) or improved or enhanced users’ access to data (61 projects; Table 4 ). Most of the DI projects (78) involved establishing linkages between various data sources relevant to CER. One project reported enhancing the Centers for Medicare & Medicaid Services’ Chronic Conditions Warehouse by increasing the size of the Medi-
www.futuremedicine.com
593
Research Article O’Day, Kieffer, Forrestal & Esposito care population in the database and adding 10 years of dual-eligible (Medicare and Medicaid) beneficiaries to it. A third of projects attempted to establish links between clinical datasets and administrative data sources, which included commercial health plan claims; Medicare or Medicaid claims; or other relevant administrative data (such as billing data, vital records, economic data and demographic data). Many of these investments (43) also addressed the dissemination of data to users through the infrastructure created. Reported progress toward goals
Most investigators who were leading ARRA CER DI investments reported in the final evaluation phase that their projects had met or were on their way to meeting their goals. For example, staff reported that projects had automated data collection and aggregation of data from multiple data sources, such as clinical data
housed in EHRs, health system utilization or claims data, biologic samples and patient-reported data to conduct CER. Investigators reported developing databases and datasets (including public use files) that incorporated data from various sources and were large enough that projects could address questions specific to IOM priority populations. Projects reported successfully integrating informatics tools and algorithms into EHRs for clinical use and for conducting CER, such as by demonstrating the effectiveness of search algorithms in identifying patients by condition and improving researchers’ and clinicians’ access to data through the development of data extraction systems, direct-query methods and end-user interfaces. Projects expanded existing research networks (e.g., of primary care practices) to include new practices or health centers to conduct prospective CER. This included adding new sources of data as well as expanding existing DI. Investigators linked disparate
Table 2. Project design characteristics for projects in the American Recovery and Reinvestment Act comparative effectiveness research portfolio with primary comparative effectiveness research data infrastructure elements. Project design characteristics
Projects (n)
All projects (%)
Type of DI being developed or enhanced: – Brought existing data sources together for CER – Developed new databases for CER – Enhanced existing databases for CER
60 10 9
61.9 10.3 9.3
Data sources used: – Medicare and/or Medicaid – Private administrative databases – Multiple administrative and/or clinical data – EHR data – Single source administrative data – VA, SAMHSA – Clinical trial data – Enhances or augments data for randomized controlled trials
26 16 46 46 5 4 2 20
26.8 16.5 47.4 47.4 5.2 4.1 2.1 20.6
Populations covered: – Children only – Adults only – All populations
9 36 47
9.3 37.1 48.5
Lives (n): – ≤100,000 – >100,000, ≤1 million – >1 million, ≤5 million – >5 million
35 19 13 21
36.1 19.6 13.4 21.6
Geographic spread: – One locale or state – Multiple states – National
23 39 23
23.7 40.2 23.7
ARRA: American Recovery and Reinvestment Act of 2009; CER: Comparative effectiveness research; DI: Data infrastructure; SAMHSA: Substance Abuse and Mental Health Services Administration; VA: Department of Veterans Affairs. Data taken from redacted grantee and contractor proposals provided by the Office of the Secretary, the Agency for Healthcare Research and Quality and the NIH.
594
J. Comp. Eff. Res. (2014) 3(6)
future science group
American Recovery and Reinvestment Act investments in data infrastructure
Research Article
Table 3. Project design characteristics for projects in the American Recovery and Reinvestment Act comparative effectiveness research portfolio with primary comparative effectiveness research data infrastructure elements. Project design characteristics
Projects (n)
All projects (%)
Projects that addressed one or more medical conditions : – Cardiovascular diseases – Various cancers – Multiple chronic conditions – Diabetes – Kidney disease – Rheumatic disease – Obesity
56 15 9 8 7 5 3 2
57.7 15.5 9.3 8.2 7.2 5.2 3.1 2.1
Projects that addressed one or more interventions: – Pharmaceutical treatments – Strategies to manage chronic conditions – Prevention – Procedures and surgery – Diagnostic testing – Delivery system interventions – Medical devices
39 13 9 5 5 4 3 1
40.2 13.4 9.3 5.2 5.2 4.1 3.1 1.0
Projects (n)
97
100.0
†
15 projects addressed one condition. ARRA: American Recovery and Reinvestment Act of 2009; CER: Comparative effectiveness research; DI: Data infrastructure. Data taken from redacted grantee and contractor proposals provided by the Office of the Secretary, the Agency for Healthcare Research and Quality and the NIH. †
data elements across existing databases and/or medical records systems across hospitals, practices, community-based clinics, integrated delivery systems or government datasets. Multiple projects completed pilot tests to demonstrate the utility of the DI and conduct CER on highpriority topics. For example, one project reported linking claims, point-of-care, patient-reported outcomes, enrollment and utilization data from 11 partner organizations and conducting pilot studies. Another pilot study reported comparing medication treatment choices and patterns for children with attention deficit hyperactivity disorder, and another examined pre- and post-operative bariatric surgical care management and weight loss outcomes among adults with obesity. Facilitators of project success
Investigators reported several facilitators as beneficial to project success, including an existing infrastructure to build upon, a project team with relevant expertise and strong partner collaborations. These attributes are particularly important for CER DI investments because many integrated complex data across multiple organizations, using a large team representing multiple interests and fields of expertise. Moreover, all these facilitators enabled researchers to obtain data used for CER more quickly, which pro-
future science group
vided the opportunity for CER evidence to be disseminated to users (such as policymakers, providers and patients) more rapidly. Building upon an existing DI-enabled project to move ahead more quickly
According to investigator reports, many projects built upon existing infrastructure with a foundation of informatics, network architecture, clinical research and other fields of expertise. A review of applications revealed that 69 projects brought together existing data, developed and enhanced existing databases, or both. Investigators told us that they used the lessons learned from building previous DI (such as data warehouses and registries) to move ARRA CER projects forward quickly, enabling research teams to leverage existing resources to produce datasets for CER more efficiently and generate CER evidence more quickly. For example, one investigator credited previous time spent developing a standalone data warehouse and establishing a robust infrastructure of hardware and software as facilitators of his project’s success. Another investigator said he used the lessons learned from developing a previous retrospective registry to build a prospective registry. Several investigators told us they had existing practice-based research networks, clinical networks and DI in place to support clinical research and quality improve-
www.futuremedicine.com
595
Research Article O’Day, Kieffer, Forrestal & Esposito ment. Investigators reported that the ARRA CER investment enabled them to enhance these resources and broaden their scope to conduct CER. Assembling a project team with expertise & experience appropriate for the project facilitated shorter project ramp-up time
Teams with multiple types of expertise, such as clinical, informatics, programming and CER, enabled projects to address complex problems from different perspectives. Investigators said that, in particular, teams with members who had experience and skills relevant to conducting CER with data that link EHR and administrative data often found it easier to remain on schedule to meet their goals, a concern for many ARRA CER projects given the short award period (no more than 3 years). For example, principal investigators described how they were able to leverage their prior experience and existing team in developing data standardization and integration protocols when linking EHR and administrative data. They said that experience with data sources facilitated shorter ramp-up times and reduced the need for troubleshooting, which enabled researchers to use the data more quickly. Experienced teams said they had baseline methodology already in place at the start of the projects. In one case, an investigator used his previously developed algorithm templates to link maternal and infant hospitalization and other datasets.
Strong collaboration with partners helped to build trust among the team
Project staff that had existing relationships with provider networks, clinical sites, hospitals or government agencies said it was often somewhat easier to build the necessary trust to share data, establish shared governance policies, agree upon processes and procure data sources than it would have been absent in such relationships. Investigators said that they had to demonstrate a return on investment to project partners, such as clinicians and practice administrators, to obtain their trust to participate and to establish a transparent data governance structure to share their patient data. If partners did not perceive that they would gain from the database, or from the resulting research, they viewed project requirements as burdensome and did not meet the requirements in a timely manner. Barriers or challenges to project success
Investigators mentioned lacking an existing project infrastructure to build upon, experienced team members, and partner buy-in as challenges to their project’s success. In addition, they described challenges in linking data from different systems, data completeness and consistency problems and regulatory issues. Although many challenges were anticipated, investigators commonly underestimated the time it would take to resolve them. For example, some teams underestimated how long it would take to clean, format and link
Table 4. Project design characteristics for projects in the American Recovery and Reinvestment Act comparative effectiveness research portfolio with primary comparative effectiveness research data infrastructure elements. Characteristics of DI project
Projects (n)
All projects (%)
Creating data solutions and methodologies
62
63.9
Improving or enhancing access to data
61
62.9
Improving dissemination of data to users
43
44.3
Creating analytic methods and informatics tools
22
22.7
Addressing health data security issues
18
18.6
Identifying or creating data standards
13
13.4
Establishing rules for the governance of data
12
12.4
Linkages between data sources: – Clinical and administrative data – Medicare or Medicaid and/or other sources – Various administrative data sources – Various clinical sources – Clinical trials data
78 32 21 11 9 4
80.4 33.0 21.6 11.3 9.3 4.1
Number of projects
97
100.0
ARRA: American Recovery and Reinvestment Act of 2009; CER: Comparative effectiveness research; DI: Data infrastructure. Data taken from redacted grantee and contractor proposals provided by the Office of the Secretary, the Agency for Healthcare Research and Quality and the NIH.
596
J. Comp. Eff. Res. (2014) 3(6)
future science group
American Recovery and Reinvestment Act investments in data infrastructure
nonstandardized EHR data across varied sources necessary to obtain rich, detailed data to conduct CER. Investigators identified the following challenges. Linking data from different EHRs or data systems was complex
Investigators reported that collecting and linking patient data from health systems that used different EHRs and other databases to manage patient data was a challenging and complex task. The process required extensive and iterative testing to ensure, for example, that all data were transformed into the appropriate format (including, for instance, reconciliation of inconsistent terms, variables, values and time periods) and that the linked dataset included all the elements necessary for the intended research. Investigators said they found that practices often had different technological capabilities, which increased the difficulty of linking EHR data. They told us that even within the same practice or hospital, data were often housed in disparate clinical data repositories (e.g., laboratory reports, pathology results, medical records, imaging results and pharmacy data) that were developed separately to fit highly specific needs; as a result, integrating multiple repositories was difficult. Data completeness & consistency varied
Project staff reported that data completeness and consistency varied by source, a situation that often led to project delays or challenges. Investigators told us that transforming patient and administrative data into useful information for CER was hampered by incomplete, missing or inaccurate data. Across projects, we heard that some datasets had incorrect coding or inaccurate measures for some elements. One investigator reported that project staff had to conduct a supplemental survey because a dataset did not differentiate between two existing codes for a treatment the project team wanted to study. Regulatory & security issues caused delays
Many investigators reported experiencing delays due to regulatory processes that they had planned for but that took longer than anticipated. Several projects reported delays due to the time required to establish data use agreements and meet Health Insurance Portability and Accountability Act (HIPAA) requirements. Some investigators said that their projects suffered unanticipated delays because they needed to obtain approval from multiple institutional review boards (IRBs) at various organizations before a study could proceed. For example, one project worked with statewide hospital partners to integrate existing patient-level datasets and reported having to obtain
future science group
Research Article
IRB approval from each of more than 15 hospital sites. Projects reported that they had to address privacy and data security issues, such as collecting data on laptops, defining and protecting what constitutes personally identifiable information in EHRs and undergoing extensive security reviews. For example, one investigator described an institution-initiated HIPAA audit of his data warehouse project that required significant time and project resources to complete. In some cases, projects reported achieving efficiencies by using common contracting language, common multisite data use agreement language and shared IRB applications. Project sustainability
Investigators pursued creative strategies to sustain over time the DI developed under their ARRA-funded projects, including seeking additional funding sources and strategic partnerships (such as with industry or philanthropic organizations). Some projects reported developing business models or exploring the use of network participation fees to sustain the data resources. Other investigators told us that to continue, they relied upon institutional support and in-kind donations of faculty and organizations associated with the project. They also reported the importance of decreasing burden on physician practices and other institutional partners as a way of increasing institutional support for sustaining the projects. For example, some project investigators suggested that having a dedicated staff member to work with practices facilitated building trust with sites. These investigators reported that it was beneficial to have, at each site, a dedicated coordinator, supported by project funds, who maintained responsibility for day-to-day study operations such as submitting data and handling administrative issues. Investigators secured buy-in from practices by providing valuable resources such as a practice data summary with analysis and benchmarking statistics. Some projects reported that hospitals were more likely to participate and support project sustainability when they were convinced that they would receive a benefit, such as reports to help them identify ways to improve quality. Discussion DI projects assembled or enhanced existing administrative and clinical datasets, developed new databases, brought together multiple data sources and leveraged EHR data sources for CER. A large number of DI investments built infrastructure to address research questions specific to clinical conditions to increase capacity to conduct clinically relevant CER. Most of the DI projects established linkages between data, such as between clinical datasets and administrative data sources. Although claims data alone may be used
www.futuremedicine.com
597
Research Article O’Day, Kieffer, Forrestal & Esposito to identify a defined population, they are often not detailed enough for CER because claims alone may not make clear the specific types of procedures or treatments that were provided or which specific subgroups of patients were treated. Conversely, EHR data without claims data are of little value; EHR linked with claims data can prove useful for research. Some projects developed CER data infrastructure by expanding existing research networks (e.g., of primary care practices) to include new community-based or safety net practices with which to conduct prospective CER relevant to a broader range of priority populations. Although the primary focus of DI projects was to build data infrastructure to enable CER, there were also benefits from these DI investments to other areas of CER investment, such as research, methods development and the dissemination and translation of CER findings. Most of these DI projects included an original research component that tested the potential impact of these investments in their ability to achieve intervention or clinical goals supported by the new data infrastructure. Some of these projects also had a secondary focus on the development of new methods in conjunction with the creation of new or enhanced data sources for CER analyses. In addition, the development of data infrastructure can also enhance dissemination and translation of CER; almost half of these investments included a dissemination component. Based on our findings, the potential for future CER DI projects to be successful and sustainable may be dependent upon several factors: Reducing burden on practices that collect data for CER
To reduce the burden on practices that collect data for CER, investigators should consider developing data infrastructure that is readily understandable to clinicians, meets their needs, fits into their clinical workflow and can be collected as part of routine clinical care and will, in the long run, increase the quality of the medical care they provide; and providing support as needed. Establishing strong relationships with practices facilitated access to clinically rich data and helped ease the burden of collection for research purposes. Addressing data privacy & security issues
Investigators were confronted with unanticipated barriers in working with de-identified data and limited datasets, which affected project timelines as well as the conduct of CER. IRBs often insist that data be de-identified to protect patient confidentiality, but researchers found it very difficult if not impossible to conduct research using these data. Many challenges were related to linking patient data sources and studying subpopulations.
598
J. Comp. Eff. Res. (2014) 3(6)
To study subpopulations, investigators need access to rich, detailed patient data for subgroups of populations. Often, datasets were de-identified in ways that constrained researchers’ ability to conduct comprehensive CER; for example, administrative claims data often did not include dates of service, race and ethnicity identifiers and location data other than the first three digits of zip codes that were useless for determining region or location based on clinical practice patterns. Patient characteristics, socioeconomic attributes and geographic location data points could enable subpopulation comparisons and allow identification of patients for clinical trials, registries and patient-level studies. Regulatory processes, such as IRB protocols and data use agreements that allow researchers to retain identifiers, would facilitate project sustainability. Data owners are often concerned about the security of shared data. A critical first step toward addressing such concerns involves building trust among potential data partners at the outset of a project and establishing data governance strategies that are sufficient to preserve patient privacy and at the same time allow data to be shared between clinicians and researchers. For example, one project established, for all data partners, a master consortium agreement that stipulated the uses and rules for data. Other projects established distributed research networks as a solution to this problem. Addressing data standardization issues
Though investigators anticipated challenges with linking data sources, the time required to resolve the issues was more extensive than planned. The primary purpose of collecting data for most clinics is to ensure that services provided are billed properly. Uses of these data for research are considered secondarily, when considered at all. Claims data, when combined with clinical data, could be a powerful research dataset, if data linkage issues can be resolved. Data linkage issues mentioned by investigators included datasets with different identifier numbers across clinics in the same healthcare system, de-identified data, missing data, fragmented data and disparate data sources. Resolving or lessening the impact of data linkage issues would support project sustainability because it would allow investigators to work with datasets more quickly, to produce CER evidence more quickly and to mitigate potential delays in project timelines. Increasing the amount of data, by increasing the number of patients, the time period included or the number of data domains and linked datasets would provide investigators who conduct CER with larger subgroups of populations that have less common medical conditions or undergo less common treatments, enabling the generation of CER evidence for those subgroups. Multiple data sources
future science group
American Recovery and Reinvestment Act investments in data infrastructure
must include medical charts (obtaining EHR data, abstracting data from paper records or pulling from an existing registry); laboratory reports; pharmaceutical records; and patient-reported data. Building research capacity for other investigators &/or clinicians
Many projects sought to build research capacity within their organization or institution, as well as in practices and health centers, in an effort to make the infrastructure sustainable. Supporting training and capacity building means that practices, health centers and new investigators will have the skills to secure their own grants to take advantage of the infrastructure these projects developed. For example, to sustain the work beyond the ARRA funding period, several projects in the Enhancing State Data subgroup trained staff at various organizations to collect data on race and ethnicity. Research Networks projects provided training to community health center clinicians and staff who went on to seek their own funding for local-level research projects. Several projects included researchers who conducted CER with a newly developed data resource and trained other researchers in the necessary skills to analyze large datasets. Conclusion & future perspective Through our study of Recovery Act–funded projects, we identified several lessons learned that may inform considerations for future investments in CER DI.
Research Article
First, projects with established DI and technology in place to enable them to efficiently continue their data linkages into the future might provide more rapid returns on CER. Second, projects that developed data resources that can be used for CER and other complementary purposes enhanced buy-in and cooperation by providers. For example, some CER DI could be leveraged to provide practices with more data for quality improvement, meaningful use and pay-for-performance initiatives. Such uses have the added advantage of demonstrating the value of CER DI to real-world stakeholders who will be asked to buy-in by sharing their data. Third, because a priority for CER is clarifying the effectiveness of different treatment strategies in specific subgroups of patients, projects that have already successfully linked large data sources to support CER on large samples of often underrepresented subgroups should offer promising areas for continued investment. Projects that combine rich sources of clinical and administrative data across many practices also offer promising resources for future high-impact CER. For example, registries represent a potentially valuable investment because they aggregate data on a wide variety of patients with high-priority conditions from across multiple relevant clinical settings. Financial & competing interests disclosure The project on which this research was based was funded by the US Department of Health and Human Services Assistant
Executive summary • The American Recovery and Reinvestment Act comparative effectiveness research (CER) data infrastructure projects developed and/or enhanced a wide range of data resources for CER, some of which were already being incorporated into grant proposals and future projects at the time of the interviews. • More than half of the projects proposed to build infrastructure or address research questions specific to one or more medical conditions, including cardiovascular diseases and cancer. • At the time of the study, projects were making strong progress toward meeting goals, which included automating data collection and aggregating data from multiple data sources, developing databases and datasets that incorporated data from various sources and were large enough that projects could address questions specific to the Institute of Medicine priority populations, successfully integrating informatics tools and algorithms into electronic health records for clinical use and for conducting CER, expanding existing research networks and conducting pilot tests. • Project investigators reported several facilitators of project success, including an existing infrastructure to build upon, a project team with relevant expertise and strong partner collaborations. • Investigators mentioned lacking an existing project infrastructure to build upon, experienced team members or partner buy-in as challenges to their project’s success. In addition, they described challenges in linking data from different systems, data completeness and consistency problems and regulatory issues. • To sustain their data infrastructure projects, investigators should attempt to reduce burden on practices that collect data for CER, address data security and standardization issues and build research capacity for other investigators and/or clinicians. • Projects with mature data infrastructure and technology in place to enable them to efficiently continue their data linkages into the future might provide more rapid returns on CER. • Prioritizing support for data resources that can be used for CER and other complementary purposes may enhance the potential return on investment to and buy-in by providers.
future science group
www.futuremedicine.com
599
Research Article O’Day, Kieffer, Forrestal & Esposito Secretary for Planning and Evaluation. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
Ethical conduct of research
References
2
1
600
Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. National Academies Press, Washington, DC, USA (2009).
J. Comp. Eff. Res. (2014) 3(6)
The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. US Department of Health and Human Services, Washington, DC, USA (2009).
future science group
