SPECIAL FOCUS y The ARRA investment in CER Opinion For reprint orders, please contact: [email protected]
Building electronic data infrastructure for comparative effectiveness research: accomplishments, lessons learned and future steps “
The collective infrastructure assets and experiences of these American Recovery and Reinvestment Act-funded projects have laid the foundation to build sustainable learning health systems...
”
There are large gaps in our knowledge on the potential impact of diagnostics and therapeutics on outcomes of patients treated in the real world. Comparative effectiveness research aims to fill these gaps to maximize effectiveness of these interventions. Health information technology has the potential to dramatically improve the practice of medicine and of research. This is an overview of about US$100 million of American Recovery and Reinvestment Act investment in 12 projects managed by the Agency for Healthcare Research and Quality to build an electronic clinical data infrastructure that connects research with healthcare delivery. The achievements and lessons learned from these projects provided a foundation for the National Patient-Centered Clinical Research Network (PCORnet) and will help to guide future infrastructure development needed to build an efficient, scalable and sustainable learning health system. Keywords: American Recovery Reinvestment Act • clinical informatics • comparative effectiveness research • data infrastructure • distributed research • learning health system • quality improvement • registry • sustainability
Background A sustained investment in comparative effectiveness research (CER) is needed to bridge the large knowledge gaps on the outcomes of diagnostics and therapeutics on patients treated in the real world. These gaps have been documented in numerous systematic literature reviews [1] . Several factors have contributed to creating these knowledge gaps, including a lack of alignment in the priorities and incentives of researchers and clinicians, lack of ready access to data, varying quality of data created during clinical care, paucity of tools and methods to analyze ‘big data’, and lack of a health information infrastructure that seamlessly and accurately captures a patient’s experience across multiple healthcare delivery sites. The Agency for Healthcare Research and Quality (AHRQ) invested about US$100 million of American Recovery and Reinvestment Act (ARRA) funds to build electronic clinical data systems
10.2217/CER.14.73 © 2014 Future Medicine Ltd
for collecting patient-centered data that can be used for research, quality improvement and clinical care in order to address these issues. The investment had two main goals: create an electronic clinical data infrastructure for conduct of CER in diverse diseases, populations, and care delivery sites; and connect the research and healthcare delivery infrastructures to improve efficiency of research and the quality of care. Three programs (consisting of 11 grants) were started to build the infra structure: Prospective Outcome Systems using Patient-specific Electronic data to Compare Test and therapies (PROSPECT); enhanced registries for Quality Improvement (QI) and CER; and Scalable Distributed Research Networks (DRNs). An additional investment was made to create the Electronic Data Methods (EDM) Forum to foster interdisciplinary collaborations and advance the methods in clinical informatics, analytics, governance and
J. Comp. Eff. Res. (2014) 3(6), 567–572
Gurvaneet S Randhawa Center for Evidence & Practice Improvement, Agency for Healthcare Research & Quality, Rockville, MD 20850, USA Tel.: +1 301 427 1619 Fax: +1 301 427 1639 Gurvaneet.Randhawa@ahrq. hhs.gov
part of
ISSN 2042-6305
567
Opinion Randhawa learning health system by actively engaging investigators from these ARRA-funded projects and a diverse group of stakeholders. Requirements of Infrastructure Programs Although the 11 projects differed in geographic locations, number and type of care delivery settings, patient populations, specific aims and designs of CER studies, and electronic data architectures, all had a set of common requirements to: • L ink data across multiple healthcare delivery sites (e.g. inpatient, ambulatory care, and long-term care); • L ink different types of databases (e.g. claims, electronic health record [EHR], pharmacy, and diagnostic) and work across diverse information technology architectures; • F ocus on AHRQ priority populations and CER conditions (including the IOM’s priority CER topics); • Collect prospective, patient-centered outcomes; • C onduct CER designed to yield valid and generalizable conclusions; • Participate in scientific collaborations; • Create a governance plan; • Develop a sustainability plan. The PROSPECT program was designed to equally invest in infrastructure development and the conduct of CER. Some PROSPECT projects conducted randomized trials, others observational studies only. The scalable DRN program was designed to assess the capability of the infrastructure to answer diverse CER questions by creating clinically unrelated patient cohorts. The enhanced registry program was designed to ensure the infrastructure meets the needs of the research and QI communities. Highlights of achievements First instance of generalizable knowledge created in a learning health system
The ImproveCareNow network was transformed from a labor-intensive, data-sparse, QI network to an efficient, data-rich, EHR-based learning health system working towards data-in-once – that is, data collected one time for multiple purposes – clinical care, QI and CER [2] . ARRA funding enabled its rapid growth to become the nation’s largest registry of pediatric inflammatory bowel disease (IBD), which covers about a
568
J. Comp. Eff. Res. (2014) 3(6)
third of these patients. It conducted observational CER from data collected during clinical care and showed the outcomes are similar to those observed in a smaller RCT [3] .This is the first reported instance of a learning health system creating generalizable knowledge [4] . Largest private-sector diabetes registry
The Surveillance, Prevention and Management of Diabetes Mellitus (SUPREME-DM) project created a registry of about 1.3 million diabetes patients from 11 integrated healthcare delivery systems that are a part of the HMO Research Network (HMORN) [5] . First distributed research network focused on under-served populations
The design of the Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) project [6] was based on the experience of the DARTNet project, which created a DRN from a pre-existing practice-based research network. SAFTINet has built a trusted, multistate community of safety net stakeholders and researchers to lead and participate in a learning community to address evidence gaps relevant to safety net populations. Connecting a community to healthcare delivery
The Washington Heights/Inwood Infrastructure for Comparative Effectiveness Research (WICER) project created a community-based infrastructure serving an inner-city, under-served population. It conducted surveys to learn the health status and needs of the community and integrated survey data with clinical data. It linked databases derived from home care, ambulatory care, in-patient care, and a long-term care facility, to conduct CER and to improve control of high blood pressure. The project is providing tailored data visualizations to survey participants with the goal of returning data back to the community. Additionally, WICER identified major security considerations for primary data collection with tablet computers, proposed strategies for implementing data collection forms, and clarified the security, cost, and workflow of each strategy [7] . EHR-linkage to enhance surgical QI & cross-disciplinary CER
The Surgical Care and Outcomes Assessment Program was created to improve quality of surgical care; it covers about 90% of the surgical care delivered in the state of Washington. ARRA funding transformed it into a learning healthcare system called CERTAIN that automated flow of information, collected patientreported outcomes (PROs), improved patient engagement, created patient cohorts from surgical and medical care delivery sites to conduct CER, and enhanced its QI capability [8] .
future science group
Building electronic data infrastructure for comparative effectiveness research
New informatics tools to improve efficiency and scale of CER
An EDM Forum report provides an overview of the 31 informatics tools and approaches created or adapted by these projects to improve their capability in collection, access, exchange or aggregation of data; population analytics; and decision support; collectively, these tools have improved the efficiency and scale of CER [9] . The Scalable National Network for Effectiveness Research (SCANNER) project, in collaboration with the SAFTINet, led the effort to upgrade the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to a new version that could meet their CER needs. The CER-Hub project created a collaborative, web-based platform for multi-site CER using the MediClass application, which can process structured data and free text data (using natural language processing) from different EHRs. The SCANNER project created the global binary logistic regression (GLORE) tool to build shared data models without sharing data. SAFTINet created a packaged data tool – reusable OMOP and SAFTINet interface adaptor (ROSITA) – to transform claims-based and EHR data to the OMOP CDM. Privacy & security policy framework
The SCANNER project conducted a comprehensive analysis of privacy and security policies to inform the development of its DRN. It identified federal requirements for privacy and security and also of the states where SCANNER organizations are located. The analysis clarified several governance issues that needed to be resolved, including patient anonymity, consent for data sharing, verifying identity and authorization for data access, prior to conducting research [10] . This analysis serves as a foundation for others building DRNs. Using patient-reported outcomes in practice
The CERTAIN project created an interactive data platform for real-time use that collects PROs and assists patient and clinician communication and decisionmaking at the point-of-care. The ImproveCareNow enhanced registry created a mobile app that assesses a patient’s symptoms and well-being between clinic visits. It also provided a mechanism for parents to document non-urgent questions. The SAFTINet project used a standardized asthma questionnaire to inform clinicians on the level of asthma control in a patient.
trial, ability to report statistics on population health, a graphical platform for viewing care giver and patient assessment scores, and a software package that interprets the data to generate patient-centric care plans [9,11] . Accelerating multi-disciplinary learning through a new collaborative forum
The EDM Forum’s work is guided by a national steering committee representing diverse expertise and stakeholders. At the start, the EDM Forum met with stakeholders, conducted site visits and an environmental scan to understand the current landscape on electronic health data initiatives and clarify how best to synergistically meet the needs of the community. It has supported methods collaborations in data quality, governance and distributed analytics. It used several approaches – large meetings, focused workgroups, webinars, and collaborative methods projects – to build a collaborative, credible and trusted space for diverse stakeholders.
“
A notable achievement is the launch of a new, rapidly-growing, open-access e-journal – eGEMs – in 2013 to share innovations, lessons learned, challenges and potential solutions...
”
It supported publications from a multi-disciplinary group of authors in two open-access supplements of Medical Care (released in 2012 and 2013, respectively) to advance the knowledge in analytic methods for CER, clinical informatics, governance, and building a learning health system. One paper was selected among the “Best Papers in Clinical Research Informatics” in the International Medical Informatics Association’s 2013 yearbook [12] . A notable achievement is the launch of a new, rapidly-growing, open-access e-journal – eGEMs – in 2013 to share innovations, lessons learned, challenges and potential solutions from those at the frontlines of creating, analyzing and using electronic health data for CER, QI and clinical care. It has published 70 papers with over 30,000 downloads. Other useful open-access resources created by EDM Forum include a governance toolkit, which comprises sample data use agreements, guidance on research best practices, and a set of eGEMs papers related to governance [13] ; a webinar repository; and a report on the informatics tools developed by these projects [9] .
Improving research efficiency by connecting to clinical care
Sustainability
The Indiana PROSPECT project infrastructure enhancements include development of a tool for realtime identification at the point-of-care of an eligible patient with Alzheimer’s disease to an ongoing clinical
The funding announcements asked for a sustainability plan. Sustainability was discussed in several program meetings over three years, culminating in the commission of an eGEMs issue on sustainabil-
future science group
Opinion
www.futuremedicine.com
569
Opinion Randhawa ity. The special issue includes eight case studies and perspectives from leaders in the field and highlights promising business models and partner strategies to sustainably support the use of health data to generate evidence. All are available online [14] . Almost all projects have received subsequent funding from one or more sources, which is a promising sign of their sustainability. Lessons Learned Transition from manual data collection in a QI network to automated data collection and extraction of additional data needed for CER is challenging Both CERTAIN and ImproveCareNow projects (funded by the enhanced registry program) transformed their labor-intensive and data-sparse QI networks to EHR-connected, data-rich networks capable of conducting CER. The CERTAIN project categorized automated data extraction from EHRs into three types: easy, moderate, and complex, and showed that a large proportion of EHR data are not amenable to easy extraction. Additionally, the difficulty of data extraction depends upon the configuration of a hospital system, which underlines the importance of thinking of the secondary uses of data before implementing hospital information systems [15] . The ImproveCareNow project evaluated the state-of-theart in building its infrastructure. Its experience shows that data analysis and data transmission require greater work compared with data collection and data processing [2] . Need to standardize commonly used terminology & definitions
The SUPREME-DM project found inconsistent terminology and definitions are used in prescription adherence research with electronic databases. It proposed a conceptual model and a unifying set of definitions for prescription adherence research to facilitate future CER [16] . DRN tools & experience are rapidly maturing
The SAFTINet and SCANNER projects, benefiting from EDM Forum discussions, chose to adopt the OMOP CDM. However, the existing version did not meet their CER needs. The willingness of the OMOP user community to meet the needs of researchers led to the development of a CDM version 4. The scalable partnering network for CER (SPAN) DRN project expanded the Virtual Data Warehouse (VDW) CDM to two non-HMORN organizations. This work showed the resource commitment needed to create a new VDW may limit its adoption to large, wellresourced organizations. Further, the SPAN project
570
J. Comp. Eff. Res. (2014) 3(6)
refined the PopMedNetTM tool for conducting distributed queries. PopMedNet has been adopted by a new national patient-centered clinical research network – PCORnet. It needs significant modification before it can be used for PCORnet-wide research. In a collaboration sponsored by the EDM Forum, SCANNER’s GLORE tool was successfully used (after modification to a web-based version) in the ImproveCareNow network. This experience demonstrated that a modular GLORE tool, where different modules are used for different purposes, may be more useful to diverse users. It is likely that further experience from multiple users will enable convergence towards a small set of CDMs and analytic tools that can be used in diverse settings for multiple purposes. However, it is unlikely that a one-size-fits-all strategy will work. Data quality needs substantial improvement
The variable quality of data collected during clinical care can be a major rate limiting step in the quest to link diverse clinical databases for research. The EDM Forum supported work to develop a pragmatic framework to assess data quality for EHR-based research [17] , which led to creation of a data quality collaborative to bring together different national activities assessing data quality. This collaborative has now received Patient-Centered Outcomes Research Institute (PCORI) support. Assessing data quality is only a first step, a sustained focus and investment is needed to create incentives and tools to improve quality of data collected during routine clinical care. Governance, stakeholder engagement & value are paramount
The SPAN project showed that even among organizations with a long-standing track record of collaboration, security and confidentiality concerns can prove to be a barrier for CER. It took time to work through the concerns of the governance group and network developers. Governance input into the design of the network led to a delay but, after the concerns were satisfied, the tools were rapidly installed and accepted in all sites [18] . The WICER project showed that building trust in a community and providing value to community members facilitated conduct of subsequent biomedical research projects. The ImproveCareNow and DARTNet/SAFTINet projects have shown that providing value to clinicians facilitates their participation in a collaborative network, the adoption of software, and growth of a network. The patient and clinicians need to have meaningful participation in research projects to ensure a proper focus of research questions and faster implementation of research findings.
future science group
Building electronic data infrastructure for comparative effectiveness research
CER & big data
Since these infrastructure projects have large (up to several million patients) and diverse patient populations, diverse and complex data sources and data elements, and rapidly changing data (due to linkages with EHRs), they are working with ‘big data’. The focus on big data has accelerated an ongoing paradigm shift. In the previous paradigm, researchers (e.g. epidemiologists) created datasets to be hoarded and mined for publications and grant support. In the new paradigm, the competition is not on the data but on the tools to create knowledge from data. In order to effectively use big data, we need to focus on development of analytic tools, governance, stakeholder value, and sustainability, while recognizing those who create and share “quality” data. A collaborative forum is needed to exchange innovations, lessons learned & to solve common problems
The EDM Forum has achieved remarkable success in its short existence. It rapidly grew its community of stakeholders beyond the ARRA-funded researchers because it has tackled common ‘pressure points’ across research projects such as governance, multi-site Institutional Review Board approval, and sustainability; supported inter-disciplinary collaborative methods projects to advance our knowledge in important but unglamorous areas of research such as data quality; and always focused on building trust and credibility. It has used a variety of dissemination and engagement methods to build its community and accelerate learning. The need for such collaborative activities has been described in new initiatives such as Big Data to K nowledge. Sustainability
There are several paths to sustainability. The HMORN (in existence for over 20 years) utilized its large networks and took a research-oriented path [19] . The relatively-new DARTNet Institute took a path with diverse funding streams: research, licensing software, and operation of a QI registry [20] . Although sustainability solutions will be heavily influenced by the local context, it is useful to think of four general considerations: available assets (structural and human), expansion and scalability, complexity of components, and engaging and providing value to diverse stakeholders [21] . Transparency & reproducibility of research
The assumptions and choices made in writing code and analyzing data are rarely published. The EDM Forum launched eGEMs to provide this service. The
future science group
Opinion
lack of a repository of open-access reference datasets derived from real-world data and of analytic tools with adequate documentation to compare and validate analyses continues to hinder the transparency and reproducibility of outcomes research. Achieving the future potential A close collaboration between the clinical informatics, clinical, patient, and CER communities has the potential to rapidly and efficiently generate evidence to fill our large knowledge gaps. This can be realized by creating and providing the appropriate informatics and analytic resources and tools, and providing the appropriate incentives to change the status quo. The incentives need to: • E nsure high quality of data created in a clinical encounter. Clinicians will pay greater attention to data quality when data are used in real-time to help in clinical care; • C onnect the clinical care and research infrastructures. Clinicians and care delivery organizations can benefit from the analytic capacity and skills of researchers to improve patient care. Researchers can benefit from high-quality clinical data to efficiently generate new knowledge; • Recognize creators of high-quality databases; • B uild credible and trustworthy collaborative spaces for researchers from multiple disciplines; • S upport development of useful, credible tools and methods; • S upport publication of research protocols and the ‘journey’ of research; • S upport stakeholder engagement and create value to stakeholders to ensure long-term sustainability; • E nsure patients and clinicians have a voice throughout the lifecycle of a research project. Almost all projects have received subsequent funding from one or more of the following organizations: AHRQ, Centers for Medicare & Medicaid Services, US FDA, Health Resources and Services Administration, NIH and PCORI, which demonstrates the diverse uses and value of their infrastructure. Several are now part of PCORnet. Their experience in governance has shaped PCORnet policies. They have benefited the larger community by sharing their innovations and lessons learned; by participating in large-scale, multidisciplinary, multi-site collaborations; and sharing their experiences to achieve sustainability. The collective infrastructure assets and experiences of these ARRA-
www.futuremedicine.com
571
Opinion Randhawa funded projects have laid the foundation to build sustainable learning health systems that can be used for research, QI and patient care.
position of the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.
Financial & competing interests disclosure Acknowledgements The author thanks William Lawrence (AHRQ), Yen-Pin Chiang (AHRQ) and Erin Holve (AcademyHealth) for their helpful comments and Beth Johnson (AcademyHealth) for editorial help.
Disclaimer The author of this article is responsible for the content. No statement in this article should be construed as an official
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties. No writing assistance was utilized in the production of this manuscript.
References 1
2
Randhawa GS, Slutsky JR. Building sustainable multifunctional prospective electronic clinical data systems. Med. Care 50(Suppl.), S3–S6 (2012).
12
Marsolo K. In search of a data-in-once, electronic health record-linked, multicenter registry – how far we have come and how far we still have to go. eGEMs 1(1), Article 3 (2013). http://repository.academyhealth.org/egems/vol1/iss1/3
Sittig D, Hazlehurst BL, Brown J et al. A survey of informatics platforms that enable distributed comparative effectiveness research using multi-institutional heterogeneous clinical data. Med. Care 50(Suppl.), S49–S59 (2012).
13
EDM Forum. Governance Toolkit. http://repository.academyhealth.org/govtoolkit
3
Forrest CB, Crandall WV, Bailey LC et al. Effectiveness of anti TNF for Crohn disease: research in a pediatric learning health system. Pediatrics 134(1), 37–44 (2014).
14
Wilcox A. Sustaining the effective use of health care data. eGEMs 2(2). http://repository.academyhealth.org/egems/vol2/iss2
4
Abernethy AP. Demonstrating the learning health system through practical use cases. Pediatrics 134(1), 171–172 (2014).
15
5
Nichols GA, Desai J, Lafata JE et al. Construction of a multisite datalink using electronic health records for the identification, surveillance, prevention, and management of diabetes mellitus: the SUPREME-DM project. Prev. Chron. Dis. 9, E110 (2012).
Capurro D, Yetisgen M, van Eaton E, Black R, Tarczy-Hornoch P. Availability of structured and unstructured clinical data for comparative effectiveness research and quality improvement: a multi-site assessment. eGEMs 2(1), Article 11 (2013). http://repository.academyhealth.org/egems/vol2/iss1/11
16
Raebel MA, Schmittdiel J, Karter AJ, Konieczny JL, Steiner JF. Standardizing terminology and definitions of medication adherence and persistence in research employing electronic databases. Med. Care 51(8 Suppl. 3), S11–S21 (2013).
17
Kahn MG, Raebel MA, Glanz JM, Riedlinger K, Steiner JF. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med. Care 50(Suppl.), S21–S29 (2012).
18
Daley MF. Trust and verify: Lessons learned from the SPAN
6
7
572
software (eMR-ABC). eGEMs 1(1), Article 8 (2013). http://repository.academyhealth.org/egems/vol1/iss1/8
Schilling LM, Kwan BM, Drolshagen CT et al. Scalable architecture for federated translational inquiries network (SAFTINet) technology infrastructure for a distributed data network. eGEMs 1(1), Article 11 (2013). http://repository.academyhealth.org/egems/vol1/iss1/11 Wilcox AB, Gallagher K, Bakken S. Security approaches in using tablet computers for primary data collection in clinical research. eGEMs 1(1), Article 7 (2013). http://repository.academyhealth.org/egems/vol1/iss1/7
network. EDM Forum Stakeholder Symposium. Baltimore,
8
CERTAIN Development [Internet]. www.becertain.org/about
MD, USA 22 June 2013.
9
EDM Forum. Informatics tools and approaches to facilitate the use of electronic data for CER, PCOR and QI: Resources developed by the PROSPECT, DRN, and Enhanced Registry projects. Issue Briefs and Reports 2013: http://repository.academyhealth.org/edm_briefs/11
10
Kim KK, McGraw D, Mamo L, Ohno-Machado L. Development of a privacy and security policy framework for a multistate comparative effectiveness research network. Med. Care 51(8 Suppl. 3), S66–S72 (2013).
11
Frame A, LaMantia M, Bynagri BBR, Dexter P, Boustani M. Development and implementation of an electronic decision support to manage the health of a high-risk population: the enhanced electronic medical record aging brain care
J. Comp. Eff. Res. (2014) 3(6)
http://repository.academyhealth.org 19
Steiner JF, Paolino AR, Thompson EE, Larson EB. Sustaining research networks: the twenty-year experience of the HMO research network. eGEMs 2(2), Article 1 (2013). http://repository.academyhealth.org/egems/vol2/iss2/1
20
Pace WD, Fox C, White T, Graham D, Schilling LM, West DR. The DARTNet Institute: seeking a sustainable support mechanism for electronic data enabled research networks. eGEMs 2(2), Article 6 (2013). http://repository.academyhealth.org/egems/vol2/iss2/6.
21
Wilcox A, Randhawa G, Embi P, Cao H, Kuperman G. Sustainability considerations for health research and analytic data infrastructures. eGEMs 2(2), Article 8 (2013). http://repository.academyhealth.org/egems/vol2/iss2/8
future science group
Building electronic data infrastructure for comparative effectiveness research: accomplishments, lessons learned and future steps “
The collective infrastructure assets and experiences of these American Recovery and Reinvestment Act-funded projects have laid the foundation to build sustainable learning health systems...
”
There are large gaps in our knowledge on the potential impact of diagnostics and therapeutics on outcomes of patients treated in the real world. Comparative effectiveness research aims to fill these gaps to maximize effectiveness of these interventions. Health information technology has the potential to dramatically improve the practice of medicine and of research. This is an overview of about US$100 million of American Recovery and Reinvestment Act investment in 12 projects managed by the Agency for Healthcare Research and Quality to build an electronic clinical data infrastructure that connects research with healthcare delivery. The achievements and lessons learned from these projects provided a foundation for the National Patient-Centered Clinical Research Network (PCORnet) and will help to guide future infrastructure development needed to build an efficient, scalable and sustainable learning health system. Keywords: American Recovery Reinvestment Act • clinical informatics • comparative effectiveness research • data infrastructure • distributed research • learning health system • quality improvement • registry • sustainability
Background A sustained investment in comparative effectiveness research (CER) is needed to bridge the large knowledge gaps on the outcomes of diagnostics and therapeutics on patients treated in the real world. These gaps have been documented in numerous systematic literature reviews [1] . Several factors have contributed to creating these knowledge gaps, including a lack of alignment in the priorities and incentives of researchers and clinicians, lack of ready access to data, varying quality of data created during clinical care, paucity of tools and methods to analyze ‘big data’, and lack of a health information infrastructure that seamlessly and accurately captures a patient’s experience across multiple healthcare delivery sites. The Agency for Healthcare Research and Quality (AHRQ) invested about US$100 million of American Recovery and Reinvestment Act (ARRA) funds to build electronic clinical data systems
10.2217/CER.14.73 © 2014 Future Medicine Ltd
for collecting patient-centered data that can be used for research, quality improvement and clinical care in order to address these issues. The investment had two main goals: create an electronic clinical data infrastructure for conduct of CER in diverse diseases, populations, and care delivery sites; and connect the research and healthcare delivery infrastructures to improve efficiency of research and the quality of care. Three programs (consisting of 11 grants) were started to build the infra structure: Prospective Outcome Systems using Patient-specific Electronic data to Compare Test and therapies (PROSPECT); enhanced registries for Quality Improvement (QI) and CER; and Scalable Distributed Research Networks (DRNs). An additional investment was made to create the Electronic Data Methods (EDM) Forum to foster interdisciplinary collaborations and advance the methods in clinical informatics, analytics, governance and
J. Comp. Eff. Res. (2014) 3(6), 567–572
Gurvaneet S Randhawa Center for Evidence & Practice Improvement, Agency for Healthcare Research & Quality, Rockville, MD 20850, USA Tel.: +1 301 427 1619 Fax: +1 301 427 1639 Gurvaneet.Randhawa@ahrq. hhs.gov
part of
ISSN 2042-6305
567
Opinion Randhawa learning health system by actively engaging investigators from these ARRA-funded projects and a diverse group of stakeholders. Requirements of Infrastructure Programs Although the 11 projects differed in geographic locations, number and type of care delivery settings, patient populations, specific aims and designs of CER studies, and electronic data architectures, all had a set of common requirements to: • L ink data across multiple healthcare delivery sites (e.g. inpatient, ambulatory care, and long-term care); • L ink different types of databases (e.g. claims, electronic health record [EHR], pharmacy, and diagnostic) and work across diverse information technology architectures; • F ocus on AHRQ priority populations and CER conditions (including the IOM’s priority CER topics); • Collect prospective, patient-centered outcomes; • C onduct CER designed to yield valid and generalizable conclusions; • Participate in scientific collaborations; • Create a governance plan; • Develop a sustainability plan. The PROSPECT program was designed to equally invest in infrastructure development and the conduct of CER. Some PROSPECT projects conducted randomized trials, others observational studies only. The scalable DRN program was designed to assess the capability of the infrastructure to answer diverse CER questions by creating clinically unrelated patient cohorts. The enhanced registry program was designed to ensure the infrastructure meets the needs of the research and QI communities. Highlights of achievements First instance of generalizable knowledge created in a learning health system
The ImproveCareNow network was transformed from a labor-intensive, data-sparse, QI network to an efficient, data-rich, EHR-based learning health system working towards data-in-once – that is, data collected one time for multiple purposes – clinical care, QI and CER [2] . ARRA funding enabled its rapid growth to become the nation’s largest registry of pediatric inflammatory bowel disease (IBD), which covers about a
568
J. Comp. Eff. Res. (2014) 3(6)
third of these patients. It conducted observational CER from data collected during clinical care and showed the outcomes are similar to those observed in a smaller RCT [3] .This is the first reported instance of a learning health system creating generalizable knowledge [4] . Largest private-sector diabetes registry
The Surveillance, Prevention and Management of Diabetes Mellitus (SUPREME-DM) project created a registry of about 1.3 million diabetes patients from 11 integrated healthcare delivery systems that are a part of the HMO Research Network (HMORN) [5] . First distributed research network focused on under-served populations
The design of the Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) project [6] was based on the experience of the DARTNet project, which created a DRN from a pre-existing practice-based research network. SAFTINet has built a trusted, multistate community of safety net stakeholders and researchers to lead and participate in a learning community to address evidence gaps relevant to safety net populations. Connecting a community to healthcare delivery
The Washington Heights/Inwood Infrastructure for Comparative Effectiveness Research (WICER) project created a community-based infrastructure serving an inner-city, under-served population. It conducted surveys to learn the health status and needs of the community and integrated survey data with clinical data. It linked databases derived from home care, ambulatory care, in-patient care, and a long-term care facility, to conduct CER and to improve control of high blood pressure. The project is providing tailored data visualizations to survey participants with the goal of returning data back to the community. Additionally, WICER identified major security considerations for primary data collection with tablet computers, proposed strategies for implementing data collection forms, and clarified the security, cost, and workflow of each strategy [7] . EHR-linkage to enhance surgical QI & cross-disciplinary CER
The Surgical Care and Outcomes Assessment Program was created to improve quality of surgical care; it covers about 90% of the surgical care delivered in the state of Washington. ARRA funding transformed it into a learning healthcare system called CERTAIN that automated flow of information, collected patientreported outcomes (PROs), improved patient engagement, created patient cohorts from surgical and medical care delivery sites to conduct CER, and enhanced its QI capability [8] .
future science group
Building electronic data infrastructure for comparative effectiveness research
New informatics tools to improve efficiency and scale of CER
An EDM Forum report provides an overview of the 31 informatics tools and approaches created or adapted by these projects to improve their capability in collection, access, exchange or aggregation of data; population analytics; and decision support; collectively, these tools have improved the efficiency and scale of CER [9] . The Scalable National Network for Effectiveness Research (SCANNER) project, in collaboration with the SAFTINet, led the effort to upgrade the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to a new version that could meet their CER needs. The CER-Hub project created a collaborative, web-based platform for multi-site CER using the MediClass application, which can process structured data and free text data (using natural language processing) from different EHRs. The SCANNER project created the global binary logistic regression (GLORE) tool to build shared data models without sharing data. SAFTINet created a packaged data tool – reusable OMOP and SAFTINet interface adaptor (ROSITA) – to transform claims-based and EHR data to the OMOP CDM. Privacy & security policy framework
The SCANNER project conducted a comprehensive analysis of privacy and security policies to inform the development of its DRN. It identified federal requirements for privacy and security and also of the states where SCANNER organizations are located. The analysis clarified several governance issues that needed to be resolved, including patient anonymity, consent for data sharing, verifying identity and authorization for data access, prior to conducting research [10] . This analysis serves as a foundation for others building DRNs. Using patient-reported outcomes in practice
The CERTAIN project created an interactive data platform for real-time use that collects PROs and assists patient and clinician communication and decisionmaking at the point-of-care. The ImproveCareNow enhanced registry created a mobile app that assesses a patient’s symptoms and well-being between clinic visits. It also provided a mechanism for parents to document non-urgent questions. The SAFTINet project used a standardized asthma questionnaire to inform clinicians on the level of asthma control in a patient.
trial, ability to report statistics on population health, a graphical platform for viewing care giver and patient assessment scores, and a software package that interprets the data to generate patient-centric care plans [9,11] . Accelerating multi-disciplinary learning through a new collaborative forum
The EDM Forum’s work is guided by a national steering committee representing diverse expertise and stakeholders. At the start, the EDM Forum met with stakeholders, conducted site visits and an environmental scan to understand the current landscape on electronic health data initiatives and clarify how best to synergistically meet the needs of the community. It has supported methods collaborations in data quality, governance and distributed analytics. It used several approaches – large meetings, focused workgroups, webinars, and collaborative methods projects – to build a collaborative, credible and trusted space for diverse stakeholders.
“
A notable achievement is the launch of a new, rapidly-growing, open-access e-journal – eGEMs – in 2013 to share innovations, lessons learned, challenges and potential solutions...
”
It supported publications from a multi-disciplinary group of authors in two open-access supplements of Medical Care (released in 2012 and 2013, respectively) to advance the knowledge in analytic methods for CER, clinical informatics, governance, and building a learning health system. One paper was selected among the “Best Papers in Clinical Research Informatics” in the International Medical Informatics Association’s 2013 yearbook [12] . A notable achievement is the launch of a new, rapidly-growing, open-access e-journal – eGEMs – in 2013 to share innovations, lessons learned, challenges and potential solutions from those at the frontlines of creating, analyzing and using electronic health data for CER, QI and clinical care. It has published 70 papers with over 30,000 downloads. Other useful open-access resources created by EDM Forum include a governance toolkit, which comprises sample data use agreements, guidance on research best practices, and a set of eGEMs papers related to governance [13] ; a webinar repository; and a report on the informatics tools developed by these projects [9] .
Improving research efficiency by connecting to clinical care
Sustainability
The Indiana PROSPECT project infrastructure enhancements include development of a tool for realtime identification at the point-of-care of an eligible patient with Alzheimer’s disease to an ongoing clinical
The funding announcements asked for a sustainability plan. Sustainability was discussed in several program meetings over three years, culminating in the commission of an eGEMs issue on sustainabil-
future science group
Opinion
www.futuremedicine.com
569
Opinion Randhawa ity. The special issue includes eight case studies and perspectives from leaders in the field and highlights promising business models and partner strategies to sustainably support the use of health data to generate evidence. All are available online [14] . Almost all projects have received subsequent funding from one or more sources, which is a promising sign of their sustainability. Lessons Learned Transition from manual data collection in a QI network to automated data collection and extraction of additional data needed for CER is challenging Both CERTAIN and ImproveCareNow projects (funded by the enhanced registry program) transformed their labor-intensive and data-sparse QI networks to EHR-connected, data-rich networks capable of conducting CER. The CERTAIN project categorized automated data extraction from EHRs into three types: easy, moderate, and complex, and showed that a large proportion of EHR data are not amenable to easy extraction. Additionally, the difficulty of data extraction depends upon the configuration of a hospital system, which underlines the importance of thinking of the secondary uses of data before implementing hospital information systems [15] . The ImproveCareNow project evaluated the state-of-theart in building its infrastructure. Its experience shows that data analysis and data transmission require greater work compared with data collection and data processing [2] . Need to standardize commonly used terminology & definitions
The SUPREME-DM project found inconsistent terminology and definitions are used in prescription adherence research with electronic databases. It proposed a conceptual model and a unifying set of definitions for prescription adherence research to facilitate future CER [16] . DRN tools & experience are rapidly maturing
The SAFTINet and SCANNER projects, benefiting from EDM Forum discussions, chose to adopt the OMOP CDM. However, the existing version did not meet their CER needs. The willingness of the OMOP user community to meet the needs of researchers led to the development of a CDM version 4. The scalable partnering network for CER (SPAN) DRN project expanded the Virtual Data Warehouse (VDW) CDM to two non-HMORN organizations. This work showed the resource commitment needed to create a new VDW may limit its adoption to large, wellresourced organizations. Further, the SPAN project
570
J. Comp. Eff. Res. (2014) 3(6)
refined the PopMedNetTM tool for conducting distributed queries. PopMedNet has been adopted by a new national patient-centered clinical research network – PCORnet. It needs significant modification before it can be used for PCORnet-wide research. In a collaboration sponsored by the EDM Forum, SCANNER’s GLORE tool was successfully used (after modification to a web-based version) in the ImproveCareNow network. This experience demonstrated that a modular GLORE tool, where different modules are used for different purposes, may be more useful to diverse users. It is likely that further experience from multiple users will enable convergence towards a small set of CDMs and analytic tools that can be used in diverse settings for multiple purposes. However, it is unlikely that a one-size-fits-all strategy will work. Data quality needs substantial improvement
The variable quality of data collected during clinical care can be a major rate limiting step in the quest to link diverse clinical databases for research. The EDM Forum supported work to develop a pragmatic framework to assess data quality for EHR-based research [17] , which led to creation of a data quality collaborative to bring together different national activities assessing data quality. This collaborative has now received Patient-Centered Outcomes Research Institute (PCORI) support. Assessing data quality is only a first step, a sustained focus and investment is needed to create incentives and tools to improve quality of data collected during routine clinical care. Governance, stakeholder engagement & value are paramount
The SPAN project showed that even among organizations with a long-standing track record of collaboration, security and confidentiality concerns can prove to be a barrier for CER. It took time to work through the concerns of the governance group and network developers. Governance input into the design of the network led to a delay but, after the concerns were satisfied, the tools were rapidly installed and accepted in all sites [18] . The WICER project showed that building trust in a community and providing value to community members facilitated conduct of subsequent biomedical research projects. The ImproveCareNow and DARTNet/SAFTINet projects have shown that providing value to clinicians facilitates their participation in a collaborative network, the adoption of software, and growth of a network. The patient and clinicians need to have meaningful participation in research projects to ensure a proper focus of research questions and faster implementation of research findings.
future science group
Building electronic data infrastructure for comparative effectiveness research
CER & big data
Since these infrastructure projects have large (up to several million patients) and diverse patient populations, diverse and complex data sources and data elements, and rapidly changing data (due to linkages with EHRs), they are working with ‘big data’. The focus on big data has accelerated an ongoing paradigm shift. In the previous paradigm, researchers (e.g. epidemiologists) created datasets to be hoarded and mined for publications and grant support. In the new paradigm, the competition is not on the data but on the tools to create knowledge from data. In order to effectively use big data, we need to focus on development of analytic tools, governance, stakeholder value, and sustainability, while recognizing those who create and share “quality” data. A collaborative forum is needed to exchange innovations, lessons learned & to solve common problems
The EDM Forum has achieved remarkable success in its short existence. It rapidly grew its community of stakeholders beyond the ARRA-funded researchers because it has tackled common ‘pressure points’ across research projects such as governance, multi-site Institutional Review Board approval, and sustainability; supported inter-disciplinary collaborative methods projects to advance our knowledge in important but unglamorous areas of research such as data quality; and always focused on building trust and credibility. It has used a variety of dissemination and engagement methods to build its community and accelerate learning. The need for such collaborative activities has been described in new initiatives such as Big Data to K nowledge. Sustainability
There are several paths to sustainability. The HMORN (in existence for over 20 years) utilized its large networks and took a research-oriented path [19] . The relatively-new DARTNet Institute took a path with diverse funding streams: research, licensing software, and operation of a QI registry [20] . Although sustainability solutions will be heavily influenced by the local context, it is useful to think of four general considerations: available assets (structural and human), expansion and scalability, complexity of components, and engaging and providing value to diverse stakeholders [21] . Transparency & reproducibility of research
The assumptions and choices made in writing code and analyzing data are rarely published. The EDM Forum launched eGEMs to provide this service. The
future science group
Opinion
lack of a repository of open-access reference datasets derived from real-world data and of analytic tools with adequate documentation to compare and validate analyses continues to hinder the transparency and reproducibility of outcomes research. Achieving the future potential A close collaboration between the clinical informatics, clinical, patient, and CER communities has the potential to rapidly and efficiently generate evidence to fill our large knowledge gaps. This can be realized by creating and providing the appropriate informatics and analytic resources and tools, and providing the appropriate incentives to change the status quo. The incentives need to: • E nsure high quality of data created in a clinical encounter. Clinicians will pay greater attention to data quality when data are used in real-time to help in clinical care; • C onnect the clinical care and research infrastructures. Clinicians and care delivery organizations can benefit from the analytic capacity and skills of researchers to improve patient care. Researchers can benefit from high-quality clinical data to efficiently generate new knowledge; • Recognize creators of high-quality databases; • B uild credible and trustworthy collaborative spaces for researchers from multiple disciplines; • S upport development of useful, credible tools and methods; • S upport publication of research protocols and the ‘journey’ of research; • S upport stakeholder engagement and create value to stakeholders to ensure long-term sustainability; • E nsure patients and clinicians have a voice throughout the lifecycle of a research project. Almost all projects have received subsequent funding from one or more of the following organizations: AHRQ, Centers for Medicare & Medicaid Services, US FDA, Health Resources and Services Administration, NIH and PCORI, which demonstrates the diverse uses and value of their infrastructure. Several are now part of PCORnet. Their experience in governance has shaped PCORnet policies. They have benefited the larger community by sharing their innovations and lessons learned; by participating in large-scale, multidisciplinary, multi-site collaborations; and sharing their experiences to achieve sustainability. The collective infrastructure assets and experiences of these ARRA-
www.futuremedicine.com
571
Opinion Randhawa funded projects have laid the foundation to build sustainable learning health systems that can be used for research, QI and patient care.
position of the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.
Financial & competing interests disclosure Acknowledgements The author thanks William Lawrence (AHRQ), Yen-Pin Chiang (AHRQ) and Erin Holve (AcademyHealth) for their helpful comments and Beth Johnson (AcademyHealth) for editorial help.
Disclaimer The author of this article is responsible for the content. No statement in this article should be construed as an official
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties. No writing assistance was utilized in the production of this manuscript.
References 1
2
Randhawa GS, Slutsky JR. Building sustainable multifunctional prospective electronic clinical data systems. Med. Care 50(Suppl.), S3–S6 (2012).
12
Marsolo K. In search of a data-in-once, electronic health record-linked, multicenter registry – how far we have come and how far we still have to go. eGEMs 1(1), Article 3 (2013). http://repository.academyhealth.org/egems/vol1/iss1/3
Sittig D, Hazlehurst BL, Brown J et al. A survey of informatics platforms that enable distributed comparative effectiveness research using multi-institutional heterogeneous clinical data. Med. Care 50(Suppl.), S49–S59 (2012).
13
EDM Forum. Governance Toolkit. http://repository.academyhealth.org/govtoolkit
3
Forrest CB, Crandall WV, Bailey LC et al. Effectiveness of anti TNF for Crohn disease: research in a pediatric learning health system. Pediatrics 134(1), 37–44 (2014).
14
Wilcox A. Sustaining the effective use of health care data. eGEMs 2(2). http://repository.academyhealth.org/egems/vol2/iss2
4
Abernethy AP. Demonstrating the learning health system through practical use cases. Pediatrics 134(1), 171–172 (2014).
15
5
Nichols GA, Desai J, Lafata JE et al. Construction of a multisite datalink using electronic health records for the identification, surveillance, prevention, and management of diabetes mellitus: the SUPREME-DM project. Prev. Chron. Dis. 9, E110 (2012).
Capurro D, Yetisgen M, van Eaton E, Black R, Tarczy-Hornoch P. Availability of structured and unstructured clinical data for comparative effectiveness research and quality improvement: a multi-site assessment. eGEMs 2(1), Article 11 (2013). http://repository.academyhealth.org/egems/vol2/iss1/11
16
Raebel MA, Schmittdiel J, Karter AJ, Konieczny JL, Steiner JF. Standardizing terminology and definitions of medication adherence and persistence in research employing electronic databases. Med. Care 51(8 Suppl. 3), S11–S21 (2013).
17
Kahn MG, Raebel MA, Glanz JM, Riedlinger K, Steiner JF. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med. Care 50(Suppl.), S21–S29 (2012).
18
Daley MF. Trust and verify: Lessons learned from the SPAN
6
7
572
software (eMR-ABC). eGEMs 1(1), Article 8 (2013). http://repository.academyhealth.org/egems/vol1/iss1/8
Schilling LM, Kwan BM, Drolshagen CT et al. Scalable architecture for federated translational inquiries network (SAFTINet) technology infrastructure for a distributed data network. eGEMs 1(1), Article 11 (2013). http://repository.academyhealth.org/egems/vol1/iss1/11 Wilcox AB, Gallagher K, Bakken S. Security approaches in using tablet computers for primary data collection in clinical research. eGEMs 1(1), Article 7 (2013). http://repository.academyhealth.org/egems/vol1/iss1/7
network. EDM Forum Stakeholder Symposium. Baltimore,
8
CERTAIN Development [Internet]. www.becertain.org/about
MD, USA 22 June 2013.
9
EDM Forum. Informatics tools and approaches to facilitate the use of electronic data for CER, PCOR and QI: Resources developed by the PROSPECT, DRN, and Enhanced Registry projects. Issue Briefs and Reports 2013: http://repository.academyhealth.org/edm_briefs/11
10
Kim KK, McGraw D, Mamo L, Ohno-Machado L. Development of a privacy and security policy framework for a multistate comparative effectiveness research network. Med. Care 51(8 Suppl. 3), S66–S72 (2013).
11
Frame A, LaMantia M, Bynagri BBR, Dexter P, Boustani M. Development and implementation of an electronic decision support to manage the health of a high-risk population: the enhanced electronic medical record aging brain care
J. Comp. Eff. Res. (2014) 3(6)
http://repository.academyhealth.org 19
Steiner JF, Paolino AR, Thompson EE, Larson EB. Sustaining research networks: the twenty-year experience of the HMO research network. eGEMs 2(2), Article 1 (2013). http://repository.academyhealth.org/egems/vol2/iss2/1
20
Pace WD, Fox C, White T, Graham D, Schilling LM, West DR. The DARTNet Institute: seeking a sustainable support mechanism for electronic data enabled research networks. eGEMs 2(2), Article 6 (2013). http://repository.academyhealth.org/egems/vol2/iss2/6.
21
Wilcox A, Randhawa G, Embi P, Cao H, Kuperman G. Sustainability considerations for health research and analytic data infrastructures. eGEMs 2(2), Article 8 (2013). http://repository.academyhealth.org/egems/vol2/iss2/8
future science group
