Articles A Data-Rich Recruitment Core to Support Translational Clinical Research Rhonda G. Kost, M.D.1, Lauren M. Corregano, M.S.W.1, Tyler-Lauren Rainer, B.A.1, Caroline Melendez, B.A.1,2, and Barry S. Coller, M.D.1 Abstract Background: Underenrollment of clinical studies wastes resources and delays assessment of research discoveries. We describe the organization and impact of a centralized recruitment core delivering comprehensive recruitment support to investigators. Methods: The Rockefeller University Center for Clinical and Translational Science supports a centralized recruitment core, call center, Research Volunteer Repository, data infrastructure, and staff who provide expert recruitment services to investigators. During protocol development, consultations aim to optimize enrollment feasibility, develop recruitment strategy, budget, and advertising. Services during study conduct include advertising placement, repository queries, call management, prescreening, referral, and visit scheduling. Utilization and recruitment outcomes are tracked using dedicated software. Results: For protocols receiving recruitment services during 2009–2013: median time from initiation of recruitment to the first enrolled participant was 10 days; of 4,047 first-time callers to the call center, 92% (n = 3,722) enrolled in the Research Volunteer Repository, with 99% retention; 23% of Repository enrollees subsequently enrolled in ≥1 research studies, with 89% retention. Of volunteers referred by repository queries, 49% (280/537) enrolled into the study, with 92% retained. Conclusions: Provision of robust recruitment infrastructure including expertise, a volunteer repository, data capture and real-time analysis accelerates protocol accrual. Application of recruitment science improves the quality of clinical investigation. Clin Trans Sci 2015; Volume 8: 91–99
Keywords: enrollment, accrual, clinical trial management, clinical research management Introduction
Timely enrollment of participants into clinical studies is a major challenge nationally, with as many as 75% of investigators failing to enroll the target number of subjects, and 90% of trials failing to enroll the requisite number of subjects within the proposed time period.1 In addition to the missed opportunities for scientific and medical advances, underenrollment also has economic consequences. For example, one academic medical center estimated their costs of underenrollment as exceeding one million dollars in a single year.2 Recruitment to industry-sponsored clinical trials conducted outside of academic centers commonly utilizes recruitment centers offering professional marketing strategies.3 In sharp contrast, recruitment to clinical trials and investigator-initiated studies at academic centers has historically been left to the research team, which commonly has limited recruitment experience and competing priorities.2 Academic institutions supported by Clinical and Translation Science Awards (CTSAs) have begun to offer participant registries and other recruitment resources to support enrollment of clinical protocols,4–9 but few have published the details of the organization or impact of these resources. To address the need for better support of recruitment, and to study the recruitment process itself, the Rockefeller University Center for Clinical and Translational Science (CCTS), committed CTSA resources to create a recruitment core with three specific goals: (1) to provide recruitment expertise early in protocol development in order to judge feasibility and minimize potential barriers; (2) to provide centralized recruitment services and resources throughout the protocol to insure expert execution of the recruitment strategy and to recruit participants; and (3) to capture and analyze recruitment-related outcome data in real time to inform modifications in the recruitment strategy or the protocol to optimize accrual. These data also form the basis for advancing the science of recruitment in the academic setting, defining best practices, and supporting continuous performance improvement. In this report, we describe our recruitment core
services, their utilization, and their effectiveness in improving timely participant recruitment and accrual. Methods
The Clinical Research Recruitment and Outreach Support Service The Rockefeller University Clinical Research Recruitment and Outreach Support Service (CRROSS), a component of the Regulatory Support Core, is funded by the University’s CTSA program and staffed by a full-time Clinical Research Recruitment Specialist with previous experience in commercial or academic clinical trial recruitment, and a half-time Recruitment Assistant with research experience and/or training in psychology. Thus, in addition to oversight of the program by the Director of the Regulatory Support Core, it is staffed by 1.5 full-time equivalent positions. As CRROSS is both a service provider and a scholarly enterprise, staff members are selected based on their experience in clinical research recruitment and communications, their data capture and analysis skills, and their ability to develop positive working relationships with research investigators. CRROSS services are provided free of charge to research teams and CRROSS has a limited CTSA-supported budget that can be used for cost-sharing if an investigator has insufficient funds to support commercial graphics and advertising placement. CRROSS services Table 1 provides an overview of the services provided by CRROSS throughout the life of a protocol. Teams may request the comprehensive services described below or a more limited set of recruitment services such as creation of a single advertisement. The comprehensive services begin early in protocol development during study design in coordination with an interdisciplinary
Center for Clinical and Translational Science, The Rockefeller University, New York, New York, USA; 2Delete Blood Cancer DKMS, New York, New York, USA.
1
Correspondence: Rhonda G. Kost ([email protected]) DOI: 10.1111/cts.12240
WWW.CTSJOURNAL.COM
VOLUME 8 • ISSUE 2
91
Kost et al. Data-Rich Recruitment Core n
Stage of Protocol Life Cycle Protocol design phase
Protocol submission phase
Protocol conduct phase
Workflow and data capture
Recruitment services/ deliverables
Clinical Conductor®*
iRIS®†
Comprehensive recruitment consultation
Access planning resources
Teamwork, Navigation‡
• Feasibility/burdens/incentives • Eligibility criteria • Availability of target population • Protocol design • Resources/limitations for conduct • Timeline expectations • Overall/staged strategy
• Query Repository for target population availability • Review screen/enrollment data from similar studies • Campaign strategy performance • Comparability of compensation • Comparable study attrition
• Evolving protocol design accessible/transparent • Planned within interdisciplinary team
Preparation and submission of materials
Configuration for data capture
Protocol elements for submission
• Written recruitment plan details strategy/tactics/services • Data-justified enrollment timeline • Advertising text, graphics, venues • Prescreening scripts for call center
• Protocol listing (internal) • Ad campaigns/referral sources • Prescreen/inclusion/exclusion • Study team portal
• Written recruitment plan for IRB • Screening, enrollment, and accrual targets for IRB • Advertisement graphics/text • Time line for CTSA accountability
Provisions of services
Activities
Activities/Data capture in iRIS
• Advertising placement • Call center/call management • Repository queries • Data capture
• Query Repository for eligible enrollees • Access study information, prescreen scripts, inclusion/exclusion worksheets for recruitment workflow • Consent/enroll new callers into Repository • Prescreen callers and query-identified prospects for investigators’ studies • Refer or schedule eligible prescreens • Study team access through portal
• Schedule screening visits for team • Capture screening outcome (screen fail, signed consent, noshow) • Capture enrollment outcome (e.g., screened out, on study, withdrawn, completed)
ata capture D • All callers, name, contact, demographics, referral source • “Touch” (series of interactions related to a protocol) • Referral source • Eligibility data, specifics • Prescreen outcome • Reasons for ineligibility • Screening/enrollment outcome Performance Evaluation cycle
Prescreening, screening and enrollment outcomes rates of referral and enrollment
Advertising Efficiency Reports
Enrollment Efficiency
• Advertising effectiveness • Reporting to research teams • Revisions to strategy, materials
• Ad campaign reach, call volume • Diversity—race, gender, ethnicity • Relative success/ad or referral source • % prescreens referred • % prescreens ineligible, and reasons • Enrollment performance compared to proposed timeline • Recruitment progress reports to PI
• Time elapsed from recruitment start to first screening visit • Time elapsed from recruitment start to last participant enrolled • % referred that enrolled/ completed study
Clinical Conductor (Bio-Optronics, Rochester, NY, USA) is a software application designed to support participant recruitment in clinical research studies. iRIS (iMEDRIS, Redlands, CA, USA), is a software application designed to support IRB management, study management, and subject management; the Rockefeller CCTS instance of iRIS has been customized to support CRROSS recruitment and Research Protocol Navigation support for protocol development. ‡ Research Protocol Navigation.10 * †
Table 1. Overview of recruitment services provided by CRROSS throughout the life of a protocol across data capture platforms.
Research Protocol Navigation process,10 and continue through the IRB submission, study conduct, and completion phases of the protocol cycle. During protocol development, investigators are strongly encouraged to utilize the Comprehensive Recruitment Consultation as it provides the most detailed analysis and recommendations to enhance recruitment feasibility, culminating in a detailed written recruitment plan. During protocol conduct, 92
VOLUME 8 • ISSUE 2
CRROSS services include actively recruiting, prescreening, referring, and scheduling participants for studies. The individual elements of the Comprehensive Recruitment Consultation include: Feasibility assessment The likelihood of timely enrollment is reviewed with the investigator based on: (a) the availability of the target population WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
as assessed through published prevalence data and prior experiences of the researcher or CRROSS staff in enrolling comparable populations of participants to similar protocols; queries of the Research Volunteer Repository, ResearchMatch, or other relevant registries; and the existence of, or potential for, community or advocacy group partnerships. (b) The stringency of the inclusion and exclusion criteria and the degree to which they narrow the eligible population. (c) The human and material resources the team is prepared to commit to accessing the population. (d) Protocol-specific burdens, benefits, and incentives as likely perceived by prospective participants. Burdens may include the number and duration of study visits, the type and number of study procedures, the discomfort and potential risks associated with the procedures, and the inconveniences associated with participation (e.g., extended inpatient visits; interruptions to family, child care, and work; and difficult travel). Benefits and incentives may include improvement in health, access to novel therapies, free diagnosis and therapy, education about health and disease, fulfillment of altruistic motives, and emotional support. Compensation is considered relative to inconvenience and out-of-pocket costs, with reference to the level of compensation provided to participants for joining similar protocols at our and other institutions. Early discussions with investigators lead to revisions to the protocol design to enhance recruitment feasibility by minimizing barriers to eligibility and balancing benefits and incentives versus burdens, while still preserving both the scientific integrity of the research and its ethical framework. A time frame for enrollment is estimated by considering the sample size, the anticipated access to the target population, the likely ratio of screened-to-enrolled volunteers (when data are available), the availability of the research team for study visits (total time and flexibility), access to scarce resources (e.g., MRI or PET scanner time), and the investigator’s competing priorities. The investigators and CRROSS then agree on a timeline for recruitment based on these elements. Written plan preparation Following the initial consultation, the CRROSS staff creates a customized written recruitment plan and an advertising budget. Customized protocol-specific recruitment strategies are detailed, including plans for community engagement outreach (e.g., partnerships with advocacy groups), advertising venues (e.g., print, radio, TV, video, podcasts, internet listings, social networks, grassroots, and health events), proposed advertising text and graphics (for IRB submission), and plans for use of the centralized call management service and repository. The CRROSS Call Center CRROSS provides a centralized call management and telephone prescreening service for investigators. The flow of volunteers through the Call Center is shown in Figure 1. Advertisements refer callers to a branded hotline (1–800-RUCARES) that hosts general and protocol-specific voicemail boxes and a branded email box ([email protected]). Based on standard operating procedures designed to insure consistency, CRROSS staff attempt to recontact callers within 48 hours and capture basic demographic data and IRB-approved protocol-specific eligibility data according to a script developed by CRROSS personnel with the individual investigator. Investigators delegate the return of phone calls and conduct of telephone prescreening to the Call WWW.CTSJOURNAL.COM
Center. Research teams choose whether they want eligible and interested prescreened volunteers to be referred to them for further prescreening by telephone, or to be directly scheduled by the CRROSS staff for a screening appointment with the research team. To discourage participants from missing appointments without notification, and to maintain the quality of referrals, volunteers are explicitly counseled that missing an appointment without canceling in advance may jeopardize their ability to participate in the study and to be recruited for future studies. Research volunteer repository CRROSS created an IRB-approved Research Volunteer Repository in 2009 to enable recontact of volunteers for future research studies. All individuals who contact the Call Center are offered participation in the repository according to an IRB-approved script; verbal consent is sufficient for initial repository entry, but in-person written confirmation must be obtained when a volunteer enters a study on-site.The Rockefeller University IRB informed consent template contains language offering participants in all protocols the option of joining the repository. Individuals who have enrolled in the repository are thereafter included in any protocol-specific queries of the database for recruitment purposes and may be called and invited to future studies. In order for investigators to gain access to repository volunteers or conduct queries of the repository, their protocol must be formally associated with the Research Volunteer Repository protocol at the time of IRB review. Investigators must also agree to the Repository-defined procedures for interacting with Repositoryenrolled volunteers. CRROSS staff members conduct queries of the Repository database to identify potentially eligible volunteers, and then contact, prescreen, and refer interested eligible volunteers to the research team (Figure 1). Repository participants who have not been contacted within 6 months are asked by CRROSS staff upon recontact to verbally reconfirm their repository consent before any recruitment takes place. Volunteers may ask to be removed from the repository at any time; to date fewer than 1% have requested that their name be removed from the repository. Data infrastructure CRROSS staff use recruitment-dedicated software (Clinical Conductor, Bio-Optronics, Inc., Rochester, NY, USA; ACCESS, MicroSoft, Redmond, WA, USA) to host documents and templates and capture comprehensive data during the recruitment process. Protocol-specific data include: advertising campaign venue and content; expenditures; protocol-specific prescreen script and protocol-specific eligibility details; checklists designed with the research team; a record of every contact and every caller with whom the protocol is discussed; and prescreen outcomes (referred for screening, or not-eligible). Volunteer-specific data captured for every caller include caller’s demographics, contact information, referral source (e.g., specific advertising campaign, word of mouth, practitioner referral), general health and common medical information, and preferred mode of recontact; the outcome of the verbal consent discussion regarding future contact for recruitment (Repository enrollment); a record of every discussion for every protocol discussed with that caller; and the prescreen outcome. The total number of volunteer contacts, staff efforts to contact/recontact, call backs, and prescreen outcomes is captured to provide general operational metrics. The data are analyzed per-protocol to track the effectiveness of specific advertisement campaigns and prescreen scripts. VOLUME 8 • ISSUE 2
93
Kost et al. Data-Rich Recruitment Core n
Figure 1. Flow of volunteers and prescreening through CRROSS Call Center. Volunteers reach the CRROSS Call Center via several paths: (1) in response to specific advertisements inviting research participation in specific protocols or at Rockefeller in general, (2) as a result of spontaneous calls not prompted by specific advertising, (3) by referral to the Call Center after successful initial interactions hosted by the ResearchMatch portal, and (4) as Research Volunteer Repository members who call intermittently to inquire about new research opportunities. All volunteers calling the Call Center are managed according to a standard procedure designed to collect basic contact, referral source, and demographic information, ascertain whether they are new to CRROSS, offer enrollment into the Research Volunteer Repository, and determine for which protocols they are eligible for prescreening. These interactions and outcomes are all documented in a recruitment database (Clinical Conductor, Bio-Optronics, Rochester, NY, USA). All callers, regardless of Repository status are prescreened through a common procedure, assessing and documenting interest and eligibility through use of protocolspecific prescreening scripts and recruitment software. Blue lines indicate pathways moving toward eventual screening/enrollment; orange lines indicate pathways related to Research Volunteer Repository enrollment; and green lines indicate pathways related to queries of the Repository database to identify potentially eligible volunteers. Diamond shapes indicate decisions; rectangles with vertical lines indicate prespecified processes; rhombus shape indicates data; orange rounded rhombus indicates the Repository participant database.
After volunteers have been referred to research teams, the outcomes of subsequent screening, consenting, and study enrollment are captured by research teams in the subject/study management application (iRIS; IMEDRIS Corporation, Redlands, CA, USA). For each participant referred for screening, research team members assign a transient status (“scheduled,” “screening,” “enrolled”) that is ultimately replaced by a final outcome status (e.g., “no-show,” “screened out,” “dropped from study,” “completed study”). CRROSS staff members reconcile iRIS subject statuses for every CRROSS-referred participant catalogued in the recruitment Clinical Conductor database. Workflow for recruitment includes multiple components, including queries of the Repository, review of protocol documents, creation and use of screening scripts and templates, capture of demographics for screening and enrollment outcomes, participant scheduling, and evaluation of screening and enrollment outcomes. These tasks are accomplished through the use of multiple 94
VOLUME 8 • ISSUE 2
electronic platforms; data reconciliation is conducted by CRROSS staff. (Table 1) Evaluation
Data are used to rapidly provide CRROSS staff and investigators guidance on the effectiveness of the different recruitment tactics so that they can assess whether to modify the advertisements and/ or the protocol. The most valuable initial data for these purposes are the number of callers responding to an advertisement, the percentage of those callers who meet eligibility criteria based on call center prescreening, and the specific reasons why responding callers are ineligible or decide not to enroll. Information is captured to evaluate overall operations and to measure protocol-specific outcomes. Operational metrics include tallies of protocols and investigators receiving services, advertising campaigns created and placed, and teams utilizing cost sharing. Call Center operational metrics include the number of total caller WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
Number of protocols receiving services* Services provided by CRROSS
2009–2010
2010–2011
2011–2012
2012–2013
2013–2014†
Any CRROSS service
25
37
41
42
38
Comprehensive recruitment consultations
13
9
8
11
18
Creation/placement of advertisements
20
19
26
24
23
Cost sharing for advertising
8
9
12
12
20
Number of supported ad campaigns
22
26
28
25
22
Call center services/call management
13
28
30
31
29
Screening appointment management
9
12
14
17
17
Eligible to receive results of query of the repository for feasibility and recruitment purposes†
23
31
36
39
36
* †
Includes protocols new to the current year and protocols receiving ongoing services from prior year. Partial data, July to December 2013; Investigators who do not seek IRB approval to link to the repository are not eligible for queries of the repository.
Table 2. CRROSS Delivery of services from 2009 to 2013.
judge the value of advertising content and venue and to inform changes to recruitment approach. Results
Utilization Table 2 provides information on CRROSS services provided to investigators from 2009 to 2013. During this period, the CRROSS Call Center managed 21,980 calls to and from new and returning volunteers. Of the 4,047 unique new callers who registered contact information with the Call Center hotline, 3,722 (92%) agreed to enroll in the Volunteer Repository (Figure 2). The racial and ethnic characteristics of the Research Volunteer Repository population Figure 2. Repository Growth by Calendar Year. Repository members are individuals who have consented to enroll in the Research Volunteer Repository, conducted under an IRB approval protocol. Enrollees agree to approximate those of New York City in 2010 be included in queries of the database and to be called in the future to learn about new research participation as judged by the 2010 US Census11 (Table 3). opportunities. The ages of Repository members range from 8 to 94 years old, with 50–60-year-olds contacts and the number and percentage of new callers enrolling making up the largest group (26%), followed by 40–50-yearinto the Repository. For all new protocol-specific requests to olds (21%), 30–40-year-olds (18%), and 20–30-year-olds, and query the Repository, staff members verify the IRB-approved 60–70-year-olds (each at 14%). association of the protocol to the repository prior to conducting the query. The repository is audited on an annual basis to assure compliance. Outcomes—overall Outcomes of recruitment and enrollment service There are three major paths by which volunteers make contact effectiveness are also analyzed, including the demographics with the CRROSS Call Center: (1) calls made by new callers who of enrolling participants, the percentage of volunteers passing respond to advertising or a referral by a friend, family, or medical prescreening and referred for screening/enrollment, the professional. (2) Calls made by Repository members who call the percentage of referred volunteers who actually enroll in the CRROSS team either spontaneously or in response to advertising. study, and the percentages of enrolled volunteers who remain (3) Calls made by CRROSS staff to repository members identified in the study or have completed the study. Additional outcomes through queries of the database who appear to be eligible for a that are evaluated are the times elapsed from the initiation of specific study. CRROSS recruitment efforts until the first participant enrollment From 2009 to 2013, CRROSS completed a total of 9,244 and until study enrollment is completed. Advertising costs are prescreening interviews. Of those, 4,047 prescreen conversations also tracked per protocol, per ad campaign, and per enrolled concerned recruitment of new callers to the Repository, and participant. Participants who fail to keep appointments for initial 92% of these callers (3,722 participants) agreed to enroll. The screening visits are tracked as “no-shows” in the Repository. remaining 5,197 prescreen conversations were conducted to Referral source, enrollment, and no-show data are also used to recruit participants to research studies. These prescreens included WWW.CTSJOURNAL.COM
VOLUME 8 • ISSUE 2
95
Kost et al. Data-Rich Recruitment Core n
White
Black or African American
Asian
Native American or Alaskan Native
Native Hawaiian or Pacific Islander
Some other race
Two or more races
Declined or not specified
New York City Population 2010 Census
33%
23%
13%
0.2%
–
0.7%
2%
–
Hispanic 29%
Research Volunteer Repository
35%
39%
4%
1%
1%
9%
4%
7%
20%
Table 3. Race and ethnicity demographics of CRROSS database population and New York City.11
Prescreened by CRROSS
Referred
“No-shows” 1st visit
Percent enrolled of volunteers who complete a screening visit†
Signed consent
Retained (%)
Percent enrolled of all volunteers referred*
1,468
1,255 (85)
43%
76%
–
280
257 (92)
49%
–
311
170
124 (73)
33%
55%
Screened
Volunteers directly calling or recalling the Call Center, 2009–2013 5,197
3,378
1081
1,943
Volunteers identified through queries of the Repository, 2009–2013 567
–
–
Example: Volunteers identified for specific study
13
785
512
185
Number of volunteers who signed consent/total number of volunteers referred to teams, including “no-shows.” † Number of volunteers who signed consent/total number of volunteers who kept their screening visit appointment. *
Table 4. Enrollment Progression of Volunteers Referred by CRROSS, from Prescreen through study enrollment/completion, by calendar year. Volunteers directly calling the Call Center who pass telephone prescreening for a specific protocol are referred to research teams for screening in person. Some research teams further prescreen referrals by telephone before selecting volunteers for screening visits. Participants who fail to appear for an in-person screening appointment without cancelling or rescheduling are classified as “no-shows.” Participants who complete screening with the research team and then sign informed consent are considered “enrolled.” “Retained” participants are those who have completed, or still remain enrolled in the study at the time of analysis.
new callers, returning callers, and the participants contacted as a result of Repository queries, and yielded 3,378 referrals (2,803 unique individuals) resulting in 1,468 (52%) study enrollments. Eighty-five percent of these enrollees (1,255) completed or still remained in the study at the time of the analysis. The Research Volunteer Repository was an important source of research participant recruitment. Queries conducted on behalf of 51 separate studies identified 1,137 potentially eligible volunteers. Half of those volunteers (567) were able to be reached and prescreened, and 280 enrolled in protocols. Most importantly, 92% of participants who enrolled through queries from the Repository completed the study or remained on-study at the time of analysis. Of the studies serviced by queries, 2 were NIH-sponsored, 8 were sponsored by industry, and 41 were investigator-initiated, reflecting the Rockefeller CCTS research portfolio. Of the 3,722 unique individuals enrolled in the Repository, 853 individuals (23%) accounted for 1,258 enrollments into research protocols, of which 1,119 (89%) were retained. Overall, 86% (1,258/1,468) of all enrollments into studies managed by CRROSS involved individuals who were already enrolled in the Repository or who joined the Repository as part of their recruitment into a study. Outcomes—efficiency Enrollment outcome data on all volunteers who made contact with CRROSS from 2009 to 2013 for all studies are shown in 96
VOLUME 8 • ISSUE 2
Table 4. A leaky pipe model12 is often used to graphically illustrate the attrition of volunteers at each step from initial recruitment through completion of study enrollment. Data for a specific study13 using CRROSS services are also included in Table 4 to illustrate the data that are shared with investigators on an ongoing basis during the recruitment phase. The time elapsed in days from the date of initiation of CRROSS recruitment activities until the date the first participant enrolled in the protocol is shown in Figure 3. For the years 2009–2013, in aggregate, the median time-to-first-participant-enrollment was 10 days. For comparison, in the years 2006–2008, which were prior to institution of CRROSS services, the median time elapsed from the date of IRB approval (taken as a surrogate for the when the PI’s recruitment efforts were likely to have started) until the first participant visit was 70 days. Using data to improve recruitment Based on the data detailing the reasons for callers’ ineligibility or lack of interest in a given protocol, CRROSS personnel and the investigators decide whether to consider changes in advertising, eligibility criteria, and/or protocol design. For example, when it was observed that many callers were ineligible to enroll in a study because their body mass index was below the eligibility cut off value, CRROSS staff worked with the investigators to modify the wording of the advertisement. Data collected after the advertisement was modified showed WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
Financial model CRROSS consultative services and recruitment activities were provided free to investigators. Many advertising venues incur no costs to investigators (e.g., Craigslist, Facebook, ResearchMatch, ClinicalTrials.gov). When paid advertising was necessary, CRROSS provided costsharing for advertising (jointly supported by the University and the CCTS) to trainees and to investigators using CRROSS services who had expended other recruitment resources. In 2013, CRROSS provided a total of $25,000 in cost-sharing to support 50%–100% of advertising costs for 20 of 42 protocols served. User satisfaction CRROSS deployed a user satisfaction survey in 2014 to all investigators and coordinators who used CRROSS services in the prior year. Of CRROSS service users, 70% used comprehensive CRROSS services at consultation, 14% utilized intermediate services, and 16% used a single CRROSS service. More than 90% of the 40 respondents were “very satisfied” or “satisfied” with the services they received. Two users expressed dissatisfaction that Rockefeller does not have a large patient population as might be found in an academic medical center. Figure 3. Time elapsed from the initiation of active recruitment by CRROSS until the first participant visit (days). Protocols are grouped according to the calendar year in which CRROSS recruitment activities were initiated, even if the first participant was not screened until the following year. For the years 2009–2013, data are included for 11, 14, 18, 13, and 14 protocols respectively. For the year 2012, one outlier value is indicated by “*”; the value, 473 days, was included in the calculations for the median shown, but was omitted from the graph visualization to preserve resolution.
that a higher percentage of respondents to the advertisement were now eligible for the study (Figure 4). Similar interim reports of the outcomes of recruitment efforts have been used by investigators to modify their approach and improve recruitment effectiveness. Other factors that affect timely recruitment Investigator-specific challenges can have a negative impact on timely recruitment, including lack of readiness to conduct laboratory or clinical assessments, limited time availability to see participants, protocol modifications after initial approval, addition of new eligibility criteria, or overreliance on convenience sampling or colleague referrals for recruitment. We observed that studies involving normal volunteers tended to have higher “no-show” rates, sometimes exceeding 30%, despite appointment reminder calls and confirmations. Some research teams informally suspended recruitment activities to prioritize other activities, thus prolonging the recruitment period. In response to these findings, CRROSS began to proactively attempt to identify during the initiation consultation any factors that might later introduce recruitment delays and to then incorporate this information into assessment of enrollment feasibility and the proposed timeline for completing recruitment. WWW.CTSJOURNAL.COM
Discussion
Using resources made available through the CTSA program and The Rockefeller University, we established a centralized, core research participant recruitment infrastructure offering expertise and services to investigators to maximize enrollment of individuals into clinical research protocols. We designed the core and its procedures to incorporate recruitment expertise early into the protocol development process,10 and to incorporate data capture and timely analysis, so that rapid changes could be made in the advertising or in the protocol to improve recruitment. We also created a Research Volunteer Repository to provide an evergrowing database of participants willing to participate in research protocols. Utilization of the recruitment core by research teams has grown steadily, and overall feedback about the program has been very positive. The ability to provide reports detailing the effort and activities required to successfully recruit participants educates investigators about the requirements for effective recruitment, informs planning of subsequent protocols, and demonstrates the value added by using CRROSS services. Collecting detailed data on the recruitment process also enabled us to assess the effectiveness of the different approaches to recruitment. The Research Volunteer Repository has been an important addition to our recruitment efforts, with 23% of the individuals in the Repository enrolling in at least one protocol, with 89% retention/completion. In addition, many Repository members spontaneously check with the recruitment core periodically to assess whether they VOLUME 8 • ISSUE 2
97
Kost et al. Data-Rich Recruitment Core n
We realize, however, that our experience may not be generalizable because of the nature of our institution and the population we serve. The ultimate goals of successful recruitment are timely study accrual and completion. There are, however, no consensus standards for meaningful measures of timely accrual. Studies vary in their duration, target population size and availability, and the stringency of eligibility criteria. Our near-term measure, the elapsed time from initiation of recruitment efforts to the first participant enrollment, is a median of 10 days for protocols receiving CRROSS services 2009–2013. We do not have a precise baseline comparator, since prior to the creation of CRROSS the date of initiation of recruitment was not recorded. However, for protocols starting in 2006–2008, the median time elapsed from IRB approval to Figure 4. CRROSS Call Center Outcomes for initial and amended advertisement. CRROSS staff collect first participant enrollment was 70 days. By aggregate prescreening outcome data (eligible vs. ineligible) as well as the specific reasons for ineligibility (e.g., failure to meet individual eligibility criteria, not interested, compensation too low) in order to rapidly suggest modithis measure, CRROSS services have had an fications in advertising. Data are shown for study MAP-0683, in which overweight individuals with low vitamin D important impact on speeding the initiation levels were recruited to study the effects of vitamin D repletion on serum cholesterol levels. The headline of the of enrollment. advertisement was: “Are you worried about your cholesterol?” Overall, 68% of respondents were ineligible, 35% due to a body mass index (BMI) below the study threshold. The advertisement was then amended to attract We recognize the limitations of volunteers with a higher BMI. The amended advertisement headline was: “Are you overweight? Are you worried extrapolating from our experience. In about your cholesterol?” After running the amended advertisement, the proportion of prescreened callers who particular, our institution’s research protocol were ineligible decreased to 47%, and the proportion of callers ineligible due to BMI criteria decreased to 22%. portfolio is relatively small compared to large academic medical centers. As a result our model may not be readily scalable in a large are eligible for new protocols. When the Repository has been institution. The major costs associated with expanding services queried to recruit for specific protocols, it has resulted in 50% would include those of additional expert recruitment personnel enrollment of those contacted, with 92% retention. Data capture and the software licensing costs to expand data platform, track and real-time analysis allow us to assess the most productive participants in the Repository, and enable data analysis. Since advertising venues for specific populations and specific protocols, we have found it to be important for recruitment personnel to and enable rapid modification of advertising campaigns in real be familiar with all of the protocols they offer, it may be better time to improve recruitment. It also allows us to match costs of to create additional recruitment teams, for example, to meet the advertising with outcomes, producing crucial cost-effectiveness needs of a department, rather continuing to add personnel to a analyses. As we have obtained more data, we have also been able single team when expanding services. Nonetheless, we believe to judge better how many prescreening interactions by the Call the central principles that define our program are generalizable, Center and screening visits by the research team will likely be namely, providing a range of centralized expert recruitment needed to achieve the desired enrollment. Thus, our feasibility services, hosting a call center to refer participants to the studies assessments have continued to improve. in which they are most interested, and most importantly, rapidly A number of patient/participant registries have been collecting robust data for analysis throughout the protocol life developed at research centers around the country based on cycle to inform performance improvement. participants providing consent to be contacted in the future for The lack of broadly agreed upon precisely defined criteria for recruitment or enrolling themselves in Web-based registries standardized measurements of recruitment success, including (unpublished proceedings of 5th Annual CRM workshop, New study enrollment, retention, and completion is a serious Haven, CT, USA, June 4–5, 2012).5–9 There are, however, few impediment in comparing different approaches to participant reports of the impact of these registries on timely enrollment recruitment. We hope that by reporting our experience we can and study completion, and so the cost/benefit is uncertain. encourage others to join us in trying to develop such metrics. Other approaches to speedy recruitment include aggregation Some of our studies had very long times to first enrollment. In of data from electronic health records based on consent waivers virtually all cases, the delay was due to factors beyond the recruitment or opt-out mechanisms; two major initiatives in 2014 are the program’s control. By analyzing these reasons systematically, PCOR-NET14–16 and CTSA-based data network.17 It remains to however, we have been able to proactively refine our assessment be established which approach(es) are most effective. Since our of feasibility. We also recognize the importance of ultimately institution is not part of an academic health center with a large focusing on study completion rather than study initiation, and we patient population, we do not have the opportunity to create a are currently directing our efforts to maximizing these crucial goals. general patient population registry for recruitment.18 Instead, we One of the key lessons that we learned is that the presentation created a Research Volunteer Repository in which each of the of data to scientific investigators is the persuasive form of enrollees has explicitly expressed interest in research participation. communication. Much has written about the importance of 98
VOLUME 8 • ISSUE 2
WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
having a learning health care system.19,20 We think our approach is an important step forward in developing a learning clinical research system. Our experience using different database formats has also been instructional. We initially used an in-house ACCESS database coupled to additional REDCap data capture tools. While this enabled us to capture the requisite data, extracting and analyzing data was laborious and slow. In 2013, we migrated our dataset to a software platform designed specifically to manage research participant recruitment (Clinical Conductor, Bio-Optronics, Inc.). This greatly simplified data analysis, reducing days of data aggregation to minutes of reporting. It also afforded integration of multiple aspects of recruitment tracking, including advertising campaign costs and effectiveness that were previously managed separately. We conclude that centralizing recruitment activities within a core structure staffed by expert personnel who are equipped to perform comprehensive data capture and analysis, speeds participant recruitment and enrollment. It also spares investigators from diverting their energies to an area in which few have expertise, standardizes the approach, increases the efficient use of scarce resources, and insures compliance with regulatory requirements for advertising and informed consent. Most important, it makes it possible to rigorously collect data as the basis for informing current practice and improving future practice. Acknowledgments
This project is supported in part by grant 8 UL1 TR000043 from the National Center for Research Resources (NCRR) and the National Center for Advancing Sciences (NCATS), National Institutes of Health. Conflict of Interest
The authors have no conflicts of interest to disclose. Human Subjects
The Research Volunteer Repository is conducted under a protocol approved by the Rockefeller University Institutional Review Board (IRB) according to applicable regulations. The CRROSS recruitment core verifies IRB approval of each protocol before conducting any recruitment activities for the protocol.
WWW.CTSJOURNAL.COM
References
1. Translation. IoMUFoDDDa. Transforming Clinical Reseach in the United States: Challenges and Opportunities: Workshop Summary. National Academies Press (US); 2010. Available at: http://www.ncbi.nlm.nih.gov/books/NBK50892/. Paper presented at: National Academies Press (US); 2010, 2010; Washington (DC). 2. Kitterman DR, Cheng SK, Dilts DM, Orwoll ES. The prevalence and economic impact of lowenrolling clinical studies at an academic medical center. Acad Med. 2011; 86(11): 1360–1366. 3. Anderson DL. A Guide to Patient Recruitment and Retention. Boston, Mass: Thomson Center Watch; 2004. 4. Kost RG, Mervin-Blake S, Hallarn R, Rathmann C, Kolb HR, Himmelfarb CD, D’Agostino T, Rubinstein EP, Dozier AM, Schuff KG, et al. Accrual and recruitment practices at the CTSAs: a call for expectations, expertise and evaluation. Acad Med. 2014: 1–10. 5. Washington University School of Medicine Research Participant Registry. Available at: https:// vfh.wustl.edu/. Accessed March 19, 2014. 6. University of Pittsburgh. Research Participant Registry. Available at: https://www.researchregistry. pitt.edu/. Accessed October 27, 2014. 7. Rigers JL, Johnson TR, Smith YR. Recruitment of women research participnts: the women’s health registry at the University of Michigan. J Womens Health. 2007; 16(5): 721–728. 8. ResearchMatch. Available at: www.researchmatch.org. Accessed October 27, 2014. 9. UCSF CTSI. Research Participant Registry. 2012. Available at: https://ucsfregistry.secure.force. com/. Accessed October 27, 2014. 10. Brassil D, Kost RG, Hurley A, Dowd K, Stanwix J, Coller BS. The Rockefeller University Navigation Program: a structured multidisciplinary protocol development and educational program to advance translational research. Clin Transl Sci. 2014; 7: 12–19. 11. NYC.gov. NYC Census. 2010; Available at: http://www.nyc.gov/html/dcp/html/census/ census_2010.shtml. Accessed July 14, 2014. 12. CPP Inc. Available at: http://www.drugdevelopment-technology.com/contractors/consulting/ cpp/cpp2.html. Accessed June 12, 2014. 13. Ponda MP, Dowd K, Finkielstein D, Holt PR, Breslow JL. The short-term effects of vitamin D repletion on cholesterol: a randomized, placebo-controlled trial. Arterioscler Thromb Vasc Biol. Oct 2012; 32(10): 2510–2515. 14. Kaushal R, Hripcsak G, Ascheim DD, Bloom T, Campion TR Jr, Caplan AL, Currie BP, Check T, Deland EL, Gourevitch MN, et al. Changing the research landscape: the New York City Clinical Data Research Network. J Am Med Inform Assoc. Jul 2014; 21(4): 587–590. 15. Kaushal R, Blumenthal D. Introduction and commentary for special issue on health information technology. Health Serv Res. Feb 2014; 49(1 Pt 2): 319–324. 16. Kern LM, Edwards A, Kaushal R. The patient-centered medical home, electronic health records, and quality of care. Ann Intern Med. Jun 3 2014; 160(11): 741–749. 17. Nadler L, Reis S, Toto R, Firestein G. CTSA Website, Proceedings of 2014 CTSA PI Retreat: Participant Accrual. 2014; Description of evolving CTSA patient-data-network. Available at: https: //www.ctsacentral.org/sites/default/files/documents/%233_Participant%20Accrual_Nadler_ Reis_Toto_Firestein.pdf. Accessed July 24, 2014. 18. Embi PJ, Jain A, Clark J, Harris CM. Development of an electronic health record-based clinical trial alert system to enhance recruitment at the point of care. Paper presented at: AMIA2005. 19. The CTSA Program at NIH: Opportunities for Advancing Clinical and Translational Research: The National Academies Press, Washington, D.C.; 2013. 20. Observational Studies in a Learning Health System: Workshop Summary: The National Academies Press, Washington, D.C.; 2013.
VOLUME 8 • ISSUE 2
99
Copyright of CTS: Clinical & Translational Science is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Keywords: enrollment, accrual, clinical trial management, clinical research management Introduction
Timely enrollment of participants into clinical studies is a major challenge nationally, with as many as 75% of investigators failing to enroll the target number of subjects, and 90% of trials failing to enroll the requisite number of subjects within the proposed time period.1 In addition to the missed opportunities for scientific and medical advances, underenrollment also has economic consequences. For example, one academic medical center estimated their costs of underenrollment as exceeding one million dollars in a single year.2 Recruitment to industry-sponsored clinical trials conducted outside of academic centers commonly utilizes recruitment centers offering professional marketing strategies.3 In sharp contrast, recruitment to clinical trials and investigator-initiated studies at academic centers has historically been left to the research team, which commonly has limited recruitment experience and competing priorities.2 Academic institutions supported by Clinical and Translation Science Awards (CTSAs) have begun to offer participant registries and other recruitment resources to support enrollment of clinical protocols,4–9 but few have published the details of the organization or impact of these resources. To address the need for better support of recruitment, and to study the recruitment process itself, the Rockefeller University Center for Clinical and Translational Science (CCTS), committed CTSA resources to create a recruitment core with three specific goals: (1) to provide recruitment expertise early in protocol development in order to judge feasibility and minimize potential barriers; (2) to provide centralized recruitment services and resources throughout the protocol to insure expert execution of the recruitment strategy and to recruit participants; and (3) to capture and analyze recruitment-related outcome data in real time to inform modifications in the recruitment strategy or the protocol to optimize accrual. These data also form the basis for advancing the science of recruitment in the academic setting, defining best practices, and supporting continuous performance improvement. In this report, we describe our recruitment core
services, their utilization, and their effectiveness in improving timely participant recruitment and accrual. Methods
The Clinical Research Recruitment and Outreach Support Service The Rockefeller University Clinical Research Recruitment and Outreach Support Service (CRROSS), a component of the Regulatory Support Core, is funded by the University’s CTSA program and staffed by a full-time Clinical Research Recruitment Specialist with previous experience in commercial or academic clinical trial recruitment, and a half-time Recruitment Assistant with research experience and/or training in psychology. Thus, in addition to oversight of the program by the Director of the Regulatory Support Core, it is staffed by 1.5 full-time equivalent positions. As CRROSS is both a service provider and a scholarly enterprise, staff members are selected based on their experience in clinical research recruitment and communications, their data capture and analysis skills, and their ability to develop positive working relationships with research investigators. CRROSS services are provided free of charge to research teams and CRROSS has a limited CTSA-supported budget that can be used for cost-sharing if an investigator has insufficient funds to support commercial graphics and advertising placement. CRROSS services Table 1 provides an overview of the services provided by CRROSS throughout the life of a protocol. Teams may request the comprehensive services described below or a more limited set of recruitment services such as creation of a single advertisement. The comprehensive services begin early in protocol development during study design in coordination with an interdisciplinary
Center for Clinical and Translational Science, The Rockefeller University, New York, New York, USA; 2Delete Blood Cancer DKMS, New York, New York, USA.
1
Correspondence: Rhonda G. Kost ([email protected]) DOI: 10.1111/cts.12240
WWW.CTSJOURNAL.COM
VOLUME 8 • ISSUE 2
91
Kost et al. Data-Rich Recruitment Core n
Stage of Protocol Life Cycle Protocol design phase
Protocol submission phase
Protocol conduct phase
Workflow and data capture
Recruitment services/ deliverables
Clinical Conductor®*
iRIS®†
Comprehensive recruitment consultation
Access planning resources
Teamwork, Navigation‡
• Feasibility/burdens/incentives • Eligibility criteria • Availability of target population • Protocol design • Resources/limitations for conduct • Timeline expectations • Overall/staged strategy
• Query Repository for target population availability • Review screen/enrollment data from similar studies • Campaign strategy performance • Comparability of compensation • Comparable study attrition
• Evolving protocol design accessible/transparent • Planned within interdisciplinary team
Preparation and submission of materials
Configuration for data capture
Protocol elements for submission
• Written recruitment plan details strategy/tactics/services • Data-justified enrollment timeline • Advertising text, graphics, venues • Prescreening scripts for call center
• Protocol listing (internal) • Ad campaigns/referral sources • Prescreen/inclusion/exclusion • Study team portal
• Written recruitment plan for IRB • Screening, enrollment, and accrual targets for IRB • Advertisement graphics/text • Time line for CTSA accountability
Provisions of services
Activities
Activities/Data capture in iRIS
• Advertising placement • Call center/call management • Repository queries • Data capture
• Query Repository for eligible enrollees • Access study information, prescreen scripts, inclusion/exclusion worksheets for recruitment workflow • Consent/enroll new callers into Repository • Prescreen callers and query-identified prospects for investigators’ studies • Refer or schedule eligible prescreens • Study team access through portal
• Schedule screening visits for team • Capture screening outcome (screen fail, signed consent, noshow) • Capture enrollment outcome (e.g., screened out, on study, withdrawn, completed)
ata capture D • All callers, name, contact, demographics, referral source • “Touch” (series of interactions related to a protocol) • Referral source • Eligibility data, specifics • Prescreen outcome • Reasons for ineligibility • Screening/enrollment outcome Performance Evaluation cycle
Prescreening, screening and enrollment outcomes rates of referral and enrollment
Advertising Efficiency Reports
Enrollment Efficiency
• Advertising effectiveness • Reporting to research teams • Revisions to strategy, materials
• Ad campaign reach, call volume • Diversity—race, gender, ethnicity • Relative success/ad or referral source • % prescreens referred • % prescreens ineligible, and reasons • Enrollment performance compared to proposed timeline • Recruitment progress reports to PI
• Time elapsed from recruitment start to first screening visit • Time elapsed from recruitment start to last participant enrolled • % referred that enrolled/ completed study
Clinical Conductor (Bio-Optronics, Rochester, NY, USA) is a software application designed to support participant recruitment in clinical research studies. iRIS (iMEDRIS, Redlands, CA, USA), is a software application designed to support IRB management, study management, and subject management; the Rockefeller CCTS instance of iRIS has been customized to support CRROSS recruitment and Research Protocol Navigation support for protocol development. ‡ Research Protocol Navigation.10 * †
Table 1. Overview of recruitment services provided by CRROSS throughout the life of a protocol across data capture platforms.
Research Protocol Navigation process,10 and continue through the IRB submission, study conduct, and completion phases of the protocol cycle. During protocol development, investigators are strongly encouraged to utilize the Comprehensive Recruitment Consultation as it provides the most detailed analysis and recommendations to enhance recruitment feasibility, culminating in a detailed written recruitment plan. During protocol conduct, 92
VOLUME 8 • ISSUE 2
CRROSS services include actively recruiting, prescreening, referring, and scheduling participants for studies. The individual elements of the Comprehensive Recruitment Consultation include: Feasibility assessment The likelihood of timely enrollment is reviewed with the investigator based on: (a) the availability of the target population WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
as assessed through published prevalence data and prior experiences of the researcher or CRROSS staff in enrolling comparable populations of participants to similar protocols; queries of the Research Volunteer Repository, ResearchMatch, or other relevant registries; and the existence of, or potential for, community or advocacy group partnerships. (b) The stringency of the inclusion and exclusion criteria and the degree to which they narrow the eligible population. (c) The human and material resources the team is prepared to commit to accessing the population. (d) Protocol-specific burdens, benefits, and incentives as likely perceived by prospective participants. Burdens may include the number and duration of study visits, the type and number of study procedures, the discomfort and potential risks associated with the procedures, and the inconveniences associated with participation (e.g., extended inpatient visits; interruptions to family, child care, and work; and difficult travel). Benefits and incentives may include improvement in health, access to novel therapies, free diagnosis and therapy, education about health and disease, fulfillment of altruistic motives, and emotional support. Compensation is considered relative to inconvenience and out-of-pocket costs, with reference to the level of compensation provided to participants for joining similar protocols at our and other institutions. Early discussions with investigators lead to revisions to the protocol design to enhance recruitment feasibility by minimizing barriers to eligibility and balancing benefits and incentives versus burdens, while still preserving both the scientific integrity of the research and its ethical framework. A time frame for enrollment is estimated by considering the sample size, the anticipated access to the target population, the likely ratio of screened-to-enrolled volunteers (when data are available), the availability of the research team for study visits (total time and flexibility), access to scarce resources (e.g., MRI or PET scanner time), and the investigator’s competing priorities. The investigators and CRROSS then agree on a timeline for recruitment based on these elements. Written plan preparation Following the initial consultation, the CRROSS staff creates a customized written recruitment plan and an advertising budget. Customized protocol-specific recruitment strategies are detailed, including plans for community engagement outreach (e.g., partnerships with advocacy groups), advertising venues (e.g., print, radio, TV, video, podcasts, internet listings, social networks, grassroots, and health events), proposed advertising text and graphics (for IRB submission), and plans for use of the centralized call management service and repository. The CRROSS Call Center CRROSS provides a centralized call management and telephone prescreening service for investigators. The flow of volunteers through the Call Center is shown in Figure 1. Advertisements refer callers to a branded hotline (1–800-RUCARES) that hosts general and protocol-specific voicemail boxes and a branded email box ([email protected]). Based on standard operating procedures designed to insure consistency, CRROSS staff attempt to recontact callers within 48 hours and capture basic demographic data and IRB-approved protocol-specific eligibility data according to a script developed by CRROSS personnel with the individual investigator. Investigators delegate the return of phone calls and conduct of telephone prescreening to the Call WWW.CTSJOURNAL.COM
Center. Research teams choose whether they want eligible and interested prescreened volunteers to be referred to them for further prescreening by telephone, or to be directly scheduled by the CRROSS staff for a screening appointment with the research team. To discourage participants from missing appointments without notification, and to maintain the quality of referrals, volunteers are explicitly counseled that missing an appointment without canceling in advance may jeopardize their ability to participate in the study and to be recruited for future studies. Research volunteer repository CRROSS created an IRB-approved Research Volunteer Repository in 2009 to enable recontact of volunteers for future research studies. All individuals who contact the Call Center are offered participation in the repository according to an IRB-approved script; verbal consent is sufficient for initial repository entry, but in-person written confirmation must be obtained when a volunteer enters a study on-site.The Rockefeller University IRB informed consent template contains language offering participants in all protocols the option of joining the repository. Individuals who have enrolled in the repository are thereafter included in any protocol-specific queries of the database for recruitment purposes and may be called and invited to future studies. In order for investigators to gain access to repository volunteers or conduct queries of the repository, their protocol must be formally associated with the Research Volunteer Repository protocol at the time of IRB review. Investigators must also agree to the Repository-defined procedures for interacting with Repositoryenrolled volunteers. CRROSS staff members conduct queries of the Repository database to identify potentially eligible volunteers, and then contact, prescreen, and refer interested eligible volunteers to the research team (Figure 1). Repository participants who have not been contacted within 6 months are asked by CRROSS staff upon recontact to verbally reconfirm their repository consent before any recruitment takes place. Volunteers may ask to be removed from the repository at any time; to date fewer than 1% have requested that their name be removed from the repository. Data infrastructure CRROSS staff use recruitment-dedicated software (Clinical Conductor, Bio-Optronics, Inc., Rochester, NY, USA; ACCESS, MicroSoft, Redmond, WA, USA) to host documents and templates and capture comprehensive data during the recruitment process. Protocol-specific data include: advertising campaign venue and content; expenditures; protocol-specific prescreen script and protocol-specific eligibility details; checklists designed with the research team; a record of every contact and every caller with whom the protocol is discussed; and prescreen outcomes (referred for screening, or not-eligible). Volunteer-specific data captured for every caller include caller’s demographics, contact information, referral source (e.g., specific advertising campaign, word of mouth, practitioner referral), general health and common medical information, and preferred mode of recontact; the outcome of the verbal consent discussion regarding future contact for recruitment (Repository enrollment); a record of every discussion for every protocol discussed with that caller; and the prescreen outcome. The total number of volunteer contacts, staff efforts to contact/recontact, call backs, and prescreen outcomes is captured to provide general operational metrics. The data are analyzed per-protocol to track the effectiveness of specific advertisement campaigns and prescreen scripts. VOLUME 8 • ISSUE 2
93
Kost et al. Data-Rich Recruitment Core n
Figure 1. Flow of volunteers and prescreening through CRROSS Call Center. Volunteers reach the CRROSS Call Center via several paths: (1) in response to specific advertisements inviting research participation in specific protocols or at Rockefeller in general, (2) as a result of spontaneous calls not prompted by specific advertising, (3) by referral to the Call Center after successful initial interactions hosted by the ResearchMatch portal, and (4) as Research Volunteer Repository members who call intermittently to inquire about new research opportunities. All volunteers calling the Call Center are managed according to a standard procedure designed to collect basic contact, referral source, and demographic information, ascertain whether they are new to CRROSS, offer enrollment into the Research Volunteer Repository, and determine for which protocols they are eligible for prescreening. These interactions and outcomes are all documented in a recruitment database (Clinical Conductor, Bio-Optronics, Rochester, NY, USA). All callers, regardless of Repository status are prescreened through a common procedure, assessing and documenting interest and eligibility through use of protocolspecific prescreening scripts and recruitment software. Blue lines indicate pathways moving toward eventual screening/enrollment; orange lines indicate pathways related to Research Volunteer Repository enrollment; and green lines indicate pathways related to queries of the Repository database to identify potentially eligible volunteers. Diamond shapes indicate decisions; rectangles with vertical lines indicate prespecified processes; rhombus shape indicates data; orange rounded rhombus indicates the Repository participant database.
After volunteers have been referred to research teams, the outcomes of subsequent screening, consenting, and study enrollment are captured by research teams in the subject/study management application (iRIS; IMEDRIS Corporation, Redlands, CA, USA). For each participant referred for screening, research team members assign a transient status (“scheduled,” “screening,” “enrolled”) that is ultimately replaced by a final outcome status (e.g., “no-show,” “screened out,” “dropped from study,” “completed study”). CRROSS staff members reconcile iRIS subject statuses for every CRROSS-referred participant catalogued in the recruitment Clinical Conductor database. Workflow for recruitment includes multiple components, including queries of the Repository, review of protocol documents, creation and use of screening scripts and templates, capture of demographics for screening and enrollment outcomes, participant scheduling, and evaluation of screening and enrollment outcomes. These tasks are accomplished through the use of multiple 94
VOLUME 8 • ISSUE 2
electronic platforms; data reconciliation is conducted by CRROSS staff. (Table 1) Evaluation
Data are used to rapidly provide CRROSS staff and investigators guidance on the effectiveness of the different recruitment tactics so that they can assess whether to modify the advertisements and/ or the protocol. The most valuable initial data for these purposes are the number of callers responding to an advertisement, the percentage of those callers who meet eligibility criteria based on call center prescreening, and the specific reasons why responding callers are ineligible or decide not to enroll. Information is captured to evaluate overall operations and to measure protocol-specific outcomes. Operational metrics include tallies of protocols and investigators receiving services, advertising campaigns created and placed, and teams utilizing cost sharing. Call Center operational metrics include the number of total caller WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
Number of protocols receiving services* Services provided by CRROSS
2009–2010
2010–2011
2011–2012
2012–2013
2013–2014†
Any CRROSS service
25
37
41
42
38
Comprehensive recruitment consultations
13
9
8
11
18
Creation/placement of advertisements
20
19
26
24
23
Cost sharing for advertising
8
9
12
12
20
Number of supported ad campaigns
22
26
28
25
22
Call center services/call management
13
28
30
31
29
Screening appointment management
9
12
14
17
17
Eligible to receive results of query of the repository for feasibility and recruitment purposes†
23
31
36
39
36
* †
Includes protocols new to the current year and protocols receiving ongoing services from prior year. Partial data, July to December 2013; Investigators who do not seek IRB approval to link to the repository are not eligible for queries of the repository.
Table 2. CRROSS Delivery of services from 2009 to 2013.
judge the value of advertising content and venue and to inform changes to recruitment approach. Results
Utilization Table 2 provides information on CRROSS services provided to investigators from 2009 to 2013. During this period, the CRROSS Call Center managed 21,980 calls to and from new and returning volunteers. Of the 4,047 unique new callers who registered contact information with the Call Center hotline, 3,722 (92%) agreed to enroll in the Volunteer Repository (Figure 2). The racial and ethnic characteristics of the Research Volunteer Repository population Figure 2. Repository Growth by Calendar Year. Repository members are individuals who have consented to enroll in the Research Volunteer Repository, conducted under an IRB approval protocol. Enrollees agree to approximate those of New York City in 2010 be included in queries of the database and to be called in the future to learn about new research participation as judged by the 2010 US Census11 (Table 3). opportunities. The ages of Repository members range from 8 to 94 years old, with 50–60-year-olds contacts and the number and percentage of new callers enrolling making up the largest group (26%), followed by 40–50-yearinto the Repository. For all new protocol-specific requests to olds (21%), 30–40-year-olds (18%), and 20–30-year-olds, and query the Repository, staff members verify the IRB-approved 60–70-year-olds (each at 14%). association of the protocol to the repository prior to conducting the query. The repository is audited on an annual basis to assure compliance. Outcomes—overall Outcomes of recruitment and enrollment service There are three major paths by which volunteers make contact effectiveness are also analyzed, including the demographics with the CRROSS Call Center: (1) calls made by new callers who of enrolling participants, the percentage of volunteers passing respond to advertising or a referral by a friend, family, or medical prescreening and referred for screening/enrollment, the professional. (2) Calls made by Repository members who call the percentage of referred volunteers who actually enroll in the CRROSS team either spontaneously or in response to advertising. study, and the percentages of enrolled volunteers who remain (3) Calls made by CRROSS staff to repository members identified in the study or have completed the study. Additional outcomes through queries of the database who appear to be eligible for a that are evaluated are the times elapsed from the initiation of specific study. CRROSS recruitment efforts until the first participant enrollment From 2009 to 2013, CRROSS completed a total of 9,244 and until study enrollment is completed. Advertising costs are prescreening interviews. Of those, 4,047 prescreen conversations also tracked per protocol, per ad campaign, and per enrolled concerned recruitment of new callers to the Repository, and participant. Participants who fail to keep appointments for initial 92% of these callers (3,722 participants) agreed to enroll. The screening visits are tracked as “no-shows” in the Repository. remaining 5,197 prescreen conversations were conducted to Referral source, enrollment, and no-show data are also used to recruit participants to research studies. These prescreens included WWW.CTSJOURNAL.COM
VOLUME 8 • ISSUE 2
95
Kost et al. Data-Rich Recruitment Core n
White
Black or African American
Asian
Native American or Alaskan Native
Native Hawaiian or Pacific Islander
Some other race
Two or more races
Declined or not specified
New York City Population 2010 Census
33%
23%
13%
0.2%
–
0.7%
2%
–
Hispanic 29%
Research Volunteer Repository
35%
39%
4%
1%
1%
9%
4%
7%
20%
Table 3. Race and ethnicity demographics of CRROSS database population and New York City.11
Prescreened by CRROSS
Referred
“No-shows” 1st visit
Percent enrolled of volunteers who complete a screening visit†
Signed consent
Retained (%)
Percent enrolled of all volunteers referred*
1,468
1,255 (85)
43%
76%
–
280
257 (92)
49%
–
311
170
124 (73)
33%
55%
Screened
Volunteers directly calling or recalling the Call Center, 2009–2013 5,197
3,378
1081
1,943
Volunteers identified through queries of the Repository, 2009–2013 567
–
–
Example: Volunteers identified for specific study
13
785
512
185
Number of volunteers who signed consent/total number of volunteers referred to teams, including “no-shows.” † Number of volunteers who signed consent/total number of volunteers who kept their screening visit appointment. *
Table 4. Enrollment Progression of Volunteers Referred by CRROSS, from Prescreen through study enrollment/completion, by calendar year. Volunteers directly calling the Call Center who pass telephone prescreening for a specific protocol are referred to research teams for screening in person. Some research teams further prescreen referrals by telephone before selecting volunteers for screening visits. Participants who fail to appear for an in-person screening appointment without cancelling or rescheduling are classified as “no-shows.” Participants who complete screening with the research team and then sign informed consent are considered “enrolled.” “Retained” participants are those who have completed, or still remain enrolled in the study at the time of analysis.
new callers, returning callers, and the participants contacted as a result of Repository queries, and yielded 3,378 referrals (2,803 unique individuals) resulting in 1,468 (52%) study enrollments. Eighty-five percent of these enrollees (1,255) completed or still remained in the study at the time of the analysis. The Research Volunteer Repository was an important source of research participant recruitment. Queries conducted on behalf of 51 separate studies identified 1,137 potentially eligible volunteers. Half of those volunteers (567) were able to be reached and prescreened, and 280 enrolled in protocols. Most importantly, 92% of participants who enrolled through queries from the Repository completed the study or remained on-study at the time of analysis. Of the studies serviced by queries, 2 were NIH-sponsored, 8 were sponsored by industry, and 41 were investigator-initiated, reflecting the Rockefeller CCTS research portfolio. Of the 3,722 unique individuals enrolled in the Repository, 853 individuals (23%) accounted for 1,258 enrollments into research protocols, of which 1,119 (89%) were retained. Overall, 86% (1,258/1,468) of all enrollments into studies managed by CRROSS involved individuals who were already enrolled in the Repository or who joined the Repository as part of their recruitment into a study. Outcomes—efficiency Enrollment outcome data on all volunteers who made contact with CRROSS from 2009 to 2013 for all studies are shown in 96
VOLUME 8 • ISSUE 2
Table 4. A leaky pipe model12 is often used to graphically illustrate the attrition of volunteers at each step from initial recruitment through completion of study enrollment. Data for a specific study13 using CRROSS services are also included in Table 4 to illustrate the data that are shared with investigators on an ongoing basis during the recruitment phase. The time elapsed in days from the date of initiation of CRROSS recruitment activities until the date the first participant enrolled in the protocol is shown in Figure 3. For the years 2009–2013, in aggregate, the median time-to-first-participant-enrollment was 10 days. For comparison, in the years 2006–2008, which were prior to institution of CRROSS services, the median time elapsed from the date of IRB approval (taken as a surrogate for the when the PI’s recruitment efforts were likely to have started) until the first participant visit was 70 days. Using data to improve recruitment Based on the data detailing the reasons for callers’ ineligibility or lack of interest in a given protocol, CRROSS personnel and the investigators decide whether to consider changes in advertising, eligibility criteria, and/or protocol design. For example, when it was observed that many callers were ineligible to enroll in a study because their body mass index was below the eligibility cut off value, CRROSS staff worked with the investigators to modify the wording of the advertisement. Data collected after the advertisement was modified showed WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
Financial model CRROSS consultative services and recruitment activities were provided free to investigators. Many advertising venues incur no costs to investigators (e.g., Craigslist, Facebook, ResearchMatch, ClinicalTrials.gov). When paid advertising was necessary, CRROSS provided costsharing for advertising (jointly supported by the University and the CCTS) to trainees and to investigators using CRROSS services who had expended other recruitment resources. In 2013, CRROSS provided a total of $25,000 in cost-sharing to support 50%–100% of advertising costs for 20 of 42 protocols served. User satisfaction CRROSS deployed a user satisfaction survey in 2014 to all investigators and coordinators who used CRROSS services in the prior year. Of CRROSS service users, 70% used comprehensive CRROSS services at consultation, 14% utilized intermediate services, and 16% used a single CRROSS service. More than 90% of the 40 respondents were “very satisfied” or “satisfied” with the services they received. Two users expressed dissatisfaction that Rockefeller does not have a large patient population as might be found in an academic medical center. Figure 3. Time elapsed from the initiation of active recruitment by CRROSS until the first participant visit (days). Protocols are grouped according to the calendar year in which CRROSS recruitment activities were initiated, even if the first participant was not screened until the following year. For the years 2009–2013, data are included for 11, 14, 18, 13, and 14 protocols respectively. For the year 2012, one outlier value is indicated by “*”; the value, 473 days, was included in the calculations for the median shown, but was omitted from the graph visualization to preserve resolution.
that a higher percentage of respondents to the advertisement were now eligible for the study (Figure 4). Similar interim reports of the outcomes of recruitment efforts have been used by investigators to modify their approach and improve recruitment effectiveness. Other factors that affect timely recruitment Investigator-specific challenges can have a negative impact on timely recruitment, including lack of readiness to conduct laboratory or clinical assessments, limited time availability to see participants, protocol modifications after initial approval, addition of new eligibility criteria, or overreliance on convenience sampling or colleague referrals for recruitment. We observed that studies involving normal volunteers tended to have higher “no-show” rates, sometimes exceeding 30%, despite appointment reminder calls and confirmations. Some research teams informally suspended recruitment activities to prioritize other activities, thus prolonging the recruitment period. In response to these findings, CRROSS began to proactively attempt to identify during the initiation consultation any factors that might later introduce recruitment delays and to then incorporate this information into assessment of enrollment feasibility and the proposed timeline for completing recruitment. WWW.CTSJOURNAL.COM
Discussion
Using resources made available through the CTSA program and The Rockefeller University, we established a centralized, core research participant recruitment infrastructure offering expertise and services to investigators to maximize enrollment of individuals into clinical research protocols. We designed the core and its procedures to incorporate recruitment expertise early into the protocol development process,10 and to incorporate data capture and timely analysis, so that rapid changes could be made in the advertising or in the protocol to improve recruitment. We also created a Research Volunteer Repository to provide an evergrowing database of participants willing to participate in research protocols. Utilization of the recruitment core by research teams has grown steadily, and overall feedback about the program has been very positive. The ability to provide reports detailing the effort and activities required to successfully recruit participants educates investigators about the requirements for effective recruitment, informs planning of subsequent protocols, and demonstrates the value added by using CRROSS services. Collecting detailed data on the recruitment process also enabled us to assess the effectiveness of the different approaches to recruitment. The Research Volunteer Repository has been an important addition to our recruitment efforts, with 23% of the individuals in the Repository enrolling in at least one protocol, with 89% retention/completion. In addition, many Repository members spontaneously check with the recruitment core periodically to assess whether they VOLUME 8 • ISSUE 2
97
Kost et al. Data-Rich Recruitment Core n
We realize, however, that our experience may not be generalizable because of the nature of our institution and the population we serve. The ultimate goals of successful recruitment are timely study accrual and completion. There are, however, no consensus standards for meaningful measures of timely accrual. Studies vary in their duration, target population size and availability, and the stringency of eligibility criteria. Our near-term measure, the elapsed time from initiation of recruitment efforts to the first participant enrollment, is a median of 10 days for protocols receiving CRROSS services 2009–2013. We do not have a precise baseline comparator, since prior to the creation of CRROSS the date of initiation of recruitment was not recorded. However, for protocols starting in 2006–2008, the median time elapsed from IRB approval to Figure 4. CRROSS Call Center Outcomes for initial and amended advertisement. CRROSS staff collect first participant enrollment was 70 days. By aggregate prescreening outcome data (eligible vs. ineligible) as well as the specific reasons for ineligibility (e.g., failure to meet individual eligibility criteria, not interested, compensation too low) in order to rapidly suggest modithis measure, CRROSS services have had an fications in advertising. Data are shown for study MAP-0683, in which overweight individuals with low vitamin D important impact on speeding the initiation levels were recruited to study the effects of vitamin D repletion on serum cholesterol levels. The headline of the of enrollment. advertisement was: “Are you worried about your cholesterol?” Overall, 68% of respondents were ineligible, 35% due to a body mass index (BMI) below the study threshold. The advertisement was then amended to attract We recognize the limitations of volunteers with a higher BMI. The amended advertisement headline was: “Are you overweight? Are you worried extrapolating from our experience. In about your cholesterol?” After running the amended advertisement, the proportion of prescreened callers who particular, our institution’s research protocol were ineligible decreased to 47%, and the proportion of callers ineligible due to BMI criteria decreased to 22%. portfolio is relatively small compared to large academic medical centers. As a result our model may not be readily scalable in a large are eligible for new protocols. When the Repository has been institution. The major costs associated with expanding services queried to recruit for specific protocols, it has resulted in 50% would include those of additional expert recruitment personnel enrollment of those contacted, with 92% retention. Data capture and the software licensing costs to expand data platform, track and real-time analysis allow us to assess the most productive participants in the Repository, and enable data analysis. Since advertising venues for specific populations and specific protocols, we have found it to be important for recruitment personnel to and enable rapid modification of advertising campaigns in real be familiar with all of the protocols they offer, it may be better time to improve recruitment. It also allows us to match costs of to create additional recruitment teams, for example, to meet the advertising with outcomes, producing crucial cost-effectiveness needs of a department, rather continuing to add personnel to a analyses. As we have obtained more data, we have also been able single team when expanding services. Nonetheless, we believe to judge better how many prescreening interactions by the Call the central principles that define our program are generalizable, Center and screening visits by the research team will likely be namely, providing a range of centralized expert recruitment needed to achieve the desired enrollment. Thus, our feasibility services, hosting a call center to refer participants to the studies assessments have continued to improve. in which they are most interested, and most importantly, rapidly A number of patient/participant registries have been collecting robust data for analysis throughout the protocol life developed at research centers around the country based on cycle to inform performance improvement. participants providing consent to be contacted in the future for The lack of broadly agreed upon precisely defined criteria for recruitment or enrolling themselves in Web-based registries standardized measurements of recruitment success, including (unpublished proceedings of 5th Annual CRM workshop, New study enrollment, retention, and completion is a serious Haven, CT, USA, June 4–5, 2012).5–9 There are, however, few impediment in comparing different approaches to participant reports of the impact of these registries on timely enrollment recruitment. We hope that by reporting our experience we can and study completion, and so the cost/benefit is uncertain. encourage others to join us in trying to develop such metrics. Other approaches to speedy recruitment include aggregation Some of our studies had very long times to first enrollment. In of data from electronic health records based on consent waivers virtually all cases, the delay was due to factors beyond the recruitment or opt-out mechanisms; two major initiatives in 2014 are the program’s control. By analyzing these reasons systematically, PCOR-NET14–16 and CTSA-based data network.17 It remains to however, we have been able to proactively refine our assessment be established which approach(es) are most effective. Since our of feasibility. We also recognize the importance of ultimately institution is not part of an academic health center with a large focusing on study completion rather than study initiation, and we patient population, we do not have the opportunity to create a are currently directing our efforts to maximizing these crucial goals. general patient population registry for recruitment.18 Instead, we One of the key lessons that we learned is that the presentation created a Research Volunteer Repository in which each of the of data to scientific investigators is the persuasive form of enrollees has explicitly expressed interest in research participation. communication. Much has written about the importance of 98
VOLUME 8 • ISSUE 2
WWW.CTSJOURNAL.COM
Kost et al. Data-Rich Recruitment Core n
having a learning health care system.19,20 We think our approach is an important step forward in developing a learning clinical research system. Our experience using different database formats has also been instructional. We initially used an in-house ACCESS database coupled to additional REDCap data capture tools. While this enabled us to capture the requisite data, extracting and analyzing data was laborious and slow. In 2013, we migrated our dataset to a software platform designed specifically to manage research participant recruitment (Clinical Conductor, Bio-Optronics, Inc.). This greatly simplified data analysis, reducing days of data aggregation to minutes of reporting. It also afforded integration of multiple aspects of recruitment tracking, including advertising campaign costs and effectiveness that were previously managed separately. We conclude that centralizing recruitment activities within a core structure staffed by expert personnel who are equipped to perform comprehensive data capture and analysis, speeds participant recruitment and enrollment. It also spares investigators from diverting their energies to an area in which few have expertise, standardizes the approach, increases the efficient use of scarce resources, and insures compliance with regulatory requirements for advertising and informed consent. Most important, it makes it possible to rigorously collect data as the basis for informing current practice and improving future practice. Acknowledgments
This project is supported in part by grant 8 UL1 TR000043 from the National Center for Research Resources (NCRR) and the National Center for Advancing Sciences (NCATS), National Institutes of Health. Conflict of Interest
The authors have no conflicts of interest to disclose. Human Subjects
The Research Volunteer Repository is conducted under a protocol approved by the Rockefeller University Institutional Review Board (IRB) according to applicable regulations. The CRROSS recruitment core verifies IRB approval of each protocol before conducting any recruitment activities for the protocol.
WWW.CTSJOURNAL.COM
References
1. Translation. IoMUFoDDDa. Transforming Clinical Reseach in the United States: Challenges and Opportunities: Workshop Summary. National Academies Press (US); 2010. Available at: http://www.ncbi.nlm.nih.gov/books/NBK50892/. Paper presented at: National Academies Press (US); 2010, 2010; Washington (DC). 2. Kitterman DR, Cheng SK, Dilts DM, Orwoll ES. The prevalence and economic impact of lowenrolling clinical studies at an academic medical center. Acad Med. 2011; 86(11): 1360–1366. 3. Anderson DL. A Guide to Patient Recruitment and Retention. Boston, Mass: Thomson Center Watch; 2004. 4. Kost RG, Mervin-Blake S, Hallarn R, Rathmann C, Kolb HR, Himmelfarb CD, D’Agostino T, Rubinstein EP, Dozier AM, Schuff KG, et al. Accrual and recruitment practices at the CTSAs: a call for expectations, expertise and evaluation. Acad Med. 2014: 1–10. 5. Washington University School of Medicine Research Participant Registry. Available at: https:// vfh.wustl.edu/. Accessed March 19, 2014. 6. University of Pittsburgh. Research Participant Registry. Available at: https://www.researchregistry. pitt.edu/. Accessed October 27, 2014. 7. Rigers JL, Johnson TR, Smith YR. Recruitment of women research participnts: the women’s health registry at the University of Michigan. J Womens Health. 2007; 16(5): 721–728. 8. ResearchMatch. Available at: www.researchmatch.org. Accessed October 27, 2014. 9. UCSF CTSI. Research Participant Registry. 2012. Available at: https://ucsfregistry.secure.force. com/. Accessed October 27, 2014. 10. Brassil D, Kost RG, Hurley A, Dowd K, Stanwix J, Coller BS. The Rockefeller University Navigation Program: a structured multidisciplinary protocol development and educational program to advance translational research. Clin Transl Sci. 2014; 7: 12–19. 11. NYC.gov. NYC Census. 2010; Available at: http://www.nyc.gov/html/dcp/html/census/ census_2010.shtml. Accessed July 14, 2014. 12. CPP Inc. Available at: http://www.drugdevelopment-technology.com/contractors/consulting/ cpp/cpp2.html. Accessed June 12, 2014. 13. Ponda MP, Dowd K, Finkielstein D, Holt PR, Breslow JL. The short-term effects of vitamin D repletion on cholesterol: a randomized, placebo-controlled trial. Arterioscler Thromb Vasc Biol. Oct 2012; 32(10): 2510–2515. 14. Kaushal R, Hripcsak G, Ascheim DD, Bloom T, Campion TR Jr, Caplan AL, Currie BP, Check T, Deland EL, Gourevitch MN, et al. Changing the research landscape: the New York City Clinical Data Research Network. J Am Med Inform Assoc. Jul 2014; 21(4): 587–590. 15. Kaushal R, Blumenthal D. Introduction and commentary for special issue on health information technology. Health Serv Res. Feb 2014; 49(1 Pt 2): 319–324. 16. Kern LM, Edwards A, Kaushal R. The patient-centered medical home, electronic health records, and quality of care. Ann Intern Med. Jun 3 2014; 160(11): 741–749. 17. Nadler L, Reis S, Toto R, Firestein G. CTSA Website, Proceedings of 2014 CTSA PI Retreat: Participant Accrual. 2014; Description of evolving CTSA patient-data-network. Available at: https: //www.ctsacentral.org/sites/default/files/documents/%233_Participant%20Accrual_Nadler_ Reis_Toto_Firestein.pdf. Accessed July 24, 2014. 18. Embi PJ, Jain A, Clark J, Harris CM. Development of an electronic health record-based clinical trial alert system to enhance recruitment at the point of care. Paper presented at: AMIA2005. 19. The CTSA Program at NIH: Opportunities for Advancing Clinical and Translational Research: The National Academies Press, Washington, D.C.; 2013. 20. Observational Studies in a Learning Health System: Workshop Summary: The National Academies Press, Washington, D.C.; 2013.
VOLUME 8 • ISSUE 2
99
Copyright of CTS: Clinical & Translational Science is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
