Journal of Medical Systems, Vol. 16, No. 6, 1992
Evaluating Productivity in Clinical Research Programs: The National Cancer Institute's (NCI) Community Clinical Oncology Program (CCOP) Denise M. Hynes, 1 Curtis P. MeLaughlin, 2 Arnold D. Kaluzny, ~ Leslie P. Ford, 3 and E d w a r d Sondik 3
This paper outlines an approach to studying productivity in clinical research programs that incorporates environmental, organizational, provider, and patient specific factors in the model of production process. We describe how this approach has been applied to the National Cancer lnstitute' s (NCI) Community Clinical Oncology Programs (CCOPs). Next, a practical evaluative model of the productive process in CCOPs is outlined and its use in evaluation and monitoring performance in CCOPs is discussed. Each level of the model is described and a number of factors potentially affecting each level are explored. Finally, we discuss the strengths and weaknesses of this approach and show how management can use it to study and improve the productivity of clinical research programs.
INTRODUCTION Despite the importance and potential impact of clinical trials research, little attention has been given to evaluating and improving the productivity and efficiency of organizations conducting clinical trials of medical devices for the Food and Drug Administration and other agencies. The potential of clinical trials to affect the adoption and use of medical technologies and medical treatments, as well as payment decisions by health insurers has been demonstrated. 1 A variety of different health care organizations are funded to conduct clinical trials research, including individual physicians, hospitals, university medical centers and health departments. However, the organizational and environmental factors that promote productivity in clinical research programs are not fully understood. Evaluation of productivity in research organizations can be very elusive. Like other From the 1Department of Health Policy and Administration and the Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina; 2Kenan-Flager Business School and the Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina; 3Division of Cancer Prevention and Control, National Cancer Institute, Bethesda, Maryland. 247 0148-5598/92/1200-0247506.50/0 © 1992 Plenum Publishing Corporation
248
Hynes et ai.
health care service organizations, techniques to evaluate and improve productivity on the whole, have been slow to develop. One major difficulty lies in defining the product and the inputs that go into its production. Previous work examining cancer research organizations attempting to evaluate the process of enrolling patients on clinical trials, has focused on the influence of physician and patient level decisions in this process. 2-5 The influence of the environment and organizational dimensions have not been fully examined. This paper outlines an approach to studying productivity in clinical research programs that incorporates environmental, organizational, provider, and patient specific factors in the model of production process. We describe how this approach has been applied to the National Cancer Institute's (NCI) Community Clinical Oncology Programs (CCOPs). Next, a practical evaluation model of the productive process in CCOPs is outlined and its use in evaluating and monitoring performance in CCOPs is discussed. Each level of the model is described and a number of factors potentially affecting each level are explored. Finally, we discuss the strengths and weaknesses of this approach and show how management can use it to study and improve the productivity of clinical research programs.
DESCRIPTION OF THE CCOP PROGRAMS In 1981 the National Cancer Institute (NCI) initiated the Community Clinical Oncology Program (CCOP) to help meet the objectives of the year 2000 goal to reduce cancer mortality to 25-50% of the 1980 levels. 6 The CCOP was designed to bring the benefits of clinical research to cancer patients in their local communities by providing support to enable community physicians to enter their patients onto NCI-approved cancer treatment and cancer control clinical research protocols, also referred to as clinical trials. Heretofore, these clinical trials were restricted to university affiliated hospitals and their satellites. A CCOP may consist of a group of physicians, a clinic, a hospital, a health maintenance organization or a consortium that agrees to work together through a principal investigator and with a single administrative focus. Institutional Review Boards (IRBs) within the CCOP organization review and approve research protocols for use at the CCOP institutions. Through participation in a CCOP, physicians have access to the latest anticancer agents and clinical research protocol information regarding treatment, follow-up, and overall cancer patient management. 7 This enrollment of cancer patients onto clinical protocols is done through participation in one or more NCI-funded research bases. Research bases are clinical trials cooperative groups or clinical cancer centers with demonstrated ability to conduct large scale clinical trials and develop cancer control trials. They are responsible for protocol development, data analysis and quality control. The research bases and CCOPs form a network. In 1983, 62 community programs and 19 research bases were funded in 34 states. During this time approximately 14,000 patients were enrolled onto NCI-approved cancer treatment trials through the CCOP. In 1987, a second phase of 52 CCOPs and 17 research bases were funded. At the end of the first year of phase two, over 4200 patients were entered onto NCI-approved treatment trials through their local CCOPs. Figure 1 shows the CCOP and research base resources affecting operations in an
Productivity in Clinical Research Programs
249
actual CCOP. This process involves four resource sub-groups. They are: the patient population at risk in the CCOP's catchment area; the nature of the local CCOP; the population of protocols available to that CCOP through its participation in its research bases; and the population of physicians who are in a given CCOP's area. The physician resources are complex and include the mix of primary care providers and medical, surgical and radiological oncologists available in the community and especially those affiliated with the CCOP. The availability of protocols derives from a complex set of interactions between NCI research efforts, the research bases and the local CCOPs. In this CCOP/research base network weighted accruals, or credits, for patients entering protocols currently serve as the primary measure of CCOP performance. Accrual is defined as the number of patients that are entered onto NCI-approved cancer treatment or cancer control protocols. For each patient accrued to a particular protocol, a credit is assigned. This credit is determined by the NCI and is based on the complexity of the protocol and the extent of data management and follow-up necessary to monitor the patient's progress. For example, a clinical treatment trial which includes frequent physiologic monitoring and multiple follow-up visits may have credits as high as 2.5 for every patient enrolled, whereas, a cancer control trial comparing two different screening approaches for colon cancer, requiring only one patient contact may have accrual credits as low as 0.1 per patient enrolled. Currently, each CCOP is required to enroll patients on both cancer treatment and cancer control protocols with annual achievement targets of 50 cancer treatment accrual credits and 20 cancer control credits.
DEFINITION OF PRODUCTIVITY Productivity in an organizational sense has been defined as the ratio of production outputs or services, to production inputs, or resources. 8 Inputs include the labor and capital resources and consumer demand factors that directly affect the production process. Outputs are the direct result of the production process. For example, output measures used to measure productivity in hospitals have included total number of admissions and discharges. In family planning programs output measures have included total number of visits and total number of clients served. 9 In the case of individual CCOPs for which the primary purpose is to provide a service, one can conceptualize the production process as provision of state-of-the-art cancer care to the available cancer patients. Thus, a key output is the number of-patients who enter into and complete protocols. Accruals and the number of protocol completions, then, serve as output measures. Another output measure is the accrual credits achieved by the individual CCOP. Presumably, the accrual credits received reflect the mix of protocols used by the CCOP.
MODELING
THE PRODUCTION
PROCESS
The process of producing these outputs--patients completing protocols and the resulting accrual credits---can be described as a flow of patients who reside in the local CCOP catchment area and develop cancer or cancer symptoms. A subset of these patients
250
Hynes et al.
will see or be referred to physicians participating in the local CCOP. Depending upon the specific diagnosis, patients may be considered for one of the research protocols available through the local CCOP. Appropriate patients may be offered the opportunity to participate in a specific protocol. If patients choose to participate, they are enrolled onto the protocol and the protocol-designated treatments commence. Patients may or may not complete the protocol depending upon clinical factors and personal preferences. This process can be modeled as a flow of the type shown in Figure 2. The process starts when individuals in the target population develop a cancer (CA/). This is an epidemiological event which may or may not produce symptoms. This event can be represented by incidence and prevalence rates which, presumably, are known for the population of the CCOP catchment area or a regional area. These individuals may or may not have a cancer for which a protocol is available-anywhere or at this CCOP. Once a patient is diagnosed and a protocol for that particular cancer is available at that CCOP, a patient still may or may not enter that protocol. The patient must first reach physicians who participate in that protocol directly or by referral. They must decide whether or not that patient is a candidate for the available protocol(s). If the physician decides that a patient is a candidate and offers the protocol therapy as an option, the patient must still decide whether or not to participate in that protocol. If the patient decides to participate, the patient must be entered onto the protocol, at which time the enrolled patient is counted as an accrual. To reach the desired end, however, the patient mustsurvive and continue to cooperate for a finite period. Under ideal conditions all patients in the community with cancers for which protocols are available would become enrolled and complete the course of treatment. We conceptualized this process in continuous quality improvement (CQI) terms. Therefore, we analyzed the process in terms of factors affecting throughput losses using a series of CQI tools flow charting, process flow analysis and Ishikawa diagrams for causal analysis. 10 Each of the flow rates out of each level can be represented as the proportion of those who do not proceed to the next (k + 1) stage (Nk) and those that do (1 - Nk). The advantage of using this flow analog is that one can disaggregate the system to the extent that the transition rates are known. Data analysis can then be related to specific transition rates as dependent variables, a significant gain when there are relatively few data points. Transition rate information is on incidence (Ii), protocol availability (1 Na), proportion of CCOP physicians treating or specializing in specific cancers (1 - Nh), proportion of physicians referring eligible patients to the CCOP (1 - Nb), patient eligibility for protocols (1 - Nc), proportion of eligible patients selected and offered protocol enrollment (1 - Nd), the proportion of patients offered protocol therapy who accept it (1 - Ne), the proportion of accepting patients who actually get enrolled (1 - Nf), and, the proportion of those enrolled who complete the protocol therapy (1 - Ng). The weakest data link concerns the patients not reaching referring physicians (Nc). These rates operate on the base population of the CCOP catchment area (Pi) susceptible to the ith cancer. The mathematics of the flows are given by Equations 1 and 2 which are reproduced below:
Aijt = ,E {(Iit) * (Pit)} *
(1
-
NUta) • {(1 - Nimlh) + Nimth(1 -- Nilb)} * (1 -- Nijlc)
• (1 - Nijlrnd) * (1 -- Niflme) * (1 " Niflf)
(1)
Productivity in Clinical Research Programs
251
"~o
°
E = u..=j
¢:
l
.# ;=.
o
=.. = .:,
=o
U ,', .9 = u
._~[
~
s
© rj
t
o
u o
.o
f
~ =:., -,:, N
~.
o
===
0"-
E~
=EO
0 r.j r.j 0
=0 :t ¢,J3
t ~o~
o ,,.j ,,j
0
~, ,p =,. '.J ~.,¢:
252
Hynes et al.
IPopulation
Cancer Cases in the C C O P Azea
Base
I CCOP Population
referred to
I MD Population
Protocol Population
Protocol Availability
CCOP have a
CCOP Physician Utilization a n d Referral to C C O P Physicians
patient fit on available
Patient Eligibility
select patient
Patient Selection
accept protocol
e~tered?
Patient Acceptance
Protocol Entrance
'y
Does patient
Desired Result J
Figure 2. CCOP Productivity analysis framework.
Protocol C o m p l e t i o n
Productivity in Clinical Research Programs
Rift = Aijl * (1 - Nijlg)
253
(2)
where, i = ith cancer, i = (1 . . . . I), j = jth protocol, j --= (1 . . . . J), k = kth loss rate, k = (a . . . . g), l = /th CCOP, l = (1 . . . . 52), mth type of physician, m = (1 . . . . M), m Aij l = Number of patients with the ith cancer entered on the jth protocol in the /th CCOP, Rift = Number of patients with the ith cancer enrolled on the jth protocol who completed the protocol in the/th CCOP, Ill = incidence rate of the ith cancer in the/th CCOP, PI population o f / t h CCOP community or catchment area, Not a = 1, if the/th CCOP does not have the jth protocol available for the ith cancer; = 0 otherwise, Nil b : proportion of time the primary care physicians do not refer patients to the/th CCOP for the ith cancer, proportion of physicians of type m, in the/th CCOP area, who see and treat Nilmh : patients with the ith cancer, but are not in the/th CCOP, Niflc = proportion of patients with the ith cancer, not suitable for the jth protocol in the lth CCOP service area, Nijlrnd = proportion of time that the mth type of physician active in the/th CCOP does not select an eligible patient with the ith cancer for the jth protocol, Nijlme = proportion of time that the patient with the ith cancer rejects the jth protocol presented by the mth type of physician at the/th CCOP, Nijtf = proportion of time that the patient with the ith cancer who accepts the jth protocol is not entered onto the jth protocol by the/th CCOP, = proportion of time that the patient with the ith cancer enrolled on thejth protocol Not by the/th CCOP, fails to complete that protocol. Maximum accrual is achieved when all of the loss rates (Nk) are equal to zero. Thus, the extent to which a CCOP can minimize the loss rates that it faces will potentially increase accrual and protocol completions, thus improving CCOP productivity. These losses of potential patients can be treated as a quality analysis issue. That approach is represented in Figures 3 through 9 as "fishbone" or Ishikawa diagrams which indicate the potential independent variables to be considered in detailed studies of loss of potential accruals at each stage. A social scientist might choose to think of these diagrams as the basis for a regression model dealing with each of the relevant flow rates. The usual approach in quality control circles is to do a Pareto analysis of the sources of losses, identifying the most frequent causes and concentrating on them first. It should be noted that some of these relationships are very simplified in this flow diagram and in the Ishikawa diagrams. LEVEL BY LEVEL ANALYSIS The model presented here allows targeting of the sources of loss in productivity at a CCOP. To explore why these deficiencies are occurring requires a closer examination of
254
Hynes et al.
those factors that may affect the level where the greatest loss has occurred. This section discusses the factors that may be important.
1. CCOP Catchment Area Consider first the population of the CCOP service area (Pi) (Refer to Figure 2). As in all research, the population base actually served by a health delivery entity is sketchy at best. Presumably the population base served was considered at some time when the proposal for admission into CCOP was under consideration. Once that number is agreed on, it is then possible to assess the CCOP members' share of market. This market share estimate multiplied by the cancer prevalence (or incidence) rate (Ii), can be used to estimate the number of cancer cases that are likely to find their way to the CCOP's oncologists. The resultant would be the number of cases (Cai) that the CCOP could be expected to garner, if everything were working perfectly. This number would also produce a useful denominator for ratio analysis, 11 yielding variables such as accruals per eligible case or per thousand population. A second denominator would be the dollars spent on the program such as the dollars per accrual or dollars per accrual credit. Several alternative possibilities exist for share estimation. One approach would include the proportion of oncology beds in the CCOP component hospitals or the total proportion of the community's practicing oncologists. One important analytical issue is whether or not to focus only on the specific cancers targeted by the particular CCOP. For example, CCOPs with physician specialists in breast cancer may choose to concentrate their efforts on recruiting patients for breast cancer protocols only, and perhaps this fact should be taken into account in the model. Such an approach would simplify the data gathering, but it effectively limits the productivity studied to the internal operations of that CCOP and eliminates any focus on the inter-organizational productivity issues involving the CCOPs and their research bases and linkages with technology transfer processes within NCI.
2. Protocol Availability The availability of a protocol at a particular CCOP is the next transition rate affecting productivity (Na) (Figure 3). Four factors are hypothesized to affect this rate. First, the CCOP physicians and staff may not be aware of the existence of a particular protocol. Lack of awareness can be due to lack of motivation to learn about what is available either within the existing network of affiliated research bases or through outside cooperative groups, or due to the research base's failure to communicate about a protocol's availability to the local CCOP. Second, although aware of the protocol, the CCOP may not have access to a specific protocol, because their research bases do not offer it. Some research bases specialize their protocol development and research activities on certain cancers or treatment modalities. If there is no mechanism for access to protocols offered by research bases outside of their existing network, then the CCOP can not enroll patients on that particular protocol. Only by the NCI designation of a protocol as an intergroup protocol can a CCOP enroll a patient on a protocol through another research base. Third, even when the protocol is available through the CCOP/research base network, the CCOP community hospitals may not be able to accommodate the protocol. The hospital may not
Productivity in Clinical Research Programs
255
'i '°'~×/ N,
'f"
I
o
0
E E O
z~
J
z~
°~
O
Z. o
256
Hynesetal.
have the available technology, trained personnel, or capital budget to adopt particular protocols. Furthermore, the CCOP may not choose to adopt a protocol that is available and feasible because it is not a priority for the CCOP or its consortium members. This lack of priority may be due to a wide variety of reasons. There may be no champion for it in the organization, or it may just have been overlooked, and, thus not submitted to the CCOP IRB. It may have been considered and rejected by the CCOP IRB due to cost, low priority, or unacceptable (locally) risk to patients. Regarding this risk issue, a protocol may be deemed to have unacceptable risk due to a misconception or due to risk adverseness on the part of local providers. The protocol may also be stalled in local or IRB committee processes due to bureaucracy or due to a hung jury over technical facts or values.
3. CCOP Physician Utilization and Referral Rate The next transition rate concerns access to CCOP physicians (Figure 4). This is one of the more critical levels in the process. Access to CCOP physicians depends upon the physician environment in the CCOP area and the extent to which physicians and patients in the CCOP area utilize CCOP resources. The loss of potential patients (N9) may be due to political problems in the referral process or due to the success of competing providers. Some primary care providers may prefer to send their patients to other local non-CCOP oncologists. There may be perceived costs to the primary care physicians to refer patients to the CCOP. The physicians in the CCOP may have a weak image with their colleagues or there may be a personality clash. Competing providers in the CCOP catchment area may also affect the CCOP's market share of patients. Such competition could be in the form of other local oncologists, another CCOP, or an academic cancer center. It may be that coordination problems within the CCOP institutions have turned the patients away or that an oversupply of specialists has led to a splintering of the share of market available. Physicians may not be available to treat a particular cancer due either to overall short supply of physicians or to lack of recruitment effort by the CCOP. Patients may not have access to the appropriate level of physician specialists due to shortages of trained oncologists, scheduling problems in using the available personnel, or due to ineffective use of support staff. The CCOP also may not have recruited the specialist preferred by patients or their primary physicians. This problem can be due to a number of weaknesses on the part of the CCOP including level of physician recruitment effort, lack of support for protocol, poor coordination in the CCOP, unattractiveness of the hospital, or a weak champion for the CCOP within the institutional hierarchy. Patient characteristics and accessibility issues can also turn away potential patients. Patients' socioeconomic condition may block their referral to a CCOP physician. They may feel financially constrained; they may lack understanding about the referral process due to lack of basic education, or they may hold inappropriate beliefs about cancer and the medical treatment for it. The patient may be denied access to a CCOP physician because: they lack adequate health insurance or their health plan may restrict their provider choice, or they are already too ill to overcome any barriers to care, or they may live too far away to use the CCOP facility (in their own perceptions).
Productivity in Clinical Research Programs
257
8e~ @ @
,a
Evaluating Productivity in Clinical Research Programs: The National Cancer Institute's (NCI) Community Clinical Oncology Program (CCOP) Denise M. Hynes, 1 Curtis P. MeLaughlin, 2 Arnold D. Kaluzny, ~ Leslie P. Ford, 3 and E d w a r d Sondik 3
This paper outlines an approach to studying productivity in clinical research programs that incorporates environmental, organizational, provider, and patient specific factors in the model of production process. We describe how this approach has been applied to the National Cancer lnstitute' s (NCI) Community Clinical Oncology Programs (CCOPs). Next, a practical evaluative model of the productive process in CCOPs is outlined and its use in evaluation and monitoring performance in CCOPs is discussed. Each level of the model is described and a number of factors potentially affecting each level are explored. Finally, we discuss the strengths and weaknesses of this approach and show how management can use it to study and improve the productivity of clinical research programs.
INTRODUCTION Despite the importance and potential impact of clinical trials research, little attention has been given to evaluating and improving the productivity and efficiency of organizations conducting clinical trials of medical devices for the Food and Drug Administration and other agencies. The potential of clinical trials to affect the adoption and use of medical technologies and medical treatments, as well as payment decisions by health insurers has been demonstrated. 1 A variety of different health care organizations are funded to conduct clinical trials research, including individual physicians, hospitals, university medical centers and health departments. However, the organizational and environmental factors that promote productivity in clinical research programs are not fully understood. Evaluation of productivity in research organizations can be very elusive. Like other From the 1Department of Health Policy and Administration and the Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina; 2Kenan-Flager Business School and the Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina; 3Division of Cancer Prevention and Control, National Cancer Institute, Bethesda, Maryland. 247 0148-5598/92/1200-0247506.50/0 © 1992 Plenum Publishing Corporation
248
Hynes et ai.
health care service organizations, techniques to evaluate and improve productivity on the whole, have been slow to develop. One major difficulty lies in defining the product and the inputs that go into its production. Previous work examining cancer research organizations attempting to evaluate the process of enrolling patients on clinical trials, has focused on the influence of physician and patient level decisions in this process. 2-5 The influence of the environment and organizational dimensions have not been fully examined. This paper outlines an approach to studying productivity in clinical research programs that incorporates environmental, organizational, provider, and patient specific factors in the model of production process. We describe how this approach has been applied to the National Cancer Institute's (NCI) Community Clinical Oncology Programs (CCOPs). Next, a practical evaluation model of the productive process in CCOPs is outlined and its use in evaluating and monitoring performance in CCOPs is discussed. Each level of the model is described and a number of factors potentially affecting each level are explored. Finally, we discuss the strengths and weaknesses of this approach and show how management can use it to study and improve the productivity of clinical research programs.
DESCRIPTION OF THE CCOP PROGRAMS In 1981 the National Cancer Institute (NCI) initiated the Community Clinical Oncology Program (CCOP) to help meet the objectives of the year 2000 goal to reduce cancer mortality to 25-50% of the 1980 levels. 6 The CCOP was designed to bring the benefits of clinical research to cancer patients in their local communities by providing support to enable community physicians to enter their patients onto NCI-approved cancer treatment and cancer control clinical research protocols, also referred to as clinical trials. Heretofore, these clinical trials were restricted to university affiliated hospitals and their satellites. A CCOP may consist of a group of physicians, a clinic, a hospital, a health maintenance organization or a consortium that agrees to work together through a principal investigator and with a single administrative focus. Institutional Review Boards (IRBs) within the CCOP organization review and approve research protocols for use at the CCOP institutions. Through participation in a CCOP, physicians have access to the latest anticancer agents and clinical research protocol information regarding treatment, follow-up, and overall cancer patient management. 7 This enrollment of cancer patients onto clinical protocols is done through participation in one or more NCI-funded research bases. Research bases are clinical trials cooperative groups or clinical cancer centers with demonstrated ability to conduct large scale clinical trials and develop cancer control trials. They are responsible for protocol development, data analysis and quality control. The research bases and CCOPs form a network. In 1983, 62 community programs and 19 research bases were funded in 34 states. During this time approximately 14,000 patients were enrolled onto NCI-approved cancer treatment trials through the CCOP. In 1987, a second phase of 52 CCOPs and 17 research bases were funded. At the end of the first year of phase two, over 4200 patients were entered onto NCI-approved treatment trials through their local CCOPs. Figure 1 shows the CCOP and research base resources affecting operations in an
Productivity in Clinical Research Programs
249
actual CCOP. This process involves four resource sub-groups. They are: the patient population at risk in the CCOP's catchment area; the nature of the local CCOP; the population of protocols available to that CCOP through its participation in its research bases; and the population of physicians who are in a given CCOP's area. The physician resources are complex and include the mix of primary care providers and medical, surgical and radiological oncologists available in the community and especially those affiliated with the CCOP. The availability of protocols derives from a complex set of interactions between NCI research efforts, the research bases and the local CCOPs. In this CCOP/research base network weighted accruals, or credits, for patients entering protocols currently serve as the primary measure of CCOP performance. Accrual is defined as the number of patients that are entered onto NCI-approved cancer treatment or cancer control protocols. For each patient accrued to a particular protocol, a credit is assigned. This credit is determined by the NCI and is based on the complexity of the protocol and the extent of data management and follow-up necessary to monitor the patient's progress. For example, a clinical treatment trial which includes frequent physiologic monitoring and multiple follow-up visits may have credits as high as 2.5 for every patient enrolled, whereas, a cancer control trial comparing two different screening approaches for colon cancer, requiring only one patient contact may have accrual credits as low as 0.1 per patient enrolled. Currently, each CCOP is required to enroll patients on both cancer treatment and cancer control protocols with annual achievement targets of 50 cancer treatment accrual credits and 20 cancer control credits.
DEFINITION OF PRODUCTIVITY Productivity in an organizational sense has been defined as the ratio of production outputs or services, to production inputs, or resources. 8 Inputs include the labor and capital resources and consumer demand factors that directly affect the production process. Outputs are the direct result of the production process. For example, output measures used to measure productivity in hospitals have included total number of admissions and discharges. In family planning programs output measures have included total number of visits and total number of clients served. 9 In the case of individual CCOPs for which the primary purpose is to provide a service, one can conceptualize the production process as provision of state-of-the-art cancer care to the available cancer patients. Thus, a key output is the number of-patients who enter into and complete protocols. Accruals and the number of protocol completions, then, serve as output measures. Another output measure is the accrual credits achieved by the individual CCOP. Presumably, the accrual credits received reflect the mix of protocols used by the CCOP.
MODELING
THE PRODUCTION
PROCESS
The process of producing these outputs--patients completing protocols and the resulting accrual credits---can be described as a flow of patients who reside in the local CCOP catchment area and develop cancer or cancer symptoms. A subset of these patients
250
Hynes et al.
will see or be referred to physicians participating in the local CCOP. Depending upon the specific diagnosis, patients may be considered for one of the research protocols available through the local CCOP. Appropriate patients may be offered the opportunity to participate in a specific protocol. If patients choose to participate, they are enrolled onto the protocol and the protocol-designated treatments commence. Patients may or may not complete the protocol depending upon clinical factors and personal preferences. This process can be modeled as a flow of the type shown in Figure 2. The process starts when individuals in the target population develop a cancer (CA/). This is an epidemiological event which may or may not produce symptoms. This event can be represented by incidence and prevalence rates which, presumably, are known for the population of the CCOP catchment area or a regional area. These individuals may or may not have a cancer for which a protocol is available-anywhere or at this CCOP. Once a patient is diagnosed and a protocol for that particular cancer is available at that CCOP, a patient still may or may not enter that protocol. The patient must first reach physicians who participate in that protocol directly or by referral. They must decide whether or not that patient is a candidate for the available protocol(s). If the physician decides that a patient is a candidate and offers the protocol therapy as an option, the patient must still decide whether or not to participate in that protocol. If the patient decides to participate, the patient must be entered onto the protocol, at which time the enrolled patient is counted as an accrual. To reach the desired end, however, the patient mustsurvive and continue to cooperate for a finite period. Under ideal conditions all patients in the community with cancers for which protocols are available would become enrolled and complete the course of treatment. We conceptualized this process in continuous quality improvement (CQI) terms. Therefore, we analyzed the process in terms of factors affecting throughput losses using a series of CQI tools flow charting, process flow analysis and Ishikawa diagrams for causal analysis. 10 Each of the flow rates out of each level can be represented as the proportion of those who do not proceed to the next (k + 1) stage (Nk) and those that do (1 - Nk). The advantage of using this flow analog is that one can disaggregate the system to the extent that the transition rates are known. Data analysis can then be related to specific transition rates as dependent variables, a significant gain when there are relatively few data points. Transition rate information is on incidence (Ii), protocol availability (1 Na), proportion of CCOP physicians treating or specializing in specific cancers (1 - Nh), proportion of physicians referring eligible patients to the CCOP (1 - Nb), patient eligibility for protocols (1 - Nc), proportion of eligible patients selected and offered protocol enrollment (1 - Nd), the proportion of patients offered protocol therapy who accept it (1 - Ne), the proportion of accepting patients who actually get enrolled (1 - Nf), and, the proportion of those enrolled who complete the protocol therapy (1 - Ng). The weakest data link concerns the patients not reaching referring physicians (Nc). These rates operate on the base population of the CCOP catchment area (Pi) susceptible to the ith cancer. The mathematics of the flows are given by Equations 1 and 2 which are reproduced below:
Aijt = ,E {(Iit) * (Pit)} *
(1
-
NUta) • {(1 - Nimlh) + Nimth(1 -- Nilb)} * (1 -- Nijlc)
• (1 - Nijlrnd) * (1 -- Niflme) * (1 " Niflf)
(1)
Productivity in Clinical Research Programs
251
"~o
°
E = u..=j
¢:
l
.# ;=.
o
=.. = .:,
=o
U ,', .9 = u
._~[
~
s
© rj
t
o
u o
.o
f
~ =:., -,:, N
~.
o
===
0"-
E~
=EO
0 r.j r.j 0
=0 :t ¢,J3
t ~o~
o ,,.j ,,j
0
~, ,p =,. '.J ~.,¢:
252
Hynes et al.
IPopulation
Cancer Cases in the C C O P Azea
Base
I CCOP Population
referred to
I MD Population
Protocol Population
Protocol Availability
CCOP have a
CCOP Physician Utilization a n d Referral to C C O P Physicians
patient fit on available
Patient Eligibility
select patient
Patient Selection
accept protocol
e~tered?
Patient Acceptance
Protocol Entrance
'y
Does patient
Desired Result J
Figure 2. CCOP Productivity analysis framework.
Protocol C o m p l e t i o n
Productivity in Clinical Research Programs
Rift = Aijl * (1 - Nijlg)
253
(2)
where, i = ith cancer, i = (1 . . . . I), j = jth protocol, j --= (1 . . . . J), k = kth loss rate, k = (a . . . . g), l = /th CCOP, l = (1 . . . . 52), mth type of physician, m = (1 . . . . M), m Aij l = Number of patients with the ith cancer entered on the jth protocol in the /th CCOP, Rift = Number of patients with the ith cancer enrolled on the jth protocol who completed the protocol in the/th CCOP, Ill = incidence rate of the ith cancer in the/th CCOP, PI population o f / t h CCOP community or catchment area, Not a = 1, if the/th CCOP does not have the jth protocol available for the ith cancer; = 0 otherwise, Nil b : proportion of time the primary care physicians do not refer patients to the/th CCOP for the ith cancer, proportion of physicians of type m, in the/th CCOP area, who see and treat Nilmh : patients with the ith cancer, but are not in the/th CCOP, Niflc = proportion of patients with the ith cancer, not suitable for the jth protocol in the lth CCOP service area, Nijlrnd = proportion of time that the mth type of physician active in the/th CCOP does not select an eligible patient with the ith cancer for the jth protocol, Nijlme = proportion of time that the patient with the ith cancer rejects the jth protocol presented by the mth type of physician at the/th CCOP, Nijtf = proportion of time that the patient with the ith cancer who accepts the jth protocol is not entered onto the jth protocol by the/th CCOP, = proportion of time that the patient with the ith cancer enrolled on thejth protocol Not by the/th CCOP, fails to complete that protocol. Maximum accrual is achieved when all of the loss rates (Nk) are equal to zero. Thus, the extent to which a CCOP can minimize the loss rates that it faces will potentially increase accrual and protocol completions, thus improving CCOP productivity. These losses of potential patients can be treated as a quality analysis issue. That approach is represented in Figures 3 through 9 as "fishbone" or Ishikawa diagrams which indicate the potential independent variables to be considered in detailed studies of loss of potential accruals at each stage. A social scientist might choose to think of these diagrams as the basis for a regression model dealing with each of the relevant flow rates. The usual approach in quality control circles is to do a Pareto analysis of the sources of losses, identifying the most frequent causes and concentrating on them first. It should be noted that some of these relationships are very simplified in this flow diagram and in the Ishikawa diagrams. LEVEL BY LEVEL ANALYSIS The model presented here allows targeting of the sources of loss in productivity at a CCOP. To explore why these deficiencies are occurring requires a closer examination of
254
Hynes et al.
those factors that may affect the level where the greatest loss has occurred. This section discusses the factors that may be important.
1. CCOP Catchment Area Consider first the population of the CCOP service area (Pi) (Refer to Figure 2). As in all research, the population base actually served by a health delivery entity is sketchy at best. Presumably the population base served was considered at some time when the proposal for admission into CCOP was under consideration. Once that number is agreed on, it is then possible to assess the CCOP members' share of market. This market share estimate multiplied by the cancer prevalence (or incidence) rate (Ii), can be used to estimate the number of cancer cases that are likely to find their way to the CCOP's oncologists. The resultant would be the number of cases (Cai) that the CCOP could be expected to garner, if everything were working perfectly. This number would also produce a useful denominator for ratio analysis, 11 yielding variables such as accruals per eligible case or per thousand population. A second denominator would be the dollars spent on the program such as the dollars per accrual or dollars per accrual credit. Several alternative possibilities exist for share estimation. One approach would include the proportion of oncology beds in the CCOP component hospitals or the total proportion of the community's practicing oncologists. One important analytical issue is whether or not to focus only on the specific cancers targeted by the particular CCOP. For example, CCOPs with physician specialists in breast cancer may choose to concentrate their efforts on recruiting patients for breast cancer protocols only, and perhaps this fact should be taken into account in the model. Such an approach would simplify the data gathering, but it effectively limits the productivity studied to the internal operations of that CCOP and eliminates any focus on the inter-organizational productivity issues involving the CCOPs and their research bases and linkages with technology transfer processes within NCI.
2. Protocol Availability The availability of a protocol at a particular CCOP is the next transition rate affecting productivity (Na) (Figure 3). Four factors are hypothesized to affect this rate. First, the CCOP physicians and staff may not be aware of the existence of a particular protocol. Lack of awareness can be due to lack of motivation to learn about what is available either within the existing network of affiliated research bases or through outside cooperative groups, or due to the research base's failure to communicate about a protocol's availability to the local CCOP. Second, although aware of the protocol, the CCOP may not have access to a specific protocol, because their research bases do not offer it. Some research bases specialize their protocol development and research activities on certain cancers or treatment modalities. If there is no mechanism for access to protocols offered by research bases outside of their existing network, then the CCOP can not enroll patients on that particular protocol. Only by the NCI designation of a protocol as an intergroup protocol can a CCOP enroll a patient on a protocol through another research base. Third, even when the protocol is available through the CCOP/research base network, the CCOP community hospitals may not be able to accommodate the protocol. The hospital may not
Productivity in Clinical Research Programs
255
'i '°'~×/ N,
'f"
I
o
0
E E O
z~
J
z~
°~
O
Z. o
256
Hynesetal.
have the available technology, trained personnel, or capital budget to adopt particular protocols. Furthermore, the CCOP may not choose to adopt a protocol that is available and feasible because it is not a priority for the CCOP or its consortium members. This lack of priority may be due to a wide variety of reasons. There may be no champion for it in the organization, or it may just have been overlooked, and, thus not submitted to the CCOP IRB. It may have been considered and rejected by the CCOP IRB due to cost, low priority, or unacceptable (locally) risk to patients. Regarding this risk issue, a protocol may be deemed to have unacceptable risk due to a misconception or due to risk adverseness on the part of local providers. The protocol may also be stalled in local or IRB committee processes due to bureaucracy or due to a hung jury over technical facts or values.
3. CCOP Physician Utilization and Referral Rate The next transition rate concerns access to CCOP physicians (Figure 4). This is one of the more critical levels in the process. Access to CCOP physicians depends upon the physician environment in the CCOP area and the extent to which physicians and patients in the CCOP area utilize CCOP resources. The loss of potential patients (N9) may be due to political problems in the referral process or due to the success of competing providers. Some primary care providers may prefer to send their patients to other local non-CCOP oncologists. There may be perceived costs to the primary care physicians to refer patients to the CCOP. The physicians in the CCOP may have a weak image with their colleagues or there may be a personality clash. Competing providers in the CCOP catchment area may also affect the CCOP's market share of patients. Such competition could be in the form of other local oncologists, another CCOP, or an academic cancer center. It may be that coordination problems within the CCOP institutions have turned the patients away or that an oversupply of specialists has led to a splintering of the share of market available. Physicians may not be available to treat a particular cancer due either to overall short supply of physicians or to lack of recruitment effort by the CCOP. Patients may not have access to the appropriate level of physician specialists due to shortages of trained oncologists, scheduling problems in using the available personnel, or due to ineffective use of support staff. The CCOP also may not have recruited the specialist preferred by patients or their primary physicians. This problem can be due to a number of weaknesses on the part of the CCOP including level of physician recruitment effort, lack of support for protocol, poor coordination in the CCOP, unattractiveness of the hospital, or a weak champion for the CCOP within the institutional hierarchy. Patient characteristics and accessibility issues can also turn away potential patients. Patients' socioeconomic condition may block their referral to a CCOP physician. They may feel financially constrained; they may lack understanding about the referral process due to lack of basic education, or they may hold inappropriate beliefs about cancer and the medical treatment for it. The patient may be denied access to a CCOP physician because: they lack adequate health insurance or their health plan may restrict their provider choice, or they are already too ill to overcome any barriers to care, or they may live too far away to use the CCOP facility (in their own perceptions).
Productivity in Clinical Research Programs
257
8e~ @ @
,a
