International Journal of Technology Assessment In Health Care, 8:4 (1992), 683-693. Copyright © 1992 Cambridge University Press. Printed in the U.S.A.
THE CONSENSUS DEVELOPMENT PROGRAM Detecting Changes in Medical Practice following a Consensus Conference on the Treatment of Prostate Cancer Charles R. Sherman U.S. National Institutes of Health
Arnold L. Potosky U.S. National Cancer Institute
Kathleen A. Weis U.S. Agency for Health Care Policy and Research
John H. Ferguson U.S. National Institutes of Health
Abstract The treatment of prostate cancer was reviewed at a U.S. National Institutes of Health Consensus Development Conference in June 1987. Data from the U.S. National Cancer Institute's Surveillance, Epidemiology, and End Results tumor registries were analyzed and showed that the proportion of eligible prostate cancer patients receiving the recommended therapies did not increase at a faster rate after the conference than before.
Consensus development panels are convened on appropriate occasions in the hope of expediting the transfer of new medical technologies into the practice of medicine for the benefit of patients. Several European countries have or are establishing new consensus development programs based, in part, on the 13-year experience of the Consensus Development Program (CDP) at the National Institutes of Health (NIH) in the United States. However, the effectiveness of such programs in influencing medical practice is not well studied. As part of a renewed commitment by the NIH to evaluating the CDP's influence on medical practice, a variety of methods are being used to monitor and evaluate changes in physician behavior following a consensus conference. This paper describes one such evaluation strategy that assesses trends in patterns of treatment using secondary clinical data from two large databases that were not originally designed for this purpose. 683
Sherman et al.
This approach contributes to the understanding of the potential uses and limitations of these databases for evaluating trends in treatment patterns. "The Management of Clinically Localized Prostate Cancer" was the subject of a Consensus Development Conference (CDC) held at the NIH in June 1987. For three days, 14 expert panelists heard and questioned 23 speakers and received comments from many people, both in a local public audience of 271 and in remote audiences of over 675 others who watched through closed-circuit televisions at 168 hospitals and cancer centers across the United States. The final consensus statement recommended the use of radical prostatectomy as the definitive treatment for patients with tumors that are confined to the prostate gland and the use of radiation therapy as the definitive treatment for those with tumors that are either confined to the prostate or limited to periprostatic tissue. Treatments of patients with incidental carcinoma of the prostate and patients with evidence of metastasis were not addressed in these recommendations. The panel also recommended the broad adoption of uniform staging and treatment outcome classifications and urged further basic and long-term clinical research. The consensus statement was widely disseminated via direct mailings to specialty societies and individual physicians and via articles in mass print media. Publication in two major biomedical journals occurred some time afterward (24;25). The principal objective of this analysis is to evaluate whether trends in the treatment of patients with localized prostate cancer changed following the conference in ways that are consistent with the recommendations of the consensus statement. Did the proportion of eligible patients with prostate cancer who received either radiation therapy or radical prostatectomy as the definitive, first course of therapy increase at a faster rate after the CDC was convened than before? The comparative assessment of the utility of two large databases for this analysis is a secondary objective of the study. METHODS
To evaluate the effectiveness of the CDP on medical practice, time series data were derived from two sources: the U.S. National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER) tumor registries and the Medicare Statistical System. SEER Program Data
The SEER Program is a comprehensive system for tracking the incidence of cancer and patient survival in the United States (12;29) and frequently has been quoted as the source of national estimates of cancer incidence and survival (1;2;5;22). Additionally, the data have also been used for a wide variety of studies relating to the prevention and treatment of cancer (17). SEER data are derived from a series of population-based tumor registries in nine geographically distinct areas, covering approximately 10% of the U.S. population. The goal of each SEER registry is to collect information on all of the cancer patients who live in a defined geographic area at the time of diagnosis using standardized abstracts of hospital, clinic, and nursing home records, death certificates, and records from private laboratories and radiotherapy units in order to obtain complete disease histories. Information such as the stage of the cancer at diagnosis, histology, tumor grade, and vital status at follow-up have been routinely collected since 1973. The data are also subject to rigorous quality control procedures (12). Information on the first course of cancer-directed therapy has been available since 1983. Patient care is tracked for four months following initial treatment, even if pa684
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
Impact of consensus on prostate cancer treatment
tients transfer between hospitals. Medical records are abstracted to obtain information from inpatient stays as well as outpatient visits to hospitals and radiation clinics. Such comprehensive tracking is required to avoid double-counting of patients and to ensure the completeness of treatment records. In the case of prostate cancer, this is particularly necessary because men whose tumors are initially detected via transurethral resection of the prostate (TURP) in one hospital may later receive either radical surgery or radiation therapy in another location. The denominators for calculating the percentage of patients who were receiving the recommended therapies included new cases (56,406) of prostate cancer with available staging information in the SEER tumor registry, diagnosed from January 1983 through December 1989. Although the month of diagnosis is available, cases were grouped by quarter-year of diagnosis to minimize variability while providing enough observations (28 consecutive quarters from 1983 through 1989) to discern any pattern of change in treatment following the conference. A total of 34,606 cases with tumors confined to the prostate ("localized") were identified. There were an additional 7,100 cases with stage-C disease (tumor spread to periprostatic tissue), a condition for which radiation therapy, without surgery, was recommended by the consensus statement. Cases with evidence of metastases were excluded from all analyses. Three time series were tested independently to determine if the percentages of patients having any of the recommended therapies increased at a faster rate after the conference: 1. percentage of localized tumors treated with radical prostatectomy; 2. percentage of localized or stage-C tumors treated with radiation; and 3. percentage of localized or stage-C tumors treated with either radical prostatectomy or radiation. Men receiving both radical prostatectomy and radiation were counted in time series #1 but not in #2, since it is likely that in such cases the radiation was prescribed as adjuvant therapy, not as the definitive first course of therapy. Time series #3 was added because the recommendation permitted either therapy for localized tumors. Analyses of radical prostatectomy were also performed separately for patients who were 70 years old and older (62% of all patients) and for those younger than 70 (38%). This division was necessary because elderly men typically are not recommended as candidates for radical prostatectomy (4;28). A linear "piecewise" regression was used to model all five time series to test for the possibility of a change occurring at the time of the conference (27). This model allows estimates of time trends in the percentages of cases that received the treatments under investigation, as well as estimates of changes in the trends following the CDC. (The technical note below contains a more detailed description of the approach.) A logistic regression also was performed using patient-level data to check for consistency. The results were found to be in substantial agreement with those using grouped data. Medicare Claims Data
Data from the U.S. Medicare Statistical System also were used in this study. Medicare is the federally funded insurance program administered by the Health Care Financing Administration (HCFA) for the elderly (ages 65 and older) and disabled. Claims for covered medical services that are provided to Medicare beneficiaries are maintained by HCFA for program evaluation. The records include claims for hospital inpatient facility charges, hospital outpatient facility charges, and charges submitted by physiINTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
685
Sherman et al.
cians for professional services and suppliers of durable medical supplies. More detailed descriptions of Medicare claims data are available elsewhere (21). HCFA maintains a separate file of claims records for a 5% sample of patients that are linkable, for individual patients, longitudinally across all providers. The 5% random sample files for hospital inpatient and outpatient services were used in this study. Both files contain a Health Insurance Claim number (HIC), by which the claims are linked, up to five diagnosis codes, and up to three procedure codes (ICD-9-CM codes). In records after April 1987, HCFA common procedure codes (HCPCS) are used to code procedures in the outpatient file. To assess changes in treatment of prostate cancer, mutually exclusive cohorts were identified for each calendar year from 1984 through 1989. All records for both inpatient stays and visits to hospital outpatient clinics were examined to identify the earliest date of care for any of the five diagnoses that are coded as prostate cancer (ICD-9-CM = 185). In an attempt to remove prevalent cases, the 1984 and 1985 cohorts were dropped. Data for 1989 were not complete enough at the time of this study to identify a full cohort but were used to follow patients who had been previously identified. A total of 18,793 cases, diagnosed from 1986 through 1988, remained for analysis of trends in prostate cancer treatment. The annual cohorts were followed to determine whether radical prostatectomy or radiation therapy was administered at any time following the earliest date associated with the index diagnosis of prostate cancer. The percentages of all cases of prostate cancer that were treated with radiation therapy, with radical prostatectomy, and with either radiation or radical prostatectomy were calculated. Comparably defined SEER data for patients over age 65, regardless of tumor stage, were summarized for comparison with the results that were obtained from Medicare claims data.
RESULTS Results of the SEER Analysis
The SEER program data showing the three time series for treatment of prostate tumors of men of all ages are presented in Figure 1; Table 1 shows the estimates of the regression parameters and associated levels of significance. The bottom series in Figure 1 shows a pattern of increasing use of radical prostatectomy in the treatment of localized prostate cancer beginning over two years before the NIH CDC; the overall rate of change between 1983 and 1989 of 2% per year did not increase following the conference. During the entire time period, there was no change in the use of radiation therapy (middle line of Figure 1), although there was a very small increase following the conference that was of borderline statistical significance (p=.O47). To assess the possibility that increases in radiation were offset by substituting surgery for radiation, we also examined the percentage of cases that received either therapy, shown in the top series. The percentage increased steadily by almost 2% per year between 1983 and 1989; there was no detectable change following the conference. For patients ages 70 and over (Figure 2), the use of radical prostatectomy is low and, while its use increased by about 1% per year overall between 1983 and 1989, the rate of increase did not change after the conference (Table 1). The same result was observed for the group of younger patients, but the overall change between 1983 and 1989 was greater. As in the case for older men, there was no observed conference effect.
686
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
Impact of consensus on prostate cancer treatment NIH Consensus Conference
-•-
Radiation or Surgery*
-•-
Radiation*
- * - Surgery" * Treatment for stage A or B or C ° Treatment for stage A or B
Quarter Year
Figure 1. Percentage of localized prostate cancers treated with radical prostatectomy and radiation, 1983-1989.
Results of the Medicare Analysis
Figure 3 compares the trends in Medicare and SEER data with respect to the percentages of men ages 65 and over who were treated with radiation, radical prostatectomy, and either radiation or prostatectomy. The percentage of cases that were treated with radiation appears in the Medicare data to be less than one-fifth of that observed in the SEER data, and the SEER trend of increasing use of radiation is not apparent in Medicare. For those having radical prostatectomy, the Medicare data show only half the percentage of cases that SEER does; however, both sources show an increase in the use of surgery in the calendar year (1988) that began six months following the con-
Table 1. Trends in the Treatment of Localized Prostate Cancer by Quarter-year of Diagnosis, 1983-1989
Age of patients
Treatment
All ages
Radical prostatectomy
All ages
Radiation
All ages Age 70 or older
Either radiation or radical prostatectomy Radical prostatectomy
Age 69 or younger
Radical prostatectomy
Overall trend .53** (.03) .00 (.04) .44** (.04) .27** (0.3) 1.48** (.10)
Change in trend (after CDC) -.01 (.02) .04* (.02) .02 (.02) .01 (.01) -.04 (.05)
* p< .05 **p< .01
Measurement unit is quarterly change in the percentage of patients receiving named treatment. Standard error is in parentheses.
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
687
Sherman et al. NIH Consensus Conference 70
Ages
THE CONSENSUS DEVELOPMENT PROGRAM Detecting Changes in Medical Practice following a Consensus Conference on the Treatment of Prostate Cancer Charles R. Sherman U.S. National Institutes of Health
Arnold L. Potosky U.S. National Cancer Institute
Kathleen A. Weis U.S. Agency for Health Care Policy and Research
John H. Ferguson U.S. National Institutes of Health
Abstract The treatment of prostate cancer was reviewed at a U.S. National Institutes of Health Consensus Development Conference in June 1987. Data from the U.S. National Cancer Institute's Surveillance, Epidemiology, and End Results tumor registries were analyzed and showed that the proportion of eligible prostate cancer patients receiving the recommended therapies did not increase at a faster rate after the conference than before.
Consensus development panels are convened on appropriate occasions in the hope of expediting the transfer of new medical technologies into the practice of medicine for the benefit of patients. Several European countries have or are establishing new consensus development programs based, in part, on the 13-year experience of the Consensus Development Program (CDP) at the National Institutes of Health (NIH) in the United States. However, the effectiveness of such programs in influencing medical practice is not well studied. As part of a renewed commitment by the NIH to evaluating the CDP's influence on medical practice, a variety of methods are being used to monitor and evaluate changes in physician behavior following a consensus conference. This paper describes one such evaluation strategy that assesses trends in patterns of treatment using secondary clinical data from two large databases that were not originally designed for this purpose. 683
Sherman et al.
This approach contributes to the understanding of the potential uses and limitations of these databases for evaluating trends in treatment patterns. "The Management of Clinically Localized Prostate Cancer" was the subject of a Consensus Development Conference (CDC) held at the NIH in June 1987. For three days, 14 expert panelists heard and questioned 23 speakers and received comments from many people, both in a local public audience of 271 and in remote audiences of over 675 others who watched through closed-circuit televisions at 168 hospitals and cancer centers across the United States. The final consensus statement recommended the use of radical prostatectomy as the definitive treatment for patients with tumors that are confined to the prostate gland and the use of radiation therapy as the definitive treatment for those with tumors that are either confined to the prostate or limited to periprostatic tissue. Treatments of patients with incidental carcinoma of the prostate and patients with evidence of metastasis were not addressed in these recommendations. The panel also recommended the broad adoption of uniform staging and treatment outcome classifications and urged further basic and long-term clinical research. The consensus statement was widely disseminated via direct mailings to specialty societies and individual physicians and via articles in mass print media. Publication in two major biomedical journals occurred some time afterward (24;25). The principal objective of this analysis is to evaluate whether trends in the treatment of patients with localized prostate cancer changed following the conference in ways that are consistent with the recommendations of the consensus statement. Did the proportion of eligible patients with prostate cancer who received either radiation therapy or radical prostatectomy as the definitive, first course of therapy increase at a faster rate after the CDC was convened than before? The comparative assessment of the utility of two large databases for this analysis is a secondary objective of the study. METHODS
To evaluate the effectiveness of the CDP on medical practice, time series data were derived from two sources: the U.S. National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER) tumor registries and the Medicare Statistical System. SEER Program Data
The SEER Program is a comprehensive system for tracking the incidence of cancer and patient survival in the United States (12;29) and frequently has been quoted as the source of national estimates of cancer incidence and survival (1;2;5;22). Additionally, the data have also been used for a wide variety of studies relating to the prevention and treatment of cancer (17). SEER data are derived from a series of population-based tumor registries in nine geographically distinct areas, covering approximately 10% of the U.S. population. The goal of each SEER registry is to collect information on all of the cancer patients who live in a defined geographic area at the time of diagnosis using standardized abstracts of hospital, clinic, and nursing home records, death certificates, and records from private laboratories and radiotherapy units in order to obtain complete disease histories. Information such as the stage of the cancer at diagnosis, histology, tumor grade, and vital status at follow-up have been routinely collected since 1973. The data are also subject to rigorous quality control procedures (12). Information on the first course of cancer-directed therapy has been available since 1983. Patient care is tracked for four months following initial treatment, even if pa684
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
Impact of consensus on prostate cancer treatment
tients transfer between hospitals. Medical records are abstracted to obtain information from inpatient stays as well as outpatient visits to hospitals and radiation clinics. Such comprehensive tracking is required to avoid double-counting of patients and to ensure the completeness of treatment records. In the case of prostate cancer, this is particularly necessary because men whose tumors are initially detected via transurethral resection of the prostate (TURP) in one hospital may later receive either radical surgery or radiation therapy in another location. The denominators for calculating the percentage of patients who were receiving the recommended therapies included new cases (56,406) of prostate cancer with available staging information in the SEER tumor registry, diagnosed from January 1983 through December 1989. Although the month of diagnosis is available, cases were grouped by quarter-year of diagnosis to minimize variability while providing enough observations (28 consecutive quarters from 1983 through 1989) to discern any pattern of change in treatment following the conference. A total of 34,606 cases with tumors confined to the prostate ("localized") were identified. There were an additional 7,100 cases with stage-C disease (tumor spread to periprostatic tissue), a condition for which radiation therapy, without surgery, was recommended by the consensus statement. Cases with evidence of metastases were excluded from all analyses. Three time series were tested independently to determine if the percentages of patients having any of the recommended therapies increased at a faster rate after the conference: 1. percentage of localized tumors treated with radical prostatectomy; 2. percentage of localized or stage-C tumors treated with radiation; and 3. percentage of localized or stage-C tumors treated with either radical prostatectomy or radiation. Men receiving both radical prostatectomy and radiation were counted in time series #1 but not in #2, since it is likely that in such cases the radiation was prescribed as adjuvant therapy, not as the definitive first course of therapy. Time series #3 was added because the recommendation permitted either therapy for localized tumors. Analyses of radical prostatectomy were also performed separately for patients who were 70 years old and older (62% of all patients) and for those younger than 70 (38%). This division was necessary because elderly men typically are not recommended as candidates for radical prostatectomy (4;28). A linear "piecewise" regression was used to model all five time series to test for the possibility of a change occurring at the time of the conference (27). This model allows estimates of time trends in the percentages of cases that received the treatments under investigation, as well as estimates of changes in the trends following the CDC. (The technical note below contains a more detailed description of the approach.) A logistic regression also was performed using patient-level data to check for consistency. The results were found to be in substantial agreement with those using grouped data. Medicare Claims Data
Data from the U.S. Medicare Statistical System also were used in this study. Medicare is the federally funded insurance program administered by the Health Care Financing Administration (HCFA) for the elderly (ages 65 and older) and disabled. Claims for covered medical services that are provided to Medicare beneficiaries are maintained by HCFA for program evaluation. The records include claims for hospital inpatient facility charges, hospital outpatient facility charges, and charges submitted by physiINTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
685
Sherman et al.
cians for professional services and suppliers of durable medical supplies. More detailed descriptions of Medicare claims data are available elsewhere (21). HCFA maintains a separate file of claims records for a 5% sample of patients that are linkable, for individual patients, longitudinally across all providers. The 5% random sample files for hospital inpatient and outpatient services were used in this study. Both files contain a Health Insurance Claim number (HIC), by which the claims are linked, up to five diagnosis codes, and up to three procedure codes (ICD-9-CM codes). In records after April 1987, HCFA common procedure codes (HCPCS) are used to code procedures in the outpatient file. To assess changes in treatment of prostate cancer, mutually exclusive cohorts were identified for each calendar year from 1984 through 1989. All records for both inpatient stays and visits to hospital outpatient clinics were examined to identify the earliest date of care for any of the five diagnoses that are coded as prostate cancer (ICD-9-CM = 185). In an attempt to remove prevalent cases, the 1984 and 1985 cohorts were dropped. Data for 1989 were not complete enough at the time of this study to identify a full cohort but were used to follow patients who had been previously identified. A total of 18,793 cases, diagnosed from 1986 through 1988, remained for analysis of trends in prostate cancer treatment. The annual cohorts were followed to determine whether radical prostatectomy or radiation therapy was administered at any time following the earliest date associated with the index diagnosis of prostate cancer. The percentages of all cases of prostate cancer that were treated with radiation therapy, with radical prostatectomy, and with either radiation or radical prostatectomy were calculated. Comparably defined SEER data for patients over age 65, regardless of tumor stage, were summarized for comparison with the results that were obtained from Medicare claims data.
RESULTS Results of the SEER Analysis
The SEER program data showing the three time series for treatment of prostate tumors of men of all ages are presented in Figure 1; Table 1 shows the estimates of the regression parameters and associated levels of significance. The bottom series in Figure 1 shows a pattern of increasing use of radical prostatectomy in the treatment of localized prostate cancer beginning over two years before the NIH CDC; the overall rate of change between 1983 and 1989 of 2% per year did not increase following the conference. During the entire time period, there was no change in the use of radiation therapy (middle line of Figure 1), although there was a very small increase following the conference that was of borderline statistical significance (p=.O47). To assess the possibility that increases in radiation were offset by substituting surgery for radiation, we also examined the percentage of cases that received either therapy, shown in the top series. The percentage increased steadily by almost 2% per year between 1983 and 1989; there was no detectable change following the conference. For patients ages 70 and over (Figure 2), the use of radical prostatectomy is low and, while its use increased by about 1% per year overall between 1983 and 1989, the rate of increase did not change after the conference (Table 1). The same result was observed for the group of younger patients, but the overall change between 1983 and 1989 was greater. As in the case for older men, there was no observed conference effect.
686
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
Impact of consensus on prostate cancer treatment NIH Consensus Conference
-•-
Radiation or Surgery*
-•-
Radiation*
- * - Surgery" * Treatment for stage A or B or C ° Treatment for stage A or B
Quarter Year
Figure 1. Percentage of localized prostate cancers treated with radical prostatectomy and radiation, 1983-1989.
Results of the Medicare Analysis
Figure 3 compares the trends in Medicare and SEER data with respect to the percentages of men ages 65 and over who were treated with radiation, radical prostatectomy, and either radiation or prostatectomy. The percentage of cases that were treated with radiation appears in the Medicare data to be less than one-fifth of that observed in the SEER data, and the SEER trend of increasing use of radiation is not apparent in Medicare. For those having radical prostatectomy, the Medicare data show only half the percentage of cases that SEER does; however, both sources show an increase in the use of surgery in the calendar year (1988) that began six months following the con-
Table 1. Trends in the Treatment of Localized Prostate Cancer by Quarter-year of Diagnosis, 1983-1989
Age of patients
Treatment
All ages
Radical prostatectomy
All ages
Radiation
All ages Age 70 or older
Either radiation or radical prostatectomy Radical prostatectomy
Age 69 or younger
Radical prostatectomy
Overall trend .53** (.03) .00 (.04) .44** (.04) .27** (0.3) 1.48** (.10)
Change in trend (after CDC) -.01 (.02) .04* (.02) .02 (.02) .01 (.01) -.04 (.05)
* p< .05 **p< .01
Measurement unit is quarterly change in the percentage of patients receiving named treatment. Standard error is in parentheses.
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 8:4, 1992
687
Sherman et al. NIH Consensus Conference 70
Ages
