BRIEF COMMUNICATION
Concordance of Hospital Cancer Registry- and Physician-Collected Data for Patients With Melanoma
Despite the proliferation of hospital tumor registries and the development of standardized high-quality software to support large cancer patient databases, many physicians continue to collect patient data for research and clinical care evaluation by conducting their own medical record reviews. Although the primary data source is the same in both cases—i.e., the patient's medical record—the variability of the collected data across these different collectors and their methods of collection is unknown. In this study, we examined the degree of agreement between physician-collected data and certified tumor registrar-collected data on six variables. These variables included stage at diagnosis (local versus regional or distant metastasis), site (location of primary lesion), histology (superficial spreading versus other specified versus not otherwise specified), depth of invasion as measured by Clark's Level (/), tumor size as measured by Breslow's measurement of tumor thickness (2), and date of diagnosis (year, month, and day). To assess the reliability of the data, we examined the frequency with which physicians and registrars had identical values for the above variables. We assessed the accuracy of stage (including Clark's Level) and site codes Vol. 84, No. 22, November 18, 1992
tumor size variables, with the large majority of the discrepant cases coded as unspecified by either the physician or the registrar. The main source of this information for both physician and registrar was the "final diagnosis" and microscopic description on the pathology report of the excisional biopsy (if available). Although 95% confidence intervals were somewhat wide, reflecting small sample size and unequal distribution of responses within cells, z scores for stage, site, and histology were statistically significant (3.58, P = .0004; 3.93, P = .0001; and 2.50, P = .01; respectively) and indicated agreement beyond chance level. On the month of diagnosis, however, registrars and physicians disagreed on over 40% of the cases, and agreement on the day of diagnosis was not above chance levels. The actual degree of discrepancy was slight, however, with the majority of discrepant dates (85%) falling within 1 month or less of each other. No consistent difference in the pattern of dating the time of diagnosis was discernible. The results of this preliminary study showed that the quality of tumor r e g i s t r a r - c o l l e c t e d and physician-collected datasets was highly variable specific. Stage and site variables were relatively reliable variables, and date of diagnosis was a relatively unreliable variable. Physician-collected data were shown to be less accurate than tumor registrar-collected data. This difference most likely reflects the registrars' training and adherence to abstracting and coding rules. The disagreement on date of diagnosis is potentially the most problematic because of its implications for survival
Downloaded from http://jnci.oxfordjournals.org/ at University of Sussex on August 23, 2015
Laurel Lockwood Hourani,* James Jakowatz, Matthew Goodman, Greg Mueller, Grace Brodie, AH Jaffery
by independently comparing discrepant values with the medical record. Case patients included all patients diagnosed and/or who had completed a first course of treatment at the medical center for a single primary cutaneous melanoma between January 1, 1984, and December 31, 1990, and for whom laboratory data were available (N = 60). Registry data were obtained from the University of California Irvine Medical Center Tumor Registry database software CANSUR/ NET (3). As part of a cancer program' approved by the California Tumor Registry (4,5) and the American College of Surgeons (6), quality control of data from the California Tumor Registry included computer data edits internal to the software itself (i.e., the software program prevents data entry of invalid codes) and comparative computerized data edits at the regional level. Physician-collected data were abstracted from physicians' office charts to data collection forms designed specifically for a retrospective study of biological markers in melanoma patients. This abstraction was performed by two surgical residents supervised by their attending physician (J. Jakowatz). Quality control included review of forms by a second-year research fellow (G. Mueller). The K test statistic was used as a measure of reliability that corrects for chance-expected agreement in categorical variables (7). Table 1 shows the amount of agreement in the datasets on the variables of interest. Physicians and registrars disagreed on six of the 48 cases that had specified stage of disease. In the majority of the discrepant cases, the physician had recorded the current stage rather than the stage at diagnosis. Concordance was very high for site, with only two disagreements out of 60 specified cases: One lesion on the lower back of the neck was coded to the trunk by a physician, and one back-of-shoulder lesion was coded to the trunk by a registrar. Complete agreement was found on Clark's Level, which is used as a staging variable by both the registrars and physicians. Less agreement was shown on histology and
Received April 20, 1992; revised September 9, 1992; accepted September 16, 1992. University of California Irvine Medical Center, Orange, Calif. We thank Ms. Susan Chavez, C.T.R., and the staff of the University of California Irvine Medical Center Tumor Registry for their support and assistance.
*Correspondence
to: Laurel Lockwood
Hourani, Ph.D., Clinical Cancer Center, University of California Irvine Medical Center, 101 The City Drive, Rt. 81, Bldg. 44, Orange, CA 92668.
BRIEF COMMUNICATION
1749
Table 1. Agreement between data collected by certified tumor registrars and physicians for patients with melanoma
Physician
Registrar Local 39 1
Regional or distant 5 3
Head or neck 10 0 0 0
Trunk 1 22 0 0
Stage at diagnosis Local Regional or distant Site Head or neck Trunk Upper limb Lower limb
Upper limb 0 1 14 0
Histology Superficial spreading Other specified Not otherwise specified
Superficial spreading 13 0 12
Other specified
2 6 5
Lower limb 0 0 0 12
Not otherwise specified g 2 10
K*
95% Confidence interval*
P
0.g8
0.48
0.22-0.74
.0004
0.96
0.95
0.83-1.0
.0001
0.48
0.21
0.04-0.37
.01
Observed agreement
1.0
NA
0.66
NA
NA
Date of diagnosis Year Month Day
0.93 0.58 0.23
NA NA NA
NA NA NA
•NA = not applicable.
analysis; i.e., length of survival is often calculated from date of diagnosis. Although unlikely to have an effect on melanoma survival rates, since melanoma has a relatively high 5-year survival rate, this problem may be important in cancers such as liver or pancreatic cancer, where average survival is a matter of months. The reliability of the data also appears to be dependent on the availability of the pathology report at the time of medical record abstraction. The fact that registrars typically abstracted from hospital charts of inpatients and from physicians' office charts of
outpatients underscores the importance of the timely receipt of copies of medical reports into both companion charts of a patient's unified medical record.
(4) CANCER SURVEILLANCE SECTION, STATE OF CALIFORNIA DEPARTMENT OF HEALTH
References
(5) SEIFFERT
(/) CLARK
WH
JR: The
histogenesis
STATE
HEALTH
OF CALIFORNIA
SERVICES:
DEPARTMENT
CANSUR/NET
OF
User
Manual, Version 1.34, California Ed., May 1991
J,
PRICE
W,
GORDON
B:
The
California tumor registry: A state-of-the-art model for a regionalized, automated, population-based registry. Top Health Rec Manage 11:59-73, 1990
and
biological behavior of primary malignant melanoma of the skin. Cancer Res 29:705717, 1969 (2) BRESLOW A: Prognosis in cutaneous melanoma: Tumor thickness as a guide to treatment. Pathol Annu Part 1:1-20, 1980 (i)
SERVICES: Cancer Reporting in California: Abstracting and Coding Procedures for Hospitals, vol I, 2nd ed. California Cancer Reporting System Standards. Emeryville, Calif: State of California Department of Health Services, 1989
(6) COMMITTEE ON APPROVALS, AMERICAN COLLEGE OF SURGEONS, COMMISSION ON
CANCER: Cancer Program Manual, 1991. Chicago: American College of Surgeons, 1991 (7) FLEISS JL: Statistical Methods for Rates and Proportions, 2nd ed. New York: John Wiley & Sons, 1981
Need the latest information on clinical trials for HIV and AIDS? Call the AIDS Clinical Trials Information Service:
1-800-TRIALS-A (1-800-874-2572) Monday through Friday, 9 a.m. to 7 p.m. eastern time. • Free up-to-date information about more than 200 federally and privately sponsored HIV and AIDS clinical trials and the drugs used in the trials.
Personalized assistance from English- and Spanish-speaking health specialists. TTY/TDD access: 1-800-243-7012.
A service of the U.S. Department of Health and Human Services. Public Health Service. The Centers for Disease Control, Food and Drug Administration. National-Institute of Allergy and Infectious Diseases, and National Library of Medicine have collaborated to provide this service.
1750
Journal of the National Cancer Institute
Downloaded from http://jnci.oxfordjournals.org/ at University of Sussex on August 23, 2015
1.0
Tumor size—Breslow's measurement of thickness, mm
Clark's Level—depth of invasion
Concordance of Hospital Cancer Registry- and Physician-Collected Data for Patients With Melanoma
Despite the proliferation of hospital tumor registries and the development of standardized high-quality software to support large cancer patient databases, many physicians continue to collect patient data for research and clinical care evaluation by conducting their own medical record reviews. Although the primary data source is the same in both cases—i.e., the patient's medical record—the variability of the collected data across these different collectors and their methods of collection is unknown. In this study, we examined the degree of agreement between physician-collected data and certified tumor registrar-collected data on six variables. These variables included stage at diagnosis (local versus regional or distant metastasis), site (location of primary lesion), histology (superficial spreading versus other specified versus not otherwise specified), depth of invasion as measured by Clark's Level (/), tumor size as measured by Breslow's measurement of tumor thickness (2), and date of diagnosis (year, month, and day). To assess the reliability of the data, we examined the frequency with which physicians and registrars had identical values for the above variables. We assessed the accuracy of stage (including Clark's Level) and site codes Vol. 84, No. 22, November 18, 1992
tumor size variables, with the large majority of the discrepant cases coded as unspecified by either the physician or the registrar. The main source of this information for both physician and registrar was the "final diagnosis" and microscopic description on the pathology report of the excisional biopsy (if available). Although 95% confidence intervals were somewhat wide, reflecting small sample size and unequal distribution of responses within cells, z scores for stage, site, and histology were statistically significant (3.58, P = .0004; 3.93, P = .0001; and 2.50, P = .01; respectively) and indicated agreement beyond chance level. On the month of diagnosis, however, registrars and physicians disagreed on over 40% of the cases, and agreement on the day of diagnosis was not above chance levels. The actual degree of discrepancy was slight, however, with the majority of discrepant dates (85%) falling within 1 month or less of each other. No consistent difference in the pattern of dating the time of diagnosis was discernible. The results of this preliminary study showed that the quality of tumor r e g i s t r a r - c o l l e c t e d and physician-collected datasets was highly variable specific. Stage and site variables were relatively reliable variables, and date of diagnosis was a relatively unreliable variable. Physician-collected data were shown to be less accurate than tumor registrar-collected data. This difference most likely reflects the registrars' training and adherence to abstracting and coding rules. The disagreement on date of diagnosis is potentially the most problematic because of its implications for survival
Downloaded from http://jnci.oxfordjournals.org/ at University of Sussex on August 23, 2015
Laurel Lockwood Hourani,* James Jakowatz, Matthew Goodman, Greg Mueller, Grace Brodie, AH Jaffery
by independently comparing discrepant values with the medical record. Case patients included all patients diagnosed and/or who had completed a first course of treatment at the medical center for a single primary cutaneous melanoma between January 1, 1984, and December 31, 1990, and for whom laboratory data were available (N = 60). Registry data were obtained from the University of California Irvine Medical Center Tumor Registry database software CANSUR/ NET (3). As part of a cancer program' approved by the California Tumor Registry (4,5) and the American College of Surgeons (6), quality control of data from the California Tumor Registry included computer data edits internal to the software itself (i.e., the software program prevents data entry of invalid codes) and comparative computerized data edits at the regional level. Physician-collected data were abstracted from physicians' office charts to data collection forms designed specifically for a retrospective study of biological markers in melanoma patients. This abstraction was performed by two surgical residents supervised by their attending physician (J. Jakowatz). Quality control included review of forms by a second-year research fellow (G. Mueller). The K test statistic was used as a measure of reliability that corrects for chance-expected agreement in categorical variables (7). Table 1 shows the amount of agreement in the datasets on the variables of interest. Physicians and registrars disagreed on six of the 48 cases that had specified stage of disease. In the majority of the discrepant cases, the physician had recorded the current stage rather than the stage at diagnosis. Concordance was very high for site, with only two disagreements out of 60 specified cases: One lesion on the lower back of the neck was coded to the trunk by a physician, and one back-of-shoulder lesion was coded to the trunk by a registrar. Complete agreement was found on Clark's Level, which is used as a staging variable by both the registrars and physicians. Less agreement was shown on histology and
Received April 20, 1992; revised September 9, 1992; accepted September 16, 1992. University of California Irvine Medical Center, Orange, Calif. We thank Ms. Susan Chavez, C.T.R., and the staff of the University of California Irvine Medical Center Tumor Registry for their support and assistance.
*Correspondence
to: Laurel Lockwood
Hourani, Ph.D., Clinical Cancer Center, University of California Irvine Medical Center, 101 The City Drive, Rt. 81, Bldg. 44, Orange, CA 92668.
BRIEF COMMUNICATION
1749
Table 1. Agreement between data collected by certified tumor registrars and physicians for patients with melanoma
Physician
Registrar Local 39 1
Regional or distant 5 3
Head or neck 10 0 0 0
Trunk 1 22 0 0
Stage at diagnosis Local Regional or distant Site Head or neck Trunk Upper limb Lower limb
Upper limb 0 1 14 0
Histology Superficial spreading Other specified Not otherwise specified
Superficial spreading 13 0 12
Other specified
2 6 5
Lower limb 0 0 0 12
Not otherwise specified g 2 10
K*
95% Confidence interval*
P
0.g8
0.48
0.22-0.74
.0004
0.96
0.95
0.83-1.0
.0001
0.48
0.21
0.04-0.37
.01
Observed agreement
1.0
NA
0.66
NA
NA
Date of diagnosis Year Month Day
0.93 0.58 0.23
NA NA NA
NA NA NA
•NA = not applicable.
analysis; i.e., length of survival is often calculated from date of diagnosis. Although unlikely to have an effect on melanoma survival rates, since melanoma has a relatively high 5-year survival rate, this problem may be important in cancers such as liver or pancreatic cancer, where average survival is a matter of months. The reliability of the data also appears to be dependent on the availability of the pathology report at the time of medical record abstraction. The fact that registrars typically abstracted from hospital charts of inpatients and from physicians' office charts of
outpatients underscores the importance of the timely receipt of copies of medical reports into both companion charts of a patient's unified medical record.
(4) CANCER SURVEILLANCE SECTION, STATE OF CALIFORNIA DEPARTMENT OF HEALTH
References
(5) SEIFFERT
(/) CLARK
WH
JR: The
histogenesis
STATE
HEALTH
OF CALIFORNIA
SERVICES:
DEPARTMENT
CANSUR/NET
OF
User
Manual, Version 1.34, California Ed., May 1991
J,
PRICE
W,
GORDON
B:
The
California tumor registry: A state-of-the-art model for a regionalized, automated, population-based registry. Top Health Rec Manage 11:59-73, 1990
and
biological behavior of primary malignant melanoma of the skin. Cancer Res 29:705717, 1969 (2) BRESLOW A: Prognosis in cutaneous melanoma: Tumor thickness as a guide to treatment. Pathol Annu Part 1:1-20, 1980 (i)
SERVICES: Cancer Reporting in California: Abstracting and Coding Procedures for Hospitals, vol I, 2nd ed. California Cancer Reporting System Standards. Emeryville, Calif: State of California Department of Health Services, 1989
(6) COMMITTEE ON APPROVALS, AMERICAN COLLEGE OF SURGEONS, COMMISSION ON
CANCER: Cancer Program Manual, 1991. Chicago: American College of Surgeons, 1991 (7) FLEISS JL: Statistical Methods for Rates and Proportions, 2nd ed. New York: John Wiley & Sons, 1981
Need the latest information on clinical trials for HIV and AIDS? Call the AIDS Clinical Trials Information Service:
1-800-TRIALS-A (1-800-874-2572) Monday through Friday, 9 a.m. to 7 p.m. eastern time. • Free up-to-date information about more than 200 federally and privately sponsored HIV and AIDS clinical trials and the drugs used in the trials.
Personalized assistance from English- and Spanish-speaking health specialists. TTY/TDD access: 1-800-243-7012.
A service of the U.S. Department of Health and Human Services. Public Health Service. The Centers for Disease Control, Food and Drug Administration. National-Institute of Allergy and Infectious Diseases, and National Library of Medicine have collaborated to provide this service.
1750
Journal of the National Cancer Institute
Downloaded from http://jnci.oxfordjournals.org/ at University of Sussex on August 23, 2015
1.0
Tumor size—Breslow's measurement of thickness, mm
Clark's Level—depth of invasion
