Volume 164 l\umber I. Part I
membranes may have an increased risk of intrauterine growth retardation. Wladimiroff and CampbeiF studied urine production in fetuses with intrauterine growth retardation and found that urine output was much lower than that expected for a normally grown fetus in three of 12 determinations. In this study fetal urine output in pregnancies complicated by premature rupture of membranes was not reduced and did not vary as a function of residual amniotic fluid volume. Fetal urine production was unchanged even in the presence of severe oligohydramnios. We conclude that residual amniotic fluid volume
Amniotic fluid after premature rupture of membranes
after premature rupture of membranes does not depend on, or alter, fetal renal function. Residual amniotic fluid volume may depend on mechanical factors, such as maternal position, the site of leakage, and the size of the defect in the fetal membranes. REFERENCES I. Campbell S, Wladimiroff JW, Dewhurst CJ. The antenatal
measurement of fetal urine production. J Obstet Gynaecol Br Commonw 1973;80:680-6. 2. Wladimiroff JW, Campbell S. Fetal urine production rates in normal and complicated pregnancies. Lancet 1974; 1: 151-4.
The diagnosis of ovarian cancer by pathologists: How often do diagnoses by contributing pathologists agree with a panel of gynecologic pathologists? Carl W. Tyler, Jr., MD,. Nancy C. Lee, MD: Stanley J. Robboy, MD; RobertJ. Kurman, MD,d Allen L. Paris, MD,-·f Phyllis A. Wingo, MS: and G. David Williamson, PhD" Atlanta, Georgia, Newark, New Jersey, Baltimore, Maryland, and West Plains, Missouri The Cancer and Steroid Hormone Study, a multicenter, population-based, case-control study of ovarian, breast, and endometrial cancer in women 20 to 54 years of age, permitted the diagnoses of contributing pathologists to be compared with those of a panel of three gynecologic pathologists. A diagnosis of ovarian cancer was made by contributing pathologists on 477 subjects. Agreement between the two groups of pathologists was 97% for primary epithelial ovarian cancer and 89% for primary nonepithelial ovarian malignancies. Agreement on diagnosis of major cellular subtypes of ovarian malignancy ranged between 73% for endometrioid cancer and 100% for clear cell carcinomas. We conclude that the diagnosis of pathologic features of primary ovarian cancer is highly predictable. Nonetheless, diagnosis by histologic type varies sufficiently that a review process should be considered for clinical or investigative decisions involving specific histologic diagnoses of ovarian cancer. (AM J OBSTET GVNECOL 1991 ;164: 65-70.)
Key words: Ovarian neoplasms, diagnosis, predictive value of tests, pathology From the Office of the Director, Centers for Disease Control," Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control/ Department of Pathology, New Jersey Medical School,' Departments of Gynecology, Obstetrics and Pathology, Johns Hopkins Hospital, d Centerfor Environmental Health and Injury Control, Centers for Disease Control,' Department of Pathology, Ozarks Medical Center,! and Epidemiology Program Office, Centers for Disease Control.' The Cancer and Steroid Hormone Study was supported by interagency agreement 3-Y01-HD-8-I037 between the Centers for Disease Control and the National Institute of Child Health and Human Development, with additional support from the National Cancer Institute. Received for publication November 30, 1989; revised August 6, 1990; accepted August 13, 1990. Reprint requests: Carl W. T.vler,Jr., MD, Centers for Disease Control, Mailstop E-42, Atlanta, GA 30333. 6/1/24575
Ovarian cancer is a significant cause of death for women in the United States. In 1989 an estimated 20,000 American women were newly diagnosed as having ovarian cancer, and 12,000 died from this malignancy. I The diagnosis of ovarian cancer by pathologists not only guides patient treatment but also provides the basis for evaluation of different approaches to treatment in multicenter clinical trials and in epidemiologic studies of risk factors. As an illustration, if a cancer that metastasized to the ovary were not diagnosed correctly on a pathology report, the patient might be treated as if the tumor were primary in the ovary when another site was the one of origin; this might lead to inappro-
65
66
Tyler et al.
priate therapy. If this occurred often enough, studies that evaluate surgical or chemotherapeutic approaches to primary ovarian cancer would incorrectly include tumors from other organs. This problem would also extend to epidemiologic investigations of the causes of ovarian cancer. In the period 1980 to 1982 the Centers for Disease Control, with support from the National Institute of Child Health and Human Development and from the National Cancer Institute, conducted the Cancer and Steroid Hormone Study, a multicenter, populationbased case-control study of ovarian, breast, and endometrial cancer in women 20 to 54 years of age. 2. :I The study provided the opportunity to compare the diagnoses of ovarian malignancies made by contributing pathologists with those of a panel of three gynecologic pathologists. In this article we describe a measure used to make quantitative comparisons of the diagnoses, give a brief review of the methods used in the Cancer and Steroid Hormone Study, and report our findings. Background
The accuracy of diagnostic tests, as exemplified by surgical pathology reports, has always been important. Even though we have always known that no diagnostic test is perfect, most of us assume that if an accurate diagnostic test for a particular condition does exist, it must be one reported by a skilled pathologist. Nonetheless, scientific articles have documented discrepancies among pathologists in the diagnosis of ovarian cancer!-7 Investigators now quantify these discrepancies. In this report we will focus on the predictive value positive" as the measure for evaluation of diagnoses of histopathologic features of ovarian cancer. The predictive value positive is a measure of the probability that disease exists, given that the report of a diagnostic test is positive. It can be calculated with a numerator of patients known by a widely accepted, highly accurate standard to have ovarian cancer, who also have a histopathology report positive for ovarian cancer. The denominator is all the histopathology reports of ovarian cancer, regardless of whether all the patients included have disease. (Other measures of the characteristics of diagnostic tests include sensitivity, specificity, and predictive value negative. They are not included here because they require information about negative test reports and the absence of disease not reflected in the data we analyzed.) In analyzing the predictive value positive of reports for a disease, the reports of interest (in this case, those of ovarian cancer) need to be compared with an acceptable standard. In this analysis we compared the reports of ovarian cancer made by all pathologists in
January 1991 Am J Obslel Gynecn)
the geographic area of each of the Surveillance, Epidemiology, and End Results (SEER) centers who contribute their reports (designated as contributing pathologists hereafter) with the consensus diagnosis of the panel of pathologists. In calculating the predictive value positive, the reports of the contributing pathologists serve as the denominatOl-. Those confirmed by the panel are the numerator. Therefore, if 50 cases were reported to be ovarian cancer by SEER centers on the basis of the reports given them by contributing pathologists, but the panel confirmed only 45 of them, then the predictive value positive would be as follows: PVP
= 45/50
x 100
= 90%
Methods
The general methods of the Cancer and Steroid Hormone Study have been described previously.2.3 Subjects included in this analysis were enrolled through population-based tumor registries participating in the SEER program of the National Cancer Institute in six areas of the United States in which both interviews and retrieval of histopathology reports could be done (the metropolitan areas of Atlanta, Detroit, San Francisco, and Seattle, and the states of Connecticut and Iowa). The subjects of this analysis were all women with primary ovarian cancer 20 to 54 years of age who resided in any of the reporting areas and who were first diagnosed as having ovarian cancer in the period Dec. I, 1980, to Dec. 31, 1982. Of the 8 I 6 women who satisfied the case definition for ovarian cancer in the original Cancer and Steroid Hormone Study, 579 (71 %) were interviewed. Slide retrieval was attempted only for the interviewed study subjects. Because only the six SEER centers mentioned in this article participated in retrieval of the slides of histologic specimens, the 40 women with ovarian cancer interviewed for the study from the other two centers (those in New Mexico and Utah) were excluded from this analysis. Furthermore, 62 additional subjects could not be included for the following reasons: slides could not be retrieved (18), inadequate specimens (8), possible previous history of ovarian cancer (4), and original diagnosis of the contributing pathologist not available (32). Therefore the subjects in this analysis included 477 women with primary ovarian malignancy as diagnosed by pathologists participating in the aforementioned population-based cancer registries. All slides of histopathologic specimens were initially read by the contributing pathologists, and the results were then reported to the SEER centers. Because the tumor registries are population-based, the pathologists making these initial readings represent those practicing in the entire community. Slides that were the basis for the original diagnosis made in the community in which the study subjects
Agreement of pathologists on diagnosis of ovarian cancer
Volume 164 Number I, Part I
67
Table I. Histopathologic diagnosis by contributing pathologists compared with those of a panel of gynecologic pathologists, by histologic category of ovarian tumor* Panel diagnosest Contributing diagnoses
Total
Benign
Epithelial cancer
Epithelial cancer Nonepithelial cancer
432 45 477
1 I 2
420 4 424
TOTAL
I
N onepithelial cancer
2 40 42
I
Metastatic cancer
9 0 9
*A total of 62 subjects are excluded from this table because of the following reasons: 32 diagnoses not available, 18 slides not retrieved, 8 inadequate slides, and 4 subjects with a previous or uncertain history of ovarian cancer, tPredictive value positive for all ovarian primary cancers [(424 + 42)/477] = 98%; predictive value positive for epithelial ovarian cancer (420/432) = 97%; predictive value positive for nonepithelial ovarian cancer (40/45) = 89%.
lived were then reviewed by a panel of three pathologists expert in the diagnosis of histopathologic features of ovarian cancer (Drs. Robboy, Kurman, and Paris). When required the panel requested additional slides for review. The panel reviewed each histologic specimen independently. If there were differences in the reviews of panel members, individual specimens were reviewed in greater detail with a microscope with eyepieces that permitted the panelists to examine and discuss each specimen simultaneously until consensus was reached. No written record of such differences were kept by the panel because the differences occurred so infrequently. Each ovarian tumor was categorized by the panel as malignant (primary), malignant (metastatic), or benign. The malignant primary tumors were then classified as epithelial or nonepithelial, and assigned a histologic type including characterization as malignant or low malignant potential (borderline). Tumors oflow malignant potential were grouped with malignant tumors when analyzed by histologic type. The classification scheme used was that of the World Health Organization. 9 In analyzing differences, trivial discrepancies in wording, such as papillary serous rather than serous, were not considered different. Further, as many ovarian tumors show blends of features (e.g., serous and endometrioid) the panel attempted not to consider such diagnoses as different from those reported by the contributing pathologists unless the discrepancy was readily apparent. In 79 cases a substantial portion of the tumor warranted a second diagnosis. Both diagnoses were considered in assessing concurrence between contributing and panel pathologists. Statistical differences between predictive value positives were evaluated with X2 tests. 'O These differences were regarded as significant for p < 0.05.
Results Among the women with primary ovarian tumors, approximately one third (35%) were in the age range 20 to 39 years; one third (34%) 40 to 49 years; and one
third (32%) 50 to 54 years at the time of diagnosis, (Design of the Cancer and Steroid Hormone Study excluded women older than 54 years.) Almost 90% (88%) were white. Slightly more than four fifths (83%) were high school graduates. Approximately one quarter (26%) were nulliparous. More than 40% were no longer menstruating regularly. In this investigation, the contributing pathologists diagnosed 477 tumors as primary ovarian cancer, and the panel pathologists agreed with 466 (98%), as shown in Table I. The predictive value positive for epithelial cancers (97%) is significantly greater than that for nonepithelial neoplasms (89%, p < 0.05). For cancer of low malignant potential (borderline), the predictive value positive is 93%, whereas that for invasive cancer is 85% (p < 0.05, Table II). Contributing pathologists and the panel of gynecologic pathologists agreed on the histologic type of ovarian cancer less often than they did on the diagnosis of malignancy. In this analysis we considered four specific types of epithelial ovarian cancers. There were too few cases of the other types of epithelial ovarian cancer to be included. Table III shows the panel agreed with the 21 cases originally diagnosed by the contributing pathologists as clear cell carcinoma. The remaining three histologic types-serous, mucinous, and endometrioid-all had predictive value positives significantly lower than that for clear cell cancers (p < 0.05), but not different from each other. The two types of nonepithelial cancer-germ cell and sex cord-had predictive values of 92% and 88%, respectively; these values are not significantly different from each other or from the predictive value positives for the serous, mucinous, or endometrioid tumors. The disagreements between the contributing pathologists and the expert panel focused on three issues. First, of the 11 instances in which the contributing and panel pathologists did not agree on the diagnosis of primary ovarian cancer (Table I), nine were diagnosed as metastatic tumors by the panel. Next, of the 63 cases in which disagreement occurred on the diagnosis of
68 Tyler et al.
January 1991 Am J Obstet Gynecol
Table II. Histopathologic diagnoses of ovarian cancers by contributing pathologists compared with those of a panel of gynecologic pathologists, by low malignant potential (borderline) and invasive cancer* Panel diagnoses Contributing diagnoses
Total
Predictive value positive (%)
Low malignant potential (borderline) Invasive cancer
86
93
80
6
378
85
321
57
Agree
I
Do not agree
p < 0.05. *Excludes 11 cases judged benign or metastatic and two diagnoses not available.
Table III. Histopathologic diagnoses by contributing pathologists compared with a panel of gynecologic pathologists by histologic type of ovarian cancer* Panel diagnoses Contributing diagnoses
Total
Predictive value positive (%) t
Serous Mucinous Endometrioid Clear cell Germ cell Sex cord
256 85 52 21 24 17
80 87 73 100 92 88
205 74 38 21 22 15
51 11 14 0 2 2
Overall
455
82
375
80
Agree
I
Do not agree
*A small number of other types of epithelial (8) and nonepithelial (3) ovarian cancers are not included in this tabulation. The 11 metastatic and benign tumors shown in Table I are also excluded from this table. tSee text for discussion of the significance of the differences in predictive value positive values.
low malignant potential or borderline (Table II), the panel made a diagnosis of low malignant potential rather than invasive cancer in 57 (90%) and a diagnosis of invasive malignancy in six (10%) instead of low malignant potential. In this group of 63 cases, there was substantial concurrence on cell type. Finally, analysis of histologic type showed that the panel did not agree with the diagnosis of the contributing pathologists in 85 instances. In more than half of these cases (48 or 56%) the disagreement involved the distinction between endometrioid and serous cancer. In 36 instances the diagnosis of contributing pathologists was serous carcinoma, whereas the diagnosis of the panel pathologists was endometrioid cancer; the diagnoses were made in the reverse order for the other 12 instances. An analysis of interview data showed that differences in diagnoses between the contributing pathologists and the panel of experts were not related to geographic region, age, race, menopausal status, estrogen use, use of oral contraceptives, or history of infertility. Comment
This analysis highlights three findings. First, a diagnosis of primary ovarian cancer by contributing pathologists agreed with that of the panel in 98% of cases included in the Cancer and Steroid Hormone Study. In fact, of 477 cases reported as ovarian cancer, only two were judged by the panel to be benign; the re-
maining nine cases were diagnosed by the panel as metastatic (i.e., not primary ovarian cancer). Next, tumors of low malignant potential and invasive cancer were also differentiated with high (93% and 85%, respectively), albeit significantly different, levels of agreement. Of the cases on which there was disagreement, the panel diagnosed 90% (57 of 63) as tumors of low malignant potential rather than invasive cancer. Finally, the panel agreed with the contributing pathologists on 100% of cases reported to be clear cell ovarian carcinoma, but overall agreement on histologic type of tumor (375 of 455, or 82%) did not reach the same level attained in the correct diagnosis of primary ovarian cancer, or in the differentiation of tumors of low malignant potential from invasive cancer. There are several limitations of this analysis. First, it is based on data from a study that focused on the epidemiologic relationship of cancers of the ovary, breast, and endometrium with administration of steroid hormone to women 20 to 54 years old. Therefore questions concerning ovarian cancer pathologic findings were not of highest priority. If they had been, the study design would almost surely have been different (e.g., it might have included women of all ages). Next, the pathologists on the panel knew the original diagnoses made by the contributing pathologists. We cannot be sure that this introduced bias into our results, but if it did it probably increased the predictive value positive. Fi-
Volume 164 Number I, Part 1
Agreement of pathologists on diagnosis of ovarian cancer
69
Table IV. Summary of reports from the recent literature showing predictive value positive estimates First author
Comment
Baak"
1982
Panel of 4 pathologists
Cooke 5
1984
Quality control survey
Hernandez6
1984
Panel of 2 pathologists
Cramer 7*
1987
Panel of 7 surgical patholog-ists
Tumor characteristics
Mucinous ovarian tumors Mucinous ovarian cancer, low malignant potential Ovarian epithelial cancer Histologic type Histologic grade Ovarian epithelial and nonepithelial cancer Mucinous Serous Clear cell Endometrioid
Predictive value positive (%)
49
76
175
70
68 68
60 60
14 25 15 30
43 32 20 17
*Predictive value positive for this study is taken as the authors' "unanimity or near-unanimity (at most, one discordant pathologist)" category.
nally, the predictive value positive gives only part of the picture. Whereas all 21 reports of clear cell carcinoma made by the contributing pathologists had the concurrence of the expert panel, the panel members diagnosed 13 other clear cell malignancies that the contributing pathologists did not. In comparing our results with those reported earlier, we found that each of the reports dealt with a different approach to investigation of the accuracy of cancer diagnosis (see Table IV). In addition, three of them focus more attention on mucinous carcinoma than the analysis reported here. The work of Baak et al.: Hernandez and his colleagues,6 and Cramer and his coworkers 7 each involved the comparison of individual members of a group of expert pathologists (four, two, and seven experts, respectively). It is important to emphasize that whereas the predictive value positives differ from study to study, differences in study design and clinical information known by the pathologists limit the value of comparing these investigations in detail. The specimens reviewed in the Cancer and Steroid Hormone Study were collected in a 2-year period that ended in December 1982; it is possible that the predictive value positive for the diagnosis of ovarian cancer by pathologists might have improved since that date. The studies cited here all have predictive value positive estimates that are near or less than those reported from the Cancer and Steroid Hormone Study. This suggests that if a change occurred, it was not appreciable; nonetheless, differences in study design limit the value of such a comparison. The approach taken to validate the diagnosis of primary ovarian cancer in our analysis is more like that used in a study that assessed the diagnosis of lymphomas. lI Nonetheless, the variability in the diagnoses by pathologists of these hematologic malignancies was sub-
stantial, ranging from 52% to 89%. The investigators concluded that variability in the diagnoses by pathologists decreases as an observer's experience increases and that the variability is least among expert pathologists. 11 The statistical approach in some studies 7 of this problem make use of the K test. Fleiss lO points out that the K statistic is suitable when all those making a set of observations are equally experienced; otherwise, he states, other tests may be more appropriate. Maclure and Willett,'· in their assessment of 13 papers in which the K test had been used, point out that it is "... a measure of agreement between two observers classifying subjects into two nominal categories...." They state that under circumstances like those in this analysis of ovarian cancer, alternative analytic approaches would be more appropriate. These results from the Cancer and Steroid Hormone Study indicate that primary ovarian cancer is diagnosed with a very high level of predictability in the United States. Nonetheless, cancer that metastasizes to the ovary made up a large proportion of the very small percentage of malignancies that are not correctly diagnosed. For those few patients in whom this occurs, the consequences may be serious because the primary cancer could go untreated. In addition, this analysis indicates that ovarian cancer of low malignant potential is interpreted as invasive malignancy in almost one of every eight cases (57 of 464, or 12%) of primary ovarian cancers, as shown in Table II. To the extent that this leads to treatment that has more frequent or more serious side effects, this problem with diagnosis also influences patients with this disease. These results lead us to recommend that research on the different histologic types of ovarian cancer be reviewed by a panel of pathologists experienced with the
70 Tyler et ai.
diagnosis of gynecologic disorders. The analysis of clear cell ovarian cancer supports our concern. Whereas the panel agreed wth the contributing pathologists on all 21 clear cell ovarian cancers, the panel found 13 other instances in which tumors reported as endometrioid were reclassified by the panel as clear cell ovarian cancer, indicating that the histologic type of 38% (13 of 34 clear cell cancers) of malignancies with this cell type was revised as a result of the review. Guidance for clinical practitioners is more difficult to suggest. Although ovarian cancer that has metastasized to the ovary from some other primary site is rare, the possibility of a distant primary site is the first and most serious factor to be considered in the diagnosis of ovarian cancer. In addition, if treatment will be influenced by the histologic type of primary ovarian cancer, review of such specimens by competent colleagues (even if only one is available) deserves serious consideration. No matter what role a physician plays in caring for women at risk of ovarian cancer, their survival depends on correct identification of the origin of the malignancy and may be influenced by histologic type as well as by clinical characteristics. For these reasons, the precise diagnosis of ovarian cancer by pathologists is important for researchers, clinicians, and women with this condition. l3 We acknowledge the contributors to the Cancer and Steroid Hormone Study. Study design and cordination: Division of Reproductive Health, Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control; principal investigator: George L. Rubin, MB; project director: Phyllis A. Wingo, MS; project associates: Nancy C. Lee, MD, Michele G. Mandel, BA, and Herbert B. Peterson, MD; data collection centers principal investigators: Atlanta, Raymond Greenberg, MD; Connecticut,]. Wister Meigs, MD; and W. Douglas Thompson, PhD; Detroit, G. Marie Swanson, PhD; Iowa, Elaine Smith, PhD; New Mexico, Charles Key, MD, and Dorothy Pathak, PhD; San Francisco, Donald Austin, MD; Seattle, David Thomas, MD; Utah, Joseph Lyon, MD, and Dee West, PhD; pathology review principal investigators: Fred Gorstein, MD, Robert McDivitt, MD, and Stanley]. Robboy, MD; project
January 1991 Am J Obstet Gynecol
consultants: Lonnie Burnett, MD, Robert Hoover, MD, Peter M. Layde, MD, MSc, Howard W. Ory, MD, MSc, James]. Schlesselman, PhD, David Schottenfeld, MD, Bruce Stadel, MD, Linda A. Webster, MSPH, and Colin White, MB, BS; pathology consultants: Walter Bauer, MD, William Christopherson, MD, Deborah Gersell, MD, Robert]. Kurman, MD, Allen L. Paris, MD, and Frank Vellios, MD.
REFERENCES I. American Cancer Society. Cancer facts and figures1989. Atlanta: American Cancer Society, 1989. 2. Wingo PA, Ory HW, Layde PM, et al. The evaluation of the data collection process for a multicenter, populationbased, case-control design. Am J Epidemiol 1988; 128: 206-17. 3. The Cancer and Steroid Hormone Study of the Centers for Disease Control and the National Institute of Child Health and Human Development. The reduction in risk of ovarian cancer associated with oral-contraceptive use. N EnglJ Med 1987;316:650-5. 4. Baak JPA, Lindeman J, Overdiep SH, Langley FA. Disagreement of histopathological diagnoses of different pathologists in ovarian tumors-with some theoretical considerations. Eur J Obstet Gynec Reprod BioI 1982; 13: 51-5. 5. Cooke RA. Quality control in anatomic pathology: experience in Australia and New Zealand. In: Somers SC, Rosen PP, eds. Pathology annual. Connecticut: AppletonCentury-Crofts, 1984; 19:221-48. 6. Hernandez E, Bhagavan BS, Parmley TH, Rosenshein NB. Interobserver variability in the interpretation of epithelial ovarian cancer. Gynecol Oncol 1984; 17: 117-23. 7. Cramer SF, Roth LM, Ulbright TM, et al. Evaluation of the reproducibility of the World Health Organization classification of common ovarian cancers. Arch Pathol Lab Med 1987;111:819-29. 8. Fletcher RH, Fletcher SW, Wagner EH. Clinical epidemiology-the essentials. Baltimore: Williams and Wilkins, 1982. 9. Serov SF, Scully RE, Sobin LH. International histological classification of tumors. Geneva: World Health Organization, 1973. 10. Fleiss JL. Statistical methods for rates and proportions. New York: John Wiley & Sons, 1981. II. Kim H, Zelman RJ, Fox MA, et al. Pathology panel for lymphoma clinical studies: a comprehensive analysis of cases accumulated since its inception. JNCI 1982;68:4367. 12. Maclure M, Willett WC. Misinterpretation and misuse of the K statistic. AmJ EpidemioI1987;126:161-169. 13. Aure JC, Hoeg K, Kolstad P. Clinical and histologic studies of ovarian cancer. Ohstet GynecoI1971;37:1-9.
membranes may have an increased risk of intrauterine growth retardation. Wladimiroff and CampbeiF studied urine production in fetuses with intrauterine growth retardation and found that urine output was much lower than that expected for a normally grown fetus in three of 12 determinations. In this study fetal urine output in pregnancies complicated by premature rupture of membranes was not reduced and did not vary as a function of residual amniotic fluid volume. Fetal urine production was unchanged even in the presence of severe oligohydramnios. We conclude that residual amniotic fluid volume
Amniotic fluid after premature rupture of membranes
after premature rupture of membranes does not depend on, or alter, fetal renal function. Residual amniotic fluid volume may depend on mechanical factors, such as maternal position, the site of leakage, and the size of the defect in the fetal membranes. REFERENCES I. Campbell S, Wladimiroff JW, Dewhurst CJ. The antenatal
measurement of fetal urine production. J Obstet Gynaecol Br Commonw 1973;80:680-6. 2. Wladimiroff JW, Campbell S. Fetal urine production rates in normal and complicated pregnancies. Lancet 1974; 1: 151-4.
The diagnosis of ovarian cancer by pathologists: How often do diagnoses by contributing pathologists agree with a panel of gynecologic pathologists? Carl W. Tyler, Jr., MD,. Nancy C. Lee, MD: Stanley J. Robboy, MD; RobertJ. Kurman, MD,d Allen L. Paris, MD,-·f Phyllis A. Wingo, MS: and G. David Williamson, PhD" Atlanta, Georgia, Newark, New Jersey, Baltimore, Maryland, and West Plains, Missouri The Cancer and Steroid Hormone Study, a multicenter, population-based, case-control study of ovarian, breast, and endometrial cancer in women 20 to 54 years of age, permitted the diagnoses of contributing pathologists to be compared with those of a panel of three gynecologic pathologists. A diagnosis of ovarian cancer was made by contributing pathologists on 477 subjects. Agreement between the two groups of pathologists was 97% for primary epithelial ovarian cancer and 89% for primary nonepithelial ovarian malignancies. Agreement on diagnosis of major cellular subtypes of ovarian malignancy ranged between 73% for endometrioid cancer and 100% for clear cell carcinomas. We conclude that the diagnosis of pathologic features of primary ovarian cancer is highly predictable. Nonetheless, diagnosis by histologic type varies sufficiently that a review process should be considered for clinical or investigative decisions involving specific histologic diagnoses of ovarian cancer. (AM J OBSTET GVNECOL 1991 ;164: 65-70.)
Key words: Ovarian neoplasms, diagnosis, predictive value of tests, pathology From the Office of the Director, Centers for Disease Control," Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control/ Department of Pathology, New Jersey Medical School,' Departments of Gynecology, Obstetrics and Pathology, Johns Hopkins Hospital, d Centerfor Environmental Health and Injury Control, Centers for Disease Control,' Department of Pathology, Ozarks Medical Center,! and Epidemiology Program Office, Centers for Disease Control.' The Cancer and Steroid Hormone Study was supported by interagency agreement 3-Y01-HD-8-I037 between the Centers for Disease Control and the National Institute of Child Health and Human Development, with additional support from the National Cancer Institute. Received for publication November 30, 1989; revised August 6, 1990; accepted August 13, 1990. Reprint requests: Carl W. T.vler,Jr., MD, Centers for Disease Control, Mailstop E-42, Atlanta, GA 30333. 6/1/24575
Ovarian cancer is a significant cause of death for women in the United States. In 1989 an estimated 20,000 American women were newly diagnosed as having ovarian cancer, and 12,000 died from this malignancy. I The diagnosis of ovarian cancer by pathologists not only guides patient treatment but also provides the basis for evaluation of different approaches to treatment in multicenter clinical trials and in epidemiologic studies of risk factors. As an illustration, if a cancer that metastasized to the ovary were not diagnosed correctly on a pathology report, the patient might be treated as if the tumor were primary in the ovary when another site was the one of origin; this might lead to inappro-
65
66
Tyler et al.
priate therapy. If this occurred often enough, studies that evaluate surgical or chemotherapeutic approaches to primary ovarian cancer would incorrectly include tumors from other organs. This problem would also extend to epidemiologic investigations of the causes of ovarian cancer. In the period 1980 to 1982 the Centers for Disease Control, with support from the National Institute of Child Health and Human Development and from the National Cancer Institute, conducted the Cancer and Steroid Hormone Study, a multicenter, populationbased case-control study of ovarian, breast, and endometrial cancer in women 20 to 54 years of age. 2. :I The study provided the opportunity to compare the diagnoses of ovarian malignancies made by contributing pathologists with those of a panel of three gynecologic pathologists. In this article we describe a measure used to make quantitative comparisons of the diagnoses, give a brief review of the methods used in the Cancer and Steroid Hormone Study, and report our findings. Background
The accuracy of diagnostic tests, as exemplified by surgical pathology reports, has always been important. Even though we have always known that no diagnostic test is perfect, most of us assume that if an accurate diagnostic test for a particular condition does exist, it must be one reported by a skilled pathologist. Nonetheless, scientific articles have documented discrepancies among pathologists in the diagnosis of ovarian cancer!-7 Investigators now quantify these discrepancies. In this report we will focus on the predictive value positive" as the measure for evaluation of diagnoses of histopathologic features of ovarian cancer. The predictive value positive is a measure of the probability that disease exists, given that the report of a diagnostic test is positive. It can be calculated with a numerator of patients known by a widely accepted, highly accurate standard to have ovarian cancer, who also have a histopathology report positive for ovarian cancer. The denominator is all the histopathology reports of ovarian cancer, regardless of whether all the patients included have disease. (Other measures of the characteristics of diagnostic tests include sensitivity, specificity, and predictive value negative. They are not included here because they require information about negative test reports and the absence of disease not reflected in the data we analyzed.) In analyzing the predictive value positive of reports for a disease, the reports of interest (in this case, those of ovarian cancer) need to be compared with an acceptable standard. In this analysis we compared the reports of ovarian cancer made by all pathologists in
January 1991 Am J Obslel Gynecn)
the geographic area of each of the Surveillance, Epidemiology, and End Results (SEER) centers who contribute their reports (designated as contributing pathologists hereafter) with the consensus diagnosis of the panel of pathologists. In calculating the predictive value positive, the reports of the contributing pathologists serve as the denominatOl-. Those confirmed by the panel are the numerator. Therefore, if 50 cases were reported to be ovarian cancer by SEER centers on the basis of the reports given them by contributing pathologists, but the panel confirmed only 45 of them, then the predictive value positive would be as follows: PVP
= 45/50
x 100
= 90%
Methods
The general methods of the Cancer and Steroid Hormone Study have been described previously.2.3 Subjects included in this analysis were enrolled through population-based tumor registries participating in the SEER program of the National Cancer Institute in six areas of the United States in which both interviews and retrieval of histopathology reports could be done (the metropolitan areas of Atlanta, Detroit, San Francisco, and Seattle, and the states of Connecticut and Iowa). The subjects of this analysis were all women with primary ovarian cancer 20 to 54 years of age who resided in any of the reporting areas and who were first diagnosed as having ovarian cancer in the period Dec. I, 1980, to Dec. 31, 1982. Of the 8 I 6 women who satisfied the case definition for ovarian cancer in the original Cancer and Steroid Hormone Study, 579 (71 %) were interviewed. Slide retrieval was attempted only for the interviewed study subjects. Because only the six SEER centers mentioned in this article participated in retrieval of the slides of histologic specimens, the 40 women with ovarian cancer interviewed for the study from the other two centers (those in New Mexico and Utah) were excluded from this analysis. Furthermore, 62 additional subjects could not be included for the following reasons: slides could not be retrieved (18), inadequate specimens (8), possible previous history of ovarian cancer (4), and original diagnosis of the contributing pathologist not available (32). Therefore the subjects in this analysis included 477 women with primary ovarian malignancy as diagnosed by pathologists participating in the aforementioned population-based cancer registries. All slides of histopathologic specimens were initially read by the contributing pathologists, and the results were then reported to the SEER centers. Because the tumor registries are population-based, the pathologists making these initial readings represent those practicing in the entire community. Slides that were the basis for the original diagnosis made in the community in which the study subjects
Agreement of pathologists on diagnosis of ovarian cancer
Volume 164 Number I, Part I
67
Table I. Histopathologic diagnosis by contributing pathologists compared with those of a panel of gynecologic pathologists, by histologic category of ovarian tumor* Panel diagnosest Contributing diagnoses
Total
Benign
Epithelial cancer
Epithelial cancer Nonepithelial cancer
432 45 477
1 I 2
420 4 424
TOTAL
I
N onepithelial cancer
2 40 42
I
Metastatic cancer
9 0 9
*A total of 62 subjects are excluded from this table because of the following reasons: 32 diagnoses not available, 18 slides not retrieved, 8 inadequate slides, and 4 subjects with a previous or uncertain history of ovarian cancer, tPredictive value positive for all ovarian primary cancers [(424 + 42)/477] = 98%; predictive value positive for epithelial ovarian cancer (420/432) = 97%; predictive value positive for nonepithelial ovarian cancer (40/45) = 89%.
lived were then reviewed by a panel of three pathologists expert in the diagnosis of histopathologic features of ovarian cancer (Drs. Robboy, Kurman, and Paris). When required the panel requested additional slides for review. The panel reviewed each histologic specimen independently. If there were differences in the reviews of panel members, individual specimens were reviewed in greater detail with a microscope with eyepieces that permitted the panelists to examine and discuss each specimen simultaneously until consensus was reached. No written record of such differences were kept by the panel because the differences occurred so infrequently. Each ovarian tumor was categorized by the panel as malignant (primary), malignant (metastatic), or benign. The malignant primary tumors were then classified as epithelial or nonepithelial, and assigned a histologic type including characterization as malignant or low malignant potential (borderline). Tumors oflow malignant potential were grouped with malignant tumors when analyzed by histologic type. The classification scheme used was that of the World Health Organization. 9 In analyzing differences, trivial discrepancies in wording, such as papillary serous rather than serous, were not considered different. Further, as many ovarian tumors show blends of features (e.g., serous and endometrioid) the panel attempted not to consider such diagnoses as different from those reported by the contributing pathologists unless the discrepancy was readily apparent. In 79 cases a substantial portion of the tumor warranted a second diagnosis. Both diagnoses were considered in assessing concurrence between contributing and panel pathologists. Statistical differences between predictive value positives were evaluated with X2 tests. 'O These differences were regarded as significant for p < 0.05.
Results Among the women with primary ovarian tumors, approximately one third (35%) were in the age range 20 to 39 years; one third (34%) 40 to 49 years; and one
third (32%) 50 to 54 years at the time of diagnosis, (Design of the Cancer and Steroid Hormone Study excluded women older than 54 years.) Almost 90% (88%) were white. Slightly more than four fifths (83%) were high school graduates. Approximately one quarter (26%) were nulliparous. More than 40% were no longer menstruating regularly. In this investigation, the contributing pathologists diagnosed 477 tumors as primary ovarian cancer, and the panel pathologists agreed with 466 (98%), as shown in Table I. The predictive value positive for epithelial cancers (97%) is significantly greater than that for nonepithelial neoplasms (89%, p < 0.05). For cancer of low malignant potential (borderline), the predictive value positive is 93%, whereas that for invasive cancer is 85% (p < 0.05, Table II). Contributing pathologists and the panel of gynecologic pathologists agreed on the histologic type of ovarian cancer less often than they did on the diagnosis of malignancy. In this analysis we considered four specific types of epithelial ovarian cancers. There were too few cases of the other types of epithelial ovarian cancer to be included. Table III shows the panel agreed with the 21 cases originally diagnosed by the contributing pathologists as clear cell carcinoma. The remaining three histologic types-serous, mucinous, and endometrioid-all had predictive value positives significantly lower than that for clear cell cancers (p < 0.05), but not different from each other. The two types of nonepithelial cancer-germ cell and sex cord-had predictive values of 92% and 88%, respectively; these values are not significantly different from each other or from the predictive value positives for the serous, mucinous, or endometrioid tumors. The disagreements between the contributing pathologists and the expert panel focused on three issues. First, of the 11 instances in which the contributing and panel pathologists did not agree on the diagnosis of primary ovarian cancer (Table I), nine were diagnosed as metastatic tumors by the panel. Next, of the 63 cases in which disagreement occurred on the diagnosis of
68 Tyler et al.
January 1991 Am J Obstet Gynecol
Table II. Histopathologic diagnoses of ovarian cancers by contributing pathologists compared with those of a panel of gynecologic pathologists, by low malignant potential (borderline) and invasive cancer* Panel diagnoses Contributing diagnoses
Total
Predictive value positive (%)
Low malignant potential (borderline) Invasive cancer
86
93
80
6
378
85
321
57
Agree
I
Do not agree
p < 0.05. *Excludes 11 cases judged benign or metastatic and two diagnoses not available.
Table III. Histopathologic diagnoses by contributing pathologists compared with a panel of gynecologic pathologists by histologic type of ovarian cancer* Panel diagnoses Contributing diagnoses
Total
Predictive value positive (%) t
Serous Mucinous Endometrioid Clear cell Germ cell Sex cord
256 85 52 21 24 17
80 87 73 100 92 88
205 74 38 21 22 15
51 11 14 0 2 2
Overall
455
82
375
80
Agree
I
Do not agree
*A small number of other types of epithelial (8) and nonepithelial (3) ovarian cancers are not included in this tabulation. The 11 metastatic and benign tumors shown in Table I are also excluded from this table. tSee text for discussion of the significance of the differences in predictive value positive values.
low malignant potential or borderline (Table II), the panel made a diagnosis of low malignant potential rather than invasive cancer in 57 (90%) and a diagnosis of invasive malignancy in six (10%) instead of low malignant potential. In this group of 63 cases, there was substantial concurrence on cell type. Finally, analysis of histologic type showed that the panel did not agree with the diagnosis of the contributing pathologists in 85 instances. In more than half of these cases (48 or 56%) the disagreement involved the distinction between endometrioid and serous cancer. In 36 instances the diagnosis of contributing pathologists was serous carcinoma, whereas the diagnosis of the panel pathologists was endometrioid cancer; the diagnoses were made in the reverse order for the other 12 instances. An analysis of interview data showed that differences in diagnoses between the contributing pathologists and the panel of experts were not related to geographic region, age, race, menopausal status, estrogen use, use of oral contraceptives, or history of infertility. Comment
This analysis highlights three findings. First, a diagnosis of primary ovarian cancer by contributing pathologists agreed with that of the panel in 98% of cases included in the Cancer and Steroid Hormone Study. In fact, of 477 cases reported as ovarian cancer, only two were judged by the panel to be benign; the re-
maining nine cases were diagnosed by the panel as metastatic (i.e., not primary ovarian cancer). Next, tumors of low malignant potential and invasive cancer were also differentiated with high (93% and 85%, respectively), albeit significantly different, levels of agreement. Of the cases on which there was disagreement, the panel diagnosed 90% (57 of 63) as tumors of low malignant potential rather than invasive cancer. Finally, the panel agreed with the contributing pathologists on 100% of cases reported to be clear cell ovarian carcinoma, but overall agreement on histologic type of tumor (375 of 455, or 82%) did not reach the same level attained in the correct diagnosis of primary ovarian cancer, or in the differentiation of tumors of low malignant potential from invasive cancer. There are several limitations of this analysis. First, it is based on data from a study that focused on the epidemiologic relationship of cancers of the ovary, breast, and endometrium with administration of steroid hormone to women 20 to 54 years old. Therefore questions concerning ovarian cancer pathologic findings were not of highest priority. If they had been, the study design would almost surely have been different (e.g., it might have included women of all ages). Next, the pathologists on the panel knew the original diagnoses made by the contributing pathologists. We cannot be sure that this introduced bias into our results, but if it did it probably increased the predictive value positive. Fi-
Volume 164 Number I, Part 1
Agreement of pathologists on diagnosis of ovarian cancer
69
Table IV. Summary of reports from the recent literature showing predictive value positive estimates First author
Comment
Baak"
1982
Panel of 4 pathologists
Cooke 5
1984
Quality control survey
Hernandez6
1984
Panel of 2 pathologists
Cramer 7*
1987
Panel of 7 surgical patholog-ists
Tumor characteristics
Mucinous ovarian tumors Mucinous ovarian cancer, low malignant potential Ovarian epithelial cancer Histologic type Histologic grade Ovarian epithelial and nonepithelial cancer Mucinous Serous Clear cell Endometrioid
Predictive value positive (%)
49
76
175
70
68 68
60 60
14 25 15 30
43 32 20 17
*Predictive value positive for this study is taken as the authors' "unanimity or near-unanimity (at most, one discordant pathologist)" category.
nally, the predictive value positive gives only part of the picture. Whereas all 21 reports of clear cell carcinoma made by the contributing pathologists had the concurrence of the expert panel, the panel members diagnosed 13 other clear cell malignancies that the contributing pathologists did not. In comparing our results with those reported earlier, we found that each of the reports dealt with a different approach to investigation of the accuracy of cancer diagnosis (see Table IV). In addition, three of them focus more attention on mucinous carcinoma than the analysis reported here. The work of Baak et al.: Hernandez and his colleagues,6 and Cramer and his coworkers 7 each involved the comparison of individual members of a group of expert pathologists (four, two, and seven experts, respectively). It is important to emphasize that whereas the predictive value positives differ from study to study, differences in study design and clinical information known by the pathologists limit the value of comparing these investigations in detail. The specimens reviewed in the Cancer and Steroid Hormone Study were collected in a 2-year period that ended in December 1982; it is possible that the predictive value positive for the diagnosis of ovarian cancer by pathologists might have improved since that date. The studies cited here all have predictive value positive estimates that are near or less than those reported from the Cancer and Steroid Hormone Study. This suggests that if a change occurred, it was not appreciable; nonetheless, differences in study design limit the value of such a comparison. The approach taken to validate the diagnosis of primary ovarian cancer in our analysis is more like that used in a study that assessed the diagnosis of lymphomas. lI Nonetheless, the variability in the diagnoses by pathologists of these hematologic malignancies was sub-
stantial, ranging from 52% to 89%. The investigators concluded that variability in the diagnoses by pathologists decreases as an observer's experience increases and that the variability is least among expert pathologists. 11 The statistical approach in some studies 7 of this problem make use of the K test. Fleiss lO points out that the K statistic is suitable when all those making a set of observations are equally experienced; otherwise, he states, other tests may be more appropriate. Maclure and Willett,'· in their assessment of 13 papers in which the K test had been used, point out that it is "... a measure of agreement between two observers classifying subjects into two nominal categories...." They state that under circumstances like those in this analysis of ovarian cancer, alternative analytic approaches would be more appropriate. These results from the Cancer and Steroid Hormone Study indicate that primary ovarian cancer is diagnosed with a very high level of predictability in the United States. Nonetheless, cancer that metastasizes to the ovary made up a large proportion of the very small percentage of malignancies that are not correctly diagnosed. For those few patients in whom this occurs, the consequences may be serious because the primary cancer could go untreated. In addition, this analysis indicates that ovarian cancer of low malignant potential is interpreted as invasive malignancy in almost one of every eight cases (57 of 464, or 12%) of primary ovarian cancers, as shown in Table II. To the extent that this leads to treatment that has more frequent or more serious side effects, this problem with diagnosis also influences patients with this disease. These results lead us to recommend that research on the different histologic types of ovarian cancer be reviewed by a panel of pathologists experienced with the
70 Tyler et ai.
diagnosis of gynecologic disorders. The analysis of clear cell ovarian cancer supports our concern. Whereas the panel agreed wth the contributing pathologists on all 21 clear cell ovarian cancers, the panel found 13 other instances in which tumors reported as endometrioid were reclassified by the panel as clear cell ovarian cancer, indicating that the histologic type of 38% (13 of 34 clear cell cancers) of malignancies with this cell type was revised as a result of the review. Guidance for clinical practitioners is more difficult to suggest. Although ovarian cancer that has metastasized to the ovary from some other primary site is rare, the possibility of a distant primary site is the first and most serious factor to be considered in the diagnosis of ovarian cancer. In addition, if treatment will be influenced by the histologic type of primary ovarian cancer, review of such specimens by competent colleagues (even if only one is available) deserves serious consideration. No matter what role a physician plays in caring for women at risk of ovarian cancer, their survival depends on correct identification of the origin of the malignancy and may be influenced by histologic type as well as by clinical characteristics. For these reasons, the precise diagnosis of ovarian cancer by pathologists is important for researchers, clinicians, and women with this condition. l3 We acknowledge the contributors to the Cancer and Steroid Hormone Study. Study design and cordination: Division of Reproductive Health, Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control; principal investigator: George L. Rubin, MB; project director: Phyllis A. Wingo, MS; project associates: Nancy C. Lee, MD, Michele G. Mandel, BA, and Herbert B. Peterson, MD; data collection centers principal investigators: Atlanta, Raymond Greenberg, MD; Connecticut,]. Wister Meigs, MD; and W. Douglas Thompson, PhD; Detroit, G. Marie Swanson, PhD; Iowa, Elaine Smith, PhD; New Mexico, Charles Key, MD, and Dorothy Pathak, PhD; San Francisco, Donald Austin, MD; Seattle, David Thomas, MD; Utah, Joseph Lyon, MD, and Dee West, PhD; pathology review principal investigators: Fred Gorstein, MD, Robert McDivitt, MD, and Stanley]. Robboy, MD; project
January 1991 Am J Obstet Gynecol
consultants: Lonnie Burnett, MD, Robert Hoover, MD, Peter M. Layde, MD, MSc, Howard W. Ory, MD, MSc, James]. Schlesselman, PhD, David Schottenfeld, MD, Bruce Stadel, MD, Linda A. Webster, MSPH, and Colin White, MB, BS; pathology consultants: Walter Bauer, MD, William Christopherson, MD, Deborah Gersell, MD, Robert]. Kurman, MD, Allen L. Paris, MD, and Frank Vellios, MD.
REFERENCES I. American Cancer Society. Cancer facts and figures1989. Atlanta: American Cancer Society, 1989. 2. Wingo PA, Ory HW, Layde PM, et al. The evaluation of the data collection process for a multicenter, populationbased, case-control design. Am J Epidemiol 1988; 128: 206-17. 3. The Cancer and Steroid Hormone Study of the Centers for Disease Control and the National Institute of Child Health and Human Development. The reduction in risk of ovarian cancer associated with oral-contraceptive use. N EnglJ Med 1987;316:650-5. 4. Baak JPA, Lindeman J, Overdiep SH, Langley FA. Disagreement of histopathological diagnoses of different pathologists in ovarian tumors-with some theoretical considerations. Eur J Obstet Gynec Reprod BioI 1982; 13: 51-5. 5. Cooke RA. Quality control in anatomic pathology: experience in Australia and New Zealand. In: Somers SC, Rosen PP, eds. Pathology annual. Connecticut: AppletonCentury-Crofts, 1984; 19:221-48. 6. Hernandez E, Bhagavan BS, Parmley TH, Rosenshein NB. Interobserver variability in the interpretation of epithelial ovarian cancer. Gynecol Oncol 1984; 17: 117-23. 7. Cramer SF, Roth LM, Ulbright TM, et al. Evaluation of the reproducibility of the World Health Organization classification of common ovarian cancers. Arch Pathol Lab Med 1987;111:819-29. 8. Fletcher RH, Fletcher SW, Wagner EH. Clinical epidemiology-the essentials. Baltimore: Williams and Wilkins, 1982. 9. Serov SF, Scully RE, Sobin LH. International histological classification of tumors. Geneva: World Health Organization, 1973. 10. Fleiss JL. Statistical methods for rates and proportions. New York: John Wiley & Sons, 1981. II. Kim H, Zelman RJ, Fox MA, et al. Pathology panel for lymphoma clinical studies: a comprehensive analysis of cases accumulated since its inception. JNCI 1982;68:4367. 12. Maclure M, Willett WC. Misinterpretation and misuse of the K statistic. AmJ EpidemioI1987;126:161-169. 13. Aure JC, Hoeg K, Kolstad P. Clinical and histologic studies of ovarian cancer. Ohstet GynecoI1971;37:1-9.
