The Tahoe Study Bias in the Interpretation of Papanicolaou Test Results When Human Papillomavirus Status Is Known Ann T. Moriarty, MD; Ritu Nayar, MD; Terry Arnold, CT(ASCP); Lisa Gearries, CT(ASCP); Andrew Renshaw, MD; Nicole Thomas, MPH, CT(ASCP); Rhona Souers, MS
Context.—Knowledge of human papillomavirus (HPV) status is expected to bias the morphologic evaluation of Papanicolaou (Pap) test results. Objective.—To characterize Pap test result interpretive bias when the HPV status is known at the microscopic evaluation. Design.—Forty HPV-positive liquid-based Pap test results initially interpreted as negative for squamous intraepithelial lesion or malignancy were selected from a quality assurance program, separated into 2 groups of 20 slides each, and circulated in 2 groups to 22 members of the College of American Pathologists Cytopathology Committee. Each member reviewed each case and indicated whether the result was negative for squamous intraepithelial lesion or malignancy or was an epithelial cell abnormality (ECA). The participants assessed the severity of ECAs using the Bethesda System. The participants were not informed of the HPV status in the initial
review round. Each group of 20 slides was then distributed to the opposite group (to avoid slide recall), and the participants were informed that all slides were from patients who were high-risk HPV positive. Differences in the responses between groups were analyzed by v2 test and Cochran-Mantel-Haenszel test at the .05 significance level. Results.—Without knowledge of the HPV status, slides were more often categorized as negative for squamous intraepithelial lesion or malignancy and less likely identified as an ECA (P , .001). There was an increase across all categories of ECAs in the biased responses compared with the unbiased responses (P ¼ .002). Conclusions.—Knowledge of positive HPV status biases morphologic Pap test result interpretation. If the HPV status is positive, observers are more likely to report a Pap test result as abnormal across all categories of ECAs. (Arch Pathol Lab Med. 2014;138:1182–1185; doi: 10.5858/arpa.2012-0115-CP)
T
the occurrence of extensive invasive carcinoma. Second, the long natural history of invasive cervical cancer is such that there are multiple opportunities to identify the precursor lesions over time before invasion.2 Pap tests are variably sensitive for detecting cervical intraepithelial neoplasm (CIN). The reported sensitivity of the Pap test for detecting CIN2 or higher (CIN2þ) ranges from 55.4% to 92%.3,4 The effect of variable sensitivity of the Pap test is counterbalanced by the multiple opportunities to identify abnormalities during the development of carcinoma. Human papillomavirus (HPV) infection is common, and most infections are transient; it is the persistence of HPV and the integration of HPV into the host genome that are essential to cancer development.5,6 Most current HPV tests detect the presence of oncogenic HPV but do not differentiate between women with transient lesions and those with an integrated virus genome. With the introduction of the HPV vaccine, oncogenic HPV infections have decreased in those vaccinated, resulting in a reduction in the number of women with significant cervical lesions that may develop into invasive cancer.7,8 Regardless of the screening test used, detecting the vanishing significant abnormality becomes more difficult as the lowrisk population increases: the ‘‘haystack’’ is increasing in size, and there are fewer ‘‘needles’’ to find.9
he Papanicolaou (Pap) smear is the most successful cancer screening test in the United States, with a 70% decrease in the incidence of invasive cervical cancer since its implementation in the 1950s.1 The success of the Pap smear as a screening test may be a result of 2 factors. First, the Pap test detects abnormalities of the cervix before cancer is present, allowing early intervention and treatment before Accepted for publication November 8, 2013. From Esoteric Testing, AmeriPath Indiana (Dr Moriarty), and Cytology, Mid America Clinical Laboratories (Mr Arnold and Ms Gearries), Indianapolis; the Department of Cytopathology, Northwestern University Medical Center, Chicago (Dr Nayar), and Surveys Department, College of American Pathologists (Ms Thomas), and Biostatistics Department, College of American Pathologists (Ms Souers), Northfield, Illinois; and the Department of Pathology, Baptist Memorial Hospital, Miami, Florida (Dr Renshaw). The authors have no relevant financial interest in the products or companies described in this article. Presented as a platform abstract at the 59th Annual Scientific Meeting of the American Society of Cytopathology; November 6, 2011; Baltimore, Maryland. Reprints: Ann T. Moriarty, MD, Esoteric Testing, AmeriPath Indiana, Indianapolis, IN 46219 (e-mail: Amoriarty@ameripath. com). Drs Moriarty (chair) and Nayar are members of the College of American Pathologists Cytopathology Committee. 1182 Arch Pathol Lab Med—Vol 138, September 2014
The Tahoe Study—Moriarty et al
Human papillomavirus virus testing is a more sensitive test for detecting women with significant precursor lesions among those older than 30 years, with sensitivities exceeding 95% reported for CIN2þ.10,11 Because of its increased sensitivity, there is an interest in using HPV as a primary screening test. While HPV testing is more sensitive than Pap tests, it is less specific (94.1% versus 96.8%).3 If HPV testing is used as a primary screening test, without a secondary triage test, the rate of colposcopies would be markedly increased.3,12 In the scenario of primary HPV testing, the Pap test has been identified as a promising second test after a positive HPV primary screen.3,11 If HPV testing is used as a primary screening test, will the Pap test perform differently using the same morphologic criteria that are now used? Specifically, if the observer knows that the HPV status is positive, will the Pap test result more likely be abnormal? Because HPV infection, persistence, and integration are considered necessary precursors to the development of most cervical cancers,5,6 it follows that a population entirely composed of HPV-positive women would be ‘‘enriched’’ for those with morphologic abnormalities. The cytological findings would be expected to encompass those cytological entities without significant patient effect (atypical squamous cells of undetermined significance [ASC-US] and low-grade squamous intraepithelial lesions [LSILs]), as well as those that may have clinical influence (high-grade squamous intraepithelial lesions [HSILs]). While the abnormal rate in an HPV-positive population is expected to be higher, will there be an additional propensity to ‘‘overcall’’ minor cellular changes as epithelial cell abnormalities (ECAs) when the reviewer knows the HPV status before review of the morphology? In this study, members of the College of American Pathologists Cytopathology Committee sought to investigate the scope of change in Pap test result interpretation with and without the knowledge of the HPV status. MATERIALS AND METHODS All slides in this study had been used in an active quality assurance program (Mid America Clinical Laboratories, Indianapolis, Indiana). The slides were used as part of a retrospective review of negative slides with a positive HPV test result to produce a population enriched with high-risk patients among whom falsenegative Pap test results could be identified. Forty slides (ThinPrep; Hologic, Marlborough, Massachusetts) were selected that had been originally interpreted as NILM from a laboratory with a pool of 14 cytotechnologists. All 40 cases had corresponding positive highrisk HPV results obtained from the original vial. A DNA test (Digene Hybrid Capture 2; Qiagen, Venlo, the Netherlands) was used for all HPV testing. These slides had been used for quality assurance between 2009 and 2010. The slides consisted of various final interpretations that ranged from NILM to HSIL. The cases were divided into 2 sets of 20 slides each; the slide sets were identical in terms of the final quality assurance interpretation (9 cases of NILM and 11 cases of ECA).
Table 1.
There were 22 members (18 pathologists and 4 cytotechnologists) of the College of American Pathologists Cytopathology Committee available for slide review during a committee meeting at Lake Tahoe, California, in July 2011. The participating committee members were unaware of the aim of the study, the ultimate interpretation of the slides, or the origin of the slides to be examined. The participants were told only that they were to review the Pap test results independently and to record their interpretation on the provided standardized result form. The committee was divided into 2 groups. The 2 slide sets were given to each group of 11 members. Using the Bethesda System,13 the committee members indicated whether slides met the criteria for the general categories of NILM or ECA. If a slide was considered an ECA, observers were instructed to indicate one of the following specific reference interpretations: ASC-US, LSIL, or HSIL. During the initial review, the HPV status was withheld from the participants. After completion of the initial review, the slides were collected and relabeled as ‘‘B’’ sets and distributed to the opposite group (to avoid slide recall), and the participants were informed before review that all slides were from patients who were high-risk HPV positive. The participants were then to apply the same criteria and indicate their interpretation as NILM or ECA; if abnormal, they were to subcategorize as ASC-US, LSIL, or HSIL. All reviews were completed the same day; the initial review without HPV status knowledge was completed in the morning, and the second review was completed in the afternoon. Responses were recorded, the responders’ identities were hidden, and each response was evaluated against the general categories of NILM and ECA, as well as the descriptive categories of ASC-US, LSIL, or HSIL. Differences in the responses between groups were analyzed statistically by v2 test and Cochran-MantelHaenszel test at the .05 significance level.
RESULTS From a total of 960 possible responses, 41 were unavailable; one observer failed to review both sets of slides, and 2 responses were given for a single slide by one reviewer. There were no significant differences between the 2 observer groups (P ¼ .41), confirming that the groups were similar in diagnostic interpretation (Table 1). Knowledge of a positive HPV result decreased the frequency of NILM designation (43.1% without knowledge of the HPV status versus 31.3% with knowledge of positive HPV status) and increased the frequency of all ECA categories (56.9% versus 68.7%) (P , .001 for both) (Table 2). Knowledge of the HPV status increased the frequency of all ECA categories (27.2% versus 24.0% for ASC-US, 22.4% versus 15.5% for LSIL, and 19.1% versus 17.4% for HSIL; P ¼ .002) (Table 3). COMMENT Pap testing has been used successfully to decrease the prevalence of squamous cervical cancer in the United States. Over the course of screening, the Pap test should identify significant lesions before the development of invasive cervical cancer, allowing intervention before invasion.
Confirmation of No Significant Difference Between the Response Rates of the 2 Observer Groups (P ¼ .41) Response Category, No. (%)
Variable
NILM
ASC-US
LSIL
HSIL
Total
Group 1 Group 2
136 (37.9) 176 (36.7)
91 (25.3) 124 (25.8)
60 (16.7) 99 (20.6)
72 (20.1) 81 (16.9)
359 480
Abbreviations: ASC-US, atypical squamous cells of undetermined significance; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy. Arch Pathol Lab Med—Vol 138, September 2014
The Tahoe Study—Moriarty et al 1183
Table 2.
General Category Response With Knowledge of the HPV Status (Biased) and Without Knowledge of the HPV Status (Unbiased) (P , .001) Response Category, No. (%)
Variable
NILM
ECA
Total
Observers know positive HPV status Observers do not know the HPV status
131 (31.3) 181 (43.1)
288 (68.7) 239 (56.9)
419 420
Abbreviations: ECA, epithelial cell abnormality; HPV, human papillomavirus; NILM, negative for intraepithelial lesion or malignancy.
Morphologic changes detected by Pap tests are most often those that do not carry an increased risk of invasive carcinoma. The most common abnormalities reported nationally are ASC-US and LSIL.14 These represent low risk of HSIL or invasive carcinoma. Reportedly, 43% of CIN2 and 32% of CIN3 (HSIL) regress.10 Despite that the Pap test successfully detects these morphologic abnormalities, potential harm occurs in patients who are subjected to therapeutic intervention for lesions that may regress. The Pap test is a subjective screening test; although there are well-established criteria for ECAs, the human process of detecting abnormality and uniformly applying diagnostic criteria is fraught with difficulties. The precision in the diagnosis of significant ECAs (HSIL and cancers) is lower than the precision of the interpretation of LSIL and inflammatory cell changes.15 Positive bias is easily introduced into the process of morphologic detection and interpretation of Pap test results because of its subjective nature. An epithelial abnormality may be reported in a Pap test result when no lesion is present (false positive), leading to intervention and patient harm. With the discovery that HPV is an essential factor in the development of invasive squamous cell carcinoma, modern screening and management guidelines have included reflex HPV testing and cotesting of individuals.16 The addition of appropriate HPV testing enhances the appropriate triage of women who may have persistent HPV.17 Human papillomavirus tests detect women with HPV but do not discern between women who have transient HPV versus those who have persistent HPV. If HPV testing is used as a primary screening test, a secondary triage test is necessary to identify those with persistent infection. The Pap test as a triage instrument preliminarily seems to be the most efficient tool.18 Benoy et al19 recently reported a colposcopy controlled study evaluating interpretive bias of cytology with knowledge of the HPV status. In contrast, the present study is not a colposcopy controlled, outcomes study. The results herein of the Tahoe Study confirm that bias occurs in the morphologic interpretation of Pap test results when HPV results are positive. A commercial HVP test (Digene Hybrid Capture 2, Qiagen) was used in the Tahoe Study, while the pathologists in the study by Benoy et al had access to the quantitative results of an HPV polymerase chain reaction. Table 3.
Their biased group identified more ECAs, with a significant difference noted only in LSIL interpretation, while our study identified a significant difference in all specific reference categories of ASC-US, LSIL, and HSIL. Sensitivity increased in the colposcopy controlled study from 58.7% for CIN2þ without prior knowledge of the HPV status to 76.0% with prior knowledge of the HPV status. The improvement in sensitivity differed by age, with women older than 30 years having the greatest increase (from 78.5% to 85.7%) if HPV results were known when interpreting Pap test results. In the study by Benoy et al, specificity decreased in the younger population (from 92.3% to 87.4%) and remained essentially unchanged for older women when observers were biased by HPV results. Benoy et al concluded that the use of HPV as a primary screening modality, followed by biased cytology triage, would increase the detection of CIN2þ in women older than 30 years. According to the Tahoe Study, when cytologists review a Pap test result with knowledge of positive HPV status, an ECA is more likely to be identified than without knowledge of the HPV status. Our study identified an increase in all categories of ECAs and not LSIL alone. This study does not reflect the usual working conditions of most laboratories. The test sets were fabricated to include epithelial abnormalities at a higher rate than might be encountered in a typical laboratory. The participants were not under usual workload constraints, evaluated slides in an unfamiliar environment, and were undoubtedly curious as to the aim of the study. These factors add another source of bias to the study. More generally, the extent of bias is difficult to extrapolate from this study to screened populations. Laboratory practices vary with respect to the availability of HPV status before the review of Pap test results. In the settings that review HPV results before reporting Pap test results, bias has already affected Pap test performance in the populations served by those laboratories. It is apparent from this study that knowledge of the HPV status biases Pap test result interpretation. If observers know that HPV test results are positive, they are more likely to report the Pap test result as abnormal, with greater frequency in all categories of ECAs. This bias may affect the performance of Pap testing in cervical cancer screening programs that use primary HPV testing. The bias may lead
Specific Reference Category With Knowledge of the HPV Status (Biased) and Without Knowledge of the HPV Status (Unbiased) (P ¼ .002) Response Category, No. (%)
Variable
NILM
ASC-US
LSIL
HSIL
Total
Observers know positive HPV status Observers do not know the HPV status
131 (31.3) 181 (43.1)
114 (27.2) 101 (24.0)
94 (22.4) 65 (15.5)
80 (19.1) 73 (17.4)
419 420
Abbreviations: ASC-US, atypical squamous cells of undetermined significance; HPV, human papillomavirus; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy. 1184 Arch Pathol Lab Med—Vol 138, September 2014
The Tahoe Study—Moriarty et al
to increased sensitivity, decreased specificity, increased patient intervention, and potential harm. References 1. Siegel R, Naishadham D, Jemal A. Cancer statistics, 2012. CA Cancer J Clin. 2012;62(1):10–29. 2. Wilbur DC. Gynecologic cytology: approach to the slide and normal morphology. In: Wilbur DC, Henry MR, eds. Gynecologic Cytopathology: Morphology Management and Molecular Methods. Northfield IL: College of American Pathologists; 2008. 3. Mayrand MH, Duarte-Franco E, Rodrigues I, et al. Human papillomavirus DNA versus Papanicolaou screening tests for cervical cancer. N Engl J Med. 2007; 357(16):1579–1588. 4. Kitchener HC, Blanks R, Cubie H, et al; MAVARIC Trial Study Group. MAVARIC: a comparison of automation-assisted and manual cervical screening: a randomised controlled trial. Health Technol Assess. 2011;15(3):iii–iv, ix–xi, 1– 170. 5. Wright TC, Schiffman M. Adding a test for human papillomavirus DNA to cervical cancer screening. N Engl J Med. 2003;348(6):489–490. 6. Schiffman M, Castle PE, Jeronimo J, Rodrigues AC, Wacholder S, Humanpapillomavirus and cervical cancer. Lancet. 2007;370(9590):890–907. ´ J, et al; HPV PATRICIA Study Group. Efficacy 7. Paavonen J, Naud P, Salmeron of human papillomavirus (HPV)-16/18 AS04-adjuvanted vaccine against cervical infection and precancer caused by oncogenic HPV types (PATRICIA): final analysis of a double-blind, randomised study in young women [published correction appears in Lancet. 2010;376(9746):1054]. Lancet. 2009;374(9686): 301–314. 8. Cuzick J, Castanon A, Sasieni P. Predicted impact of vaccination against human papillomavirus 16/18 on cancer incidence and cervical abnormalities in women aged 20–29 in the UK. Br J Cancer. 2010;102(5):933–939. 9. Franco EL, Mahmud SM, Tota J, Ferenczy A, Coutlee F. The expected impact of HPV vaccination on the accuracy of cervical cancer screening: the need for a paradigm change. Arch Med Res. 2009;40(6):478–485, 10. Vesco KK, Whitlock EP, Eder M, et al. Screening for Cervical Cancer: A Systematic Evidence Review for the U.S. Preventive Services Task Force.
Arch Pathol Lab Med—Vol 138, September 2014
Rockville, MD: Agency for Healthcare Research and Quality; May 2011. Evidence Syntheses, No. 86. AHRQ publication 11-05156-EF-1. 11. Leinonen M, Nieminen P, Kotaniemi-Talonen L, et al. Age-specific evaluation of primary human papillomavirus screening vs conventional cytology in a randomized setting. J Natl Cancer Inst. 2009;101(23):1612–1623. 12. Ronco G, Giorgi-Rossi P, Carozzi F, et al; New Technologies for Cervical Cancer Screening (NTCC) Working Group. Efficacy of human papillomavirus testing for the detection of invasive cervical cancers and cervical intraepithelial neoplasia: a randomised controlled trial. Lancet Oncol. 2010;11(3):249–257. 13. Solomon D, Nayar R, eds. The Bethesda System for Reporting Cervical Cytology: Definitions, Criteria and Explanatory Notes. 2nd ed. New York, NY: Springer-Verlag; 2004. 14. Eversole G, Moriarty AT, Schwartz MR, et al. Practices of participants in the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology 2006. Arch Pathol Lab Med. 2010;134(3):331–335. 15. Renshaw A, Davey DD, Birdsong GG, et al. Precision in gynecologic cytologic interpretation: a study from the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology. Arch Pathol Lab Med. 2003;127(11):1413–1420. 16. Massad LS, Einstein MH, Huh WK, et al; 2012 ASCCP Consensus Guidelines Conference. 2012 Updated consensus guidelines for the management of abnormal cervical cancer screening tests and cancer precursors [published correction appears in J Low Genit Tract Dis. 2013;17(3):367]. J Low Genit Tract Dis. 2103;17(5):S1–S27. 17. Saslow D, Solomon D, Lawson HW, et al. American Cancer Society, American Society for Colposcopy and Cervical Pathology, and American Society for Clinical Pathology screening guidelines for the prevention and early detection of cervical cancer. J Low Genit Tract Dis. 2012;16(3):175–204. 18. Kulasingam SL, Havrilesky L, Ghebre R, Myers ER. Screening for Cervical Cancer: A Decision Analysis for the U.S. Preventive Services Task Force. Rockville, MD: Agency for Healthcare Research and Quality; May 2011. AHRQ publication 11-05157-EF-1. 19. Benoy IH, Vanden Broeck DV, Ruymbeke MJ, et al. Prior knowledge of HPV status improves detection of CIN2þ by cytology screening. Am J Obstet Gynecol. 2011;205(6):569.e1–569.e7. http://www.ajog.org/article/ S0002-9378(11)00896-9/fulltext. Accessed November 19, 2013.
The Tahoe Study—Moriarty et al 1185
Copyright of Archives of Pathology & Laboratory Medicine is the property of College of American Pathologists and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Context.—Knowledge of human papillomavirus (HPV) status is expected to bias the morphologic evaluation of Papanicolaou (Pap) test results. Objective.—To characterize Pap test result interpretive bias when the HPV status is known at the microscopic evaluation. Design.—Forty HPV-positive liquid-based Pap test results initially interpreted as negative for squamous intraepithelial lesion or malignancy were selected from a quality assurance program, separated into 2 groups of 20 slides each, and circulated in 2 groups to 22 members of the College of American Pathologists Cytopathology Committee. Each member reviewed each case and indicated whether the result was negative for squamous intraepithelial lesion or malignancy or was an epithelial cell abnormality (ECA). The participants assessed the severity of ECAs using the Bethesda System. The participants were not informed of the HPV status in the initial
review round. Each group of 20 slides was then distributed to the opposite group (to avoid slide recall), and the participants were informed that all slides were from patients who were high-risk HPV positive. Differences in the responses between groups were analyzed by v2 test and Cochran-Mantel-Haenszel test at the .05 significance level. Results.—Without knowledge of the HPV status, slides were more often categorized as negative for squamous intraepithelial lesion or malignancy and less likely identified as an ECA (P , .001). There was an increase across all categories of ECAs in the biased responses compared with the unbiased responses (P ¼ .002). Conclusions.—Knowledge of positive HPV status biases morphologic Pap test result interpretation. If the HPV status is positive, observers are more likely to report a Pap test result as abnormal across all categories of ECAs. (Arch Pathol Lab Med. 2014;138:1182–1185; doi: 10.5858/arpa.2012-0115-CP)
T
the occurrence of extensive invasive carcinoma. Second, the long natural history of invasive cervical cancer is such that there are multiple opportunities to identify the precursor lesions over time before invasion.2 Pap tests are variably sensitive for detecting cervical intraepithelial neoplasm (CIN). The reported sensitivity of the Pap test for detecting CIN2 or higher (CIN2þ) ranges from 55.4% to 92%.3,4 The effect of variable sensitivity of the Pap test is counterbalanced by the multiple opportunities to identify abnormalities during the development of carcinoma. Human papillomavirus (HPV) infection is common, and most infections are transient; it is the persistence of HPV and the integration of HPV into the host genome that are essential to cancer development.5,6 Most current HPV tests detect the presence of oncogenic HPV but do not differentiate between women with transient lesions and those with an integrated virus genome. With the introduction of the HPV vaccine, oncogenic HPV infections have decreased in those vaccinated, resulting in a reduction in the number of women with significant cervical lesions that may develop into invasive cancer.7,8 Regardless of the screening test used, detecting the vanishing significant abnormality becomes more difficult as the lowrisk population increases: the ‘‘haystack’’ is increasing in size, and there are fewer ‘‘needles’’ to find.9
he Papanicolaou (Pap) smear is the most successful cancer screening test in the United States, with a 70% decrease in the incidence of invasive cervical cancer since its implementation in the 1950s.1 The success of the Pap smear as a screening test may be a result of 2 factors. First, the Pap test detects abnormalities of the cervix before cancer is present, allowing early intervention and treatment before Accepted for publication November 8, 2013. From Esoteric Testing, AmeriPath Indiana (Dr Moriarty), and Cytology, Mid America Clinical Laboratories (Mr Arnold and Ms Gearries), Indianapolis; the Department of Cytopathology, Northwestern University Medical Center, Chicago (Dr Nayar), and Surveys Department, College of American Pathologists (Ms Thomas), and Biostatistics Department, College of American Pathologists (Ms Souers), Northfield, Illinois; and the Department of Pathology, Baptist Memorial Hospital, Miami, Florida (Dr Renshaw). The authors have no relevant financial interest in the products or companies described in this article. Presented as a platform abstract at the 59th Annual Scientific Meeting of the American Society of Cytopathology; November 6, 2011; Baltimore, Maryland. Reprints: Ann T. Moriarty, MD, Esoteric Testing, AmeriPath Indiana, Indianapolis, IN 46219 (e-mail: Amoriarty@ameripath. com). Drs Moriarty (chair) and Nayar are members of the College of American Pathologists Cytopathology Committee. 1182 Arch Pathol Lab Med—Vol 138, September 2014
The Tahoe Study—Moriarty et al
Human papillomavirus virus testing is a more sensitive test for detecting women with significant precursor lesions among those older than 30 years, with sensitivities exceeding 95% reported for CIN2þ.10,11 Because of its increased sensitivity, there is an interest in using HPV as a primary screening test. While HPV testing is more sensitive than Pap tests, it is less specific (94.1% versus 96.8%).3 If HPV testing is used as a primary screening test, without a secondary triage test, the rate of colposcopies would be markedly increased.3,12 In the scenario of primary HPV testing, the Pap test has been identified as a promising second test after a positive HPV primary screen.3,11 If HPV testing is used as a primary screening test, will the Pap test perform differently using the same morphologic criteria that are now used? Specifically, if the observer knows that the HPV status is positive, will the Pap test result more likely be abnormal? Because HPV infection, persistence, and integration are considered necessary precursors to the development of most cervical cancers,5,6 it follows that a population entirely composed of HPV-positive women would be ‘‘enriched’’ for those with morphologic abnormalities. The cytological findings would be expected to encompass those cytological entities without significant patient effect (atypical squamous cells of undetermined significance [ASC-US] and low-grade squamous intraepithelial lesions [LSILs]), as well as those that may have clinical influence (high-grade squamous intraepithelial lesions [HSILs]). While the abnormal rate in an HPV-positive population is expected to be higher, will there be an additional propensity to ‘‘overcall’’ minor cellular changes as epithelial cell abnormalities (ECAs) when the reviewer knows the HPV status before review of the morphology? In this study, members of the College of American Pathologists Cytopathology Committee sought to investigate the scope of change in Pap test result interpretation with and without the knowledge of the HPV status. MATERIALS AND METHODS All slides in this study had been used in an active quality assurance program (Mid America Clinical Laboratories, Indianapolis, Indiana). The slides were used as part of a retrospective review of negative slides with a positive HPV test result to produce a population enriched with high-risk patients among whom falsenegative Pap test results could be identified. Forty slides (ThinPrep; Hologic, Marlborough, Massachusetts) were selected that had been originally interpreted as NILM from a laboratory with a pool of 14 cytotechnologists. All 40 cases had corresponding positive highrisk HPV results obtained from the original vial. A DNA test (Digene Hybrid Capture 2; Qiagen, Venlo, the Netherlands) was used for all HPV testing. These slides had been used for quality assurance between 2009 and 2010. The slides consisted of various final interpretations that ranged from NILM to HSIL. The cases were divided into 2 sets of 20 slides each; the slide sets were identical in terms of the final quality assurance interpretation (9 cases of NILM and 11 cases of ECA).
Table 1.
There were 22 members (18 pathologists and 4 cytotechnologists) of the College of American Pathologists Cytopathology Committee available for slide review during a committee meeting at Lake Tahoe, California, in July 2011. The participating committee members were unaware of the aim of the study, the ultimate interpretation of the slides, or the origin of the slides to be examined. The participants were told only that they were to review the Pap test results independently and to record their interpretation on the provided standardized result form. The committee was divided into 2 groups. The 2 slide sets were given to each group of 11 members. Using the Bethesda System,13 the committee members indicated whether slides met the criteria for the general categories of NILM or ECA. If a slide was considered an ECA, observers were instructed to indicate one of the following specific reference interpretations: ASC-US, LSIL, or HSIL. During the initial review, the HPV status was withheld from the participants. After completion of the initial review, the slides were collected and relabeled as ‘‘B’’ sets and distributed to the opposite group (to avoid slide recall), and the participants were informed before review that all slides were from patients who were high-risk HPV positive. The participants were then to apply the same criteria and indicate their interpretation as NILM or ECA; if abnormal, they were to subcategorize as ASC-US, LSIL, or HSIL. All reviews were completed the same day; the initial review without HPV status knowledge was completed in the morning, and the second review was completed in the afternoon. Responses were recorded, the responders’ identities were hidden, and each response was evaluated against the general categories of NILM and ECA, as well as the descriptive categories of ASC-US, LSIL, or HSIL. Differences in the responses between groups were analyzed statistically by v2 test and Cochran-MantelHaenszel test at the .05 significance level.
RESULTS From a total of 960 possible responses, 41 were unavailable; one observer failed to review both sets of slides, and 2 responses were given for a single slide by one reviewer. There were no significant differences between the 2 observer groups (P ¼ .41), confirming that the groups were similar in diagnostic interpretation (Table 1). Knowledge of a positive HPV result decreased the frequency of NILM designation (43.1% without knowledge of the HPV status versus 31.3% with knowledge of positive HPV status) and increased the frequency of all ECA categories (56.9% versus 68.7%) (P , .001 for both) (Table 2). Knowledge of the HPV status increased the frequency of all ECA categories (27.2% versus 24.0% for ASC-US, 22.4% versus 15.5% for LSIL, and 19.1% versus 17.4% for HSIL; P ¼ .002) (Table 3). COMMENT Pap testing has been used successfully to decrease the prevalence of squamous cervical cancer in the United States. Over the course of screening, the Pap test should identify significant lesions before the development of invasive cervical cancer, allowing intervention before invasion.
Confirmation of No Significant Difference Between the Response Rates of the 2 Observer Groups (P ¼ .41) Response Category, No. (%)
Variable
NILM
ASC-US
LSIL
HSIL
Total
Group 1 Group 2
136 (37.9) 176 (36.7)
91 (25.3) 124 (25.8)
60 (16.7) 99 (20.6)
72 (20.1) 81 (16.9)
359 480
Abbreviations: ASC-US, atypical squamous cells of undetermined significance; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy. Arch Pathol Lab Med—Vol 138, September 2014
The Tahoe Study—Moriarty et al 1183
Table 2.
General Category Response With Knowledge of the HPV Status (Biased) and Without Knowledge of the HPV Status (Unbiased) (P , .001) Response Category, No. (%)
Variable
NILM
ECA
Total
Observers know positive HPV status Observers do not know the HPV status
131 (31.3) 181 (43.1)
288 (68.7) 239 (56.9)
419 420
Abbreviations: ECA, epithelial cell abnormality; HPV, human papillomavirus; NILM, negative for intraepithelial lesion or malignancy.
Morphologic changes detected by Pap tests are most often those that do not carry an increased risk of invasive carcinoma. The most common abnormalities reported nationally are ASC-US and LSIL.14 These represent low risk of HSIL or invasive carcinoma. Reportedly, 43% of CIN2 and 32% of CIN3 (HSIL) regress.10 Despite that the Pap test successfully detects these morphologic abnormalities, potential harm occurs in patients who are subjected to therapeutic intervention for lesions that may regress. The Pap test is a subjective screening test; although there are well-established criteria for ECAs, the human process of detecting abnormality and uniformly applying diagnostic criteria is fraught with difficulties. The precision in the diagnosis of significant ECAs (HSIL and cancers) is lower than the precision of the interpretation of LSIL and inflammatory cell changes.15 Positive bias is easily introduced into the process of morphologic detection and interpretation of Pap test results because of its subjective nature. An epithelial abnormality may be reported in a Pap test result when no lesion is present (false positive), leading to intervention and patient harm. With the discovery that HPV is an essential factor in the development of invasive squamous cell carcinoma, modern screening and management guidelines have included reflex HPV testing and cotesting of individuals.16 The addition of appropriate HPV testing enhances the appropriate triage of women who may have persistent HPV.17 Human papillomavirus tests detect women with HPV but do not discern between women who have transient HPV versus those who have persistent HPV. If HPV testing is used as a primary screening test, a secondary triage test is necessary to identify those with persistent infection. The Pap test as a triage instrument preliminarily seems to be the most efficient tool.18 Benoy et al19 recently reported a colposcopy controlled study evaluating interpretive bias of cytology with knowledge of the HPV status. In contrast, the present study is not a colposcopy controlled, outcomes study. The results herein of the Tahoe Study confirm that bias occurs in the morphologic interpretation of Pap test results when HPV results are positive. A commercial HVP test (Digene Hybrid Capture 2, Qiagen) was used in the Tahoe Study, while the pathologists in the study by Benoy et al had access to the quantitative results of an HPV polymerase chain reaction. Table 3.
Their biased group identified more ECAs, with a significant difference noted only in LSIL interpretation, while our study identified a significant difference in all specific reference categories of ASC-US, LSIL, and HSIL. Sensitivity increased in the colposcopy controlled study from 58.7% for CIN2þ without prior knowledge of the HPV status to 76.0% with prior knowledge of the HPV status. The improvement in sensitivity differed by age, with women older than 30 years having the greatest increase (from 78.5% to 85.7%) if HPV results were known when interpreting Pap test results. In the study by Benoy et al, specificity decreased in the younger population (from 92.3% to 87.4%) and remained essentially unchanged for older women when observers were biased by HPV results. Benoy et al concluded that the use of HPV as a primary screening modality, followed by biased cytology triage, would increase the detection of CIN2þ in women older than 30 years. According to the Tahoe Study, when cytologists review a Pap test result with knowledge of positive HPV status, an ECA is more likely to be identified than without knowledge of the HPV status. Our study identified an increase in all categories of ECAs and not LSIL alone. This study does not reflect the usual working conditions of most laboratories. The test sets were fabricated to include epithelial abnormalities at a higher rate than might be encountered in a typical laboratory. The participants were not under usual workload constraints, evaluated slides in an unfamiliar environment, and were undoubtedly curious as to the aim of the study. These factors add another source of bias to the study. More generally, the extent of bias is difficult to extrapolate from this study to screened populations. Laboratory practices vary with respect to the availability of HPV status before the review of Pap test results. In the settings that review HPV results before reporting Pap test results, bias has already affected Pap test performance in the populations served by those laboratories. It is apparent from this study that knowledge of the HPV status biases Pap test result interpretation. If observers know that HPV test results are positive, they are more likely to report the Pap test result as abnormal, with greater frequency in all categories of ECAs. This bias may affect the performance of Pap testing in cervical cancer screening programs that use primary HPV testing. The bias may lead
Specific Reference Category With Knowledge of the HPV Status (Biased) and Without Knowledge of the HPV Status (Unbiased) (P ¼ .002) Response Category, No. (%)
Variable
NILM
ASC-US
LSIL
HSIL
Total
Observers know positive HPV status Observers do not know the HPV status
131 (31.3) 181 (43.1)
114 (27.2) 101 (24.0)
94 (22.4) 65 (15.5)
80 (19.1) 73 (17.4)
419 420
Abbreviations: ASC-US, atypical squamous cells of undetermined significance; HPV, human papillomavirus; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy. 1184 Arch Pathol Lab Med—Vol 138, September 2014
The Tahoe Study—Moriarty et al
to increased sensitivity, decreased specificity, increased patient intervention, and potential harm. References 1. Siegel R, Naishadham D, Jemal A. Cancer statistics, 2012. CA Cancer J Clin. 2012;62(1):10–29. 2. Wilbur DC. Gynecologic cytology: approach to the slide and normal morphology. In: Wilbur DC, Henry MR, eds. Gynecologic Cytopathology: Morphology Management and Molecular Methods. Northfield IL: College of American Pathologists; 2008. 3. Mayrand MH, Duarte-Franco E, Rodrigues I, et al. Human papillomavirus DNA versus Papanicolaou screening tests for cervical cancer. N Engl J Med. 2007; 357(16):1579–1588. 4. Kitchener HC, Blanks R, Cubie H, et al; MAVARIC Trial Study Group. MAVARIC: a comparison of automation-assisted and manual cervical screening: a randomised controlled trial. Health Technol Assess. 2011;15(3):iii–iv, ix–xi, 1– 170. 5. Wright TC, Schiffman M. Adding a test for human papillomavirus DNA to cervical cancer screening. N Engl J Med. 2003;348(6):489–490. 6. Schiffman M, Castle PE, Jeronimo J, Rodrigues AC, Wacholder S, Humanpapillomavirus and cervical cancer. Lancet. 2007;370(9590):890–907. ´ J, et al; HPV PATRICIA Study Group. Efficacy 7. Paavonen J, Naud P, Salmeron of human papillomavirus (HPV)-16/18 AS04-adjuvanted vaccine against cervical infection and precancer caused by oncogenic HPV types (PATRICIA): final analysis of a double-blind, randomised study in young women [published correction appears in Lancet. 2010;376(9746):1054]. Lancet. 2009;374(9686): 301–314. 8. Cuzick J, Castanon A, Sasieni P. Predicted impact of vaccination against human papillomavirus 16/18 on cancer incidence and cervical abnormalities in women aged 20–29 in the UK. Br J Cancer. 2010;102(5):933–939. 9. Franco EL, Mahmud SM, Tota J, Ferenczy A, Coutlee F. The expected impact of HPV vaccination on the accuracy of cervical cancer screening: the need for a paradigm change. Arch Med Res. 2009;40(6):478–485, 10. Vesco KK, Whitlock EP, Eder M, et al. Screening for Cervical Cancer: A Systematic Evidence Review for the U.S. Preventive Services Task Force.
Arch Pathol Lab Med—Vol 138, September 2014
Rockville, MD: Agency for Healthcare Research and Quality; May 2011. Evidence Syntheses, No. 86. AHRQ publication 11-05156-EF-1. 11. Leinonen M, Nieminen P, Kotaniemi-Talonen L, et al. Age-specific evaluation of primary human papillomavirus screening vs conventional cytology in a randomized setting. J Natl Cancer Inst. 2009;101(23):1612–1623. 12. Ronco G, Giorgi-Rossi P, Carozzi F, et al; New Technologies for Cervical Cancer Screening (NTCC) Working Group. Efficacy of human papillomavirus testing for the detection of invasive cervical cancers and cervical intraepithelial neoplasia: a randomised controlled trial. Lancet Oncol. 2010;11(3):249–257. 13. Solomon D, Nayar R, eds. The Bethesda System for Reporting Cervical Cytology: Definitions, Criteria and Explanatory Notes. 2nd ed. New York, NY: Springer-Verlag; 2004. 14. Eversole G, Moriarty AT, Schwartz MR, et al. Practices of participants in the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology 2006. Arch Pathol Lab Med. 2010;134(3):331–335. 15. Renshaw A, Davey DD, Birdsong GG, et al. Precision in gynecologic cytologic interpretation: a study from the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology. Arch Pathol Lab Med. 2003;127(11):1413–1420. 16. Massad LS, Einstein MH, Huh WK, et al; 2012 ASCCP Consensus Guidelines Conference. 2012 Updated consensus guidelines for the management of abnormal cervical cancer screening tests and cancer precursors [published correction appears in J Low Genit Tract Dis. 2013;17(3):367]. J Low Genit Tract Dis. 2103;17(5):S1–S27. 17. Saslow D, Solomon D, Lawson HW, et al. American Cancer Society, American Society for Colposcopy and Cervical Pathology, and American Society for Clinical Pathology screening guidelines for the prevention and early detection of cervical cancer. J Low Genit Tract Dis. 2012;16(3):175–204. 18. Kulasingam SL, Havrilesky L, Ghebre R, Myers ER. Screening for Cervical Cancer: A Decision Analysis for the U.S. Preventive Services Task Force. Rockville, MD: Agency for Healthcare Research and Quality; May 2011. AHRQ publication 11-05157-EF-1. 19. Benoy IH, Vanden Broeck DV, Ruymbeke MJ, et al. Prior knowledge of HPV status improves detection of CIN2þ by cytology screening. Am J Obstet Gynecol. 2011;205(6):569.e1–569.e7. http://www.ajog.org/article/ S0002-9378(11)00896-9/fulltext. Accessed November 19, 2013.
The Tahoe Study—Moriarty et al 1185
Copyright of Archives of Pathology & Laboratory Medicine is the property of College of American Pathologists and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
