Vol. 29, No. 2
JOURNAL OF CLINICAL MICROBIOLOGY, Feb. 1991, p. 283-286
0095-1137/91/020283-04$02.00/0
Usefulness of a New Serological Test (Bio-Rad) To Diagnose Helicobacter pylori-Associated Gastritis H. L. A. VAN DEN GEVER,1 R. J. L. F. LOFFELD,2 AND E. E. STOBBERINGH1* Departments of Medical Microbiology' and Internal Medicine,2 University of Limburg, P.O. Box 616, 6200 MD Maastricht, The Netherlands Received 23 July 1990/Accepted 9 November 1990
A semiquantitative serological test (G.A.P. test; Bio-Rad) to diagnose Helicobacter (Campylobacter) pyloni-associated gastritis has recently become commercially available. This test was evaluated with sera from 72 dyspeptic patients with known H. pylori status. When the instructions provided by the manufacturer were applied strictly, the sensitivity of the test was only 32.7%. With a slight modification of the instructions, the performance characteristics were still unsatisfactory (specificity, 43.5%). When a new cutoff value was determined by a receiver operating curve, the quality of the test (positive predictive value, 95.6%; negative predictive value, 77.8%) was slightly lower than that of a noncommercial enzyme-linked immunosorbent assay (positive predictive value, 95.7%; negative predictive value, 84.0%). It is concluded that the Bio-Rad G.A.P. test has low diagnostic yield, and it is stressed that commercial tests should be evaluated extensively before being applied in daily practice.
GmbH, Munich, Federal Republic of Germany), and the suspension was stored at a final concentration of 1,500 mg/ml, preserved with 0.1% sodium azide, at -20°C until used. Optimal concentrations were determined by checkerboard titration. To coat the wells, we pipetted 100 ,ul of the antigen suspension, diluted 1:1,000 in 60 mM carbonate buffer (pH 9.6), or 100 p.l of buffer only as a control into the wells of flexible Titertek immunoassay plates (Flow Laboratories, Amsterdam, The Netherlands) and let the plates stand overnight at 4°C. After the plates were washed three times with PBS containing 0.5% Tween 20 (Merck-Schuchard, Hohenbrunn, Federal Republic of Germany), uncoated active sites were saturated by incubation with 100 p.1 of 1% bovine serum albumin (Merck, Darmstadt, Federal Republic of Germany) in PBS for 1 h at 37°C. The plates were washed, sealed with Parafilm, and stored at 4°C. For the test assay, 50 p.1 of serum, diluted 1:100 in PBS, was pipetted into both coated and control wells in triplicate and incubated at 37°C for 1 h. The washing step was repeated, and 50 p.1 of a 1:1,000 dilution of peroxidase-conjugated rabbit immunoglobulins to human immunoglobulin G (IgG) (DAKO Immunoglobulins, Copenhagen, Denmark) in PBS was added. The plates were incubated at 37°C for 1 h and washed. For the color reaction, we used 50 p.1 of a solution of 20 p.g of ortho-phenylenediamine in 50 ml of citratephosphate buffer (pH 5.0) to which 10 p.1 of 30% hydrogen peroxide was added. The reaction was stopped after 10 min with 50 p.1 of 4.0 M sulfuric acid and read at 492 nm. Sample scores were expressed as positive/negative (P/N) ratios by dividing the optical densities (OD) of the patient sera by the OD of a pooled negative serum included in every run. Bio-Rad G.A.P. test. The G.A.P. test, a semiquantitative test for the detection of H. pylori-specific IgG antibodies, was obtained from the Clinical Division, Bio-Rad Laboratories. H. pylori group-specific antigens were purified and immobilized on microwells. All procedures were carried out as described by the manufacturer. Positive control, negative control, and patient sera were added to the appropriate coated wells. One hundred microliters of diluted patient serum (25 p.1 of patient serum and 5 ml of diluent buffer
Since the first description by Marshall and Warren (6a) of a curved bacillus in the stomachs of patients with gastritis and peptic ulceration, Helicobacter pylori has been recognized as a common cause of type B (antral) gastritis and probably peptic ulcers. For the diagnosis of H. pyloriassociated gastritis, histological staining and/or culturing of biopsy specimens has been considered the "gold standard" (1). However, to facilitate the diagnosis, noninvasive techniques, such as serological detection of anti-H. pylori antibodies, would be preferable (7, 9). In a previous study, the diagnostic value of an enzyme-linked immunosorbent assay (ELISA) with a sonicated bacterial suspension as an antigen was described (5). In the present study, the usefulness of a recently available commercial ELISA (G.A.P. test; Bio-Rad Laboratories, Milan, Italy) was compared with that of the one previously described with sera from dyspeptic patients with histologically proven gastritis associated with H. pylori infections and from dyspeptic patients with normal antral mucosae. MATERIALS AND METHODS Serum samples. Serum samples from 72 patients with dyspepsia and described in detail by Loffeld et al. (5) were tested. By histological examination (hematoxylin, eosin, and modified Giemsa stains) and culturing of antral biopsies, 49 patients were found positive and 23 patients were found negative for H. pylori. The cross-reactivity of these sera with Campylobacterjejuni was excluded as described previously
(5).
ELISA of SBS. The ELISA for the sonicated bacterial suspension (SBS) was slightly modified from that in earlier reports (3, 5, 6). Five H. pylori strains were cultured at 37°C on blood agar under microaerobic conditions (10% CO2 and 5% 02). After 5 to 6 days of incubation, the cells were harvested, washed twice in phosphate-buffered saline (PBS), heated at 60°C for 30 min, and sonicated three times for 20 s each time. The protein concentration of the suspension was determined by standard methods (Bio-Rad Laboratories * Corresponding author. 283
284
J. CLIN. MICROBIOL.
VAN DEN OEVER ET AL.
P/N
5.0
TABLE 1. Diagnostic value of the two ELISA procedures
-
No. of patients with: Test and result
Antral gastritis and H. pylori
No antral gastritis or
45 4
2 21
16 33
0 23
Method II Positive Midrange (+/-) Negative
45 2 2
11 2 10
Method III Positive Negative
43 6
2 21
H.
pylori
SBS ELISA
3.0
Positive
I
4.0
Negative Bio-Rad G.A.P. test Method I Positive
-
Negative
2.0 -
I
1.40 I
1.0
I
0.0
H. pylori Positive histology
FIG. 1. SBS ELISA
scores
H. pylori
Negative histology
of the positive and negative patient
sera.
[specifications not given]) was added to the wells in duplicate. After incubation for 1 h at room temperature, each well was washed three times with 3 ml of washing buffer (specifications not given). After the wells were dried, 100 ,ul of the anti-IgG-enzyme conjugate reagent was added (5 ml of antibody conjugate diluent in anti-IgG-enzyme conjugate concentrate). After incubation for 1 h at room temperature and repeated washings, 100 ,ul of freshly prepared substrate reagent (6 ml of substrate buffer and one substrate tablet) was added. After 30 min, 1 drop of stopping solution (specifications not given) was added to stop the color reaction. The A405 was read. The readings were expressed as P/N ratios by dividing the measured OD by the OD of the negative control. An IgG-positive control specific for H. pylori is provided by the manufacturer. It has two purposes. First, serial twofold dilutions are used as the standard cutoff values for the semiquantitative analysis; second, the dilutions indicate whether all reagents are working properly. Statistics. Student's t test and Pearson's coefficient of correlation were computed with the Crunch Interactive Statistical Package (Crunch Software Corp., San Francisco, Calif.).
RESULTS
SBS ELISA. Figure 1 shows the SBS ELISA scores of the positive and negative patient sera. The mean P/N ratio of the positives was 2.92 (standard deviation [SD] 1.01), and that of
the negatives was 0.83 (SD, 0.58). This difference was significant (P < 0.0001). With a receiver operating curve (ROC) (described below), an optimal cutoff value was determined at a P/N ratio of 1.40 (sensitivity, 91.8%; specificity, 91.3%). The positive predictive value (PPV) of the test at this cutoff value was 95.7%; the negative predictive value (NPV) was 84.0% (Table 1). ROC. An ROC is a graph of pairs of true-positive rates (sensitivity) and false-positive rates (100% minus specificity) that correspond to each possible cutoff value for the diagnostic test result. This strategy can be used to compare the usefulness of different cutoff values. The cutoff values are plotted in an ROC (sensitivity on the horizontal axis and 100% minus specificity on the vertical axis); the cutoff value that lies farthest to the "northwest" is the most accurate value or, more formally, the cutoff value whose ROC encloses the largest area is the most accurate value (8). Bio-Rad G.A.P. test. The scores of the patient sera and positive control dilutions in the G.A.P. test are shown in Fig. 2. The mean P/N ratios of the positive and negative patient sera were 5.19 (SD, 1.37) and 2.53 (SD, 1.12), respectively (P < 0.0001). For analysis of the data, three methods were used. First, interpretation of the data along the guidelines provided by Bio-Rad (method I) revealed a specificity of 100%, but two-thirds of the proven positive patient sera showed negative P/N ratios (sensitivity, 32.7%; PPV, 100%; NPV, 58.9%). This low sensitivity was due to the manufacturer's instruction to consider all results less than 6.0 as negative. Semiquantitative interpretation could not be performed, because the P/N ratios of all positive control dilutions were also less than 6.0 (Fig. 2). Therefore, another method of datum analysis was performed. For this method (method II), the cutoff value of 6.0 was ignored and only the positive controls provided by the manufacturer were used to interpret the data semiquantitatively. (In other words, the positive control was used to assess the cutoff value, whereas in method I this serum was used only as a positive control.) This was achieved by classifying the results for different control dilutions into one of five categories: -, +/-, 1+, 2+, and 3+. Results lower
NEW TEST FOR H. PYLORI-ASSOCIATED GASTRITIS DIAGNOSIS?
VOL. 29, 1991
than the P/N ratio of positive control dilution 1:16 were classified as negative, the other positive control dilutions (1:8, 1:4, and 1:2) establishing the cutoff points for the remaining categories. This classification resulted in a sensitivity of 91.8%, but now the specificity decreased to 43.5% (Fig. 2). The PPV and NPV were 80.4 and 83.3%, respectively, but no predictions could be made for the four patient sera classified in the midrange (two positives and two negatives, as determined by histological examination). Finally, all instructions for datum interpretation were disregarded, and an ROC determination was applied (method III). An optimal cutoff value was found at a P/N ratio of 3.9, resulting in a sensitivity of 87.8%, a specificity of 91.3%, a PPV of 95.6%, and an NPV of 77.8% (Table 1). Correlation of ELISA scores. Pearson's correlation coefficient for the SBS ELISA scores and the results of the Bio-Rad G.A.P. test with a cutoff value of 3.9 was 0.88 (P < 0.0001; Fig. 3); at a cutoff value of 6.0 no correlation was present.
P/N
8.0 -
i 6.00
6.0 -
I
(12)(1:4)-±
I
4.0
3.90
-
a
(l.8)t
DISCUSSION For a reliable determination of the cutoff point of an ELISA, sera from patients proven to be positive or negative for H. pylori are necessary. In the present study, culturing and histological examination of the biopsy specimens were used as the reference methods. With calculated cutoff points of 1.40 and 3.90 for the SBS and Bio-Rad ELISAs, respectively, both tests were comparable in terms of sensitivity and specificity, the NPV of the SBS ELISA being slightly higher than that of the G.A.P. test (84 versus 78%, respectively). However, it must be borne in mind that the cutoff value for the G.A.P. test was modified by us and cannot be used to increase sensitivity and specificity values per se. Interpretation according to instructions provided by the manufacturer yielded worse results because of the high OD of the negative control, as measured in several independent
(1:16) -
2.0 -
0.0 H. pylori H. pylori Positive control Positive Negative dilutions histology histology FIG. 2. Scores of the positive and negative patient sera and the positive control dilutions in the G.A.P. test.
5.0
SBS ELISA (P/N)
285
3.90
4.0
.-U
,
a a
*-
'
a"
a
V
a
3.0 a
a
.
o
1.40 1.0
3.0 4.0 5.0 6.0 7.0 8.0 Bio-Rad G.A.P. Test (P/N) FIG. 3. Correlation of the G.A.P. test and the SBS ELISA (Pearson's correlation coefficient, 0.88; P < 0.0001). Symbols: *, correct in both tests; O, false in both tests; 0, false only in the SBS ELISA; V, false only in the G.A.P. test.
0.0
1.0
2.0
286
J. CLIN. MICROBIOL.
VAN DEN OEVER ET AL.
experiments, compared with the OD provided by the manufacturer. In the Bio-Rad manual, a calculation example was given with a mean OD for the negative control serum of 0.029. This low value made the results hard to reproduce in another laboratory, because an increase of only a few decimals results in a dramatic decrease in the P/N ratio. Therefore, it might be better to use a reference serum that is very close to the cutoff point. Differences in laboratory circumstances would then influence the OD of the reference serum in proportion to the OD of the sera tested. When the cutoff value of 6.0 was disregarded for the reasons mentioned above, PPV and NPV of 80.4 and 83.3%, respectively, were achieved. Four patients were classified as +/-, and no prediction could be made for them at all. Since the P/N ratios of the positive control dilutions were measured and calculated in a manner similar to that used for the patient sera, different laboratory circumstances could not explain these low performance characteristics. Regarding these values, it may be questioned whether the Bio-Rad G.A.P. test is a useful diagnostic tool in its present form. The main difference between the SBS ELISA and the G.A.P. test was an antigen purification step done by BioRad. The technique used by Bio-Rad to refine the antigen was not documented, but it did not render their test superior to the SBS ELISA. Moreover, the results of the present study confirmed the observations of Hirschl et al. that the use of a mixture of antigens (e.g., sonicated whole cells) yields better discrimination than does the use of any protein fraction prepared by electrophoresis (2). In addition, multiple dominant antigenic cell fragments have been found throughout the H. pylori outer membrane and flagella (4). Despite differences in antigen preparation, a high correlation was found between the results of the two tests when the cutoff value for the Bio-Rad test was changed. For the patients with a false-positive reaction, no explanation was revealed by their clinical data. Four of 49 patients with an H. pylori-positive biopsy (by culturing and histological examination) had negative reactions in both tests, suggesting that for these patients, measuring the IgG titer was not a valid diagnostic approach. Bio-Rad does not provide detailed data on their reference population. In our view, this is an omission. The manufacturer states that 100 controls with no history of gastric ulcers and 100 patients with a history of gastric ulcers were used. This, of course, is not the correct way to establish a reference population. First, the correlation between gastric ulcers and complaints is rather weak, and endoscopy and a
biopsy should be done. Second, it is known from many studies that the prevalence of H. pylori infections in healthy individuals without any history of upper abdominal complaints is about 25% and increases with age (6). Hence, it is to be expected that a large number of negative controls actually have H. pylori infection. This can be the reason why the cutoff value has to be very high and why there were so many false-negatives in the Bio-Rad G.A.P. test. Further studies on the usefulness of the ELISA for the diagnosis and follow-up H. pylori-associated gastritis, especially in patients not referred to an endoscopy department (e.g., patients visiting their general practitioners), are necessary and will be performed in the near future. All commercial tests should be evaluated extensively with known reference sera before they are applied in daily practice. REFERENCES 1. Barthel, J. S., and E. D. Everett. 1990. Diagnosis of Campylo-
bacter pylori infections: the "gold standard" and the alternatives. Rev. Infect. Dis. 12:S107-S114. 2. Hirschl, A. M., M. Pletschette, M. H. Hirschl, J. Berger, G. Stanek, and M. L. Rotter. 1988. Comparison of different antigen preparations in an evaluation of the immune response to Campylobacterpylori. Eur. J. Clin. Microbiol. Infect. Dis. 7:570-575. 3. Kaldor, J., W. Tee, P. McCarthy, J. Watson, and B. Dwyer. 1985. Immune response to Campylobacter pyloridis in patients with peptic ulceration. Lancet i:921. 4. Loffeld, R. J. L. F., E. Stobberingh, and J. W. Arends. 1989. Detection of Campylobacter pylori infection. Lancet ii:569-570. 5. Loffeld, R. J. L. F., E. Stobberingh, J. A. Flendrig, J. P. van Spreeuwel, and J. W. Arends. 1989. Diagnostic value of an immunoassay to detect anti-Campylobacter pylori antibodies in non-ulcer dyspepsia. Lancet i:1182-1185. 6. Loffeld, R. J. L. F., E. Stobberingh, J. P. van Spreeuwel, J. A. Flendrig, and J. W. Arends. 1990. The prevalence of antiHelicobacter (Campylobacter) pylori antibodies in patients and healthy blood donors. J. Med. Microbiol. 32:105-109. 6a.Marshall, B. J., and J. R. Warren. 1984. Unidentified curved bacilli in the stomach of patients with gastritis and peptic ulcers. Lancet i:1311-1314. 7. Perez-Perez, G. I., B. M. Dworkin, J. E. Chodes, and M. J.
Blaser. 1988. Campylobacter antibodies in humans. Ann. Intern. Med. 109:11-17. 8. Sackett, D. L., R. B. Haynes, and P. Tugwell. 1985. Clinical epidemiology. A basic science for clinical medicine, p. 107. Little, Brown & Co., Boston. 9. Wyatt, J. I., and B. J. Rathbone. 1989. The role of serology in the
diagnosis of Campylobacter pylori infection. Scand. J. Gastroenterol. 24:S27-S34.
JOURNAL OF CLINICAL MICROBIOLOGY, Feb. 1991, p. 283-286
0095-1137/91/020283-04$02.00/0
Usefulness of a New Serological Test (Bio-Rad) To Diagnose Helicobacter pylori-Associated Gastritis H. L. A. VAN DEN GEVER,1 R. J. L. F. LOFFELD,2 AND E. E. STOBBERINGH1* Departments of Medical Microbiology' and Internal Medicine,2 University of Limburg, P.O. Box 616, 6200 MD Maastricht, The Netherlands Received 23 July 1990/Accepted 9 November 1990
A semiquantitative serological test (G.A.P. test; Bio-Rad) to diagnose Helicobacter (Campylobacter) pyloni-associated gastritis has recently become commercially available. This test was evaluated with sera from 72 dyspeptic patients with known H. pylori status. When the instructions provided by the manufacturer were applied strictly, the sensitivity of the test was only 32.7%. With a slight modification of the instructions, the performance characteristics were still unsatisfactory (specificity, 43.5%). When a new cutoff value was determined by a receiver operating curve, the quality of the test (positive predictive value, 95.6%; negative predictive value, 77.8%) was slightly lower than that of a noncommercial enzyme-linked immunosorbent assay (positive predictive value, 95.7%; negative predictive value, 84.0%). It is concluded that the Bio-Rad G.A.P. test has low diagnostic yield, and it is stressed that commercial tests should be evaluated extensively before being applied in daily practice.
GmbH, Munich, Federal Republic of Germany), and the suspension was stored at a final concentration of 1,500 mg/ml, preserved with 0.1% sodium azide, at -20°C until used. Optimal concentrations were determined by checkerboard titration. To coat the wells, we pipetted 100 ,ul of the antigen suspension, diluted 1:1,000 in 60 mM carbonate buffer (pH 9.6), or 100 p.l of buffer only as a control into the wells of flexible Titertek immunoassay plates (Flow Laboratories, Amsterdam, The Netherlands) and let the plates stand overnight at 4°C. After the plates were washed three times with PBS containing 0.5% Tween 20 (Merck-Schuchard, Hohenbrunn, Federal Republic of Germany), uncoated active sites were saturated by incubation with 100 p.1 of 1% bovine serum albumin (Merck, Darmstadt, Federal Republic of Germany) in PBS for 1 h at 37°C. The plates were washed, sealed with Parafilm, and stored at 4°C. For the test assay, 50 p.1 of serum, diluted 1:100 in PBS, was pipetted into both coated and control wells in triplicate and incubated at 37°C for 1 h. The washing step was repeated, and 50 p.1 of a 1:1,000 dilution of peroxidase-conjugated rabbit immunoglobulins to human immunoglobulin G (IgG) (DAKO Immunoglobulins, Copenhagen, Denmark) in PBS was added. The plates were incubated at 37°C for 1 h and washed. For the color reaction, we used 50 p.1 of a solution of 20 p.g of ortho-phenylenediamine in 50 ml of citratephosphate buffer (pH 5.0) to which 10 p.1 of 30% hydrogen peroxide was added. The reaction was stopped after 10 min with 50 p.1 of 4.0 M sulfuric acid and read at 492 nm. Sample scores were expressed as positive/negative (P/N) ratios by dividing the optical densities (OD) of the patient sera by the OD of a pooled negative serum included in every run. Bio-Rad G.A.P. test. The G.A.P. test, a semiquantitative test for the detection of H. pylori-specific IgG antibodies, was obtained from the Clinical Division, Bio-Rad Laboratories. H. pylori group-specific antigens were purified and immobilized on microwells. All procedures were carried out as described by the manufacturer. Positive control, negative control, and patient sera were added to the appropriate coated wells. One hundred microliters of diluted patient serum (25 p.1 of patient serum and 5 ml of diluent buffer
Since the first description by Marshall and Warren (6a) of a curved bacillus in the stomachs of patients with gastritis and peptic ulceration, Helicobacter pylori has been recognized as a common cause of type B (antral) gastritis and probably peptic ulcers. For the diagnosis of H. pyloriassociated gastritis, histological staining and/or culturing of biopsy specimens has been considered the "gold standard" (1). However, to facilitate the diagnosis, noninvasive techniques, such as serological detection of anti-H. pylori antibodies, would be preferable (7, 9). In a previous study, the diagnostic value of an enzyme-linked immunosorbent assay (ELISA) with a sonicated bacterial suspension as an antigen was described (5). In the present study, the usefulness of a recently available commercial ELISA (G.A.P. test; Bio-Rad Laboratories, Milan, Italy) was compared with that of the one previously described with sera from dyspeptic patients with histologically proven gastritis associated with H. pylori infections and from dyspeptic patients with normal antral mucosae. MATERIALS AND METHODS Serum samples. Serum samples from 72 patients with dyspepsia and described in detail by Loffeld et al. (5) were tested. By histological examination (hematoxylin, eosin, and modified Giemsa stains) and culturing of antral biopsies, 49 patients were found positive and 23 patients were found negative for H. pylori. The cross-reactivity of these sera with Campylobacterjejuni was excluded as described previously
(5).
ELISA of SBS. The ELISA for the sonicated bacterial suspension (SBS) was slightly modified from that in earlier reports (3, 5, 6). Five H. pylori strains were cultured at 37°C on blood agar under microaerobic conditions (10% CO2 and 5% 02). After 5 to 6 days of incubation, the cells were harvested, washed twice in phosphate-buffered saline (PBS), heated at 60°C for 30 min, and sonicated three times for 20 s each time. The protein concentration of the suspension was determined by standard methods (Bio-Rad Laboratories * Corresponding author. 283
284
J. CLIN. MICROBIOL.
VAN DEN OEVER ET AL.
P/N
5.0
TABLE 1. Diagnostic value of the two ELISA procedures
-
No. of patients with: Test and result
Antral gastritis and H. pylori
No antral gastritis or
45 4
2 21
16 33
0 23
Method II Positive Midrange (+/-) Negative
45 2 2
11 2 10
Method III Positive Negative
43 6
2 21
H.
pylori
SBS ELISA
3.0
Positive
I
4.0
Negative Bio-Rad G.A.P. test Method I Positive
-
Negative
2.0 -
I
1.40 I
1.0
I
0.0
H. pylori Positive histology
FIG. 1. SBS ELISA
scores
H. pylori
Negative histology
of the positive and negative patient
sera.
[specifications not given]) was added to the wells in duplicate. After incubation for 1 h at room temperature, each well was washed three times with 3 ml of washing buffer (specifications not given). After the wells were dried, 100 ,ul of the anti-IgG-enzyme conjugate reagent was added (5 ml of antibody conjugate diluent in anti-IgG-enzyme conjugate concentrate). After incubation for 1 h at room temperature and repeated washings, 100 ,ul of freshly prepared substrate reagent (6 ml of substrate buffer and one substrate tablet) was added. After 30 min, 1 drop of stopping solution (specifications not given) was added to stop the color reaction. The A405 was read. The readings were expressed as P/N ratios by dividing the measured OD by the OD of the negative control. An IgG-positive control specific for H. pylori is provided by the manufacturer. It has two purposes. First, serial twofold dilutions are used as the standard cutoff values for the semiquantitative analysis; second, the dilutions indicate whether all reagents are working properly. Statistics. Student's t test and Pearson's coefficient of correlation were computed with the Crunch Interactive Statistical Package (Crunch Software Corp., San Francisco, Calif.).
RESULTS
SBS ELISA. Figure 1 shows the SBS ELISA scores of the positive and negative patient sera. The mean P/N ratio of the positives was 2.92 (standard deviation [SD] 1.01), and that of
the negatives was 0.83 (SD, 0.58). This difference was significant (P < 0.0001). With a receiver operating curve (ROC) (described below), an optimal cutoff value was determined at a P/N ratio of 1.40 (sensitivity, 91.8%; specificity, 91.3%). The positive predictive value (PPV) of the test at this cutoff value was 95.7%; the negative predictive value (NPV) was 84.0% (Table 1). ROC. An ROC is a graph of pairs of true-positive rates (sensitivity) and false-positive rates (100% minus specificity) that correspond to each possible cutoff value for the diagnostic test result. This strategy can be used to compare the usefulness of different cutoff values. The cutoff values are plotted in an ROC (sensitivity on the horizontal axis and 100% minus specificity on the vertical axis); the cutoff value that lies farthest to the "northwest" is the most accurate value or, more formally, the cutoff value whose ROC encloses the largest area is the most accurate value (8). Bio-Rad G.A.P. test. The scores of the patient sera and positive control dilutions in the G.A.P. test are shown in Fig. 2. The mean P/N ratios of the positive and negative patient sera were 5.19 (SD, 1.37) and 2.53 (SD, 1.12), respectively (P < 0.0001). For analysis of the data, three methods were used. First, interpretation of the data along the guidelines provided by Bio-Rad (method I) revealed a specificity of 100%, but two-thirds of the proven positive patient sera showed negative P/N ratios (sensitivity, 32.7%; PPV, 100%; NPV, 58.9%). This low sensitivity was due to the manufacturer's instruction to consider all results less than 6.0 as negative. Semiquantitative interpretation could not be performed, because the P/N ratios of all positive control dilutions were also less than 6.0 (Fig. 2). Therefore, another method of datum analysis was performed. For this method (method II), the cutoff value of 6.0 was ignored and only the positive controls provided by the manufacturer were used to interpret the data semiquantitatively. (In other words, the positive control was used to assess the cutoff value, whereas in method I this serum was used only as a positive control.) This was achieved by classifying the results for different control dilutions into one of five categories: -, +/-, 1+, 2+, and 3+. Results lower
NEW TEST FOR H. PYLORI-ASSOCIATED GASTRITIS DIAGNOSIS?
VOL. 29, 1991
than the P/N ratio of positive control dilution 1:16 were classified as negative, the other positive control dilutions (1:8, 1:4, and 1:2) establishing the cutoff points for the remaining categories. This classification resulted in a sensitivity of 91.8%, but now the specificity decreased to 43.5% (Fig. 2). The PPV and NPV were 80.4 and 83.3%, respectively, but no predictions could be made for the four patient sera classified in the midrange (two positives and two negatives, as determined by histological examination). Finally, all instructions for datum interpretation were disregarded, and an ROC determination was applied (method III). An optimal cutoff value was found at a P/N ratio of 3.9, resulting in a sensitivity of 87.8%, a specificity of 91.3%, a PPV of 95.6%, and an NPV of 77.8% (Table 1). Correlation of ELISA scores. Pearson's correlation coefficient for the SBS ELISA scores and the results of the Bio-Rad G.A.P. test with a cutoff value of 3.9 was 0.88 (P < 0.0001; Fig. 3); at a cutoff value of 6.0 no correlation was present.
P/N
8.0 -
i 6.00
6.0 -
I
(12)(1:4)-±
I
4.0
3.90
-
a
(l.8)t
DISCUSSION For a reliable determination of the cutoff point of an ELISA, sera from patients proven to be positive or negative for H. pylori are necessary. In the present study, culturing and histological examination of the biopsy specimens were used as the reference methods. With calculated cutoff points of 1.40 and 3.90 for the SBS and Bio-Rad ELISAs, respectively, both tests were comparable in terms of sensitivity and specificity, the NPV of the SBS ELISA being slightly higher than that of the G.A.P. test (84 versus 78%, respectively). However, it must be borne in mind that the cutoff value for the G.A.P. test was modified by us and cannot be used to increase sensitivity and specificity values per se. Interpretation according to instructions provided by the manufacturer yielded worse results because of the high OD of the negative control, as measured in several independent
(1:16) -
2.0 -
0.0 H. pylori H. pylori Positive control Positive Negative dilutions histology histology FIG. 2. Scores of the positive and negative patient sera and the positive control dilutions in the G.A.P. test.
5.0
SBS ELISA (P/N)
285
3.90
4.0
.-U
,
a a
*-
'
a"
a
V
a
3.0 a
a
.
o
1.40 1.0
3.0 4.0 5.0 6.0 7.0 8.0 Bio-Rad G.A.P. Test (P/N) FIG. 3. Correlation of the G.A.P. test and the SBS ELISA (Pearson's correlation coefficient, 0.88; P < 0.0001). Symbols: *, correct in both tests; O, false in both tests; 0, false only in the SBS ELISA; V, false only in the G.A.P. test.
0.0
1.0
2.0
286
J. CLIN. MICROBIOL.
VAN DEN OEVER ET AL.
experiments, compared with the OD provided by the manufacturer. In the Bio-Rad manual, a calculation example was given with a mean OD for the negative control serum of 0.029. This low value made the results hard to reproduce in another laboratory, because an increase of only a few decimals results in a dramatic decrease in the P/N ratio. Therefore, it might be better to use a reference serum that is very close to the cutoff point. Differences in laboratory circumstances would then influence the OD of the reference serum in proportion to the OD of the sera tested. When the cutoff value of 6.0 was disregarded for the reasons mentioned above, PPV and NPV of 80.4 and 83.3%, respectively, were achieved. Four patients were classified as +/-, and no prediction could be made for them at all. Since the P/N ratios of the positive control dilutions were measured and calculated in a manner similar to that used for the patient sera, different laboratory circumstances could not explain these low performance characteristics. Regarding these values, it may be questioned whether the Bio-Rad G.A.P. test is a useful diagnostic tool in its present form. The main difference between the SBS ELISA and the G.A.P. test was an antigen purification step done by BioRad. The technique used by Bio-Rad to refine the antigen was not documented, but it did not render their test superior to the SBS ELISA. Moreover, the results of the present study confirmed the observations of Hirschl et al. that the use of a mixture of antigens (e.g., sonicated whole cells) yields better discrimination than does the use of any protein fraction prepared by electrophoresis (2). In addition, multiple dominant antigenic cell fragments have been found throughout the H. pylori outer membrane and flagella (4). Despite differences in antigen preparation, a high correlation was found between the results of the two tests when the cutoff value for the Bio-Rad test was changed. For the patients with a false-positive reaction, no explanation was revealed by their clinical data. Four of 49 patients with an H. pylori-positive biopsy (by culturing and histological examination) had negative reactions in both tests, suggesting that for these patients, measuring the IgG titer was not a valid diagnostic approach. Bio-Rad does not provide detailed data on their reference population. In our view, this is an omission. The manufacturer states that 100 controls with no history of gastric ulcers and 100 patients with a history of gastric ulcers were used. This, of course, is not the correct way to establish a reference population. First, the correlation between gastric ulcers and complaints is rather weak, and endoscopy and a
biopsy should be done. Second, it is known from many studies that the prevalence of H. pylori infections in healthy individuals without any history of upper abdominal complaints is about 25% and increases with age (6). Hence, it is to be expected that a large number of negative controls actually have H. pylori infection. This can be the reason why the cutoff value has to be very high and why there were so many false-negatives in the Bio-Rad G.A.P. test. Further studies on the usefulness of the ELISA for the diagnosis and follow-up H. pylori-associated gastritis, especially in patients not referred to an endoscopy department (e.g., patients visiting their general practitioners), are necessary and will be performed in the near future. All commercial tests should be evaluated extensively with known reference sera before they are applied in daily practice. REFERENCES 1. Barthel, J. S., and E. D. Everett. 1990. Diagnosis of Campylo-
bacter pylori infections: the "gold standard" and the alternatives. Rev. Infect. Dis. 12:S107-S114. 2. Hirschl, A. M., M. Pletschette, M. H. Hirschl, J. Berger, G. Stanek, and M. L. Rotter. 1988. Comparison of different antigen preparations in an evaluation of the immune response to Campylobacterpylori. Eur. J. Clin. Microbiol. Infect. Dis. 7:570-575. 3. Kaldor, J., W. Tee, P. McCarthy, J. Watson, and B. Dwyer. 1985. Immune response to Campylobacter pyloridis in patients with peptic ulceration. Lancet i:921. 4. Loffeld, R. J. L. F., E. Stobberingh, and J. W. Arends. 1989. Detection of Campylobacter pylori infection. Lancet ii:569-570. 5. Loffeld, R. J. L. F., E. Stobberingh, J. A. Flendrig, J. P. van Spreeuwel, and J. W. Arends. 1989. Diagnostic value of an immunoassay to detect anti-Campylobacter pylori antibodies in non-ulcer dyspepsia. Lancet i:1182-1185. 6. Loffeld, R. J. L. F., E. Stobberingh, J. P. van Spreeuwel, J. A. Flendrig, and J. W. Arends. 1990. The prevalence of antiHelicobacter (Campylobacter) pylori antibodies in patients and healthy blood donors. J. Med. Microbiol. 32:105-109. 6a.Marshall, B. J., and J. R. Warren. 1984. Unidentified curved bacilli in the stomach of patients with gastritis and peptic ulcers. Lancet i:1311-1314. 7. Perez-Perez, G. I., B. M. Dworkin, J. E. Chodes, and M. J.
Blaser. 1988. Campylobacter antibodies in humans. Ann. Intern. Med. 109:11-17. 8. Sackett, D. L., R. B. Haynes, and P. Tugwell. 1985. Clinical epidemiology. A basic science for clinical medicine, p. 107. Little, Brown & Co., Boston. 9. Wyatt, J. I., and B. J. Rathbone. 1989. The role of serology in the
diagnosis of Campylobacter pylori infection. Scand. J. Gastroenterol. 24:S27-S34.
