Journal of Applied Bacteriology 1992, 73, 210-216

Laboratory performance in a food microbiology proficiency testing scheme M. Peterz Biology Division, National Food Administration, Box 622, S-75126 Uppsala. Sweden

4010/10/91 : accepted 2 April

1992

Results from two shipments in a proficiency testing scheme in which almost 200 food microbiology laboratories participated are summarized. Freeze-dried mixtures of bacteria were used as simulated food samples. Four and six samples, respectively, were examined. T h e statistical procedures used to evaluate the performance of participating laboratories are described. It is shown that laboratories which had been in t h e scheme for a long time perform, on average, better than those that had been i n the scheme for a short time. T h e former laboratories produced fewer false and outlying results, and were more accurate M. PETERZ. 1992.

and precise in their determinations.

INTRODUCTION it is important that high standards are maintained in laboratories where food and water samples are examined for offcia1 purposes. Internal quality control and quality assurance systems, monitoring staff, equipment, methods and culture media are thcrefore essential. In the future, accreditation of food and water laboratorics will probably be required, and the success of internal quality assurance procedures and laboratory performance may have to be evaluated by an independent organization. Proficiency testing, in which samples are distributed by mail, has for many years served as a basis for evaluating the perfnrmance of clinical laboratories. Concepts, materials and statistical methods for evaluation of laboratory performance have been developcd (Gavan 1974; Tillett & Crone 1976; Snell et af. 1982; Whitby ef al. 1982; Griffin et ul. 1982, 1986a,b; De Mello & Snell 1985). Unlike tests in clinical microbiology, most of those used in food and water microbiology are quantitative. Samples used for performance evaluations in food and water microbiology must therefore be stable and homogeneous. In addition, the statistical procedures used to assess the performance of participating laboratories must themselves be suitable for quantitative tests. Since the beginning of the 1980s a proficiency testing scheme for microbiological food laboratories has been organizcd on a regular basis in the Nordic countries. Simulated food samples in the form of freeze-dried bacteria in small glass vials are distributed. These samples satisfy the necessary requirements of stability and homogeneity (Peterz & Norberg 1983, 1986). T h e types of microorganisms in the samples are those that may be found in

foods, and in particular are those for which tests are made routinely in food microbiology laboratories. During thc years 1981-1990, 20 shipments, each normally including four or six samples, were made. Eighty-five samples were distributed, on which a total of 275 different tests were made. New laboratories have joined the scheme from time to time, and the number of' participating laboratories has increased from ca 30 to ca 200. The aim of the present study was to evaluate the performance of food microbiology laboratories which participated in the examination of the samples in two shipments in this scheme and to see if there was any correlation between the performance of a laboratory and the number of sample shipments it received.

MATERIALS AND METHODS

Samples

T h e study includes the results from shipments number 19 and 20 in the Nordic food microbiology proficiency testing scheme, which were distributed in September and December 1990, respectively (Peterz et al. 1990, 1991). Six samples were distributed in shipment 19, in which there were two pairs of blind duplicates. Four samples were distributed in shipment 20. Most of the organisms in the samples had been isolated from foods (Table 1). Freeze-dried samples were prepared as described by Peterz 8i Norberg (1983, 1986) but with inositol serum broth as freeze-drying medium (De Mello & Snell 1985). T h e samples were prepared about two months before the trials. Each sample was given a unique code number and

PERFORMANCE OF FOOD MICROBIOLOGY LABORATORIES 211

Table 1 Micro-organisms in samples distributed in two

shipments of the Nordic food microbiology proficiency testing scheme

Sample Shipment 19

Shipment 20

A

Pseudomonas aeruginosa Escherichia coli Salmonella saint-paul

Klebsiella oxytoca Escherichia coli Candida albicans Aeromonas hydrophila

B

Morganella morganii Alcaligenes faecalis Klebsiella oxytoca Campylobacter coli

Morganella morganii Enterobacter cloacae Bacillus cereus Saccharomyces cerevisiae

C

Acinetobacter calcoaceticus Hafnia alvei Proteus mirabilis Salmonella dublin Campylobacter jejuni

Escherichia coli Bacillus cereus Proteus mirabilis

Enterococcus faecium Staphylococcus saprophyticus Citrobacter freundii Salmonella senftenberg

Bacillus cereus Aeromonas hydrophila Dekkera

D

24 tests in shipment 19 (four tests on each of six samples) and from 16 in shipment 20 (four tests, four samples). Not all laboratories, however, were able to perform all the tests requested. I n order to normalize the data before statistical calculations, all colony counts from the quantitative analyses were converted to log,, . Outlying results were identified by the methods described by Filliben (1975) and Ryan et al. (1981), and the number of outliers was recorded for each laboratory. After the elimination of outliers, the mean (m)and standard deviation (s) for each of the tests were calculated by standard statistical methods. In order to make results from different tests comparable, all data (including outliers) were transformed to standard-score form with the formula : z=-

I-m S

these were mailed two weeks before the trials to 199 laboratories (shipment no. 19) and 200 laboratories (shipment no. 20).

where I = log colony count of the individual laboratory, m = mean log colony count for all laboratories, and s = standard deviation for the log colony counts of all laboratories. After this transformation the standard-scores (z) from all tests had a mean of zero and a standard deviation of unity. Standard-scores higher than 4 or lower than - 4, respectively, were assigned these two values. Laboratory performance

Sample examlnation

In the trials all samples were examined during a specified two-week period. At the time of examination the freezedried micro-organisms in each glass vial were reconstituted in 54 ml of sterile peptone water. This liquid was treated as the sample for testing. T h e tests performed are shown in Table 2. The laboratories were asked to examine the samples by their own routine procedures. In most cases, media and methods corresponded to those approved by the Nordic Committee on Food Analyses (NMKL) (Anon. 1982, 1986, 1990a,b, 1991) Statistical analyses

The results obtained were reported to the organizing laboratory. Altogether each laboratory reported its results from Table 2 Tests performed on the samples

Shipment 19

Shipment 20

Aerobic plate count 30°C Coliform bacteria 37°C

Coliform bacteria 37°C Thermotolerant coliform bacteria

Salmonella Campylobacter

Bacillus cereus Yeasts

T h e average performance for each laboratory was obtained by calculating the mean and standard deviation of all its standard-scores. T h e mean describes the average deviation, in standard deviation units, from the means of all tests performed in each shipment and is a measure of the accuracy. A value close to zero indicates good accuracy whereas one distant from zero indicates bad accuracy. The direction of the deviation from zero, i.e. results consistently above or below the mean for all laboratories, is indicated by the sign of the mean standard-score. T h e standard deviation for all standard-scores for a laboratory is a measure of the precision and should be as small as possible. T h e number of false results, negative and positive, were recorded for each laboratory. In shipment 19, failures to isolate coliforms in samples A-D, or salmonellas in samples A, C and D, were regarded as false negative results. Reports on isolations of salmonellas in sample B were regarded as false positive results. In shipment 20, failures to isolate coliforms in samples A, B and C, thermotolerant coliforms in samples A and C, Bacillus cereus in samples B, C and D, and yeasts in samples A, B and D were regarded as false negative results. Reports on isolations of coliforms in sample D, thermotolerant coliforms in samples B and D, B . cereus in sample A, and yeasts in sample C were regarded as false positive results.

212 M . PETERZ

T h e correlation between laboratory performance and the number of sample shipments a laboratory has received in the scheme were studied by linear regression analyses. The absolute value of the mean and the standard deviation of the standard-scores, the numbers of outliers and false results reportcd for each laboratory were related to the number of trials in which the laboratory had participated.

Shipment

20

RESULTS

In shipment 19 and 20, 195 and 197 laboratories, respectively, reportcd their results to the organizing laboratory. Results from the campylobacter tests in shipment 19 are not included in this study as only about half of the laboratories performed these tests and many of them were not familiar with these organisms.

0

I

2

3

4

5

6

2

7

NO.of outlyinq results

Outlying results

There were 147 outlying results in shipment 19, which is equal to 6.5% out of the results reported from the quantitative tests. However, there were differences between tests. A maximum number of outliers were reported for coliforms in sample B (14%) and a minimum for the aerobic plate count in sample A (3.6%). The majority of laboratories (62%) did not report any outlying result, and 24% reported only one outlying result (Fig. 1). In shipment 20, 188 outliers were identified. This corresponds to 10% of the results (maximum for thermotolerant coliforms in sample C, 22% ; minimum for yeasts in sample

Shipment 19

Fig. 2 Number of outlying results reported by laboratories participating in shipment 20

D, 24%). In this shipment 55% of the laboratories did not report any outlying results, whereas 17% rcported one (Fig. 2). Laboratory accuracy

O n average, there was a good standard of accuracy in the results of the quantitative tests. In shipments 19 and 20, 85 and 82% of the laboratories, respectively, had a mean of their standard-score data within rf: 1 from the overall mean (Figs 3 and 4). Of the laboratories, 3.1 and 4.6%, respectively, had on average a negative displacement in their determinations of more than - 1.5 standard-score in shipments 19 and 20; 3.6 and 3.0% of the laboratories, respectively, had a positive displacement of more than 1.5. Laboratory preclslon

0

1

2

3 4 5 6 No of outlying results

2

7

Fig. 1 Number of outlying results reported by laboratories participating in shipment 19 of the Nordic food microbiology proficiency testing scheme

T h e standard deviations for the laboratories standardscores ranged from 0.27 to 2.83 in shipment 19, and from 0-21 to 3.35 in shipment 20 (Figs 5 and 6). T h e mean for all standard deviations in shipments 19 and 20 were 1.02 and 1.26, respectively. T h e distributions for all results are positively skewed, but most laboratories had standard deviations below two (Figs 5 and 6). In shipment 19, 4.1% of the participants exceeded a standard deviation of two, whereas in shipment 20, 15% of the participants exceeded this value. False results

More than half of the laboratories (52%) did not report any false results in shipment 19 (Fig. 7). A total of 203 false

PERFORMANCE OF FOOD MICROBIOLOGY L A B O R A T O R I E S 213

50

c 0 0

*

20

10

0 -4

-3

-2

-I

0

I

3

2

-4

4

-3

-2

-I

0

I

2

3

4

Mean standard score

Mean standard score

deviation = 0.83

Fig. 4 Mean standard-score calculated for laboratories participating in shipment 20. Mean = -0.10, standard deviation = 0.80

results were reported, equal to 6.7% of the tests performed. Most of the false results were from the salmonella tests. In particular, there were problems in isolating Salmonella dublin from sample C , especially in laboratories that used tetrathionate-type enrichment media or who enriched samples at a temperature above 42°C (cf. Peterz et al. 1989, 1990a). In shipment 20, only 19% of the laboratories did not

report any false results (Fig. 8). A total of 287 false results were reported, equal to 10% of the tests performed. Of the 117 false positive results, most were misidentifications of Aeromonas hydrophila as a coliform in sample D and of Enterobacter cloacae as a thermotolerant coliform in sample B. T h e number of false negative results was 170, mainly because of failures to isolate B. cereus in sample C as a result of the spread of Proteus mirabilis over the plates.

Fig. 3 Mean standard-score calculated for laboratories participating in shipment 19. Mean = 0.03, standard

Shipment 20

Shipment I 9

I

0

I

2

....I

I

3

I

I

I

I

4

Standard deviation o f standard score

Fig. 5 Standard deviation of standard-scores calculated for laboratories participating in shipment 19

0

I

2

3

--I

4

Standard deviation of stondard score

Fig. 6 Standard deviation of standard-scores calculated for laboratories participating in shipment 20

214 M . PETERZ

120

r Shipment 19

Table 3 Comparison of average performance (see text), as determined in shipment 19, with number of shipments received

Shipments

I00

received

z 15 7-13 3-6 1-2

No. of' laboratories

Absolute mean

Standard deviation

Outliers

47 26 106 16

0.50

0.69

0.77 1.10

u.21 0.92

0.56

1.07

0.82

0.98

1.19

1.63

False results

0.83 1.04 1.14 1.13

c

0 0

z

40

Table 4 Comparison of average performance (see text), as

determined in shipment 20, with number of shipments received

20

Shipments received

0 0

1

2

3

4

5

No. of laboratories

Absolute mean

Standard deviation

Outliers

False results

51

0.46 0.58 064 0.7 1

1.16

0.76 0.73 1.07 1.50

0.73 1.68 1.72 1.67

6

No. o f folse results

Fig. 7 Number of false results reported hy laboratories participating in shipment 19

Impact on performance The participating laboratories were placed into four subgroups on the basis of the number of shipments they had received. In shipment 19, 24% of the laboratories had participated 15 times or more in the scheme, in shipment 20, 26% of the laboratories had participated 15 times or more (Tables 3 and 4). Of the laboratories, 8 and 6%, respectively, participated for their first or second time in the scheme. The average performance for each subgroup is

2 15 7-13 3-6 1-2

37 97

12

1.10

1.32 1.70

shown in Tables 3 and 4. I n most cases the laboratories that had been in the scheme for a long time performed much better than the others, or the recent arrivals performed much worse than the others. There were negative correlations between all performance parameters and the number of shipments a laboratory received (Table 5). T h e coefficients for the regression lines were significantly different from zero in all cases except for the absolute mean value and the number of false results in shipment 19. DISCUSSION

Shipment 20

lo0l I

; 80 L

Laboratories that have been participating in the Nordic food microbiology proficiency testing scheme for a long time produced, on average, fewer false and outlying results and obtained more accurate and precise results than those that had recently joined the scheme. Similar findings are

Table 5 Pearson correlation coefficients for relations between

number of shipments received and different performance parameters (see text)

No of false r e s u l t s

Fig. 8 Number of false results reported by laboratories pnrticipating in shipment 20

Performance parameter

Shipment 19

Shipment 20

Absolute mean Standard deviation No. of outlying results No. of false results

-0.108 -0,2517 -0.1947 -0.103

-0*167*

* P < 0.05; t P < 0.01; $ P < 0.001.

0.179* ---0~162* - 0,3871 -

PERFORMANCE OF FOOD MICROBIOLOGY LABORATORIES 215

reported by Griffin et al. (1982, 1986a) and Whitby et a!. (1982), who found that clinical laboratories which participated in performance evaluation programmes in bacteriology improved in performance during the time period studied. Peddecord & Cada (1980) showed that clinical laboratories which had maintained accreditation for a longer time performed best in proficiency tests. Although there are statistically significant correlations between several of the performance parameters and the number of times a laboratory has participated in the scheme, this only partly explains the laboratory performance. There are some other factors that may influence the results, e.g. size of the laboratory, education, training and experience of the laboratory staff, equipment, and methods used (cf. Whitby et al. 1982). Griffin et al. (1986b) report that clinical laboratories that test large numbers of samples perform better than those that test smaller numbers. Another factor could be the time spent on the samples. The freeze-dried samples are easily identifiable and are probably often given special attention in many laboratories. In fact, proficiency tests in most cases reflect what laboratories can do at their best and may not represent performance in actual practice (LaMotte et al. 1977; Whitby e l al. 1982). Based on the results of the proficiency tests presented in this study and in earlier shipments in this scheme, limits for detection of poor performers can be set up. The accuracy of a laboratory measured as the mean of the standardscores based on at least five, preferably more than 10 results, should not exceed & 1.5, and the precision measured as the standard deviation of the standard-scores should be below 2-0. Both figures must be evaluated together, as there may be laboratories that show excellent accuracy but poor precision, or the reverse. These limits for detecting poor performers must be regarded as guidelines. They are influenced by the numbers and type of organisms examined in a particular proficiency test as well as the competing flora in the samples. Furthermore, as the limits are relative the demands will increase when the laboratory performance increases. The total number of outlying results in the proficiency tests generally ranged between 5 and lo%, differing with the type of organism examined. In collaborative study procedures used to validate the methods of examinations no more than 22% of outliers are permitted (Horwitz 1988). An outlying result may occur by pure chance; thus, a single outlier reported by a laboratory cannot be regarded as crucial. If a laboratory reports several outliers, it may be tested whether the laboratory is a ‘poor performer’ by methods described by Friedman et al. (1983) or Tillett & Lightfoot (1991). I t is not necessary, however, to have limits for a maximum number of outliers reported by a laboratory because in the calculation of standard-scores,

outlying results receive large values which will affect the accuracy and precision estimates for the laboratory (laboratory mean and laboratory standard deviation, respectively). I n qualitative tests in clinical microbiology an average score of correct results of at least 80% is required (August et al. 1990). This is approximately equal to 20% false negative results, but varies from 0 to 50% depending on the organism examined, whether it appeared in a pure or in a mixed culture type sample (Gavan 1974; Tillett & Crone 1976; Griffin et al. 1986a), and on the selective isolation medium used (e.g. Beckers et al. 1985; Tablan et al. 1987). Statistical analyses to compare success rates in isolating pathogenic organisms by different clinical laboratories have been suggested (Tillett & Crone 1976; Snell et al. 1982; Griffin et al. 1986b). In guidelines for collaborative studies of methods, Horwitz (1988) stated that false positives and/or false negatives should not exceed about lo%, otherwise the analyses become uninterpretable. T h e need for quality assurance and quality control in food microbiology laboratories has been adequately justified by the consistent findings of significant differences in performance. Thus, it is necessary to have a system that will detect day-to-day errors and recognize potential sources of error. Proficiency testing schemes can hardly be organized with intervals close enough to replace an internal quality assurance system. This is not the aim of a proficiency testing scheme. It should primarily test the proficiency of the laboratory and check the adequacy of the internal quality control programme. ACKNOWLEDGEMENTS

T h e author wishes to thank Torbjorn Lindberg and MarieLouise Danielsson-Tham for constructive comments on earlier versions of the manuscript and Ann-Charlotte Nilsson for technical assistance. REFERENCES A N O N . (1982) Bacillus cereus. Determination in foods. Method No. 67, 2nd edn. Nordic Committee on Food Analyses (NMKL), c/o Technical Research Centre of Finland, Food Research Laboratory, PO Box 203, SF-02150 Espoo, Finland. A N O N . (1986) Aerobic microorganisms. Enumeration at 30°C in meat and meat products. Method No. 86, 2nd edn. Nordic Committee on Food Analyses (NMKL), C/O Technical Research Centre of Finland, Food Research Laboratory, PO Box 203, SF-02150 Espoo, Finland. A N o N . (1990a) Campylobacter jejuni/coli. Detection in foods. Method No. 119, 2nd edn. Nordic Committee on Food Analyses (NMKL), c/o Technical Research Centre of Finland, Food Research Laboratory, PO Box 203, SF-021 50 Espoo, Finland. A N O N . (1990b) Colifrm bacteria. Detection in foods. Method No. 44, 3rd edn. Nordic Committee on Food Analyses (NMKL),

216 M . PETERZ

c/o Technical Research Centre of Finland, Food Research Laboratory, PO Box 203, SF-02150 Espoo, Finland. A N O N . (1991) Salmoneila bacteria. Detection in foods. Method No. 71, 4th edn. Nordic Committee on Food Analyses (NMKL), c/o Technical Research Centre of Finland, Food Research Laboratory, PO Box 203, SF-02150 Espoo, Finland. A U G U S T , M.J., IIINDLER, J.A., H U B E R , T . W . & S E W E L LD , . L . (1990) Cumitech 3A. Quality Control and Quality Assurance Practices in Clinical Microbiology. Coordinating ed. Weissfeld, AS. Washington, DC: American Society for Microbiology. BECKERS,H . J . , V A N LEUSDEN,F . M . , M E I J S S E N , R, (1985) Reference M . J . M . & K A M P E L M A C H E E.H. material for evaluation of a standard method for the detection of Salmonella in foods and feeding stuffs. Journal of Applied Bacteriology 59, 507-5 12. D E MEI.I.O, J . V . & SNEI.L,J . J . S . (1985) Preparation of simulated clinical material for bacteriological examination. Journal of Applied Bacteriology 59,421-436. F I LI. I B E N , J . J . (1975) The probability plot correlation coefficient test for normality. Technometrics 17, 111-1 17. F R I E D M A NL.C., , BRADFORD,W . L . & P E A R T , D.B. (1988) Application of binomial distributions to quality assurance of quantitative chemical analyses. Journal of Environmental Scrence and Health A18, 561-570. Ci A V A N T , .L . (1974) A summary of the bacteriology portion of the 1972 basic, comprehensive, and special, College of American Pathologists (CAP) quality evaluation program. American Journul of Clinical Pathology 61, 971-979. G R I F F I N111, C . W . , M E H A F F E YM, . A . & C O O K ,E.C. (1982) Review of the Centres for Disease Control proficiency testing programme in bacteriology. Archiv fu’v Lebensrnittelhygiene 33, 166-173. G R I F F I N111, C.W., COOK, E.C. & M E H A F F E YM.A. , (1 986a) Centers for Disease Control performance evaluation program in bacteriology: 1980 to 1985. Journal of Clinical Microbiology 24, 1004-1012. G R I F F I N111, C . W . , M E H A F F E YM, . A . , C O O K , E . C . , B L U M E RS.O. , & PODESZWIK P .,A . (1986b) Relationship between performance in three of the Centers for Disease Control microbiology proficiency testing programs and the number of actual patient specimens tested by participating laboratories. Journal of Clinical Microbiology 23, 24G250. H O R W I T ZW. , (1988) Guidelines for collaborative study procedure to validate characteristics of a method of analysis. Journai of the Association of Oficial Analytical Chemists 71, 160-1 72. L A M O T T E ,L . C . , G U E R R A N TG, . O . , L E W I S , D . S . & H AI,L , C .T . ( 1977) Comparison of laboratory performance with blind and mail-distributed proficiency testing samples. Public Health Reports 92, 554.~560.

PEDDECORD, K . M . & C A D A ,R.1,. (1980) Clinical laboratory proficiency test performance. Its relationship to structural, process and environmental variables. American Journal of Clinical Pathology 73, 380-385. P E T E R ZM , . & N O R B E R GP,. (1983) Freeze-dried mixed cultures as samples for proficiency tests and collaborative studies in food microbiology. Journal of the Association of Ojjicial Analytical Chemists 66, 1510-1513. PETERZM , . & N O R B E R GP, . (1986) A split sample study of microbiological food laboratories using freeze-dried mixed cultures. International Journal of Food Microbiology 3, 161-166. P E T E R ZM , . , W I B E R GC , . & N O R B E R GP,. (1989) The effect of incubation temperature and magnesium chloride concentration on growth of salmonella in home-made and in commercially available dehydrated Rappaport-Vassiliadis broths. Journal of Applied Bacteriology 66, 523-528. , (1990) P E T E R Z ,M . , N I L S S O N ,A.-C. & N O R B E R GP. Resultat frdn interkalibrering av mikrobiologiska livsmedelsanalyser, September 1990. (Proficiency tests of microbiological food laboratories, September 1990. In Swedish.) S L Y rapport 1990 : 11, 28 pp. National Food Administration, Box 622, S-75 1 26 Uppsala, Sweden. P E T E R Z ,M . , N I L S S O N ,A.-C. & N O R B E R GP, . (1991) Resultat frdn interkalibrering av mikrobiologiska livsmedelsanalyser, December 1990. (Proficiency tests of microbiological food laboratories, December 1990. In Swedish.) S L Y rappor! 1991: 2, 24 pp. National Food Administration, Box 622, S-751 26 Uppsala, Sweden. R Y A N ,T . A . , J O I N E R , B . L . & R Y A N ,B . F . (1981) Minitub Reference Manual. Pennsylvania State University. SNELL, J.J.S., DE MELI.O,J . V . & G A R D N E RP.S. , (1982) The United Kingdom national microbiological quality assessment scheme. Journal of Clinical Parthology 35, 8P-92. TAFILAN, O.C., CARSON,L . A . , CUSICK,L . B . , B L A N D , & J A R V I S , W . R . (1987) LaborL . A . , M A R T O N EW.J. , atory proficiency test results on use of selective media for isolating Pseudomonas cepacia from simulated sputum specimens of patients with cystic fibrosis. Journal of Clinical Microbiology 25, 485-487. T I L L E T T , H . E . & CRONE, P . B . (1976) Quality control of the isolation rate of pathogens in medical microbiology laboratories. Journal of Hygiene, Cambridge 77, 359-367. TILLETT, H.E. & LIGHTFOOT, N . F . (1991) Preliminary statistical assessment of UK water quality control trials. Water Science and Technology 24, 57-60. W H I T B Y , J. L . , B L A C K , W . A . , RICHARDSON,H . 8i WOOD,D . (1982) System for laboratory proficiency testing in bacteriology : organisation and impact on microbiology laboratories in health care facilities funded by the Ontario Government. Journal of Clinical Pathology 35,96100.