Mutation Research, 266 (1992) 27-42

27

© 1992 Elsevier Science Publishers B.V. All fights reserved 0027-5107/92/$05.00

MUTREV 07312

INTERNATIONAL COHHISSION FOR PROTECTION AGAINST ENVIRONHENTAL MUTAGENS AND CARCINOGENS

A method for comparing and combining short-term genotoxicity test data: The optimal use of dose information D.H. Moore II a, M.L. Mendelsohn a and P.H.M. Lohman b a Biomedical Sciences, Lawrence Lit,ermore National Laboratory, P.O. Box 5507, Licennore, CA 94550 (U.S.A.) and b MGC - - Laboratory of Radiation Genetics and Chemical Mutagenesis, Unicersity of Leiden (The Netherlands)

(Received 9 September 1991) (Accepted 7 October 1991)

Keywords: Short-term genotoxicity test data, comparing and combining; Dose information, optimal use of

In the previous paper, Lohman et al. (1992), we describe a general approach toward creating a composite scoring system for genotoxicity test data. The method combines dose, metabolic activation and sign of outcome information in a way that accommodates negative and positive outcomes, and hierarchically combines replicates within a given test, tests within a class, classes within a family, and finally families within a single agent score for each chemical. A crucial aspect of the method is the development of a sound framework for entering dose information. We use the concept of log-defining doses, as initially developed by Waters et al. (1988) for their Genetic Activity Profiles. Thus for a given literature entry, test, chemical, and dose series, if the result has been scored as a positive then the defining dose is taken to be the logarithm of the lowest dose with a positive outcome. If the result has been scored as a negative

Correspondence: Dr. D.H. Moore !1, Biomedical Sciences, Lawrence Livermore National Laboratory, P.O. Box 5507, Livermore, CA 94550 (U.S.A.).

then the defining dose is the logarithm of the highest dose with a negative outcome. The log-defining dose in genotoxicity testing carries many different connotations. Within a given test, the magnitude of a positive response inversely reflects both the chemical property of mutagenic potency and the rigor at which the testing was done. Thus a lower-dose positive result suggests a more potent chemical and a more convincingly tested outcome, while a higher-dose positive suggests less chemical potency or less rigorous testing. For negative responses, the logdefining dose directly reflects nontoxicity and nonmutagenicity of the chemical, as well as rigor of testing. Thus a higher-dose negative result implies less chance of mutagenicity, less toxicity and more rigor, while a lower-dose negative suggests the chemical is less convincingly nonmutagenie, more likely toxic or less rigorously tested. When comparing between different types of tests, the log-defining dose also connotes general differences in the sensitivities to chemical of the involved tests. While genetic toxicologists have some interest in such sensitivities for designing and selecting tests, the efficiency with which a

28

|

J Fig. 1. Distribution of log-defining doses for all positive and negative outcomes in the database. Frequencies for positive outcomes are shown ahove the horizontal line. Frequencies for negative outcomes are show below the line. Each is roughly gaussian, with a skewing to the left. The dotted vertical line is at log dose zero. Numbers of entries and means (solid lines) are shown for each distribution. Note the displacement of the negative distribution one log to the right of the positive distribution. The ripples of local maxima reflect the non-random,clumped selection of doses. test uses chemicals is quite secondary to the interpretation of that chemical's genotoxicity. For this reason, we set out to find some way to

compensate out the general efficiency or sensitivity of a test, while keeping the ability to use dose within test for comparative purposes. The second goal of this dose-oriented analysis is to find an appropriate way to combine positive and negative results. Outcome in genotoxicity testing is usually a dichotomous response associated with some quantitation from either dose or magnitude of response. Typical genotoxicity data have mixed positive and negative results, and the agglomeration of results requires a good method to combine the dichotomous and quantitative scales into a composite figure of merit.

Comparison of log-defining doses by class The means and distributions of log-defining doses were collected and analyzed for the 2871 positive and 1581 negative entries in our database of 113 chemicals and 85 tests. Fig. 1 displays the distribution of results pooled across all classes and separated by sign of outcome. It shows two gaussian distributions, each with a skew towards lower doses. Negative outcomes lie, on average, 1 log higher than positive outcomes (negative mean, A6 Gone mutatinn- mammalian celia

20

B4 SCE, somatic- mammal 20,

loi ~5 0

0',

E

210

10" 20.

20 Log definingdose

Log definingdose

Fig, 2. ClaTs distributions of log-defining doses, A sample of classes is displayed. The paired distributions are similar to Fig. 1, but with differing displacement of positive and negative means. Class B4 (SCE. somatic - mammal) shows an almost 2 log difference in positive and negative means, while class A6 (gene mutation - mammalian cells) shows less than one log difference. Class B5 (micronuclei, somatic - mammal) has the largest displacement of the positive and negative means from log dose 0. Class A4 (gene mutation - prokaryoles) differs little from the average behavior shown in Fig. !.

29 TABLE 1 TEST REPLICATE SUMMARY Test

Class

Number of chemicals tested

Number of chem/ outcms with reps.

BRD BSD ECD ECL ERD

AI A1 AI AI AI

13 25 20 18 16

3 2 3 3 8

SCG SCH BSM UHF UHL

A2 A2 A3 A3 A3

56 52 9 24 17

UHT UIA UIH URP EC2

A3 A3 A3 A3 A4

ECF ECK ECR ECW SAL

Total number of reps.

Pooled s.d. of reps.

Test

Class

Number of chemicals tested

Number of chem/ outcms with reps.

Total number of reps.

Pooled s.d. of reps.

7 5 9 6 18

0.76 0.80 0.29 0.60 0.65

CIA CIC CIH CIR CIS

A9 A9 A9 A9 A9

6 65 19 22 10

1 40 5 1 2

2 148 10 2 4

0.06 0.80 1.49 0.21 0.99

30 17

85 42

0.84 0.86

0.57 0.43 0.'t4

0.82 0.62

19 25 27 6 28

2 12 6

26 5

A9 AI0 AI0 AI0 A 10

! 6 3

11 2

CIT T7S TBM TCL TCM

7

16

0.48

24 23 12 48 30

6 7 4 20 5

12 18 8 62 16

0.51 1.39 0.75 1.01 0.98

TCS TRR UPR UVA UVM

A10 AI0 BI BI BI

40 21 12 6 6

16 3 3

53 6 6

0.72 1.35 0.61

!

2

0.64

A4 A4 A4 A4 A4

10 19 26 44 i I1

4 7 10 115

8 19 25 750

0.67 0.66 0.74 0.83

UVR DMM MST SLH SVA

BI B2 B3 B4 B4

6 23 13 28 47

13 I 14 20

32 3 37 80

1.04 0.30 0.77 0.63

NCF NCR SCF SCR SZF

A5 A5 A5 A5 A5

8 9 18 46 25

2 3 6 13 3

6 8 20 35 9

0.40 0.69 0,42 0.66 0,75.

SVH MVC MVM MVR CBA

B4 B5 B5 B5 B6

5 11 75 13 65

I 2 34 2 23

3 5 133 7 94

0.06 0,37 0.56 0,46 0.69

GSI GST GgH Ggo GCO

A0 A6 A6 A5 A6

16 49 49 17 33

3 19 21 1 16

6 5! 60 2 35

0.69 0.69 0.63 0.48 0.57

CBH CLA CLH CVA DMH

B6 B6 B6 B6 B7

6 9 37 12 16

I 4 17 2 2

2 !2 58 4 4

0.07 !.01 0.99 0.26 0,34

GIA SCN SHF SHL SIA

A6 A7 A8 A8 A8

26 21 9 50 6

7 5 2 26

19 10 5 105

0.87 0.45 0,97 0.97

DML DMX SLO DLM DLR

B7 137 138 B9 B9

9 80 6 58 22

2 34 1 27 6

5 97 3 95 13

0,06 0.88 0,28 0,45 0.78

SIC SIH SIM SIR

A8 A8 A8 A8

71 20 6 14

54 3 2 2

202 7 4 4

0,83 i.42 0,87 0.49

MHT CCC CGC CGG

BI0 B11 B! 1 BI1

6 6 i4 16

2

7

0.31

2 3

6 11

0.23 0.38

SIS SIT CHF CHL

A8 A8 A9 A9

6 11 11 56

4 2 31

11 4 134

0.64 0.84 0,90

COE SPM SPR Totals

BI ! BI2 BI2

11 30 5 2115

4 6

13 16

0.37 0.46

794

2867

0.796

30

2.3(I; positive mean, 1.12). Presumably, this 10-fold difference stems directly from the primary strategy of picking highest dose in the case of negatives and lowest dose in the case of positivcs in the presence of about l log of uncertainty. The distributions of log-defining dose were also looked at by class. These display three phenomena: (1) the same shapc and relative relationship of positive and negative values already described for the pooled data, (2) wide variations in amount of data per class, with dcgcneratc distributions when class size becomes too small; and (3) striking differences among classes for the mean log-defining doses. The variation of means with class has a range of 2-3 logs, and the positive and ncgative means by class arc positively correlated. The variance across classes is relatively stable and similar to that of the composite distributions. See Fig. 2 for a sampling of class distributions. Based on the clear differences in log-defining doses among classes, it was decided to analyze thc full database by test, using variation of replicate data within outcome as a yardstick of resolution.

Replication variance Table 1 summarizes replicate information from the database arranged by test. The test Salmonella rcversc mutation (SAL) was by far the most replicated in the database with a total of 750 replicates spread over 111 of the 113 chemicals. The second most popular test was sister chromatid exchange in Chinese hamster cells in vitro (SIC) with 202 replicates from 71 chemicals tested. Table 1 also shows the standard deviation of replication of log-defining dose pooled over chemicals and outcomes for each test. The standard deviation of replication for the total database is 0.80 (pooled over all tests, chemicals and outcomes, and weighted by the number of degrees of frecdum for each test). To give an overall picture of variability among tests, Fig. 3 plots each test's replicate variance against its degrees of freedom (equal to the total number of replicates minus the number of chemicals tested). Upper and lower bounds, based on the chi-square distribution, were drawn to show the 95% confidence limits expected for each test

10

['. :

c

.g ,u

0.1

-o

0.0!

Q 0

•

~."Lr..

: i : . - : .:: :'.. • " :~L : • b~_~cP Dch ....... ~ G C ECD ,

-

0.001

1

10

100

1000

Degrees of freedom Fig. 3, Replication variance by test as a function of degrees of freedom. The degrees of freedom for each test are equal to the total number of replicates minus the number of chemicals tested. The chi-square distribution was used to construct upper and lower 95% confidence limits (curved lines). 7 tests lie above the upper 95% confidence limit, and 11 tests lie below the lower 95% confidence limit. The most frequently replicated test is the Salmonella mutation assay with 750 entries in the database.

variance based on the overall pooled value and the test's degrees of freedom. 7 tests are above the upper bound and I1 fall below the lower bound, compared to the two outliers in each direction that wore expected (2.5% of 78, the number of replicated tests). UIA, unscheduled DNA synthesis in other animal cells, falls quite a bit above the upper bound, and should probably be considered significant. The number below the lower bound is much larger than expected, suggesting that some tests - - particularly MVM, micronucleus test in mice, which was tested on 34 different chemicals, and DLM, dominant lethal in mice, which was tested on 27 different chemicals have significantly smaller replication variance than the bulk of tests. Note that many falling below the lower bound are based on very few replications and represent tests which were not used much in the database. Perhaps it would be worth evaluating a few of these tests, such as DNA damage in E. coil (ECD), chromosome aberration in ooeytes or embryos (COE) and forward gene mutation in Saccharomyces cerevisiae (SCF), on a larger database to determine whether their apparently good replication reliability will hold up for more chemicals.

31

However, for present purposes it suffices to know that the overall replication variance of logdefining dose is smaU in comparison to the total variance, and that apart from a few outliers the distribution of replication errors is reasonably well behaved. We are encouraged, therefore, to continue toward a full analysis of dose by test.

Comparison of log-defining dose by test within and among chemicals Now we turn to the problem of comparing log-defining doses across tests. An examination of the entire database reveals that some tests are more sensitive (i.e. have positive outcomes at lower doses) than others. For example, Fig. 4 shows the log-defining doses in the database by test for the chemical aflatoxin BI. Each data point is the log-defining dose for a single observation on a particular test (within a class). Widely varying clusters of points are evident in the figure. This variation can be subdivided into three parts: that due to outcome (positive vs. negative result); that due to class; and that due to test within class. The mean log-defining dose for all the positive results is -0.39 while that for the negatives is 1.04, reflecting again what was seen in Figs. I and 2. As an example of class variation, it is apparent that class A4 tests (prokaryote

1, ~,~',~

l.I¸

!,

o1'~1 ~!i

•

?}

i°

i

i

A3 A4 AS A6 ,4? J48 A9 Aq0 B1 82 B3 B4 8S B6 B? B8 B9 B10811B12 Class

Fig. 4. Individual log-defining doses for aflatoxin BI by test within class. The classe~ are demarcated by the vertical lines, and tests within class are arranged in alphabetical order from left to right between the lines. Positive results (closed circles) and negative results (open circles) are shown separately. Note the large differences for tests and classes, and that the few negative outcomes lie toward the top.

,,

,

;0

,

o

I )

•

•

)

J"

~ . *

•

io -

~

lel

j ,,

1°

$

•

; !

|

,

, "1'1 J'=.2,

A1 ~ , / ~ ~1 AS A6 A7 , ~ A9 A10 B'I B2 ~3 B4 B5 B6 B? B8 R9 810B11B12

Class

Fig. 5. M e a n

log-defining

doses for all chemicals

in the

database by test within class, plotted as in Fig. 4. Note again the patterning of results by tes~ and by class, and the distinct layering of negative outcomes above positive outcomes.

mutation) give positive res~lt,s at much lower doses than class A2 tests (lower eukaryote DNA damage). The mean for the class A4 tests, which for this chemical consists entirely of Salmonella reverse mutation, is - 1.48 while that for class A2 is 1,95. The largest within class variation appears to be in class A3 (mammalian cell primary DNA damage) where the 6 tests have means ranging from - 0 . 5 2 for UHT and U R P to 1.50 for UHL (although individual values range from -2.52 for a single U R P entry to 1.50 for UHL). Fig. 5 shows the same relationships for all chemicals combined, each point now being the test mean for either positive or negative logdefining doses. The mean for all positive results is 1.26 while that for all negatives is 2.16, again consistent with the results obtained from Fig. 1. The test means vary widely across the panel of classes, and in some cases even within classes. The overall variability is many times larger than the replication error, and would make it almost useless to pool results across tests and classes. There are three attributable sources of the variability shown in Fig. 5. The first is the replication error which is largely random and operates almost uniformly across the database. The second is structural patterns of test efficiency reflecting in the mean outcomes by sign of each test. And

32 o f n u m e r i c a l a n a l y s i s for a n y g i v e n - s i z e d s a m p l e , test e f f i c i e n c y is largely i r r e l e v a n t to e i t h e r test comparisons or chemical evaluation, and the

t h e t h i r d is c h e m i c a l d i f f e r e n c e s i n the sign a n d s t r e n g t h o f g e n o t o x i c i t y a n d toxicity signals. O f these, replication defines the limiting resolution

TABLE 2 EXAMPLE OF EM ALGORITHM Initial estimates Chemical

Test CBA

CHL

CIC

Acryloniteile Aflatoxin BI Benzene Benzo[a]pyrene

0.90 2.12

- 1.61 0.95

1.27 0.13 2.76 1.82

2.88

Column average Column effect

1.51 0.20

1.49 0.18

2.38 1.07

- 0.33 - 1.64

DLM 1.88

Avg.

Row effect

!.27 0,32 !.95 2.35

-0.04 -0.99 0.64 1.04

1.31

Grand mean

First iteration Chemical

Test CBA

CHL

CIC

DLM

Avg.

Acrylonitrile Aflatoxin BI Benzene Benzo[a]pyrene

1.47 0.90 2.12 2.$$

- 0.37 - 1.61 0.95 0.71

!.27 0.13 2.76 1.82

2.33 1.88 3.01 2.88

1.17 0.32 2.21 1.99

Column average Column effect

1.76 0.34

- 0.08 - 1.50

1.49 0.07

2.52 1.10

1.42

CBA

CHL

CIC

DLM

Avg.

Benzene

I.$1 0.90 2.12

Benzo[a]pyrene

2.32

-0.33 - 1.61 0.95 0.48

1.27 0,13 2.76 1,82

2.27 1,88 3.31 2.88

1.18 0.32 2.29 1.88

Column average Column effect

1.71 0.30

- 0.13 1.54

1.49 0,08

2.58 1.17

1,42

-

Row effect -

0.25

i.10 0.79 0.56

Second iteration

~hemical Acwlonitrile Aflatoxin BI

Test

Row

effect -0.24 1.09 0.87 0.46

-

Third iteration ,,,,,,,,,,

Chemical

Test

Row

CBA

CHL

CIC

DLM

Avg.

effect

Acrylonitrile Aflatoxin B! Benzene Benzo[a]pyrene

1.48 0.90 2.12 2.17

-0.36 - 1.61 0.95 0.33

1.27 0.13 2.76 1,82

2.35 1.88 3.45 2.88

1.18 0.32 2.32 1.80

-0.23

Column average Column effect

1,67 0.26

- 0.17 - 1.58

1.49 0.09

2.64 1,23

1.41

Bold entries are estimated from the Model: X~j = M + R~ + Cr M, Grand mean. R,, Row effect estimated from previous iteration, Cj, Column effect estimated from previous iteration.

-

1,08

0.92 0.39

33

chemical effects are often the definitive information desired from testing but may be the cause of considerable confounding when trying to evaluate the tests in incompletely filled database matrices. The problem, then, is to normalize out the test sensitivities in the presence of incompletely filled data containing large chemical effects. The following statistical model restates the problem and provides the structure for finding the solution: X i i k = M + C i + Tj + S k + e i j k ,

where X0.k represents the log-defining dose for the jth test (j = 1. . . . . 85) on the ith chemical (i = 1,..., 113) with outcome k (k = 1 for a positive result and 2 for a negative result). M is the grand mean (over all chemicals, all tests and both signs), C i is the deviance from this mean due to the ith chemical, Tj is the deviance due to the jth test, Sk is the deviance due to the sign of the outcome (positive or negative) and e~ik is a random error term. This definition of Sk is based on the observation that the positive and negative mean log-defining doses are closely coupled across most tests. S k effectively estimates the average distance between positive and negative outcomes for the entire database. When there are no missing data, an estimate for the grand mean M is obtained by summing all entries and dividing by the number of entries. An estimate for the C,. would be obtained by summing, for each chemical, over all tests and both outcomes pertaining to that chemical and then subtracting M. Similarly estimates for T~ are obtained by summing, for each test, over all chemicals and outcomes and subtracting M. The estimate for Sk is formed by summing, for each outcome, over all chemicals and all tests and then subtracting M. Unfortunately the usual procedure does not work here since all tests have not been applied to each chemical nor do we have all chemicals for each test. (The Analysis of Variance procedure depends on the assumption that the sums X;Tj, X~Ci, and X;Sk are equal to zero when taken over all tests, chemicals and outcomes respectively. This is because T~, Ci and $1, measure deviances from the grand mean.) In our

case, because of the numerous missing entries (in fact 87.6% of the entries are missing), there is no guarantee that the sums will be zero. Fortunately there is a simple procedure for estimating the main effects (i.e. the Tj, C i and S k) when there are missing entries. The procedure is known as the EM algorithm, for Expectation of Means and Maximization of Likelihood (Dempster et al., 1977). With the algorithm, the values for t h e main effects and for the missing data are estimated alternatively by successive iteration. Each round of the iteration begins with current estimates for M, Tj, Ci and S~ to provide best guesses for the missing values, and concludes with new estimates for the main effects using the best guesses. The procedure continues until changes in the estimates for the main effects become sufficiently small. Usually this procedure is applied to matrices with sparse missing data; in fact, we have been unable to find an application where the volume of missing data is as large as ours. However, the theory behind the method predicts monotonic convergence toward maximum likelihood answers as long as there are enough elements in every row and column of the matrix to estimate the row and column means. Table 2 demonstrates a simple example of the EM algorithm involving 4 tests on 4 chemicals with missing data. In the first section, the initial estimates, based on row, column and overall (grand) means, are calculated from the available data. In subsequent iterations, the effects (row or column means minus the grand mean) are used to estimate values for missing data (shown in bold type). Results of the first three iterations are shown and it is clear that convergence is beginning since the changes in estimates for the effects are decreasing in absolute magnitude. When we applied the EM algorithm to our database, convergence was reasonably fast, and by the end of 20 iterations the estimates of main effects were fluctuating only in the third significant figure. As a further check of the method, 10 random subsamples, each containing 99% of our data, were selected and the EM algorithm was applied to each for 20 iterations. Variation introduced by the EM algorithm amounted to less than 2% of the replication variance.

34

The main effects, according to our current best estimate, are: M, overall mean of log-defining dose S,, offsets due to outcome positive negative eo~, random errors

+ 0.29 - 0.29 0.94

Ti. offsets due to tests

see Table 3

1.67

The estimztes for positive and negative main effects illustrate the bias inherent in nonEMbased estimates. The difference in the EM-based estimates for positive and negative main effects is smaller (0,58) than that based on differences between the means of positive and negative outcomes, which was nearer to 1 log. The bias of nonEM-based estimates is caused by the varying numbers of positive and negative outcomes across tests and chemicals. If all tests were carried out on all chemicals, there would be no bias. The effect of using the Ty and Sk values is best visualized by subtracting them from the log-defining dose of each datum. Results are shown in Fig. 6 (for the Aflatoxin data of Fig. 4) and in Fig. 7 (for the entire database). Clearly there has been a dramatic reduction in the variability of the data across the entire spectrum of classes and tests. The occasional gross outliers in Fig. 7, such as the large negative result in Class A9, are examples based on a single test/chemical entry. A more precise measure of effectiveness of the adjust-

•

" llJ -I

.4, At A2 ~

Aa 41 A6 A? All AgA1O B! 9 l 83 84 8S 66 BT' U 69 B10811B~2

C)ass

Fig. 6. Adjusted Iog-ttefining doses for tests-of the chemical afiatoxin B1. This plot is identical to Fig, 4 except for the adjustment of log-defining dose for differences in test efficiency.

At A2 A3 A4 A5 A6 A7 AB A9 A10 Bl B2 B3 B4 EIS 86 B7 B8 B9 B10Bllla12 Class

Fig. 7. Mean adjusted log-defining doses for all chemicals in the database by test within class. The plot is identical to Fig. 5 except for the adjustment of log-defining dose for differences in lest sensitivity. There has been a major coalescence of the data, including a disappearance of the structured difference between positive and negative outcomes.

ment is the resulting overall 38% reduction in the among test within chemical variance, reducing the residual variation to 37% above the replication variance. Individual test adjustments are listed in Table 3 (as Tj + M). Tests with large positive adjustments (e.g, NCR and SCF in class A5) have large log-defining doses (i.e. were positive or negative at high doses). In contrast, tests with small logdefining doses (e.g. SVH and SLH in class B4) are given negative adjustments to bring them into alignment. Fig. 8 shows the overall pattern of these test data by plotting Tj, the displacement from the mean, against test within class. For the in vitro tests, the largest displacements (and hence the largest doses) are coming from tests using yeast and bacteria (upper left), while the smallest displacements are from mammalian tests that require dividing cells (low entries in A6 and A8A10). For in vivo tests, most lie near zero displacement, but tests using Drosophila (D_ in B2 and B7) have the highest doses, and tests using human cells (4 lower values in B4 and B6) have the lowest. It is unclear at present how much of this patterning is the geometry of the test systems and how much is the innate tolerance of the biological material to high doses.

35

C o n s o l i d a t i o n o f positive a n d negative results T h e a n a l y s i s u p t o this p o i n t has k e p t p o s i t i v e a n d negative results in separate categories. How-

ever, t h e e v e n t u a l p l a n for t h e a d j u s t e d l o g - d e f i n ing dose v a l u e s is to c o m b i n e r e s u l t s o f b o t h outcomes into a composite score for the chemical o r test. T h e s i m p l e s t a p p r o a c h w o u l d p u t t h e

TABLE 3 TEST DOSE ADJUSTMENT FACTORS Test

Class

BRD BSD ECD ECL ERD

A1 AI AI AI A!

SCG SCH UHF UHL UHT

Positive

Negative

Test

Class

2.48 2.16 2.36 1.12 1.96

3.07 2.74 2.94 1.70 2,54

CIA CIC CIH CI R CIS

A9 A9 A9 A9 A9

1.82 1.48 0.73 1.08 0.29

2.40 2.06 1.32 1.66 0.87

A2 A2 A3 A3 A3

2.22 2.69 1.58 2.39 0.97

2.80 3.28 2.16 2.97 1.55

CIT T7S TBM TCL TCM

A9 AI0 Al0 A 10 A10

0.60 1.27 0.61 1.09 1.01

1.18 1.85 1.19 1.67 1.59

UIA UIH URP EC2 ECF

A3 A3 A3 A4 A4

1.14 1.77 1.08 2.19 1.81

1.72 2.35 1.66 2.77 2.39

TCS TRR UPR OVA UVM

AI0 AI0 B1 BI BI

0.31 0.06 1.57 0.78 1.39

0.89 0.64 2.15 1.36 1.97

ECK ECR ECW SAL BSM

A4 A4 A4 A4 A4

2.38 2.09 1.63 1.74 2.24

2.96 2.67 2.21 2.32 2,82

UVR DMM MST SLH SVA

B! B2 B3 B4 B4

1.46 2.66 1.72 - 0.15 1.34

2.04 3.24 2.30 0.44 1,93

NCF NCR SCF SCR SZF

A5 A5 A5 A5 A5

2.45 3.07 2.56 2,13 1.47

3.03 3,65 3,14 2,71 2,05

SVH MVC MVM MVR CBA

B4 B5 B5 B5 B6

- 0.32 1.55 1,52 1.40

0.26 .". 19 2,13 2.10 1.98

G51 G5T G9H Ggo GCO

A6 A6 A6 A6 A6

1.28 !.33 1.17 0.93 1.25

1.86 1.91 1.75 1.51 1.83

CBH CLA CLH CVA DMH

B6 B6 B6 B6 B7

0.41 1.35 -0.35 0.96 2.34

0.99 1.94 0.23 1.54 2.92

GIA SCN SHF SHL SIA

A6 A7 A8 A8 A8

0.81 1.81 0.48 0.81 1.66

1.40 2.39 1.06 1.39 2.24

DML DMX SLO DLM DLR

B7 B7 B8 B9 B9

2.13 2.12 1,87 1.52 1.25

2.72 2.71 2.45 2.10 1.83

SIC SIH SIM SIR SIS

A8 A8 A8 A8 A8

0.91 0.48 - 0.01 0.73 0.40

1.50 1.06 0.58 !.31 0.98

MHT CCC CGC CGG COE

BI0 B 11 B 11 B11 B 11

1.46 1.45 1,51 1.28 1.70

2.05 2.03 2,09 1.86 2.28

SIT CHF CHL

A8 A9 A9

- 0.06

0.52 1.24 1.51

SPM SPR

B12 BI2

2.03 1.78

2.61 2.36

0.66 0.92

Positive

1.61

Negative

36

o

.

-

:i

-

:

•

~

:

i

~T

:•

~'.7a:

:

'

:

;

:

! ~

:

:s~'va).c~6...-

...... ................. ................ ~ ......... ................, ~ s . . . . . . ; . . . . : .......~v~............

0

UIA

o

,.~

.o9o

cm:

.

sic q~

:

:

"

.sul

s~!

:

roe:"

................2 ...............~ ¢ ....: ....

',

i

!

i

,

:

.

, qm

!

: -2

,

" ~

ci,,

d'~qr.~s,i sin's

es

;

:.

!

!

:

sea :

!

'cus !

AI A2 A3 A4 AS A6 A7 A8 A9 A10 B1 B2 B3 B4 B5 B6 B7 BS B9 B10B11B12

Class Fig. 8. Displacement (T~) from the mean ( M ) of test adjustments plotted by test within class. Positive displacement indicates tests that use relatively large doses independently of which chemicals they were used on, while negative displacement reflects tests using relatively small doses, The patterning in the data is discussed in the text.

dose scores in the range from - 1 to + 1 and would center the normalized positives at +0.5 and negatives at -0.5, such that the distributions for each are equally scaled, and do not cross zero. Also, directionality must be properly oriented so that strength of positive outcome goes toward the right, and strength of negative outcome goes toward the left. This can be accomplished with the following conversion: For positive results, Standardized

Dose

With such standardization, dose and other data for tests and chemicals can be entered into the basic analytical scheme presented in the previous paper (Lehman et al,, 1991) and will provide appropriately weighted test, class, family and agent scores summarizing genotoxicity. Two examples of this are shown in Figs. 10 and 11. Fig. 10 plots the relationship between agent score and the proportion of positive outcomes for every chemical in the database. Clearly, these two mea-

Score

= 0.1 • (5- (log(definingdose)- (T~+ M))) and for negative results,

i

StandardizedDoseScore •= - O. I * ( ( l o g (

defining

dose)

- (~

+ M )) + 5),

Values for (Tj + M) are given in Table 3. Using standardized dose scores, the data in Fig. 1 are now converted into the distributions shown in Fig. 9. Positive and negative outcomes now group into two near gaussian distributions, each slightly skewed to the right, with means and standard deviations as shown.

-1

-u.=

0 ~anOa~llxed~se scorn

O.S

1

Fig. 9. The distribution of standardized dose scores for the entire database. Positive and negative outcomes now form two similar gaussian distributions lying completely to each side of zero.

The

mean that

(+SD)

for the

for the negatives

positives

is 0.53 (_+0.13)

i s - 0 . 5 2 ( _+ 0 . 1 2 ) .

and

37 50 40

.

.

.

,

.

.

.

i

.

.

.

i

.

.

.

J

identification is not shown here, but the chemicals are ranked roughly as one would expect from their behavior on individual tests. Also the individual classes show reasonable consistency of class score compared to agent score. However, Fig. 11 shows a pronounced banding of the class scores above and below the agent scores. This is a dramatic demonstration of the effect of the bimodal distribution of standardized dose scores on the results. Also we strongly suspect that the orderly results in Fig. 10 may be due in large part to the overemphasis on sign of outcome when using the bimodal standardized dose scores.

"O~' ~ "l

"

.............................................................................................................................

30 0

.~

20

.................................- - ~

........................................ o .................................. ~,...........o~-----o-~So---~.-~ ~ 8o~ oo -.

Io

~'

O ~ o

o--~--.eo~---~

........................................

0

0

0

0

0 0

~.,.

-^

o . . .o . . ~ . -. o 3 - o ...............................: ..............o~........................... 0

0

0

0

0

0

o

-10 .....................................................................................................: i °oO°O°~r o o " ° ° o o o ~ o -20 o .............cP -Q""""~""~- ................................................................................................. " ...........

o •30

, 0

,

~ , ,°, l • • ' : " ' ' : ' 0.2 0.4 0.6 0.8 Proportion of outcomes that are positive

'

'

Fig. 10. Agent score as a function of proportion of positive outcomes by chemical for the entire database• Each point represents a single chemical. The correlation between the two measures is 0•80.

Further refinement of standardized dose score

The final stage of dose modification is the minimization of bimodality in the distribution of standardized dose scores with maximal preservation of the chemically discriminating dose information. Thus the approach taken will be to optimize transformations of the dosage scale that can preserve the effects of chemicalness while minimizing the effects of other sources of variation (i.e. sign of outcome, differences among tests and among replicates of the same tes~~o,

sures are highly correlated over the 113 chemicals. Fig. 11 provides another perspective by rank ordering the chemicals by their agent scores, and then plotting both the agent score and the distribution of class scores by chemical. The agent scores are broadly distributed, ranging from strongly negative to strongly positive. Chemical 100

'

.

.

so

"

•

-

."

..

.

• 40

• l

"

•

"

.*

•

• "-

"20

- • l

401

i I • j-

I

•

•

;

•

It

',''

:

"

I .

i

.d°Z ,Z

,, ,,*, •

t , ,, 5

*

•

-":'*'" " *

.|,

~ Dee • I j

.

, •

.

t I I • •

•

""

";I

"

~.ll

,"

:

el

• ,,,"

-

*

l •t

ee

•

*

Is9 I 01 • •

• • •

io

• • • •

ie

•

•

•

~" -

I i • •

i

•

0 •

.

•

,, • , , , : ,

•

, , , n. •

" l , r l . e . .wv ' ;•, : - .

" ..... ".*" .,..~ - ,

.

•

o o •

• I e ii

•

• •

el I I • lle de •

•

..,

-~,,.l'...?..'.::.V'~Ti.:

0• i •* o• • • • • • • .joeeee4I • • era| •lIe~eoe • -

• •tS; * I,

•

,,, ,

•

DI~

.

l i

"-

....

eI

ee

_, t , , . . I ~ ,

-6out,,g,:ol:P,;:.l:"

"801

I~

"

...

..,..,.....:

•

,,

".. • ' , "

. .¢g o. , %'." l | 1 %- "• . . ' •, - . -. i-: - I~N ' ; . l l •

. e"

I

•

..

-

",'1",-, I ... -. •

.

"'"','.'" •

.

. 01..•. e•-"

eel

I I ~lJI. I I~ie~ ¢6®o •

IIIeII" . . . .

"

•

• ,,'"~.,,:,,

i .....!.

.. ,

:" ". -

I ,•

•

•

•" % •o

"'

,. ..

:

• • • I ;

• , ¢

•

.

20

.

.

ii

o

I

• i

•

I

, .0.',, ,,,.| : I I Ii I • i iI • q.. • • : : ! . I . ,, •

•

,

•

.

"

-100 Chemicals ordered by agent score Fig. ] |. Class and agent scores for chemicals ordered

by their agent scores. Agent

plotted

diagonal

as p l u s s e s ( + )

and appear

as a n i n t e r m i t t e n t

line running

upward

scores, which from

are averages of class scores, are

left to right in the figure. Class scores,

p l o t t e d as s o l i d c i r c l e s ( e ) , a p p e a r i n t w o d i s t i n , ' t b a n d s , r e f l e c t i n g a n o v e r e m p h a s i s

on sign of outcome.

38

We consider the famiiy of transformations given by

Y = sign(X)

{ IX~cl)P

No shift

500 400

o" LI.

1 ------~

300 200 100

-1

where X is the standardized dose score. The shift parameter c is a nonnegative constant which is subtracted from positive outcomes (which have positive X values) and added to negative outcomes (which have negative X values). Applying c results in shifting the two gaussian shaped distributions of Fig. 9 closer together, as shown in Fig. 12. The power parameter p changes the shapes of the distributions which also results in pulling the distributions together (see Fig. 13). The factor ( 1 - c) in the denominator ensures that the range of transformed values is maintained at - 1 to + 1. The term sign(X) preserves the sign of the outcome so that Y has the same sign as X. The largest reasonable value for the shift parameter is c = 0.20. Even at this value 21 negative outcomes (with values between -0.20 and 0) and 8 positive outcomes (with values between 0 and +0.20) had to be assigned zero values to avoid changing sign. Further shifting would result in a spike of zero values which would destroy differences among small doses. Fig. 13 demonstrates that increasing the power parameter (p) changes the shape of the distribution by a nonlinear shifting of valu¢s toward zero. Values near the extremes, zero and :t: 1, are shifted less than those near ¢ 0,5 in contrast to the shift transformation which shifts all values equally, The final step is to make an optimal choice for the two parameters, c and p, based on some formal criterion. To accomplish this, we partition the variance into two components: one for variations among chemicals and the other, termed residual, for all other sources of variation. Our optimization criterion is the ratio of variation among chemicals divided by residual variation. This ensures that after transformation we will attain the maximum dispersion of scores for chemicals. We employ the statistical method of one-way analysis of variance (ANOVA) to accomplish the partitioning. The results for various

500-

0

1

o

1

Shiit = _+0.10

p, 300 i, 200

400

100.

500 400 S h i ~ 300 u.

2011100

•1

0

1

Transformed standardized dose

Fig. 12. Effect of changing the shift parameter in the shiftpower tr,nsformation. The Iow(:r panels show the distribution of standardized dose scores after shifting negative outcomes to the right (toward zero) and the positive outcomes to the left (toward zero) by the amount shown in each panel. In the upper panel there is no shift so the distribution is the same as in Fig. 9. Increasing the magnitude of the shift parameter reduces the distance between the two distributions, eventually causing them to overlap. At c = 0.20. the potential overlaps have been held at each side of zero and do not perturb the trough. Further shifting would cause a piling up of such overlaps, and would be counterproductive.

combinations of the parameters c and p are summarized in Table 4 where our strategy was to first optimize c and then optimize p. Table 4 shows a consistent improvement in score as c increases to 0.2, its largest reasonable value. With c fixed at 0.2, increasing values of p first increase and then decrease the score, with the best score occurring at p = 2. Fig. 14 plots additional results for c = 0, and compares them to the tabular values for c = 0.2, with each dataset given a quadratic fit. The same conclusion is

39 500'

46

Power = 1

400

300

S ¢J o~

h

i

f

t

-

P

o

w

e

~

44.

200

IL

100' to

-1.0 800

-0.5

0.5

0

1.0

~~

Power=3

42,

40

600 400

I1

38

0

200

-1.0

-0.5

0

0.5

1.0

2000 Power = 5

1

2

3

4

Power Fig. 14. Relative performance of the shift-powe~ transformation of standardized dose. Chemical difference,~ as a percent of total sum of squares arc plotted against parameters of the transformation. The data for c = 0 and c = 0.20 are fit by quadratic polynomials. To the best that present data can resolve, the optimal transformation is at c = 0.20 and p = 2.

1500

|

u.

1000

500.

I

-1.0

I

.0.5

,

0

:

0.5

:

1,0

Transformed standardizeddose

Fig. 13. Effect of changing the power parameter in the shiftpower transformation. Each panel shows the distribution of standardized dose scores after raising the (unshifted) values to a power. Increasing the power reduces the range of values and fills up the center of the distribution.

Given the optimal transformation, the definitive distribution of transformed values is shown in Fig. 15 and described statistically in Table 5. When comparing the untransformed and shiftpower transformed, standardized dose scores, the positive and negative means are closer together, the skew of negative values has shifted from right to left and a greater proportion of the positive distribution has been shifted to the tails as measured by the increase in kurtosis. Note that the 600'

reached, namely that for present purposes the optimal transformation parameters are c ffi 0.2 and p -- 2.

TABLE 4

400,

200

EFFECTS O F SHIFT-POWER T R A N S F O R M A T I O N ON V A R I A N C E COMPONENTS Parameters

Variance components

p

c

Chemicals

Residual

Ratio

1 I 1 2 3

0 0. ! 0.2 0.2 0.2

4.2296 3.6630 3.0393 0.9675 0.3630

0,1694 0.1385 0.1057 0.0296 0.0122

25.0 26.4 28.8 32.7 29.8

0 11

-0.S

05

1

Shift power transformed standar0ized dose score

Fig. 15. Distribution of shift-power transformed standardized dose scores. Compared to the distribution in Fig. 9 (untransformed data) the gap between positive and negative outcomes has been removed and there are relaf~'ely more data in the tails of each distributioa.

40

Fig. 16 uses the shift-power transformation to show class and agent scores plotted against chemicals ordered by their agent scores. The transformation completely eliminates the parallel banding of Fig. 11. It also reduces the range of agent scores, but the overall response and uniformity of the data are considerably better than anything we have seen in the evolution of the method.

TABLE 5 C O M P A R I S O N OF S U M M A R Y STATISTICS Untransformed

Transformed

Negatices Number Mean Median Standard Deviation Interquarti|e range Minimum Maximum Skewness Kurtosis

1581 - 0.522 - 0.530 0.120 0.144 - 0.831 - 0.032 0.559 0.808

- 0.185 - 0.170 0.114 0.149 - 0.622

Evaluation of final dose score

0

- 0.719 0.531

The appropriateness of the mathematical model and the shift-power transformation used to describe the data can be evaluated by examining the residuals:

Positices Number Mean Median Standard D~viation lnterquartile range Minimum Maximum Skewness Kurtosis

2871 0.533 0.524 0.132 0.171 0.120 1 0,291 0,210

0.201 0.164 0.150 0.176 0 1 !.468 3.044

%k = X, jk -- M - T~ - q - Sk.

Fig. 17 shows residuals plotted against expected deviates from a standard normal (gaussian) distribution. The appropriateness of the model is judged by how closely the plot approaches a straight line. In the upper part, which shows residuals before transformation, negative residuals, making up approximately 5% of the data, deviate from the line. This is the result of a negative skew. The coefficient of skewness for

interquartile range, which covers the central 50% of the values for each distribution, is relatively unaffected by the transformation.

00]

:',[

•

'

60'

"

4

.,

40

• ,, •

•

,, I

• * " " • . ,, oo;. ,,,•

,

•

t

•

,

, * oh

•''.

" * , •|0 |

**l

*

*

4,0

•

.:,......n, : e

"

i-

•

•

SIS* •

I

•

I. • •-i

I'~*.l "l "m° •. • .i • i ! •:, . ..

•

•

Is

• ..

•. • •

"

* e • •Q • • •

," i t:d.llJ'*nl;.Plq .j .iLTo,, h l l | ' , ' i . l ~ ] : ~ : ~. I,

| . .* •l.'~

....

•

I

iI

•

,,

,~• O • , ,:e

ee

• I|

i I hV

**'1

i!nh|.*~lil

**rT"

|

J • . . * . • .~ * ~.1:~.'* , . o * * I . ' • l .~ ~ • . . . ; ** • ** ,, "•l,••Lh:h:,•.* ,, ::l, ,: .t :',•, , I,~ • • 1:*•. p h : ,** ,," :

•

:.

.

i- *".~".~;|:'||l'|i • :. ~.,.~s,:! •,::, •';. • ': i *0" .. • "..i" -" ...•* d" • I% * *• ° " " . • " • cog I

•6 0

.

•* *

Be

"

."

40

:--|

,, •

"20

"

'""

*%

|

• I

.•. • : l" *

***,

4

"

• •

•-

Ib

I

•

•

• •

m

•

I

••

~

Chemicals ordered by agent score

Fig. 16, Class a n d agent scores f o r chemicals o r d e r e d by t h e i r agent scores a f t e r t r a n s f o r m a t i o n , A g e n t scores ( + ) a p p e a r as a d i a g o n a l line r u n n i n g u p w a r d f r o m left t o right in the figure, Class scores ( o ) are n o w f r e e o f the b a n d i n g e f f e c t seen in Fig. 11.

41 4 2 0 .-g ft.

-2 -4

.,-................................................

.6 .01 .I

80

1

51020

50

8090

99 99.9

AfterShift-PowerTransformation

40

•[

"

-2i°o.,o2°

"~=

-60,01 .1

1

5

10 20

50

80 90

99 99.9

Normal distribution percentile Fig, 17. Normal probability plots for residuals of log-defining doses before and after transformation. The upper figure shows residuals for untransformed log-defining doses after fitting to the A N O V A model. The lower figure shows the same for residuals for shift-power transformed scores. There are over 2000 data points in each figure so that the observed deviations from a straight line (expected for samples from a normal distribution) are both statistically significant, The smaller number of points deviating from the line in the lower figure indicates a better fit under the shift-power transformation.

these data is -0.412. In the lower part, showing residuals after the shift-power transformation, fewer than 1% of the largest residuals deviate from the straight line. The coefficient of skewness for these data is 0.189. Comparison of the absolute values for the skewness coefficients indicates that the shift-power model fits the data better than the untransformed model (although both are beyond the 1% tail value, _+0.13, for samples from a normal distribution). A similar comparison of the coefficient for kurtosis, which measures the proportion of the distribution in the tails relative to a normal distribution, supports the assertion of better fit after shift-power transformation. The kurtosis is 0.963 on untrans-

formed data and 0.822 after shift-power transformation (both are larger than the 1% upper tail value of 0.28 for samples from a normal distribution). Finally, we wish to verify that our scoring scheme performs as intended by making use of both dose information and sign of outcome. To do this we return to data where Fositive and negative outcomes have yet to be combined. In Fig. 18, shift-power transformed replicate scores for all entries in the database are plotted against chemicals ranked by agent scores. It is clear from the figure that chemicals with low agent scores are those with mostly negative outcomes as evidenced by a greater proportion of data points below the zero horizontal line. The relatively infrequent positive outcomes for these chemicals tend to be at high doses (i.e., are weak positives) and therefore plot near zero. The opposite holds true for chemicals with high agent scores. For these, the majority of replicates have positive outcomes and high scores. There is a broad continuum in going from the strong negatives to the strong positives characterized by a gradual increase in both the proportion of outcomes that are positive and the magnitude of the positive entry scores. There is a concomitant decrease in the magnitude of the negative entry scores proceeding from left to right in the plot. These results indicate a strong consistency between the outcome and the dose-derived strength of the outcome in the entire database. Thus our method of dose transformation adequately meets the design criteria. Discussion Table 3 is one definitive outcome of this exercise, and is a listing of the dose adjustment factors for each of the 85 genotoxicity tests we have studied. It satisfies our need for a set of factors that best adjust each test for its relative efficiency in the use of chemical. The factors have been estimated in the presence of extensive confounding with chemical effects, and with a matrix well under half full. The adjustment factors may be of interest to those concerned with comparative efficiencies of genot.oxic tests, although the primary

42 | .=

°.L

:

•

•

•"

i

•

.

•

•

"O

0.4,

•

• .......,,..

. -.... :....,

. . .

. • • "4: •" ~ , . . . . . .. , •

,-

.

~

• •

Q)

• ."

. I;..n

•

• •

•

=|

•

•

|

|

- l ' , ' ~ g " h

• .-. • " , • :, ",," . .'."m " -

=1, :'1:" |" .,..pJl.'i

• I" :-,1-'-:,,

.l-

•

"

m|

|

1.!.,. , ,..I ~.~" 1.......:.,.....,,.,.1:1. ... -,.• ,.,.~ m, -,,,. t,ºJ!!.llJl

•. :, ,.., ' ..,,, ",.,; -.,...:. ,'.;, "-.;.,e:e!|:.!.a,LI " '" " I,I,)::.,! I ,. '14..I".r ~,,l;:l I.,iri i:,..~. "ill .:.'i!a. ":: :" I~l;q-~Uli ,pil.l]llr'. -i#,,:..4, ,'. - 5 ~i-~i . ' ; J • t . . . p . . , i st. r F ' - . ~ - J ~ ' - ; ' ~ ' u =.J~ "~; L _ ~ - . ~ ' ~ u . - r r p ,:,1-

0,0 . ¢ .

•

•

":

., : . . u . . . .

-'"

' """

"

"

."

.0.4

,1~:i-; . . . .

" I I "I "Y " ' : r •il~'l*

II

•

• .1., . . ,,, . . . . ~:s.m.. ~ - : i r 1:. ! ....

e-

¢/)

'J

"d"

. ..

,..l

--.-*r,:

•

,''..;-,-

'hi " ~ e I ~ . . , t , [. ;; .! %'0." i " ~.~.i.I 1..,. i,i ,h.,. rl -..', ,4 • • ", • • ..,,,!,f ,.h:h~ ~. 1.d1!:i ,.k.,~,,., ,,-,!lºJ, .>.,..,. --0'; • ! .' el i ..,

Q.

i

.. ...¢ me 1.;., . . . . . !. .. . .. .. . .•. . * I.

I "'iP..~ "" " - . . . . . |.

;.'~" ..'..

:'"

" "

.

.'"

|

i

Ill.

, , •

•

* •

... .

.•

•

..

•

•

;

Chemicals ordered by agent score Fig. 18. Shift-power transformed replicate scores for the 113 chemicals in the database. Each replicate observation for a particular chc,mical subjected Ioa particular test is plotted as a single data point. Those above the zero horizontal are positive outcomes, those below arc negative outcomes, and no averaging has taken place. The plot shov,; both the magnitude of the score and the relative amount of positive and negative outcomes. As agent score increases, there is a concomitant increase in the proportion o f outcomes that are positive and the magnitude of positive results, while the magnitude of negative results decreases. Clearly the outcome and dose.related information are well preserved using the adjusted and shift-power transformed data,

interest to us has been to be able to put this effect aside. The second definitive result is the demonstration of the striking effects of dichotomization of outcome on the pooling of quantitative scores of genotoxicity. While there are important advantages to having yes/no answers about a biological phenomenon, anyone working with ensembles of genotoxicity data learns sooner or later how difficult it is to come to terms with such a black and white outcome in the face of so much variation of response. The third and final definitive outcome is the ability to bypass the dichotomization problem with an appropriate transformation resulting in the display of well-behaved, collective, quantitative genotoxicity data. This was the ultimate methodological goal of our methods development, T h e r e now remains the demonstration of overall performance and early experiences with the method, as

will be described in the next paper of this series (Mendelsohn et al., 1992).

Acknowledgement Some of the work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract W-7405-Eng-48. References

Dempster, A.P., N.M. Laird and P.B. Rubin (1977) M a x i m u m l i k e l i h o o d f r o m incomplete data via the EM algorithm, J. Roy, Statist. See., Ser. B 39, 1-38. Lehman, P,H.M., M,L. Mendelsohn, D,H. M o o r 11, M.D. Waters, D.J. Brusick, J, Ashby and WJ.A. Lehman (1992) A method f o r comparing and combining short-term genotoxicity test data: The basic system, Mutation Ros., 266, 7-25. Mendelsohn, M,L,, D,H, Moore I! and P,H,M. Lehman (1992) A method f o r comparing and combining short-term genotoxicity test data: Results and interpretation, Mutation

Res., 266, 43-60. Waters, M.D., H.F. Stack, A,L. Brady, P.H.M. Lehman, L. Haroun and H. Vainio (1988) Use of computerized data listings and activity profiles of genetic and related effects in the review of 195 compounds, Mutation Res., 205, 295-312.