Br. J. clin. Pharmac. (1991), 32, 407-408
Spontaneous reporting: how many cases are required to trigger a warning? P. TUBERT', B. BEGAUD2, F. HARAMBURU2 & J. C. PERE2 'Unite INSERM 169-16, Avenue Paul Vaillant Couturier-94807 Villejuif Cedex and 2Centre de Pharmacovigilance, Hopital Pellegrin, 33076 Bordeaux Cedex, France
A statistical method is proposed to aid decisions on the recognition of new adverse drug reactions on the basis of spontaneous reports. The maximal number of associations between drug exposure and event is calculated from an independence test between drug and event, taking into account the expected incidence of that particular event in the general population. An example is given which illustrates the method.
Introduction
The chance of detecting rare and severe adverse drug reactions (ADRs) during Phase 3 clinical trials is extremely low. Such ADRs usually emerge only after the drug has been marketed and prescribed to several thousand patients. Spontaneous reporting remains the best method of detecting new ADRs. Two main problems require consideration. Only a fraction of ADRs, which is unknown, is detected and reported, and the causal relation between drug intake and the event is usually uncertain. It is generally impossible to identify a new ADR from the first report as the clinical features are rarely pathognomic of an adverse drug reaction. Although many procedures have been proposed for assessing adverse drug reactions, considerable difficulties remain in identifying ADRs. Associations of drug exposure with events which are not causal can be expected due to chance. However, when drug treatment is associated with the given event frequently this can no longer be considered coincidental. The purpose of this paper is to define the critical number of events in a population exposed to the drug required to trigger a warning of an ADR, while limiting the risk of false alerts of a drugdisease association. The statistical decision rule proposed is based on the population size, the frequency of drug exposure, and the incidence or prevalence of the event in the general population.
incidence of the event or symptom is I. If drug treatment and the event were independent, the probability distribution of the total number of associations (X) approximates to a Poisson distribution with the parameter I-T being the expected number of associations between drug and event during the period. The probability of having at least x associations between drug and event, assuming independence of the drug and event, is x-1
Prob (X-x) = 1-
E e-m
mk
- , where m = IT
k=o0k
For a given value of m, the test of independence between the event and exposure to the drug is performed by limiting the risk of rejecting erroneously the hypothesis of independence Ho to a given value a. The critical number of events (Xcr) is given by:
Prob (X 2 Xcr when Ho) a and Prob (X 2 Xcr - 1 when Ho) > a. -
If the observed number of associations between drug and event, X, is higher than Xcr, the hypothesis of independence is rejected. Table 1 shows the critical numbers (Xcr) obtained for 91 combinations of m (ranging from 0.1 to 100) and alpha (from 1% to 30%). Values of m depend upon both symptoms incidence and the number of subjects exposed to the drug. For example, m = 10 may correspond to a low incidence of the symptom (I = 1/1,000) and a low exposure to a drug (T = 10,000), or to a very low incidence of the symptom (I = 1/100,000) and a higher risk of exposure (T = 1 million). In this case the critical number is 15 when the type 1 error is 10%.
Method
Let us consider that in a given geographical area and defined period of time (e.g. 1 year) the number of patients treated with the drug is T, and the background
Correspondence: Dr P. Tubert, Unite INSERM 169-16, Avenue Paul Vaillant Couturier-94807 Villejuif Cedex, France
407
408
P. Tubert et al.
Table 1 Critical number of associations between drug and event, according to m = I*T and a, where I is the background incidence of the event, T the number of patients treated by the drug, and a the significance level
quently confirmed by the French National Board for Drug Safety. Discussion
Xcr
m
0.01
0.05
0.10
a 0.15
100 50 40 30 20 10 5 4 3 2 1 0.5 0.1
125 68 56 44 32 19 12 10 9 7 5 4 2
118 63 52 40 29 16 10 9 7 6 4 3 1
114 60 49 38 27 15 9 8 6 5 3 2 1
111 58 48 37 26 14 8 7 6 4 3 2
109 57 46 36 25 14 8 7 5 4 3 2
1
1
0.20
0.25
0.30
The approach outlined above has its limitations:
108 56 45 35 24 13 7 6 5 4 3 2 1
106 55 44 34 23 13 7 6 5
1 It does not signal a causal relationship, only a statistical association which must be investigated further. 2 It is based on spontaneous reporting, a method which captures only a small fraction of all associations between drug and event. In addition to under-reporting of possible ADRs, which is well recognised, prescribers tend to notify only those, events which are suspected to be adverse drug reactions, and not events for which is a clear alternative explanation. The rate of reporting may vary greatly between and within countries, and it may be increased by published case reports or by reports in the media. These factors will only decrease the power of the statistical test provided that for each report the patient was actually exposed to the drug and did have the event under consideration. The statistical power of the method can be increased to take account of possible under-reporting by accepting a significance level higher than that usually chosen in statistical inference. However, it can be seen from Table 1 that the critical values Xcr are not greatly affected when alpha varies between 0.01 and 0.3. This is a consequence of the relatively small dispersion of the Poisson distribution. 3 It can be difficult to estimate the background incidence of the event which is required for the calculation of m. The incidence in patients likely to be treated with the drug may be different from the incidence in the general population. However, it is often possible to reach a satisfactory conclusion despite these difficulties. In the example cited for instance, an m value of 10 for a = 0.05 would have been necessary to accept Ho. This would have corresponded to an annual incidence of acute liver injury of 1/200, a value which would clearly have been unrealistic.
4 2 2 1
Example
Eleven episodes of hepatitis reported to the regional monitoring center of Bordeaux during 1985 involved patients treated by fluindione, an oral anticoagulant. The number of patients treated with this drug in the relevant area, during this period, was estimated as T = 2,000 from prescription data for the drug while the annual background incidence of similar liver injury in a similar population was estimated to be 1/2,500 by a group of experts. In this case m = 0.8 and the critical number (Xcr) was below 11 for values of risk alpha in the range c 30%. Moreover, using the Poisson distribution, it was calculated that the P-value Prob (X - 11 when Ho) was smaller than 0.0001. For values of alpha greater than this critical level the hypothesis of independence could be rejected. Thus fluindione (which is pharmacologically different from other indane-dione derivatives) was considered to be possibly hepatotoxic. The reality of this risk, despite its relatively low incidence, was subse-
(Received 4 September 1990, accepted 20 May 1991)
Spontaneous reporting: how many cases are required to trigger a warning? P. TUBERT', B. BEGAUD2, F. HARAMBURU2 & J. C. PERE2 'Unite INSERM 169-16, Avenue Paul Vaillant Couturier-94807 Villejuif Cedex and 2Centre de Pharmacovigilance, Hopital Pellegrin, 33076 Bordeaux Cedex, France
A statistical method is proposed to aid decisions on the recognition of new adverse drug reactions on the basis of spontaneous reports. The maximal number of associations between drug exposure and event is calculated from an independence test between drug and event, taking into account the expected incidence of that particular event in the general population. An example is given which illustrates the method.
Introduction
The chance of detecting rare and severe adverse drug reactions (ADRs) during Phase 3 clinical trials is extremely low. Such ADRs usually emerge only after the drug has been marketed and prescribed to several thousand patients. Spontaneous reporting remains the best method of detecting new ADRs. Two main problems require consideration. Only a fraction of ADRs, which is unknown, is detected and reported, and the causal relation between drug intake and the event is usually uncertain. It is generally impossible to identify a new ADR from the first report as the clinical features are rarely pathognomic of an adverse drug reaction. Although many procedures have been proposed for assessing adverse drug reactions, considerable difficulties remain in identifying ADRs. Associations of drug exposure with events which are not causal can be expected due to chance. However, when drug treatment is associated with the given event frequently this can no longer be considered coincidental. The purpose of this paper is to define the critical number of events in a population exposed to the drug required to trigger a warning of an ADR, while limiting the risk of false alerts of a drugdisease association. The statistical decision rule proposed is based on the population size, the frequency of drug exposure, and the incidence or prevalence of the event in the general population.
incidence of the event or symptom is I. If drug treatment and the event were independent, the probability distribution of the total number of associations (X) approximates to a Poisson distribution with the parameter I-T being the expected number of associations between drug and event during the period. The probability of having at least x associations between drug and event, assuming independence of the drug and event, is x-1
Prob (X-x) = 1-
E e-m
mk
- , where m = IT
k=o0k
For a given value of m, the test of independence between the event and exposure to the drug is performed by limiting the risk of rejecting erroneously the hypothesis of independence Ho to a given value a. The critical number of events (Xcr) is given by:
Prob (X 2 Xcr when Ho) a and Prob (X 2 Xcr - 1 when Ho) > a. -
If the observed number of associations between drug and event, X, is higher than Xcr, the hypothesis of independence is rejected. Table 1 shows the critical numbers (Xcr) obtained for 91 combinations of m (ranging from 0.1 to 100) and alpha (from 1% to 30%). Values of m depend upon both symptoms incidence and the number of subjects exposed to the drug. For example, m = 10 may correspond to a low incidence of the symptom (I = 1/1,000) and a low exposure to a drug (T = 10,000), or to a very low incidence of the symptom (I = 1/100,000) and a higher risk of exposure (T = 1 million). In this case the critical number is 15 when the type 1 error is 10%.
Method
Let us consider that in a given geographical area and defined period of time (e.g. 1 year) the number of patients treated with the drug is T, and the background
Correspondence: Dr P. Tubert, Unite INSERM 169-16, Avenue Paul Vaillant Couturier-94807 Villejuif Cedex, France
407
408
P. Tubert et al.
Table 1 Critical number of associations between drug and event, according to m = I*T and a, where I is the background incidence of the event, T the number of patients treated by the drug, and a the significance level
quently confirmed by the French National Board for Drug Safety. Discussion
Xcr
m
0.01
0.05
0.10
a 0.15
100 50 40 30 20 10 5 4 3 2 1 0.5 0.1
125 68 56 44 32 19 12 10 9 7 5 4 2
118 63 52 40 29 16 10 9 7 6 4 3 1
114 60 49 38 27 15 9 8 6 5 3 2 1
111 58 48 37 26 14 8 7 6 4 3 2
109 57 46 36 25 14 8 7 5 4 3 2
1
1
0.20
0.25
0.30
The approach outlined above has its limitations:
108 56 45 35 24 13 7 6 5 4 3 2 1
106 55 44 34 23 13 7 6 5
1 It does not signal a causal relationship, only a statistical association which must be investigated further. 2 It is based on spontaneous reporting, a method which captures only a small fraction of all associations between drug and event. In addition to under-reporting of possible ADRs, which is well recognised, prescribers tend to notify only those, events which are suspected to be adverse drug reactions, and not events for which is a clear alternative explanation. The rate of reporting may vary greatly between and within countries, and it may be increased by published case reports or by reports in the media. These factors will only decrease the power of the statistical test provided that for each report the patient was actually exposed to the drug and did have the event under consideration. The statistical power of the method can be increased to take account of possible under-reporting by accepting a significance level higher than that usually chosen in statistical inference. However, it can be seen from Table 1 that the critical values Xcr are not greatly affected when alpha varies between 0.01 and 0.3. This is a consequence of the relatively small dispersion of the Poisson distribution. 3 It can be difficult to estimate the background incidence of the event which is required for the calculation of m. The incidence in patients likely to be treated with the drug may be different from the incidence in the general population. However, it is often possible to reach a satisfactory conclusion despite these difficulties. In the example cited for instance, an m value of 10 for a = 0.05 would have been necessary to accept Ho. This would have corresponded to an annual incidence of acute liver injury of 1/200, a value which would clearly have been unrealistic.
4 2 2 1
Example
Eleven episodes of hepatitis reported to the regional monitoring center of Bordeaux during 1985 involved patients treated by fluindione, an oral anticoagulant. The number of patients treated with this drug in the relevant area, during this period, was estimated as T = 2,000 from prescription data for the drug while the annual background incidence of similar liver injury in a similar population was estimated to be 1/2,500 by a group of experts. In this case m = 0.8 and the critical number (Xcr) was below 11 for values of risk alpha in the range c 30%. Moreover, using the Poisson distribution, it was calculated that the P-value Prob (X - 11 when Ho) was smaller than 0.0001. For values of alpha greater than this critical level the hypothesis of independence could be rejected. Thus fluindione (which is pharmacologically different from other indane-dione derivatives) was considered to be possibly hepatotoxic. The reality of this risk, despite its relatively low incidence, was subse-
(Received 4 September 1990, accepted 20 May 1991)
