1338
Letters to the Editors
Response
PROS AND CONS OF RESULT-SPECIFIC
In my essay, I said, “If one wishes to interpret a particular test response for a particular patient, one needs to known the sensitivity and specificity of that particular response . . . rather than for some arbitrary categorical threshold.” The ratio of sensitivity for “that particular response” to 1 - specificity for “that particular response” is identical to the result-specific likelihood ratio referred to by Irwig. I therefore fully agree that result-specific likelihood ratios can be used as an aid to the Bayesian interpretation of an individual patient’s test response [l, 21. I disagree, however, with the statement that result-specific likelihood ratios “provide a clinically useful measure of test accuracy.” Because the likelihood ratio (especially the result-specific likelihood ratio) represents only one of a spectrum of possible test responses, and because the likelihood ratio does not consider the frequency of occurrence of the test response. It cannot serve as a measure of a test’s discriminant accuracy. Rather, the area under a receiver-operation characteristic curve [3,4], or
LIKELIHOOD
RATIOS
the information content [l, 21 derived from the entire distribution of responses in the diseased and non-diseased populations are more proper measures of the test’s discriminant accuracy. GEORGE A. DIAMOND Cedars -Sinai Medical Center Los Angeles, CA !20048 U.S.A.
REFERENCES Diamond GA, Forrester JS, Hirsch M, Staniloff HM,
‘. Vas R, Berman DS, Swan HJC. Application of
2,
3. 4.
conditional probability analysis to the clinical diagnosis of coronary artery disease. J Clin Invest 1980; 65: 1210-1221. Diamond GA, Hirsch M, Forrester JS, Staniloff HM, Vas R, Halpern SW, Swan HJC. Application of information theory to clinical diagnostic testing. The electrocardiographic stress test. Circulation 1981; 63: 915-921. Swets JA. Measuring the accuracy of diagnostic systems. Science 1988; 24th 1285-1289. Diamond GA. Alhrmative actions. Can the discriminant accuracy of a test be determined in the face of selection bias? Mad De& Making 1991; 11: 48-56.
Letters to the Editors THE IGNOBLE LIE
In a recent issue of this journal an editor (Alvan Feinstein) and a reviewer (Kathi Kemper) drew attention to a hoax which had been perpetrated upon them by a social worker, William Epstein [I, 21. As part of his research into the process of peer review Epstein had submitted a false manuscript, the response of the reviewer being the object of his study. Both Kemper and Feinstein discuss the ethics of this behaviour drawing parallels to the problem of consent in clinical trials but coming to slightly different
conclusions. The object of this note is to point out (1) that Epstein’s approach is unethical because it flouts a fundamental principle, (2) that any attempt at blinding subjects without agreeing the procedure beforehand is scientifically worthless, and (3) that there is one particular aspect of many clinical trials which is unethical and unscientific in a very similar way to Epstein’s research. The fundamental principle which is being flouted is Kant’s categorical imperative which
Letters to the Editors has been phrased, “always act in such a way that the principle underlying the action could be conceived as turned into a general law” [3]. I do not regard Epstein’s behaviour as unethical because Feinstein and Kemper were hoaxed (as a practical joke it might possibly be acceptable). I regard it as unethical and unscientific because this sort of research creates distrust whilst itself requiring trust to be effective. It thus debases for others the coin in which it seeks payment. This point of view is implicit in Kemper’s commentary where she states “Future research is jeopardized if potential subjects suspect they will be deceived”. If Epstein’s research is regularly repeated it will then be seen quite clearly that no advantage can be claimed by such deception which would not be available to a study with informed consent. I should like to propose two particular applications of Kant’s categorical imperative for all research involving blinding of subjects. First that the .researcher should not expect higher standards of his subjects than he employs himself. If he deceives them he deserves to be deceived by them. Secondly he should not suppose that he has greater knowledge or intelligence than his subjects. If he hopes to outwit them he deserves to be outwitted. If a researcher does not tell his subjects the truth, what right has he to expect that they are going to play fair with him and why, for that matter, should the reader believe the researcher’s account of what happened? The ethical policy to adopt in all such research is one of mutually agreed deception [4]. For this purpose one should use experiments with “open protocols and hidden allocations”. This was a point recognized a long time ago by R. A. Fisher in his famous description of a tea-tasting experiment. Fisher writes, “The subject has been told in advance of what the test will consist, namely that she will be asked to taste eight cups, that these shall be four of each kind, and that they shall be presented to her in a random order,” [5]. Although he does not cover the point explicitly, Fisher clearly rejects as inadmissible any attempt to “improve” the experiment by not informing the subject that the allocation is to be blocked in blocks of eight. It is a pity that the same acumen is not shown by certain modem-day trialists who insist on regarding the blinding as having been improved by not publishing the block size. An experimenter of lesser ability than Fisher might regard the probability of the lady’s guess-
1339
ing correctly for each cup whether tea or milk was poured in first as having been reduced from 1 in 70 (the number of possible arrangements of twice four objects of two different sorts) to 1 in 256 (the total number of ways of filling eight places with two different kinds of object) if she is not informed that there will be four cups of each kind. This argument requires him to assume, however, that the lady has not guessed that he will use a balanced allocation process rather than a completely random one. If he is not careful it is he who will end up being deceived by the lady rather than vice versa. A similar approach to Epstein’s is pointlessly used in clinical trials which employ so called placebo run-ins. Such run-ins are unethical and unscientific because they involve a deception which can only hope to be successful if the protocol is hidden from the patient. If the patient is told there will be a placebo run-in there is no point in having one. This is quite different from the open protocol blinding which is possible if the patient is told that he will be allocated placebo or verum at random. Placebo run-ins rely on a “stupid patient” argument. The investigator must assume that the patient is too stupid or ignorant to realize that clinical trials frequently have placebo runins. Presumably the protocols of such clinical trials must exclude statisticians and doctors as potential patients and of course if we had a society in which the ordinary citizen was trained in the theory of clinical trial (and why not?) they would have to exclude everybody. To return to the original context, Feinstein considers that the problem of “guarding the guardians” is serious enough to justify deception without consent [l]. Plato did, in fact, allow the noble lie, although he reserved it for a different purpose. I consider, however, that it has no part in solving Feinstein’s or any other scientific problem and I should like to recommend the following maxim as worth the serious consideration of all who carry out research on human beings. “In any experiment in which the investigator deceives without consent somebody is indeed being fooled but it is not always the subject”. STEPHEN SENN Ciba -Geigy Ltd CH 402 Basle Switzerland
1340
Letters to the Editors
REFERENCES 1. Feinstein A. Construction, consent and condemnation in research on peer review. J Clin Epidemiol 199 1; 441 339-341. 2. Kemper, K. Pride and prejudice in peer review. J Clln Epidemiol 1991; 44: 343-345.
3. Vesey G. and Foulkes P. Collins Dictionary of Philasophy. London: Collins; 1990.
4.
Freedman B. Placebo controlled trials and the logic of clinical purpose. IRB: A Review of Human Subjects Research 1990; 12: l-6. 5. Fisher R. A. The Design of Experiments; reprinted in: Bennet H, Ed. Statistical Methods, Experimental Design and Scientific Inference. Oxford University Press; 1990.
Response When Kant proposed his categorical imperative, I doubt that he thought it might be applied in research testing whether journal editors or reviewers are biased in appraising manuscripts. As I indicated in my previous comments, this type of research is very different from a clinical trial or other experimental intervention; and I feared that the act of getting the editors’ informed consent about a particular “test” manuscript would destroy the opportunity to reveal their bias. In a recent brief report of a test for “reviewer bias”, Ernst et al. [l] said that their “study would not have been possible with informed consent”. Senn’s other comments may or may not be pertinent for all clinical trials. For example, I disagree that a placebo run-in necessarily
implies that the investigators believe patients are “stupid”. Regardless of the scope, depth, and inflexibility of Senn’s concluding “maxim”, however, and regardless of the ethical (or scientific) delinquencies in “Epstein’s behaviour”, I would still like to know how to design a suitably blinded but fully consented and pragmatically feasible study of editorial bias. ALVAN R. FEINSTEIN Yale University School of Medicine 333 Cedar Street New Haven, CT 06510-8025 U.S.A.
REFERENCE 1. Ernst E, Resch KL, Uher EM. (Letter to Editors) Reviewer bias. Ann Intern Med 1992; 116: 958.
Editors’ Note-In accordance with the Journal’s custom, Dr Senn’s letter was also sent to Dr Kathi Kemper. She preferred not to respond.
Letters to the Editors
Response
PROS AND CONS OF RESULT-SPECIFIC
In my essay, I said, “If one wishes to interpret a particular test response for a particular patient, one needs to known the sensitivity and specificity of that particular response . . . rather than for some arbitrary categorical threshold.” The ratio of sensitivity for “that particular response” to 1 - specificity for “that particular response” is identical to the result-specific likelihood ratio referred to by Irwig. I therefore fully agree that result-specific likelihood ratios can be used as an aid to the Bayesian interpretation of an individual patient’s test response [l, 21. I disagree, however, with the statement that result-specific likelihood ratios “provide a clinically useful measure of test accuracy.” Because the likelihood ratio (especially the result-specific likelihood ratio) represents only one of a spectrum of possible test responses, and because the likelihood ratio does not consider the frequency of occurrence of the test response. It cannot serve as a measure of a test’s discriminant accuracy. Rather, the area under a receiver-operation characteristic curve [3,4], or
LIKELIHOOD
RATIOS
the information content [l, 21 derived from the entire distribution of responses in the diseased and non-diseased populations are more proper measures of the test’s discriminant accuracy. GEORGE A. DIAMOND Cedars -Sinai Medical Center Los Angeles, CA !20048 U.S.A.
REFERENCES Diamond GA, Forrester JS, Hirsch M, Staniloff HM,
‘. Vas R, Berman DS, Swan HJC. Application of
2,
3. 4.
conditional probability analysis to the clinical diagnosis of coronary artery disease. J Clin Invest 1980; 65: 1210-1221. Diamond GA, Hirsch M, Forrester JS, Staniloff HM, Vas R, Halpern SW, Swan HJC. Application of information theory to clinical diagnostic testing. The electrocardiographic stress test. Circulation 1981; 63: 915-921. Swets JA. Measuring the accuracy of diagnostic systems. Science 1988; 24th 1285-1289. Diamond GA. Alhrmative actions. Can the discriminant accuracy of a test be determined in the face of selection bias? Mad De& Making 1991; 11: 48-56.
Letters to the Editors THE IGNOBLE LIE
In a recent issue of this journal an editor (Alvan Feinstein) and a reviewer (Kathi Kemper) drew attention to a hoax which had been perpetrated upon them by a social worker, William Epstein [I, 21. As part of his research into the process of peer review Epstein had submitted a false manuscript, the response of the reviewer being the object of his study. Both Kemper and Feinstein discuss the ethics of this behaviour drawing parallels to the problem of consent in clinical trials but coming to slightly different
conclusions. The object of this note is to point out (1) that Epstein’s approach is unethical because it flouts a fundamental principle, (2) that any attempt at blinding subjects without agreeing the procedure beforehand is scientifically worthless, and (3) that there is one particular aspect of many clinical trials which is unethical and unscientific in a very similar way to Epstein’s research. The fundamental principle which is being flouted is Kant’s categorical imperative which
Letters to the Editors has been phrased, “always act in such a way that the principle underlying the action could be conceived as turned into a general law” [3]. I do not regard Epstein’s behaviour as unethical because Feinstein and Kemper were hoaxed (as a practical joke it might possibly be acceptable). I regard it as unethical and unscientific because this sort of research creates distrust whilst itself requiring trust to be effective. It thus debases for others the coin in which it seeks payment. This point of view is implicit in Kemper’s commentary where she states “Future research is jeopardized if potential subjects suspect they will be deceived”. If Epstein’s research is regularly repeated it will then be seen quite clearly that no advantage can be claimed by such deception which would not be available to a study with informed consent. I should like to propose two particular applications of Kant’s categorical imperative for all research involving blinding of subjects. First that the .researcher should not expect higher standards of his subjects than he employs himself. If he deceives them he deserves to be deceived by them. Secondly he should not suppose that he has greater knowledge or intelligence than his subjects. If he hopes to outwit them he deserves to be outwitted. If a researcher does not tell his subjects the truth, what right has he to expect that they are going to play fair with him and why, for that matter, should the reader believe the researcher’s account of what happened? The ethical policy to adopt in all such research is one of mutually agreed deception [4]. For this purpose one should use experiments with “open protocols and hidden allocations”. This was a point recognized a long time ago by R. A. Fisher in his famous description of a tea-tasting experiment. Fisher writes, “The subject has been told in advance of what the test will consist, namely that she will be asked to taste eight cups, that these shall be four of each kind, and that they shall be presented to her in a random order,” [5]. Although he does not cover the point explicitly, Fisher clearly rejects as inadmissible any attempt to “improve” the experiment by not informing the subject that the allocation is to be blocked in blocks of eight. It is a pity that the same acumen is not shown by certain modem-day trialists who insist on regarding the blinding as having been improved by not publishing the block size. An experimenter of lesser ability than Fisher might regard the probability of the lady’s guess-
1339
ing correctly for each cup whether tea or milk was poured in first as having been reduced from 1 in 70 (the number of possible arrangements of twice four objects of two different sorts) to 1 in 256 (the total number of ways of filling eight places with two different kinds of object) if she is not informed that there will be four cups of each kind. This argument requires him to assume, however, that the lady has not guessed that he will use a balanced allocation process rather than a completely random one. If he is not careful it is he who will end up being deceived by the lady rather than vice versa. A similar approach to Epstein’s is pointlessly used in clinical trials which employ so called placebo run-ins. Such run-ins are unethical and unscientific because they involve a deception which can only hope to be successful if the protocol is hidden from the patient. If the patient is told there will be a placebo run-in there is no point in having one. This is quite different from the open protocol blinding which is possible if the patient is told that he will be allocated placebo or verum at random. Placebo run-ins rely on a “stupid patient” argument. The investigator must assume that the patient is too stupid or ignorant to realize that clinical trials frequently have placebo runins. Presumably the protocols of such clinical trials must exclude statisticians and doctors as potential patients and of course if we had a society in which the ordinary citizen was trained in the theory of clinical trial (and why not?) they would have to exclude everybody. To return to the original context, Feinstein considers that the problem of “guarding the guardians” is serious enough to justify deception without consent [l]. Plato did, in fact, allow the noble lie, although he reserved it for a different purpose. I consider, however, that it has no part in solving Feinstein’s or any other scientific problem and I should like to recommend the following maxim as worth the serious consideration of all who carry out research on human beings. “In any experiment in which the investigator deceives without consent somebody is indeed being fooled but it is not always the subject”. STEPHEN SENN Ciba -Geigy Ltd CH 402 Basle Switzerland
1340
Letters to the Editors
REFERENCES 1. Feinstein A. Construction, consent and condemnation in research on peer review. J Clin Epidemiol 199 1; 441 339-341. 2. Kemper, K. Pride and prejudice in peer review. J Clln Epidemiol 1991; 44: 343-345.
3. Vesey G. and Foulkes P. Collins Dictionary of Philasophy. London: Collins; 1990.
4.
Freedman B. Placebo controlled trials and the logic of clinical purpose. IRB: A Review of Human Subjects Research 1990; 12: l-6. 5. Fisher R. A. The Design of Experiments; reprinted in: Bennet H, Ed. Statistical Methods, Experimental Design and Scientific Inference. Oxford University Press; 1990.
Response When Kant proposed his categorical imperative, I doubt that he thought it might be applied in research testing whether journal editors or reviewers are biased in appraising manuscripts. As I indicated in my previous comments, this type of research is very different from a clinical trial or other experimental intervention; and I feared that the act of getting the editors’ informed consent about a particular “test” manuscript would destroy the opportunity to reveal their bias. In a recent brief report of a test for “reviewer bias”, Ernst et al. [l] said that their “study would not have been possible with informed consent”. Senn’s other comments may or may not be pertinent for all clinical trials. For example, I disagree that a placebo run-in necessarily
implies that the investigators believe patients are “stupid”. Regardless of the scope, depth, and inflexibility of Senn’s concluding “maxim”, however, and regardless of the ethical (or scientific) delinquencies in “Epstein’s behaviour”, I would still like to know how to design a suitably blinded but fully consented and pragmatically feasible study of editorial bias. ALVAN R. FEINSTEIN Yale University School of Medicine 333 Cedar Street New Haven, CT 06510-8025 U.S.A.
REFERENCE 1. Ernst E, Resch KL, Uher EM. (Letter to Editors) Reviewer bias. Ann Intern Med 1992; 116: 958.
Editors’ Note-In accordance with the Journal’s custom, Dr Senn’s letter was also sent to Dr Kathi Kemper. She preferred not to respond.
