Making sense of prior probabilities in research Lex (ALB) Rutten1, Robert T. Mathie2, and Raj K. Manchanda3 1

Homeopathic physician and independent researcher, Aard 10, 4813 NN Breda, The Netherlands British Homeopathic Association, Hahnemann House, 29 Park Street West, Luton LUI 3BE, UK 3 Central Council for Research in Homoeopathy, Ministry of Health and Family Welfare, Government of India; Secretary for Research, Liga Medicorum Homoeopathica Internationalis (LMHI), 61–65 Janak Puri Institutional Area, Delhi 110058, India 2

In a recent article, Gorski and Novella state that prior probabilities can be so low that putting them to the test makes no sense [1]. A few decades ago the randomised controlled trial (RCT) was demanded because of the low prior probability of clinical methods such as homeopathy. Interestingly, the mention of prior chance and its updating using Bayes’ theorem arose after a considerable number of RCTs of homeopathy were subjected to meta-analysis; this concluded that the results were not compatible with the placebo hypothesis for homeopathy [2]. Extremely low priors, based on theory, are now proposed as sufficient reason to disregard scientific evidence. It is stated that a prior probability is infinitesimally low if it ‘violates well-established laws of physics and chemistry’. Quantum mechanics seemed at one time to violate well-established laws; it did not overturn all existing knowledge, but supplemented it. The Copernican worldview was once ridiculed: the daily turn of the world would cause a devastating wind. Nowadays homeopathy is ridiculed because sometimes ‘there is no molecule in it’, though it is now importantly recognised that nanoparticles may remain [3]. We cannot yet conclude which mechanism might explain homeopathy, but over 2000 basic research experiments indicate an effect of ultra-molecular dilutions (http://www.carstens-stiftung.de/hombrex/index.php). Prior chances must be updated by new evidence using Bayes’ theorem. This is not a one-step process, and all possible evidence should be used for sequential updating [4]. In sequential updating, the posterior odds after one piece of evidence serve as prior odds for the following calculation based on new information. Bayes’ formula is: Posterior odds ¼ likelihood ratio  prior odds


where odds = chance/(1 – chance), and chance = odds/(1 + odds). Evidence from RCTs can be translated into a likelihood ratio (LR). Rosendaal and Bouter assigned one significantly positive RCT an LR of 16 [5]. Their estimate of the prior that homeopathy ‘works’ was 1 in 106. Sequential updating after eight positive trials generates the results shown in Table 1. Very low priors are updated surprisingly quickly by Bayes’ formula [6]. In Corresponding author: Rutten, L. ([email protected]) Keywords: prior chance; homeopathy; Bayes’ theorem. 1471-4914/ ß 2014 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.molmed.2014.09.007

eight trials the probability of efficacy increased from one in a million to more than 99%. The only existing comparative review of homeopathy and conventional medicine showed similar indications of efficacy, with better quality in the homeopathy trials [7]. The funnel plot of homeopathy trials shows no difference from the comparable plot of conventional trials. Most spots on the graph indicate beneficial effects for both homeopathy and conventional medicine, and the median sample sizes are the same (65.5 vs 65.0). Gorski and Novella, however, stated that the conclusions of the review were negative or inconclusive. This was based on testing a post hoc hypothesis of different efficacy of ‘larger trials of higher methodological quality’, with unspecified subsets of eight homeopathy and six conventional medicine trials. These subsets appeared to be incomparable and highly heterogeneous [8]. The negative conclusion depended on various subjective choices. Before RCT evidence in complementary and alternative medicine (CAM) became available, this type of research was requested by opponents of CAM. Declaring now that this evidence is pseudoscientific, based only on plausibility, is suggestive of post hoc reasoning. Is there perfection in the evidence-based medicine (EBM) paradigm, based on basic science, cell culture, in vitro studies, animal models, and clinical trials? Ezzo concluded, after reviewing 160 Cochrane reviews, that 43.8% of conventional treatments were (possibly) efficacious [9]. Daily reality illustrates that CAM helps many patients after conventional medicine has failed [10]. It is unlikely that the present physicochemical model is complete; how unlikely, therefore, is a future explanation for homeopathy? A low theoretical prior is a questionable argument for rejecting CAM trials. The theory is imprecise; it is not about whether homeopathy acts in the same way as Table 1. Updating beliefs in eight steps after new evidence: the posterior chance after the first evidence is the prior chance before the second evidence, and so on [6] RCT number 1 2 3 4 5 6 7 8 a

Prior chance 0.000001 0.000016 0.000256 0.004079 0.061505 0.511856 0.943748 0.996289

Posterior chance 0.000016 0.000256 0.004079 0.061505 0.511856 0.943748 0.996289 0.999767

LR+ a 16 16 16 16 16 16 16 16

LR+, likelihood ratio for positive results. Trends in Molecular Medicine, November 2014, Vol. 20, No. 11


Letters conventional medicine, but reflects the possibility of another mechanism of action. The statement that accepting homeopathy would overthrow existing knowledge is unscientific. The conventional paradigm is incomplete, leaving room for additional mechanisms of action that could fill gaps in medicine. Patients experiencing effectiveness after conventional medicine has failed should be taken seriously. Existing RCT evidence seems to be incompatible with the placebo hypothesis, and quality bias does not explain this finding. Prior chance combined with Bayes’ theorem cannot be used as a one-step method to interpret scientific evidence because, when used correctly, it demonstrates that extremely low priors are consecutively increased by new positive evidence. To accept the possibility that homeopathy works because of an as-yet unestablished different mechanism of action is more compelling than to ignore the experience of patients, basic research, and RCT evidence. References 1 Gorski, D.H. and Novella, S.P. (2014) Clinical trial of integrative medicine: testing whether magic works. Trends Mol. Med. 20, 473–476

Trends in Molecular Medicine November 2014, Vol. 20, No. 11

2 Vandenbroucke, J.P. and de Craen, A.J.M. (2001) Alternative medicine: a ‘mirror image’ for scientific reasoning in conventional medicine. Ann. Intern. Med. 135, 507–513 3 Bell, I.R. and Schwartz, G.E. (2013) Adaptive network nanomedicine: an integrated model for homeopathic medicine. Front. Biosci. (Schol. Ed.) S5, 685–708 4 Gill, C.J. et al. (2005) Why clinicians are natural bayesians. BMJ 330, 1080–1083 5 Rosendaal, F.R. and Bouter, L.M. (2002) Errors in methodology (conclusion). XXXIX. The ultimate truth. Ned. Tijdschr. Geneeskd. 146, 304–309 (article in Dutch) 6 Rutten, A.L.B. (2008) How can we change beliefs? A Bayesian perspective. Homeopathy 97, 214–219 7 Shang, A. et al. (2005) Are the clinical effects of homeopathy placebo effects? Comparative study of placebo-controlled trials of homeopathy and allopathy. Lancet 366, 726–732 8 Lu¨dtke, R. and Rutten, A.L. (2008) The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials. J. Clin. Epidemiol. 61, 1197–1204 9 Ezzo, J. et al. (2001) Reviewing the reviews. How strong is the evidence? How clear are the conclusions? Int. J. Technol. Assess. Health Care 17, 457–466 10 Marian, F. et al. (2008) Patient satisfaction and side effects in primary care: an observational study comparing homeopathy and conventional medicine. BMC Complement. Altern. Med. 8, 52

Considering prior plausibility in clinical trials does not mean ignoring scientific evidence David H. Gorski1,2 and Steven P. Novella3 1

Michael and Marian Illitch Department of Surgery, Wayne State University School of Medicine, 3990 John R St., Detroit, MI 48201, USA 2 Molecular Therapeutics Program, Wayne State University, Barbara Ann Karmanos Cancer Institute, 4100 John R St., Detroit, MI 48201, USA 3 Department of Neurology, Yale University, 40 Temple St., Suite 6C, New Haven CT 06510, USA

The response of Rutten et al. to our recent article [1], in which we contend that it is unscientific and unethical to test highly implausible treatments such as homeopathy and reiki in randomized clinical trials (RCTs), represents a common and misguided complaint by advocates of alternative medicine against interpreting clinical trials through the lens of prior plausibility. It is a complaint that is, from a scientific viewpoint, unjustified and relies on a misunderstanding of the history of science, a straw man characterization of arguments for science-based medicine, and an incorrect interpretation of Bayes’ theorem applied to clinical trials of homeopathy. They mischaracterize our core argument as stating that ‘extremely low priors, based on theory, are now proposed as sufficient reason to disregard scientific evidence’; yet nowhere did we write that prior plausibility alone is sufficient reason to disregard scientific evidence. Rather, we argued that prior plausibility, when Corresponding author: Gorski, D.H. ([email protected]) Keywords: evidence-based medicine; science-based medicine; clinical trials; complementary and alternative medicine; integrative medicine. 1471-4914/ ß 2014 Published by Elsevier Ltd. http://dx.doi.org/10.1016/j.molmed.2014.10.001


so low as to be indistinguishable from zero (as is the case for homeopathy), can be sufficient reason to conclude that clinical trials are so unlikely to be informative that carrying out such trials would be both unethical and scientifically unproductive. The extremely low prior probability of homeopathy is not based solely on theory, as implied by Rutten et al. [1]. A scientific theory, after all, is not merely a hunch, but rather ‘a well-substantiated explanation of some aspect of the natural world that can incorporate facts, laws, inferences, and tested hypotheses’ (http://www.nap.edu/openbook.php?record_id=6024&page=2). A theory, such as the theory of relativity or evolution, is accepted by the community of relevant scientists based on overwhelming observational and experimental evidence supporting it. Contrary to how it is portrayed by Rutten et al., quantum theory was necessary to explain compelling observations that the then-current theory could not account for. Currently, there are no such compelling observations that can only be explained by homeopathy. Moreover, the underpinnings of physics and chemistry, in both theory and evidence, that conclude that homeopathy is about as close to impossible as one can imagine are so well established, rooted in 200 years

Making sense of prior probabilities in research.

Making sense of prior probabilities in research. - PDF Download Free
238KB Sizes 0 Downloads 5 Views