Opinion
VIEWPOINT
John P. A. Ioannidis, MD, DSc Stanford Prevention Research Center (SPRC) and MetaResearch Innovation Center at Stanford (METRICS), Stanford, California.
Author Reading at jama.com
Corresponding Author: John P.A. Ioannidis, MD, DSc, Stanford Prevention Research Center (SPRC) and Meta-Research Innovation Center at Stanford (METRICS), 1265 Welch Rd, Medical School Office Bldg, Room X306, Stanford, CA 94305 (jioannid @stanford.edu).
Stealth Research Is Biomedical Innovation Happening Outside the Peer-Reviewed Literature? of venipuncture.5 Several patents have been filed and approved. A search in the JUSTIA patent database using Theranos as a search term yielded 71 items retrieved as of January 5, 2015.6 However, it is practically impossible to judge the validity of the science based only on patents with titles such as “Methods and Systems for Assessing Clinical Outcomes.” Theranos is just one example among many for which major efforts and major claims about biomedical progress seem to be happening outside the peer-reviewed scientific literature. Many of these efforts and claims have a biotechnology flavor, and the people involved often include a blend of engineers, physical scientists, and venture capitalists. The main motive appears to be to develop products and services, rather than report new discoveries as research scholarship. Products, services, and profit appear to be more important than scientific publications. For some biotechnology innovators, peerreviewed publication may be perceived as unnecessary. Publication makes knowledge public and communal, whereas the efforts of some companies build on the spirit of private ultracompetitive entrepreneurship. At a time of increasing recognition of the importance of transparency in laboratory and clinical research, this approach seems paradoxical. The path to publication also involves peer review; for disruptive innoHowever, stealth research creates total vators, this may be perceived as an ordeal because they would have to respond ambiguity about what evidence can be to and satisfy the concerns and potentrusted in a mix of possibly brilliant ideas, tial biases of mainstream reviewers who represent the very core traditions that aggressive corporate announcements, the innovation specifically aims to disand mass media hype. rupt. Some evidence suggests that has operated in stealth mode for more than a decade, peer reviewers can be hostile to innovation.7,8 If so, not publishing anything in the literature while prepar- circumventing the peer review process and maintaining to change the entire health system: “One closely ing new discoveries in a stealth mode until there is guarded secret is… how exactly the technology confidence that it will be possible to disrupt the status behind its blood test works.”1 quo effectively appears reasonable and perhaps even At a TEDMED talk on the web,5 the founder and CEO justifiable. However, stealth research creates total ambiguity advocates that all people should be able to order their own diagnostic tests—as many as they wish and whenever and about what evidence can be trusted in a mix of possibly as often as they wish—so as to not die of a disease diag- brilliant ideas, aggressive corporate announcements, and nosed too late. Choice and human rights are evoked as mass media hype. The unquestionable success of comsupportingargumentsforthisstatement.Thereisnomen- puter science, engineering, and social media technolotion of overdiagnosis, false-positive findings, or the po- gies has created reasonable hope that these technolotential for escalation of iatrogenic disease secondary to gies can also improve health in ways that the biomedical misplaced and perhaps overly zealous diagnostic and and life sciences have failed to do until now. But then how screening efforts. Other less contestable and clearly wor- can the validity of the claims made be assessed, if the thy goals mentioned are lowering testing cost, improv- evidence is not within reach of other scientists to evaluing access to diagnostic testing, and diminishing the pain ate and scrutinize?
Information about Theranos, a privately held biotechnology company that has developed novel approaches for laboratory diagnostic testing, has appeared in The Wall Street Journal, Business Insider, San Francisco Business Times, Fortune, Forbes, Medscape, and Silicon Valley Business Journal—but not in the peer-reviewed biomedical literature. As of January 5, 2015, a search in PubMed using Theranos as a search term identified affiliations for only 2 unrelated articles coauthored by Theranos Inc employees, although these 2 reports do not offer insights about their company. Conversely, according to the non–peer-reviewed sources mentioned above, Theranos is “revolutionizing the blood test” and this “is a golden idea”: “the company can run hundreds of tests on a drop of blood far more quickly than could be done with whole vials in the past— and it costs a lot less.”1 The company is estimated to be worth $9 billion.2 Test results are “near-instantaneous.”3 Moreover, the company has teamed with Walgreens pharmacies in Palo Alto and Arizona to create “Theranos Wellness Centers.” A footnote in the respective Walgreens webpage4 mentions that the laboratories are Clinical Laboratory Improvement Act– certified. According to the same sources, Theranos
jama.com
(Reprinted) JAMA February 17, 2015 Volume 313, Number 7
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a University of Leeds User on 05/17/2015
663
Opinion Viewpoint
The concept of stealth research is not necessarily new. Influential classified scientific achievements such as the nuclear bomb or aspects of satellite technology were, for military reasons, developed without communicating each and every step to the wider scientific community. Do solutions to health and disease problems require a similar stealth operation? A problem with such a secretive approach is that even for successful, influential ideas, it is impossible to discern eventually whether the success of those ideas resulted from better science or simply a better financial or advertising model. As an illustration, for laboratory testing, most of the diagnostic cost in the United States is attributable to overhead and indirect personnel costs rather than the cost of the technology, and the amount of money charged can be vastly greater than the direct cost. If a company can diminish charges with a different operational model, this may be useful for health care, even if there has been no progress in the basic and applied science of laboratory testing per se. Without knowing why and how something worked, when it does (seem to) work, it will not be possible to design rationally the next steps for developing even better technologies and strategies and for achieving better patient outcomes. To solve this conundrum, it may be necessary to find ways to realign the reward system for innovation.9 One possibility is to make the scientific literature more receptive to innovators. This could include models in which reports of disruptive discoveries that are in dissonance with the mainstream can still be communicated as preprints without prior peer review, perhaps in the same way as the successful example of arXiv in the physical sciences, which has now reached 1 million e-print articles.10 That there has been no peer review of these initial reports should be transparent to researchers and the public. This would permit public discourse ARTICLE INFORMATION Conflict of Interest Disclosures: The author has completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. REFERENCES 1. Loria K. This Woman’s Revolutionary Idea Made Her a Billionaire—and Could Change Medicine. Business Insider website. http://www.businessinsider .com/theranos-founder-elizabeth-holmes-is-a -billionaire-2014-9#ixzz3JNgdYNBQ. September 29, 2014. Accessed December 22, 2014. 2. Topol EJ. Creative Disruption? She’s 29 and Set to Reboot Lab Medicine. Medscape website.
664
about the innovation. Only a small fraction of these claims may survive open scrutiny, because major innovations are rare. Preprint communication could primarily help innovators who want academic and scientific credit for their bold ideas. For innovators who primarily seek financial success, publications alone would be less important. Thus, some better regulatory process is needed so that innovative ideas for financially successful applications can be scrutinized by the wider scientific community as to their validity. A company should not be forced to disclose its science secrets in detail, especially while its efforts are still exploratory trial-anderror and while creating basic elements for its products and services. However, if a product or service reaches the point at which it generates substantial revenue, the science behind it should then be communicated in detail to ensure adequate review. Additional incentives also could be established for more adequate communication. For example, communication of the scientific information may be used as a prerequisite for extending patent protection. A company also may build its further communication and marketing strategy using as an extra asset the fact that it had its methods and technologies scrutinized and validated by independent scientists. Both the scientific and corporate communities and their mass mediaoutletscanpromotethenotionthatindependentvalidationprovides powerful testimony for the real worth of a technology. Biomedical innovation and discovery based on research and development by private and public companies and institutions are essential for advancing medical science and improving clinical care. However, unless stealth research adopts more scientific transparency, investors, physicians, patients, and healthy people will not be able to judge whether some proposed innovation is worth $9 billion, $900 billion, or just $9—let alone if the innovation will improve the health and well-being of individuals.
http://www.medscape.com/viewarticle/814233. November 18, 2013. Accessed December 22, 2014. 3. Parloff R. This CEO is out for blood. Fortune. June 12, 2014. http://fortune.com/2014/06/12/theranos -blood-holmes/. Accessed December 22, 2014. 4. The Blood Tests That Need Just a Tiny Sample: Walgreens Partners With Theranos to Provide Lab Services. Walgreens website. http://www.walgreens .com/pharmacy/lab-testing/home.jsp. Accessed November 17, 2014.
7. Armstrong JS. Peer review for journals: evidence on quality control, fairness, and innovation. Sci Eng Ethics. 1997;3:63-84. 8. Horrobin DF. The philosophical basis of peer review and the suppression of innovation. JAMA. 1990;263(10):1438-1441. 9. Ioannidis JPA. How to make more published research true. PLoS Med. 2014;11(10):e1001747. 10. arXiv.org website. http://arxiv.org/. Accessed November 17, 2014.
5. TEDMED Speakers: Elizabeth Holmes. TEDMED website. http://www.tedmed.com/speakers/show ?id=308981. Accessed November 17, 2014. 6. JUSTIA Patents website. http://patents.justia.com /search?q=theranos. Accessed January 5, 2015.
JAMA February 17, 2015 Volume 313, Number 7 (Reprinted)
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a University of Leeds User on 05/17/2015
jama.com
VIEWPOINT
John P. A. Ioannidis, MD, DSc Stanford Prevention Research Center (SPRC) and MetaResearch Innovation Center at Stanford (METRICS), Stanford, California.
Author Reading at jama.com
Corresponding Author: John P.A. Ioannidis, MD, DSc, Stanford Prevention Research Center (SPRC) and Meta-Research Innovation Center at Stanford (METRICS), 1265 Welch Rd, Medical School Office Bldg, Room X306, Stanford, CA 94305 (jioannid @stanford.edu).
Stealth Research Is Biomedical Innovation Happening Outside the Peer-Reviewed Literature? of venipuncture.5 Several patents have been filed and approved. A search in the JUSTIA patent database using Theranos as a search term yielded 71 items retrieved as of January 5, 2015.6 However, it is practically impossible to judge the validity of the science based only on patents with titles such as “Methods and Systems for Assessing Clinical Outcomes.” Theranos is just one example among many for which major efforts and major claims about biomedical progress seem to be happening outside the peer-reviewed scientific literature. Many of these efforts and claims have a biotechnology flavor, and the people involved often include a blend of engineers, physical scientists, and venture capitalists. The main motive appears to be to develop products and services, rather than report new discoveries as research scholarship. Products, services, and profit appear to be more important than scientific publications. For some biotechnology innovators, peerreviewed publication may be perceived as unnecessary. Publication makes knowledge public and communal, whereas the efforts of some companies build on the spirit of private ultracompetitive entrepreneurship. At a time of increasing recognition of the importance of transparency in laboratory and clinical research, this approach seems paradoxical. The path to publication also involves peer review; for disruptive innoHowever, stealth research creates total vators, this may be perceived as an ordeal because they would have to respond ambiguity about what evidence can be to and satisfy the concerns and potentrusted in a mix of possibly brilliant ideas, tial biases of mainstream reviewers who represent the very core traditions that aggressive corporate announcements, the innovation specifically aims to disand mass media hype. rupt. Some evidence suggests that has operated in stealth mode for more than a decade, peer reviewers can be hostile to innovation.7,8 If so, not publishing anything in the literature while prepar- circumventing the peer review process and maintaining to change the entire health system: “One closely ing new discoveries in a stealth mode until there is guarded secret is… how exactly the technology confidence that it will be possible to disrupt the status behind its blood test works.”1 quo effectively appears reasonable and perhaps even At a TEDMED talk on the web,5 the founder and CEO justifiable. However, stealth research creates total ambiguity advocates that all people should be able to order their own diagnostic tests—as many as they wish and whenever and about what evidence can be trusted in a mix of possibly as often as they wish—so as to not die of a disease diag- brilliant ideas, aggressive corporate announcements, and nosed too late. Choice and human rights are evoked as mass media hype. The unquestionable success of comsupportingargumentsforthisstatement.Thereisnomen- puter science, engineering, and social media technolotion of overdiagnosis, false-positive findings, or the po- gies has created reasonable hope that these technolotential for escalation of iatrogenic disease secondary to gies can also improve health in ways that the biomedical misplaced and perhaps overly zealous diagnostic and and life sciences have failed to do until now. But then how screening efforts. Other less contestable and clearly wor- can the validity of the claims made be assessed, if the thy goals mentioned are lowering testing cost, improv- evidence is not within reach of other scientists to evaluing access to diagnostic testing, and diminishing the pain ate and scrutinize?
Information about Theranos, a privately held biotechnology company that has developed novel approaches for laboratory diagnostic testing, has appeared in The Wall Street Journal, Business Insider, San Francisco Business Times, Fortune, Forbes, Medscape, and Silicon Valley Business Journal—but not in the peer-reviewed biomedical literature. As of January 5, 2015, a search in PubMed using Theranos as a search term identified affiliations for only 2 unrelated articles coauthored by Theranos Inc employees, although these 2 reports do not offer insights about their company. Conversely, according to the non–peer-reviewed sources mentioned above, Theranos is “revolutionizing the blood test” and this “is a golden idea”: “the company can run hundreds of tests on a drop of blood far more quickly than could be done with whole vials in the past— and it costs a lot less.”1 The company is estimated to be worth $9 billion.2 Test results are “near-instantaneous.”3 Moreover, the company has teamed with Walgreens pharmacies in Palo Alto and Arizona to create “Theranos Wellness Centers.” A footnote in the respective Walgreens webpage4 mentions that the laboratories are Clinical Laboratory Improvement Act– certified. According to the same sources, Theranos
jama.com
(Reprinted) JAMA February 17, 2015 Volume 313, Number 7
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a University of Leeds User on 05/17/2015
663
Opinion Viewpoint
The concept of stealth research is not necessarily new. Influential classified scientific achievements such as the nuclear bomb or aspects of satellite technology were, for military reasons, developed without communicating each and every step to the wider scientific community. Do solutions to health and disease problems require a similar stealth operation? A problem with such a secretive approach is that even for successful, influential ideas, it is impossible to discern eventually whether the success of those ideas resulted from better science or simply a better financial or advertising model. As an illustration, for laboratory testing, most of the diagnostic cost in the United States is attributable to overhead and indirect personnel costs rather than the cost of the technology, and the amount of money charged can be vastly greater than the direct cost. If a company can diminish charges with a different operational model, this may be useful for health care, even if there has been no progress in the basic and applied science of laboratory testing per se. Without knowing why and how something worked, when it does (seem to) work, it will not be possible to design rationally the next steps for developing even better technologies and strategies and for achieving better patient outcomes. To solve this conundrum, it may be necessary to find ways to realign the reward system for innovation.9 One possibility is to make the scientific literature more receptive to innovators. This could include models in which reports of disruptive discoveries that are in dissonance with the mainstream can still be communicated as preprints without prior peer review, perhaps in the same way as the successful example of arXiv in the physical sciences, which has now reached 1 million e-print articles.10 That there has been no peer review of these initial reports should be transparent to researchers and the public. This would permit public discourse ARTICLE INFORMATION Conflict of Interest Disclosures: The author has completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. REFERENCES 1. Loria K. This Woman’s Revolutionary Idea Made Her a Billionaire—and Could Change Medicine. Business Insider website. http://www.businessinsider .com/theranos-founder-elizabeth-holmes-is-a -billionaire-2014-9#ixzz3JNgdYNBQ. September 29, 2014. Accessed December 22, 2014. 2. Topol EJ. Creative Disruption? She’s 29 and Set to Reboot Lab Medicine. Medscape website.
664
about the innovation. Only a small fraction of these claims may survive open scrutiny, because major innovations are rare. Preprint communication could primarily help innovators who want academic and scientific credit for their bold ideas. For innovators who primarily seek financial success, publications alone would be less important. Thus, some better regulatory process is needed so that innovative ideas for financially successful applications can be scrutinized by the wider scientific community as to their validity. A company should not be forced to disclose its science secrets in detail, especially while its efforts are still exploratory trial-anderror and while creating basic elements for its products and services. However, if a product or service reaches the point at which it generates substantial revenue, the science behind it should then be communicated in detail to ensure adequate review. Additional incentives also could be established for more adequate communication. For example, communication of the scientific information may be used as a prerequisite for extending patent protection. A company also may build its further communication and marketing strategy using as an extra asset the fact that it had its methods and technologies scrutinized and validated by independent scientists. Both the scientific and corporate communities and their mass mediaoutletscanpromotethenotionthatindependentvalidationprovides powerful testimony for the real worth of a technology. Biomedical innovation and discovery based on research and development by private and public companies and institutions are essential for advancing medical science and improving clinical care. However, unless stealth research adopts more scientific transparency, investors, physicians, patients, and healthy people will not be able to judge whether some proposed innovation is worth $9 billion, $900 billion, or just $9—let alone if the innovation will improve the health and well-being of individuals.
http://www.medscape.com/viewarticle/814233. November 18, 2013. Accessed December 22, 2014. 3. Parloff R. This CEO is out for blood. Fortune. June 12, 2014. http://fortune.com/2014/06/12/theranos -blood-holmes/. Accessed December 22, 2014. 4. The Blood Tests That Need Just a Tiny Sample: Walgreens Partners With Theranos to Provide Lab Services. Walgreens website. http://www.walgreens .com/pharmacy/lab-testing/home.jsp. Accessed November 17, 2014.
7. Armstrong JS. Peer review for journals: evidence on quality control, fairness, and innovation. Sci Eng Ethics. 1997;3:63-84. 8. Horrobin DF. The philosophical basis of peer review and the suppression of innovation. JAMA. 1990;263(10):1438-1441. 9. Ioannidis JPA. How to make more published research true. PLoS Med. 2014;11(10):e1001747. 10. arXiv.org website. http://arxiv.org/. Accessed November 17, 2014.
5. TEDMED Speakers: Elizabeth Holmes. TEDMED website. http://www.tedmed.com/speakers/show ?id=308981. Accessed November 17, 2014. 6. JUSTIA Patents website. http://patents.justia.com /search?q=theranos. Accessed January 5, 2015.
JAMA February 17, 2015 Volume 313, Number 7 (Reprinted)
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://jama.jamanetwork.com/ by a University of Leeds User on 05/17/2015
jama.com
