COMMENTARY
Puppies On The Doorstep
All sins have their origin in a sense of inferiority otherwise called ambition.—Cesare Pavase
A
basket of puppies on your doorstep would likely trigger a biphasic set of feelings. Initially, they are cute and cuddly, and they cause oxytocin release. Then they grow up. And they destroy the house; bark all night; eat all your food; shed hair; and require vaccinations, licensing, and long walks with plastic bags. They cost unbudgeted time, effort, and money. Similarly, most people initially cotton to rules designed to create justice. As recently as the mid-1990s, clinical trial research was largely done on the honor system. Researchers were given the presumption of honesty about their endpoints. Almost no one asked to see original copies of IRB filings, to verify if the endpoints reported were what the trial intended. About the only place where protocols were filed and then revisited were at the FDA, where research aims (or endpoints) are proposed in advance to support marketing claims. In other words, industry concerns and their research teams would bring a protocol to FDA for premarket review, ostensibly to get some idea of whether the trial would support a particular labeling of a drug or device. Outside of the FDA, journal editors, reviewers, and readers largely had to trust the researchers. But then too many investigators were accused of surreptitiously moving the target. In one notorious case, a researcher studying the role of antibiotics in treating otitis media was accused of changing the time for the main endpoint from 8 weeks to 4 weeks. The investigator had received funding from the National Institutes of Health (NIH), as well as hundreds of thousands of dollars from companies that manufacture antibiotics. The result of this event, known by some in the field as the “Bluestone affair,” led to hearings by three University of Pittsburgh committees, three panels of the NIH, a congressional subcommittee, a federal district court, and the U.S. Court of Appeals. Whether or not the researcher did anything wrong remains a matter of opinion, but that case, and others like it, led to a proverbial Act of Congress that established the trial registration system, clinicaltrials.gov. The legislation states:
The author has no relevant financial information or potential conflicts of interest to disclose.
354 354
ISSN 1069-6563 354 PII ISSN 1069-6563583
Section 113 of FDAMA required that the National Institutes of Health (NIH) create a public information resource on certain clinical trials regulated by the Food and Drug Administration (FDA). Specifically, FDAMA 113 required that the registry include information about federally or privately funded clinical trials conducted under investigational new drug applications (INDs) to test the effectiveness of experimental drugs for patients with serious or life-threatening diseases or condition.1 Other registry systems exist, but ClinicalTrials.gov is the oldest and largest and the only registry that, in theory, can lead to federal prosecution of investigators for noncompliance. The World Health Organization has a voluntary registry platform, the International Clinical Trials Registry Platform (ICTRP), which defines a clinical trial as: For the purposes of registration, a clinical trial is any research study that prospectively assigns human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes. Clinical trials may also be referred to as interventional trials. Interventions include but are not restricted to drugs, cells and other biological products, surgical procedures, radiologic procedures, devices, behavioral treatments, process-of-care changes, preventive care, etc. This definition includes Phase I to Phase IV trials.2 Similarly, ClinicalTrials.gov defines its subject matter as follows: A clinical trial is a research study in which human volunteers are assigned to interventions (for example, a medical product, behavior, or procedure) based on a protocol (or plan) and are then evaluated for effects on biomedical or health outcomes. ClinicalTrials.gov also includes records describing observational studies and programs providing access to investigational drugs outside of clinical trials (expanded access). The original intent of ClinicalTrials.gov was to ensure integrity, transparency, and honesty in clinical trial reporting to the FDA. The key here is the latter prepositional phrase. A nonvolatile, curated registry was and is needed to ensure that investigators, possibly financially conflicted, and under pressure of reciprocity, do not modify trial endpoints to fit the data. In addition, ClinicalTrials.gov was meant to serve as a
© 2015 by the Society for Academic Emergency Medicine doi: 10.1111/acem.12594
ACADEMIC EMERGENCY MEDICINE • March 2015, Vol. 22, No. 3 • www.aemj.org
site for consumers and patients to locate trials about specific disease processes. However, at the same time, the federal government delivered puppies to the doorstep of the non–industry-funded independent researcher. ClinicalTrials.gov began life as cuddly as a basket of puppies, engendering warm feelings. Analogously to puppies growing up, ClinicalTrials.gov has morphed from a steward of information into a complex, time-wasting, obtuse system of queries, quality checks, and illogical requests from the curators. Especially at the time of study closure, the process becomes an odious, painstaking, frustrating, and pointless exercise in deciphering a poorly designed and inflexible data entry system. Responding to curator inquiries requires unbudgeted time, and assuming that time equals cost, this represents an unfunded mandate. The main negative consequence to public health is that this process stifles innovation of investigator-initiated research. Unlike industry, independent researchers cannot charge more for their product to account for governmental mandates. It cannot be reasonably argued that the administrative costs for this process are covered by indirect costs on grants. Here is a typical request for a registered study of diagnostic accuracy that had no arms to it and the outcome was a diagnosis (NCT00368836): Specifically, the Arm Titles and Number of Participants Analyzed need to be revised in Baseline Characteristics to avoid double-counting participants, which is not permitted in the Baseline Characteristics module. The Unit of Measure in Outcome Measures 1 is invalid and needs to be updated, as appropriate. Outcome Measure 2 is vague and additional information is needed in the Measure Description to clearly understand what was assessed and reported in the data table.3 This particular request was received on October 15, 2014, for an investigator-initiated, single-arm study of diagnostic accuracy of a device that did not have an IDE. The study was closed in September 2008. In addition to its administrative burden, the information placed on ClinicalTrials.gov breeds animosity in peer review, promoting a game of “gotcha.” Too often, reviewers use information on the ClinicalTrials.gov website out of context to make assumptions. This leads to spurious charges of inconsistency in what is reported in a manuscript versus what was listed on the registry. For example, a clinical trial of a test article (a drug or device) may only upload the primary functional endpoint, such as the 6-minute walk test for the purpose of a drug label. However, the original application read by the IRB, and possibly peer reviewers at funding agencies, may clearly state the scientific hypothesis that the test article should affect a biomarker, known to reflect the severity of the illness, but not have any particular patient-oriented importance. Indeed, the biomarker may have been the primary scientific basis for funding. For this reason, the biomarker may be left off of the website, because after
355
all, the purpose of the registry was to establish the primary outcome for public to see, not to post the entire protocol. With the clout of the federal government, bolstered by the pursuit of social sanctity, the International Committee of Medical Journal Editors (ICMJE) recommends “. . . that all medical journal editors require, registration of clinical trials in a public trials registry at or before the time of first patient enrollment as a condition of consideration for publication.”4 Therefore, most respected peer-reviewed clinical journals have installed roadblocks in their submission processes that mandate a registration number for prospective studies of human subjects. Some have gone as far as to mandate registration for any prospective study, even in the absence of an intervention. The process has led to the risible consequence of investigators “registering” their studies post hoc, on the same day of submission, solely to obtain an NCT number to satisfy the requirement for manuscript submission (traverse to the ICMJE statement). Here again, the puppies metaphor barks again. Journals had the intention of integrity, but ended up with obstructive bureaucracy for many authors. The solution lies in a pragmatic approach that is being taken by Academic Emergency Medicine. First, journals should abide by the law as specified in section 114 of the FDAMA. That part is easy. Therefore, any study that satisfies the ClinicalTrials.gov definition of a clinical trial (described above) and has an IDE or IND, or federal funding, must be preregistered. Second, any randomized controlled trial of a commercially available drug or device, funded by the industry concern, must be preregistered. For investigator-initiated research without an IND, IDE, or federal funding, including randomized trials, a reasonable and prudent approach is warranted. Academic Emergency Medicine will continue to promulgate policy that encourages preregistration for such trials, but going forward, will not set a hardline mandate for registration of studies that meet the ClinicalTrials.gov definition, but do not have an IND, IDE, or federal funding. Most importantly, Academic Emergency Medicine will not require or even allow post hoc registration. Too often, the masses of the innocent are punished by the bad deeds of a few. It is time to correct this overcorrection. Jeffrey A. Kline, MD ([email protected]) Senior Associate Editor Academic Emergency Medicine Emergency Medicine and Physiology Indiana University School of Medicine Indianapolis, IN
Supervising Editor: David C. Cone, MD.
356
References 1. U.S. Food and Drug Administration. Regulatory Information. Full text of FDAMA Law. Section 113. Information Program on Clinical Trials for Serious or Life-threatening Diseases. Available at: http://www. fda.gov/RegulatoryInformation/Legislation/FederalFood DrugandCosmeticActFDCAct/SignificantAmendment stotheFDCAct/FDAMA/FullTextofFDAMAlaw/default. htm#SEC.%20113. Accessed Dec 1, 2014. 2. World Health Organization. International Clinical Trials Registry Platform (ICTRP). Available at:
Kline • PUPPIES ON THE DOORSTEP
http://www.who.int/ictrp/en/. Accessed Nov 27, 2014. 3. National Institutes of Health. ClinicalTrials.gov: History, Policy, and Laws. Available at: http://www.clinicaltrials.gov/ct2/about-site/history. Accessed Nov 27, 2014. 4. International Committee of Medical Journal Editors. Clinical Trial Registration. Available at: http:// www.icmje.org/recommendations/browse/publishingand-editorial-issues/clinical-trial-registration.html. Accessed Nov 27, 2014.
Puppies On The Doorstep
All sins have their origin in a sense of inferiority otherwise called ambition.—Cesare Pavase
A
basket of puppies on your doorstep would likely trigger a biphasic set of feelings. Initially, they are cute and cuddly, and they cause oxytocin release. Then they grow up. And they destroy the house; bark all night; eat all your food; shed hair; and require vaccinations, licensing, and long walks with plastic bags. They cost unbudgeted time, effort, and money. Similarly, most people initially cotton to rules designed to create justice. As recently as the mid-1990s, clinical trial research was largely done on the honor system. Researchers were given the presumption of honesty about their endpoints. Almost no one asked to see original copies of IRB filings, to verify if the endpoints reported were what the trial intended. About the only place where protocols were filed and then revisited were at the FDA, where research aims (or endpoints) are proposed in advance to support marketing claims. In other words, industry concerns and their research teams would bring a protocol to FDA for premarket review, ostensibly to get some idea of whether the trial would support a particular labeling of a drug or device. Outside of the FDA, journal editors, reviewers, and readers largely had to trust the researchers. But then too many investigators were accused of surreptitiously moving the target. In one notorious case, a researcher studying the role of antibiotics in treating otitis media was accused of changing the time for the main endpoint from 8 weeks to 4 weeks. The investigator had received funding from the National Institutes of Health (NIH), as well as hundreds of thousands of dollars from companies that manufacture antibiotics. The result of this event, known by some in the field as the “Bluestone affair,” led to hearings by three University of Pittsburgh committees, three panels of the NIH, a congressional subcommittee, a federal district court, and the U.S. Court of Appeals. Whether or not the researcher did anything wrong remains a matter of opinion, but that case, and others like it, led to a proverbial Act of Congress that established the trial registration system, clinicaltrials.gov. The legislation states:
The author has no relevant financial information or potential conflicts of interest to disclose.
354 354
ISSN 1069-6563 354 PII ISSN 1069-6563583
Section 113 of FDAMA required that the National Institutes of Health (NIH) create a public information resource on certain clinical trials regulated by the Food and Drug Administration (FDA). Specifically, FDAMA 113 required that the registry include information about federally or privately funded clinical trials conducted under investigational new drug applications (INDs) to test the effectiveness of experimental drugs for patients with serious or life-threatening diseases or condition.1 Other registry systems exist, but ClinicalTrials.gov is the oldest and largest and the only registry that, in theory, can lead to federal prosecution of investigators for noncompliance. The World Health Organization has a voluntary registry platform, the International Clinical Trials Registry Platform (ICTRP), which defines a clinical trial as: For the purposes of registration, a clinical trial is any research study that prospectively assigns human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes. Clinical trials may also be referred to as interventional trials. Interventions include but are not restricted to drugs, cells and other biological products, surgical procedures, radiologic procedures, devices, behavioral treatments, process-of-care changes, preventive care, etc. This definition includes Phase I to Phase IV trials.2 Similarly, ClinicalTrials.gov defines its subject matter as follows: A clinical trial is a research study in which human volunteers are assigned to interventions (for example, a medical product, behavior, or procedure) based on a protocol (or plan) and are then evaluated for effects on biomedical or health outcomes. ClinicalTrials.gov also includes records describing observational studies and programs providing access to investigational drugs outside of clinical trials (expanded access). The original intent of ClinicalTrials.gov was to ensure integrity, transparency, and honesty in clinical trial reporting to the FDA. The key here is the latter prepositional phrase. A nonvolatile, curated registry was and is needed to ensure that investigators, possibly financially conflicted, and under pressure of reciprocity, do not modify trial endpoints to fit the data. In addition, ClinicalTrials.gov was meant to serve as a
© 2015 by the Society for Academic Emergency Medicine doi: 10.1111/acem.12594
ACADEMIC EMERGENCY MEDICINE • March 2015, Vol. 22, No. 3 • www.aemj.org
site for consumers and patients to locate trials about specific disease processes. However, at the same time, the federal government delivered puppies to the doorstep of the non–industry-funded independent researcher. ClinicalTrials.gov began life as cuddly as a basket of puppies, engendering warm feelings. Analogously to puppies growing up, ClinicalTrials.gov has morphed from a steward of information into a complex, time-wasting, obtuse system of queries, quality checks, and illogical requests from the curators. Especially at the time of study closure, the process becomes an odious, painstaking, frustrating, and pointless exercise in deciphering a poorly designed and inflexible data entry system. Responding to curator inquiries requires unbudgeted time, and assuming that time equals cost, this represents an unfunded mandate. The main negative consequence to public health is that this process stifles innovation of investigator-initiated research. Unlike industry, independent researchers cannot charge more for their product to account for governmental mandates. It cannot be reasonably argued that the administrative costs for this process are covered by indirect costs on grants. Here is a typical request for a registered study of diagnostic accuracy that had no arms to it and the outcome was a diagnosis (NCT00368836): Specifically, the Arm Titles and Number of Participants Analyzed need to be revised in Baseline Characteristics to avoid double-counting participants, which is not permitted in the Baseline Characteristics module. The Unit of Measure in Outcome Measures 1 is invalid and needs to be updated, as appropriate. Outcome Measure 2 is vague and additional information is needed in the Measure Description to clearly understand what was assessed and reported in the data table.3 This particular request was received on October 15, 2014, for an investigator-initiated, single-arm study of diagnostic accuracy of a device that did not have an IDE. The study was closed in September 2008. In addition to its administrative burden, the information placed on ClinicalTrials.gov breeds animosity in peer review, promoting a game of “gotcha.” Too often, reviewers use information on the ClinicalTrials.gov website out of context to make assumptions. This leads to spurious charges of inconsistency in what is reported in a manuscript versus what was listed on the registry. For example, a clinical trial of a test article (a drug or device) may only upload the primary functional endpoint, such as the 6-minute walk test for the purpose of a drug label. However, the original application read by the IRB, and possibly peer reviewers at funding agencies, may clearly state the scientific hypothesis that the test article should affect a biomarker, known to reflect the severity of the illness, but not have any particular patient-oriented importance. Indeed, the biomarker may have been the primary scientific basis for funding. For this reason, the biomarker may be left off of the website, because after
355
all, the purpose of the registry was to establish the primary outcome for public to see, not to post the entire protocol. With the clout of the federal government, bolstered by the pursuit of social sanctity, the International Committee of Medical Journal Editors (ICMJE) recommends “. . . that all medical journal editors require, registration of clinical trials in a public trials registry at or before the time of first patient enrollment as a condition of consideration for publication.”4 Therefore, most respected peer-reviewed clinical journals have installed roadblocks in their submission processes that mandate a registration number for prospective studies of human subjects. Some have gone as far as to mandate registration for any prospective study, even in the absence of an intervention. The process has led to the risible consequence of investigators “registering” their studies post hoc, on the same day of submission, solely to obtain an NCT number to satisfy the requirement for manuscript submission (traverse to the ICMJE statement). Here again, the puppies metaphor barks again. Journals had the intention of integrity, but ended up with obstructive bureaucracy for many authors. The solution lies in a pragmatic approach that is being taken by Academic Emergency Medicine. First, journals should abide by the law as specified in section 114 of the FDAMA. That part is easy. Therefore, any study that satisfies the ClinicalTrials.gov definition of a clinical trial (described above) and has an IDE or IND, or federal funding, must be preregistered. Second, any randomized controlled trial of a commercially available drug or device, funded by the industry concern, must be preregistered. For investigator-initiated research without an IND, IDE, or federal funding, including randomized trials, a reasonable and prudent approach is warranted. Academic Emergency Medicine will continue to promulgate policy that encourages preregistration for such trials, but going forward, will not set a hardline mandate for registration of studies that meet the ClinicalTrials.gov definition, but do not have an IND, IDE, or federal funding. Most importantly, Academic Emergency Medicine will not require or even allow post hoc registration. Too often, the masses of the innocent are punished by the bad deeds of a few. It is time to correct this overcorrection. Jeffrey A. Kline, MD ([email protected]) Senior Associate Editor Academic Emergency Medicine Emergency Medicine and Physiology Indiana University School of Medicine Indianapolis, IN
Supervising Editor: David C. Cone, MD.
356
References 1. U.S. Food and Drug Administration. Regulatory Information. Full text of FDAMA Law. Section 113. Information Program on Clinical Trials for Serious or Life-threatening Diseases. Available at: http://www. fda.gov/RegulatoryInformation/Legislation/FederalFood DrugandCosmeticActFDCAct/SignificantAmendment stotheFDCAct/FDAMA/FullTextofFDAMAlaw/default. htm#SEC.%20113. Accessed Dec 1, 2014. 2. World Health Organization. International Clinical Trials Registry Platform (ICTRP). Available at:
Kline • PUPPIES ON THE DOORSTEP
http://www.who.int/ictrp/en/. Accessed Nov 27, 2014. 3. National Institutes of Health. ClinicalTrials.gov: History, Policy, and Laws. Available at: http://www.clinicaltrials.gov/ct2/about-site/history. Accessed Nov 27, 2014. 4. International Committee of Medical Journal Editors. Clinical Trial Registration. Available at: http:// www.icmje.org/recommendations/browse/publishingand-editorial-issues/clinical-trial-registration.html. Accessed Nov 27, 2014.
