CanJPsychiatry 2015;60(1):4–5
Editorial
Problems From the Past and Prevention for the Future Scott B Patten, MD, PhD1 1
Editor-in-Chief, The Canadian Journal of Psychiatry, Ottawa, Ontario; Professor, Departments of Community Health Sciences and Psychiatry, University of Calgary, Calgary, Alberta; Member, Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta. Correspondence: Department of Community Health Sciences, 3rd Floor, TRW Building, University of Calgary, 3280 Hospital Drive NW, Calgary, AB T2N 4Z6; [email protected].
Key Words: Randomized controlled trials, major depression, measurementbased care, antidepressant, pragmatic trials, trial registration, STAR*D
open access
T
his issue of The Canadian Journal of Psychiatry (The CJP) includes a Perspective article written by Dr H Edmund Pigott.1 Dr Pigott articulates a series of criticisms of the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) trial. Some of these arguments attack the trial methodology and reporting, whereas others take issue with the broader concept of measurement-based care. There is a reason for the ascendancy of RCTs. The process of randomization helps to ensure that potential confounding variables are equally distributed between treatment groups. Unique among procedures for controlling confounding (for example, matching and statistical modelling), randomization controls for unmeasured, and even unknown, confounders. As such, it is a uniquely powerful study design. The CJP’s Perspectives format is intended to be open to expressions of opinion. Authors of Perspectives articles are encouraged to take positions on controversial topics,2,3 as Dr Pigott has done. In an accompanying Guest Editorial,4 Dr Raymond W Lam and Dr Sidney H Kennedy take issue with 3 of the arguments put forward by Dr Pigott: the assertion that a primary objective of the STAR*D trial was modified post hoc; the idea that remission is not a suitable outcome measure in depression research; and the idea that the STAR*D story provides a general indictment of the principles of measurement-based care. Questions surrounding modification of a priori hypotheses in clinical trials are not trivial. Statistical analyses are vulnerable to error if there is a lack of clarity surrounding their primary outcomes. Problems also arise when primary outcomes are not clearly distinguished from secondary and exploratory ones. In the worst-case scenario, data are analyzed and investigators then selectively report the findings that they prefer. This is the proverbial fishing expedition seen in undisciplined research. In this disastrous scenario, readers cannot correctly interpret reported P levels or confidence intervals as these no longer reflect their intended probabilities. Instead, they partially reflect decisions made by the authors. This does not mean that investigators should be denied full rein to explore their data. However, if exploratory analyses are to be conducted, they must be carefully identified as such. Results from exploratory analyses are provisional. They require additional replication to ensure that the results were not merely statistical outliers. In their Guest Editorial, Lam and Kennedy4 argue that Pigott has misunderstood some aspects of the STAR*D protocol. They feel that outcomes in question, those for the Quick Inventory of Depression Symptoms—Self-Report remission, were adequately characterized as post hoc analyses, whereas Pigott feels that these were obscured. This issue is one of transparency of reporting. Progress is being made in addressing such problems. For example, The CJP requires that clinical trials be registered in a suitable archive at or before the onset of subject enrolment. A suitable registry must be accessible to the public at no charge; be open to all prospective registrants; be managed by a not-for-profit organization; have a mechanism to ensure the validity of the registration data; and be electronically searchable. Registration protects the integrity of trials, both from methodological
4 W La Revue canadienne de psychiatrie, vol 60, no 1, janvier 2015
www.LaRCP.ca
Problems From the Past and Prevention for the Future
manipulation and from accusations of such manipulation. However, registration, in itself, is not always enough. After all, the STAR*D trial protocol was registered,5 but this did not prevent concerns and controversies from arising. Some journals (for example, The Journal of the American Medical Association) now require clinical trial protocols, including the complete statistical analysis plan, be submitted along with submissions of clinical trial reports. Increasingly, investigators are choosing to archive their complete RCT protocols, including their detailed a priori analysis plans. In contrast, Dr Pigott reports that he needed a Freedom of Information Act request to obtain the STAR*D protocol.
of urgency for greater progress against depression. Sakina J Rizvi and collaborators6 report unemployment– disability rates of 30.3% of a sample of depressed primary care patients and of 41.4% in a tertiary care sample. An updated description of the general population’s prevalence of major depressive disorder documents continuing high prevalence,7 associated dysfunction, comorbidity, and stigmatization.8 Better publication and reporting standards will help us to keep our sights on the true enemy: depression itself.
In the systematic review literature, it is a long-standing practice to publish or archive review protocols prior to the reviews being conducted. The CJP’s recently initiated Systematic Reviews category strongly encourages authors to register their review protocols in a suitable registry (for example, PROSPERO) or to archive or publish their full protocols.3
1. Pigott HE. The STAR*D trial: it is time to reexamine the clinical beliefs that guide the treatment of major depression. Can J Psychiatry. 2015;60(1):9–13. 2. Paris J, Patten S. Reviews and perspectives. Can J Psychiatry. 2014;59(1):1–2. 3. The Canadian Journal of Psychiatry. Information for contributors [Internet]. Ottawa (ON): Canadian Psychiatric Association; 2014 Feb 18 [cited 2014 Nov 13]. Available from: http://publications.cpa-apc.org/browse/documents/6. 4. Lam RW, Kennedy SH. STAR*D and measurement-based care for depression: don’t toss out the baby! Can J Psychiatry. 2015;60(1):6–8. 5. National Library of Medicine (NLM) US National Institutes of Health (NIH). ClinicalTrials.gov database: clinical trial NCT00021528 [Internet]. Bethesda (MA): NLM US NIH; 2001 Jul 20 [cited 2014 Nov 18]. Available from: http://clinicaltrials.gov/ show/NCT00021528. 6. Rizvi SJ, Cyriac A, Grima E, et al. Depression and employment status in primary and tertiary care settings. Can J Psychiatry. 2015;60(1):14–22. 7. Patten SB, Williams JVA, Lavorato DH, et al. The prevalence of major depression is not changing. Can J Psychiatry. 2015;60(1):31–34. 8. Patten SB, Williams JVA, Lavorato DH, et al. Descriptive epidemiology of major depressive disorder in Canada in 2012. Can J Psychiatry. 2015;60(1):23–30.
While most major journals, this one included, require registration for trials, they usually do not require it for other types of studies. Ultimately, it would be a good idea for authors of all studies that use statistical analysis, such as epidemiologic studies or brain imaging studies, to register their protocols in a similar fashion. This would assist readers in their interpretation of the statistics reported in the ensuing papers. It would also protect them from later accusations that they have modified their analysis after the fact. While they disagree on many points, the authors of these papers1,4 agree that the STAR*D results were disappointing. Other papers in this issue of The CJP amplify this sense
www.TheCJP.ca
References
The Canadian Journal of Psychiatry, Vol 60, No 1, January 2015 W 5
Editorial
Problems From the Past and Prevention for the Future Scott B Patten, MD, PhD1 1
Editor-in-Chief, The Canadian Journal of Psychiatry, Ottawa, Ontario; Professor, Departments of Community Health Sciences and Psychiatry, University of Calgary, Calgary, Alberta; Member, Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta. Correspondence: Department of Community Health Sciences, 3rd Floor, TRW Building, University of Calgary, 3280 Hospital Drive NW, Calgary, AB T2N 4Z6; [email protected].
Key Words: Randomized controlled trials, major depression, measurementbased care, antidepressant, pragmatic trials, trial registration, STAR*D
open access
T
his issue of The Canadian Journal of Psychiatry (The CJP) includes a Perspective article written by Dr H Edmund Pigott.1 Dr Pigott articulates a series of criticisms of the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) trial. Some of these arguments attack the trial methodology and reporting, whereas others take issue with the broader concept of measurement-based care. There is a reason for the ascendancy of RCTs. The process of randomization helps to ensure that potential confounding variables are equally distributed between treatment groups. Unique among procedures for controlling confounding (for example, matching and statistical modelling), randomization controls for unmeasured, and even unknown, confounders. As such, it is a uniquely powerful study design. The CJP’s Perspectives format is intended to be open to expressions of opinion. Authors of Perspectives articles are encouraged to take positions on controversial topics,2,3 as Dr Pigott has done. In an accompanying Guest Editorial,4 Dr Raymond W Lam and Dr Sidney H Kennedy take issue with 3 of the arguments put forward by Dr Pigott: the assertion that a primary objective of the STAR*D trial was modified post hoc; the idea that remission is not a suitable outcome measure in depression research; and the idea that the STAR*D story provides a general indictment of the principles of measurement-based care. Questions surrounding modification of a priori hypotheses in clinical trials are not trivial. Statistical analyses are vulnerable to error if there is a lack of clarity surrounding their primary outcomes. Problems also arise when primary outcomes are not clearly distinguished from secondary and exploratory ones. In the worst-case scenario, data are analyzed and investigators then selectively report the findings that they prefer. This is the proverbial fishing expedition seen in undisciplined research. In this disastrous scenario, readers cannot correctly interpret reported P levels or confidence intervals as these no longer reflect their intended probabilities. Instead, they partially reflect decisions made by the authors. This does not mean that investigators should be denied full rein to explore their data. However, if exploratory analyses are to be conducted, they must be carefully identified as such. Results from exploratory analyses are provisional. They require additional replication to ensure that the results were not merely statistical outliers. In their Guest Editorial, Lam and Kennedy4 argue that Pigott has misunderstood some aspects of the STAR*D protocol. They feel that outcomes in question, those for the Quick Inventory of Depression Symptoms—Self-Report remission, were adequately characterized as post hoc analyses, whereas Pigott feels that these were obscured. This issue is one of transparency of reporting. Progress is being made in addressing such problems. For example, The CJP requires that clinical trials be registered in a suitable archive at or before the onset of subject enrolment. A suitable registry must be accessible to the public at no charge; be open to all prospective registrants; be managed by a not-for-profit organization; have a mechanism to ensure the validity of the registration data; and be electronically searchable. Registration protects the integrity of trials, both from methodological
4 W La Revue canadienne de psychiatrie, vol 60, no 1, janvier 2015
www.LaRCP.ca
Problems From the Past and Prevention for the Future
manipulation and from accusations of such manipulation. However, registration, in itself, is not always enough. After all, the STAR*D trial protocol was registered,5 but this did not prevent concerns and controversies from arising. Some journals (for example, The Journal of the American Medical Association) now require clinical trial protocols, including the complete statistical analysis plan, be submitted along with submissions of clinical trial reports. Increasingly, investigators are choosing to archive their complete RCT protocols, including their detailed a priori analysis plans. In contrast, Dr Pigott reports that he needed a Freedom of Information Act request to obtain the STAR*D protocol.
of urgency for greater progress against depression. Sakina J Rizvi and collaborators6 report unemployment– disability rates of 30.3% of a sample of depressed primary care patients and of 41.4% in a tertiary care sample. An updated description of the general population’s prevalence of major depressive disorder documents continuing high prevalence,7 associated dysfunction, comorbidity, and stigmatization.8 Better publication and reporting standards will help us to keep our sights on the true enemy: depression itself.
In the systematic review literature, it is a long-standing practice to publish or archive review protocols prior to the reviews being conducted. The CJP’s recently initiated Systematic Reviews category strongly encourages authors to register their review protocols in a suitable registry (for example, PROSPERO) or to archive or publish their full protocols.3
1. Pigott HE. The STAR*D trial: it is time to reexamine the clinical beliefs that guide the treatment of major depression. Can J Psychiatry. 2015;60(1):9–13. 2. Paris J, Patten S. Reviews and perspectives. Can J Psychiatry. 2014;59(1):1–2. 3. The Canadian Journal of Psychiatry. Information for contributors [Internet]. Ottawa (ON): Canadian Psychiatric Association; 2014 Feb 18 [cited 2014 Nov 13]. Available from: http://publications.cpa-apc.org/browse/documents/6. 4. Lam RW, Kennedy SH. STAR*D and measurement-based care for depression: don’t toss out the baby! Can J Psychiatry. 2015;60(1):6–8. 5. National Library of Medicine (NLM) US National Institutes of Health (NIH). ClinicalTrials.gov database: clinical trial NCT00021528 [Internet]. Bethesda (MA): NLM US NIH; 2001 Jul 20 [cited 2014 Nov 18]. Available from: http://clinicaltrials.gov/ show/NCT00021528. 6. Rizvi SJ, Cyriac A, Grima E, et al. Depression and employment status in primary and tertiary care settings. Can J Psychiatry. 2015;60(1):14–22. 7. Patten SB, Williams JVA, Lavorato DH, et al. The prevalence of major depression is not changing. Can J Psychiatry. 2015;60(1):31–34. 8. Patten SB, Williams JVA, Lavorato DH, et al. Descriptive epidemiology of major depressive disorder in Canada in 2012. Can J Psychiatry. 2015;60(1):23–30.
While most major journals, this one included, require registration for trials, they usually do not require it for other types of studies. Ultimately, it would be a good idea for authors of all studies that use statistical analysis, such as epidemiologic studies or brain imaging studies, to register their protocols in a similar fashion. This would assist readers in their interpretation of the statistics reported in the ensuing papers. It would also protect them from later accusations that they have modified their analysis after the fact. While they disagree on many points, the authors of these papers1,4 agree that the STAR*D results were disappointing. Other papers in this issue of The CJP amplify this sense
www.TheCJP.ca
References
The Canadian Journal of Psychiatry, Vol 60, No 1, January 2015 W 5
