MESSAGE FROM THE EDITOR
Have the Annals Editors Added Value?
T
he only real utility of any journal today is judging the value of submissions and improving them whenever possible. There are plenty of places to publish research—a journal for every article—so it is not enough to provide an outlet for work in the field. Rather, the journal’s utility is defined by the editorial process and peer review, and the standards that they raise for investigators. When a reader picks up a copy of the Annals, looks through our table of contents online, or sees an Annals article in a list of PubMed output, we hope that the standards we apply will encourage a deeper dive. This increases exposure of the work and, by extension, the impact of our publishing authors. As we come to the end of our term, we have been asking ourselves, “How good a job have we done?” Has your trust in us been well placed? Can we be proud of our work as an editorial team? Unfortunately (or perhaps fortunately), the data to address these critical questions are very limited. Citations are often accepted as the fundamental currency of individual articles and journals. Faculty promotion committees are increasingly scrutinizing article citations and the h index, a measure of the number of an investigator’s articles with that number or greater citations, has become a common metric used to compare career researcher productivity. Journals themselves tout their Impact Factors, based on article citations, as primary proof of prestige, as can been seen by the flurry of press releases after the annual announcement of the updated numbers. By the Impact Factor metric, we seem to deserve a pat on the back. Although not a monotonic increase, we have gained in the last 10 years while continuing to publish a large number of research articles and few review articles, which tend to be cited more frequently1 (Figure 1). The clinical neurosciences have also gained more robust funding during that time so some of this gain may simply be due to more articles published in our field generating more citations for us and for all other neuroscience journals. However, other journals in our field have not seen gains in the last decade. Furthermore, another measure of relative article impact, the Article Influence Score, has also climbed steadily since its introduction in 2007 (Figure 1) and this is standardized with respect to other journals.
FIGURE 1: The current editorial board began its tenure in October 2005 and since then Impact Factor and Article Influence indices reflect steady gains. The Article Influence Score was first introduced in 2007.
Of course, one of the reasons the Annals can publish impactful articles is that it receives great manuscripts from investigators. Perceived prestige drives submissions in a sort of self-fulfilling prophecy. Riding the wave of prestige is certainly easy for any editorial team, so we have not congratulated ourselves much for our Impact Factor. However, looking into articles that we have accepted, we asked a slightly different question. Are we actually good judges of impact? For this we did two studies, one prospective and one retrospective. In the prospective study, each editor was asked at the time of submission to judge whether a manuscript was likely to be high-impact. Such manuscripts were notated and the remaining steps of the review process progressed as usual. Between January 2008 and July 2013, 34 submissions were noted to be potentially high-impact, of which four have not yet been published and seven were rejected after outside review. Among the 23 ultimately published in the Annals, 11 were cited more than average for the month of publication, and the average standardized citation rate was similar between those rated “high-impact” and others (p50.41). Thus, in this experiment, there was no evidence the editors could select the high-impact articles C 2013 American Neurological Association V A7
ANNALS
of Neurology
FIGURE 2: In our retrospective study, we developed a standardized article citation ratio to assess editors’ expected citations to actual citations. Tracking 1054 accepted manuscripts, the trend reflected more citations for highly rated submissions.
prior to external review. However, the prospective study had many flaws, including poor adherence to study protocol by several of the editors, who only inconsistently participated. Alas, the difficulties of research involving humans. . . In the retrospective study, we asked each editor to review all the manuscripts they had handled and were ultimately accepted during their term of service and rate the manuscripts on a Likert Scale based on quality (from 1, constituting a close call, to 5, a game changer for the field). We could not shield editors from information about how the article was received after publications but none of us tracks citations of articles we handle, so bias from the retrospective design was not a huge threat. Noting that citations increased monotonically after publication, we created a standardized article citation ratio by
TABLE. Published Articles Cited More and Less than Expected Based on Editors Ratings
Rating
Published Manuscript
Times Cited
Standardized Citation Ratio
Full-Text Downloads
Rated Low but Cited Highly 1
SV Ramagopalan et al. Rare variants in the CYP27B1 gene are associated with multiple sclerosis
42
3.1
2,556
2
SR Irani et al. Faciobrachial Dystonic Seizures Precede Lgi1 Antibody Limbic Encephalitis
60
3.2
3,873
2
DM Rentz et al. Cognition, Reserve, and Amyloid Deposition in Normal Aging
97
3.2
1,946
2
VL Villemagne et al. Longitudinal Assessment of A beta and Cognition in Aging and Alzheimer Dise ase
116
5.2
2,835
Rated High but Poorly Cited 4
C Qiu et al. Microvascular Lesions in the Brain and Retina: The Age. Gene/Environment Susceptibility-Reykjavik Study
8
0.2
427
4
S Jafarian et al. Sumatriptan for prevention of acute mountain sickness: Randomized clinical trial
8
0.1
622
4
W Eriksen et al. Register Data Suggest Lower Intelligence in Men Born the Year After Flu Pandemic
3
0.1
376
5
A Jacquier et al. Alsin/Rac1 signaling controls survival and growth of spinal motoneurons
33
0.5
1,319
5
AJ Barkowich et al. A developmental classification of malformations of the brainstem
24
0.5
1,548
A8
Volume 74, No. 5
:
dividing the actual number of citations by the expected citations based on months since publication from a fitted regression model. In this study, which included 1054 published manuscripts, we did a little better. Submissions we rated more highly were more likely to receive more citations (Figure 2; p < 0.001 for trend). Most of our predictive ability came from differentiating the highest impact articles from others while there were no real differences at the lower end of the rating scale. Though the p value is impressive, the association between rating and citation impact is actually fairly weak. We only predicted 1.5% of the variance in article citation. Some of this is certainly related to our rating impact rather than our attempting to predict citation. The outliers tell this story well (Table). We defined these outliers as those with at least three-fold greater than expected citations and editors’ ratings of one or two, and those receiving half the expected citations that were rated four or five.2–9 Not shown in the table is the article with the highest standardized citation ratio, a cerebrospinal fluid biomarker study using data from the Alzheimer Disease Neuroimaging Initiative.10 We knew this was an important study, contributed by a great group of investigators, but it only received a rating of 3 because we saw substantial overlap between this publication and several others submitted around the same time or just afterwards; a common issue for studies relying on these data. On rereview, we remain surprised by the articles cited well and still like the articles with fewer than expected citations. Article downloads correlated with citations, suggesting it is not just a function of citations alone. Of course, there are many of examples of masterpieces not appreciated in their time, and more startling findings may take longer to discover. Seeking to publish manuscripts that ultimately receive multiple citations may also come with an important risk. There is a troubling – but perhaps not surprising – linear correlation between the impact factor of a journal and its rate of retractions11. Even in the current era of modern neuroimaging and advanced molecular and cellular biology, transformative findings are uncommon, and journals that require dramatic data for publication are likely at higher risk of accepting manuscripts that simply don’t pan out. As Carl Sagan famously quipped, “extraordinary claims require extraordinary evidence”. Although we seek impactful manuscripts, an even higher priority is to choose those that will stand the test of time.
November 2013
Taken together, we are not convinced by our own data one way or the other. We have certainly worked hard and are glad the articles we have published have been generally well received, keeping the Annals in its position of selectivity. Could we have done better? Almost certainly but we also could have done worse. Now it is time for the new team to come along and demonstrate the real potential of the journal. S. Claiborne Johnston, MD, PhD Donna M. Ferriero, MD S. Andrew Josephson, MD Daniel H. Lowenstein, MD Robert O. Messing, MD Jorge Oksenberg, PhD Adam Stewart Stephen L. Hauser, MD Editors
References 1.
E Garfield (Sept 2005) The Agony and the Ecstasy—The History and Meaning of the Journal Impact Factor presented at the International Congress on Peer Review And Biomedical Publication, Chicago. Presentation retrieved from http://garfield.library.upenn. edu/papers/jifchicago2005.pdf
2.
SV Ramagopalan et al, Rare variants in the CYP27B1 gene are associated with multiple sclerosis, Ann Neurol 2011;70:881–886.
3.
SR Irani et al, Faciobrachial Dystonic Seizures Precede Lgi1 Antibody Limbic Encephalitis, Ann Neurol 2011;69:892–900.
4.
DM Rentz et al, Cognition, Reserve, and Amyloid Deposition in Normal Aging, Ann Neurol 2010;67:353–64.
5.
C Qiu et al, Microvascular Lesions in the Brain and Retina: The Age, Gene/Environment Susceptibility-Reykjavik Study, Ann Neurol 2009;65:569–576.
6.
S Jafarian et al, Sumatriptan for prevention of acute mountain sickness: Randomized clinical trial, Ann Neurol 2007;62:273–277.
7.
W Eriksen et al, Register Data Suggest Lower Intelligence in Men Born the Year After Flu Pandemic, Ann Neurol 2009;66:284–289.
8.
A Jacquier et al, Alsin/Rac1 signaling controls survival and growth of spinal motoneurons, Ann Neurol 2006;60:105–117.
9.
AJ Barkowich et al, A developmental classification of malformations of the brainstem, Ann Neurol 2007;62:625–639.
10.
LM Shaw et al, Cerebrospinal fluid biomarker signature in Alzheimer’s disease neuroimaging initiative subjects Ann Neurol 2009; 65:403–413.
11.
FC Fang, A Casadevall, RP Morrison, Retracted Science and the Retraction Index Infect. Immun. 2011;79:3855–3859.
DOI: 10.1002/ana.24077
A9
Have the Annals Editors Added Value?
T
he only real utility of any journal today is judging the value of submissions and improving them whenever possible. There are plenty of places to publish research—a journal for every article—so it is not enough to provide an outlet for work in the field. Rather, the journal’s utility is defined by the editorial process and peer review, and the standards that they raise for investigators. When a reader picks up a copy of the Annals, looks through our table of contents online, or sees an Annals article in a list of PubMed output, we hope that the standards we apply will encourage a deeper dive. This increases exposure of the work and, by extension, the impact of our publishing authors. As we come to the end of our term, we have been asking ourselves, “How good a job have we done?” Has your trust in us been well placed? Can we be proud of our work as an editorial team? Unfortunately (or perhaps fortunately), the data to address these critical questions are very limited. Citations are often accepted as the fundamental currency of individual articles and journals. Faculty promotion committees are increasingly scrutinizing article citations and the h index, a measure of the number of an investigator’s articles with that number or greater citations, has become a common metric used to compare career researcher productivity. Journals themselves tout their Impact Factors, based on article citations, as primary proof of prestige, as can been seen by the flurry of press releases after the annual announcement of the updated numbers. By the Impact Factor metric, we seem to deserve a pat on the back. Although not a monotonic increase, we have gained in the last 10 years while continuing to publish a large number of research articles and few review articles, which tend to be cited more frequently1 (Figure 1). The clinical neurosciences have also gained more robust funding during that time so some of this gain may simply be due to more articles published in our field generating more citations for us and for all other neuroscience journals. However, other journals in our field have not seen gains in the last decade. Furthermore, another measure of relative article impact, the Article Influence Score, has also climbed steadily since its introduction in 2007 (Figure 1) and this is standardized with respect to other journals.
FIGURE 1: The current editorial board began its tenure in October 2005 and since then Impact Factor and Article Influence indices reflect steady gains. The Article Influence Score was first introduced in 2007.
Of course, one of the reasons the Annals can publish impactful articles is that it receives great manuscripts from investigators. Perceived prestige drives submissions in a sort of self-fulfilling prophecy. Riding the wave of prestige is certainly easy for any editorial team, so we have not congratulated ourselves much for our Impact Factor. However, looking into articles that we have accepted, we asked a slightly different question. Are we actually good judges of impact? For this we did two studies, one prospective and one retrospective. In the prospective study, each editor was asked at the time of submission to judge whether a manuscript was likely to be high-impact. Such manuscripts were notated and the remaining steps of the review process progressed as usual. Between January 2008 and July 2013, 34 submissions were noted to be potentially high-impact, of which four have not yet been published and seven were rejected after outside review. Among the 23 ultimately published in the Annals, 11 were cited more than average for the month of publication, and the average standardized citation rate was similar between those rated “high-impact” and others (p50.41). Thus, in this experiment, there was no evidence the editors could select the high-impact articles C 2013 American Neurological Association V A7
ANNALS
of Neurology
FIGURE 2: In our retrospective study, we developed a standardized article citation ratio to assess editors’ expected citations to actual citations. Tracking 1054 accepted manuscripts, the trend reflected more citations for highly rated submissions.
prior to external review. However, the prospective study had many flaws, including poor adherence to study protocol by several of the editors, who only inconsistently participated. Alas, the difficulties of research involving humans. . . In the retrospective study, we asked each editor to review all the manuscripts they had handled and were ultimately accepted during their term of service and rate the manuscripts on a Likert Scale based on quality (from 1, constituting a close call, to 5, a game changer for the field). We could not shield editors from information about how the article was received after publications but none of us tracks citations of articles we handle, so bias from the retrospective design was not a huge threat. Noting that citations increased monotonically after publication, we created a standardized article citation ratio by
TABLE. Published Articles Cited More and Less than Expected Based on Editors Ratings
Rating
Published Manuscript
Times Cited
Standardized Citation Ratio
Full-Text Downloads
Rated Low but Cited Highly 1
SV Ramagopalan et al. Rare variants in the CYP27B1 gene are associated with multiple sclerosis
42
3.1
2,556
2
SR Irani et al. Faciobrachial Dystonic Seizures Precede Lgi1 Antibody Limbic Encephalitis
60
3.2
3,873
2
DM Rentz et al. Cognition, Reserve, and Amyloid Deposition in Normal Aging
97
3.2
1,946
2
VL Villemagne et al. Longitudinal Assessment of A beta and Cognition in Aging and Alzheimer Dise ase
116
5.2
2,835
Rated High but Poorly Cited 4
C Qiu et al. Microvascular Lesions in the Brain and Retina: The Age. Gene/Environment Susceptibility-Reykjavik Study
8
0.2
427
4
S Jafarian et al. Sumatriptan for prevention of acute mountain sickness: Randomized clinical trial
8
0.1
622
4
W Eriksen et al. Register Data Suggest Lower Intelligence in Men Born the Year After Flu Pandemic
3
0.1
376
5
A Jacquier et al. Alsin/Rac1 signaling controls survival and growth of spinal motoneurons
33
0.5
1,319
5
AJ Barkowich et al. A developmental classification of malformations of the brainstem
24
0.5
1,548
A8
Volume 74, No. 5
:
dividing the actual number of citations by the expected citations based on months since publication from a fitted regression model. In this study, which included 1054 published manuscripts, we did a little better. Submissions we rated more highly were more likely to receive more citations (Figure 2; p < 0.001 for trend). Most of our predictive ability came from differentiating the highest impact articles from others while there were no real differences at the lower end of the rating scale. Though the p value is impressive, the association between rating and citation impact is actually fairly weak. We only predicted 1.5% of the variance in article citation. Some of this is certainly related to our rating impact rather than our attempting to predict citation. The outliers tell this story well (Table). We defined these outliers as those with at least three-fold greater than expected citations and editors’ ratings of one or two, and those receiving half the expected citations that were rated four or five.2–9 Not shown in the table is the article with the highest standardized citation ratio, a cerebrospinal fluid biomarker study using data from the Alzheimer Disease Neuroimaging Initiative.10 We knew this was an important study, contributed by a great group of investigators, but it only received a rating of 3 because we saw substantial overlap between this publication and several others submitted around the same time or just afterwards; a common issue for studies relying on these data. On rereview, we remain surprised by the articles cited well and still like the articles with fewer than expected citations. Article downloads correlated with citations, suggesting it is not just a function of citations alone. Of course, there are many of examples of masterpieces not appreciated in their time, and more startling findings may take longer to discover. Seeking to publish manuscripts that ultimately receive multiple citations may also come with an important risk. There is a troubling – but perhaps not surprising – linear correlation between the impact factor of a journal and its rate of retractions11. Even in the current era of modern neuroimaging and advanced molecular and cellular biology, transformative findings are uncommon, and journals that require dramatic data for publication are likely at higher risk of accepting manuscripts that simply don’t pan out. As Carl Sagan famously quipped, “extraordinary claims require extraordinary evidence”. Although we seek impactful manuscripts, an even higher priority is to choose those that will stand the test of time.
November 2013
Taken together, we are not convinced by our own data one way or the other. We have certainly worked hard and are glad the articles we have published have been generally well received, keeping the Annals in its position of selectivity. Could we have done better? Almost certainly but we also could have done worse. Now it is time for the new team to come along and demonstrate the real potential of the journal. S. Claiborne Johnston, MD, PhD Donna M. Ferriero, MD S. Andrew Josephson, MD Daniel H. Lowenstein, MD Robert O. Messing, MD Jorge Oksenberg, PhD Adam Stewart Stephen L. Hauser, MD Editors
References 1.
E Garfield (Sept 2005) The Agony and the Ecstasy—The History and Meaning of the Journal Impact Factor presented at the International Congress on Peer Review And Biomedical Publication, Chicago. Presentation retrieved from http://garfield.library.upenn. edu/papers/jifchicago2005.pdf
2.
SV Ramagopalan et al, Rare variants in the CYP27B1 gene are associated with multiple sclerosis, Ann Neurol 2011;70:881–886.
3.
SR Irani et al, Faciobrachial Dystonic Seizures Precede Lgi1 Antibody Limbic Encephalitis, Ann Neurol 2011;69:892–900.
4.
DM Rentz et al, Cognition, Reserve, and Amyloid Deposition in Normal Aging, Ann Neurol 2010;67:353–64.
5.
C Qiu et al, Microvascular Lesions in the Brain and Retina: The Age, Gene/Environment Susceptibility-Reykjavik Study, Ann Neurol 2009;65:569–576.
6.
S Jafarian et al, Sumatriptan for prevention of acute mountain sickness: Randomized clinical trial, Ann Neurol 2007;62:273–277.
7.
W Eriksen et al, Register Data Suggest Lower Intelligence in Men Born the Year After Flu Pandemic, Ann Neurol 2009;66:284–289.
8.
A Jacquier et al, Alsin/Rac1 signaling controls survival and growth of spinal motoneurons, Ann Neurol 2006;60:105–117.
9.
AJ Barkowich et al, A developmental classification of malformations of the brainstem, Ann Neurol 2007;62:625–639.
10.
LM Shaw et al, Cerebrospinal fluid biomarker signature in Alzheimer’s disease neuroimaging initiative subjects Ann Neurol 2009; 65:403–413.
11.
FC Fang, A Casadevall, RP Morrison, Retracted Science and the Retraction Index Infect. Immun. 2011;79:3855–3859.
DOI: 10.1002/ana.24077
A9
