Special Ar ticles • Original Research Rosenkrantz and Harisinghani Metrics for Original Research Articles in the AJR
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Special Articles Original Research
Metrics for Original Research Articles in the AJR: From First Submission to Final Publication Andrew B. Rosenkrantz1 Mukesh Harisinghani2 Rosenkrantz AB, Harisinghani M
Keywords: journalism, publication, radiology research DOI:10.2214/AJR.14.13944 Received October 15, 2014; accepted after revision October 28, 2014. 1 Department of Radiology, Center for Biomedical Imaging, NYU School of Medicine, NYU Langone Medical Center, 660 First Ave, 3rd Fl, New York, NY 10016. Address correspondence to A. B. Rosenkrantz ([email protected]). 2 Department of Radiology, Massachusetts General Hospital, Boston, MA.
AJR 2015; 204:1152–1156 0361–803X/15/2046–1152 © American Roentgen Ray Society
1152
OBJECTIVE. The objective of our study was to evaluate manuscript metrics pertaining to AJR submissions, assessing the pathway from manuscript submission to publication, including the reviewer allocation time, decisions rendered, timing of decisions rendered, and time to publication. MATERIALS AND METHODS. Six hundred ninety-six unsolicited Original Research manuscripts submitted to the AJR between July 1, 2012, and December 21, 2012, were included in this retrospective analysis. Metrics pertaining to manuscripts’ decision status and associated timelines were extracted by journal editorial staff and assessed using standard summary statistics. RESULTS. For new submissions, decisions rendered were as follows: Accept, 0.3%; Minor Revision, 8.5%; Major Revision, 19.7%; Reject, 65.1%; and Reject Without Review, 6.5%. For first and second resubmissions, 40.0–55.2% of manuscripts representing a Major Revision and 91.5–94.7% of manuscripts representing a Minor Revision were accepted; 100% of manuscripts undergoing a third resubmission were accepted; 98.3% and 84.7% of manuscripts receiving at first submission a decision of Minor Revision and Major Revision, respectively, ultimately achieved acceptance. The time (mean ± SD) to review a new submission was 30.5 ± 43.1 days (Accept), 42.7 ± 27.4 days (Minor Revision), 39.4 ± 17.6 days (Major Revision), and 40.2 ± 20.3 days (Reject) and decreased with each subsequent resubmission to 6.3 ± 6.3 days (Accept) for third resubmissions. The mean days for authors to submit a first resubmission was 21.1 ± 15.3 days (Minor Revision) and 73.7 ± 65.1 days (Major Revision) and decreased with each subsequent resubmission to 9.8 ± 11.3 days (Minor Revision) and 27.0 ± 0.0 days (Major Revision) for third resubmissions. The mean time from acceptance to publication was 242.5 ± 47.5 days. CONCLUSION. The observed metrics may provide valuable insights for authors and for AJR editorial staff in ongoing efforts to shorten turnaround times from manuscript submission to publication.
R
apid publication of the results of scientific research is important for allowing prompt dissemination of new knowledge and the application of advances for patient care [1]. However, the process of publishing new manuscripts can be prolonged depending on the timing of the sequence of steps that occur after submission, including the initial editorial office review, invitation of reviewers, completion of reviews, rendering of a decision, revision by the authors, one or more additional rounds of review until acceptance, and finally production of the accepted manuscript for publication [2–4]. Digital workflows and modifications in editorial office operations can be useful in accelerating this process. For instance, the submission and re-
view processes can be performed in an entirely electronic fashion [5–7], and release of electronic copies of accepted manuscripts before publication of the print version [8] has become standard for many journals. Additional options, such as continuous online publication [9] or conversion to an onlineonly format [10], are also increasingly being considered. A reduction in publication turnaround time through such efforts can improve a journal’s impact factor [11] and influence authors to be more likely to submit a manuscript to a given journal [12]. The AJR is a leading journal in the field of radiology, having a circulation of nearly 25,000 and being highly competitive in its selection of submitted manuscripts for publication [13, 14]. Achieving as rapid a publica-
AJR:204, June 2015
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Metrics for Original Research Articles in the AJR tion turnaround time as possible would benefit not only the journal’s authors and readers, but also the radiology community in general. Toward this end, a heightened understanding of the timeline for review and publication of newly submitted manuscripts would be useful for guiding efforts to improve publication times, in addition to providing authors with key insights regarding the handling of their submissions. Therefore, in this study, our aim was to evaluate metrics pertaining to manuscripts submitted to the AJR, assessing the pathway from submission to publication including the reviewer allocation time, decisions rendered, timing of decisions rendered, and time to publication. Materials and Methods This retrospective study did not pertain to human subjects research and therefore did not require institutional review board approval. Data were obtained from the AJR’s Editorial Manager system (Aries Systems) through the assistance of the journal’s office staff. Initially, data were obtained for 982 manuscripts that underwent first submission between July 1, 2012, and December 31, 2012. Of these, submissions that were not unsolicited Original Research articles (i.e., Reviews, Technical Innovations, Clinical Perspectives, Editorials, Opinions, Commentaries, Letters to the Editor, and solicited Original Research articles) were excluded. Exclusion of these manuscripts left a final cohort of 696 unsolicited Original Research manuscripts submitted during this 6-month period that were evaluated in this study. This period was selected to allow sufficient time for not only review and revision of all manuscripts, but also for assignment of a final publication date for all accepted manuscripts in the analysis.
For the identified manuscripts, the following metrics were recorded: days for initial editorial office processing; days for manuscript review, which incorporated the time needed for solicitation of reviewers, completion of reviews, and rendering of a final decision; and decision rendered (Accept, Minor Revision, Major Revision, Reject, and Reject Without Review). A decision of Major Revision generally entails repeating a considerable portion of the work that preceded drafting the manuscript, such as acquiring additional data or repeating key aspects of the data analysis [15]. In comparison, a decision of Minor Revision generally entails modifications to the presentation of the existing data—for instance, clarification of methodologic details or acknowledging additional prior studies or limitations in the Discussion. These metrics were also recorded for each resubmission of the manuscript, along with the days needed for the authors to perform the revision. For accepted manuscripts, the time from acceptance to publication was recorded. Metrics for resubmitted manuscripts were stratified by whether related to a Major or Minor Revision, as well as by the round of resubmission since the first submission. Metrics regarding decisions rendered were also stratified by journal section or category. Metrics were initially assessed using standard summary statistics (Excel for Mac 2011, Microsoft). Subsequently, decisions rendered were compared between rounds of submission using McNemar tests, whereas days for editorial office processing, manuscript review, and author revisions were compared between rounds of submission and between manuscripts receiving various decisions using paired Wilcoxon and MannWhitney tests, respectively (MedCalc, version 9.1, MedCalc Software). Metrics pertaining to third resubmissions were not included in the testing for statistical significance given the inclusion of only
six manuscripts in that category. All comparisons are two-sided and considered statistically significant at p < 0.05.
Results Decisions Rendered Table 1 summarizes data regarding decisions rendered. For new submissions, decisions were as follows: Accept, 0.3%; Minor Revision, 8.5%; Major Revision, 19.7%; Reject, 65.1%; and Reject Without Review, 6.5%. For both first and second resubmissions, there were more Accept (40.0–94.7%) or Minor Revision (2.6–40.0%) decisions and fewer Major Revision (0.0–10.0%) or Reject (1.7–10.4%) decisions. Furthermore, for both the first and second resubmissions, roughly half of manuscripts representing a Major Revision and more than 90% of manuscripts representing a Minor Revision were accepted. In comparison, approximately 10% of manuscripts representing a Major Revision and less than 3% of manuscripts representing a Minor Revision were rejected. Among the small number of manuscripts undergoing a third resubmission, 100% (6/6) were accepted. Across all rounds of review, authors revised and resubmitted 99.0% (102/103) and 97.3% (145/149) of manuscripts invited for Minor and Major Revision, respectively. Ultimately, 98.3% (58/59) and 84.7% (116/137) of manuscripts receiving a decision of Minor or Major Revision at first submission, respectively, achieved a final acceptance after one or more subsequent rounds of revision. Overall, 25.3% (n = 176) of the 696 Original Research articles in this analysis achieved eventual acceptance, once considering all rounds of resubmission. Table 2 shows the distribution of
TABLE 1: Distribution of Decisions Rendered Stratified by Manuscript Submission Number and Whether a Minor or Major Revision Previous Decision First Resubmission Decision Rendered
Second Resubmission
Third Resubmission
New Submission
Major Revision
Minor Revision
Major Revision
Minor Revision
Major Revision
Minor Revision
0.3 (2/696)
55.2 (74/134)
91.5 (54/59)
40.0 (4/10)
94.7 (36/38)
100.0 (1/1)
100.0 (5/5)
Minor Revision
8.5 (59/696)
26.1 (35/134)
6.8 (4/59)
40.0 (4/10)
2.6 (1/38)
0.0 (0/1)
0.0 (0/5)
Major Revision
19.7 (137/696)
8.2 (11/134)
0.0 (0/59)
10.0 (1/10)
0.0 (0/38)
0.0 (0/1)
0.0 (0/5)
Reject
65.1 (453/696)
10.4 (14/134)
1.7 (1/59)
10.0 (1/10)
2.6 (1/38)
0.0 (0/1)
0.0 (0/5)
6.5 (45/696)
—
—
—
—
—
—
696
134
59
10
38
1
5
Decision rendered Accept
Reject Without Review Total no. of manuscripts
Note—Unless indicated otherwise, data are presented as percentages (no. of manuscripts/total no. of manuscripts). Dash (—) indicates not applicable.
AJR:204, June 2015 1153
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Rosenkrantz and Harisinghani Revision (range of mean values, 9.8 ± 11.3 to 21.1 ± 15.3 days). Overall when considering the various components of the review and resubmission pro cess, the mean time from the date of the first submission for a new manuscript to the date of final acceptance after all rounds of revision was 128.6 ± 83.8 days (range, 6–539 days; median, 115 days). The mean time from final acceptance to publication in the print version of the journal was 242.5 ± 47.5 days.
rendered. In particular, for resubmissions, manuscripts receiving a decision of Accept required less time for review (p ≤ 0.008) than manuscripts receiving other decisions (range of mean values of 2.3 ± 2.6 to 11.9 ± 13.2 days vs range of mean values of 9.0 ± 0.0 to 28.9 ± 13.8 days, respectively). Table 3 also shows the time required for authors to complete manuscript revisions. This time decreased for subsequent rounds of resubmission (p < 0.001) from a range of 21.1 ± 15.3 to 73.7 ± 65.1 days for first resubmissions to a range of 9.8 ± 11.3 to 27.0 ± 0.0 days for third resubmissions, although these times varied substantially depending on the previous decision. In particular, authors required considerably more time (p < 0.001) for responding to a previous decision of Major Revision (range of mean values, 27.0 ± 0.0 to 73.7 ± 65.1 days) rather than Minor
decisions rendered for new submissions and final acceptance rate after all rounds of revision stratified by journal section or category. Timelines The mean time for initial editorial office processing after submission was shorter (p < 0.001) for new submissions (mean ± SD, 3.5 ± 7.6 days) than for resubmissions (8.7 ± 7.1, 8.2 ± 7.2, and 7.3 ± 2.3 days, for first, second, and third resubmissions, respectively). Table 3 shows the time required for subsequent manuscript review stratified by the submission number and decision rendered. This time consistently decreased with each subsequent round of resubmission after the first submission (p < 0.001), from 30.5 ± 43.1 to 42.7 ± 27.4 days for new submissions to 6.3 ± 6.3 days for third resubmissions, although the time also varied by the decision
Discussion In this study, we evaluated metrics pertaining to manuscripts submitted to the AJR from first submission through eventual publication. Our observations regarding decisions rendered should be reassuring to authors given that among new research articles invited for revision, the large majority were ultimate-
TABLE 2: Distribution of Decisions Rendered for New Submissions and Final Acceptance Rate After All Rounds of Revision Stratified by Journal Section or Category Decision No. of Manuscripts
Section or Category
Accept
Minor Revision
Major Revision
Reject Without Review
Reject
Final Acceptance Rate
Section Cardiopulmonary Imaging
98
0.0 (0/98)
3.1 (3/98)
13.3 (13/98)
81.6 (80/98)
2.0 (2/98)
16.3 (16/98)
Gastrointestinal Imaging
123
0.0 (0/123)
7.3 (9/123)
14.6 (18/123)
75.6 (93/123)
2.4 (3/123)
22.0 (27/123)
Genitourinary Imaging
42
0.0 (0/42)
4.8 (2/42)
38.1 (16/42)
57.1 (24/42)
0.0 (0/42)
40.5 (17/42)
Health Care Policy and Quality
14
0.0 (0/14)
28.6 (4/14)
28.6 (4/14)
28.6 (4/14)
14.3 (2/14)
57.1 (8/14)
Medical Physics and Informatics
32
6.3 (2/32)
15.6 (5/32)
18.8 (6/32)
31.3 (10/32)
28.1 (9/32)
40.6 (13/32)
Musculoskeletal Imaging
77
0.0 (0/77)
3.9 (3/77)
20.8 (16/77)
72.7 (56/77)
2.6 (2/77)
19.5 (15/77)
Neuroradiology/Head and Neck Imaging
68
0.0 (0/68)
5.9 (4/68)
30.9 (21/68)
60.3 (41/68)
2.9 (2/68)
32.4 (22/68)
Nuclear Medicine and Molecular Imaging
23
0.0 (0/23)
8.7 (2/23)
4.3 (1/23)
87.0 (20/23)
0.0 (0/23)
13.0 (3/23)
Pediatric Imaging
44
0.0 (0/44)
20.5 (9/44)
15.9 (7/44)
63.6 (28/44)
0.0 (0/44)
31.8 (14/44)
Vascular and Interventional Radiology
69
0.0 (0/69)
8.7 (6/69)
14.5 (10/69)
72.5 (50/69)
4.3 (3/69)
18.8 (13/69)
Women’s Imaging
73
0.0 (0/73)
8.2 (6/73)
27.4 (20/73)
39.7 (29/73)
24.7 (18/73)
26.0 (19/73)
33
0.0 (0/33)
18.2 (6/33)
15.2 (5/33)
54.5 (18/33)
12.1 (4/33)
27.3 (9/33)
Categories Special Articles and Other
Note—Unless indicated otherwise, data are presented as percentages (no. of manuscripts/total no. of manuscripts).
TABLE 3: Time Required for Various Aspects of Manuscript Review and Revision Process Time for Authors to Prepare Resubmission (d) Manuscript New submission
Minor Revision
Time for Reviewers’ and Section Editor’s Review, Stratified by Decision Rendered (d)
Major Revision
Accept
Minor Revision
Major Revision
Reject
Reject Without Review
—
—
30.5 ± 43.1
42.7 ± 27.4
39.4 ± 17.6
40.2 ± 20.3
2.3 ± 4.2
First resubmission
21.1 ± 15.3
73.7 ± 65.1
11.9 ± 13.2
21.8 ± 19.9
28.9 ± 13.8
25.6 ± 15.5
—
Second resubmission
11.1 ± 8.7
57.1 ± 69.8
2.3 ± 2.6
18.6 ± 14.8
9.0 ± 0.0
10.0 ± 12.7
—
Third resubmission
9.8 ± 11.3
27.0 ± 0.0
6.3 ± 6.3
—
—
—
—
Note—Data are mean ± SD. Dash (—) indicates not applicable.
1154
AJR:204, June 2015
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Metrics for Original Research Articles in the AJR ly accepted, whether the revisions were major or minor. In addition, a final decision of Accept or Reject was reached after a single resubmission in almost 95% of included studies, highlighting an ability of reviewers and section editors to efficiently identify those articles meriting publication. Moreover, approximately 95% of studies requiring a second or third resubmission were accepted, suggesting that authors should view these additional rounds of revision as a favorable sign that indicates a desire by the journal to publish the article, albeit after additional modifications. Awareness of these trends is important for authors when evaluating reviewer comments and considering whether to prepare and submit a revised manuscript to the journal. Indeed, a revised manuscript was submitted in nearly 100% of opportunities to do so, signifying the level of encouragement felt by authors when provided this chance. Even if an Original Research manuscript is rejected by the AJR, past work indicates that these manuscripts are likely to eventually be accepted and published by an alternate journal [13], thereby allowing dissemination of the results of the authors’ research. We observed that 6.5% of new submissions were rejected without review. This decision generally occurs because of a technical matter detected by the editorial staff during the initial evaluation of the manuscript for adherence to International Committee of Medical Journal Editors and AJR author guidelines. The journal maintains comprehensive policies, made available to authors on the journal website, regarding article categories, authorship requirements, content limits, and other details [16]. For instance, authors are required to submit a detailed cover letter providing information regarding any previous publications or conference presentations relating to the submitted manuscript along with copies of prior work. In addition, CrossCheck (CrossRef) is applied to all manuscripts to assess for overlap with previous publications, including those published by the authors themselves [17]. Authors can play an active role in ensuring timely handling of their new manuscripts by being aware of these myriad issues and taking sufficient care to follow all journal policies [18]. Authors should also be aware that a manuscript may be rejected without review if the manuscript content is not well suited for the intended general audience of the AJR, such as a highly technical research paper that may better fit a subspecialty journal. For manuscripts proceeding to review, the reviews took more time when requesting re-
visions than when accepting the submission, consistent with the expected lengthier and more complex nature of a review that identifies weaknesses of a manuscript as well as specific areas for improvement. Moreover, authors took more time for preparing revisions that were Major in nature, consistent as well with the greater effort inherent in responding to more extensive comments. Both of these areas suggest potential opportunities to shorten the overall process, namely by reducing the amount of time allowed for these steps. As of this writing, the journal allows 14 business days for completion of a review and 6 months for completion of a Major Revision. Regarding the allowed review time, reviewers may decline an invitation to review a manuscript if unable to complete the review by the specified deadline. Regarding the allowed revision time, authors are typically eager to have their work accepted and will make all efforts possible to revise a manuscript within the provided window. Thus, we anticipate that the time allotments for these steps could be reasonably shortened to accelerate the overall turnaround time without impeding general journal operations. Although uncommon, a small number of manuscripts did require four rounds of submission before achieving acceptance. These serial rounds of revision and further review inherently prolong the publication process, with one manuscript requiring approximately 1.5 years to undergo these multiple rounds of review. Although shortening the time allowed each round, as noted previously, will yield a more efficient process, an additional consideration is to limit either the number of allowed revisions or the number of opportunities for Major Revision for an individual manuscript. This approach would require authors to strive to meaningfully address all reviewer comments at the earliest possible opportunity during the revision cycle and ideally mitigate the frequency of manuscripts that become outliers in terms of requiring repetitive rounds of revision and associated greatly prolonged publication times. In particular, authors need to be sure to read and understand the reviewer comments and to address these comments through changes to the manuscript and in the rebuttal letter [15, 19]. Ultimately, the longest component of the total turnaround time was the time from acceptance to publication, taking more than 6 months on average. In the current era of digital communications [20, 21], in which social media and other web applications allow nearly instantaneous dissemination of information and opinions, it is critical that this time be substan-
tially shortened. Although there are inherent delays in the time required for preparation and production of the print version, workflow redesigns can be pursued to accelerate the process by which accepted manuscripts become available in electronic format. Indeed, most physicians today rely solely on electronic versions of journals for accessing published manuscripts [20]. Ideally, it will be possible to make all accepted manuscripts available in electronic form shortly after acceptance to make the results available to the radiology community in as timely a fashion as possible even if there are additional delays until the article eventually appears in a print issue. One further consideration for shortening the publication time is to reduce the percentage of submitted manuscripts receiving acceptance. This action would reduce the backlog of manuscripts awaiting publication, thereby allowing accepted manuscripts to move through the production process more quickly. To reduce the fraction of accepted manuscripts would pose a challenge given the large number of high-quality manuscripts submitted to the AJR but would be anticipated to further increase the quality of accepted manuscripts, as judged by the journal’s reviewers. In this study, we observed an overall acceptance rate of 25.3%, which essentially matches the target that the journal provides to reviewers of recommending acceptance for the top 25% of manuscripts reviewed. Nevertheless, a lower targeted acceptance rate would effectively reduce any accumulation of accepted manuscripts awaiting publication and be expected to directly yield shorter publication times. This analysis has a number of limitations. First, although we have made broad suggestions regarding potential approaches to accelerate the publication process, the distribution in decisions rendered was variable among journal sections and categories. Nevertheless, we believe that the most powerful reforms to produce a meaningful across-the-board reduction in publication times will likely pertain to all of the journal’s sections in a global fashion. In addition, we assessed publication times for only a single imaging journal. To our knowledge, external benchmarks for performance have not been established, and we have not analyzed how the observed turnaround times compare with those from other journals in the field. Finally, although we have suggested a number of ways in which publication times may be shortened, we have not tested or implemented these ideas and their actual impact in practice remains uncertain.
AJR:204, June 2015 1155
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Rosenkrantz and Harisinghani In conclusion, we have summarized the pathway followed by unsolicited Original Research manuscripts in the AJR from first submission to final publication. We hope that this information will provide useful insights to authors submitting manuscripts to the AJR and to AJR editorial staff in ongoing efforts to shorten publication turnaround times. References 1. Palese A, Coletti S, Dante A. Publication efficiency among the higher impact factor nursing journals in 2009: a retrospective analysis. Int J Nurs Stud 2013; 50:543–551 2. Cornelius JL. Reviewing the review process: identifying sources of delay. Australas Med J 2012; 5:26–29 3. Kljaković-Gaspić M, Hren D, Marusić A, Marusić M. Peer review time: how late is late in a small medical journal? Arch Med Res 2003; 34:439–443 4. Tite L, Schroter S. Why do peer reviewers decline to review? A survey. J Epidemiol Community Health 2007; 61:9–12 5. Shelton JS, Beach J. Electronic manuscript submission and peer review. Prim Care Companion J
1156
Clin Psychiatry 2004; 6:232–233 6. Govender P, Buckley O, McAuley G, O’Brien J, Torreggiani WC. Does online submission of manuscripts improve efficiency? JBR-BTR 2008; 91:231–234 7. Schroeder TV, Beard JD. Web manuscript submission and tracking system for peer review: the journal moves with the time. Eur J Vasc Endovasc Surg 2002; 23:95–96 8. Heneberg P. Effects of print publication lag in dual format journals on scientometric indicators. PLoS ONE 2013; 8:e59877 9. Anderton S, Harvey L. Continuous publication: ready, set, cite! Br J Radiol 2013; 86:20130252 10. Berquist TH. Print versus online: the debate goes on! AJR 2010; 195:815–816 11. Labanaris AP, Vassiliadu AP, Polykandriotis E, Tjiawi J, Arkudas A, Horch RE. Impact factors and publication times for plastic surgery journals. Plast Reconstr Surg 2007; 120:2076–2081 12. Soreide K, Winter DC. Global survey of factors influencing choice of surgical journal for manuscript submission. Surgery 2010; 147:475–480 13. Chew FS. Fate of manuscripts rejected for publication in the AJR. AJR 1991; 156:627–632
14. American Roentgen Ray Society website. About AJR. www.arrs.org/Publications/AJR/about.aspx. Accessed October 12, 2014 15. Provenzale JM. Revising a manuscript: ten principles to guide success for publication. AJR 2010; 195:[web]W382–W387 16. American Roentgen Ray Society. AJR guidelines for authors: Rev 09.2014. www.arrs.org/uploadedFiles/ ARRS/Publications/authorGuidelines.pdf. Accessed October 12, 2014 17. Berquist TH. Scientific integrity: has CrossCheck made a difference? AJR 2013; 201:703 18. Berquist TH. Submission to final decision: what can authors do to facilitate the process? AJR 2014; 202:1165 19. Provenzale JM. Ten principles to improve the likelihood of publication of a scientific manuscript. AJR 2007; 188:1179–1182 20. Kanne JP. The digital media revolution: what it means for the AJR. AJR 2011; 197:6–7 21. Hawkins CM, Duszak R, Rawson JV. Social media in radiology: early trends in Twitter micro blogging at radiology’s largest international meeting. J Am Coll Radiol 2014; 11:387–390
AJR:204, June 2015
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Special Articles Original Research
Metrics for Original Research Articles in the AJR: From First Submission to Final Publication Andrew B. Rosenkrantz1 Mukesh Harisinghani2 Rosenkrantz AB, Harisinghani M
Keywords: journalism, publication, radiology research DOI:10.2214/AJR.14.13944 Received October 15, 2014; accepted after revision October 28, 2014. 1 Department of Radiology, Center for Biomedical Imaging, NYU School of Medicine, NYU Langone Medical Center, 660 First Ave, 3rd Fl, New York, NY 10016. Address correspondence to A. B. Rosenkrantz ([email protected]). 2 Department of Radiology, Massachusetts General Hospital, Boston, MA.
AJR 2015; 204:1152–1156 0361–803X/15/2046–1152 © American Roentgen Ray Society
1152
OBJECTIVE. The objective of our study was to evaluate manuscript metrics pertaining to AJR submissions, assessing the pathway from manuscript submission to publication, including the reviewer allocation time, decisions rendered, timing of decisions rendered, and time to publication. MATERIALS AND METHODS. Six hundred ninety-six unsolicited Original Research manuscripts submitted to the AJR between July 1, 2012, and December 21, 2012, were included in this retrospective analysis. Metrics pertaining to manuscripts’ decision status and associated timelines were extracted by journal editorial staff and assessed using standard summary statistics. RESULTS. For new submissions, decisions rendered were as follows: Accept, 0.3%; Minor Revision, 8.5%; Major Revision, 19.7%; Reject, 65.1%; and Reject Without Review, 6.5%. For first and second resubmissions, 40.0–55.2% of manuscripts representing a Major Revision and 91.5–94.7% of manuscripts representing a Minor Revision were accepted; 100% of manuscripts undergoing a third resubmission were accepted; 98.3% and 84.7% of manuscripts receiving at first submission a decision of Minor Revision and Major Revision, respectively, ultimately achieved acceptance. The time (mean ± SD) to review a new submission was 30.5 ± 43.1 days (Accept), 42.7 ± 27.4 days (Minor Revision), 39.4 ± 17.6 days (Major Revision), and 40.2 ± 20.3 days (Reject) and decreased with each subsequent resubmission to 6.3 ± 6.3 days (Accept) for third resubmissions. The mean days for authors to submit a first resubmission was 21.1 ± 15.3 days (Minor Revision) and 73.7 ± 65.1 days (Major Revision) and decreased with each subsequent resubmission to 9.8 ± 11.3 days (Minor Revision) and 27.0 ± 0.0 days (Major Revision) for third resubmissions. The mean time from acceptance to publication was 242.5 ± 47.5 days. CONCLUSION. The observed metrics may provide valuable insights for authors and for AJR editorial staff in ongoing efforts to shorten turnaround times from manuscript submission to publication.
R
apid publication of the results of scientific research is important for allowing prompt dissemination of new knowledge and the application of advances for patient care [1]. However, the process of publishing new manuscripts can be prolonged depending on the timing of the sequence of steps that occur after submission, including the initial editorial office review, invitation of reviewers, completion of reviews, rendering of a decision, revision by the authors, one or more additional rounds of review until acceptance, and finally production of the accepted manuscript for publication [2–4]. Digital workflows and modifications in editorial office operations can be useful in accelerating this process. For instance, the submission and re-
view processes can be performed in an entirely electronic fashion [5–7], and release of electronic copies of accepted manuscripts before publication of the print version [8] has become standard for many journals. Additional options, such as continuous online publication [9] or conversion to an onlineonly format [10], are also increasingly being considered. A reduction in publication turnaround time through such efforts can improve a journal’s impact factor [11] and influence authors to be more likely to submit a manuscript to a given journal [12]. The AJR is a leading journal in the field of radiology, having a circulation of nearly 25,000 and being highly competitive in its selection of submitted manuscripts for publication [13, 14]. Achieving as rapid a publica-
AJR:204, June 2015
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Metrics for Original Research Articles in the AJR tion turnaround time as possible would benefit not only the journal’s authors and readers, but also the radiology community in general. Toward this end, a heightened understanding of the timeline for review and publication of newly submitted manuscripts would be useful for guiding efforts to improve publication times, in addition to providing authors with key insights regarding the handling of their submissions. Therefore, in this study, our aim was to evaluate metrics pertaining to manuscripts submitted to the AJR, assessing the pathway from submission to publication including the reviewer allocation time, decisions rendered, timing of decisions rendered, and time to publication. Materials and Methods This retrospective study did not pertain to human subjects research and therefore did not require institutional review board approval. Data were obtained from the AJR’s Editorial Manager system (Aries Systems) through the assistance of the journal’s office staff. Initially, data were obtained for 982 manuscripts that underwent first submission between July 1, 2012, and December 31, 2012. Of these, submissions that were not unsolicited Original Research articles (i.e., Reviews, Technical Innovations, Clinical Perspectives, Editorials, Opinions, Commentaries, Letters to the Editor, and solicited Original Research articles) were excluded. Exclusion of these manuscripts left a final cohort of 696 unsolicited Original Research manuscripts submitted during this 6-month period that were evaluated in this study. This period was selected to allow sufficient time for not only review and revision of all manuscripts, but also for assignment of a final publication date for all accepted manuscripts in the analysis.
For the identified manuscripts, the following metrics were recorded: days for initial editorial office processing; days for manuscript review, which incorporated the time needed for solicitation of reviewers, completion of reviews, and rendering of a final decision; and decision rendered (Accept, Minor Revision, Major Revision, Reject, and Reject Without Review). A decision of Major Revision generally entails repeating a considerable portion of the work that preceded drafting the manuscript, such as acquiring additional data or repeating key aspects of the data analysis [15]. In comparison, a decision of Minor Revision generally entails modifications to the presentation of the existing data—for instance, clarification of methodologic details or acknowledging additional prior studies or limitations in the Discussion. These metrics were also recorded for each resubmission of the manuscript, along with the days needed for the authors to perform the revision. For accepted manuscripts, the time from acceptance to publication was recorded. Metrics for resubmitted manuscripts were stratified by whether related to a Major or Minor Revision, as well as by the round of resubmission since the first submission. Metrics regarding decisions rendered were also stratified by journal section or category. Metrics were initially assessed using standard summary statistics (Excel for Mac 2011, Microsoft). Subsequently, decisions rendered were compared between rounds of submission using McNemar tests, whereas days for editorial office processing, manuscript review, and author revisions were compared between rounds of submission and between manuscripts receiving various decisions using paired Wilcoxon and MannWhitney tests, respectively (MedCalc, version 9.1, MedCalc Software). Metrics pertaining to third resubmissions were not included in the testing for statistical significance given the inclusion of only
six manuscripts in that category. All comparisons are two-sided and considered statistically significant at p < 0.05.
Results Decisions Rendered Table 1 summarizes data regarding decisions rendered. For new submissions, decisions were as follows: Accept, 0.3%; Minor Revision, 8.5%; Major Revision, 19.7%; Reject, 65.1%; and Reject Without Review, 6.5%. For both first and second resubmissions, there were more Accept (40.0–94.7%) or Minor Revision (2.6–40.0%) decisions and fewer Major Revision (0.0–10.0%) or Reject (1.7–10.4%) decisions. Furthermore, for both the first and second resubmissions, roughly half of manuscripts representing a Major Revision and more than 90% of manuscripts representing a Minor Revision were accepted. In comparison, approximately 10% of manuscripts representing a Major Revision and less than 3% of manuscripts representing a Minor Revision were rejected. Among the small number of manuscripts undergoing a third resubmission, 100% (6/6) were accepted. Across all rounds of review, authors revised and resubmitted 99.0% (102/103) and 97.3% (145/149) of manuscripts invited for Minor and Major Revision, respectively. Ultimately, 98.3% (58/59) and 84.7% (116/137) of manuscripts receiving a decision of Minor or Major Revision at first submission, respectively, achieved a final acceptance after one or more subsequent rounds of revision. Overall, 25.3% (n = 176) of the 696 Original Research articles in this analysis achieved eventual acceptance, once considering all rounds of resubmission. Table 2 shows the distribution of
TABLE 1: Distribution of Decisions Rendered Stratified by Manuscript Submission Number and Whether a Minor or Major Revision Previous Decision First Resubmission Decision Rendered
Second Resubmission
Third Resubmission
New Submission
Major Revision
Minor Revision
Major Revision
Minor Revision
Major Revision
Minor Revision
0.3 (2/696)
55.2 (74/134)
91.5 (54/59)
40.0 (4/10)
94.7 (36/38)
100.0 (1/1)
100.0 (5/5)
Minor Revision
8.5 (59/696)
26.1 (35/134)
6.8 (4/59)
40.0 (4/10)
2.6 (1/38)
0.0 (0/1)
0.0 (0/5)
Major Revision
19.7 (137/696)
8.2 (11/134)
0.0 (0/59)
10.0 (1/10)
0.0 (0/38)
0.0 (0/1)
0.0 (0/5)
Reject
65.1 (453/696)
10.4 (14/134)
1.7 (1/59)
10.0 (1/10)
2.6 (1/38)
0.0 (0/1)
0.0 (0/5)
6.5 (45/696)
—
—
—
—
—
—
696
134
59
10
38
1
5
Decision rendered Accept
Reject Without Review Total no. of manuscripts
Note—Unless indicated otherwise, data are presented as percentages (no. of manuscripts/total no. of manuscripts). Dash (—) indicates not applicable.
AJR:204, June 2015 1153
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Rosenkrantz and Harisinghani Revision (range of mean values, 9.8 ± 11.3 to 21.1 ± 15.3 days). Overall when considering the various components of the review and resubmission pro cess, the mean time from the date of the first submission for a new manuscript to the date of final acceptance after all rounds of revision was 128.6 ± 83.8 days (range, 6–539 days; median, 115 days). The mean time from final acceptance to publication in the print version of the journal was 242.5 ± 47.5 days.
rendered. In particular, for resubmissions, manuscripts receiving a decision of Accept required less time for review (p ≤ 0.008) than manuscripts receiving other decisions (range of mean values of 2.3 ± 2.6 to 11.9 ± 13.2 days vs range of mean values of 9.0 ± 0.0 to 28.9 ± 13.8 days, respectively). Table 3 also shows the time required for authors to complete manuscript revisions. This time decreased for subsequent rounds of resubmission (p < 0.001) from a range of 21.1 ± 15.3 to 73.7 ± 65.1 days for first resubmissions to a range of 9.8 ± 11.3 to 27.0 ± 0.0 days for third resubmissions, although these times varied substantially depending on the previous decision. In particular, authors required considerably more time (p < 0.001) for responding to a previous decision of Major Revision (range of mean values, 27.0 ± 0.0 to 73.7 ± 65.1 days) rather than Minor
decisions rendered for new submissions and final acceptance rate after all rounds of revision stratified by journal section or category. Timelines The mean time for initial editorial office processing after submission was shorter (p < 0.001) for new submissions (mean ± SD, 3.5 ± 7.6 days) than for resubmissions (8.7 ± 7.1, 8.2 ± 7.2, and 7.3 ± 2.3 days, for first, second, and third resubmissions, respectively). Table 3 shows the time required for subsequent manuscript review stratified by the submission number and decision rendered. This time consistently decreased with each subsequent round of resubmission after the first submission (p < 0.001), from 30.5 ± 43.1 to 42.7 ± 27.4 days for new submissions to 6.3 ± 6.3 days for third resubmissions, although the time also varied by the decision
Discussion In this study, we evaluated metrics pertaining to manuscripts submitted to the AJR from first submission through eventual publication. Our observations regarding decisions rendered should be reassuring to authors given that among new research articles invited for revision, the large majority were ultimate-
TABLE 2: Distribution of Decisions Rendered for New Submissions and Final Acceptance Rate After All Rounds of Revision Stratified by Journal Section or Category Decision No. of Manuscripts
Section or Category
Accept
Minor Revision
Major Revision
Reject Without Review
Reject
Final Acceptance Rate
Section Cardiopulmonary Imaging
98
0.0 (0/98)
3.1 (3/98)
13.3 (13/98)
81.6 (80/98)
2.0 (2/98)
16.3 (16/98)
Gastrointestinal Imaging
123
0.0 (0/123)
7.3 (9/123)
14.6 (18/123)
75.6 (93/123)
2.4 (3/123)
22.0 (27/123)
Genitourinary Imaging
42
0.0 (0/42)
4.8 (2/42)
38.1 (16/42)
57.1 (24/42)
0.0 (0/42)
40.5 (17/42)
Health Care Policy and Quality
14
0.0 (0/14)
28.6 (4/14)
28.6 (4/14)
28.6 (4/14)
14.3 (2/14)
57.1 (8/14)
Medical Physics and Informatics
32
6.3 (2/32)
15.6 (5/32)
18.8 (6/32)
31.3 (10/32)
28.1 (9/32)
40.6 (13/32)
Musculoskeletal Imaging
77
0.0 (0/77)
3.9 (3/77)
20.8 (16/77)
72.7 (56/77)
2.6 (2/77)
19.5 (15/77)
Neuroradiology/Head and Neck Imaging
68
0.0 (0/68)
5.9 (4/68)
30.9 (21/68)
60.3 (41/68)
2.9 (2/68)
32.4 (22/68)
Nuclear Medicine and Molecular Imaging
23
0.0 (0/23)
8.7 (2/23)
4.3 (1/23)
87.0 (20/23)
0.0 (0/23)
13.0 (3/23)
Pediatric Imaging
44
0.0 (0/44)
20.5 (9/44)
15.9 (7/44)
63.6 (28/44)
0.0 (0/44)
31.8 (14/44)
Vascular and Interventional Radiology
69
0.0 (0/69)
8.7 (6/69)
14.5 (10/69)
72.5 (50/69)
4.3 (3/69)
18.8 (13/69)
Women’s Imaging
73
0.0 (0/73)
8.2 (6/73)
27.4 (20/73)
39.7 (29/73)
24.7 (18/73)
26.0 (19/73)
33
0.0 (0/33)
18.2 (6/33)
15.2 (5/33)
54.5 (18/33)
12.1 (4/33)
27.3 (9/33)
Categories Special Articles and Other
Note—Unless indicated otherwise, data are presented as percentages (no. of manuscripts/total no. of manuscripts).
TABLE 3: Time Required for Various Aspects of Manuscript Review and Revision Process Time for Authors to Prepare Resubmission (d) Manuscript New submission
Minor Revision
Time for Reviewers’ and Section Editor’s Review, Stratified by Decision Rendered (d)
Major Revision
Accept
Minor Revision
Major Revision
Reject
Reject Without Review
—
—
30.5 ± 43.1
42.7 ± 27.4
39.4 ± 17.6
40.2 ± 20.3
2.3 ± 4.2
First resubmission
21.1 ± 15.3
73.7 ± 65.1
11.9 ± 13.2
21.8 ± 19.9
28.9 ± 13.8
25.6 ± 15.5
—
Second resubmission
11.1 ± 8.7
57.1 ± 69.8
2.3 ± 2.6
18.6 ± 14.8
9.0 ± 0.0
10.0 ± 12.7
—
Third resubmission
9.8 ± 11.3
27.0 ± 0.0
6.3 ± 6.3
—
—
—
—
Note—Data are mean ± SD. Dash (—) indicates not applicable.
1154
AJR:204, June 2015
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Metrics for Original Research Articles in the AJR ly accepted, whether the revisions were major or minor. In addition, a final decision of Accept or Reject was reached after a single resubmission in almost 95% of included studies, highlighting an ability of reviewers and section editors to efficiently identify those articles meriting publication. Moreover, approximately 95% of studies requiring a second or third resubmission were accepted, suggesting that authors should view these additional rounds of revision as a favorable sign that indicates a desire by the journal to publish the article, albeit after additional modifications. Awareness of these trends is important for authors when evaluating reviewer comments and considering whether to prepare and submit a revised manuscript to the journal. Indeed, a revised manuscript was submitted in nearly 100% of opportunities to do so, signifying the level of encouragement felt by authors when provided this chance. Even if an Original Research manuscript is rejected by the AJR, past work indicates that these manuscripts are likely to eventually be accepted and published by an alternate journal [13], thereby allowing dissemination of the results of the authors’ research. We observed that 6.5% of new submissions were rejected without review. This decision generally occurs because of a technical matter detected by the editorial staff during the initial evaluation of the manuscript for adherence to International Committee of Medical Journal Editors and AJR author guidelines. The journal maintains comprehensive policies, made available to authors on the journal website, regarding article categories, authorship requirements, content limits, and other details [16]. For instance, authors are required to submit a detailed cover letter providing information regarding any previous publications or conference presentations relating to the submitted manuscript along with copies of prior work. In addition, CrossCheck (CrossRef) is applied to all manuscripts to assess for overlap with previous publications, including those published by the authors themselves [17]. Authors can play an active role in ensuring timely handling of their new manuscripts by being aware of these myriad issues and taking sufficient care to follow all journal policies [18]. Authors should also be aware that a manuscript may be rejected without review if the manuscript content is not well suited for the intended general audience of the AJR, such as a highly technical research paper that may better fit a subspecialty journal. For manuscripts proceeding to review, the reviews took more time when requesting re-
visions than when accepting the submission, consistent with the expected lengthier and more complex nature of a review that identifies weaknesses of a manuscript as well as specific areas for improvement. Moreover, authors took more time for preparing revisions that were Major in nature, consistent as well with the greater effort inherent in responding to more extensive comments. Both of these areas suggest potential opportunities to shorten the overall process, namely by reducing the amount of time allowed for these steps. As of this writing, the journal allows 14 business days for completion of a review and 6 months for completion of a Major Revision. Regarding the allowed review time, reviewers may decline an invitation to review a manuscript if unable to complete the review by the specified deadline. Regarding the allowed revision time, authors are typically eager to have their work accepted and will make all efforts possible to revise a manuscript within the provided window. Thus, we anticipate that the time allotments for these steps could be reasonably shortened to accelerate the overall turnaround time without impeding general journal operations. Although uncommon, a small number of manuscripts did require four rounds of submission before achieving acceptance. These serial rounds of revision and further review inherently prolong the publication process, with one manuscript requiring approximately 1.5 years to undergo these multiple rounds of review. Although shortening the time allowed each round, as noted previously, will yield a more efficient process, an additional consideration is to limit either the number of allowed revisions or the number of opportunities for Major Revision for an individual manuscript. This approach would require authors to strive to meaningfully address all reviewer comments at the earliest possible opportunity during the revision cycle and ideally mitigate the frequency of manuscripts that become outliers in terms of requiring repetitive rounds of revision and associated greatly prolonged publication times. In particular, authors need to be sure to read and understand the reviewer comments and to address these comments through changes to the manuscript and in the rebuttal letter [15, 19]. Ultimately, the longest component of the total turnaround time was the time from acceptance to publication, taking more than 6 months on average. In the current era of digital communications [20, 21], in which social media and other web applications allow nearly instantaneous dissemination of information and opinions, it is critical that this time be substan-
tially shortened. Although there are inherent delays in the time required for preparation and production of the print version, workflow redesigns can be pursued to accelerate the process by which accepted manuscripts become available in electronic format. Indeed, most physicians today rely solely on electronic versions of journals for accessing published manuscripts [20]. Ideally, it will be possible to make all accepted manuscripts available in electronic form shortly after acceptance to make the results available to the radiology community in as timely a fashion as possible even if there are additional delays until the article eventually appears in a print issue. One further consideration for shortening the publication time is to reduce the percentage of submitted manuscripts receiving acceptance. This action would reduce the backlog of manuscripts awaiting publication, thereby allowing accepted manuscripts to move through the production process more quickly. To reduce the fraction of accepted manuscripts would pose a challenge given the large number of high-quality manuscripts submitted to the AJR but would be anticipated to further increase the quality of accepted manuscripts, as judged by the journal’s reviewers. In this study, we observed an overall acceptance rate of 25.3%, which essentially matches the target that the journal provides to reviewers of recommending acceptance for the top 25% of manuscripts reviewed. Nevertheless, a lower targeted acceptance rate would effectively reduce any accumulation of accepted manuscripts awaiting publication and be expected to directly yield shorter publication times. This analysis has a number of limitations. First, although we have made broad suggestions regarding potential approaches to accelerate the publication process, the distribution in decisions rendered was variable among journal sections and categories. Nevertheless, we believe that the most powerful reforms to produce a meaningful across-the-board reduction in publication times will likely pertain to all of the journal’s sections in a global fashion. In addition, we assessed publication times for only a single imaging journal. To our knowledge, external benchmarks for performance have not been established, and we have not analyzed how the observed turnaround times compare with those from other journals in the field. Finally, although we have suggested a number of ways in which publication times may be shortened, we have not tested or implemented these ideas and their actual impact in practice remains uncertain.
AJR:204, June 2015 1155
Downloaded from www.ajronline.org by NYU Langone Med Ctr-Sch of Med on 07/03/15 from IP address 128.122.253.228. Copyright ARRS. For personal use only; all rights reserved
Rosenkrantz and Harisinghani In conclusion, we have summarized the pathway followed by unsolicited Original Research manuscripts in the AJR from first submission to final publication. We hope that this information will provide useful insights to authors submitting manuscripts to the AJR and to AJR editorial staff in ongoing efforts to shorten publication turnaround times. References 1. Palese A, Coletti S, Dante A. Publication efficiency among the higher impact factor nursing journals in 2009: a retrospective analysis. Int J Nurs Stud 2013; 50:543–551 2. Cornelius JL. Reviewing the review process: identifying sources of delay. Australas Med J 2012; 5:26–29 3. Kljaković-Gaspić M, Hren D, Marusić A, Marusić M. Peer review time: how late is late in a small medical journal? Arch Med Res 2003; 34:439–443 4. Tite L, Schroter S. Why do peer reviewers decline to review? A survey. J Epidemiol Community Health 2007; 61:9–12 5. Shelton JS, Beach J. Electronic manuscript submission and peer review. Prim Care Companion J
1156
Clin Psychiatry 2004; 6:232–233 6. Govender P, Buckley O, McAuley G, O’Brien J, Torreggiani WC. Does online submission of manuscripts improve efficiency? JBR-BTR 2008; 91:231–234 7. Schroeder TV, Beard JD. Web manuscript submission and tracking system for peer review: the journal moves with the time. Eur J Vasc Endovasc Surg 2002; 23:95–96 8. Heneberg P. Effects of print publication lag in dual format journals on scientometric indicators. PLoS ONE 2013; 8:e59877 9. Anderton S, Harvey L. Continuous publication: ready, set, cite! Br J Radiol 2013; 86:20130252 10. Berquist TH. Print versus online: the debate goes on! AJR 2010; 195:815–816 11. Labanaris AP, Vassiliadu AP, Polykandriotis E, Tjiawi J, Arkudas A, Horch RE. Impact factors and publication times for plastic surgery journals. Plast Reconstr Surg 2007; 120:2076–2081 12. Soreide K, Winter DC. Global survey of factors influencing choice of surgical journal for manuscript submission. Surgery 2010; 147:475–480 13. Chew FS. Fate of manuscripts rejected for publication in the AJR. AJR 1991; 156:627–632
14. American Roentgen Ray Society website. About AJR. www.arrs.org/Publications/AJR/about.aspx. Accessed October 12, 2014 15. Provenzale JM. Revising a manuscript: ten principles to guide success for publication. AJR 2010; 195:[web]W382–W387 16. American Roentgen Ray Society. AJR guidelines for authors: Rev 09.2014. www.arrs.org/uploadedFiles/ ARRS/Publications/authorGuidelines.pdf. Accessed October 12, 2014 17. Berquist TH. Scientific integrity: has CrossCheck made a difference? AJR 2013; 201:703 18. Berquist TH. Submission to final decision: what can authors do to facilitate the process? AJR 2014; 202:1165 19. Provenzale JM. Ten principles to improve the likelihood of publication of a scientific manuscript. AJR 2007; 188:1179–1182 20. Kanne JP. The digital media revolution: what it means for the AJR. AJR 2011; 197:6–7 21. Hawkins CM, Duszak R, Rawson JV. Social media in radiology: early trends in Twitter micro blogging at radiology’s largest international meeting. J Am Coll Radiol 2014; 11:387–390
AJR:204, June 2015
