EDITORIAL
The impact factor: yesterday’s metric?
F. S. Haddad From The British Editorial Society of Bone & Joint Surgery, London, United Kingdom
F. S. Haddad, MD(Res), MCh(Orth), FRCS(Orth), Professor of Orthopaedic Surgery, Editor-in-Chief The Bone & Joint Journal, 22 Buckingham Street, London, WC2N 6ET, UK. Correspondence should be sent to Professor F. S. Haddad: [email protected] ©2014 The British Editorial Society of Bone & Joint Surgery doi:10.1302/0301-620X.96B3. 33905 $2.00 Bone Joint J 2014;96-B:289–90.
VOL. 96-B, No. 3, MARCH 2014
One of the core objectives of The Bone & Joint Journal (BJJ) remains to publish the best, highest impact and most clinically relevant articles possible. We believe that we cover the most clinically challenging areas and that many of our papers change practice.1-9 Yet this does not automatically translate to a traditional twoyear impact factor (IF) high enough to reflect the importance of the material published. The impact factor is embedded in academic thinking, not least because it is easy to calculate and most people can understand the concept. Unfortunately, the IF does not fully measure the true clinical and practical impact of journals like BJJ that have a largely clinical readership, many of whom read the articles and change their day-to-day practice based on them, but don't cite them. The impact factor remains an area of great debate amongst researchers, publishers and academic institutions. Journals in Trauma and Orthopaedics, along with other surgical specialties, tend to have lower IFs in comparison to specialties with higher basic science and less craft emphasis. Impact factors are used as part of promotion and tenure decisions. They are used by researchers for selecting which journal to submit to, they are used by university administrators for ranking academic and research programmes within and across universities, and they are used by grant giving bodies as a way of assessing investigators. Impact factors are used by journals and their publishers to attract subscriptions, sponsored supplements and advertising. There are some very high-ranking clinical journals that have a much lower circulation and less clinical relevance to most practitioners than the BMJ (IF 2012: 17.215), which is more widely read, such as New England Journal of Medicine (51.658), The Lancet (39.060) and JAMA (29.978). By comparison, the highest ranked orthopaedic journal is currently American Journal of Sports Medicine (4.439), now just ahead of the much less clinical Osteoarthritis and Cartilage (4.262) which held the top
spot in this category for many years. On the other hand the translational effects of the work in BJJ and its predecessor JBJS Br (2.735), such as product patent and clinical practice change, are massive. Therefore the impact factor per se doesn’t truly reflect the clinical influence it has. In this issue, Kodumuri et al10 evaluated the impact of 13 journals in Trauma and Orthopaedics looking at the longer-term impact of their publications. Surgical papers had a lower citation rate at two years compared with basic science or clinical medical papers but, by four years, the citation rates were similar. This reinforces the view that a longer period is required for citation rate / IF to be appropriate for clinical surgery, and that the standard two year IF as it stands may not be the best way to judge a product such as BJJ. Current limitations of impact factor include: 1) A positive discrimination towards journals publishing review articles; these tend to be more highly cited than original research articles. 2) There is wide disparity in IF across different scientific disciplines depending on the method of citation. For example, Mathematics researchers rarely cite more than one or two references, whereas a typical paper in molecular biology cites more than ten. 3) The two year citation window used in the calculation method is considered too short to detect the real impact of the publication in slowly evolving disciplines such as orthopaedics. This is supported by the fact some types of journals are found to have higher impacts when increasing the two-year period used to calculate the impact factor to five years or more. For example, the five-year IF for BJJ (formerly that of JBJS Br) is 3.353 compared to 2.735 for the normal two year IF. 4) A high impact factor value of one journal does not correlate to the high citation rate of each article in that journal. A study in Nature in 2005 showed that 89% of their citations came from 25% of their articles.11 The concept of citable documents is not operationalised adequately. 289
290
F. S. HADDAD
The “non-source” items (for the denominator), such as editorials, book reviews, and letters to the editor, do not count as “citable articles”, yet citations to such “non-source” articles are counted in the numerator of the impact factor. 5) There is no normalisation for reference practices and traditions in different fields. 6) There is no distinction in regard to the quality and nature of the citing journals, a point addressed by an alternative metric, the Eigenfactor. 7) The impact factor also says nothing about the peer review process. 8) Impact factors may also be manipulated by authors and editors (i.e. self-citation, coercive citation). 9) Citation practices are also inconsistent. Some scientific articles tend to cite only scientific articles whereas clinical articles tend to cite both types. Measures have been taken to improve the situation. IFs can be compared between journals in the same field. There is also more transparency now around authors and journals that have a higher percentage of self-citation. Many also now accept that looking at five year data is more relevant than two year data. A fairer evaluation of research impact may be a new basket of metrics for journals, researchers, and institutions that take into account the way that the literature is accessed in 2014. We need to tackle the problem that the readership may vary between clinicians and scientists. The clinical impact is not citation-based and translation and knowledge transfer do not correlate with the IF. It is critical to refine the IF or look at ways of evaluating papers and journals in another way based on their role within any profession. We need to define metrics to look at the clinical impact of clinical and research papers that is not currently reflected by citation, and to strive for a balance between science and clinical practicality. Plans for a journal usage factor, which would rank journals by actual usage rather than citations, are well advanced, with trials by a number of leading scientific and medical publishers. We now also provide article-level metrics in BJJ from Altmetric, which show downloads, tweets, other social and news media mentions, citations etc. In addition, BJJ is a beta partner with other publishers developing a new service, Kudos, to be launched in May 2014. This will provide a simple toolkit for our authors to use
email and social media to promote their articles to their own research networks, as well as linking the article to other relevant sites - e.g. authors' own websites, conference programmes and presentations, videos etc. Meanwhile biomedical research funders are increasingly seeking to measure not only the academic impact of research in terms of citations, but the real-world effects on clinical practice and patient outcomes. This is apparent in the new assessment criteria of the new Research Excellence Framework (REF) in the UK, which include an assessment of the “reach and significance” of research as well as its originality and rigour. We will, in the short-term, continue to be judged against our peers using the current IF, but we will also continue to grapple with the best means of defining the genuine influence of any single piece of work or group of studies. This is another area where we will see significant change in the next few years. In the interim, we will not be diverted from our aim of publishing the most up-to-date, clinically relevant and practice changing work in trauma and orthopaedics.
References 1. Parker M, Cawley S, Palial V. Internal fixation of intracapsular fractures of the hip using a dynamic locking plate: Two-year follow-up of 320 patients. Bone Joint J 2013;95-B:1402–1405. 2. Abram SG, Pollard TC, Andrade AJ. Inadequate 'three-point' proximal fixation predicts failure of the Gamma nail. Bone Joint J 2013;95-B:825–830. 3. Parker L, Garlick N, McCarthy I, et al. Screw fixation of medial malleolar fractures: a cadaveric biomechanical study challenging the current AO philosophy. Bone Joint J 2013;95-B:1662–1666. 4. Inglis M, McClelland B, Sutherland LM, Cundy PJ. Synthetic versus plaster of Paris casts in the treatment of fractures of the forearm in children: a randomised trial of clinical outcomes and patient satisfaction. Bone Joint J 2013;95-B:1285– 1289. 5. Liddle AD, Pandit H, O'Brien S, et al. Cementless fixation in Oxford unicompartmental knee replacement: a multicentre study of 1000 knees. Bone Joint J 2013;95B:181–187. 6. Meding JB, Ritter MA, Davis KE, Farris A. Meeting increased demand for total knee replacement and follow-up: determining optimal follow-up. Bone Joint J 2013;95-B:1484–1489. 7. Rao BM, Kamal TT, Vafaye J, Moss M. Tantalum cones for major osteolysis in revision knee replacement. Bone Joint J 2013;95-B:1069-1074. :. 8. Sri-Ram K, Salmon LJ, Pinczewski LA, Roe JP. The incidence of secondary pathology after anterior cruciate ligament rupture in 5086 patients requiring ligament reconstruction. Bone Joint J 2013;95-B:59–64. 9. Balain B, Jaiswal A, Trivedi JM, et al. The Oswestry Risk Index: an aid in the treatment of metastatic disease of the spine. Bone Joint J 2013;95-B:210–216. 10. Kodumuri P, Ollivere B, Holley J, Moran CG. The impact factor of a journal is a poor measure of the clinical relevance of its papers. Bone Joint J 2014;96-B:414– 419. 11. No authors listed. Not-so-deep impact. Nature 2005;435:1003–1004.
THE BONE & JOINT JOURNAL
The impact factor: yesterday’s metric?
F. S. Haddad From The British Editorial Society of Bone & Joint Surgery, London, United Kingdom
F. S. Haddad, MD(Res), MCh(Orth), FRCS(Orth), Professor of Orthopaedic Surgery, Editor-in-Chief The Bone & Joint Journal, 22 Buckingham Street, London, WC2N 6ET, UK. Correspondence should be sent to Professor F. S. Haddad: [email protected] ©2014 The British Editorial Society of Bone & Joint Surgery doi:10.1302/0301-620X.96B3. 33905 $2.00 Bone Joint J 2014;96-B:289–90.
VOL. 96-B, No. 3, MARCH 2014
One of the core objectives of The Bone & Joint Journal (BJJ) remains to publish the best, highest impact and most clinically relevant articles possible. We believe that we cover the most clinically challenging areas and that many of our papers change practice.1-9 Yet this does not automatically translate to a traditional twoyear impact factor (IF) high enough to reflect the importance of the material published. The impact factor is embedded in academic thinking, not least because it is easy to calculate and most people can understand the concept. Unfortunately, the IF does not fully measure the true clinical and practical impact of journals like BJJ that have a largely clinical readership, many of whom read the articles and change their day-to-day practice based on them, but don't cite them. The impact factor remains an area of great debate amongst researchers, publishers and academic institutions. Journals in Trauma and Orthopaedics, along with other surgical specialties, tend to have lower IFs in comparison to specialties with higher basic science and less craft emphasis. Impact factors are used as part of promotion and tenure decisions. They are used by researchers for selecting which journal to submit to, they are used by university administrators for ranking academic and research programmes within and across universities, and they are used by grant giving bodies as a way of assessing investigators. Impact factors are used by journals and their publishers to attract subscriptions, sponsored supplements and advertising. There are some very high-ranking clinical journals that have a much lower circulation and less clinical relevance to most practitioners than the BMJ (IF 2012: 17.215), which is more widely read, such as New England Journal of Medicine (51.658), The Lancet (39.060) and JAMA (29.978). By comparison, the highest ranked orthopaedic journal is currently American Journal of Sports Medicine (4.439), now just ahead of the much less clinical Osteoarthritis and Cartilage (4.262) which held the top
spot in this category for many years. On the other hand the translational effects of the work in BJJ and its predecessor JBJS Br (2.735), such as product patent and clinical practice change, are massive. Therefore the impact factor per se doesn’t truly reflect the clinical influence it has. In this issue, Kodumuri et al10 evaluated the impact of 13 journals in Trauma and Orthopaedics looking at the longer-term impact of their publications. Surgical papers had a lower citation rate at two years compared with basic science or clinical medical papers but, by four years, the citation rates were similar. This reinforces the view that a longer period is required for citation rate / IF to be appropriate for clinical surgery, and that the standard two year IF as it stands may not be the best way to judge a product such as BJJ. Current limitations of impact factor include: 1) A positive discrimination towards journals publishing review articles; these tend to be more highly cited than original research articles. 2) There is wide disparity in IF across different scientific disciplines depending on the method of citation. For example, Mathematics researchers rarely cite more than one or two references, whereas a typical paper in molecular biology cites more than ten. 3) The two year citation window used in the calculation method is considered too short to detect the real impact of the publication in slowly evolving disciplines such as orthopaedics. This is supported by the fact some types of journals are found to have higher impacts when increasing the two-year period used to calculate the impact factor to five years or more. For example, the five-year IF for BJJ (formerly that of JBJS Br) is 3.353 compared to 2.735 for the normal two year IF. 4) A high impact factor value of one journal does not correlate to the high citation rate of each article in that journal. A study in Nature in 2005 showed that 89% of their citations came from 25% of their articles.11 The concept of citable documents is not operationalised adequately. 289
290
F. S. HADDAD
The “non-source” items (for the denominator), such as editorials, book reviews, and letters to the editor, do not count as “citable articles”, yet citations to such “non-source” articles are counted in the numerator of the impact factor. 5) There is no normalisation for reference practices and traditions in different fields. 6) There is no distinction in regard to the quality and nature of the citing journals, a point addressed by an alternative metric, the Eigenfactor. 7) The impact factor also says nothing about the peer review process. 8) Impact factors may also be manipulated by authors and editors (i.e. self-citation, coercive citation). 9) Citation practices are also inconsistent. Some scientific articles tend to cite only scientific articles whereas clinical articles tend to cite both types. Measures have been taken to improve the situation. IFs can be compared between journals in the same field. There is also more transparency now around authors and journals that have a higher percentage of self-citation. Many also now accept that looking at five year data is more relevant than two year data. A fairer evaluation of research impact may be a new basket of metrics for journals, researchers, and institutions that take into account the way that the literature is accessed in 2014. We need to tackle the problem that the readership may vary between clinicians and scientists. The clinical impact is not citation-based and translation and knowledge transfer do not correlate with the IF. It is critical to refine the IF or look at ways of evaluating papers and journals in another way based on their role within any profession. We need to define metrics to look at the clinical impact of clinical and research papers that is not currently reflected by citation, and to strive for a balance between science and clinical practicality. Plans for a journal usage factor, which would rank journals by actual usage rather than citations, are well advanced, with trials by a number of leading scientific and medical publishers. We now also provide article-level metrics in BJJ from Altmetric, which show downloads, tweets, other social and news media mentions, citations etc. In addition, BJJ is a beta partner with other publishers developing a new service, Kudos, to be launched in May 2014. This will provide a simple toolkit for our authors to use
email and social media to promote their articles to their own research networks, as well as linking the article to other relevant sites - e.g. authors' own websites, conference programmes and presentations, videos etc. Meanwhile biomedical research funders are increasingly seeking to measure not only the academic impact of research in terms of citations, but the real-world effects on clinical practice and patient outcomes. This is apparent in the new assessment criteria of the new Research Excellence Framework (REF) in the UK, which include an assessment of the “reach and significance” of research as well as its originality and rigour. We will, in the short-term, continue to be judged against our peers using the current IF, but we will also continue to grapple with the best means of defining the genuine influence of any single piece of work or group of studies. This is another area where we will see significant change in the next few years. In the interim, we will not be diverted from our aim of publishing the most up-to-date, clinically relevant and practice changing work in trauma and orthopaedics.
References 1. Parker M, Cawley S, Palial V. Internal fixation of intracapsular fractures of the hip using a dynamic locking plate: Two-year follow-up of 320 patients. Bone Joint J 2013;95-B:1402–1405. 2. Abram SG, Pollard TC, Andrade AJ. Inadequate 'three-point' proximal fixation predicts failure of the Gamma nail. Bone Joint J 2013;95-B:825–830. 3. Parker L, Garlick N, McCarthy I, et al. Screw fixation of medial malleolar fractures: a cadaveric biomechanical study challenging the current AO philosophy. Bone Joint J 2013;95-B:1662–1666. 4. Inglis M, McClelland B, Sutherland LM, Cundy PJ. Synthetic versus plaster of Paris casts in the treatment of fractures of the forearm in children: a randomised trial of clinical outcomes and patient satisfaction. Bone Joint J 2013;95-B:1285– 1289. 5. Liddle AD, Pandit H, O'Brien S, et al. Cementless fixation in Oxford unicompartmental knee replacement: a multicentre study of 1000 knees. Bone Joint J 2013;95B:181–187. 6. Meding JB, Ritter MA, Davis KE, Farris A. Meeting increased demand for total knee replacement and follow-up: determining optimal follow-up. Bone Joint J 2013;95-B:1484–1489. 7. Rao BM, Kamal TT, Vafaye J, Moss M. Tantalum cones for major osteolysis in revision knee replacement. Bone Joint J 2013;95-B:1069-1074. :. 8. Sri-Ram K, Salmon LJ, Pinczewski LA, Roe JP. The incidence of secondary pathology after anterior cruciate ligament rupture in 5086 patients requiring ligament reconstruction. Bone Joint J 2013;95-B:59–64. 9. Balain B, Jaiswal A, Trivedi JM, et al. The Oswestry Risk Index: an aid in the treatment of metastatic disease of the spine. Bone Joint J 2013;95-B:210–216. 10. Kodumuri P, Ollivere B, Holley J, Moran CG. The impact factor of a journal is a poor measure of the clinical relevance of its papers. Bone Joint J 2014;96-B:414– 419. 11. No authors listed. Not-so-deep impact. Nature 2005;435:1003–1004.
THE BONE & JOINT JOURNAL
