J Neurosurg 120:746–755, 2014 ©AANS, 2014

An analysis of publication productivity for 1225 academic neurosurgeons and 99 departments in the United States Clinical article Nickalus R. Khan, M.D.,1 Clinton J. Thompson, M.S., 2 Douglas R. Taylor, B.S., 3 Garrett T. Venable, B.Sc., 3 R. Matthew Wham, B.Sc., 3 L. Madison Michael II, M.D.,1,4 and Paul Klimo Jr., M.D., M.P.H.1,4 Department of Neurosurgery, 3University of Tennessee Health Science Center; 4Semmes-Murphey Neurologic & Spine Institute, Memphis, Tennessee; and 2School of Public Health and Health Services, The George Washington University, Washington, DC

1

Object. Bibliometrics is defined as the study of statistical and mathematical methods used to quantitatively analyze scientific literature. The application of bibliometrics in neurosurgery is in its infancy. The authors calculate a number of publication productivity measures for almost all academic neurosurgeons and departments within the US. Methods. The h-index, g-index, m-quotient, and contemporary h-index (hc-index) were calculated for 1225 academic neurosurgeons in 99 (of 101) programs listed by the Accreditation Council for Graduate Medical Education in January 2013. Three currently available citation databases were used: Google Scholar, Scopus, and Web of Science. Bibliometric profiles were created for each surgeon. Comparisons based on academic rank (that is, chairperson, professor, associate, assistant, and instructor), sex, and subspecialties were performed. Departments were ranked based on the summation of individual faculty h-indices. Calculations were carried out from January to February 2013. Results. The median h-index, g-index, hc-index, and m-quotient were 11, 20, 8, and 0.62, respectively. All indices demonstrated a positive relationship with increasing academic rank (p < 0.001). The median h-index was 11 for males (n = 1144) and 8 for females (n = 81). The h-index, g-index and hc-index significantly varied by sex (p < 0.001). However, when corrected for academic rank, this difference was no longer significant. There was no difference in the m-quotient by sex. Neurosurgeons with subspecialties in functional/epilepsy, peripheral nerve, radiosurgery, neurooncology/skull base, and vascular have the highest median h-indices; general, pediatric, and spine neurosurgeons have the lowest median h-indices. By summing the manually calculated Scopus h-indices of all individuals within a department, the top 5 programs for publication productivity are University of California, San Francisco; Barrow Neurological Institute; Johns Hopkins University; University of Pittsburgh; and University of California, Los Angeles. Conclusions. This study represents the most detailed publication analysis of academic neurosurgeons and their programs to date. The results for the metrics presented should be viewed as benchmarks for comparison purposes. It is our hope that organized neurosurgery will adopt and continue to refine bibliometric profiling of individuals and departments. (http://thejns.org/doi/abs/10.3171/2013.11.JNS131708)

A

Key Words      •      h-index      •      academic neurosurgery      •      bibliometrics      •      citation impact      •      Scopus      •      Google Scholar      •      Web of Science      •      g-index      •    contemporary h-index      •      m-quotient

cademic advancement in medicine is dependent on multiple factors, such as clinical volume and outcomes, teaching, board certification, number of years in practice, membership in organizations, participation in administrative duties, acquisition of funding, and conference presentations. Research and the end result of one’s work—publishing—have been shown to be one of the most important determinants of promotion.3,8,10,48 The simplest method of quantifying one’s research activity is to count publications; however, not all publications should be considered equal. A book chapter, case report, or review article should not be given the same weight as original research. Among original research, how does one differentiate papers that have substantial impact on a per-

746

son’s specialty against those that do not? In essence, can we establish research quality and impact, and if so, how? Bibliometrics is defined as the application of statistical and mathematical methods to quantitatively analyze scholarly documents in an effort to establish indicators of research performance.11,18,35,55 Bibliometrics is heavily based on citations analysis, which is the study of references cited in the bibliographies of scholarly publications. Many metrics have been introduced to evaluate academic productivity, but none is more famous or controversial than the h-index.6,16 The h-index was first described in 2005 by physicist Jorge E. Hirsch from the University of California, San Diego.28 The h-index is defined as an individual having h papers with at least h citations. For J Neurosurg / Volume 120 / March 2014

Bibliometrics in neurosurgery example, if an author has 20 papers cited at least 20 times his or her h-index would be 20. This simple yet intuitively descriptive index has generated much discussion in various fields in medicine, including anesthesiology,36 hepatology,39 otolaryngology,47 radiation oncology,40 radiology,41 surgery, 50 urology,9 and neurosurgery.2,33,38,45 As of July 2013, Hirsch’s landmark paper has been cited almost 3200 times (per Google Scholar). Although the h-index is the most well-known citation metric, it has a number of weaknesses.5,25,43,52 For example, the h-index favors senior researchers since h-index numbers can never decrease; the index also relies on time for researchers to generate a sufficient number of papers and then more time for those papers to be cited. As a result, other metrics have been proposed to counteract the shortcomings of the h-index.1 The contemporary h-index (hc-index)44 and m-quotient28 are examples of metrics that attempt to establish parity when young researchers are compared with more seasoned ones. The hc-index described by Antonis Sidiropoulos et al. in 2006 considers the age of publications and assigns more citations to recent publications.44 The m-quotient is defined as the hindex divided by the number of years since the author’s first publication. Once a publication is counted toward the h-index, it will have no further impact on the h-index, despite garnering more subsequent citations. Thus, many argue that the h-index values quantity more so than quality. The g-index19,20 and e-index54 are metrics that are able to differentiate scientists with highly cited works that the h-index fails to capture. An alternative is Google’s i10 index; it is the number of articles with 10 or more citations. Another drawback of the h-index is that it is dependent on the size of the field. Researchers in small fields—such as neurosurgery—with consequently a smaller readership will, in general, have lower indices than researchers in fields with larger audiences, such as general medicine.33 Thus, the h-index should not be used to compare researchers of different areas of expertise. The h-index is also susceptible to self-citation and counts review articles, which are often highly cited, equal to original research and gives equal value to all authors listed on a publication. Citation analysis and metrics are founded in databases, of which there are currently 3. Prior to 2004, Thomson Reuters’ Web of Science was the only database available. In 2004, both Elsevier’s Scopus and Google Scholar were released. The h-index was included as an indicator in Web of Science, Scopus, and Google Scholar within 2 years of its publication. The h-index, other metrics, and the citation analysis that creates them have been shown to vary depending on which database is used and, depending on how the search is performed, even within each database.4,23,32 Each database has its advantages and disadvantages. For example, Google Scholar is free, updated several times a week, and has a broader coverage than both Scopus and Web of Science. However, Google Scholar does not provide a list of its sources and includes citations in non–peer-reviewed publications, such as conference proceedings, white papers, and books.27 This article evaluates almost all of academic neurosurgery (99 of 101 departments and 1225 academicians) to benchmark the h-index, m-quotient, g-index, and conJ Neurosurg / Volume 120 / March 2014

temporary h-index across academic ranks, departments, and subspecialties using all 3 currently available databases (Scopus, Google Scholar, and Web of Science). Selection of Programs

Methods

A listing of the 2012 neurosurgery residency–training programs was obtained from the Accreditation Council for Graduate Medical Education (https://www.acgme.org/ads/ Public/Reports/ReportRun?ReportId=1&CurrentYear=201 2&SpecialtyId=35). Departmental websites were consulted for names, academic ranks, and subspecialties. Nonneurosurgical faculty (for example, neurologists, non–M.D. Ph.D.s, and radiologists) were excluded from this study. If all relevant information could not be obtained from the department’s website, we contacted the department via email or telephone. Two programs were excluded (Cleveland Clinic and Walter Reed). This was due to insufficient information obtainable from departmental websites and unsuccessful subsequent attempts to gain this information. Calculations were carried out from January to February 2013.

Definition of Citation Metrics

h-Index. The h-index is defined as an individual having h papers with at least h citations. In other words, it corresponds to the point where the number of citations crosses the publications listed in decreasing order of citations. m-Quotient. The m-quotient is the h-index divided by the number of years since the author’s first publication.

hc-Index. The contemporary h-index is derived by multiplying the citation count of article by 4, then dividing by the number of years since publication. Thus, the number of citations an article published in this year (2013) would be multiplied by 4, a paper from 4 years ago would have its citation count multiplied by 1, and a paper from 6 years ago would have its citation count multiple by 4/6.

g-Index. With articles ranked in decreasing order of the number of citations that they received, the g-index is the largest number such that the top g articles received (together) at least g2 citations. Calculation of Citation Metrics by Citation Database

Scopus. The automated h-index from Scopus (http:// www.scopus.com) was obtained using the “Author Search” function. Because Scopus does not count citations prior to 1996, a “manually calculated” h-index was calculated for each individual by looking at each of the author’s manuscripts (accounting for citations prior to 1996 in Scopus). The m-quotient was calculated by dividing Scopus’ manually calculated h-index by the years since the first publication. Google Scholar. Harzing’s Publish or Perish (http:// www.harzing.com/pop.htm) application was used to access Google Scholar for h-index, g-index, and contemporary h-index (hc). Publish or Perish uses the Advanced Scholar Search capabilities of Google Scholar.26 747

N. R. Khan et al. Web of Science. The Web of Science (http://wokinfo. com/products_tools/multidisciplinary/webofscience/) was used to determine an individual’s h-index using the “Author” search function for each individual author. An author’s first and last initials were used within search strings. Careful examination of the results from each search was performed to determine if the author had a preferred way of listing his or her initials. Further analysis was performed on each search result to determine if, indeed, it represented the individual being searched. This method included looking at the titles of articles, titles of journals, affiliations of authors, and in some instances reading the articles published. Citation metrics were then summarized for groups of individuals based on academic rank (instructor, assistant, associate, professor, and chairperson), sex, and subspecialty (spine, pediatrics, neuro-oncology/skull base, vascular, general, functional/epilepsy, peripheral nerve, and radiosurgery). A mean departmental h-index was calculated using Scopus, Google Scholar, and Web of Science. We then ranked departments by summing the manually calculated Scopus h-indices of all individuals within a department. Statistical Analysis

The following a priori statistical comparisons were performed (mean values used): 1) h-index (Scopus, Google Scholar, Web of Science), m-quotient (Scopus), hc-index (Google Scholar), and g-index (Google Scholar) versus academic rank; 2) h-index (Scopus, Google Scholar, Web of Science), m-quotient (Scopus), hc-index (Google Scholar), and gindex (Google Scholar) versus sex (raw analysis and then stratified for academic rank); and 3) h-index (Scopus, Google Scholar, Web of Science), m-quotient (Scopus), hc-index (Google Scholar), and g-index (Google Scholar) versus neurosurgical subspecialty. All statistics were calculated using SPSS (version 21, IBM SPSS). Significant values were considered to be p < 0.05. Mean values are presented with ± SD. Natural logarithms were used to transform nonparametric data.

Results Citation Metrics Based on Database Used

Data were obtained from 99 departmental websites for a total of 1225 academic neurosurgeons. The overall mean and median of the calculated citation metrics across all 3 databases are shown in Table 1. The distribution of all metrics was positively skewed so that the means are larger than the medians. h-Index. Using Scopus, the h-index was calculated for 1180 individuals. The mean manually calculated hindex was 14.6 ± 12.7 with a range of 0–76 and a median of 11. The mean automated Scopus h-index was 12.1 ± 10.4 with a range of 0–68 and a median of 9. For Google Scholar, 1210 neurosurgeons were analyzed. The mean hindex was 14.1 ± 12.9 with a range of 0–83 and a median of 10. Finally, using Web of Science, the mean value for

748

1216 neurosurgeons was 13 ± 13 with a range of 0–77 and a median of 9.

m-Quotient. The m-quotient was calculated for 1167 individuals using Scopus. The mean was 0.71 ± 0.47 (range of 0–3.67) and the median was 0.62.

hc-Index. The mean hc index was 9.3 ± 7.1 with a range of 0–42 and a median of 8 (Google Scholar, n = 1210). g-Index. The mean g-index was 26.5 ± 23.8 with a range of 0–131 and a median of 20 (Google Scholar, n = 1210). Citation Metrics by Groups of Individuals

Table 2 shows the mean and median h-indices for all 3 databases, m-quotient, hc-index, and g-index by academic rank, sex, and subspecialty. Academic Rank. Citation metric calculations were carried out for 99 chairs, 290 professors, 261 associate professors, 472 assistant professors, and 21 instructors. In general there was good agreement between the 3 databases. The range in average values for the h-index, mquotient, hc-index, and g-index between an instructor and chairman were 5–31, 0.51–1.02, 5–17, and 12–55, respectively. All citation metrics were positively correlated with increasing academic rank (Kruskal-Wallis, p < 0.001).

Sex. There were 1144 male and 81 female neurosurgeons. The h-index appeared to vary with sex for all 3 databases (2-tailed Mann-Whitney, p < 0.001); however, when corrected for academic rank, this difference was no longer significant (2-way ANOVA, p = 0.111, p = 0.291, and p = 0.423). The m-quotient did not vary with sex (2-tailed Mann-Whitney, p = 0.211). Similar to the h-index, the hc-index and g-index did not vary with sex when corrected for academic rank (2-way ANOVA, p = 0.167 and p = 0.380, respectively).

Subspecialty. There was a statistically significant difference in h-indices among the various neurosurgical subspecialties (Kruskal-Wallis, p < 0.001). The distribution of values seemed to cluster into 2 groups: 1) general (n = 106), pediatrics (n = 190), and spine (n = 317) with an average hindex range of 10–14; and 2) functional/epilepsy (n = 123), peripheral nerve (n = 17), radiosurgery (n = 43), neuro-oncology/skull base (n = 195), and vascular (n = 167) with an average h-index range of 15–19. The same differences were seen for the m-quotient (0.58–0.64 vs 0.70–0.87), hc-index (8–9 vs 10–13), and g-index (22–25 vs 29–35). Departmental h-Indices. Table 3 lists a ranking of the 99 programs based on the cumulative manually calculated Scopus h-index. Each department had at least 3 faculty members listed on their website. The top 5 programs were University of California, San Francisco; Barrow Neurological Institute; Johns Hopkins University; University of Pittsburgh; and University of California, Los Angeles.

Discussion

Bibliometrics (or citation metrics) is a powerful and useful tool. It allows one to quantitatively evaluate, in as J Neurosurg / Volume 120 / March 2014

Bibliometrics in neurosurgery TABLE 1: Mean and median of calculated citation metrics across the 3 databases used* Citation Metric Database

h-Index

m-Quotient

Scopus†  manual  automated GS (n = 1210) WOS (n = 1216)

14.6 [11] 12.1 [9] 14.1 [10] 13 [9]

0.71 [0.62]

hc-Index

g-Index

9.3 [8]

26.5 [20]

*  Mean values are listed outside the brackets. Median values are listed in brackets. GS = Google Scholar; WOS = Web of Science. †  The h-index was calculated for 1180 neurosurgeons and the m-quotient was calculated for 1167 neurosurgeons.

much detail as he or she would like, the scope, breadth, and impact of an author’s publication output. These metrics can be used to compare one’s achievements with those of his or her peers. It is a tool that can be used by a dean or department chair to determine, along with other factors, promotion or tenure. Some governments, particularly those with centralized and socialized health care delivery systems, are using citation analysis as a means of establishing research quality and impact. For example, the Australian government established the Research Quality Framework to determine the quality and impact of government-funded research, one method of which

was the use of quantitative metrics. Some “collectable” quality indicators included 1) the number of highly-cited articles published; 2) the number of articles in highquality journals; and 3) the number of citations of articles within articles published in high-quality journals.13 In 2004, INSERM (the French National Institutes of Health and Medical Research) introduced bibliometrics as part of its research assessment procedures.42 The use of citation metrics in neurosurgery is still in its infancy.2 It began in 2009 when Lee et al. demonstrated a positive correlation between academic rank and a small group of neurosurgeons from 30 programs.33 Two

TABLE 2: Mean and median citation metrics by groups of individuals* Citation Metric Grouping academic rank  chair  professor   associate professor   assistant professor  instructor   p value sex  male  female   p value subspecialty  vascular  functional/epilepsy  radiosurgery   neuro-onc/skull base  pediatrics   peripheral nerve  spine  general   p value

m-Quotient (Scopus†)

hc-Index (GS)

g-Index (GS)

31/29/27 [29/22/26] 24/24/23 [22/17/23] 12/12/11 [11/10/11] 8/7/7 [6/6/6] 7/5/7 [4/4/3]

An analysis of publication productivity for 1225 academic neurosurgeons and 99 departments in the United States.

Bibliometrics is defined as the study of statistical and mathematical methods used to quantitatively analyze scientific literature. The application of...
1MB Sizes 0 Downloads 0 Views