Original Research—General Otolaryngology
Do AAO-HNSF CORE Grants Predict Future NIH Funding Success? Jean Anderson Eloy, MD1,2,3, Peter F. Svider, MD4, Vivek V. Kanumuri1, Adam J. Folbe, MD4, Michael Setzen, MD5,6, and Soly Baredes, MD1,3
Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article.
Abstract Objective. To determine (1) whether academic otolaryngologists who have received an American Academy of Otolaryngology— Head and Neck Surgery Foundation (AAO-HNSF) Centralized Otolaryngology Research Efforts (CORE) grant are more likely to procure future National Institutes of Health (NIH) funding; (2) whether CORE grants or NIH Career Development (K) awards have a stronger association with scholarly impact. Study Design and Setting. Historical cohort. Methods. Scholarly impact, as measured by the h-index, publication experience, and prior grant history, were determined for CORE-funded and non–CORE-funded academic otolaryngologists. All individuals were assessed for NIH funding history. Results. Of 192 academic otolaryngologists with a CORE funding history, 39.6% had active or prior NIH awards versus 15.1% of 1002 non–CORE-funded faculty (P \ .0001). Higher proportions of CORE-funded otolaryngologists have received K-series and R-series grants from the NIH (P-values \ .05). K-grant recipients had higher h-indices than CORE recipients (12.6 vs 7.1, P \ .01). Upon controlling for rank and experience, this difference remained significant among junior faculty. Conclusions. A higher proportion of academic otolaryngologists with prior AAO-HNSF CORE funding have received NIH funding relative to their non–CORE-funded peers, suggesting that the CORE program may be successful in its stated goals of preparing individuals for the NIH peer review process, although further prospective study is needed to evaluate a ‘‘cause and effect’’ relationship. Individuals with current or prior NIH K-grants had greater research productivity than those with CORE funding history. Both cohorts had higher scholarly impact values than previously published figures among academic otolaryngologists, highlighting that both CORE grants and NIH K-grants awards are effective career development resources.
Otolaryngology– Head and Neck Surgery 2014, Vol. 151(2) 246–252 Ó American Academy of Otolaryngology—Head and Neck Surgery Foundation 2014 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/0194599814533647 http://otojournal.org
Keywords scholarly productivity, scholarly impact, research output, research productivity, academic promotion, academic advancement, academic appointment, National Institutes of Health funding, AAO-HNSF CORE grants, CORE Received October 3, 2013; revised March 14, 2014; accepted April 9, 2014.
Introduction Maintaining active research interests and having an impact on scholarship within the field play important roles in the hiring and promotions process in academic otolaryngology, as well as other fields of medicine.1-11 Identifying and retaining faculty with the ability to secure funding from external sources is advantageous for an institution and takes on special importance in light of recent cutbacks in research support.12,13 With these issues in mind, several grant programs are focused on providing support for junior practitioners to help them mature into advanced researchers. One of the grant types offered by the National Institutes of Health (NIH) is the Career Development Award, designated as ‘‘K-series’’ grant. Some of these, such as the mentored clinical scientist development awards (K01, K08, K23), are usually 1
Department of Otolaryngology–Head and Neck Surgery, Rutgers New Jersey Medical School, Newark, New Jersey, USA 2 Department of Neurological Surgery, Rutgers New Jersey Medical School, Newark, New Jersey, USA 3 Center for Skull Base and Pituitary Surgery, Rutgers New Jersey Medical School, Newark, New Jersey, USA 4 Department of Otolaryngology–Head and Neck Surgery, Wayne State University School of Medicine, Detroit, Michigan, USA 5 Rhinology Section, North Shore University Hospital, Manhasset, New York, USA 6 Department of Otolaryngology, New York University School of Medicine, New York, New York, USA Corresponding Author: Jean Anderson Eloy, Department of Otolaryngology–Head and Neck Surgery, Rutgers New Jersey Medical School, 90 Bergen St., Suite 8100, Newark, NJ 07103, USA. Email: [email protected]
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
Eloy et al
247
focused on supporting junior faculty early in their careers14; additionally some of these, such as K12 awards, are institutional awards. Other career development grants include K18 awards (Career Enhancement Award for Stem Cell Research, aimed at midcareer individuals) and K24 (Midcareer Investigator in Patient-Oriented Research) awards. Research (R) Grants are the most common funding mechanism offered by the NIH, particularly the R01 program, which is open to researchers at all levels of their career. An alternate source of support for early-career otolaryngology investigators includes grants awarded by the American Academy of Otolaryngology—Head and Neck Surgery Foundation (AAO-HNSF) Centralized Otolaryngology Research Efforts (CORE) grants program. Created in 1985, CORE grants are intended to promote career development for investigators hoping to make basic, clinical, or translational science a component of their careers. This program was designed with an eye toward future participation in the NIH peer review and funding process.15 A previous analysis demonstrated that procurement of AAO-HNSF CORE grant is associated with greater scholarly impact among academic otolaryngologists.16 A primary objective of the present analysis was to determine whether receiving a CORE grant ultimately leads to future NIH funding success. Furthermore, we were interested in evaluating whether CORE grant recipients in academic otolaryngology receive Career Development (K) NIH grants, Research (R) NIH grants, and multiple NIH grants at a higher rate than non–CORE-funded academic otolaryngologists. Additionally, we were interested in exploring whether CORE grants or Career Development (K) NIH grants have a stronger association with greater scholarly impact.
Materials and Methods This analysis compares two cohorts: (1) CORE-funded academic otolaryngologists and (2) non–CORE-funded academic otolaryngologists, comparing the proportion of individuals in each of these cohorts who have received NIH funding. We obtained a listing of all CORE grant recipients from 1985 to 2012 from the AAO-HNSF. This listing contained 494 grants awarded to 432 different individuals. An Internet search of these individuals with the term otolaryngology was conducted to determine their practice setting. Only individuals confirmed to be full-time faculty in an academic otolaryngology department were included in this analysis, while the following practitioners were excluded: those not possessing an MD or MD-equivalent degree, parttime faculty, and voluntary faculty. The NIH Reporter website (http://projectreporter.nih.gov/reporter.cfm) was searched to determine whether these individuals had active or previous NIH grants and what types of grants they had received. This resource contains NIH grants dating back to 1989. In further comparison of CORE-funded faculty versus Kgrant recipients, we considered only individuals receiving one of these grants (not both) to facilitate comparison of independent groups, unless mentioned otherwise. In other words, ‘‘CORE-funded faculty’’ includes those who received
CORE funding but not K-grants (unless otherwise mentioned), while individuals receiving K-series grants do not include CORE-funded faculty for the purposes of this analysis. The Scopus database was used to determine publication experience and scholarly impact, as measured by the hindex, of all faculty included in this analysis. An individual’s h-index is the number (h) of their papers that have been cited in the literature at least h times each.17,18 Its definition, strengths, and weaknesses have been widely discussed in the literature.11,17-31 While other databases such as Publish or Perish and ISI’s Web of Knowledge contain this information, Scopus has been used in a multitude of analyses utilizing the h-index, and results from this database have been shown to correlate highly with other commonly used bibliometric databases, such as Google Scholar.2,9-11,19,24-26,32-43 Publication experience was defined as the number of years from an individual’s first publication to the present. NIH funding status was similarly determined for practitioners included in the non–CORE ‘‘control’’ cohort. The American Medical Association’s FREIDA database was used to find a list of academic otolaryngology departments, and an Internet search of online faculty profiles was used to gather all available academic otolaryngologists meeting inclusion criteria as specified previously. Any individuals with a CORE funding history were excluded from this group, as they were already counted in the ‘‘CORE-funded’’ academician cohort. Academic rank, scholarly impact (as measured by the h-index), and publication range were determined for NIH-funded individuals. Data collection was completed in May 2013. This study qualified as nonhuman subject research and was thus exempted from needing Institutional Review Board approval per the standing policy of the Institutional Review Board of Rutgers New Jersey Medical School.
Statistical Analysis Mann-Whitney U-tests and Pearson’s chi-square tests were used for comparison of continuous and categorical data, respectively, with threshold for significance set at P \ .05 (two-tailed). SPSS version 20 (an IBM Company, Chicago, Illinois) was used for statistical calculations.
Results Out of 432 individual CORE grant recipients since 1985, 192 were confirmed to be current full-time MD faculty serving in an academic otolaryngology department as per our methodology. This group was used for comparison to the non–CORE-funded cohort of 1002 academic otolaryngologists included in this analysis (Table 1). As previously stated, the present analysis compares 2 cohorts, specifically (1) CORE-funded academic otolaryngologists and (2) non–CORE-funded otolaryngologists. Seventysix of 192 (39.6%) academicians with a CORE-funding history have received NIH funding, a statistically greater proportion than the 15.1% (152 out of 1002) of non–CORE-funded otolaryngologists who have received NIH funding (P \ .01)
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
248
Otolaryngology–Head and Neck Surgery 151(2)
Table 1. Sizes of Cohorts Included in This Analysis.a Number Non–CORE-Funded
Number CORE-Funded
Number NIH-Funded
Number K-Grant-Funded
1002 439 240 323
192 90 50 41
227 49 60 115
79 27 23 28
Overall Assistant Associate Professor a
Among CORE-funded academicians, 11 full-time academic faculty did not have a designated rank.
Table 2. Comparison of NIH Funding Success between CORE and Non–CORE-Funded Academicians.
NIH (any) funding history K-series grants R-series grants Multiple NIH grants
CORE-Funded (n)
Non-CORE (n)
Difference (95% CI)
P Value
39.6% (76) 15.6% (30) 20.8% (40) 8.9% (17)
15.1% (151) 4.9% (49) 9.6% (96) 5.4% (54)
24.5% (17.5%-31.9%) 10.7% (6.0%-1.7%) 11.2% (5.7%-17.8%) 3.5% (–0.02% to 8.5%)
\ .0001 \ .001 \ .001 .06
(Table 2) (P \ .0001). Out of these 76 academicians with both a CORE and NIH funding history, 71 (93.4%) received at least 1 NIH grant after receiving a CORE award. In addition to comparing overall NIH funding success between CORE and non–CORE-funded otolaryngologists, the authors were interested in seeing whether differences in NIH success were present between these 2 groups upon further examination of specific grant types. Therefore, the proportion of CORE-funded otolaryngologists who have received multiple NIH grants, any K-grants, and any R-series grants were compared to the proportions of non–CORE-funded who have received multiple NIH grants, any K-grants, and any R-series grants. A statistically greater proportion of CORE-funded otolaryngologists received K-grants (15.6%) than the proportion of non–CORE-funded otolaryngologists receiving K-grants (4.9%) (P \ .001) (Table 2). Similarly, a statistically greater proportion of CORE-funded otolaryngologists received Rgrants (20.8%) than the proportion of non–CORE-funded otolaryngologists receiving R-grants (9.6%) (P \ .001) (Table 2). The proportion of otolaryngologists receiving multiple NIH grants did not differ upon comparison of the CORE-funded and non–CORE-funded cohorts (9.4% vs 5.4%), P = .06, 95% CI of difference, 0.02%-8.5% (Table 2). Upon assessment of scholarly impact, individuals receiving AAO-HNSF CORE grants (but not receiving NIH Career Development K awards) had a statistically lower h-index than those receiving K-grants (P \ .0001) (Figure 1). We considered only individuals receiving 1 of these grants to facilitate comparison of independent groups. When considering the 30 academicians who have received both CORE and K-grants, there was a statistical trend toward higher scholarly impact in these individuals than those receiving only CORE grants (P = .09); additionally, individuals receiving both CORE and K-grants had lower h-indices than individuals in the K-grant cohort (P = .01).
Figure 1. Scholarly impact, as measured by the h-index, organized by career development resource. K represents NIH K-series career development awards, CORE represents AAO-HNSF CORE awards. Both represents individuals who have received both of these grant types. Error bars represent standard deviations.
We further compared the relative scholarly impact of these groups by controlling for academic rank and publication experience. CORE recipients at the assistant professor level had lower scholarly impact, as measured by the hindex, than their K-grant counterparts (H = 7.1 6 4.9 SD vs 12.6 6 7.5 SD, P \ .01). These differences disappeared upon comparison of promoted faculty, as there was no statistical difference in scholarly impact among associate professors (H = 13.5 6 5.9 SD vs 14.4 6 5.5 SD, P = .69) and
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
Eloy et al
249
Figure 2. Scholarly impact, as measured by the h-index, organized by publication experience. Top (solid) line represents individuals who have received K-series career development awards from the NIH (but not AAO-HNSF CORE awards), bottom (dashed) line represents individuals who have received AAO-HNSF CORE (but not K) grants.
professors (H = 21.2 6 9.8 SD vs 23.2 6 9.4 SD, P = .39). Seventy-five of 162 (46.3%) ranked individuals who received CORE grants (but no K-grants) were senior faculty (ie, associate professors or professors), significantly lower than the 71.4% of faculty with K-grants at these ranks. Overall, R-series grants recipients were the individuals with the highest scholarly impact (H = 19.2 6 10.2 SD, P-values \ .05). Individuals with a CORE funding history (but no Kgrants) had a statistically lower mean publication experience (15.5 years) than those who had received NIH career development grants (but no CORE funding) (20.5 years) (P \ .001). As K-series grants and CORE grants are both intended to promote career development for investigators, the authors felt it to be logical to compare scholarly impact values among individuals in these 2 cohorts. Upon controlling for publication range, individuals receiving NIH Career Development awards had higher scholarly impact among individuals with 10 or less years of experience (P = .0002) (Figure 2, Table 3). These differences disappeared among faculty beyond this point, with no statistical difference noted among K-funded and CORE-funded individuals upon comparison of 5- and 10-year experience intervals (P-value for comparison of individuals with 11-20 years of experience = 0.14, P-value for 21-30 years = .25) (Figure 2).
Discussion Individuals with current or prior NIH Career Development awards had greater scholarly impact, as
measured by the h-index, than their colleagues with a CORE funding history (Figure 1). Interestingly, a far higher proportion of K-grant recipients were senior faculty relative to CORE-grant recipients (71.4% vs 46.3%). As mentioned, these differences in scholarly impact were only present among junior faculty, emphasizing that differences in research production may trump the prestige or types of grants one procures in the academic advancement process in many departments. When controlling for years of experience (Figure 2), K-grant recipients had marked and significantly higher h-indices than CORE-funded academicians among those in the first decade of their careers, likely related to the finding that a higher proportion are promoted faculty. Despite an association with greater scholarly impact among K-grant recipients, a higher proportion of academicians with a CORE funding history successfully received Kseries grants relative to their non–CORE-funded colleagues (Table 2). This relationship persisted when considering Rgrants and overall NIH funding (Table 2). This suggests that the CORE program may be successful in its stated goals of preparing individuals for the NIH peer review application process.15 Further supporting this is that the vast majority (93.4%) of academicians receiving support from both organizations had CORE grants that preceded NIH grants. However, the present analysis was not a prospective study that can attribute causation. In other words, it is not clear if individuals who are more likely to pursue future NIH funding opportunities are the ones drawn toward applying for CORE funding or if the CORE program process facilitates the skills and research maturation necessary for a career that includes NIH-supported projects. For example, individuals in the former scenario may have had a more comprehensive research background, collaborated with highly effective (and well-funded) mentors, or simply had more dedicated research time set aside. These possibilities were beyond the scope of our study design but present interesting issues for further exploration in a prospective and/or survey-based study. In addition to deficits in scholarly impact, CORE-funded individuals (without K-grants) had a statistically lower mean publication experience than their peers with K-grants (P \ .001). There are several potential explanations for this finding, an obvious one being the monetary value of NIH awards, with one analysis reporting this to average over $500,000 for otolaryngologists receiving grants.41 Although fiscal concerns in recent years have resulted in declining
Table 3. Comparison of Scholarly Impact between K-Grant Recipients and CORE Grant Recipients.
Overall H-index H-index at \ 10 years of experience H-index at 11-20 years of experience H-index at 21-30 years of experience
CORE Granta
K-Granta
Difference (95% CI)
P Value
11.7 4.9 12.7 19.0
18.2 16.0 15.6 21.6
6.5 (5.2-6.5) 11.1 (9.2-11.1) 2.9 (1.5-4.3) 2.6 (0.15-5.0)
\ .0001 .0002 .14 .25
a
To facilitate comparison of independent groups, this refers to individuals with only one type of these grants, not individuals with both. Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
250
Otolaryngology–Head and Neck Surgery 151(2)
NIH funding,13 the size of these awards is still many times the magnitude of CORE funding, as the median CORE award in the database provided to use was $10,000. This perspective is valuable in reinforcing the CORE program’s primary goal of preparation for the rigorous NIH peer review process. Additionally, it should be emphasized that CORE grants are aimed toward trainees and early-career investigators, which may partially explain the discrepancies in publication experience we report. Individuals receiving K-grants but no CORE grants (H = 18.2) (Figure 1), CORE grants but no K-grants (H = 11.7) (Figure 1), both K and CORE grants (H = 13.1) (Figure 1), and R-grants (regardless of other grants; H = 19.2) all had far higher scholarly impact than previously cited values for otolaryngologists. One study of scholarly impact in various surgical subspecialties reported a mean h-index of 7.8 among academic otolaryngologists.43 Comparison with another analysis of otolaryngologists organized by academic rank puts the scholarly impact of NIH- and CORE-funded individuals in this analysis above associate professors (previously reported to have an h-index of 8.13) and for some groups, above the level of professors (previously calculated to be 15.6).11 We have previously explored the relationship between scholarly impact, as measured by the h-index, and NIH funding among faculty from the top 20 NIH-funded otolaryngology departments.41 That analysis demonstrated a strong relationship between NIH funding and scholarly productivity, as NIH-funded colleagues had far higher h-indices than their non-funded peers at the same 20 departments. Our present analysis builds on this analysis by analyzing the relationship between procuring CORE grants and future NIH funding success, as well as comparing 2 grant programs intended as career development research support tools, namely, CORE funding and NIH Career Development (K) grants. This analysis highlights that individuals benefitting from either of these programs have greater scholarly impact. While Kgrant recipients have higher h-indices, receiving CORE funding may play a role in preparing young investigators for future success with the NIH application process. A limitation inherent to this analysis is our use of the hindex to assess scholarly impact. Departments and institutions may take into account a variety of factors, subjective and objective, when considering research output. Subjective factors, such as an individual’s reputation or the number of one’s ‘‘significant’’ publications, have obvious limitations in that they are not uniform and are considered differently by different individuals.17 Commonly used objective factors, such as number of publications, also have limitations. Number of publications does not relay the strength, relevance, or quality of a person’s research. One’s total number of citations in the literature may be disproportionately influenced by a single or a handful of papers. We chose to look at the h-index because it is an objective bibliometric that indicates the consistency at which an individual is producing impactful work and has been shown to have a strong association with scholarly impact, academic promotion, NIH funding, and even fellowship training in a multitude of specialties, including otolaryngology.2,9-11,19,24-26,29-39,41,43-53
Its definition, strengths, and weaknesses have been discussed extensively in the literature.2,9,17,18,21,24,26,31,35,54,55 This analysis evaluated the 192 CORE-funded practitioners confirmed to be currently in academic practice in an attempt to compare practitioners in similar practice settings. It is likely that the 230 other CORE-funded persons may possibly not be as productive in terms of scholarly impact, whether using measurements such as NIH funding procurement or objective bibliometrics such as the h-index. Another limitation is the retrospective design of this analysis, which makes it impossible for us to attribute causation. A future analysis of interest may include a prospective design focusing on AAO-HNSF CORE grant applicants and other otolaryngologists-in-training along with their subsequent research trajectories. In addition, an attempt was made to create a more robust prediction model based on some of the predictive factors for NIH funding and hindex suggested in this article (including K-grants, CORE grants, years of publications); however, analysis suggested that these factors were not independent using the DurbinWatson statistic. Another limitation in this analysis relates to the NIH RePORTER database only reporting awards dating back to 1989, representing a 4-year gap since the start of the CORE funding program in 1985 in which CORE-funded individuals may have received NIH funding that was not included in this analysis. Finally, this analysis was not able to nor was it intended to take into account numerous other sources of support for research in the field, including local organizations, intra-institutional funding opportunities, and even internal departmental mechanisms. These potential sources may have value in encouraging individuals to incorporate research into their careers and providing the skills necessary to mature into advanced researchers.
Conclusions A higher proportion of academic otolaryngologists with an AAO-HNSF CORE funding history have received NIH funding relative to their non–CORE-funded peers. This is true when further considering individuals awarded Career Development (K) awards and Research (R-Series) grants from the NIH, as a higher proportion of CORE-funded academicians are successfully awarded these grants than their non–CORE-funded academic colleagues. This suggests that the CORE program may be successful in its stated goals of preparing individuals for the NIH peer review application process, although further prospective study is needed to evaluate whether this association represents a cause and effect relationship. Upon further comparison of relative scholarly impact, individuals with current or prior NIH Kgrants had greater research productivity than those with a CORE funding history. Both of these cohorts have higher scholarly impact values than previously published figures among all academic otolaryngologists, highlighting that both AAO-HNSF CORE grants and NIH K-grants awards are effective career development resources.
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
Eloy et al
251
Acknowledgments The authors would like to thank Stephanie Jones and the CORE grant committee members for their assistance in providing a database of previous grant recipients.
14.
Author Contributions
15.
Jean Anderson Eloy, conception, design, analysis, and interpretation; revision; final approval; Peter F. Svider, conception, design, data acquisition, analysis; drafting; final approval; Vivek V. Kanumuri, data acquisition; revision; final approval; Adam J. Folbe, analysis; revision; final approval; Michael Setzen, analysis; revision; final approval; Soly Baredes, analysis; revision; final approval.
16.
Disclosures
18.
Competing interests: Michael Setzen, speaker for TEVA and MEDA on their Speakers Bureau (not related to current subject).
19.
17.
Sponsorships: None. Funding source: None. 20.
References
21.
1. Carr P, Friedman RH, Moskowitz MA, Kazis LE, Weed HG. Research, academic rank, and compensation of women and men faculty in academic general internal medicine. J Gen Intern Med. 1992;7:418-423. 2. Colaco M, Svider PF, Mauro KM, Eloy JA, Jackson-Rosario I. Is there a relationship between National Institutes of Health funding and research impact on academic urology? J Urol. 2013;190:999-1003. 3. Hauptman JS, Chow DS, Martin NA, Itagaki MW. Research productivity in neurosurgery: trends in globalization, scientific focus, and funding. J Neurosurg. 2011;115:1262-1272. 4. Atasoylu AA, Wright SM, Beasley BW, et al. Promotion criteria for clinician-educators. J Gen Intern Med. 2003;18:711-716. 5. Dixon AK. Publishing and academic promotion. Singapore Med J. 2009;50:847-850. 6. Holden G, Rosenberg G, Barker K. Bibliometrics: a potential decision making aid in hiring, reappointment, tenure and promotion decisions. Soc Work Health Care. 2005;41:67-92. 7. Sanfey H. Promotion to professor: a career development resource. Am J Surg. 2010;200:554-557. 8. Sanfey H, Hollands C. Career development resource: promotion to associate professor. Am J Surg. 2012;204:130-134. 9. Svider PF, Mady LJ, Husain Q, et al. Geographic differences in academic promotion practices, fellowship training, and scholarly impact. Am J Otolaryngol. 2013;34:464-470. 10. Svider PF, Blake DM, Setzen M, Folbe AJ, Baredes S, Eloy JA. Rhinology fellowship training and its scholarly impact. Am J Rhinol Allergy. 2013;27:e131-e134. 11. Svider PF, Choudhry ZA, Choudhry OJ, Baredes S, Liu JK, Eloy JA. The use of the h-index in academic otolaryngology. Laryngoscope. 2013;123:103-106. 12. Jacob BA, Lefgren L. The impact of research grant funding on scientific productivity. J Public Econ. 2011;95:1168-1177. 13. Garrison HH, Drehman B. NIH research funding trends: FY1995-2013. http://www.faseb.org/Policy-and-Government-
22. 23. 24.
25.
26.
27.
28.
29. 30.
31.
32.
Affairs/Data-Compilations/NIH-Research-Funding-Trends.aspx# sthash.PW3i8pW4.dpbs. K Kiosk—information about NIH Career Development Awards.Accessed June 29, 2013. http://grantsnihgov/training/ careerdevelopmentawardshtm. 2013 Centralized Otolaryngology Research Efforts (CORE) Program. http://www.entnet.org/EducationAndResearch/COREGrants.cfm. Eloy JA, Svider PF, Folbe AJ, Setzen M, Baredes S. AAOHNSF CORE grant acquisition is associated with greater scholarly impact. Otolaryngol Head Neck Surg. 2014;150:53-60. Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005;102:1656916572. Hirsch JE. Does the H index have predictive power? Proc Natl Acad Sci U S A. 2007;104:19193-19198. Benway BM, Kalidas P, Cabello JM, Bhayani SB. Does citation analysis reveal association between h-index and academic rank in urology? Urology. 2009;74:30-33. Castillo M. Measuring academic output: the H-index. AJNR Am J Neuroradiol. 2010;31:783-784. Engqvist L, Frommen JG. The h-index and self-citations. Trends Ecol Evol. 2008;23:250-252. Gaster N, Gaster M. A critical assessment of the h-index. Bioessays. 2012;34:830-832. Jeang KT. H-index, mentoring-index, highly-cited and highlyaccessed: how to evaluate scientists? Retrovirology. 2008;5:106. Pagel PS, Hudetz JA. H-index is a sensitive indicator of academic activity in highly productive anaesthesiologists: results of a bibliometric analysis. Acta Anaesthesiol Scand. 2011;55:1085-1089. Pagel PS, Hudetz JA. Bibliometric analysis of anaesthesia journal editorial board members: correlation between journal impact factor and the median h-index of its board members. Br J Anaesth. 2011;107:357-361. Pagel PS, Hudetz JA. Scholarly productivity of United States academic cardiothoracic anesthesiologists: influence of fellowship accreditation and transesophageal echocardiographic credentials on h-index and other citation bibliometrics. J Cardiothorac Vasc Anesth. 2011;25:761-765. Poynard T, Thabut D, Jabre P, et al. Ranking hepatologists: which Hirsch’s h-index to prevent the ‘‘e-crise de foi-e’’? Clin Res Hepatol Gastroenterol. 2011;35:375-386. Quigley MR, Holliday EB, Fuller CD, Choi M, Thomas CR Jr. Distribution of the h-index in radiation oncology conforms to a variation of power law: implications for assessing academic productivity. J Cancer Educ. 2012;27:463-466. Rad AE, Brinjikji W, Cloft HJ, Kallmes DF. The H-index in academic radiology. Acad Radiol. 2010;17:817-821. Rezek I, McDonald RJ, Kallmes DF. Is the h-index predictive of greater NIH funding success among academic radiologists? Acad Radiol. 2011;18:1337-1340. Sharma B, Boet S, Grantcharov T, Shin E, Barrowman NJ, Bould MD. The h-index outperforms other bibliometrics in the assessment of research performance in general surgery: a province-wide study. Surgery. 2013;153:493-501. Lee J, Kraus KL, Couldwell WT. Use of the h index in neurosurgery. Clinical article. J Neurosurg. 2009;111:387-392.
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
252
Otolaryngology–Head and Neck Surgery 151(2)
33. Pagel PS, Hudetz JA. An analysis of scholarly productivity in United States academic anaesthesiologists by citation bibliometrics. Anaesthesia. 2011;66:873-878. 34. Eloy JA, Svider P, Chandrasekhar SS, et al. Gender disparities in scholarly productivity within academic otolaryngology departments. Otolaryngol Head Neck Surg. 2013;148:215-222. 35. Eloy JA, Svider PF, Cherla DV, et al. Gender disparities in research productivity among 9952 academic physicians. Laryngoscope. 2013;123:1865-1875. 36. Eloy JA, Svider PF, Kovalerchik O, Baredes S, Kalyoussef E, Chandrasekhar SS. Gender differences in successful NIH grant funding in otolaryngology. Otolaryngol Head Neck Surg. 2013;149:77-83. 37. Eloy JA, Svider PF, Mauro KM, Setzen M, Baredes S. Impact of fellowship training on research productivity in academic otolaryngology. Laryngoscope. 2012;122:2690-2694. 38. Eloy JA, Svider PF, Folbe AJ, Couldwell WT, Liu JK. Comparison of plaintiff and defendant expert witness qualification in malpractice litigation in neurological surgery. J Neurosurg. 2014;120:185-190. 39. Eloy JA, Svider PF, Patel D, Setzen M, Baredes S. Comparison of plaintiff and defendant expert witness qualification in malpractice litigation in otolaryngology. Otolaryngol Head Neck Surg. 2013;148:764-769. 40. Eloy JA, Svider PF, Patel D, Setzen M, Baredes S. In response to ‘‘comparison of plaintiff and defendant expert witness qualification in malpractice litigation in otolaryngology.’’Otolaryngol Head Neck Surg. 2013;149:649-650. 41. Svider PF, Mauro KM, Sanghvi S, Setzen M, Baredes S, Eloy JA. Is NIH funding predictive of greater research productivity and impact among academic otolaryngologists? Laryngoscope. 2013;123:118-122. 42. Svider PF, Husain Q, Folbe AJ, Couldwell WT, Liu JK, Eloy JA. Assessing National Institutes of Health funding and scholarly impact in neurological surgery. J Neurosurg. 2014;120:191-196. 43. Svider PF, Pashkova AA, Choudhry Z, et al. Comparison of scholarly impact among surgical specialties: an examination of 2429 academic surgeons. Laryngoscope. 2013;123:884-889. 44. Pashkova AA, Svider PF, Chang CY, Diaz L, Eloy JA, Eloy JD. Gender disparity among US anaesthesiologists: are women underrepresented in academic ranks and scholarly productivity? Acta Anaesthesiol Scand. 2013;57:1058-1064.
45. Agarwal N, Clark S, Svider PF, Couldwell WT, Eloy JA, Liu JK. Impact of fellowship training on research productivity in academic neurological surgery. World Neurosurg. 2013;80:738-744. 46. Svider PF, Lopez SA, Husain Q, Bhagat N, Eloy JA, Langer PD. The association between scholarly impact and National Institutes of Health funding in ophthalmology. Ophthalmology. 2014;121:423-428. 47. Eloy JA, Svider PF, Setzen M, Baredes S, Folbe AJ. Does receiving an American Academy of Otolaryngology—Head and Neck Surgery Foundation Centralized Otolaryngology Research Efforts grant influence career path and scholarly impact among fellowship-trained rhinologists? Int Forum Allergy Rhinol. 2014;4(1):85-90. 48. Svider PF, Husain Q, Mauro KM, Folbe AJ, Baredes S, Eloy JA. Impact of mentoring medical students on scholarly productivity. Int Forum Allergy Rhinol. 2014;4(2):138-142. 49. Sunaryo PL, Svider PF, Jackson-Rosario I, Eloy JA. Expert witness testimony in urology malpractice litigation. J Urol. 2014;83(4):704-709. 50. Kasabwala K, Morton CM, Svider PF, Nahass TA, Eloy JA, Jackson-Rosario I. Factors influencing scholarly impact: does urology fellowship-training affect research output? [published online December 30, 2013]. J Surg Educ. doi:http://dx.doi.org/ 10.1016/j.jsurg.2013.10.010 51. Tomei KL, Nahass MM, Husain Q, et al. A gender-based comparison of academic rank and scholarly productivity in academic neurological surgery [published online December 8, 2013]. J Clin Neurosci. doi:10.1016/j.jocn.2013.11.006 52. Eloy JA, Mady LJ, Svider PF, et al. Regional differences in gender promotion and scholarly productivity in otolaryngology. Otolaryngol Head Neck Surg. 2014;150(3):371-377. 53. Folbe AJ, Svider PF, Setzen M, Zuliani G, Lin H, Eloy JA. Scientific inquiry into rhinosinusitis: who is receiving funding from the National Institutes of Health? [published online November 15, 2013] Laryngoscope. doi:10.1002/lary.24525 54. Purvis A. The h index: playing the numbers game. Trends Ecol Evol. 2006;21:422. 55. Kelly CD, Jennions MD. H-index: age and sex make it unreliable. Nature. 2007;449:403.
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
Do AAO-HNSF CORE Grants Predict Future NIH Funding Success? Jean Anderson Eloy, MD1,2,3, Peter F. Svider, MD4, Vivek V. Kanumuri1, Adam J. Folbe, MD4, Michael Setzen, MD5,6, and Soly Baredes, MD1,3
Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article.
Abstract Objective. To determine (1) whether academic otolaryngologists who have received an American Academy of Otolaryngology— Head and Neck Surgery Foundation (AAO-HNSF) Centralized Otolaryngology Research Efforts (CORE) grant are more likely to procure future National Institutes of Health (NIH) funding; (2) whether CORE grants or NIH Career Development (K) awards have a stronger association with scholarly impact. Study Design and Setting. Historical cohort. Methods. Scholarly impact, as measured by the h-index, publication experience, and prior grant history, were determined for CORE-funded and non–CORE-funded academic otolaryngologists. All individuals were assessed for NIH funding history. Results. Of 192 academic otolaryngologists with a CORE funding history, 39.6% had active or prior NIH awards versus 15.1% of 1002 non–CORE-funded faculty (P \ .0001). Higher proportions of CORE-funded otolaryngologists have received K-series and R-series grants from the NIH (P-values \ .05). K-grant recipients had higher h-indices than CORE recipients (12.6 vs 7.1, P \ .01). Upon controlling for rank and experience, this difference remained significant among junior faculty. Conclusions. A higher proportion of academic otolaryngologists with prior AAO-HNSF CORE funding have received NIH funding relative to their non–CORE-funded peers, suggesting that the CORE program may be successful in its stated goals of preparing individuals for the NIH peer review process, although further prospective study is needed to evaluate a ‘‘cause and effect’’ relationship. Individuals with current or prior NIH K-grants had greater research productivity than those with CORE funding history. Both cohorts had higher scholarly impact values than previously published figures among academic otolaryngologists, highlighting that both CORE grants and NIH K-grants awards are effective career development resources.
Otolaryngology– Head and Neck Surgery 2014, Vol. 151(2) 246–252 Ó American Academy of Otolaryngology—Head and Neck Surgery Foundation 2014 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/0194599814533647 http://otojournal.org
Keywords scholarly productivity, scholarly impact, research output, research productivity, academic promotion, academic advancement, academic appointment, National Institutes of Health funding, AAO-HNSF CORE grants, CORE Received October 3, 2013; revised March 14, 2014; accepted April 9, 2014.
Introduction Maintaining active research interests and having an impact on scholarship within the field play important roles in the hiring and promotions process in academic otolaryngology, as well as other fields of medicine.1-11 Identifying and retaining faculty with the ability to secure funding from external sources is advantageous for an institution and takes on special importance in light of recent cutbacks in research support.12,13 With these issues in mind, several grant programs are focused on providing support for junior practitioners to help them mature into advanced researchers. One of the grant types offered by the National Institutes of Health (NIH) is the Career Development Award, designated as ‘‘K-series’’ grant. Some of these, such as the mentored clinical scientist development awards (K01, K08, K23), are usually 1
Department of Otolaryngology–Head and Neck Surgery, Rutgers New Jersey Medical School, Newark, New Jersey, USA 2 Department of Neurological Surgery, Rutgers New Jersey Medical School, Newark, New Jersey, USA 3 Center for Skull Base and Pituitary Surgery, Rutgers New Jersey Medical School, Newark, New Jersey, USA 4 Department of Otolaryngology–Head and Neck Surgery, Wayne State University School of Medicine, Detroit, Michigan, USA 5 Rhinology Section, North Shore University Hospital, Manhasset, New York, USA 6 Department of Otolaryngology, New York University School of Medicine, New York, New York, USA Corresponding Author: Jean Anderson Eloy, Department of Otolaryngology–Head and Neck Surgery, Rutgers New Jersey Medical School, 90 Bergen St., Suite 8100, Newark, NJ 07103, USA. Email: [email protected]
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
Eloy et al
247
focused on supporting junior faculty early in their careers14; additionally some of these, such as K12 awards, are institutional awards. Other career development grants include K18 awards (Career Enhancement Award for Stem Cell Research, aimed at midcareer individuals) and K24 (Midcareer Investigator in Patient-Oriented Research) awards. Research (R) Grants are the most common funding mechanism offered by the NIH, particularly the R01 program, which is open to researchers at all levels of their career. An alternate source of support for early-career otolaryngology investigators includes grants awarded by the American Academy of Otolaryngology—Head and Neck Surgery Foundation (AAO-HNSF) Centralized Otolaryngology Research Efforts (CORE) grants program. Created in 1985, CORE grants are intended to promote career development for investigators hoping to make basic, clinical, or translational science a component of their careers. This program was designed with an eye toward future participation in the NIH peer review and funding process.15 A previous analysis demonstrated that procurement of AAO-HNSF CORE grant is associated with greater scholarly impact among academic otolaryngologists.16 A primary objective of the present analysis was to determine whether receiving a CORE grant ultimately leads to future NIH funding success. Furthermore, we were interested in evaluating whether CORE grant recipients in academic otolaryngology receive Career Development (K) NIH grants, Research (R) NIH grants, and multiple NIH grants at a higher rate than non–CORE-funded academic otolaryngologists. Additionally, we were interested in exploring whether CORE grants or Career Development (K) NIH grants have a stronger association with greater scholarly impact.
Materials and Methods This analysis compares two cohorts: (1) CORE-funded academic otolaryngologists and (2) non–CORE-funded academic otolaryngologists, comparing the proportion of individuals in each of these cohorts who have received NIH funding. We obtained a listing of all CORE grant recipients from 1985 to 2012 from the AAO-HNSF. This listing contained 494 grants awarded to 432 different individuals. An Internet search of these individuals with the term otolaryngology was conducted to determine their practice setting. Only individuals confirmed to be full-time faculty in an academic otolaryngology department were included in this analysis, while the following practitioners were excluded: those not possessing an MD or MD-equivalent degree, parttime faculty, and voluntary faculty. The NIH Reporter website (http://projectreporter.nih.gov/reporter.cfm) was searched to determine whether these individuals had active or previous NIH grants and what types of grants they had received. This resource contains NIH grants dating back to 1989. In further comparison of CORE-funded faculty versus Kgrant recipients, we considered only individuals receiving one of these grants (not both) to facilitate comparison of independent groups, unless mentioned otherwise. In other words, ‘‘CORE-funded faculty’’ includes those who received
CORE funding but not K-grants (unless otherwise mentioned), while individuals receiving K-series grants do not include CORE-funded faculty for the purposes of this analysis. The Scopus database was used to determine publication experience and scholarly impact, as measured by the hindex, of all faculty included in this analysis. An individual’s h-index is the number (h) of their papers that have been cited in the literature at least h times each.17,18 Its definition, strengths, and weaknesses have been widely discussed in the literature.11,17-31 While other databases such as Publish or Perish and ISI’s Web of Knowledge contain this information, Scopus has been used in a multitude of analyses utilizing the h-index, and results from this database have been shown to correlate highly with other commonly used bibliometric databases, such as Google Scholar.2,9-11,19,24-26,32-43 Publication experience was defined as the number of years from an individual’s first publication to the present. NIH funding status was similarly determined for practitioners included in the non–CORE ‘‘control’’ cohort. The American Medical Association’s FREIDA database was used to find a list of academic otolaryngology departments, and an Internet search of online faculty profiles was used to gather all available academic otolaryngologists meeting inclusion criteria as specified previously. Any individuals with a CORE funding history were excluded from this group, as they were already counted in the ‘‘CORE-funded’’ academician cohort. Academic rank, scholarly impact (as measured by the h-index), and publication range were determined for NIH-funded individuals. Data collection was completed in May 2013. This study qualified as nonhuman subject research and was thus exempted from needing Institutional Review Board approval per the standing policy of the Institutional Review Board of Rutgers New Jersey Medical School.
Statistical Analysis Mann-Whitney U-tests and Pearson’s chi-square tests were used for comparison of continuous and categorical data, respectively, with threshold for significance set at P \ .05 (two-tailed). SPSS version 20 (an IBM Company, Chicago, Illinois) was used for statistical calculations.
Results Out of 432 individual CORE grant recipients since 1985, 192 were confirmed to be current full-time MD faculty serving in an academic otolaryngology department as per our methodology. This group was used for comparison to the non–CORE-funded cohort of 1002 academic otolaryngologists included in this analysis (Table 1). As previously stated, the present analysis compares 2 cohorts, specifically (1) CORE-funded academic otolaryngologists and (2) non–CORE-funded otolaryngologists. Seventysix of 192 (39.6%) academicians with a CORE-funding history have received NIH funding, a statistically greater proportion than the 15.1% (152 out of 1002) of non–CORE-funded otolaryngologists who have received NIH funding (P \ .01)
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
248
Otolaryngology–Head and Neck Surgery 151(2)
Table 1. Sizes of Cohorts Included in This Analysis.a Number Non–CORE-Funded
Number CORE-Funded
Number NIH-Funded
Number K-Grant-Funded
1002 439 240 323
192 90 50 41
227 49 60 115
79 27 23 28
Overall Assistant Associate Professor a
Among CORE-funded academicians, 11 full-time academic faculty did not have a designated rank.
Table 2. Comparison of NIH Funding Success between CORE and Non–CORE-Funded Academicians.
NIH (any) funding history K-series grants R-series grants Multiple NIH grants
CORE-Funded (n)
Non-CORE (n)
Difference (95% CI)
P Value
39.6% (76) 15.6% (30) 20.8% (40) 8.9% (17)
15.1% (151) 4.9% (49) 9.6% (96) 5.4% (54)
24.5% (17.5%-31.9%) 10.7% (6.0%-1.7%) 11.2% (5.7%-17.8%) 3.5% (–0.02% to 8.5%)
\ .0001 \ .001 \ .001 .06
(Table 2) (P \ .0001). Out of these 76 academicians with both a CORE and NIH funding history, 71 (93.4%) received at least 1 NIH grant after receiving a CORE award. In addition to comparing overall NIH funding success between CORE and non–CORE-funded otolaryngologists, the authors were interested in seeing whether differences in NIH success were present between these 2 groups upon further examination of specific grant types. Therefore, the proportion of CORE-funded otolaryngologists who have received multiple NIH grants, any K-grants, and any R-series grants were compared to the proportions of non–CORE-funded who have received multiple NIH grants, any K-grants, and any R-series grants. A statistically greater proportion of CORE-funded otolaryngologists received K-grants (15.6%) than the proportion of non–CORE-funded otolaryngologists receiving K-grants (4.9%) (P \ .001) (Table 2). Similarly, a statistically greater proportion of CORE-funded otolaryngologists received Rgrants (20.8%) than the proportion of non–CORE-funded otolaryngologists receiving R-grants (9.6%) (P \ .001) (Table 2). The proportion of otolaryngologists receiving multiple NIH grants did not differ upon comparison of the CORE-funded and non–CORE-funded cohorts (9.4% vs 5.4%), P = .06, 95% CI of difference, 0.02%-8.5% (Table 2). Upon assessment of scholarly impact, individuals receiving AAO-HNSF CORE grants (but not receiving NIH Career Development K awards) had a statistically lower h-index than those receiving K-grants (P \ .0001) (Figure 1). We considered only individuals receiving 1 of these grants to facilitate comparison of independent groups. When considering the 30 academicians who have received both CORE and K-grants, there was a statistical trend toward higher scholarly impact in these individuals than those receiving only CORE grants (P = .09); additionally, individuals receiving both CORE and K-grants had lower h-indices than individuals in the K-grant cohort (P = .01).
Figure 1. Scholarly impact, as measured by the h-index, organized by career development resource. K represents NIH K-series career development awards, CORE represents AAO-HNSF CORE awards. Both represents individuals who have received both of these grant types. Error bars represent standard deviations.
We further compared the relative scholarly impact of these groups by controlling for academic rank and publication experience. CORE recipients at the assistant professor level had lower scholarly impact, as measured by the hindex, than their K-grant counterparts (H = 7.1 6 4.9 SD vs 12.6 6 7.5 SD, P \ .01). These differences disappeared upon comparison of promoted faculty, as there was no statistical difference in scholarly impact among associate professors (H = 13.5 6 5.9 SD vs 14.4 6 5.5 SD, P = .69) and
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
Eloy et al
249
Figure 2. Scholarly impact, as measured by the h-index, organized by publication experience. Top (solid) line represents individuals who have received K-series career development awards from the NIH (but not AAO-HNSF CORE awards), bottom (dashed) line represents individuals who have received AAO-HNSF CORE (but not K) grants.
professors (H = 21.2 6 9.8 SD vs 23.2 6 9.4 SD, P = .39). Seventy-five of 162 (46.3%) ranked individuals who received CORE grants (but no K-grants) were senior faculty (ie, associate professors or professors), significantly lower than the 71.4% of faculty with K-grants at these ranks. Overall, R-series grants recipients were the individuals with the highest scholarly impact (H = 19.2 6 10.2 SD, P-values \ .05). Individuals with a CORE funding history (but no Kgrants) had a statistically lower mean publication experience (15.5 years) than those who had received NIH career development grants (but no CORE funding) (20.5 years) (P \ .001). As K-series grants and CORE grants are both intended to promote career development for investigators, the authors felt it to be logical to compare scholarly impact values among individuals in these 2 cohorts. Upon controlling for publication range, individuals receiving NIH Career Development awards had higher scholarly impact among individuals with 10 or less years of experience (P = .0002) (Figure 2, Table 3). These differences disappeared among faculty beyond this point, with no statistical difference noted among K-funded and CORE-funded individuals upon comparison of 5- and 10-year experience intervals (P-value for comparison of individuals with 11-20 years of experience = 0.14, P-value for 21-30 years = .25) (Figure 2).
Discussion Individuals with current or prior NIH Career Development awards had greater scholarly impact, as
measured by the h-index, than their colleagues with a CORE funding history (Figure 1). Interestingly, a far higher proportion of K-grant recipients were senior faculty relative to CORE-grant recipients (71.4% vs 46.3%). As mentioned, these differences in scholarly impact were only present among junior faculty, emphasizing that differences in research production may trump the prestige or types of grants one procures in the academic advancement process in many departments. When controlling for years of experience (Figure 2), K-grant recipients had marked and significantly higher h-indices than CORE-funded academicians among those in the first decade of their careers, likely related to the finding that a higher proportion are promoted faculty. Despite an association with greater scholarly impact among K-grant recipients, a higher proportion of academicians with a CORE funding history successfully received Kseries grants relative to their non–CORE-funded colleagues (Table 2). This relationship persisted when considering Rgrants and overall NIH funding (Table 2). This suggests that the CORE program may be successful in its stated goals of preparing individuals for the NIH peer review application process.15 Further supporting this is that the vast majority (93.4%) of academicians receiving support from both organizations had CORE grants that preceded NIH grants. However, the present analysis was not a prospective study that can attribute causation. In other words, it is not clear if individuals who are more likely to pursue future NIH funding opportunities are the ones drawn toward applying for CORE funding or if the CORE program process facilitates the skills and research maturation necessary for a career that includes NIH-supported projects. For example, individuals in the former scenario may have had a more comprehensive research background, collaborated with highly effective (and well-funded) mentors, or simply had more dedicated research time set aside. These possibilities were beyond the scope of our study design but present interesting issues for further exploration in a prospective and/or survey-based study. In addition to deficits in scholarly impact, CORE-funded individuals (without K-grants) had a statistically lower mean publication experience than their peers with K-grants (P \ .001). There are several potential explanations for this finding, an obvious one being the monetary value of NIH awards, with one analysis reporting this to average over $500,000 for otolaryngologists receiving grants.41 Although fiscal concerns in recent years have resulted in declining
Table 3. Comparison of Scholarly Impact between K-Grant Recipients and CORE Grant Recipients.
Overall H-index H-index at \ 10 years of experience H-index at 11-20 years of experience H-index at 21-30 years of experience
CORE Granta
K-Granta
Difference (95% CI)
P Value
11.7 4.9 12.7 19.0
18.2 16.0 15.6 21.6
6.5 (5.2-6.5) 11.1 (9.2-11.1) 2.9 (1.5-4.3) 2.6 (0.15-5.0)
\ .0001 .0002 .14 .25
a
To facilitate comparison of independent groups, this refers to individuals with only one type of these grants, not individuals with both. Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
250
Otolaryngology–Head and Neck Surgery 151(2)
NIH funding,13 the size of these awards is still many times the magnitude of CORE funding, as the median CORE award in the database provided to use was $10,000. This perspective is valuable in reinforcing the CORE program’s primary goal of preparation for the rigorous NIH peer review process. Additionally, it should be emphasized that CORE grants are aimed toward trainees and early-career investigators, which may partially explain the discrepancies in publication experience we report. Individuals receiving K-grants but no CORE grants (H = 18.2) (Figure 1), CORE grants but no K-grants (H = 11.7) (Figure 1), both K and CORE grants (H = 13.1) (Figure 1), and R-grants (regardless of other grants; H = 19.2) all had far higher scholarly impact than previously cited values for otolaryngologists. One study of scholarly impact in various surgical subspecialties reported a mean h-index of 7.8 among academic otolaryngologists.43 Comparison with another analysis of otolaryngologists organized by academic rank puts the scholarly impact of NIH- and CORE-funded individuals in this analysis above associate professors (previously reported to have an h-index of 8.13) and for some groups, above the level of professors (previously calculated to be 15.6).11 We have previously explored the relationship between scholarly impact, as measured by the h-index, and NIH funding among faculty from the top 20 NIH-funded otolaryngology departments.41 That analysis demonstrated a strong relationship between NIH funding and scholarly productivity, as NIH-funded colleagues had far higher h-indices than their non-funded peers at the same 20 departments. Our present analysis builds on this analysis by analyzing the relationship between procuring CORE grants and future NIH funding success, as well as comparing 2 grant programs intended as career development research support tools, namely, CORE funding and NIH Career Development (K) grants. This analysis highlights that individuals benefitting from either of these programs have greater scholarly impact. While Kgrant recipients have higher h-indices, receiving CORE funding may play a role in preparing young investigators for future success with the NIH application process. A limitation inherent to this analysis is our use of the hindex to assess scholarly impact. Departments and institutions may take into account a variety of factors, subjective and objective, when considering research output. Subjective factors, such as an individual’s reputation or the number of one’s ‘‘significant’’ publications, have obvious limitations in that they are not uniform and are considered differently by different individuals.17 Commonly used objective factors, such as number of publications, also have limitations. Number of publications does not relay the strength, relevance, or quality of a person’s research. One’s total number of citations in the literature may be disproportionately influenced by a single or a handful of papers. We chose to look at the h-index because it is an objective bibliometric that indicates the consistency at which an individual is producing impactful work and has been shown to have a strong association with scholarly impact, academic promotion, NIH funding, and even fellowship training in a multitude of specialties, including otolaryngology.2,9-11,19,24-26,29-39,41,43-53
Its definition, strengths, and weaknesses have been discussed extensively in the literature.2,9,17,18,21,24,26,31,35,54,55 This analysis evaluated the 192 CORE-funded practitioners confirmed to be currently in academic practice in an attempt to compare practitioners in similar practice settings. It is likely that the 230 other CORE-funded persons may possibly not be as productive in terms of scholarly impact, whether using measurements such as NIH funding procurement or objective bibliometrics such as the h-index. Another limitation is the retrospective design of this analysis, which makes it impossible for us to attribute causation. A future analysis of interest may include a prospective design focusing on AAO-HNSF CORE grant applicants and other otolaryngologists-in-training along with their subsequent research trajectories. In addition, an attempt was made to create a more robust prediction model based on some of the predictive factors for NIH funding and hindex suggested in this article (including K-grants, CORE grants, years of publications); however, analysis suggested that these factors were not independent using the DurbinWatson statistic. Another limitation in this analysis relates to the NIH RePORTER database only reporting awards dating back to 1989, representing a 4-year gap since the start of the CORE funding program in 1985 in which CORE-funded individuals may have received NIH funding that was not included in this analysis. Finally, this analysis was not able to nor was it intended to take into account numerous other sources of support for research in the field, including local organizations, intra-institutional funding opportunities, and even internal departmental mechanisms. These potential sources may have value in encouraging individuals to incorporate research into their careers and providing the skills necessary to mature into advanced researchers.
Conclusions A higher proportion of academic otolaryngologists with an AAO-HNSF CORE funding history have received NIH funding relative to their non–CORE-funded peers. This is true when further considering individuals awarded Career Development (K) awards and Research (R-Series) grants from the NIH, as a higher proportion of CORE-funded academicians are successfully awarded these grants than their non–CORE-funded academic colleagues. This suggests that the CORE program may be successful in its stated goals of preparing individuals for the NIH peer review application process, although further prospective study is needed to evaluate whether this association represents a cause and effect relationship. Upon further comparison of relative scholarly impact, individuals with current or prior NIH Kgrants had greater research productivity than those with a CORE funding history. Both of these cohorts have higher scholarly impact values than previously published figures among all academic otolaryngologists, highlighting that both AAO-HNSF CORE grants and NIH K-grants awards are effective career development resources.
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
Eloy et al
251
Acknowledgments The authors would like to thank Stephanie Jones and the CORE grant committee members for their assistance in providing a database of previous grant recipients.
14.
Author Contributions
15.
Jean Anderson Eloy, conception, design, analysis, and interpretation; revision; final approval; Peter F. Svider, conception, design, data acquisition, analysis; drafting; final approval; Vivek V. Kanumuri, data acquisition; revision; final approval; Adam J. Folbe, analysis; revision; final approval; Michael Setzen, analysis; revision; final approval; Soly Baredes, analysis; revision; final approval.
16.
Disclosures
18.
Competing interests: Michael Setzen, speaker for TEVA and MEDA on their Speakers Bureau (not related to current subject).
19.
17.
Sponsorships: None. Funding source: None. 20.
References
21.
1. Carr P, Friedman RH, Moskowitz MA, Kazis LE, Weed HG. Research, academic rank, and compensation of women and men faculty in academic general internal medicine. J Gen Intern Med. 1992;7:418-423. 2. Colaco M, Svider PF, Mauro KM, Eloy JA, Jackson-Rosario I. Is there a relationship between National Institutes of Health funding and research impact on academic urology? J Urol. 2013;190:999-1003. 3. Hauptman JS, Chow DS, Martin NA, Itagaki MW. Research productivity in neurosurgery: trends in globalization, scientific focus, and funding. J Neurosurg. 2011;115:1262-1272. 4. Atasoylu AA, Wright SM, Beasley BW, et al. Promotion criteria for clinician-educators. J Gen Intern Med. 2003;18:711-716. 5. Dixon AK. Publishing and academic promotion. Singapore Med J. 2009;50:847-850. 6. Holden G, Rosenberg G, Barker K. Bibliometrics: a potential decision making aid in hiring, reappointment, tenure and promotion decisions. Soc Work Health Care. 2005;41:67-92. 7. Sanfey H. Promotion to professor: a career development resource. Am J Surg. 2010;200:554-557. 8. Sanfey H, Hollands C. Career development resource: promotion to associate professor. Am J Surg. 2012;204:130-134. 9. Svider PF, Mady LJ, Husain Q, et al. Geographic differences in academic promotion practices, fellowship training, and scholarly impact. Am J Otolaryngol. 2013;34:464-470. 10. Svider PF, Blake DM, Setzen M, Folbe AJ, Baredes S, Eloy JA. Rhinology fellowship training and its scholarly impact. Am J Rhinol Allergy. 2013;27:e131-e134. 11. Svider PF, Choudhry ZA, Choudhry OJ, Baredes S, Liu JK, Eloy JA. The use of the h-index in academic otolaryngology. Laryngoscope. 2013;123:103-106. 12. Jacob BA, Lefgren L. The impact of research grant funding on scientific productivity. J Public Econ. 2011;95:1168-1177. 13. Garrison HH, Drehman B. NIH research funding trends: FY1995-2013. http://www.faseb.org/Policy-and-Government-
22. 23. 24.
25.
26.
27.
28.
29. 30.
31.
32.
Affairs/Data-Compilations/NIH-Research-Funding-Trends.aspx# sthash.PW3i8pW4.dpbs. K Kiosk—information about NIH Career Development Awards.Accessed June 29, 2013. http://grantsnihgov/training/ careerdevelopmentawardshtm. 2013 Centralized Otolaryngology Research Efforts (CORE) Program. http://www.entnet.org/EducationAndResearch/COREGrants.cfm. Eloy JA, Svider PF, Folbe AJ, Setzen M, Baredes S. AAOHNSF CORE grant acquisition is associated with greater scholarly impact. Otolaryngol Head Neck Surg. 2014;150:53-60. Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005;102:1656916572. Hirsch JE. Does the H index have predictive power? Proc Natl Acad Sci U S A. 2007;104:19193-19198. Benway BM, Kalidas P, Cabello JM, Bhayani SB. Does citation analysis reveal association between h-index and academic rank in urology? Urology. 2009;74:30-33. Castillo M. Measuring academic output: the H-index. AJNR Am J Neuroradiol. 2010;31:783-784. Engqvist L, Frommen JG. The h-index and self-citations. Trends Ecol Evol. 2008;23:250-252. Gaster N, Gaster M. A critical assessment of the h-index. Bioessays. 2012;34:830-832. Jeang KT. H-index, mentoring-index, highly-cited and highlyaccessed: how to evaluate scientists? Retrovirology. 2008;5:106. Pagel PS, Hudetz JA. H-index is a sensitive indicator of academic activity in highly productive anaesthesiologists: results of a bibliometric analysis. Acta Anaesthesiol Scand. 2011;55:1085-1089. Pagel PS, Hudetz JA. Bibliometric analysis of anaesthesia journal editorial board members: correlation between journal impact factor and the median h-index of its board members. Br J Anaesth. 2011;107:357-361. Pagel PS, Hudetz JA. Scholarly productivity of United States academic cardiothoracic anesthesiologists: influence of fellowship accreditation and transesophageal echocardiographic credentials on h-index and other citation bibliometrics. J Cardiothorac Vasc Anesth. 2011;25:761-765. Poynard T, Thabut D, Jabre P, et al. Ranking hepatologists: which Hirsch’s h-index to prevent the ‘‘e-crise de foi-e’’? Clin Res Hepatol Gastroenterol. 2011;35:375-386. Quigley MR, Holliday EB, Fuller CD, Choi M, Thomas CR Jr. Distribution of the h-index in radiation oncology conforms to a variation of power law: implications for assessing academic productivity. J Cancer Educ. 2012;27:463-466. Rad AE, Brinjikji W, Cloft HJ, Kallmes DF. The H-index in academic radiology. Acad Radiol. 2010;17:817-821. Rezek I, McDonald RJ, Kallmes DF. Is the h-index predictive of greater NIH funding success among academic radiologists? Acad Radiol. 2011;18:1337-1340. Sharma B, Boet S, Grantcharov T, Shin E, Barrowman NJ, Bould MD. The h-index outperforms other bibliometrics in the assessment of research performance in general surgery: a province-wide study. Surgery. 2013;153:493-501. Lee J, Kraus KL, Couldwell WT. Use of the h index in neurosurgery. Clinical article. J Neurosurg. 2009;111:387-392.
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
252
Otolaryngology–Head and Neck Surgery 151(2)
33. Pagel PS, Hudetz JA. An analysis of scholarly productivity in United States academic anaesthesiologists by citation bibliometrics. Anaesthesia. 2011;66:873-878. 34. Eloy JA, Svider P, Chandrasekhar SS, et al. Gender disparities in scholarly productivity within academic otolaryngology departments. Otolaryngol Head Neck Surg. 2013;148:215-222. 35. Eloy JA, Svider PF, Cherla DV, et al. Gender disparities in research productivity among 9952 academic physicians. Laryngoscope. 2013;123:1865-1875. 36. Eloy JA, Svider PF, Kovalerchik O, Baredes S, Kalyoussef E, Chandrasekhar SS. Gender differences in successful NIH grant funding in otolaryngology. Otolaryngol Head Neck Surg. 2013;149:77-83. 37. Eloy JA, Svider PF, Mauro KM, Setzen M, Baredes S. Impact of fellowship training on research productivity in academic otolaryngology. Laryngoscope. 2012;122:2690-2694. 38. Eloy JA, Svider PF, Folbe AJ, Couldwell WT, Liu JK. Comparison of plaintiff and defendant expert witness qualification in malpractice litigation in neurological surgery. J Neurosurg. 2014;120:185-190. 39. Eloy JA, Svider PF, Patel D, Setzen M, Baredes S. Comparison of plaintiff and defendant expert witness qualification in malpractice litigation in otolaryngology. Otolaryngol Head Neck Surg. 2013;148:764-769. 40. Eloy JA, Svider PF, Patel D, Setzen M, Baredes S. In response to ‘‘comparison of plaintiff and defendant expert witness qualification in malpractice litigation in otolaryngology.’’Otolaryngol Head Neck Surg. 2013;149:649-650. 41. Svider PF, Mauro KM, Sanghvi S, Setzen M, Baredes S, Eloy JA. Is NIH funding predictive of greater research productivity and impact among academic otolaryngologists? Laryngoscope. 2013;123:118-122. 42. Svider PF, Husain Q, Folbe AJ, Couldwell WT, Liu JK, Eloy JA. Assessing National Institutes of Health funding and scholarly impact in neurological surgery. J Neurosurg. 2014;120:191-196. 43. Svider PF, Pashkova AA, Choudhry Z, et al. Comparison of scholarly impact among surgical specialties: an examination of 2429 academic surgeons. Laryngoscope. 2013;123:884-889. 44. Pashkova AA, Svider PF, Chang CY, Diaz L, Eloy JA, Eloy JD. Gender disparity among US anaesthesiologists: are women underrepresented in academic ranks and scholarly productivity? Acta Anaesthesiol Scand. 2013;57:1058-1064.
45. Agarwal N, Clark S, Svider PF, Couldwell WT, Eloy JA, Liu JK. Impact of fellowship training on research productivity in academic neurological surgery. World Neurosurg. 2013;80:738-744. 46. Svider PF, Lopez SA, Husain Q, Bhagat N, Eloy JA, Langer PD. The association between scholarly impact and National Institutes of Health funding in ophthalmology. Ophthalmology. 2014;121:423-428. 47. Eloy JA, Svider PF, Setzen M, Baredes S, Folbe AJ. Does receiving an American Academy of Otolaryngology—Head and Neck Surgery Foundation Centralized Otolaryngology Research Efforts grant influence career path and scholarly impact among fellowship-trained rhinologists? Int Forum Allergy Rhinol. 2014;4(1):85-90. 48. Svider PF, Husain Q, Mauro KM, Folbe AJ, Baredes S, Eloy JA. Impact of mentoring medical students on scholarly productivity. Int Forum Allergy Rhinol. 2014;4(2):138-142. 49. Sunaryo PL, Svider PF, Jackson-Rosario I, Eloy JA. Expert witness testimony in urology malpractice litigation. J Urol. 2014;83(4):704-709. 50. Kasabwala K, Morton CM, Svider PF, Nahass TA, Eloy JA, Jackson-Rosario I. Factors influencing scholarly impact: does urology fellowship-training affect research output? [published online December 30, 2013]. J Surg Educ. doi:http://dx.doi.org/ 10.1016/j.jsurg.2013.10.010 51. Tomei KL, Nahass MM, Husain Q, et al. A gender-based comparison of academic rank and scholarly productivity in academic neurological surgery [published online December 8, 2013]. J Clin Neurosci. doi:10.1016/j.jocn.2013.11.006 52. Eloy JA, Mady LJ, Svider PF, et al. Regional differences in gender promotion and scholarly productivity in otolaryngology. Otolaryngol Head Neck Surg. 2014;150(3):371-377. 53. Folbe AJ, Svider PF, Setzen M, Zuliani G, Lin H, Eloy JA. Scientific inquiry into rhinosinusitis: who is receiving funding from the National Institutes of Health? [published online November 15, 2013] Laryngoscope. doi:10.1002/lary.24525 54. Purvis A. The h index: playing the numbers game. Trends Ecol Evol. 2006;21:422. 55. Kelly CD, Jennions MD. H-index: age and sex make it unreliable. Nature. 2007;449:403.
Downloaded from oto.sagepub.com at NORTH CAROLINA STATE UNIV on May 11, 2015
