Using bibliometrics to evaluate translational science training: evidence for early career success of KL2 scholars

Abstract

INTRODUCTION: Evaluating clinical and translational research (CTR) mentored training programs is challenging because no two programs are alike. Careful selection of appropriate metrics is required to make valid comparisons between individuals and between programs. The KL2 program provides mentored-training for early stage CTR investigators. Clinical and Translational Awards (CSTAs) across the country have unique KL2 programs. The evaluation of KL2 programs has begun to incorporate bibliometrics to measure KL2 scholar and program impact. METHODS: This study investigated demographic differences in bibliometric performance and post-K award funding of KL2 scholars and compared the bibliometric performance and post-K award federal funding of KL2 scholars and other mentored-K awardees at the same institution. Data for this study included SciVal and iCite bibliometrics and NIH RePORTER grant information for mentored-K awardees (K08, K23, and KL2) at Case Western Reserve University (CWRU) between 2005 and 2013. RESULTS: Results showed no demographics differences within the KL2 program scholars. Bibliometric differences between KL2 and other mentored-K awardee indicated an initial KL2 advantage for the number of publications at five years post-matriculation (i.e., the start of the K-award). Regression analyses indicated the number of initial publications was a significant predictor of federal grant funding at the same time point. Analysis beyond the five-year post matriculation point did not result in a sustained, significant KL2-advantage. CONCLUSIONS: Factors that contributed to the grant funding advantage need to be determined. Additionally, differences between translational and clinical bibliometrics must be interpreted with caution and appropriate metrics for translational science must be established.

Growing the next generation of trauma surgeon-scientists: Reflections on 20 years of research investment

“…13 K awardees are much more likely to receive a subsequent R01 than those who never held career development award with the mentoring component of the award identified as a key factor to success especially for women. 14 In addition to obtaining extramural funding, the success of our SCH recipients in terms of publication productivity was also higher than that reported by other society recipient awardees. The American College of Gastroenteology reported a mean of 45 peer-reviewed publications after a society scholarship award.…”

“…2). Overall h-index was 16 (IQR, 12-21), and there was no difference in h-index between those >5 years versus ≤5 years from their SCH (16 [IQR, [14][15][16][17][18][19][20][21][22][23][24][25] vs. 15 [IQR, 9-19], p = NS). Distribution of h-index for SCH recipients is shown in Figure 3.…”

“…This is up from earlier work on KL2 programs across the country with awardees from 2006 to 2013 showing only a 39% of conversion to R01 funding 13 . K awardees are much more likely to receive a subsequent R01 than those who never held career development award with the mentoring component of the award identified as a key factor to success especially for women 14 …”

“…For those remaining in academics (90% SCH), PUBs were higher for those >5 years (66 [IQR,) versus <5 years from their ; p = 0.05), but there was no difference in h-index (16 [IQR, 14-25] vs. 15 [9-19], p = NS). Comparing the most recent 5 years of SCH to INV group, there was no difference in academic productivity as measured by total PUBs (SCH, 33 [IQR, vs. INV, 34 [IQR,; p = 0.7) or h-index (INV, 9 [IQR,(5)(6)(7)(8)(9)(10)(11)(12)(13)(14); p = 0.1). However, no attendee held research funding before INV, but 31.6% (6 of 19 attendees) have subsequently acquired ≥1 funding award (11 non-NIH, 1 NIH) in the short interval since participation.…”

Big splashes & ripple effects: a narrative review of the short- & long-term impact of publications supported by an NIH CTSA pediatrics program

“…Indeed, others have suggested that most current citation factors are illsuited for comparing across scientific disciplines and that the use of multiple indicators is crucial (16,17). Traditional bibliometrics have been used to provide overviews that focus on long-term influence in particular areas of pediatric research (18)(19)(20)(21), and among CTSA-supported programs (8,22). However, this is the first analysis to present complimentary bibliometric tools used together to understand and discuss both short-term attention and long-term influence of publications and shed light on how pediatric research is being used within both academic and public networks.…”