Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study

Abstract

Background Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal. Objective This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators. Methods The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results. Results A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001). Conclusions Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.

Growing the next generation of trauma surgeon-scientists: Reflections on 20 years of research investment

“…Qua et al 11 found similar findings with 44% of KL2 and 51% of K08/K23 scholars from their institution subsequently holding R01 funding. Guo et al 12 also found 59% of institutional KL2 scholars acquired subsequent funding.…”

“…More recent data have shown that 80% of surgeon K award holders applied for R01 awards with 39% achieving successful R01 funding. 10 Qua et al 11 found similar findings with 44% of KL2 and 51% of K08/K23 scholars from their institution subsequently holding R01 funding. Guo et al 12 also found 59% of institutional KL2 scholars acquired subsequent funding.…”

“…A study from Case Western Reserve University found that KL2 and K08/K23 recipients had a median of 15 and 10 publications within 5 years of their awards and 27 and 20, respectfully within 8 years of their award. 11 In contrast, our SCH recipients within the last 5 years have a median of 34, and INV, a median of 33 peer-reviewed publications.…”

“…For those remaining in academics (90% SCH), PUBs were higher for those >5 years (66 [IQR,) versus <5 years from their ; p = 0.05), but there was no difference in h-index (16 [IQR, 14-25] vs. 15 [9-19], p = NS). Comparing the most recent 5 years of SCH to INV group, there was no difference in academic productivity as measured by total PUBs (SCH, 33 [IQR, vs. INV, 34 [IQR,; p = 0.7) or h-index (INV, 9 [IQR,(5)(6)(7)(8)(9)(10)(11)(12)(13)(14); p = 0.1). However, no attendee held research funding before INV, but 31.6% (6 of 19 attendees) have subsequently acquired ≥1 funding award (11 non-NIH, 1 NIH) in the short interval since participation.…”

Evaluating the impact of a CTSA program from 2008 to 2021 through bibliometrics, social network analysis, and altmetrics

“…Recent evaluation studies by CTSAs also confirmed the validity and feasibility of bibliometrics as a critical approach to CTSA-supported translational research evaluation [12,14]. For example, bibliometrics has been applied to assess (1) an https://doi.org/10.1017/cts.2022.530 Published online by Cambridge University Press individual CTSA program hub [11], a group of CTSA program hubs [14], overall CTSA consortium or a specific program across CTSAs [12,13,15,21]; (2) research productivity and citation impact using both basic publication/citation counts and advanced citation impact indicators (e.g., iCite's relative citation ratio, Elsevier's Field-Weighted-Citation-Impact, and Web of Science's Category Normalized Citation Impact) [11][12][13][14][15]21]; (3) interdisciplinary or inter-CTSA collaborations [11,12]; and (4) research areas align with the translational spectrum [11,12].…”