Impact for whom? Mapping the users of public research with lexicon-based text mining

Bonaccorsi, Andrea; Chiarello, Filippo; Fantoni, Gualtiero

doi:10.1007/s11192-020-03803-z

Cited by 9 publications

(2 citation statements)

References 84 publications

(66 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Much of this work primarily focuses on the qualitative data that arises from the universities' narrative impact case studies. The approaches include topic modelling [25,30], social network analysis [31,32], and comparative linguistic analysis of high-scoring versus low-scoring submissions [33,34]. These provide a wealth of information on the narrative features, such as the work of Reichard et al [33], which used text mining to evidence statistically significant variation in writing style between the high-scoring and low-scoring impact case study submissions.…”

Section: Emerging Machine Learning Techniques For Research Evaluationmentioning

confidence: 99%

Exploring the application of machine learning to expert evaluation of research impact

et al. 2023

View full text Add to dashboard Cite

The objective of this study is to investigate the application of machine learning techniques to the large-scale human expert evaluation of the impact of academic research. Using publicly available impact case study data from the UK’s Research Excellence Framework (2014), we trained five machine learning models on a range of qualitative and quantitative features, including institution, discipline, narrative style (explicit and implicit), and bibliometric and policy indicators. Our work makes two key contributions. Based on the accuracy metric in predicting high- and low-scoring impact case studies, it shows that machine learning models are able to process information to make decisions that resemble those of expert evaluators. It also provides insights into the characteristics of impact case studies that would be favoured if a machine learning approach was applied for their automated assessment. The results of the experiments showed strong influence of institutional context, selected metrics of narrative style, as well as the uptake of research by policy and academic audiences. Overall, the study demonstrates promise for a shift from descriptive to predictive analysis, but suggests caution around the use of machine learning for the assessment of impact case studies.

show abstract

Section: Emerging Machine Learning Techniques For Research Evaluationmentioning

confidence: 99%

Exploring the application of machine learning to expert evaluation of research impact

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Automated tools commonly used in literature reviews analyse textual data at a lexical level (see Smith & Humphreys, 2006;Blei et al, 2003;Marrone & Hammerle, 2016;Bonaccorsi et al, 2021), meaning that they disregard the semantic relations between words. The concern with carrying out any analysis at the lexical level is that homographs (i.e., words with the same spelling but different meanings) are not disambiguated.…”

Section: Introductionmentioning

confidence: 99%

Entity linking systems for literature reviews

2022

View full text Add to dashboard Cite

Computer-assisted methods and tools can help researchers automate the coding process of literature reviews and accelerate the literature review process. However, existing approaches for coding textual data do not account for lexical ambiguity; that is, instances in which individual words have multiple meanings. To counter this, we developed a method to conduct rapid and comprehensive analyses of diverse literature types. Our method uses entity linking and keyword analysis and is embedded into a literature review framework. Next, we apply the framework to review the literature on digital disruption and digital transformation. We outline the method’s advantages and its applicability to any research topic.

show abstract

“How Do You Measure a Relationship?” Assessment and Evaluation Challenges of Knowledge Exchange Activities in Information Work

O'Brien,

McDavid,

Yao

2023

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

Today there is increasing emphasis on knowledge exchange (KE), the movement of knowledge and expertise amongst diverse groups to enhance research uptake, use, and impact in healthcare, government, and community settings. Library and information science (LIS) professionals are central actors in KE though community engagement, scholarly communication, literacy, and cultural heritage initiatives, but (how) is this work formally documented and evaluated? Through interviews with 24 information professionals working in varied settings, we considered how KE activities fit into the current library assessment and evaluation landscape. Using thematic analysis, we identified challenges with placing this work within current assessment practices and evaluation frameworks and showing its value, as well as a desire for alternative, more dynamic assessment and evaluation methods. We discuss these findings with respect to previous research in LIS and KE more broadly to consider professional and organizational implications.

show abstract

Impact for whom? Mapping the users of public research with lexicon-based text mining

Cited by 9 publications

References 84 publications

Exploring the application of machine learning to expert evaluation of research impact

Exploring the application of machine learning to expert evaluation of research impact

Entity linking systems for literature reviews

“How Do You Measure a Relationship?” Assessment and Evaluation Challenges of Knowledge Exchange Activities in Information Work

Contact Info

Product

Resources

About