The use of ChatGPT to find similar institutions for institutional benchmarking

Abstract

In evaluative bibliometrics and higher education studies, one is frequently confronted with the task of comparing institutions with similar institutions. In this Letter to the Editor, a simple approach is discussed which applies ChatGPT. Although the approach seems to produce promising results (tested with an example at the level of research institute and of a university), it is necessary to investigate it systematically based on a sample including many institutions before it should be applied in research evaluation. The challenge in systematic investigations is that ChatGPT provides the user with different answers on the sane request (missing reliability).

How to measure Interdisciplinary Research? A `systemic' design for the model of measurement

“…Alternative methods to identify a Z matrix of similarity events may involve adopting scores generated by Large Langange Models (LLM). However, these could lack explainability and reproducibility: especially for unexpected similarities it can be difficult to discern whether the model has uncovered meaningful but hidden interdisciplinary connections, or it is just hallucinating an answer (Ray, 2023;Bornmann and Lepori, 2024;Thelwall, 2024).…”