Terms in journal articles associating with high quality: can qualitative research be world-leading?

Abstract

PurposeScholars often aim to conduct high quality research and their success is judged primarily by peer reviewers. Research quality is difficult for either group to identify, however and misunderstandings can reduce the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek quantitative evidence of aspects of research that are associated with high or low quality.Design/methodology/approachWe extracted the words and 2–5-word phrases most strongly associated with different quality scores in each of 34 Units of Assessment (UoAs) in the Research Excellence Framework (REF) 2021. We extracted the terms from 122,331 journal articles 2014–2020 with individual REF2021 quality scores.FindingsThe terms associating with high- or low-quality scores vary between fields but relate to writing styles, methods and topics. We show that the first-person writing style strongly associates with higher quality research in many areas because it is the norm for a set of large prestigious journals. We found methods and topics that associate with both high- and low-quality scores. Worryingly, terms associated with educational and qualitative research attract lower quality scores in multiple areas. REF experts may rarely give high scores to qualitative or educational research because the authors tend to be less competent, because it is harder to do world leading research with these themes, or because they do not value them.Originality/valueThis is the first investigation of journal article terms associating with research quality.

In which fields are citations indicators of research quality?

“…3.2.2. of: Thelwall et al, 2022). The bucket size of 25 seems like a reasonable compromise between too fine grained and too broad.…”

“…The range of scores for each UoA are available online (fig. 3.2.2 of the main report : Thelwall et al, 2022) alongside other background information about the dataset. The articles were spread reasonably evenly between 2014 and 2020, from 11% in 2014 to 16% in 2018.…”

Do altmetric scores reflect article quality? Evidence from the UK Research Excellence Framework 2021

“…Provisional REF2021 scores from March 2022 for 148,977 journal articles from 2014 to 2020 were supplied by the REF team as part of an unrelated project (Thelwall et al, 2022). These are either the final scores or with a few article scores changed.…”

“…Exact numbers for each field and year are in the supplementary materials (https://doi.org/10.6084/m9.figshare.21938234) but there were low numbers for the arts and humanities UoAs. In the unrelated project report (Thelwall et al, 2022), see tab. 3.6.1 for overall duplicate information and fig.…”

Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paper

“…In contrast, there was no evidence of association in the arts and humanities, and the decision sciences seemed to benefit from fewer authors (Thelwall, Kousha, Abdoli, Stuart, Makita, Wilson, & Levitt, 2023d). For the United Kingdom, after controlling for the effect of collaboration, having international (rather than national) co‐authors associates with higher quality research in 27 out of the 34 Units of Assessment, with collaboration with other advanced economies being particularly advantageous and collaboration with weaker economies tending to be a disadvantage from a quality perspective (Thelwall et al, 2022d).…”