A machine‐learning approach to coding book reviews as quality indicators: Toward a theory of megacitation

Abstract

A theory of “megacitation” is introduced and used in an experiment to demonstrate how a qualitative scholarly book review can be converted into a weighted bibliometric indicator. We employ a manual human‐coding approach to classify book reviews in the field of history based on reviewers' assessments of a book author's scholarly credibility (SC) and writing style (WS). In total, 100 book reviews were selected from the American Historical Review and coded for their positive/negative valence on these two dimensions. Most were coded as positive (68% for SC and 47% for WS), and there was also a small positive correlation between SC and WS (r = 0.2). We then constructed a classifier, combining both manual design and machine learning, to categorize sentiment‐based sentences in history book reviews. The machine classifier produced a matched accuracy (matched to the human coding) of approximately 75% for SC and 64% for WS. WS was found to be more difficult to classify by machine than SC because of the reviewers' use of more subtle language. With further training data, a machine‐learning approach could be useful for automatically classifying a large number of history book reviews at once. Weighted megacitations can be especially valuable if they are used in conjunction with regular book/journal citations, and “libcitations” (i.e., library holding counts) for a comprehensive assessment of a book/monograph's scholarly impact.

Revisiting an open access monograph experiment: measuring citations and tweets 5 years later

“…When the library-generated data is compared with the available citation data, again the same pattern emerges: at best a weak correlation between the ‘alternative’ metrics and citations. Quite a different approach is used by Zuccala et al (2014), who use machine-learning techniques to automatically classify the conclusions of book reviews in the field of history. However, the reported results derive from a pilot experiment, and no correlation to citations is described.…”

Language, Culture and Traversing the Scholarly Evaluation Landscape

“…This does not mean, however, that the products of humanistic research cannot be quantified. What it means is that when we work with textual and symbolic material quantitatively versus qualitatively there is potential to obtain different types and levels of insight (e.g., see Zuccala et al 2014). In fact, the history of 'citation-ology' (i.e., the study of referencing and citation behavior) has already demonstrated that what we learn from approaching the highly textual, contextual, and symbolic citation, using a qualitative method of investigation can differ greatly than when we approach the same citation using a quantitative method of analysis (Bornmann and Daniel 2008;Brooks 1985;McCain 2006;Small 1978).…”

Beyond Coverage: Toward a Bibliometrics for the Humanities

“…Thus, a system that counts written reviews could disadvantage younger and less renowned scholars. Another alternative is to view book reviews as 'mega-citations' that indicate the quality of a book (Zuccala et al 2014). This approach has many advantages, especially since book reviews play an important function in the humanities; however, many books are never reviewed, and the overall coverage is possibly too low for systematic assessment.…”