Automatic Classification of Swedish Metadata Using Dewey Decimal Classification: A Comparison of Approaches

Abstract

PurposeWith more and more digital collections of various information resources becoming available, also increasing is the challenge of assigning subject index terms and classes from quality knowledge organization systems. While the ultimate purpose is to understand the value of automatically produced Dewey Decimal Classification (DDC) classes for Swedish digital collections, the paper aims to evaluate the performance of six machine learning algorithms as well as a string-matching algorithm based on characteristics of DDC.Design/methodology/approachState-of-the-art machine learning algorithms require at least 1,000 training examples per class. The complete data set at the time of research involved 143,838 records which had to be reduced to top three hierarchical levels of DDC in order to provide sufficient training data (totaling 802 classes in the training and testing sample, out of 14,413 classes at all levels).FindingsEvaluation shows that Support Vector Machine with linear kernel outperforms other machine learning algorithms as well as the string-matching algorithm on average; the string-matching algorithm outperforms machine learning for specific classes when characteristics of DDC are most suitable for the task. Word embeddings combined with different types of neural networks (simple linear network, standard neural network, 1D convolutional neural network, and recurrent neural network) produced worse results than Support Vector Machine, but reach close results, with the benefit of a smaller representation size. Impact of features in machine learning shows that using keywords or combining titles and keywords gives better results than using only titles as input. Stemming only marginally improves the results. Removed stop-words reduced accuracy in most cases, while removing less frequent words increased it marginally. The greatest impact is produced by the number of training examples: 81.90% accuracy on the training set is achieved when at least 1,000 records per class are available in the training set, and 66.13% when too few records (often less than 100 per class) on which to train are available—and these hold only for top 3 hierarchical levels (803 instead of 14,413 classes).Research limitationsHaving to reduce the number of hierarchical levels to top three levels of DDC because of the lack of training data for all classes, skews the results so that they work in experimental conditions but barely for end users in operational retrieval systems.Practical implicationsIn conclusion, for operative information retrieval systems applying purely automatic DDC does not work, either using machine learning (because of the lack of training data for the large number of DDC classes) or using string-matching algorithm (because DDC characteristics perform well for automatic classification only in a small number of classes). Over time, more training examples may become available, and DDC may be enriched with synonyms in order to enhance accuracy of automatic classification which may also benefit information retrieval performance based on DDC. In order for quality information services to reach the objective of highest possible precision and recall, automatic classification should never be implemented on its own; instead, machine-aided indexing that combines the efficiency of automatic suggestions with quality of human decisions at the final stage should be the way for the future.Originality/valueThe study explored machine learning on a large classification system of over 14,000 classes which is used in operational information retrieval systems. Due to lack of sufficient training data across the entire set of classes, an approach complementing machine learning, that of string matching, was applied. This combination should be explored further since it provides the potential for real-life applications with large target classification systems.

Labelling the past: data set creation and multi-label classification of Dutch archaeological excavation reports

“…While multi-label classification is a well-studied subject, in this paper we perform this task on a noisy data set in an expert domain, making the process more challenging. Even though the difficulty of the task is high, we achieve decent results: we achieve comparable or better scores when compared to similar studies in other domains (Golub et al, 2020;Kleppe et al, 2019). We also specifically test which pre-processing methods have a positive effect on classification, and provide the created data in an online repository 2 .…”

“…These characteristics are not unique to the archaeology domain, and are also often encountered in e.g. the biomedical domain (Laza et al, 2011) and library domain (Golub et al, 2020).…”

Determining subject headings of documents using information retrieval models

“…The evaluation is therefore performed on the unit of subject headings instead of documents, which further differentiates this work with a standard document retrieval task. Our idea is supported by Golub et al [19] who concluded that automatic subject heading assignment should never be implemented on its own; instead, a system should combine the efficiency of automatic suggestions with quality of human decisions at the final stage. They found that applying purely automatic subject heading classification does not work, because there are a large number of subject headings classes.…”

“…later, sapon-white and hansbrough [9] showed that the dissertations with subject headings are found to be more likely to circulate than those without subject headings. Some researchers have also studied subject heading for various purposes, such as: analysing subjects listed in different subject heading [10]- [12], generating map of science [13], [14], developing simplified subject heading list [15], [16], and assigning subject heading automatically [17]- [19].…”

“…Some previous work that are mostly related to our work are summarized in Table 1. They investigated the ways to assign subject headings automatically [17]- [19]. Our work is different to them in which we formulate our task as a subject heading recommendation/retrieval instead of subject heading assignment or classification.…”