Knowledge Annotation for Intelligent Textbooks

Abstract

With the increased popularity of electronic textbooks, there is a growing interests in developing a new generation of "intelligent textbooks", which have the ability to guide the readers according to their learning goals and current knowledge. The intelligent textbooks extend regular textbooks by integrating machine-manipulatable knowledge such as a knowledge map or a prerequisite-outcome relationship between sections, among which, the most popular integrated knowledge is a list of unique knowledge concepts associated with each section. With the help of these concept, multiple intelligent operations, such as content linking, content recommendation or student modeling, can be performed. However, annotating a reliable set of concepts to a textbook section is a challenge. Automatic unsupervised methods for extracting key-phrases as the concepts are known to have insufficient accuracy. Manual annotation by experts is considered as a preferred approach and can be used to produce both the target outcome and the labeled data for training supervised models. However, most researchers in education domain still consider the concept annotation process as an ad-hoc activity rather than an engineering task, resulting in low-quality annotated data. In this paper, we present a textbook knowledge engineering method to obtain reliable concept annotations. The approach has been applied to produce annotated concepts for Introduction to Information Retrieval textbook. As shown by the data we collected, the inter-annotator agreement gradually increased along with our procedure, and the concept annotations we produced led to better results in document linking and student modeling tasks. The outcomes of our work include a validated knowledge engineering procedure, a code-book for technical concept annotation, and a set of concept annotations for the target textbook, which could be used as gold standard in further research.

Annotation Protocol for Textbook Enrichment with Prerequisite Knowledge Graph

“…Manual annotation is known to be error-prone, as well-recognised in the literature (Fort et al, 2011;Dickinson, 2015;Wang et al, 2021) and also studied in our own work (Alzetta et al 2020a). Therefore, PREAP recommends a revision phase (Annotation revision task in Fig.…”

“…We refer to "concepts" similarly to other works in the literature on prerequisite annotation (Talukdar and Cohen, 2012;Wang et al, 2016;Liang et al, 2017;Pan et al, 2017a;Zhou & Xiao, 2019;Adorni et al, 2019;Alzetta et al, 2019;Limongelli et al, 2015;Xiao et al, 2022) that basically associate concepts to terms, intended as lexical units composed of single or multiple words with unambiguous sense in the given context. Similarly to (Chau et al, 2020b;Wang et al, 2021), we identify terms representing concepts from an educational text as a subset of words therein (more precisely, noun phrases) that convey a domain-specific meaning Cabré (1999). This perspective borrows from the approaches in terminology research, according to which the terminology of a domain provides as many lexical units as there are concepts in its subspace (Sager, 1990), and also from information extraction, which addresses automatic keyword extraction (Augenstein et al, 2017;Shen et al, 2014;Martinez-Rodriguez et al, 2020).…”

“…In fact, PRs can concern prerequisite relations between university courses (Yang et al, 2015;Liang et al, 2017;Li et al, 2019), MOOCs (Chaplot et al, 2016;Pan et al, 2017a;Roy et al, 2019;Zhao et al, 2020), MOOC videos (Pan et al, 2017c;Huang et al, 2021;Wen et al, 2021;Xiao et al, 2021), learning objects (Gasparetti, 2022), scientific databases (Gordon et al, 2017) or Wikipedia pages (Talukdar and Cohen, 2012;Gasparetti et al, 2018;Miaschi et al, 2019;Zhou & Xiao, 2019;Sayyadiharikandeh et al, 2019;Bai et al, 2021;Hu et al, 2021), all represented as PR relations between concept pairs. Alternatively, concepts can be relevant domain terms acquired from a text, as in (Wang et al, 2016;Lu et al, 2019;Adorni et al, 2019;Alzetta et al, 2019;Chau et al, 2020b;Wang et al, 2021), and in our approach.…”

“…In fact, although properly defining an annotation task is vital to reduce annotation inconsistencies (Ide & Pustejovsky, 2017), the available PR-annotated datasets are mostly poorly documented, and annotation guidelines tend to be absent or fairly basic, mostly relying on a naive definition of prerequisite relation. PREAP tries to fill this gap in PR literature as it defines a systematic procedure for annotating educational texts: we could not find any other knowledge engineering procedure for prerequisite annotation and PR dataset creation, while methods exist for the mere task of concept annotation, including a recent one from (Wang et al, 2021).…”

Measuring the Quality of Domain Models Extracted from Textbooks with Learning Curves Analysis

“…Over the last decade, a range of approaches have been introduced capable of extracting different elements of domain semantics from domain-oriented documents and user data [6,7,11]. However, very few studies examined the applicability of automatically extracted domain semantics to the task of modeling student knowledge [13].…”