Performance and Configuration of Artificial Intelligence in Educational Settings. Introducing a New Reliability Concept Based on Content Analysis

Abstract

Learning analytics represent a promising approach for fostering personalized learning processes. Most applications of this technology currently do not use textual data for providing information on learning, or for deriving recommendations for further development. This paper presents the results of three studies aiming to make textual information usable. In the first study, the iota concept is introduced as a new content analysis measure to evaluate inter-coder reliability. The main advantage of this new concept is that it provides a reliability estimation for every single category, allowing deeper insight into the quality of textual analysis. The second study simulates the process of content analysis, comparing the new iota concept with well-established measures (e.g., Krippendorff’s Alpha, percentage agreement). The results show that the new concept covers the true reliability of a coding scheme, and is not affected by the number of coders or categories, the sample size, or the distribution of data. Furthermore, cut-off values are derived for judging the quality of the analysis. The third study employs the new concept, as it analyzes the performance of different artificial intelligence (AI) approaches for interpreting textual data based on 90 different constructs. The texts used here were either created by apprentices, students, and pupils, or were taken from vocational textbooks. The paper shows that AI can reliably interpret textual information for learning purposes, and also provides recommendations for optimal AI configuration.

Iota Reliability Concept of the Second Generation

“…Yet the range of its applications is not limited to research. Data generated by content analysis can be a valuable source of information in other fields like education in the social sciences (Berding et al, 2022). In educational settings, textual data is omnipresent, manifested in artefacts such as explanations in school books, tasks on worksheets, in written essays or exams, in lesson plans or curricula, or in written communication between teachers and learners in digital learning environments.…”

“…This leads to the application of artificial intelligence (AI), a technology that attempts to simulate human actions (Kleesiek et al, 2020, p. 24). Within educational settings, AI has to understand the textual information, summarize the information in categories of scientific models and theories, and derive the impact of the categories on further learning to provide recommendations for learners and teachers (Berding et al, 2022). For example, if the analysis of a student's text leads to the conclusion that the student has not acquired the "right" understanding of prices, it does not make sense to teach new topics that build on a valid understanding of prices.…”

“…Recent years however have seen the advantages of Krippendorff's Alpha being questioned and controversially discussed (Feng & Zhao, 2016;Krippendorff, 2016;Zhao et al, 2018). Zhao et al (2013) analyzed different reliability measures, concluding that Krippendorff's Alpha contains problematic assumptions and produces the highest number of paradoxes and abnormalities amongst all However, the study by Berding et al (2022) is limited. The true reliability is modeled with the assumption that the reliability is the same over all categories and that the second ordinal variable is measured with perfect reliability.…”

“…Previous measures often used in content analysis such as Krippendorff's Alpha, Percentage Agreement, Scott's Pi, and Cohen's Kappa (Lovejoy et al, 2016) describe the reliability of a scale with one single numeric value assuming that the reliability is constant for the complete scale (Feng and Zhao, 2016). The Iota Concept adapts the basic ideas of modern test theory (Ayala, 2009;Baker and Kim, 2017;Bonifay, 2020;Paek and Cole, 2020) and overcomes this limitation (Berding et al, 2022). These insights can help in the construction of a coding scheme by directly showing where the coding scheme is performing well and where improvements are necessary.…”

“…In chapter 2, we present a short summary of the old Iota Concept as described in Berding et al (2022) as an introduction into the concept. The third chapter presents a refined version of the Iota Concept and compares the new version to the old one.…”