Intelligent Tutoring Systems (ITSs) are aimed at promoting acquisition of knowledge and skills by providing relevant and appropriate feedback during students' practice activities. ITSs for literacy instruction commonly assess typed responses using Natural Language Processing (NLP) algorithms. One step in this direction often requires building a scoring mechanism that matches human judgments. This paper describes the challenges encountered while implementing an automated evaluation workflow and adopting solutions for increasing performance of the tutoring system. The algorithm described here comprises multiple stages, including initial pre-processing, a rule-based system for pre-classifying self-explanations, followed by classification using a Support Virtual Machine (SVM) learning algorithm. The SVM model hyper-parameters were optimized using grid search approach with 4,109 different selfexplanations scored 0 to 3 (i.e., poor to great). The accuracy achieved for the model was 59% (adjacent accuracy = 97%; Kappa = .43).
Natural Language Processing has massively evolved during the last years and many up-to-date applications integrate different speech tools in order to create an enhanced user experiences. For obtaining a seamless integration of existing speech recognition systems, there is a trending interest for developing and improving existing speech-to-text algorithms. The aim of this paper is to improve user interaction with the ReaderBench platform, by developing and integrating a speech recognition module designed so that young pupils can dictate their self-explanations to a given text. Afterwards, the ReaderBench framework is used to automatically evaluate the employed reading strategies based on the resulted speech transcriptions. A dataset containing 160 self-explanations from students ranging from 9 to 11 years old was analysed using both original transcripts, and the ones automatically generated by our custom speech recognition system. Multiple methods designed to perform speech recognition are also compared, while a new dedicated model was trained in order to improve the quality of the existing French model for speech recognition from CMUSphinx speech recognition system. Our revised model includes a pronunciation dictionary obtained after training a Long Short-Term Memory (LSTM) Grapheme-to-Phoneme neural network. The accuracy of our system is benchmarked in relation to the automated process of identifying reading strategies implemented in our ReaderBench framework, which is applied on both manual transcriptions and automated speech-to-text inputs. The obtained results argue for the adequacy of our method as the slight decrease in terms of identification accuracy is justifiable in contrast to the effort of manually transcribing each self-explanation.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.