Deep-Learning Based Automatic Spontaneous Speech Assessment in a Data-Driven Approach for the 2017 SLaTE CALL Shared Challenge

Abstract

This paper presents a deep-learning based assessment method of a spoken computer-assisted language learning (CALL) for a non-native child speaker, which is performed in a data-driven approach rather than in a rule-based approach. Especially, we focus on the spoken CALL assessment of the 2017 SLaTE challenge. To this end, the proposed method consists of four main steps: speech recognition, meaning feature extraction, grammar feature extraction, and deep-learning based assessment. At first, speech recognition is performed on an input speech using three automatic speech recognition (ASR) systems. Second, twenty-seven meaning features are extracted from the recognized texts via the three ASRs using language models (LMs), sentence-embedding models, and wordembedding models. Third, twenty-two grammar features are extracted from the recognized text via one ASR system using linear-order LMs and hierarchical-order LMs. Fourth, the extracted forty-nine features are fed into a full-connected deep neural network (DNN) based model for the classification of acceptance or rejection. Finally, an assessment is performed by comparing the probability of a output unit of the DNN-based classifier with a predefined threshold. For the experiments of a spoken CALL assessment, we use English spoken utterances by Swiss German teenagers. It is shown from the experiments that the D score is 4.37 for the spoken CALL assessment system employing the proposed method.

Automatic Assessment of Spoken Language Proficiency of Non-native Children

“…For each test sentence formed by NW words, of which NOOV are out-of-vocabulary (OOV), we compute the following 5 features using each LM (taking inspiration for features from the works of [29,30,12,9,10]): a) log(P ) N W , that is, the average log-probability of the sentence, b) log(P OOV ) N OOV , that is, the average contribution of OOV words to the log-probability of the sentence, c) log(P )−log(P OOV ) N W , that is, the average log-difference between the two above probabilities, d) NW − N bo , where N bo is the number of back-offs applied by the LM to the input sentence (this difference is related to the frequency of n-grams in the sentence that have also been observed in the training set), e) NOOV , the number of OOVs in the sentence. Note that if word counts NW or NOOV are equal to zero (i.e.…”

Overview of the 2019 Spoken CALL Shared Task

“…This new data was selected in a similar way to the first training set, to be balanced and representative of the collected data, with the additional constraint that there should be no overlap of individual students between the first task and second task. Speech data were processed through the two best speech recognisers from the first shared task [6,8] after which the two sets of output transcriptions were merged and cleaned up by transcribers at the University of Geneva.…”

“…The cleaned, merged transcriptions were processed through four of the best assessment systems from the first shared task [6,8,7,9] to give accept/reject decisions for the language criterion. The training data could then be divided into three groups according to the agreement among the four systems.…”

“…The first edition of the task was announced at LREC 2016 [5], with training data released in July 2016 and test data in March 2017, and attracted 20 entries. Results, including seven papers, were presented at the SLaTE workshop in August 2017 (http://regulus.unige.ch/ spokencallsharedtask/; [4,6,7,8,9,10,11]).…”

Liulishuo's System for the Spoken CALL Shared Task 2018

“…Several types of machine learning models have been used in the first edition of this challenge, for example, Support Vector Machine (SVM), K-Nearest Neighbor models, and Feed-Forward Neural networks [9,15,16].…”