DNN-Based Scoring of Language Learners’ Proficiency Using Learners’ Shadowings and Native Listeners’ Responsive Shadowings

Kabashima, Suguru; Inoue, Y.; Saito, Daisuke; Minematsu, Nobuaki

doi:10.1109/slt.2018.8639645

Cited by 3 publications

(13 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [4,5], a novel method of predicting comprehensibility of an utterance was proposed, that is native listeners' reverse shadowing and it does not require any special device like EEG or eyetracker. In the conventional form of shadowing, native utterances are presented to learners, who are shadowers.…”

Section: Reverse Form Of Shadowingmentioning

confidence: 99%

“…Two speech features used were used in [4,5] to predict comprehensibility. They are accuracy of articulation and delay of shadowing.…”

Section: Comprehensibility Prediction From Natives' Shadowingmentioning

confidence: 99%

“…For a shadowing utterance with K segments, the averaged GOP score over {GOP(k)} is calculated as shadowability score for that utterance. In [4,5], Japanese sentences read by Vietnamese learners and native listeners' shadowings were used for analysis, and P (pt|ot) was calculated by a DNN-based front end of a Japanese speech recognizer, trained with CSJ [24]-based KALDI [25]. The GOP scores from natives' shadowings were shown to be very highly correlated with comprehensibilities subjectively judged by the native shadowers.…”

Section: Comprehensibility Prediction From Natives' Shadowingmentioning

confidence: 99%

“…As for delay of shadowing, in [4,5], by comparing a forced alignment result of a Vietnamese-Japanese (VJ) utterance and that of its native reverse shadowing (RS), the temporal gap between every pair of phoneme boundaries was obtained between the two utterances. The phoneme-based temporal gaps obtained from the two utterances were averaged to define delay of shadowing between the two.…”

Section: Comprehensibility Prediction From Natives' Shadowingmentioning

confidence: 99%

“…This is why the authors consider that physiological sensing is impractical and very difficult to be used on a cloud sourcing infrastructure. Our previous studies [4,5] showed another possibility only with an inexpensive device, that is a microphone, and listeners' responses were observed acoustically. Native listeners were asked to shadow given non-native speech and smoothness of their shadowing was calculated acoustically.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Native Listeners' Shadowing of Non-native Utterances as Spoken Annotation Representing Comprehensibility of the Utterances

Lin

Inoue

Trisitichoke

et al. 2019

8th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2019)

Self Cite

View full text Add to dashboard Cite

Recently, researchers' attention has been paid to pronunciation assessment not based on comparison between learners' utterances and native models, but based on comprehensibility of the utterances [1, 2, 3]. In our previous studies [4, 5], native listeners' shadowing was investigated and shown to be effective to predict comprehensibility perceived by listeners (shadowers). In this paper, native listeners' shadowings are viewed as spoken annotations that can represent comprehensibility. In [4, 5], to predict comprehensibility of a non-native utterance, the GOP scores of its corresponding native listeners' shadowings were calculated by using a DNN-based ASR front-end. Generally speaking, annotations are prepared manually and, even when some techniques are used for annotations, only stable and reliable techniques should be used. In this paper, a simpler, stabler, and more reliable method to derive comprehensibility annotations was proposed. After native listeners' shadowing, they are asked to read aloud the sentence intended by the learner. Reading is the most prepared speech and shadowing is probably the least prepared speech. DTW between the two utterances is supposed to be able to quantify and predict comprehensibility or shadowability perceived by the shadowers. In experiments, DTW between shadowings and readings shows higher correlation than the GOP scores of shadowings.

show abstract

Section: Reverse Form Of Shadowingmentioning

confidence: 99%

“…Two speech features used were used in [4,5] to predict comprehensibility. They are accuracy of articulation and delay of shadowing.…”

Section: Comprehensibility Prediction From Natives' Shadowingmentioning

confidence: 99%

Section: Comprehensibility Prediction From Natives' Shadowingmentioning

confidence: 99%

Section: Comprehensibility Prediction From Natives' Shadowingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Native Listeners' Shadowing of Non-native Utterances as Spoken Annotation Representing Comprehensibility of the Utterances

Lin

Inoue

Trisitichoke

et al. 2019

8th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2019)

Self Cite

View full text Add to dashboard Cite

show abstract

A Large Collection of Sentences Read Aloud by Vietnamese Learners of Japanese and Native Speaker's Reverse Shadowings

Ando

Lin

Trisitichoke

et al. 2019

2019 22nd Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databa

View full text Add to dashboard Cite

Analysis of Native Listeners’ Facial Microexpressions While Shadowing Non-Native Speech — Potential of Shadowers’ Facial Expressions for Comprehensibility Prediction

Trisitichoke

Ando²,

Saito

et al. 2019

Interspeech 2019

Self Cite

View full text Add to dashboard Cite

Recently, researchers' attention has been paid to pronunciation assessment not based on comparison between L2 speech and native models, but based on comprehensibility of L2 speech [1, 2, 3]. In our previous studies [4, 5, 6], native listeners' shadowing of L2 speech was examined and it was shown that delay of shadowing and accuracy of articulation in shadowing utterances, both of which were acoustically calculated, are strongly influenced by the amount of cognitive load imposed for understanding L2 speech, especially when it is with strong accents. In this paper, aside from acoustic analysis of shadowings, we focus on shadowers' facial microexpressions and examine how they are correlated with perceived comprehensibility. To extract facial expression features, two methods are tested. One is a computer-vision-based method and recorded videos of shadowers' facial expressions are analyzed. The other is a method using a physiological sensor that can detect subtle movements of facial muscles. In experiments, four shadowers' facial expressions are analyzed, each of whom shadowed approximately 800 L2 utterances. Results show that some of shadowers' facial expressions are highly correlated with perceived comprehensibility, and that those facial expressions are strongly shadowerdependent. These results indicate a high potential of shadowers' facial expressions for comprehensibility prediction.

show abstract

DNN-Based Scoring of Language Learners’ Proficiency Using Learners’ Shadowings and Native Listeners’ Responsive Shadowings

Cited by 3 publications

References 20 publications

Native Listeners' Shadowing of Non-native Utterances as Spoken Annotation Representing Comprehensibility of the Utterances

Native Listeners' Shadowing of Non-native Utterances as Spoken Annotation Representing Comprehensibility of the Utterances

A Large Collection of Sentences Read Aloud by Vietnamese Learners of Japanese and Native Speaker's Reverse Shadowings

Analysis of Native Listeners’ Facial Microexpressions While Shadowing Non-Native Speech — Potential of Shadowers’ Facial Expressions for Comprehensibility Prediction

Contact Info

Product

Resources

About