An ASR-based Reading Tutor for Practicing Reading Skills in the First Grade: Improving Performance through Threshold Adjustment

Bai, Yu; Hubers, Ferdy; Cucchiarini, Catia; Strik, Helmer

doi:10.21437/iberspeech.2021-3

Cited by 3 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the previous study [3] we found that 48 was the threshold that produced the highest agreement between the ASR scores for correct/incorrect and the human judgment [3]. After removing invalid audio recordings, the best threshold shifted to 46.…”

Section: Performance Evaluation Of Different Systemsmentioning

confidence: 76%

“…The reading tutor used by pupils during the experiments made use of the Nov-oLearning ASR engine, in which the ASR model (ASR.1) was trained with Dutch adult speech. It calculates probability scores and gives right/wrong feedback according to the adjustable threshold [3]. Two other systems (system ASR.2 and system ASR.3) were built with Kaldi after the experimental tests with children.…”

Section: Asr Systemsmentioning

confidence: 99%

“…These probability measures are compared to a threshold to obtain classification scores. In a previous paper [3], we focused on optimizing this threshold.…”

Section: Procedures For Performance Evaluation Of Asr Systemsmentioning

confidence: 99%

“…A good reading tutor for children should avoid too many false rejections (FRs), which will lower the credibility of the reading tutor and cause frustration, and detect miscues that pupils can correct [25]. In the previous study, we evaluated the agreement between DART reading tutor with Novo-ASR backend and human transcripts [3]. The highest agreement obtained measured by Cohen's Kappa is .41, which is classified as moderate agreement [18].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An ASR-Based Tutor for Learning to Read: How to Optimize Feedback to First Graders

Bai¹,

Tejedor-Garcı́a²,

Hubers³

et al. 2021

Speech and Computer

Self Cite

View full text Add to dashboard Cite

The interest in employing automatic speech recognition (ASR) in applications for reading practice has been growing in recent years. In a previous study, we presented an ASR-based Dutch reading tutor application that was developed to provide instantaneous feedback to first-graders learning to read. We saw that ASR has potential at this stage of the reading process, as the results suggested that pupils made progress in reading accuracy and fluency by using the software. In the current study, we used children's speech from an existing corpus (JAS-MIN) to develop two new ASR systems, and compared the results to those of the previous study. We analyze correct/incorrect classification of the ASR systems using human transcripts at word level, by means of evaluation measures such as Cohen's Kappa, Matthews Correlation Coefficient (MCC), precision, recall and F-measures. We observe improvements for the newly developed ASR systems regarding the agreement with human-based judgment and correct rejection (CR). The accuracy of the ASR systems varies for different reading tasks and word types. Our results suggest that, in the current configuration, it is difficult to classify isolated words. We discuss these results, possible ways to improve our systems and avenues for future research.

show abstract

Section: Performance Evaluation Of Different Systemsmentioning

confidence: 76%

Section: Asr Systemsmentioning

confidence: 99%

“…These probability measures are compared to a threshold to obtain classification scores. In a previous paper [3], we focused on optimizing this threshold.…”

Section: Procedures For Performance Evaluation Of Asr Systemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An ASR-Based Tutor for Learning to Read: How to Optimize Feedback to First Graders

Bai¹,

Tejedor-Garcı́a²,

Hubers³

et al. 2021

Speech and Computer

Self Cite

View full text Add to dashboard Cite

show abstract

“…Improvements in reading accuracy and speed were seen in different types of exercises [9]. After analyzing the data of this first experiment, we saw that a higher agreement between the system's scores and the judgements assigned by a human annotator could be obtained by adjusting the measures and thresholds in the system [10].…”

Section: Introductionmentioning

confidence: 88%

Automatic Speech Recognition Technology and Reading Skill Development in Primary School

Bai¹,

Tejedor-Garcı́a²,

Hubers³

et al. 2021

ICERI2021 Proceedings

Self Cite

View full text Add to dashboard Cite

Reading is a learned skill that children acquire through instruction and practice. A desirable feature of that practice is that children can read aloud under the guidance of a teacher. Unfortunately, this is not always possible to a sufficient extent because of general time-constraints in teacher-fronted education. For this reason, experts have been looking at Automatic Speech Recognition (ASR) technology as a possible alternative for the "listening ear" that is usually lent by teachers. Especially for reading in English, the contribution of ASR technology has been investigated from various perspectives and this research even led to relatively successful commercial products. In general, in these products the aim for ASR technology is to follow the children while reading aloud and provide some form of support when they hesitate.An important requirement for this kind of research and applications is that ASR of child speech is of sufficient quality. In turn this requires that the ASR algorithms are trained with large amounts of child speech recordings that, in general, are difficult to obtain.A compounding problem with child speech recognition is that children, as a group, display an enormous amount of variation in terms of speech characteristics, which is in part related to their variable physical characteristics. This means that large amounts of speech recordings are needed for each age cohort, which makes achieving high-quality levels of ASR performance even more difficult.As a matter of fact, the reason why so far research and applications have mostly addressed English is that for English larger amounts of child speech recordings could be obtained than for other languages. As a way of circumventing the data sparsity problem, speech technologists have investigated child speech recognition by applying several techniques that had been developed for low-resourced languages.In general, research has shown that the younger the children, the more difficult it is to achieve good ASR performance. On the other hand, one can imagine that ASR-based applications for learning to read are especially required for young children.In this paper we report on research aimed at developing and testing a Dutch ASR-based reading tutor. Innovative features of this system are that it is intended for young children in the earlier stages of learning to read, when they start developing decoding skills, and that it addresses reading in Dutch. A preliminary study conducted during the pandemic when children were primarily working from home showed encouraging results. In the present paper we report on a recent, larger study in which children used the system in the classroom. After collecting over 617 thousand words from 263 first graders from 44 different schools, we analyzed various attempts by the children at reading words and sentences before and after they received various types of ASR-based feedback. The results indicate that successive attempts were characterized by improved reading accuracy and fluency. Significant improvement in accuracy ...

show abstract

Can Artificial Intelligence Identify Reading Fluency and Level? Comparison of Human and Machine Performance

Yıldız,

Keskin,

Oyucu

et al. 2024

Reading & Writing Quarterly

View full text Add to dashboard Cite

An ASR-based Reading Tutor for Practicing Reading Skills in the First Grade: Improving Performance through Threshold Adjustment

Cited by 3 publications

References 9 publications

An ASR-Based Tutor for Learning to Read: How to Optimize Feedback to First Graders

An ASR-Based Tutor for Learning to Read: How to Optimize Feedback to First Graders

Automatic Speech Recognition Technology and Reading Skill Development in Primary School

Can Artificial Intelligence Identify Reading Fluency and Level? Comparison of Human and Machine Performance

Contact Info

Product

Resources

About