Neural Networks approaches focused on French Spoken Language Understanding: application to the MEDIA Evaluation Task

Ghannay, Sahar; Servan, Christophe; Rosset, Sophie

doi:10.18653/v1/2020.coling-main.245

Cited by 4 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the GESE oral test method, a formative hierarchical evaluation model of oral English teaching in higher vocational colleges is constructed, and its positive "backwash effect" law is obtained. Many of the research tasks proposed in [1] focus on English, and it is difficult to explore a richer variety of languages within the framework of the SLU task. Minor languages need to be updated in the training process of the SLU model.…”

Section: Introductionmentioning

confidence: 99%

Construction of the Open Oral Evaluation Model Based on the Neural Network

Zhi-xin

Zhang

et al. 2021

Scientific Programming

View full text Add to dashboard Cite

According to the problem of low efficiency and low scoring accuracy of the traditional oral language scoring system, this study builds an open oral language evaluation model based on the basic principles of deep learning technology. Firstly, the basic methods of the convolutional neural network (CNN) and long short-term memory (LSTM) neural network are introduced. Then, we combine the convolutional neural network (CNN) and long short-term memory (LSTM) neural network to design an open oral scoring model based on CNN + LSTM, which divides the oral evaluation model into the speech scoring model and text scoring model and makes a specific implementation of two scoring models, respectively. An experimental environment is then built to preprocess the data, and finally, the model built in this study is trained and simulated. The experimental results show that the CNN + LSTM network evaluation model has a better comprehensive scoring performance, higher scoring efficiency, and higher accuracy and has feasibility and practicability.

show abstract

Section: Introductionmentioning

confidence: 99%

Construction of the Open Oral Evaluation Model Based on the Neural Network

Zhi-xin

Zhang

et al. 2021

Scientific Programming

View full text Add to dashboard Cite

show abstract

“…This system is based on the fine-tuning of the French CamemBERT [26] model, on the manual transcriptions of MEDIA corpus. It achieved state-of-the-art result on manual transcriptions of MEDIA corpus [19], yielding to 7.56 of CER when there is no error in the transcription.…”

Section: Cascade Approach With Pre-trained Modelsmentioning

confidence: 97%

“…For the NLU module, we propose to use the one that achieved the state-ofthe-art result on manual transcriptions of MEDIA corpus [19]. This system is based on a fine-tuning of BERT [16] on MEDIA SLU task using the French CamemBERT [26] model.…”

Section: Bert and Camembert Modelsmentioning

confidence: 99%

Where are we in semantic concept extraction for Spoken Language Understanding?

Ghannay,

Caubrière,

Mdhaffar

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Spoken language understanding (SLU) topic has seen a lot of progress these last three years, with the emergence of end-to-end neural approaches. Spoken language understanding refers to natural language processing tasks related to semantic extraction from speech signal, like named entity recognition from speech or slot filling task in a context of human-machine dialogue. Classically, SLU tasks were processed through a cascade approach that consists in applying, firstly, an automatic speech recognition process, followed by a natural language processing module applied to the automatic transcriptions. These three last years, end-toend neural approaches, based on deep neural networks, have been proposed in order to directly extract the semantics from speech signal, by using a single neural model. More recent works on self-supervised training with unlabeled data open new perspectives in term of performance for automatic speech recognition and natural language processing. In this paper, we present a brief overview of the recent advances on the French MEDIA benchmark dataset for SLU, with or without the use of additional data. We also present our last results that significantly outperform the current state-of-the-art with a Concept Error Rate (CER) of 11.2%, instead of 13.6% for the last state-of-the-art system presented this year.

show abstract

“…They highlighted the competitiveness of Word2Vec and ELMO on the French SLU corpus: MEDIA. More recently, [12] investigated the transferability of two French pre-trained BERT models [2] and their integration in BiLSTM and BiLSTM+CNN-based architectures. They obtained state-of-the-art results on MEDIA using CamemBERT [13] model.…”

Section: Introductionmentioning

confidence: 99%

Benchmarking Transformers-based models on French Spoken Language Understanding tasks

Cattan,

Ghannay,

Servan

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In the last five years, the rise of the self-attentional Transformerbased architectures led to state-of-the-art performances over many natural language tasks. Although these approaches are increasingly popular, they require large amounts of data and computational resources. There is still a substantial need for benchmarking methodologies ever upwards on under-resourced languages in data-scarce application conditions. Most pre-trained language models were massively studied using the English language and only a few of them were evaluated on French. In this paper, we propose a unified benchmark, focused on evaluating models quality and their ecological impact on two well-known French spoken language understanding tasks. Especially we benchmark thirteen well-established Transformer-based models on the two available spoken language understanding tasks for French: MEDIA and ATIS-FR. Within this framework, we show that compact models can reach comparable results to bigger ones while their ecological impact is considerably lower. However, this assumption is nuanced and depends on the considered compression method.

show abstract

Neural Networks approaches focused on French Spoken Language Understanding: application to the MEDIA Evaluation Task

Cited by 4 publications

References 15 publications

Construction of the Open Oral Evaluation Model Based on the Neural Network

Construction of the Open Oral Evaluation Model Based on the Neural Network

Where are we in semantic concept extraction for Spoken Language Understanding?

Benchmarking Transformers-based models on French Spoken Language Understanding tasks

Contact Info

Product

Resources

About