Benchmarking Transformers-based models on French Spoken Language Understanding tasks

Abstract

In the last five years, the rise of the self-attentional Transformerbased architectures led to state-of-the-art performances over many natural language tasks. Although these approaches are increasingly popular, they require large amounts of data and computational resources. There is still a substantial need for benchmarking methodologies ever upwards on under-resourced languages in data-scarce application conditions. Most pre-trained language models were massively studied using the English language and only a few of them were evaluated on French. In this paper, we propose a unified benchmark, focused on evaluating models quality and their ecological impact on two well-known French spoken language understanding tasks. Especially we benchmark thirteen well-established Transformer-based models on the two available spoken language understanding tasks for French: MEDIA and ATIS-FR. Within this framework, we show that compact models can reach comparable results to bigger ones while their ecological impact is considerably lower. However, this assumption is nuanced and depends on the considered compression method.

Data-Efficient French Language Modeling with CamemBERTa

“…Pre-training Compute and CO2 Impact Our model was trained for 8 days on 6 A40 GPUs, compared to CamemBERT which was trained on 256 V100 GPUs for one day, which is roughly equivalent to 28 days of training on 6 A40 GPUs, since an NVIDIA A40 GPU is about 1.5x faster than a V100 GPU on language modeling tasks according to recent benchmarks. 8 Following the reports by Luccioni et al (2022) and Cattan et al (2022) on the environmental impact of language model training, we use Lannelongue et al's (2021) online carbon footprint calculator to provide the following estimates: CAMEM-BERTA's pre-training used 700kWh and emitted 36kg CO 2 compared to 3.32MWh and 170kg for CamemBERT. 9…”