BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese

Tran, Nguyen Luong; Le, Duong Minh; Nguyen, Dat Quoc

doi:10.21437/interspeech.2022-10177

Cited by 19 publications

(13 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the authors of PhoBERT -a Vietnamese pre-trained language model, instead of BERT for better Vietnamese presentation extraction (Nguyen and Nguyen 2020), have demonstrated that using models dedicated to fixed language pairs yields superior results compared to using a single multilingual model. They have introduced the BARTpho model (Tran, Le, and Nguyen 2022) along with pre-trained weights trained on Vietnamese data.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Revitalizing Bahnaric Language through Neural Machine Translation: Challenges, Strategies, and Promising Outcomes

Vo,

Le,

Phan

et al. 2024

AAAI

View full text Add to dashboard Cite

The Bahnar, a minority ethnic group in Vietnam with ancient roots, hold a language of deep cultural and historical significance. The government is prioritizing the preservation and dissemination of Bahnar language through online availability and cross-generational communication. Recent AI advances, including Neural Machine Translation (NMT), have transformed translation with improved accuracy and fluency, fostering language revitalization through learning, communication, and documentation. In particular, NMT enhances accessibility for Bahnar language speakers, making information and content more available. However, translating Vietnamese to Bahnar language faces practical hurdles due to resource limitations, particularly in the case of Bahnar language as an extremely low-resource language. These challenges encompass data scarcity, vocabulary constraints, and a lack of fine-tuning data. To address these, we propose transfer learning from selected pre-trained models to optimize translation quality and computational efficiency, capitalizing on linguistic similarities between Vietnamese and Bahnar language. Concurrently, we apply tailored augmentation strategies to adapt machine translation for the Vietnamese-Bahnar language context. Our approach is validated through superior results on bilingual Vietnamese-Bahnar language datasets when compared to baseline models. By tackling translation challenges, we help revitalize Bahnar language, ensuring information flows freely and the language thrives.

show abstract

Section: Related Workmentioning

confidence: 99%

“…BARTpho (Tran, Le, and Nguyen 2022) is a pretrained sequence-to-sequence model for Vietnamese based on the BART architecture (Lewis et al 2020) that is trained on a large corpus of Vietnamese text. BART is a sequenceto-sequence model that can generate text from text, such as summarization, translation, or text generation.…”

Section: Bahnaric-fine-tuned Bn-bartphomentioning

confidence: 99%

Revitalizing Bahnaric Language through Neural Machine Translation: Challenges, Strategies, and Promising Outcomes

Vo,

Le,

Phan

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…BARTPhoBEiT [44]. BARTPhoBEiT is a our previous novel integration of the BARTPho [47] and BEiT-3 [42] models, specifically tailored for the Vietnamese language. This innovative model incorporates pre-trained Sequence-to-Sequence and bidirectional encoder representations derived from Image Transformers.…”

Section: Transformer Approachmentioning

confidence: 99%

ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamese

Tran,

Phan,

Nguyen

et al. 2023

Preprint

View full text Add to dashboard Cite

In recent years, Visual Question Answering (VQA) has gained significant attention for its diverse applications, including intelligent car assistance, aiding visually impaired individuals, and document image information retrieval using natural language queries. VQA requires effective integration of information from questions and images to generate accurate answers. Neural models for VQA have made remarkable progress on large-scale datasets, with a primary focus on resource-rich languages like English.To address this, we introduce the ViCLEVR dataset, a pioneering collection for evaluating various visual reasoning capabilities in Vietnamese while mitigating biases. The dataset comprises over 26,000 images and 30,000 question-answer pairs (QAs), each question annotated to specify the type of reasoning involved. Leveraging this dataset, we conduct a comprehensive analysis of contemporary visual reasoning systems, offering valuable insights into their strengths and limitations.Furthermore, we present PhoVIT, a comprehensive multimodal fusion that identifies objects in images based on questions. The architecture effectively employs transformers to enable simultaneous reasoning over textual and visual data, merging both modalities at an early model stage. The experimental findings demonstrate that our proposed model achieves state-of-the-art performance across seven evaluation metrics.

show abstract

“…This strongly limits the portability to multilingual scenarios. To overcome the above-mentioned limitation, prior works have attempted to train transformer-based models trained in languages other than English such as French [23], Vietnamese [24], or Chinese [25]. Limited research efforts have been devoted to the Italian language, i.e., [9,[26][27][28].…”

Section: Italian Language Modelingmentioning

confidence: 99%

BART-IT: An Efficient Sequence-to-Sequence Model for Italian Text Summarization

Quatra

Cagliero

2022

Future Internet

View full text Add to dashboard Cite

The emergence of attention-based architectures has led to significant improvements in the performance of neural sequence-to-sequence models for text summarization. Although these models have proved to be effective in summarizing English-written documents, their portability to other languages is limited thus leaving plenty of room for improvement. In this paper, we present BART-IT, a sequence-to-sequence model, based on the BART architecture that is specifically tailored to the Italian language. The model is pre-trained on a large corpus of Italian-written pieces of text to learn language-specific features and then fine-tuned on several benchmark datasets established for abstractive summarization. The experimental results show that BART-IT outperforms other state-of-the-art models in terms of ROUGE scores in spite of a significantly smaller number of parameters. The use of BART-IT can foster the development of interesting NLP applications for the Italian language. Beyond releasing the model to the research community to foster further research and applications, we also discuss the ethical implications behind the use of abstractive summarization models.

show abstract

BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese

Cited by 19 publications

References 5 publications

Revitalizing Bahnaric Language through Neural Machine Translation: Challenges, Strategies, and Promising Outcomes

Revitalizing Bahnaric Language through Neural Machine Translation: Challenges, Strategies, and Promising Outcomes

ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamese

BART-IT: An Efficient Sequence-to-Sequence Model for Italian Text Summarization

Contact Info

Product

Resources

About