Using Massive Multilingual Pre-Trained Language Models Towards Real Zero-Shot Neural Machine Translation in Clinical Domain

Abstract

Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating super powers and the preknowledge they acquire for downstream tasks. In this work, we investigate whether MM-PLMs can be applied to zero-shot machine translation (MT) towards entirely new language pairs and new domains. We carry out experimental investigation using Meta-AI's MM-PLMs "wmt21-dense-24-wide-en-X and X-en (WMT21fb)" which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and opposite direction. We fine-tune these MM-PLMs towards English-Spanish language pair which did not exist at all in their original pre-trained corpora both implicitly and explicitly. We prepare carefully aligned clinical domain data for this fine-tuning, which is different from their original mixed domain knowledge as well. Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain EN-ES pairs / sentences for three sub-task translation testings: clinical cases, clinical terms and ontology concepts. It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a highresource setting in the pre-training. To the best of our knowledge, this is the first work on using MMPLMs towards real zero-shot NMT successfully for totally unseen languages during pre-training, and also the first in clinical domain for such study.

Effective Natural Language Processing Algorithms for Gout Flare Early Alert from Chief Complaints

“…All three discriminative LLMs use a bidirectional encoder as BERT [27]. The encoder part of these models was used to encode each CC text, and the “[CLS]” token was used as the dense representation.…”

“…These technologies have the potential to revolutionize the healthcare industry by enhancing medical decision-making, patient care, and biomedical research. Some tasks in NLP could be automated using LLM such as text classification [8, 9], keyword Extraction [10, 11], machine translation [12], and text summarization [13]. Furthermore, NLP and LLM can assist in the early detection and diagnosis of diseases by sifting through vast datasets to identify patterns, symptoms, and risk factors.…”

“…Some tasks in NLP could be automated using LLM such as text classification [4, 5, 6], keyword Extraction [7, 8, 9], machine translation [10], and text summarization [11]. Furthermore, NLP and LLM can assist in the early detection and diagnosis of diseases by sifting through vast datasets to identify patterns, symptoms, and risk factors.…”