Natural Language Generation Using Transformer Network in an Open-Domain Setting

Varshney, Deeksha; Ekbal, Asif; Nagaraja, Ganesh Prasad; Tiwari, Mrigank; Gopinath, Abhijith Athreya Mysore; Bhattacharyya, Pushpak

doi:10.1007/978-3-030-51310-8_8

Cited by 8 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Gillioz et al, 2020). Not long after its introduction, the research community in Natural Language Processing began to explore its application to some sequential tasks such as text generation (Varshney et al, 2020;Mishra et al, 2021), machine translation and language understanding (Banar et al, 2021).…”

Section: Recent Advancements and Future Directionsmentioning

confidence: 99%

Deep Learning: Historical Overview from Inception to Actualization, Models, Applications and Future Trends

Ekundayo,

Ezugwu

2024

Preprint

View full text Add to dashboard Cite

Deep learning stands at the forefront of contemporary machine learning techniques and is well-known for its outstanding predictive accuracy, adaptability to data variability, and remarkable ability to generalize across diverse domains. These attributes have spurred rapid progress and the emergence of novel iterations within the discipline. Yet, this swift evolution often obscures the foundational breakthroughs, with even trailblazing researchers at risk of fading into obscurity despite their seminal contributions. This study aims to provide a historical narrative of deep learning, tracing its origins from the cybernetic era to its current state-of-the-art status. We critically examine the contributions of individual pioneer scholars who have profoundly influenced the development of deep neural networks under the taxonomy of supervised, unsupervised, and reinforcement learning. Furthermore, the study also discusses the trending deep neural network architectures, explaining their operational principles, confronting associated challenges, exploring real-world applications, and outlining potential future trajectories that could offer a starting point for aspiring researchers in the field.

show abstract

Section: Recent Advancements and Future Directionsmentioning

confidence: 99%

Deep Learning: Historical Overview from Inception to Actualization, Models, Applications and Future Trends

Ekundayo,

Ezugwu

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, the transformer architecture started to substitute RNNs for the language model training [44][45][46][47]. One of the major advantages of transformers is that they achieve independency while making predictions on different dialogue stages based on the selfattention mechanism.…”

Section: Dialogue Management Modulementioning

confidence: 99%

Development of Dialogue Management System for Banking Services

2021

View full text Add to dashboard Cite

Rapid increase in conversational AI and user chat data lead to intensive development of dialogue management systems (DMS) for various industries. Yet, for low-resource languages, such as Azerbaijani, very little research has been conducted. The main purpose of this work is to experiment with various DMS pipeline set-ups to decide on the most appropriate natural language understanding and dialogue manager settings. In our project, we designed and evaluated different DMS pipelines with respect to the conversational text data obtained from one of the leading retail banks in Azerbaijan. In the work, the main two components of DMS—Natural language Understanding (NLU) and Dialogue Manager—have been investigated. In the first step of NLU, we utilized a language identification (LI) component for language detection. We investigated both built-in LI methods such as fastText and custom machine learning (ML) models trained on the domain-based dataset. The second step of the work was a comparison of the classic ML classifiers (logistic regression, neural networks, and SVM) and Dual Intent and Entity Transformer (DIET) architecture for user intention detection. In these experiments we used different combinations of feature extractors such as CountVectorizer, Term Frequency-Inverse Document Frequency (TF-IDF) Vectorizer, and word embeddings for both word and character n-gram based tokens. To extract important information from the text messages, Named Entity Extraction (NER) component was added to the pipeline. The best NER model was chosen among conditional random fields (CRF) tagger, deep neural networks (DNN), models and build in entity extraction component inside DIET architecture. Obtained entity tags fed to the Dialogue Management module as features. All NLU set-ups were followed by the Dialogue Management module that contains a Rule-based Policy to handle FAQs and chitchats as well as a Transformer Embedding Dialogue (TED) Policy to handle more complex and unexpected dialogue inputs. As a result, we suggest a DMS pipeline for a financial assistant, which is capable of identifying intentions, named entities, and a language of text followed by policies that allow generating a proper response (based on the designed dialogues) and suggesting the best next action.

show abstract

“…They were dis covered in Vaswani et al (2017) and were able to beat several state-of-the-art approaches on different NLP tasks. They have been employed in machine language translation ( Sefara et al., 2021 ), conversational agents ( Golovanov et al., 2020 ), sentiment analysis ( Pipalia, Bhadja & Shukla, 2020 ), language generation ( Varshney et al., 2020 ), text summarization ( Luo, Guo & Guo, 2019 ) and so on bringing a huge advancement to the literature in different tasks and domains. Transformers have been already successfully applied to perform music genre classification and generation ( Qiu, Li & Sung, 2021 ; Huang et al., 2018 ).…”

Section: Introductionmentioning

confidence: 99%

A transformers-based approach for fine and coarse-grained classification and generation of MIDI songs and soundtracks

Angioni

Lincoln-DeCusatis

Ibba

et al. 2023

PeerJ Computer Science

View full text Add to dashboard Cite

Music is an extremely subjective art form whose commodification via the recording industry in the 20th century has led to an increasingly subdivided set of genre labels that attempt to organize musical styles into definite categories. Music psychology has been studying the processes through which music is perceived, created, responded to, and incorporated into everyday life, and, modern artificial intelligence technology can be exploited in such a direction. Music classification and generation are emerging fields that gained much attention recently, especially with the latest discoveries within deep learning technologies. Self attention networks have in fact brought huge benefits for several tasks of classification and generation in different domains where data of different types were used (text, images, videos, sounds). In this article, we want to analyze the effectiveness of Transformers for both classification and generation tasks and study the performances of classification at different granularity and of generation using different human and automatic metrics. The input data consist of MIDI sounds that we have considered from different datasets: sounds from 397 Nintendo Entertainment System video games, classical pieces, and rock songs from different composers and bands. We have performed classification tasks within each dataset to identify the types or composers of each sample (fine-grained) and classification at a higher level. In the latter, we combined the three datasets together with the goal of identifying for each sample just NES, rock, or classical (coarse-grained) pieces. The proposed transformers-based approach outperformed competitors based on deep learning and machine learning approaches. Finally, the generation task has been carried out on each dataset and the resulting samples have been evaluated using human and automatic metrics (the local alignment).

show abstract

Natural Language Generation Using Transformer Network in an Open-Domain Setting

Cited by 8 publications

References 23 publications

Deep Learning: Historical Overview from Inception to Actualization, Models, Applications and Future Trends

Deep Learning: Historical Overview from Inception to Actualization, Models, Applications and Future Trends

Development of Dialogue Management System for Banking Services

A transformers-based approach for fine and coarse-grained classification and generation of MIDI songs and soundtracks

Contact Info

Product

Resources

About