Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation

Bonetta, Giovanni; Cancelliere, R.; Liu, Ding; Vozila, Paul

doi:10.32473/flairs.v34i1.128369

Cited by 6 publications

(2 citation statements)

References 6 publications

(1 reference statement)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, RETRO Borgeaud et al (2021) enhances the model architecture not by increasing the number of parameters or the size of training data, but rather through the retrieval of information relevant for each sample. Similarly, Bonetta et al (2021) uses memorized similarity information from the training data for retrieval at at inference time.…”

Section: Retrieval Mechanismsmentioning

confidence: 99%

MQRetNN: Multi-Horizon Time Series Forecasting with Retrieval Augmentation

Yang¹,

Eisenach²,

Madeka³

2022

Preprint

View full text Add to dashboard Cite

Multi-horizon probabilistic time series forecasting has wide applicability to real-world tasks such as demand forecasting. Recent work in neural time-series forecasting mainly focus on the use of Seq2Seq architectures Sutskever et al. (2014). For example, MQTransformer (Eisenach et al., 2020) -an improvement of MQCNN (Wen et al., 2017) -has shown the state-of-the-art performance in probabilistic demand forecasting. In this paper, we consider incorporating crossentity information to enhance model performance by adding a cross-entity attention mechanism along with a retrieval mechanism to select which entities to attend over. We demonstrate how our new neural architecture, MQRetNN, leverages the encoded contexts from a pretrained baseline model on the entire population to improve forecasting accuracy. Using MQCNN as the baseline model (due to computational constraints, we do not use MQTransformer), we first show on a small demand forecasting dataset that it is possible to achieve ∼3% improvement in test loss by adding a cross-entity attention mechanism where each entity attends to all others in the population. We then evaluate the model with our proposed retrieval methods -as a means of approximating an attention over a large population -on a large-scale demand forecasting application with over 2 million products and observe ∼1% performance gain over the MQCNN baseline.

show abstract

Section: Retrieval Mechanismsmentioning

confidence: 99%

MQRetNN: Multi-Horizon Time Series Forecasting with Retrieval Augmentation

Yang¹,

Eisenach²,

Madeka³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The idea in such systems is to retrieve suitable responses from the dataset of recorded dialogs and adapt them to the context of the current dialog if needed. Retrieval-based approaches have been commonly applied in the domains of question-answering (Q&A) systems and general purpose conversational systems [17,18,19,20]. However, these approaches are not very common in the research literature for building CRS, where nowadays language generation models are predominant.…”

Section: Introductionmentioning

confidence: 99%

Towards Retrieval-based Conversational Recommendation

Manzoor,

Jannach

2021

Preprint

View full text Add to dashboard Cite

Conversational recommender systems have attracted immense attention recently. The most recent approaches rely on neural models trained on recorded dialogs between humans, implementing an end-to-end learning process. These systems are commonly designed to generate responses given the user's utterances in natural language. One main challenge is that these generated responses both have to be appropriate for the given dialog context and must be grammatically and semantically correct. An alternative to such generation-based approaches is to retrieve responses from pre-recorded dialog data and to adapt them if needed. Such retrieval-based approaches were successfully explored in the context of general conversational systems, but have received limited attention in recent years for CRS. In this work, we re-assess the potential of such approaches and design and evaluate a novel technique for response retrieval and ranking. A user study (N=90) revealed that the responses by our system were on average of higher quality than those of two recent generation-based systems. We furthermore found that the quality ranking of the two generation-based approaches is not aligned with the results from the literature, which points to open methodological questions. Overall, our research underlines that retrieval-based approaches should be considered an alternative or complement to language generation approaches. 1

show abstract

Transformer technology in molecular science

Jiang,

Ke,

Chen

et al. 2024

WIREs Comput Mol Sci

View full text Add to dashboard Cite

A transformer is the foundational architecture behind large language models designed to handle sequential data by using mechanisms of self‐attention to weigh the importance of different elements, enabling efficient processing and understanding of complex patterns. Recently, transformer‐based models have become some of the most popular and powerful deep learning (DL) algorithms in molecular science, owing to their distinctive architectural characteristics and proficiency in handling intricate data. These models leverage the capacity of transformer architectures to capture complex hierarchical dependencies within sequential data. As the applications of transformers in molecular science are very widespread, in this review, we only focus on the technical aspects of transformer technology in molecule domain. Specifically, we will provide an in‐depth investigation into the algorithms of transformer‐based machine learning techniques in molecular science. The models under consideration include generative pre‐trained transformer (GPT), bidirectional and auto‐regressive transformers (BART), bidirectional encoder representations from transformers (BERT), graph transformer, transformer‐XL, text‐to‐text transfer transformer, vision transformers (ViT), detection transformer (DETR), conformer, contrastive language‐image pre‐training (CLIP), sparse transformers, and mobile and efficient transformers. By examining the inner workings of these models, we aim to elucidate how their architectural innovations contribute to their effectiveness in processing complex molecular data. We will also discuss promising trends in transformer models within the context of molecular science, emphasizing their technical capabilities and potential for interdisciplinary research. This review seeks to provide a comprehensive understanding of the transformer‐based machine learning techniques that are driving advancements in molecular science.This article is categorized under: Data Science > Chemoinformatics Data Science > Artificial Intelligence/Machine Learning

show abstract

Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation

Cited by 6 publications

References 6 publications

MQRetNN: Multi-Horizon Time Series Forecasting with Retrieval Augmentation

MQRetNN: Multi-Horizon Time Series Forecasting with Retrieval Augmentation

Towards Retrieval-based Conversational Recommendation

Transformer technology in molecular science

Contact Info

Product

Resources

About