Recognizing Semantic Relations: Attention-Based Transformers vs. Recurrent Models

Abstract

Automatically recognizing an existing semantic relation (such as "is a", "part of", "property of", "opposite of" etc.) between two arbitrary words (phrases, concepts, etc.) is an important task affecting many information retrieval and artificial intelligence tasks including query expansion, common-sense reasoning, question answering, and database federation. Currently, two classes of approaches exist to classify a relation between words (concepts) X and Y: (1) path-based and (2) distributional. While the path-based approaches look at word-paths connecting X and Y in text, the distributional approaches look at statistical properties of X and Y separately, not necessary in the proximity of each other. Here, we suggest how both types can be improved and empirically compare them using several standard benchmarking datasets. For our distributional approach, we are suggesting using an attentionbased transformer. While they are known to be capable of supporting knowledge transfer between different tasks, and recently set a number of benchmarking records in various applications, we are the first to successfully apply them to the task of recognizing semantic relations. To improve a path-based approach, we are suggesting our original neural word path model that combines useful properties of convolutional and recurrent networks, and thus addressing several shortcomings from the prior path-based models. Both our models significantly outperforms the state-of-the-art within its type accordingly. Our transformer-based approach outperforms current state-of-the-art by 1-12% points on 4 out of 6 standard benchmarking datasets. This results in 15-40% error reduction and is closing the gap between the automated and human performance by up to 50%. It also needs much less training data than prior approaches. For the ease of re-producing our results, we make our source code and trained models publicly available.During the last few years, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have resulted in major breakthroughs and are behind

Sentence Level Human Translation Quality Estimation with Attention-based Neural Networks

“…There has been some research on different ways for integration of LSTMs and CNNs, since the two methods for building the neural networks are somewhat complementary. Roussinov et al (2020) studied the use of LSTMs (or pre-trained transformers) with convolution filters to predict the lexical relations for pairs of concepts, for example, Tom Cruise is an actor or a rat has a tail. Most similar to our work is the study by (Zhou et al, 2016), which also used a stacked architecture with LSTM followed by twodimensional pooling to obtain a fixed-length representation for text classification tasks.…”