Making the Relation Matters: Relation of Relation Learning Network for Sentence Semantic Matching

Abstract

Sentence semantic matching is one of the fundamental tasks in natural language processing, which requires an agent to determine the semantic relation among input sentences. Recently, deep neural networks have achieved impressive performance in this area, especially BERT. Despite the effectiveness of these models, most of them treat output labels as meaningless one-hot vectors, underestimating the semantic information and guidance of relations that these labels reveal, especially for tasks with a small number of labels. To address this problem, we propose a Relation of Relation Learning Network (R2-Net) for sentence semantic matching. Specifically, we first employ BERT to encode the input sentences from a global perspective. Then a CNN-based encoder is designed to capture keywords and phrase information from a local perspective. To fully leverage labels for better relation information extraction, we introduce a self-supervised relation of relation classification task for guiding R2-Net to consider more about labels. Meanwhile, a triplet loss is employed to distinguish the intra-class and inter-class relations in a finer granularity. Empirical experiments on two sentence semantic matching tasks demonstrate the superiority of our proposed model. As a byproduct, we have released the codes to facilitate other researches.

GBT: Generative Boosting Training Approach for Paraphrase Identification

“…The generation length in Chinese is Models QQP ACC. F1 CENN 80.7 \ L.D.C (Wang et al, 2016) 85.6 \ BiMPM (Wang et al, 2017) 88.2 \ DIIN (Gong et al, 2017) 89.1 \ DRCN (Kim et al, 2019) 90.2 \ R 2 -Net (Zhang et al, 2021) 91.6 \ ISG (RoBERTa-large) (Xu et al, 2022) shorter and the difficulty is lower, which makes the generation effect better. Therefore, the boosting effect for Chinese PI is better than that of English.…”

“…Inspired by ResNet (He et al, 2016a), Kim et al (2019) propose a densely-connected recurrent and co-attentive Network (DRCN), which combines residual operation with RNN and attention mechanism. Zhang et al (2021) propose a Relation Learning Network (R 2 -Net) based on BERT, which is characterized by interactive relationship perception of multi-granularity linguistic units. The recently proposed ISG-BERT (Xu et al, 2022) integrates syntactic alignments and semantic matching signals of two sentences into an association graph to gain a fine granularity matching process.…”

“…A variety of neural PI approaches have been proposed Wang et al, 2017;Gong et al, 2017;Lai et al, 2019;Kim et al, 2019). Recently, the large PLMbased Models are leveraged as crucial supportive encoders for neural PI (Devlin et al, 2019;Liu et al, 2019;Cui et al, 2021;Zhang et al, 2021; Sentences Label Edit Distance…”

Enhancing sentence semantic matching with isotropic batch normalization and generalized pooling operator

“…𝐿 ∑ 𝑦 true log 𝑦 pred 1 𝑦 true log 1 𝑦 pred (4) ytrue represents the distribution of true labels, while ypred represents the predicted label distribution of the trained model. Crossentropy loss can measure the similarity between y true and y pred .…”

“…Inspired by DenseNet [2], Kim et al proposed a densely connected co-attention recursive neural network that effectively preserves both original and co-attention features across layers [3]. Zhang et al discussed the importance of sentence semantic matching in natural language processing and highlighted the limitations of existing models like BERT, which treat output labels as single-point vectors [4]. To address this issue, the authors introduced the Relation of Relation Learning Network (R 2 Net).…”