BertGCN: Transductive Text Classification by Combining GCN and BERT

Lin, Yuxiao; Meng, Yuxian; Sun, Xiaofei; Han, Qinghong; Kuang, Kun; Li, Jiwei; Wu, Fei

doi:10.48550/arxiv.2105.05727

Cited by 24 publications

(21 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We first compare the proposed method with several state-ofthe-art methods on the text classification task, including HAN [28], TextGCN [30], XLNet [6], BertGCN [37]. Micro-F1 and Weighted-F1 are adopted as evaluation metrics.…”

Section: Event Text Classification Resultsmentioning

confidence: 99%

Joint Learning With BERT-GCN and Multi-Attention for Event Text Classification and Event Assignment

She¹,

Chen²,

Chen³

2022

IEEE Access

View full text Add to dashboard Cite

Government hotline is closely related to people's lives and plays an important role in solving social problems and maintaining social stability in China. However, the event text of the hotline is inconsistent in length and unclear in elements, so it is a challenge for the operator to manually complete the assignment tasks of hotline event. To address these problems, we propose a joint learning method for event text classification and event assignment for Chinese government hotline. Firstly, graph convolution network (GCN) and BERT are used to process the event text respectively to obtain the corresponding representation vector. Then, the obtained two representation vectors are fused by the dynamic fusion gate to get fusion vector and classified the fusion vector through the text classification. Secondly, we use multi-attention mechanism to process the GCN result vector, BERT result vector and the "sanding" vector to obtain attentive "eventsanding" representation vector and calculate the corresponding department probability distribution. Finally, the historical prior knowledge based reorder model is used to sort the results of the "event-sanding" matching module and output the optimal assignment department of government hotline event. Experimental results show that our method can achieve better performance compared with several baseline approaches. The ablation experiments also demonstrate the validity of each proposed module in our model.

show abstract

Section: Event Text Classification Resultsmentioning

confidence: 99%

Joint Learning With BERT-GCN and Multi-Attention for Event Text Classification and Event Assignment

She¹,

Chen²,

Chen³

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…trieve a set of relevant information to improve the model performance under the merit that an openbook exam is easier than a close-book exam. Recent success on various NLP tasks has shown the effectiveness of retrieval augmented models in improving the quality of neural NLP models, such as language modeling (Khandelwal et al, 2019;, question answering (Guu et al, 2020;Lewis et al, 2020a,b;Xiong et al, 2020), text classification (Lin et al, 2021b), dialog generation Thulke et al, 2021;Weston et al, 2018) and neural machine translation (Khandelwal et al, 2019;Meng et al, 2021a;Wang et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

$k$NN-NER: Named Entity Recognition with Nearest Neighbor Search

Wang¹,

Li²,

Meng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Inspired by recent advances in retrieval augmented methods in NLP (Khandelwal et al., 2019(Khandelwal et al., , 2020 , in this paper, we introduce a k nearest neighbor NER (kNN-NER) framework, which augments the distribution of entity labels by assigning k nearest neighbors retrieved from the training set. This strategy makes the model more capable of handling long-tail cases, along with better few-shot learning abilities. kNN-NER requires no additional operation during the training phase, and by interpolating k nearest neighbors search into the vanilla NER model, kNN-NER consistently outperforms its vanilla counterparts: we achieve a new state-of-the-art F1score of 72.03 (+1.25) on the Chinese Weibo dataset and improved results on a variety of widely used NER benchmarks. Additionally, we show that kNN-NER can achieve comparable results to the vanilla NER model with 40% less amount of training data. 1

show abstract

“…Graph attention network (GAT) [43] leverages masked self-attentional layers to address the shortcomings of prior work based on graph convolutions or their approximations. GNN has demonstrated effectiveness in a wide variety of tasks such as text classification [24], question answering [6], recommendation [44] and relation extraction [23]. For example, Guo et al [9] presents Star-Transformer which reduces the computation complexity of the standard Transformer by replacing the fully-connected structure in self-attention with a star-like topology.…”

Section: Graph Neural Networkmentioning

confidence: 99%

GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval

Liu¹,

Yang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, retrieval models based on dense representations are dominant in passage retrieval tasks, due to their outstanding ability in terms of capturing semantics of input text compared to the traditional sparse vector space models. A common practice of dense retrieval models is to exploit a dual-encoder architecture to represent a query and a passage independently. Though efficient, such a structure loses interaction between the query-passage pair, resulting in inferior accuracy. To enhance the performance of dense retrieval models without loss of efficiency, we propose a GNN-encoder model in which query (passage) information is fused into passage (query) representations via graph neural networks that are constructed by queries and their top retrieved passages. By this means, we maintain a dual-encoder structure, and retain some interaction information between query-passage pairs in their representations, which enables us to achieve both efficiency and efficacy in passage retrieval. Evaluation results indicate that our method significantly outperforms the existing models on MSMARCO, Natural Questions and TriviaQA datasets, and achieves the new state-of-the-art on these datasets.

show abstract

BertGCN: Transductive Text Classification by Combining GCN and BERT

Cited by 24 publications

References 25 publications

Joint Learning With BERT-GCN and Multi-Attention for Event Text Classification and Event Assignment

Joint Learning With BERT-GCN and Multi-Attention for Event Text Classification and Event Assignment

$k$NN-NER: Named Entity Recognition with Nearest Neighbor Search

GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval

Contact Info

Product

Resources

About