BertGCN: Transductive Text Classification by Combining GNN and BERT

Lin, Yuxiao; Meng, Yuxian; Sun, Xiaofei; Han, Qinghong; Kuang, Kun; Li, Jiwei; Wu, Fei

doi:10.18653/v1/2021.findings-acl.126

Cited by 100 publications

(46 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On this account, it is easy to see how graph architectures can also be integrated with deep language models. BertGCN [100], for example, trains a GCN jointly with a BERT-like model, in order to leverage the advantages of both pre-trained language models and graph-based approaches. Document nodes are initialised through BERT-style embeddings and updated iteratively by the GCN layers.…”

Section: Successful Approachesmentioning

confidence: 99%

A Survey on Text Classification Algorithms: From Text to Predictions

et al. 2022

View full text Add to dashboard Cite

In recent years, the exponential growth of digital documents has been met by rapid progress in text classification techniques. Newly proposed machine learning algorithms leverage the latest advancements in deep learning methods, allowing for the automatic extraction of expressive features. The swift development of these methods has led to a plethora of strategies to encode natural language into machine-interpretable data. The latest language modelling algorithms are used in conjunction with ad hoc preprocessing procedures, of which the description is often omitted in favour of a more detailed explanation of the classification step. This paper offers a concise review of recent text classification models, with emphasis on the flow of data, from raw text to output labels. We highlight the differences between earlier methods and more recent, deep learning-based methods in both their functioning and in how they transform input data. To give a better perspective on the text classification landscape, we provide an overview of datasets for the English language, as well as supplying instructions for the synthesis of two new multilabel datasets, which we found to be particularly scarce in this setting. Finally, we provide an outline of new experimental results and discuss the open research challenges posed by deep learning-based language models.

show abstract

Section: Successful Approachesmentioning

confidence: 99%

A Survey on Text Classification Algorithms: From Text to Predictions

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Lee et al [13] adopted a perplexity-based approach in the few-shot learning, which assumes that the given claim may be fake if the corresponding perplexity score from evidence-conditioned language models is high. BertGCN [14] is proposed by integrating the advantages of large-scale pre-trained models and graph neural networks for fake news detection, which is able to learn the representations from the massive amount of pre-trained data and the label influence through the propagation. MCAN [6] adopts a large-scale pre-trained NLP model and a pre-trained computer vision (CV) model for extracting features from text and images, respectively.…”

Section: Fake News Detectionmentioning

confidence: 99%

Team Yao at Factify 2022: Utilizing Pre-trained Models and Co-attention Networks for Multi-Modal Fact Verification

Wang¹,

Peng²

2022

Preprint

View full text Add to dashboard Cite

In recent years, social media has enabled users to get exposed to a myriad of misinformation and disinformation; thus, misinformation has attracted a great deal of attention in research fields and as a social issue. To address the problem, we propose a framework, Pre-CoFact, composed of two pre-trained models for extracting features from text and images, and multiple co-attention networks for fusing the same modality but different sources and different modalities. Besides, we adopt the ensemble method by using different pre-trained models in Pre-CoFact to achieve better performance. We further illustrate the effectiveness from the ablation study and examine different pre-trained models for comparison. Our team, Yao, won the fifth prize (F1-score: 74.585%) in the Factify challenge hosted by De-Factify @ AAAI 2022, which demonstrates that our model achieved competitive performance without using auxiliary tasks or extra information. The source code of our work is publicly available 1 .

show abstract

“…GNN have gained popularity due to their powerful expressive ability, and they are also used to solve the problem of text classification [31][32][33][34].…”

Section: Short Text Classification Based Gcnmentioning

confidence: 99%

Concept and Dependencies Enhanced Graph Convolutional Networks for Short Text Classification

Zhang

Ping

2022

Preprint

View full text Add to dashboard Cite

Short text classification task is a special kind of text classification task in that the text to be classified is generally short, typically generating a sparse text representation that lacks rich semantic information. Given this shortcoming, scholars worldwide have explored improved short text classification methods based on deep learning. However, existing methods cannot effectively use concept knowledge and long-distance word dependencies. Therefore, based on graph neural networks from the perspective of text composition, we propose concept and dependencies enhanced graph convolutional networks for short text classification. First, the co-occurrence relationship between words, the inclusion relationship between documents and words, long-distance word dependencies, and the association relationship between external concepts and words are defined. Then, a text graph is constructed for an entire text corpus based on the four relationships. Finally, the text graph is input into graph convolutional neural networks, and the category of each document node is predicted after two layers of convolution. Experimental results show that our proposed method overall best on multiple classical English text classification datasets.

show abstract

BertGCN: Transductive Text Classification by Combining GNN and BERT

Cited by 100 publications

References 46 publications

A Survey on Text Classification Algorithms: From Text to Predictions

A Survey on Text Classification Algorithms: From Text to Predictions

Team Yao at Factify 2022: Utilizing Pre-trained Models and Co-attention Networks for Multi-Modal Fact Verification

Concept and Dependencies Enhanced Graph Convolutional Networks for Short Text Classification

Contact Info

Product

Resources

About