DGTR: Dynamic graph transformer for rumor detection

Abstract

Social media rumors have the capacity to harm the public perception and the social progress. The news propagation pattern is a key clue for detecting rumors. Existing propagation-based rumor detection methods represent propagation patterns as a static graph structure. They simply consider the structure information of news distribution in social networks and disregard the temporal information. The dynamic graph is an effective modeling tool for both the structural and temporal information involved in the process of news dissemination. Existing dynamic graph representation learning approaches struggle to capture the long-range dependence of the structure and temporal sequence as well as the rich semantic association between full graph features and individual parts. We build a transformer-based dynamic graph representation learning approach for rumor identification DGTR to address the aforementioned challenges. We design a position embedding format for the graph data such that the original transformer model can be utilized for learning dynamic graph representations. The model can describe the structural long-range reliance between the dynamic graph nodes and the temporal long-range dependence between the temporal snapshots by employing a self-attention mechanism. In addition, the CLS token in transformer may model the rich semantic relationships between the complete graph and each subpart. Extensive experiments demonstrate the superiority of our model when compared to the state of the art.

Dynamic Graph Representation Learning With Neural Networks: A Survey

“…As a model originally designed for sequence to sequence problems, Transformers architectures [23] are increasingly being employed in static graph representation learning and extended to dynamic graphs. We have already mentioned above how models use Transformer as a temporal [101], [108], [121], [123] or spatial encoder [101], [123], [136], as well as implementing the positional encoding technique, all of which show its great potential for learning dynamic graph representations.…”

“…The third combination is a parallel encoding of the DG by independent transformer and GNN layers, followed by a combination of their encoded hidden states [120], merging the strengths of both layers. Additionally, some dynamic graph models [73], [98], [99], [122], [123], [125], [127] use transformers exclusively as graph encoders, exploiting the self-attention mechanism for node hidden states propagation without relying on traditional GNN architectures.…”

“…The second method utilizes (global) graph spectral information [99], such as the Laplacian or its eigenvectors. The third method relies on spatial domain information or local structures, such as embedding the node degrees or random walks [73], [99], [101], [122], [123], [127], [139]. Finally, as mentioned in section III-E, almost all the dynamic graph transformers use time embedding to represent the timestamp or snapshot index with a vector.…”

“…Concerning Self-Attention (SA), there are four solutions on dynamic graphs, as shown in Fig. 12 (right): Spatial SA: when a token represents a node in a snapshot, SA operates at the spatial level, equivalent to the graph encoder f G , with SA matrix dimensions being the square of the number of nodes N t in the snapshot t [101], [120]- [123], [125], [136].…”

“…Temporal SA: When a token represents the same node (or the entire snapshot) across multiple snapshots, SA operates at the temporal level and acts as a temporal encoder f T , its SA matrix dimensions thus are the square of the number of time steps K [101], [120], [123], [125], [136]. Spatial-Temporal SA: Following Temporal Edge Modeling III-B, creating replicas for each node in K snapshots, forming Spatial-Temporal Attention, with the Attention matrix size being the square of N t or K × T [98]- [101], [139].…”