Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer

Fan, Ziwei; Liu, Zhiwei; Zhang, Jiawei; Xiong, Yun; Zheng, Lei; Yu, Philip S.

doi:10.1145/3459637.3482242

Cited by 115 publications

(47 citation statements)

References 47 publications

(67 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Early applications of transformers in graph domain focus on the sequential recommendation because the positional embeddings are easy to be defined based on the user interaction sequence and timestamps, such as SASRec [22], BERT4Rec [42], SSE-PT [47], TGSRec [15], etc. Some other works have tried to adopt transformers into static graph tasks and have achieved competitive performance compared with state-of-the-art GNN-based models.…”

Section: Transformersmentioning

confidence: 99%

Graph Masked Autoencoders with Transformers

Zhang¹,

Chen²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Transformers have achieved state-of-the-art performance in learning graph representations. However, there are still some challenges when applying transformers to real-world scenarios due to the fact that deep transformers are hard to be trained from scratch and the memory consumption is large. To address the two challenges, we propose Graph Masked Autoencoders (GMAE), a self-supervised model for learning graph representations, where vanilla graph transformers are used as the encoder and the decoder. GMAE takes partially masked graphs as input, and reconstructs the features of the masked nodes. We adopt asymmetric encoder-decoder design, where the encoder is a deep graph transformer and the decoder is a shallow graph transformer. The masking mechanism and the asymmetric design make GMAE a memory-efficient model compared with conventional transformers. We show that, compared with training from scratch, the graph transformer pre-trained using GMAE can achieve much better performance after fine-tuning. We also show that, when serving as a conventional self-supervised graph representation model and using an SVM model as the downstream graph classifier, GMAE achieves state-of-the-art performance on 5 of the 7 benchmark datasets.

show abstract

Section: Transformersmentioning

confidence: 99%

Graph Masked Autoencoders with Transformers

Zhang¹,

Chen²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Along with recent developments in the sequential recommendation, Time-LSTM [Zhu et al, 2017], MTAM [Ji et al, 2020], TiSASRec [Li, Wang, and McAuley, 2020], and TGSRec [Fan et al, 2021] incorporate time intervals between successive interactions into LSTM, Memory-Network, and SA; SLi-Rec [Yu et al, 2019b] and TASER [Ye et al, 2020] consider both time intervals and the time of the prediction. Each design has its own modifications that refine a few specific components to be time-aware.…”

Section: Temporal-aware Recommendationmentioning

confidence: 99%

Leaping Through Time with Gradient-based Adaptation for Recommendation

Chairatanakul¹,

Nt²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Modern recommender systems are required to adapt to the change in user preferences and item popularity. Such a problem is known as the temporal dynamics problem, and it is one of the main challenges in recommender system modeling. Different from the popular recurrent modeling approach, we propose a new solution named LeapRec to the temporal dynamic problem by using trajectory-based meta-learning to model time dependencies. LeapRec characterizes temporal dynamics by two complement components named global time leap (GTL) and ordered time leap (OTL). By design, GTL learns long-term patterns by finding the shortest learning path across unordered temporal data. Cooperatively, OTL learns short-term patterns by considering the sequential nature of the temporal data. Our experimental results show that LeapRec consistently outperforms the state-of-the-art methods on several datasets and recommendation metrics. Furthermore, we provide an empirical study of the interaction between GTL and OTL, showing the effects of long-and short-term modeling.

show abstract

“…Recommender systems [6,9,25,26,44] become crucial components in web applications [23], which provide personalized item lists by modeling interactions between users and items. Sequential recommendation (SR) attracts a lot of attention from both the academic community and industry due to its success and scalability.…”

Section: Introductionmentioning

confidence: 99%

Sequential Recommendation via Stochastic Self-Attention

Fan¹,

Liu²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Sequential recommendation models the dynamics of a user's previous behaviors in order to forecast the next item, and has drawn a lot of attention. Transformer-based approaches, which embed items as vectors and use dot-product self-attention to measure the relationship between items, demonstrate superior capabilities among existing sequential methods. However, users' real-world sequential behaviors are uncertain rather than deterministic, posing a significant challenge to present techniques. We further suggest that dot-product-based approaches cannot fully capture collaborative transitivity, which can be derived in item-item transitions inside sequences and is beneficial for cold start items. We further argue that BPR loss has no constraint on positive and sampled negative items, which misleads the optimization.We propose a novel STOchastic Self-Attention (STOSA) to overcome these issues. STOSA, in particular, embeds each item as a stochastic Gaussian distribution, the covariance of which encodes the uncertainty. We devise a novel Wasserstein Self-Attention module to characterize item-item position-wise relationships in sequences, which effectively incorporates uncertainty into model training. Wasserstein attentions also enlighten the collaborative transitivity learning as it satisfies triangle inequality. Moreover, we introduce a novel regularization term to the ranking loss, which assures the dissimilarity between positive and the negative items. Extensive experiments on five real-world benchmark datasets demonstrate the superiority of the proposed model over state-of-the-art baselines, especially on cold start items. The code is available in https://github.com/zfan20/STOSA.

show abstract

Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer

Cited by 115 publications

References 47 publications

Graph Masked Autoencoders with Transformers

Graph Masked Autoencoders with Transformers

Leaping Through Time with Gradient-based Adaptation for Recommendation

Sequential Recommendation via Stochastic Self-Attention

Contact Info

Product

Resources

About