One Person, One Model, One World: Learning Continual User Representation without Forgetting

Yuan, Fajie; Zhang, Guoxiao; Karatzoglou, Alexandros; He, Xiangnan; Jose, Joemon M.; Kong, Beibei; Li, Yudong

doi:10.48550/arxiv.2009.13724

Cited by 4 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To begin with, we first train an DL-based sequential recommendation model on a large source dataset and use it as our pre-trained model. This practice has been well verified by recent studies in [11], [12] (as well as in the NLP domains [13], [14], [15]) since sequential neural network can be trained in a selfsupervised manner and thus could generate more universal user representations for effective transfer learning. In this paper, we choose the temporal CNN model NextItNet [2] as the pre-trained backbone model given its efficient network structure [16], [17], superior performance [2], [18], [19] and widespread usage in modeling sequential recommendation data [11], [12], [19], [20], [21], [22] in recent literature.…”

Section: Introductionmentioning

confidence: 60%

“…They also presented several methodologies to accelerate the training and inference processes of NextItNet. More recently, PeterRec [11] and Conure [12] demonstrated that the learned user representations by NextItNet were generic and could be transferred to solve various downstream recommendation problems. Meanwhile, self-attention based models, such as SASRec [4] and BERT4Rec [25], also showed competitive performance for the SRS task.…”

Section: Sequential Recommender Systems (Srs)mentioning

confidence: 99%

“…However, compared with NextItNet with linear complexity, these models often require quadratic time complexity to compute the selfattention matrix. Thereby, in this paper, we plan to use the well-known temporal CNN model (i.e., NextItNet) as the backbone network, given that the performance of which has been well evaluated by PeterRec [11] and Conure [12] for the CDR tasks [28].…”

Section: Sequential Recommender Systems (Srs)mentioning

confidence: 99%

“…By contrast, TLCDR does not need to train the learner from scratch for target task. Besides, compared with JLCDR, TLCDR can obtain better prediction accuracy [12]. This is because the joint learning scheme of JLCDR that trades off many objective functions usually does not guarantee the optimal performance for the target objective function [11].…”

Section: Cross-domain Recommendations (Cdr)mentioning

confidence: 99%

“…• MTL: We present a widely used multi-task learning (referred to as MTL) baseline by hard parameter sharing [46], similar to that used in [12]. MTL jointly learns the objective functions of both source and target domains rather than using the two-stage pretraining and fine-tuning framework.…”

Section: Modelsmentioning

confidence: 99%

See 4 more Smart Citations

User-specific Adaptive Fine-tuning for Cross-domain Recommendations

Chen¹,

Yuan²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Making accurate recommendations for cold-start users has been a longstanding and critical challenge for recommender systems (RS). Cross-domain recommendations (CDR) offer a solution to tackle such a cold-start problem when there is no sufficient data for the users who have rarely used the system. An effective approach in CDR is to leverage the knowledge (e.g., user representations) learned from a related but different domain and transfer it to the target domain. Fine-tuning works as an effective transfer learning technique for this objective, which adapts the parameters of a pre-trained model from the source domain to the target domain. However, current methods are mainly based on the global fine-tuning strategy: the decision of which layers of the pre-trained model to freeze or fine-tune is taken for all users in the target domain. In this paper, we argue that users in RS are personalized and should have their own fine-tuning policies for better preference transfer learning. As such, we propose a novel User-specific Adaptive Fine-tuning method (UAF), selecting which layers of the pre-trained network to fine-tune, on a per user basis. Specifically, we devise a policy network with three alternative strategies to automatically decide which layers to be fine-tuned and which layers to have their parameters frozen for each user. Extensive experiments show that the proposed UAF exhibits significantly better and more robust performance for user cold-start recommendation.

show abstract

Section: Introductionmentioning

confidence: 60%

Section: Sequential Recommender Systems (Srs)mentioning

confidence: 99%

Section: Sequential Recommender Systems (Srs)mentioning

confidence: 99%

Section: Cross-domain Recommendations (Cdr)mentioning

confidence: 99%

Section: Modelsmentioning

confidence: 99%

See 3 more Smart Citations

User-specific Adaptive Fine-tuning for Cross-domain Recommendations

Chen¹,

Yuan²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

A User-Adaptive Layer Selection Framework for Very Deep Sequential Recommender Models

Chen

Yuan

Yang

et al. 2021

AAAI

View full text Add to dashboard Cite

Sequential recommender systems (SRS) have become a research hotspot in recent studies. Because of the requirement in capturing user's dynamic interests, sequential neural network based recommender models often need to be stacked with more hidden layers (e.g., up to 100 layers) compared with standard collaborative filtering methods. However, the high network latency has become the main obstacle when deploying very deep recommender models into a production environment. In this paper, we argue that the typical prediction framework that treats all users equally during the inference phase is inefficient in running time, as well as sub-optimal in accuracy. To resolve such an issue, we present SkipRec, an adaptive inference framework by learning to skip inactive hidden layers on a per-user basis. Specifically, we devise a policy network to automatically determine which layers should be retained and which layers are allowed to be skipped, so as to achieve user-specific decisions. To derive the optimal skipping policy, we propose using gumbel softmax and reinforcement learning to solve the non-differentiable problem during backpropagation. We perform extensive experiments on three real-world recommendation datasets, and demonstrate that SkipRec attains comparable or better accuracy with much less inference time.

show abstract

Online Continual Learning in Image Classification: An Empirical Survey

Mai

Jeong

et al. 2021

Preprint

View full text Add to dashboard Cite

Online continual learning for image classification studies the problem of learning to classify images from an online stream of data and tasks, where tasks may include different classes (class incremental) or data nonstationarity (domain incremental). One of the key challenges of continual learning is to avoid catastrophic forgetting (CF), i.e., forgetting old tasks in the presence of more recent tasks. Over the past few years, a large range of methods and tricks have been introduced to address the continual learning problem, but many have not been fairly and systematically compared under a variety of realistic and practical settings.To better understand the relative advantages of various approaches and the settings where they work best, this survey aims to (1) compare stateof-the-art methods such as Maximally Interfered Retrieval (MIR), iCARL, and GDumb (a very strong baseline) and determine which works best at different memory and data settings as well as better understand the key source of CF; (2) determine if the best online class incremental methods are also competitive in domain incremental setting; and (3) evaluate the performance of 7 simple but effective trick such as "review" trick and nearest class mean (NCM) classifier to assess their relative impact. Regarding (1), we

show abstract

One Person, One Model, One World: Learning Continual User Representation without Forgetting

Cited by 4 publications

References 36 publications

User-specific Adaptive Fine-tuning for Cross-domain Recommendations

User-specific Adaptive Fine-tuning for Cross-domain Recommendations

A User-Adaptive Layer Selection Framework for Very Deep Sequential Recommender Models

Online Continual Learning in Image Classification: An Empirical Survey

Contact Info

Product

Resources

About