PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision

Wu, Chuhan; Wu, Fangzhao; Qi, Tao; Lian, Jianxun; Huang, Yongfeng; Xie, Xing

doi:10.18653/v1/2020.findings-emnlp.174

Cited by 21 publications

(19 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Personalized news recommendation is an important research problem and has been widely studied in recent years (Konstan et al, 1997;Wang and Blei, 2011;Liu et al, 2010;Bansal et al, 2015;Wu et al, 2020b;Qi et al, 2021c;Wu et al, 2020dWu et al, , 2021dWang et al, 2020;Ge et al, 2020;.…”

Section: Personalized News Recommendationmentioning

confidence: 99%

Uni-FedRec: A Unified Privacy-Preserving News Recommendation Framework for Model Training and Online Serving

Qi¹,

Wu²,

Wu³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

Self Cite

View full text Add to dashboard Cite

News recommendation is important for personalized online news services. Most existing news recommendation methods rely on centrally stored user behavior data to both train models offline and provide online recommendation services. However, user data is usually highly privacy-sensitive, and centrally storing them may raise privacy concerns and risks. In this paper, we propose a unified news recommendation framework, which can utilize user data locally stored in user clients to train models and serve users in a privacy-preserving way. Following a widely used paradigm in real-world recommender systems, our framework contains two stages. The first one is for candidate news generation (i.e., recall) and the second one is for candidate news ranking (i.e., ranking). At the recall stage, each client locally learns multiple interest representations from clicked news to comprehensively model user interests. These representations are uploaded to the server to recall candidate news from a large news pool, which are further distributed to the user client at the ranking stage for personalized news display. In addition, we propose an interest decomposer-aggregator method with perturbation noise to better protect private user information encoded in user interest representations. Besides, we collaboratively train both recall and ranking models on the data decentralized in a large number of user clients in a privacy-preserving way. Experiments on two real-world news datasets show that our method can outperform baseline methods and effectively protect user privacy.

show abstract

Section: Personalized News Recommendationmentioning

confidence: 99%

Uni-FedRec: A Unified Privacy-Preserving News Recommendation Framework for Model Training and Online Serving

Qi¹,

Wu²,

Wu³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

Self Cite

View full text Add to dashboard Cite

show abstract

Section: User Model Pre-trainingmentioning

confidence: 99%

“…In recent years, there are a few works on pretraining user models using unlabeled user behavior data via self-supervision (Wu et al, 2020a;Yuan et al, 2020;Xie et al, 2020). For example, Wu et al (2020a) proposed a PTUM approach that uses a masked behavior prediction task and a next K behaviors prediction tasks to pre-train user models. Yuan et al (2020) proposed a PeterRec approach that pre-trains user models in a masked behavior prediction task and an auto-regressive task that successively predicts user behaviors based on past ones.…”

Section: User Model Pre-trainingmentioning

confidence: 99%

See 1 more Smart Citation

UserBERT: Contrastive User Model Pre-training

Wu¹,

Wu²,

Yu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

User modeling is critical for personalized web applications. Existing user modeling methods usually train user models from user behaviors with task-specific labeled data. However, labeled data in a target task may be insufficient for training accurate user models. Fortunately, there are usually rich unlabeled user behavior data which encode rich information of user characteristics and interests. Thus, pre-training user models on unlabeled user behavior data has the potential to improve user modeling for many downstream tasks. In this paper, we propose a contrastive user model pre-training method named UserBERT. Two self-supervision tasks are incorporated in User-BERT for user model pre-training on unlabeled user behavior data to empower user modeling. The first one is masked behavior prediction, which aims to model the relatedness between user behaviors. The second one is behavior sequence matching, which aims to capture the inherent user interests that are consistent in different periods. In addition, we propose a medium-hard negative sampling framework to select informative negative samples for better contrastive pre-training. We maintain a synchronously updated candidate behavior pool and an asynchronously updated candidate behavior sequence pool to select the locally hardest negative behaviors and behavior sequences in an efficient way. Extensive experiments on two real-world datasets in different tasks show that UserBERT can effectively improve various user models.

show abstract

“…User interest modeling is a critical step for personalized news recommendation (Wu et al, 2021;Zheng et al, 2018;Wu et al, 2020c). Existing methods usually learn a single representation vector to model overall user interests from users' clicked news (Okura et al, 2017;Wu et al, 2020b;.…”

Section: Introductionmentioning

confidence: 99%

HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation

Qi¹,

Wu²,

Wu³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

User interest modeling is critical for personalized news recommendation. Existing news recommendation methods usually learn a single user embedding for each user from their previous behaviors to represent their overall interest. However, user interest is usually diverse and multi-grained, which is difficult to be accurately modeled by a single user embedding. In this paper, we propose a news recommendation method with hierarchical user interest modeling, named HieRec. Instead of a single user embedding, in our method each user is represented in a hierarchical interest tree to better capture their diverse and multi-grained interest in news. We use a three-level hierarchy to represent 1) overall user interest; 2) user interest in coarse-grained topics like sports; and 3) user interest in fine-grained topics like football. Moreover, we propose a hierarchical user interest matching framework to match candidate news with different levels of user interest for more accurate user interest targeting. Extensive experiments on two real-world datasets validate our method can effectively improve the performance of user modeling for personalized news recommendation.

show abstract

PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision

Cited by 21 publications

References 15 publications

Uni-FedRec: A Unified Privacy-Preserving News Recommendation Framework for Model Training and Online Serving

Uni-FedRec: A Unified Privacy-Preserving News Recommendation Framework for Model Training and Online Serving

UserBERT: Contrastive User Model Pre-training

HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation

Contact Info

Product

Resources

About