Exploring lottery ticket hypothesis in media recommender systems

Abstract

Media recommender systems aim to capture users’ preferences and provide precise personalized recommendation of media content. There are two critical components in the common paradigm of modern recommender models: (1) representation learning, which generates an embedding for each user and item; and (2) interaction modeling, which fits user preferences toward items based on their representations. In spite of great success, when a great amount of users and items exist, it usually needs to create, store, and optimize a huge embedding table, where the scale of model parameters easily reach millions or even larger. Hence, it naturally raises questions about the heavy recommender models: Do we really need such large‐scale parameters? We get inspirations from the recently proposed lottery ticket hypothesis (LTH), which argues that the dense and over‐parameterized model contains a much smaller and sparser sub‐model that can reach comparable performance to the full model. In this paper, we extend LTH to media recommender systems, aiming to find the winning tickets in deep recommender models. To the best of our knowledge, this is the first work to study LTH in media recommender systems. With Matrix Factorization and Light Graph Convolution Networks as the backbone models, we found that there widely exist winning tickets in recommender models. On three media convergence data sets—Yelp2018, TikTok and Kwai, the winning tickets can achieve comparable recommendation performance with only 29 % ~ 48 % , 7 % ~ 10 %, and 3 % ~ 17 % of parameters, respectively.

Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation

“…However, since the amount of information carried by diverse users and items is very likely to be different, the strategy of assigning the same embedding size may be suboptimal. Observations from prior studies such as ESAPN [26], UnKD [2], and LTH-MRS [46], suggest that inactive users and unpopular items may contain less information, necessitating smaller embedding sizes to depict their simpler characteristics. Hence, we relax the constraint on embedding size and encourage the model to dynamically discover those redundant weights during learning.…”

“…• MP-based methods: PEP [27], LTH-MRS [46], Random Pruning (RP) [62], One-shot Magnitude Pruning (OMP) [16], Without Rewinding (WR) [34].…”

“…It becomes an effective solution to reduce model size in the fields of computer vision [8,35], graph learning [6,12,13,[36][37][38][39], and natural language processing [3][4][5]. Recently, the lottery ticket hypothesis [14,46] becomes an effective strategy to guide the model pruning. It states that dense randomly initialized networks contain sparse subnetworks (aka., winning tickets) that can be trained in isolation to achieve comparable performance.…”

“…In addition, the complex procedures of optimization and performance estimation [1] also lead to an unaffordable computational cost. • Model Pruning (MP) [14,16,40,46,62] trims down the model size by removing redundant parameters from the full model. It aims to find a sparse and lightweight model that can best retain the performance of the full model.…”

“…It aims to find a sparse and lightweight model that can best retain the performance of the full model. However, it needs to empirically find intriguing sparse embedding tables by an iterative "train-prune-retrain" pipeline [14,46], which still suffers from the expensiveness of the post-training pruning.…”