Single-shot Embedding Dimension Search in Recommender System

Qu, Liang; Ye, Yong-Hong; Tang, Ningzhi; Zhang, Lixin; Shi, Yuhui; Yin, Hongzhi

doi:10.1145/3477495.3532060

Cited by 11 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model Compression [10,30,32,43,49,61] aims to obtain lightweight models by removing redundant weights. It becomes an effective solution to reduce model size in the fields of computer vision [8,35], graph learning [6,12,13,[36][37][38][39], and natural language processing [3][4][5].…”

Section: Related Workmentioning

confidence: 99%

Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation

Wang,

Sui,

et al. 2024

Proceedings of the 17th ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

In the realm of deep learning-based recommendation systems, the increasing computational demands, driven by the growing number of users and items, pose a significant challenge to practical deployment. This challenge is primarily twofold: reducing the model size while effectively learning user and item representations for efficient recommendations. Despite considerable advancements in model compression and architecture search, prevalent approaches face notable constraints. These include substantial additional computational costs from pre-training/re-training in model compression and an extensive search space in architecture design. Additionally, managing complexity and adhering to memory constraints is problematic, especially in scenarios with strict time or space limitations. Addressing these issues, this paper introduces a novel learning paradigm, Dynamic Sparse Learning (DSL), tailored for recommendation models. DSL innovatively trains a lightweight sparse model from scratch, periodically evaluating and dynamically adjusting each weight's significance and the model's sparsity distribution during the training. This approach ensures a consistent and minimal parameter budget throughout the full learning lifecycle, paving the way for "end-to-end" efficiency from training to inference. Our extensive experimental results underline DSL's effectiveness, significantly reducing training and inference costs while delivering comparable recommendation performance. We give an code link of our work: https://github.com/shuyao-wang/DSL.

show abstract

Section: Related Workmentioning

confidence: 99%

Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation

Wang,

Sui,

et al. 2024

Proceedings of the 17th ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

show abstract

“…Following the previous works [17,25], we evaluate the performance of our method using two common metrics: AUC and Logloss. AUC refers to the area under the ROC curve, which means the probability that a model will rank a randomly selected positive instance higher than a randomly selected negative one.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…The embedding tables play a fundamental role in the recommendation system, as they dominate the majority of parameters. However, most existing methods construct their proposed recommender models with large-sized embedding tables and a uniform dimension size for all possible fields [8,10,29], which may lead to overfitting, high computational cost, and poor model generalization [15,25,33]. Therefore, the first objective for an optimal DLRM is to find optimal embedding dimensions for different fields and remove redundant dimensions.…”

Section: Introductionmentioning

confidence: 99%

iHAS: Instance-wise Hierarchical Architecture Search for Deep Learning Recommendation Models

Yu,

Qi,

Yang

et al. 2023

Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Current recommender systems employ large-sized embedding tables with uniform dimensions for all features, leading to overfitting, high computational cost, and suboptimal generalizing performance. Many techniques aim to solve this issue by feature selection or embedding dimension search. However, these techniques typically select a fixed subset of features or embedding dimensions for all instances and feed all instances into one recommender model without considering heterogeneity between items or users. This paper proposes a novel instance-wise Hierarchical Architecture Search framework, iHAS, which automates neural architecture search at the instance level. Specifically, iHAS incorporates three stages: searching, clustering, and retraining. The searching stage identifies optimal instance-wise embedding dimensions across different field features via carefully designed Bernoulli gates with stochastic selection and regularizers. After obtaining these dimensions, the clustering stage divides samples into distinct groups via a deterministic selection approach of Bernoulli gates. The retraining stage then constructs different recommender models, each one designed with optimal dimensions for the corresponding group. We conduct extensive experiments to evaluate the proposed iHAS on two public benchmark datasets from a real-world recommender system. The experimental results demonstrate the effectiveness of iHAS and its outstanding transferability to widely-used deep recommendation models. CCS CONCEPTS• Information systems → Recommender systems; Online advertising.

show abstract

“…Therefore, in order to make feature representations of different dimensions that can be used seamlessly in these base models, we use a generalization framework that contains a dimension alignment layer, as shown in Figure 9. This method is also used in [41,69]. Specifically, for each field, we initialize an alignment matrix.…”

Section: A the Generalization Framework Of Fdo Approachmentioning

confidence: 99%

Towards Deeper, Lighter and Interpretable Cross Network for CTR Prediction

Wang,

Gu,

et al. 2023

Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Click Through Rate (CTR) prediction plays an essential role in recommender systems and online advertising. It is crucial to effectively model feature interactions to improve the prediction performance of CTR models. However, existing methods face three significant challenges. First, while most methods can automatically capture high-order feature interactions, their performance tends to diminish as the order of feature interactions increases. Second, existing methods lack the ability to provide convincing interpretations of the prediction results, especially for high-order feature interactions, which limits the trustworthiness of their predictions. Third, many methods suffer from the presence of redundant parameters, particularly in the embedding layer. This paper proposes a novel method called Gated Deep Cross Network (GDCN) and a Field-level Dimension Optimization (FDO) approach to address these challenges. As the core structure of GDCN, Gated Cross Network (GCN) captures explicit high-order feature interactions and dynamically filters important interactions with an information gate in each order. Additionally, we use the FDO approach to learn condensed dimensions for each field based on their importance. Comprehensive experiments on five datasets demonstrate the effectiveness, superiority and interpretability of GDCN. Moreover, we verify the effectiveness of FDO in learning various dimensions and reducing model parameters. The code is available on https://github.com/anonctr/GDCN.

show abstract

Single-shot Embedding Dimension Search in Recommender System

Cited by 11 publications

References 22 publications

Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation

Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation

iHAS: Instance-wise Hierarchical Architecture Search for Deep Learning Recommendation Models

Towards Deeper, Lighter and Interpretable Cross Network for CTR Prediction

Contact Info

Product

Resources

About