Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation

Zhang, Shengyu; Yang, Ling; Yao, Dong; Lu, Yujie; Feng, Fuli; Zhao, Zhou; Chua, Tat-Seng; Wu, Fei

doi:10.1145/3485447.3512094

Cited by 25 publications

(20 citation statements)

References 66 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Re4 [50] further takes the backward flow into account to regularize the process. Despite improved architectures and enriched information, these works generally follow the training of the general training scheme of candidate matching, where we identify several problems in our work.…”

Section: Multi-interest Learningmentioning

confidence: 99%

“…In prior works [2][3][4]50], 𝑄 is often generic sampling distributions such as log-uniform and uniform sampling, which are shown to perform relatively well in general RS learning [44]. Specifically, using log-uniform sampling on the item set sorted by popularity gives the popular items a higher probability of being selected as negative samples.…”

Section: Sampled Softmax Lossmentioning

confidence: 99%

“…A key module in multi-interest learning is items-to-interests routing, which is typically achieved with multihead attention or capsule networks [2,24]. Multi-head attentionbased multi-interest routing demonstrates higher efficiency and comparable performance compared with capsule networks [2,4], and therefore they are widely applied as the base structure in recent work [2][3][4]50]. However, we observe that after training for multiple epochs, the interests tend to over-focus on single items in the behavior sequence, as shown in Figure 1 (b).…”

Section: Routing Regularization For Routing Collapsementioning

confidence: 99%

See 2 more Smart Citations

Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems

Xie¹,

Gao²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

Existing research efforts for multi-interest candidate matching in recommender systems mainly focus on improving model architecture or incorporating additional information, neglecting the importance of training schemes. This work revisits the training framework and uncovers two major problems hindering the expressiveness of learned multi-interest representations. First, the current training objective (i.e., uniformly sampled softmax) fails to effectively train discriminative representations in a multi-interest learning scenario due to the severe increase in easy negative samples. Second, a routing collapse problem is observed where each learned interest may collapse to express information only from a single item, resulting in information loss. To address these issues, we propose the REMI framework, consisting of an Interest-aware Hard Negative mining strategy (IHN) and a Routing Regularization (RR) method. IHN emphasizes interest-aware hard negatives by proposing an ideal sampling distribution and developing a Monte-Carlo strategy for efficient approximation. RR prevents routing collapse by introducing a novel regularization term on the item-to-interest routing matrices. These two components enhance the learned multi-interest representations from both the optimization objective and the composition information. REMI is a general framework that can be readily applied to various existing multi-interest candidate matching methods. Experiments on three real-world datasets show our method can significantly improve state-of-the-art methods with easy implementation and negligible computational overhead. The source code will be released. CCS CONCEPTS• Information systems → Recommender systems.

show abstract

Section: Multi-interest Learningmentioning

confidence: 99%

Section: Sampled Softmax Lossmentioning

confidence: 99%

Section: Routing Regularization For Routing Collapsementioning

confidence: 99%

See 1 more Smart Citation

Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems

Xie¹,

Gao²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Some popular methods like Wide&Deep [4] and DeepFM [9] explore the combination among features to construct the high-order information, which reduces the burden of the subsequent classifier. Several approaches like DeepFM [39], DIN [50] and Bert4Rec [32] resort to deep learning to hierarchically extract the semantics about user interests for recommendation [19,45]. More recent works like MIMN [21], SIM [22] and Linformer [37] solve the scalability and efficiency issues in comprehensively understanding user intention from lifelong sequences.…”

Section: Related Work 21 Cloud-based Recommendationmentioning

confidence: 99%

Device-cloud Collaborative Recommendation via Meta Controller

Yao

Wang

Ding³

et al. 2022

Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

On-device machine learning enables the lightweight deployment of recommendation models in local clients, which reduces the burden of the cloud-based recommenders and simultaneously incorporates more real-time user features. Nevertheless, the cloud-based recommendation in the industry is still very important considering its powerful model capacity and the efficient candidate generation from the billion-scale item pool. Previous attempts to integrate the merits of both paradigms mainly resort to a sequential mechanism, which builds the on-device recommender on top of the cloud-based recommendation. However, such a design is inflexible when user interests dramatically change: the on-device model is stuck by the limited item cache while the cloud-based recommendation based on the large item pool do not respond without the new re-fresh feedback. To overcome this issue, we propose a meta controller to dynamically manage the collaboration between the on-device recommender and the cloud-based recommender, and introduce a novel efficient sample construction from the causal perspective to solve the dataset absence issue of meta controller. On the basis of the counterfactual samples and the extended training, extensive experiments in the industrial recommendation scenarios show the promise of meta controller in the device-cloud collaboration. CCS CONCEPTS• Information systems → Recommender systems.

show abstract

“…Modern Recommender Systems (RS) [7,19,36,40,41] often expose a list of items to the users on the edges (e.g., mobile phones) where the items are sorted by the relevance to users' interests. Waterfall RS, a popular form of RS, is a stream of recommended items consisting of successive pages that can be browsed by scrolling, as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

Intelligent Request Strategy Design in Recommender System

Qian¹,

Xu²,

Lv³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Waterfall Recommender System (RS), a popular form of RS in mobile applications, is a stream of recommended items consisting of successive pages that can be browsed by scrolling. In waterfall RS, when a user finishes browsing a page, the edge (e.g., mobile phones) would send a request to the cloud server to get a new page of recommendations, known as the paging request mechanism. RSs typically put a large number of items into one page to reduce excessive resource consumption from numerous paging requests, which, however, would diminish the RSs' ability to timely renew the recommendations according to users' real-time interest and lead to a poor user experience. Intuitively, inserting additional requests inside pages to update the recommendations with a higher frequency can alleviate the problem. However, previous attempts, including only non-adaptive strategies (e.g., insert requests uniformly), would eventually lead to resource overconsumption. To this end, we envision a new learning task of edge intelligence named Intelligent Request Strategy Design (IRSD). It aims to improve the effectiveness of waterfall RSs by determining the appropriate occasions of request insertion based on users' real-time intention. Moreover, we propose a new paradigm of adaptive request insertion strategy named Uplift-based On-edge Smart Request Framework (AdaRequest). AdaRequest 1) captures the dynamic change of users' intentions by matching their real-time behaviors with their historical interests based on attention-based neural networks. 2) estimates the counterfactual uplift of user purchase brought by an inserted request based on causal inference. 3) determines the final request insertion strategy by maximizing the utility function under online resource constraints. We conduct extensive experiments on both offline dataset and online A/B test to verify the effectiveness of AdaRequest. Remarkably, AdaRequest has been deployed on the Waterfall RS of Taobao and brought over 3% lift on Gross Merchandise Value (GMV).

show abstract

Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation

Cited by 25 publications

References 66 publications

Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems

Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems

Device-cloud Collaborative Recommendation via Meta Controller

Intelligent Request Strategy Design in Recommender System

Contact Info

Product

Resources

About