Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction

Abstract

Rich user behavior data has been proven to be of great value for Click-Through Rate (CTR) prediction applications, especially in industrial recommender, search, or advertising systems. However, it's non-trivial for real-world systems to make full use of long-term user behaviors due to the strict requirements of online serving time. Most previous works adopt the retrieval-based strategy, where a small number of user behaviors are retrieved first for subsequent attention. However, the retrieval-based methods are sub-optimal and would cause more or less information losses, and it's difficult to balance the effectiveness and efficiency of the retrieval algorithm.In this paper, we propose SDIM (Sampling-based Deep Interest Modeling), a simple yet effective sampling-based end-to-end approach for modeling long-term user behaviors. We sample from multiple hash functions to generate hash signatures of the candidate item and each item in the user behavior sequence, and obtain the user interest by directly gathering behavior items associated with the candidate item with the same hash signature. We show theoretically and experimentally that the proposed method performs on par with standard attention-based models on modeling long-term user behaviors, while being sizable times faster. We also introduce the deployment of SDIM in our system. Specifically, we decouple the behavior sequence hashing, which is the most time-consuming part, from the CTR model by designing a separate module named BSE (behavior Sequence Encoding). BSE is latency-free for the CTR server, enabling us to model extremely long user behaviors. Both offline and online experiments are conducted to demonstrate the effectiveness of SDIM. SDIM now has been deployed online in the search system of Meituan APP.

TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou

“…More recently, ETA [3] uses locality-sensitive hash (LSH) to encode item embeddings from ESU and retrieve similar items from long-term behaviors via Hamming distance (HD) based on those hash signatures. SDIM [1] samples behavior items with the same hash signature as the target item through multiround hash collision, whose ESU then linearly aggregates those sampled behavior items to obtain user interests. It is positive that ETA and SDIM adopt End2End training, so that the embeddings of the two stages are the same.…”

“…Then GSU uses Hamming distance as the target-behavior relevance metric. • SDIM [1]. GSU selects behaviors with the same hash signature as the target video through multi-round hash collision.…”

“…SIM Cluster (+), and SIM Soft use refined retrieval methods, but the compromise is that they have to use pre-trained embeddings and generate offline inverted index in advance. Note that SIM Hard and SIM Soft are simultaneously proposed [19], but SIM Soft has not yet been defeated by the follow-up work ETA and SDIM [1,3]. We do not compare with UBR4CTR [20] because its iterative training is not suitable for our streaming scenario.…”

“…ETA approximates the relevance score calculation using locality-sensitive hashing (LSH) and Hamming distance [3]. SDIM samples behaviors with the same hash signature as the targetbehavior through multi-round hash collision [1], among others. Despite being extensively studied, existing two-stage life-long behavior modeling algorithms still suffer from a crucial limitation: the inconsistency between GSU and ESU (shown in Fig 1 1 ).…”

“…As a result, GSU may probably miss relevant behaviors, but retrieve ones considered irrelevant by ESU, wasting ESU's precious computational resources. In such case, the TA in ESU, no matter how attention is allocated, 1 The Oracle is identically copied from the ESU of SIM-hard (trained from the top-100 behaviors retrieved by GSU), except that it ranks 10 4 rather than 10 2 behaviors. Experiments are conducted on a small demo dataset from Kuaishou, not feasible for normal datasets due to the extremely high cost of Oracle (details in Section 4.5).…”