End-to-End User Behavior Retrieval in Click-Through RatePrediction Model

Abstract

Click-Through Rate (CTR) prediction is one of the core tasks in recommender systems (RS). It predicts a personalized click probability for each user-item pair. Recently, researchers have found that the performance of CTR model can be improved greatly by taking user behavior sequence into consideration, especially long-term user behavior sequence. The report on an e-commerce website shows that 23% of users have more than 1000 clicks during the past 5 months. Though there are numerous works focus on modeling sequential user behaviors, few works can handle long-term user behavior sequence due to the strict inference time constraint in real world system. Two-stage methods are proposed to push the limit for better performance. At the first stage, an auxiliary task is designed to retrieve the top-𝑘 similar items from long-term user behavior sequence. At the second stage, the classical attention mechanism is conducted between the candidate item and 𝑘 items selected in the first stage. However, information gap happens between retrieval stage and the main CTR task. This goal divergence can greatly diminishing the performance gain of long-term user sequence. In this paper, inspired by Reformer, we propose a localitysensitive hashing (LSH) method called ETA (End-to-end Target Attention) which can greatly reduce the training and inference cost and make the end-to-end training with long-term user behavior sequence possible. Both offline and online experiments confirm the effectiveness of our model. We deploy ETA into a large-scale real world E-commerce system and achieve extra 3.1% improvements on GMV (Gross Merchandise Value) compared to a two-stage long user sequence CTR model.

Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction

“…UBR4CTR [14] uses a similar search-based method as SIM [13] to face challenges. Recently, ETA [3] proposes an end-to-end target attention method to model long-term user behavior sequence. They apply locality-sensitive hashing (LSH) to reduce the training and inference time cost.…”

“…Recently, some methods are proposed to model users' long-term interests from long behavior sequences [3,[12][13][14]. Among them, MIMN [12] decouples the user interest modeling from the entire model by designing a separate User Interest Center (UIC) module.…”

“…Among them, MIMN [12] decouples the user interest modeling from the entire model by designing a separate User Interest Center (UIC) module. While saving lots of online serving time, MIMN can't utilize the information from the target item, which has been proved to be crucial for modeling users' interests [3]. Therefore, MIMN can only model shallow user interests.…”

“…They retrieves top-𝑘 similar items, and feed top-𝑘 items to the subsequent target attention module [23]. As pointed by [3], the retrieve objectives of these approaches are divergent with the goal of the CTR model, and the pre-trained embedding in offline inverted index is not suiting for online learning systems. ETA [3] proposes an LSH-based method to retrieve top-𝑘 similar items from longterm user behaviors in an end-to-end fashion.…”

“…As pointed by [3], the retrieve objectives of these approaches are divergent with the goal of the CTR model, and the pre-trained embedding in offline inverted index is not suiting for online learning systems. ETA [3] proposes an LSH-based method to retrieve top-𝑘 similar items from longterm user behaviors in an end-to-end fashion. They use localitysensitive hash (LSH) to convert items into hash signatures, and then retrieve top-𝑘 similar items based on their hamming distance to the candidate item.…”