Learning Fast Matching Models from Weak Annotations

Abstract

We propose a novel training scheme for fast matching models in Search Ads, motivated by practical challenges. The first challenge stems from the pursuit of high throughput, which prohibits the deployment of inseparable architectures, and hence greatly limits model accuracy. The second problem arises from the heavy dependency on human provided labels, which are expensive and time-consuming to collect, yet how to leverage unlabeled search log data is rarely studied. The proposed training framework targets on mitigating both issues, by treating the stronger but undeployable models as annotators, and learning a deployable model from both human provided relevance labels and weakly annotated search log data. Specifically, we first construct multiple auxiliary tasks from the enumerated relevance labels, and train the annotators by jointly learning from those related tasks. The annotation models are then used to assign scores to both labeled and unlabeled training samples. The deployable model is firstly learnt on the scored unlabeled data, and then fine-tuned on scored labeled data, by leveraging both labels and scores via minimizing the proposed label-aware weighted loss. According to our experiments, compared with the baseline that directly learns from relevance labels, training by the proposed framework outperforms it by a large margin, and improves data efficiency substantially by dispensing with 80% labeled samples. The proposed framework allows us to improve the fast matching model by learning from stronger annotators while keeping its architecture unchanged. Meanwhile, it offers a principled manner to leverage search log data in the training phase, which could effectively alleviate our dependency on human provided labels.

AdsGNN: Behavior-Graph Augmented Relevance Modeling in Sponsored Search

“…Parameters in baselines are carefully tuned on the validation set to select the most desirable parameter setting. Considering the high imbalance distribution of the annotations, following the previous work [20] we select ROC-AUC score as the measurement, which represents the area under the Receiver Operating Characteristic curve. We release our code to facilitate future research (https:// github.com/ qwe35/ AdsGNN ).…”

“…This strategy confuses the relevance correlations with the click relations and thus may introduce ambiguities from two aspects. Firstly, the arbitrariness and subjectivity of user behavior lead to the misalignment between user clicks and true relevance annotations [20], which may introduce noises into the ground truth and further pollute the training set. Secondly, negative pairs sampled by data synthesizing usually share no common tokens for queries and ads, which may mislead the relevance model to view common terms as critical evidence of relevance.…”

Diversity driven Query Rewriting in Search Advertising

“…Some other works [3,12] learn embeddings of queries and ads in a shared vector space from search session, ad click, and search link click data using word2vec [28] like algorithms. A few recent works [23] exploit advances in neural information retrieval models [29] such as Deep Crossing [41] in sponsored search. An important line of work [11,13,15,27,37] that is based on the idea of performing query to query transformations, also known as query rewriting.…”