Pre-trained Language Model based Ranking in Baidu Search

Abstract

As the heart of a search engine, the ranking system plays a crucial role in satisfying users' information demands. More recently, neural rankers fine-tuned from pre-trained language models (PLMs) establish state-of-the-art ranking effectiveness. However, it is nontrivial to directly apply these PLM-based rankers to the large-scale web search system due to the following challenging issues: (1) the prohibitively expensive computations of massive neural PLMs, especially for long texts in the web-document, prohibit their deployments in an online ranking system that demands extremely low latency; (2) the discrepancy between existing ranking-agnostic pre-training objectives and the ad-hoc retrieval scenarios that demand comprehensive relevance modeling is another main barrier for improving the online ranking system; (3) a real-world search engine typically involves a committee of ranking components, and thus the compatibility of the individually fine-tuned ranking model is critical for a cooperative ranking system.In this work, we contribute a series of successfully applied techniques in tackling these exposed issues when deploying the stateof-the-art Chinese pre-trained language model, i.e., ERNIE, in the online search engine system. We first articulate a novel practice to cost-efficiently summarize the web document and contextualize the resultant summary content with the query using a cheap yet powerful Pyramid-ERNIE architecture. Then we endow an innovative paradigm to finely exploit the large-scale noisy and biased post-click behavioral data for relevance-oriented pre-training. We also propose a human-anchored fine-tuning strategy tailored for the online ranking system, aiming to stabilize the ranking signals across various online components. Extensive offline and online experimental results show that the proposed techniques significantly boost the search engine's performance.

Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

“…Note that a passage p consists of 𝑇 sentences p = {s 𝜏 } 𝑇 𝜏=1 . Following a previous study [52], a desirable re-ranker is a scoring function 𝑓 * (•, •) that maximizes the consistency between its predictions (denoted as Ŷq,P = {𝑓 (q, p 𝜅 ) | p 𝜅 ∈ P}) and the ground truth labels (denoted as 𝑌 = {𝑦 𝜅 } 𝜘 𝜅=1 ), i.e.,…”

“…To this end, we mimic human judgment and only focus on the sentence of each passage that is the most related to a query [52].…”

Pre-training Methods in Information Retrieval

“…In this section, we introduce recent works designing PTMs tailored for IR (Lee et al, 2019b;Chang et al, 2019;Ma et al, 2021b;Ma et al, 2021c;Boualili et al, 2020;Ma et al, 2021d;Zou et al, 2021;Liu et al, 2021d). General pre-trained models like BERT have achieved great success when applied to IR tasks on both the firststage retrieval and the re-ranking stage.…”

Pre-trained Language Model for Web-scale Retrieval in Baidu Search

“…Despite the above-mentioned bi-encoder models, interactionbased methods are also widely used in many information retrieval systems [9,10,44,[48][49][50][51][52]. As such, another line of research for semantic matching is to model query-document interaction with DNNs [25,28,40,44,53]. However, they cannot cache the document embeddings offline, and thus are inefficient for retrieval.…”

“…They are preferred for ranking stage, which will not be further discussed in this paper. In our search engine, interaction-based methods are exploited to build the PLM-based ranking system [53].…”