ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps

Abstract

Pre-trained models (PTMs) have become a fundamental backbone for downstream tasks in natural language processing and computer vision. Despite initial gains that were obtained by applying generic PTMs to geo-related tasks at Baidu Maps, a clear performance plateau over time was observed. One of the main reasons for this plateau is the lack of readily available geographic knowledge in generic PTMs. To address this problem, in this paper, we present POLARIS, which is a geographic pre-trained model designed and developed for improving the geo-related tasks at Baidu Maps. POLARIS is elaborately designed to learn a universal representation of geography-language by pre-training on large-scale data generated from a heterogeneous graph that contains abundant geographic knowledge. Extensive quantitative and qualitative experiments conducted on large-scale real-world datasets demonstrate the superiority and effectiveness of POLARIS. POLARIS has already been deployed in production at Baidu Maps since April 2021, which significantly benefits the performance of a wide range of downstream tasks. This demonstrates that POLARIS can serve as a fundamental backbone for geo-related tasks. CCS CONCEPTS• Information systems → Data mining.

A Hierarchy-Aware Geocoding Model Based on Cross-Attention within the Seq2Seq Framework

“…Therefore, researchers have begun to apply end-to-end deep learning models to geocoding in order to directly predict associated geographic spatial labels based on input query texts. This is often modeled as a coordinate point-based regression task and a grid-based or region-based multi-classification task [12,13,15], with low dependence on external databases, such as gazetteers, and stronger generalization ability. Therefore, it is widely used in tasks such as event geocoding and Internet text geocoding [34,35].…”

“…This approach deepens the semantic understanding of address text and can directly predict geographical spatial labels from text with a straightforward workflow and low dependency on external databases such as gazetteers. It typically includes two primary methods: modeling as a regression task based on coordinate points or as a classification task based on regions such as grids or polygons [1][2][3][12][13][14][15]. Due to the challenges involved in directly learning the mapping between text and precise coordinates in coordinate-point-based tasks [15], some researchers prefer to model geocoding as a classification problem for predicting regions based on grids or polygons.…”

“…It typically includes two primary methods: modeling as a regression task based on coordinate points or as a classification task based on regions such as grids or polygons [1][2][3][12][13][14][15]. Due to the challenges involved in directly learning the mapping between text and precise coordinates in coordinate-point-based tasks [15], some researchers prefer to model geocoding as a classification problem for predicting regions based on grids or polygons. Compared with approaches that use complex polygon structures for cities, states, and countries, geographic grids offer multi-level spatial representations without the need for external metadata, making them straightforward to understand and use.…”

“…Therefore, this study focuses on gridbased classification methods for geocoding. This approach usually discretizes Earth's surface into a series of grids and predicts the grid category corresponding to the input text based on a classification model to indirectly obtain geographical coordinates [1,2,6,[13][14][15][16][17][18]. Additionally, discrete global grid systems such as Geohash, H3, and S2 geometry play a crucial role in the spatial indexing, association, and geolocation of satellite and streetview imagery data [16,[19][20][21][22][23][24].…”

“…Taking the S2 grid as an example, when the average unit grid area is approximately 300 m², the global number of grids can reach 1649 billion. This leads to a serious dimensionality explosion in the output space, making it difficult for classification models to train and thus leading to a decrease in geocoding accuracy [15,28]. Moreover, most approaches generally overlook the inherent hierarchical structure characteristics of geographical grids and address text as well as the potential associations between them [29,30].…”