Deep Reinforcement Learning for Sponsored Search Real-time Bidding

Zhao, Jun; Qiu, Guang; Guan, Ziyu; Zhao, Wei; He, Xiaofei

doi:10.1145/3219819.3219918

Cited by 64 publications

(51 citation statements)

References 28 publications

(56 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the key advantages of DRL is that it enables RL to scale to problems with high-dimensional state and action spaces. However, most existing successful DRL applications so far have been on visual domains (e.g., Atari games), and there is still a lot of work to be done for more realistic applications [25,26] with complex dynamics, which are not necessarily vision-based.DRL has been regarded as an important component in constructing general AI systems [27] and has been successfully integrated with other techniques, e.g., search [14], planning [28], and more recently with multiagent systems, with an emerging area of multiagent deep reinforcement learning (MDRL) [29,30]. 1 Learning in multiagent settings is fundamentally more difficult than the single-agent case due to the presence of multiagent pathologies, e.g., the moving target problem (non-stationarity) [2, 5, 10], curse of dimensionality [2,5], multiagent credit assignment [31,32], global exploration [8], and relative overgeneralization [33,34,35].…”

mentioning

confidence: 99%

“…18 https://github.com/gjp1203/nui_in_madrl 19 https://github.com/gjp1203/nui_in_madrl 20 https://www.pommerman.com/ 21 https://github.com/oxwhirl/smac 22 https://github.com/oxwhirl/pymarl MuJoCo Multiagent Soccer[313] uses the MuJoCo physics engine[202]. The environment simulates a 2 vs. 2 soccer game with agents having a 3-dimensional action space 26. • Neural MMO[314] is a research platform 27 inspired by the human game genre of Massively Multiplayer Online (MMO) Role-Playing Games.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A survey and critique of multiagent deep reinforcement learning

Hernández-Leal¹,

Kartal²,

Taylor³

2019

Auton Agent Multi-Agent Syst

421

240

View full text Add to dashboard Cite

Deep reinforcement learning (RL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent learning (MAL) scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. The primary goal of this article is to provide a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Additionally, we complement the overview with a broader analysis: (i) we revisit previous key components, originally presented in MAL and RL, and highlight how they have been adapted to multiagent deep reinforcement learning settings. (ii) We provide general guidelines to new practitioners in the area: describing lessons learned from MDRL works, pointing to recent benchmarks, and outlining open avenues of research. (iii) We take a more critical tone raising practical challenges of MDRL (e.g., implementation and computational demands). We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists (e.g., RL and MAL) in a joint effort to promote fruitful research in the multiagent community.$ Earlier versions of this work had the title: "Is multiagent deep reinforcement learning the answer or the question? A brief survey" arXiv:1810.05587v3 [cs.MA] 30 Aug 2019 Go [14,15], poker [16,17], and games of two competing teams, e.g., DOTA 2 [18] and StarCraft II [19].While different techniques and algorithms were used in the above scenarios, in general, they are all a combination of techniques from two main areas: reinforcement learning (RL) [20] and deep learning [21,22].RL is an area of machine learning where an agent learns by interacting (i.e., taking actions) within a dynamic environment. However, one of the main challenges to RL, and traditional machine learning in general, is the need for manually designing quality features on which to learn. Deep learning enables efficient representation learning, thus allowing the automatic discovery of features [21,22]. In recent years, deep learning has had successes in different areas such as computer vision and natural language processing [21,22]. One of the key aspects of deep learning is the use of neural networks (NNs) that can find compact representations in high-dimensional data [23].In deep reinforcement learning (DRL) [23,24] deep neural networks are trained to approximate the optimal policy and/or the value function. In this way the deep NN, serving as function approximator, enables powerful generalization. One of the key advantages of DRL is that it enables RL to scale to problems with high-dimensional state and action spaces. However, most existing successful DRL applications so far have been on visual domains (e.g., Atari games), and there is still a lot of work to be done for more realistic applications [25,26] with complex dynamics, which are not necessarily vision-based.DRL h...

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

A survey and critique of multiagent deep reinforcement learning

Hernández-Leal¹,

Kartal²,

Taylor³

2019

Auton Agent Multi-Agent Syst

421

240

View full text Add to dashboard Cite

show abstract

“…Such a scenario is typical in nextgeneration wireless networks such as 5G highly-dense heterogeneous networks. DRL appears to be an efficient approach for solving different types of auctions such as in [174].…”

Section: ) Drl For Channel Estimation In Wireless Systemsmentioning

confidence: 99%

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

Luong

Hoang

Gong

et al. 2019

IEEE Commun. Surv. Tutorials

1,401

481

View full text Add to dashboard Cite

This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, deep reinforcement learning, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of deep reinforcement learning from fundamental concepts to advanced models. Then, we review deep reinforcement learning approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks such as 5G and beyond. Furthermore, we present applications of deep reinforcement learning for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying deep reinforcement learning.

show abstract

“…Similarly, a bid optimizing strategy called optimized cost per click was proposed in [49]. A reinforcement learning based real-time bidding strategy for sponsored search was proposed in [48]. In the industry of sponsored search, there are also some bidding strategies tools [6,18], such as Enhanced CPC, Target CPA, Maximize Conversions, and so on.…”

Section: Related Workmentioning

confidence: 99%

AiAds

Yang

Sun

Zhu

et al. 2019

Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Sponsored search has more than 20 years of history, and it has been proven to be a successful business model for online advertising. Based on the pay-per-click pricing model and the keyword targeting technology, the sponsored system runs online auctions to determine the allocations and prices of search advertisements. In the traditional setting, advertisers should manually create lots of ad creatives and bid on some relevant keywords to target their audience. Due to the huge amount of search traffic and a wide variety of ad creations, the limits of manual optimizations from advertisers become the main bottleneck for improving the efficiency of this market. Moreover, as many emerging advertising forms and supplies are growing, it's crucial for sponsored search platform to pay more attention to the ROI metrics of ads for getting the marketing budgets of advertisers.In this paper, we present the AiAds system developed at Baidu, which use machine learning techniques to build an automated and intelligent advertising system. By designing and implementing the automated bidding strategy, the intelligent targeting and the intelligent creation models, the AiAds system can transform the manual optimizations into multiple automated tasks and optimize these tasks in advanced methods. AiAds is a brand-new architecture of sponsored search system which changes the bidding language and allocation mechanism, breaks the limit of keyword targeting with end-to-end ad retrieval framework and provides global optimization of ad creation. This system can increase the advertiser's campaign performance, the user experience and the revenue of the advertising platform simultaneously and significantly. We present the overall architecture and modeling techniques for each module of the system and share our lessons learned in solving several key challenges. Finally, online A/B test and long-term grouping experiment demonstrate the advancement and effectiveness of this system. CCS CONCEPTS• Information systems → Sponsored search advertising; Computational advertising.

show abstract

Deep Reinforcement Learning for Sponsored Search Real-time Bidding

Cited by 64 publications

References 28 publications

A survey and critique of multiagent deep reinforcement learning

A survey and critique of multiagent deep reinforcement learning

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

AiAds

Contact Info

Product

Resources

About