A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning

Liu, Ning; Li, Zhe; Xu, Jiaojiao; Xu, Zhiyuan; Lin, Sheng; Qiu, Qinru; Tang, Jian; Wang, Yanzhi

doi:10.1109/icdcs.2017.123

Cited by 226 publications

(125 citation statements)

References 30 publications

Supporting

Mentioning

100

Contrasting

Unclassified

Order By: Relevance

“…Since tabular-Q learning requires iterative updating to converge, an optimal policy is difficult to find in limited time [14][15][16]. To solve this problem, deep Q-networks algorithm was proposed by combining Q-learning with deep neural networks [17][18][19][20]. Van Hasselt, Guez and Silver proposed double DQN to address the overestimation of Q-learning [17].…”

Section: Uncertain Mobile Crowdsourcing and Deep Reinforcement Learningmentioning

confidence: 99%

See 1 more Smart Citation

A trust-aware task allocation method using deep q-learning for uncertain mobile crowdsourcing

Sun

Tan

2019

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

IntroductionWith the advancing technology of mobile devices with numerous built-in sensors, mobile crowdsourcing has recently emerged as a new collaboration paradigm in numerous intelligent mobile information systems [1]. The existing mobile crowdsourcing has applications in numerous domains including urban planning, traffic monitoring, ride sharing, environmental monitoring and intelligent disaster response [2]. Mobile crowdsourcing is a combination of spatial crowdsourcing and smart phone technology that employs mobile workers to perform certain tasks in a specific AbstractMobile crowdsourcing has emerged as a promising collaboration paradigm in which each spatial task requires a set of mobile workers in near vicinity to the target location. Considering the desired privacy of the participating mobile devices, trust is considered to be an important factor to enable effective collaboration in mobile crowdsourcing. The main impediment to the success of mobile crowdsourcing is the allocation of trustworthy mobile workers to nearby spatial tasks for collaboration. This process becomes substantially more challenging for large-scale online spatial task allocations in uncertain mobile crowdsourcing systems. The uncertainty can mislead the task allocation, resulting in performance degradation. Moreover, the large-scale nature of real-world crowdsourcing poses a considerable challenge to spatial task allocation in uncertain environments. To address the aforementioned challenges, first, an optimization problem of mobile crowdsourcing task allocation is formulated to maximize the trustworthiness of workers and minimize movement distance costs. Second, for the uncertain crowdsourcing scenario, a Markov decision process-based mobile crowdsourcing model (MCMDP) is formulated to illustrate the dynamic trust-aware task allocation problem. Third, to solve large-scale MCMDP problems in a stable manner, this study proposes an improved deep Q-learning-based trust-aware task allocation (ImprovedDQL-TTA) algorithm that combines trust-aware task allocation and deep Q-learning as an improvement over the uncertain mobile crowdsourcing systems. Finally, experimental results illustrate that the ImprovedDQL-TTA algorithm can stably converge in a number of training iterations. Compared with the reference algorithm, our proposed algorithm achieves effective solutions on the experimental data sets. which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

show abstract

Section: Uncertain Mobile Crowdsourcing and Deep Reinforcement Learningmentioning

confidence: 99%

“…Specifically, the dynamic TTA optimization can be formulated as a Markov decision process (MDP) problem. The emerging deep Q-learning (DQL) algorithm shows distinct advantages for largescale MDP problems and has been widely used in dynamic sequential decision making problems [17][18][19][20]. By combining the advantages of both deep neural networks…”

mentioning

confidence: 99%

A trust-aware task allocation method using deep q-learning for uncertain mobile crowdsourcing

Sun

Tan

2019

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…Techniques derived from deep-learning have been recently proposed to address problems in the field of distributed infrastructure such as Cloud data centers and Fog systems. For example, the authors of [23] propose a deep reinforced learning technique for the management of VMs allocation in Cloud data center. Our proposal is completely orthogonal to the proposal in [6] and can be integrated with a class-based approach leveraging the VMs identification proposed in this paper.…”

Section: Related Workmentioning

confidence: 99%

A Deep-learning-based approach to VM behavior Identification in Cloud Systems

Stefanini

Lancellotti

Baraldi

et al. 2019

Proceedings of the 9th International Conference on Cloud Computing and Services Science

View full text Add to dashboard Cite

Cloud computing data centers are growing in size and complexity to the point where monitoring and management of the infrastructure become a challenge due to scalability issues. A possible approach to cope with the size of such data centers is to identify VMs exhibiting a similar behavior. Existing literature demonstrated that clustering together VMs that show a similar behavior may improve the scalability of both monitoring and management of a data center. However, available clustering techniques suffer from a trade-off between the accuracy of the clustering and the time to achieve this result. Not being able to obtain an accurate clustering in short time hinders the application of these solutions, especially in public cloud scenarios where on-demand VMs are instantiated and run for a short time span. Throughout this paper we propose a different approach where, instead of an unsupervised clustering, we rely on classifiers based on deep learning techniques to assign a newly deployed VMs to a cluster of already-known VMs. The two proposed classifiers, namely DeepConv and DeepFFT use a convolution neural network and (in the latter model) exploits Fast Fourier Transformation to classify the VMs. Our proposal is validated using a set of traces describing the behavior of VMs from a real cloud data center. The experiments compare our proposal with state-of-the-art solutions available in literature, such as the AGATE technique and PCA-based clustering, demonstrating that our proposal can achieve a very high accuracy (compared to the best performing alternatives) without the need to introduce the notion of a gray-area to take into account not-yet assigned VMs as in AGATE. Furthermore, we show that our solution is significantly faster than the alternatives as it can produce a perfect classification even with just a few samples of data, such as 4 observations (corresponding to 20 minutes of data), making our proposal viable also to classify on-demand VMs that are characterized by a short life span.

show abstract

“…An RL approach was proposed in [37] to enable automated tuning configuration of MapReduce parameters. Liu et al [27] proposed a novel hierarchical framework for solving the overall resource resource allocation and power management problem in cloud computing systems with DRL. The control problems in MapReduce and cloud systems are quite different from the problem studied here.…”

Section: Related Workmentioning

confidence: 99%

Model-free control for distributed stream data processing using deep reinforcement learning

Tang

et al. 2018

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

In this paper, we focus on general-purpose Distributed Stream Data Processing Systems (DSDPSs), which deal with processing of unbounded streams of continuous data at scale distributedly in real or near-real time. A fundamental problem in a DSDPS is the scheduling problem (i.e., assigning workload to workers/machines) with the objective of minimizing average end-to-end tuple processing time. A widelyused solution is to distribute workload evenly over machines in the cluster in a round-robin manner, which is obviously not efficient due to lack of consideration for communication delay. Model-based approaches (such as queueing theory) do not work well either due to the high complexity of the system environment.We aim to develop a novel model-free approach that can learn to well control a DSDPS from its experience rather than accurate and mathematically solvable system models, just as a human learns a skill (such as cooking, driving, swimming, etc). Specifically, we, for the first time, propose to leverage emerging Deep Reinforcement Learning (DRL) for enabling model-free control in DSDPSs; and present design, implementation and evaluation of a novel and highly effective DRL-based control framework, which minimizes average end-to-end tuple processing time by jointly learning the system environment via collecting very limited runtime statistics data and making decisions under the guidance of powerful Deep Neural Networks (DNNs). To validate and evaluate the proposed framework, we implemented it based on a widely-used DSDPS, Apache Storm, and tested it with three representative applications: continuous queries, log stream processing and word count (stream version). Extensive experimental results show 1) Compared to Storm's default scheduler and the state-of-the-art model-based method, the proposed framework reduces average tuple processing by 33.5% and 14.0% respectively on average. 2) The proposed * Accepted by VLDB 2018 framework can quickly reach a good scheduling solution during online learning, which justifies its practicability for online control in DSDPSs.

show abstract

A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning

Cited by 226 publications

References 30 publications

A trust-aware task allocation method using deep q-learning for uncertain mobile crowdsourcing

A trust-aware task allocation method using deep q-learning for uncertain mobile crowdsourcing

A Deep-learning-based approach to VM behavior Identification in Cloud Systems

Model-free control for distributed stream data processing using deep reinforcement learning

Contact Info

Product

Resources

About