DeepCAT: A Cost-Efficient Online Configuration Auto-Tuning Approach for Big Data Frameworks

Dou, Hui; Wang, Yilun; Zhang, Yiwen; Chen, Pengfei

doi:10.1145/3545008.3545018

Cited by 3 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several optimizations methods [ 24 , 25 , 26 , 27 , 28 ] are discussed to improve the performance of distributed systems. Donta et al [ 24 ] summarize various message queues and message brokers used in IoT systems, and they find out that multiple message queues handle messages as per predefined constraints, making them static in nature.…”

Section: Related Workmentioning

confidence: 99%

“…Additionally, implementing NetMQ may require specialized hardware and expertise in programmable switches. Dou et al [ 27 ] propose DeepCAT for online configuration auto-tuning for big data frameworks. DeepCAT leverages the TD3 (twin delayed deep deterministic policy gradient) algorithm and incorporates a novel reward-driven prioritized experience replay mechanism.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Towards an Optimized Distributed Message Queue System for AIoT Edge Computing: A Reinforcement Learning Approach

Xie

et al. 2023

Sensors

View full text Add to dashboard Cite

The convergence of artificial intelligence and the Internet of Things (IoT) has made remarkable strides in the realm of industry. In the context of AIoT edge computing, where IoT devices collect data from diverse sources and send them for real-time processing at edge servers, existing message queue systems face challenges in adapting to changing system conditions, such as fluctuations in the number of devices, message size, and frequency. This necessitates the development of an approach that can effectively decouple message processing and handle workload variations in the AIoT computing environment. This study presents a distributed message system for AIoT edge computing, specifically designed to address the challenges associated with message ordering in such environments. The system incorporates a novel partition selection algorithm (PSA) to ensure message order, balance the load among broker clusters, and enhance the availability of subscribable messages from AIoT edge devices. Furthermore, this study proposes the distributed message system configuration optimization algorithm (DMSCO), based on DDPG, to optimize the performance of the distributed message system. Experimental evaluations demonstrate that, compared to the genetic algorithm and random searching, the DMSCO algorithm can provide a significant improvement in system throughput to meet the specific demands of high-concurrency AIoT edge computing applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Towards an Optimized Distributed Message Queue System for AIoT Edge Computing: A Reinforcement Learning Approach

Xie

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

DeepCAT⁺: A Low-Cost and Transferrable Online Configuration Auto-Tuning Approach for Big Data Frameworks

Dou,

Wang,

Zhang

et al. 2024

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

CTuner: Automatic NoSQL Database Tuning with Causal Reinforcement Learning

Mai,

He,

et al. 2024

Proceedings of the 15th Asia-Pacific Symposium on Internetware

View full text Add to dashboard Cite

DeepCAT: A Cost-Efficient Online Configuration Auto-Tuning Approach for Big Data Frameworks

Cited by 3 publications

References 21 publications

Towards an Optimized Distributed Message Queue System for AIoT Edge Computing: A Reinforcement Learning Approach

Towards an Optimized Distributed Message Queue System for AIoT Edge Computing: A Reinforcement Learning Approach

DeepCAT⁺: A Low-Cost and Transferrable Online Configuration Auto-Tuning Approach for Big Data Frameworks

CTuner: Automatic NoSQL Database Tuning with Causal Reinforcement Learning

Contact Info

Product

Resources

About

DeepCAT: A Cost-Efficient Online Configuration Auto-Tuning Approach for Big Data Frameworks

Cited by 3 publications

References 21 publications

Towards an Optimized Distributed Message Queue System for AIoT Edge Computing: A Reinforcement Learning Approach

Towards an Optimized Distributed Message Queue System for AIoT Edge Computing: A Reinforcement Learning Approach

DeepCAT+: A Low-Cost and Transferrable Online Configuration Auto-Tuning Approach for Big Data Frameworks

CTuner: Automatic NoSQL Database Tuning with Causal Reinforcement Learning

Contact Info

Product

Resources

About

DeepCAT⁺: A Low-Cost and Transferrable Online Configuration Auto-Tuning Approach for Big Data Frameworks