Achieving autonomous power management using reinforcement learning

Abstract

System level power management must consider the uncertainty and variability that come from the environment, the application and the hardware. A robust power management technique must be able to learn the optimal decision from past events and improve itself as the environment changes. This article presents a novel on-line power management technique based on model-free constrained reinforcement learning (Q-learning). The proposed learning algorithm requires no prior information of the workload and dynamically adapts to the environment to achieve autonomous power management. We focus on the power management of the peripheral device and the microprocessor, two of the basic components of a computer. Due to their different operating behaviors and performance considerations, these two types of devices require different designs of Q-learning agent. The article discusses system modeling and cost function construction for both types of Q-learning agent. Enhancement techniques are also proposed to speed up the convergence and better maintain the required performance (or power) constraint in a dynamic system with large variations. Compared with the existing machine learning based power management techniques, the Q-learning based power management is more flexible in adapting to different workload and hardware and provides a wider range of power-performance tradeoff.

Machine learning for run-time energy optimisation in many-core systems

“…The advantage our approach employing EPD (2) during initial learning (i.e., RL step), is illustrated in Table II, highlighting the average number of explorations for three applications compared against an existing approach [21]. It can be observed that our approach benefits from reduced exploration due to the relationship between current performance and the V-F action (4) [21].…”

“…Predicting the state of the system is a key step in RL and in our methodology the expected workload is classified into a system state at the beginning of each decision epoch [8], [9]. The state of the system is represented using the CPU Cycle Count (CC), obtained using the performance monitoring unit.…”

“…It can be observed that our approach benefits from reduced exploration due to the relationship between current performance and the V-F action (4) [21]. FFT uses the fewest explorations since it exhibits less workload variations resulting in faster learning by the algorithm.…”

“…However, the majority of online approaches [8], [9] do not consider changing application performance requirements. Processor workloads are exercised differently depending on the application tasks being executed.…”

Workload Uncertainty Characterization and Adaptive Frequency Scaling for Energy Minimization of Embedded Systems

“…RESULTS The proposed run-time approach is validated on Texas Instrument's PandaBoard featuring ARM A9 cores and Intel quad-core system running Linux. The proposed approach is compared with one representative approach from each category of related works -the reinforcement learning-based technique of [4], the prediction-based DVFS technique of [1] and the multinomial logistic regression-based technique of [9].…”