Online Reinforcement Learning for Dynamic Multimedia Systems

Abstract

In our previous work, we proposed a systematic cross-layer framework for dynamic multimedia systems, which allows each layer to make autonomous and foresighted decisions that maximize the system's long-term performance, while meeting the application's real-time delay constraints. The proposed solution solved the cross-layer optimization offline, under the assumption that the multimedia system's probabilistic dynamics were known a priori, by modeling the system as a layered Markov decision process. In practice, however, these dynamics are unknown a priori and, therefore, must be learned online. In this paper, we address this problem by allowing the multimedia system layers to learn, through repeated interactions with each other, to autonomously optimize the system's long-term performance at run-time. The two key challenges in this layered learning setting are: (i) each layer's learning performance is directly impacted by not only its own dynamics, but also by the learning processes of the other layers with which it interacts; and (ii) selecting a learning model that appropriately balances time-complexity (i.e., learning speed) with the multimedia system's limited memory and the multimedia application's real-time delay constraints. We propose two reinforcement learning algorithms for optimizing the system under different design constraints: the first algorithm solves the cross-layer optimization in a centralized manner and the second solves it in a decentralized manner. We analyze both algorithms in terms of their required computation, memory, and interlayer communication overheads. After noting that the proposed reinforcement learning algorithms learn too slowly, we introduce a complementary accelerated learning algorithm that exploits partial knowledge about the system's dynamics in order to dramatically improve the system's performance. In our experiments, we demonstrate that decentralized learning can perform equally as well as centralized learning, while enabling the layers to act autonomously. Additionally, we show that existing application-independent reinforcement learning algorithms, and existing myopic learning algorithms deployed in multimedia systems, perform significantly worse than our proposed application-aware and foresighted learning methods.

A Sarsa(λ)-Based Control Model for Real-Time Traffic Light Coordination

“…Sarsa( λ ) is an eligibility trace [21] version of Sarsa. The update of Q π ( s , a ) [22] depends on $\begin{array}{c}{Q}_{t+\mathrm{1}}\left(s,a\right)=Q_t\left(s,a\right)+\alpha {\delta }_t{e}_t\left(s,a\right),\end{array}$ where δ t = r t +1 + γQ t ( s t +1 , a t +1 ) − Q t ( s t , a t ) and $\begin{array}{c}{e}_t(s,a)=\{\begin{array}{cc}\gamma \lambda {e_t}_{-\mathrm{normal1}}(s,a)+\mathrm{normal1}\hfill & \text{if}\hspace{0.17em}\hspace{0.17em}s=s_t,\hspace{0.17em}\hspace{0.17em}a=a_t\hfill \\ \gamma \lambda {e_t}_{-\mathrm{normal1}}(s,a)\hfill & \text{otherwise}.\hfill \end{array}\end{array}$ …”

Reinforcement learning for energy-efficient wireless transmission

“…PDS learning uses a sample average of the PDS value function to approximate given the experience tuple in each time slot: (9) where is a time-varying learning rate parameter.…”

“…and ) in (8). Second, updating a single PDS using (9) provides information about the state-value function at many states. This is evident from the expected PDS value function on the right-hand-side of (8).…”

“…Consequently, the statistical information obtained from the experience tuple at time can be extrapolated to other PDSs with different post-decision buffer states and different post-decision PM states. Due to space limitations, we refer the interested reader to our prior work [9], where we refer to this as virtual experience learning. Table 2 summarizes the parameters used in our MATLAB-based simulator.…”

A Q-Learning Based Self-Adaptive I/O Communication for 2.5D Integrated Many-Core Microprocessor and Memory

“…On-line machine learning based power management has been recently practiced [26], [27], [28], [29], [30], [31]. For example, Q-learning [32], [33] can be utilized to find an optimal action-selection policy from set of states.…”