Optimal cross-layer wireless control policies using TD learning

Abstract

Abstract-We present an on-line crosslayer control technique to characterize and approximate optimal policies for wireless networks. Our approach combines network utility maximization and adaptive modulation over an infinite discrete-time horizon using a class of performance measures we call time smoothed utility functions. We model the system as an averagecost Markov decision problem. Model approximations are used to find suitable basis functions for application of least squares TD-learning techniques. The approach yields network control policies that learn the underlying characteristics of the random wireless channel and that approximately optimize network performance.

Approximate dynamic programming using fluid and diffusion approximations with applications to power management

“…The preliminary version of this work [13] and the related conference article [24] use these techniques for TD-learning, and motivate the approach through Taylor series approximations. In the present work, these arguments are refined to obtain explicit bounds on the Bellman error.…”