Optimization-driven Deep Reinforcement Learning for Robust Beamforming in IRS-assisted Wireless Communications

Abstract

Intelligent reflecting surface (IRS) is a promising technology to assist downlink information transmissions from a multi-antenna access point (AP) to a receiver. In this paper, we minimize the AP's transmit power by a joint optimization of the AP's active beamforming and the IRS's passive beamforming. Due to uncertain channel conditions, we formulate a robust power minimization problem subject to the receiver's signalto-noise ratio (SNR) requirement and the IRS's power budget constraint. We propose a deep reinforcement learning (DRL) approach that can adapt the beamforming strategies from past experiences. To improve the learning performance, we derive a convex approximation as a lower bound on the robust problem, which is integrated into the DRL framework and thus promoting a novel optimization-driven deep deterministic policy gradient (DDPG) approach. In particular, when the DDPG algorithm generates a part of the action (e.g., passive beamforming), we can use the model-based convex approximation to optimize the other part (e.g., active beamforming) of the action more efficiently. Our simulation results demonstrate that the optimization-driven DDPG algorithm can improve both the learning rate and reward performance significantly compared to the conventional modelfree DDPG algorithm.

Interplay between RIS and AI in Wireless Communications: Fundamentals, Architectures, Applications, and Open Research Problems

“…To avoid huge training labels, DRLbased methods, which achieve the property of online learning and sample generation, are widely utilized. In [80], an efficient DRL method is proposed to solve the non-convex optimization problem of the phase shift deign for the RIS-aided downlink MISO wireless communication system to maximize the received SNR and, in [81], to minimize the BS transmit power by jointly optimizing the active beamforming at the BS and passive beamforming at the RIS, as shown in Fig. 8(a).…”

“…Reference [82] also investigated the joint beamforming design of transmit beamforming matrix at the BS and the phase shift matrix at the RIS by leveraging the recent advances in DRL with the model-driven DDPG approach. However, unlike the aforementioned work in [81] that applies alternating optimization to alternatively obtain the optimal transmit beamforming and phase shift matrix, the proposed method can simultaneously achieve the optimal transmit beamforming and phase shift matrix by maximizing the sum rate, which is utilized as the instant rewards to train the DRL-based algorithm. The same group of authors also investigated the joint design of digital beamforming matrix at the BS and analog beamforming matrices at the RISs by leveraging the DRL framework to combat the propagation loss in [83], which further shows that DRL-based architectures are those effective methods for tackling the non-convex optimization problems, such as NP-hard beamforming problems.…”