Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning

Abstract

Quantum machine learning (QML) has been identified as one of the key fields that could reap advantages from near-term quantum devices, next to optimization and quantum chemistry. Research in this area has focused primarily on variational quantum algorithms (VQAs), and several proposals to enhance supervised, unsupervised and reinforcement learning (RL) algorithms with VQAs have been put forward. Out of the three, RL is the least studied and it is still an open question whether VQAs can be competitive with state-of-the-art classical algorithms based on neural networks (NNs) even on simple benchmark tasks. In this work, we introduce a training method for parametrized quantum circuits (PQCs) that can be used to solve RL tasks for discrete and continuous state spaces based on the deep Q-learning algorithm. We investigate which architectural choices for quantum Q-learning agents are most important for successfully solving certain types of environments by performing ablation studies for a number of different data encoding and readout strategies. We provide insight into why the performance of a VQA-based Q-learning algorithm crucially depends on the observables of the quantum model and show how to choose suitable observables based on the learning task at hand. To compare our model against the classical DQN algorithm, we perform an extensive hyperparameter search of PQCs and NNs with varying numbers of parameters. We confirm that similar to results in classical literature, the architectural choices and hyperparameters contribute more to the agents' success in a RL setting than the number of parameters used in the model. Finally, we show when recent separation results between classical and quantum agents for policy gradient RL can be extended to inferring optimal Q-values in restricted families of environments.

Quantum algorithms applied to satellite mission planning for Earth observation

“…3) Hybrid-quantum PPO: Inspired by the quantum advantage shown in Refs. [10]- [12], [24], [25], this section investigates the utility of a hybrid-quantum neural network as a policy model for RL. Fig.…”

UAV Coverage Path Planning with Quantum-based Deep Deterministic Policy Gradient

“…The best possible action solution can be estimated to maximize objective reward through learning. --- [16], [17] -- [23], [24] --this work during a finite flight time. In [11], a deep Q-network (DQN) was designed to optimize UAV navigation using a massive multiple-input-multiple-output (MIMO) technique by selecting the optimal policy.…”

“…In [23], [24], quantum reinforcement learning outperformed classical reinforcement learning by obtaining higher rewards. In [23], classical deep reinforcement learning algorithms, such as experience replay and target networks, were transformed into a representation of variational quantum circuits.…”

“…In [23], classical deep reinforcement learning algorithms, such as experience replay and target networks, were transformed into a representation of variational quantum circuits. In [24], a training method for parametrized quantum circuits (PQCs) that can be used to solve tasks based on the deep Q-learning algorithm was introduced. Based on the aforementioned related studies, QNN promises a solution for the high computational complexity issue in classical DDPG yet achieve higher rewards.…”

Trainability Enhancement of Parameterized Quantum Circuits via Reduced-Domain Parameter Initialization

“…The core idea is exploiting PQCs to delegate classically difficult computation and variationally updating the parameters in PQCs with a powerful classical optimizer [28,29,30,31,32]. With error mitigation techniques [33,34,35,36,37,38,39,40], VQAs have been applied to various fields and the corresponding specific algorithms have been developed, including variational quantum eigensolver (VQE) [41,42,43,44,45], quantum approximate optimization algorithm [46,47,48,15,49], variational quantum simulator [50,51,52,53], quantum neural network (QNN) [54,55,56,57,58,59,60,61,62,63], and others [64,65,66,67,68,69,70,71,72,...…”