Scalable photonic reinforcement learning by time-division multiplexing of laser chaos

Abstract

Reinforcement learning involves decision-making in dynamic and uncertain environments and constitutes a crucial element of artificial intelligence. In our previous work, we experimentally demonstrated that the ultrafast chaotic oscillatory dynamics of lasers can be used to efficiently solve the two-armed bandit problem, which requires decision-making concerning a class of difficult trade-offs called the exploration–exploitation dilemma. However, only two selections were employed in that research; hence, the scalability of the laser-chaos-based reinforcement learning should be clarified. In this study, we demonstrated a scalable, pipelined principle of resolving the multi-armed bandit problem by introducing time-division multiplexing of chaotically oscillated ultrafast time series. The experimental demonstrations in which bandit problems with up to 64 arms were successfully solved are presented where laser chaos time series significantly outperforms quasiperiodic signals, computer-generated pseudorandom numbers, and coloured noise. Detailed analyses are also provided that include performance comparisons among laser chaos signals generated in different physical conditions, which coincide with the diffusivity inherent in the time series. This study paves the way for ultrafast reinforcement learning by taking advantage of the ultrahigh bandwidths of light wave and practical enabling technologies.

Dynamic channel selection in wireless communications via a multi-armed bandit algorithm using laser chaos time series

“…The photonic dynamics provide an ultrafast irregular time sequence unachievable by electronics means; hence, application to photonic ultrafast random number generation [14,15] and photonic reservoir computing [16] has been demonstrated. Naruse et al exploits such photonic physical processes in decision-making [8,9,17].…”

“…In such decision-making mechanisms, the estimated reward probability ˆi P of the selection i, has been considered important in former studies [5,8,9]. The increase or decrease in the threshold level should be attenuated based on ˆi P .…”

“…when the decision is to select the Option 0. Here the parameter α is termed the forgetting parameter to adapt to the uncertain environmental change [8,9]; a smaller α indicates rapid forgetting of past threshold values. In the case of four-armed bandits demonstrated in the following, in total five threshold values are arranged.…”

“…Otherwise, the second MSB is (8) As mentioned in Eq. (2), the number of changes in the threshold can be configured differently as specified by Ω, reflecting the estimated reward probabilities of the selections [8,9]. In the present study, however, a fixed number of changes is employed while introducing a smaller forgetting parameter α = 0.9.…”

“…The details of the flexible updating of Ω and the thresholds are found in ref. [9].Experimental wireless communication implementations. The four-channel WLAN system was constructed as follows.…”