Decision making for large-scale multi-armed bandit problems using bias control of chaotic temporal waveforms in semiconductor lasers

Morijiri, Kensei; Mihana, Takatomo; Kanno, Kazutaka; Naruse, Makoto; Uchida, Atsushi

doi:10.1038/s41598-022-12155-y

Cited by 7 publications

(2 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For comparison, we use the laser-chaos-based method with experimental data (black line in Fig. 2(b)) [8]. We also use Thompson sampling (blue line) [9] and UCB1-tuned method (green line) [10], which are well-known software-based algorithms for solving the multi-armed bandit problem.…”

Section: Resultsmentioning

confidence: 99%

Photonic decision making using optical spatiotemporal chaos generated from spatial light modulator

Uchida,

Morijiri,

Takehana

et al. 2023

Emerging Topics in Artificial Intelligence (ETAI) 2023

Self Cite

View full text Add to dashboard Cite

Photonic decision making for solving the multi-armed bandit problem has been reported recently. However, experimental studies have been limited owing to technological difficulties. In this study, we experimentally demonstrate photonic decision making for solving a large-scale multi-armed bandit problem using optical spatiotemporal chaos generated by a semiconductor laser, spatial light modulator, and camera. We implement a parallel configuration of decision making with up to 512 slot machines, which is much larger than the previous experiment by two orders of magnitude. We found better scaling characteristics between decision-making performance and the number of slot machines than those obtained from software-based algorithms.

show abstract

Section: Resultsmentioning

confidence: 99%

Photonic decision making using optical spatiotemporal chaos generated from spatial light modulator

Uchida,

Morijiri,

Takehana

et al. 2023

Emerging Topics in Artificial Intelligence (ETAI) 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Indeed, scalable decision-making principles have been demonstrated using chaotic time series up to 64-arms based on timedomain multiplexing of chaotic laser time series [27]. Recently, Morijiri et al succeeded in solving 1024-arm bandit problems by bias control of chaotic waveform [28]. Meanwhile, the utilization of chaotic itinerancy in multi-mode laser dynamics is discussed [29].…”

Section: Remarks For Futurementioning

confidence: 99%

Parallel bandit architecture based on laser chaos for reinforcement learning

et al. 2022

Self Cite

View full text Add to dashboard Cite

Accelerating artificial intelligence by photonics is an active field of study aiming to exploit the unique properties of photons. Reinforcement learning is an important branch of machine learning, and photonic decision-making principles have been demonstrated with respect to the multi-armed bandit problems. However, reinforcement learning could involve a massive number of states, unlike previously demonstrated bandit problems where the number of states is only one. Q-learning is a well-known approach in reinforcement learning that can deal with many states. The architecture of Q-learning, however, does not fit well photonic implementations due to its separation of update rule and the action selection. In this study, we organize a new architecture for multi-state reinforcement learning as a parallel array of bandit problems in order to benefit from photonic decision-makers, which we call parallel bandit architecture for reinforcement learning or PBRL in short. Taking a cart-pole balancing problem as an instance, we demonstrate that PBRL adapts to the environment in fewer time steps than Q-learning. Furthermore, PBRL yields faster adaptation when operated with a chaotic laser time series than the case with uniformly distributed pseudorandom numbers where the autocorrelation inherent in the laser chaos provides a positive effect. We also find that the variety of states that the system undergoes during the learning phase exhibits completely different properties between PBRL and Q-learning. The insights obtained through the present study are also beneficial for existing computing platforms, not just photonic realizations, in accelerating performances by the PBRL algorithms and correlated random sequences.

show abstract

Solving multi-armed bandit problems using a chaotic microresonator comb

Cuevas,

Iwami,

Uchida

et al. 2024

APL Photonics

View full text Add to dashboard Cite

The Multi-Armed Bandit (MAB) problem, foundational to reinforcement learning-based decision-making, addresses the challenge of maximizing rewards amid multiple uncertain choices. While algorithmic solutions are effective, their computational efficiency diminishes with increasing problem complexity. Photonic accelerators, leveraging temporal and spatial-temporal chaos, have emerged as promising alternatives. However, despite these advancements, current approaches either compromise computation speed or amplify system complexity. In this paper, we introduce a chaotic microresonator frequency comb (chaotic comb) to tackle the MAB problem, where each comb mode is assigned to a slot machine. Through a proof-of-concept experiment, we employ 44 comb modes to address an MAB with 44 slot machines, demonstrating performance competitive with both conventional software algorithms and other photonic methods. Furthermore, the scalability of decision making is explored with up to 512 slot machines using experimentally obtained temporal chaos in different time slots. Power-law scalability is achieved with an exponent of 0.96, outperforming conventional software-based algorithms. Moreover, we find that a numerically calculated chaotic comb accurately reproduces experimental results, paving the way for discussions on strategies to increase the number of slot machines.

show abstract

Decision making for large-scale multi-armed bandit problems using bias control of chaotic temporal waveforms in semiconductor lasers

Cited by 7 publications

References 37 publications

Photonic decision making using optical spatiotemporal chaos generated from spatial light modulator

Photonic decision making using optical spatiotemporal chaos generated from spatial light modulator

Parallel bandit architecture based on laser chaos for reinforcement learning

Solving multi-armed bandit problems using a chaotic microresonator comb

Contact Info

Product

Resources

About