Feedforward beta control in the KSTAR tokamak by deep reinforcement learning

Nuclear fusion using magnetic confinement, in particular in the tokamak configuration, is a promising path towards sustainable energy. A core challenge is to shape and maintain a high-temperature plasma within the tokamak vessel. This requires high-dimensional, high-frequency, closed-loop control using magnetic actuator coils, further complicated by the diverse requirements across a wide range of plasma configurations. In this work, we introduce a previously undescribed architecture for tokamak magnetic controller design that autonomously learns to command the full set of control coils. This architecture meets control objectives specified at a high level, at the same time satisfying physical and operational constraints. This approach has unprecedented flexibility and generality in problem specification and yields a notable reduction in design effort to produce new plasma configurations. We successfully produce and control a diverse set of plasma configurations on the Tokamak à Configuration Variable1,2, including elongated, conventional shapes, as well as advanced configurations, such as negative triangularity and ‘snowflake’ configurations. Our approach achieves accurate tracking of the location, current and shape for these configurations. We also demonstrate sustained ‘droplets’ on TCV, in which two separate plasmas are maintained simultaneously within the vessel. This represents a notable advance for tokamak feedback control, showing the potential of reinforcement learning to accelerate research in the fusion domain, and is one of the most challenging real-world systems to which reinforcement learning has been applied.

show abstract

“…Recently, Seo et al. 49 have developed feedforward signals for beta control using RL, which have then been verified on the KSTAR tokamak.…”

Section: Methodsmentioning

confidence: 99%

Magnetic control of tokamak plasmas through deep reinforcement learning

Degrave

Felici

Buchli

et al. 2022

Nature

399

218

View full text Add to dashboard Cite

show abstract

“…The integrated tokamak simulation with multiple physics codes [14] can offer virtual experiments for the RL agent, but it is still computationally expensive for millions of simulations. In this work, instead, we employ a data-driven simulator using long short-term memory (LSTM) [15]-based neural networks, which provides fast enough simulations and experimentally relevant plasma responses [11,16].…”

Section: Modelingmentioning

confidence: 99%

“…Active control of plasma parameters has been widely attempted with various techniques, such as the proportionalintegral-differential (PID) feedback algorithm [8,9] and model predictive control (MPC) [10]. Recently, feedforward control of β N has been conducted by designing the operation scenario with the deep reinforcement learning (RL) technique [11]. The RL method has also been applied for real-time magnetic coil control to form various plasma configurations, requiring sophisticated decision-making in multi-dimensional engineering space based on magneto-hydrodynamics (MHD) [12].…”

Section: Introductionmentioning

confidence: 99%

“…In the previous work [11], we first developed a data-driven simulator from the experimental data in KSTAR, and then we trained an RL-based AI in the simulation to design an operation scenario to achieve a given target β N . However, it was revealed that the control of only β N occasionally yields MHD instabilities in the experiment due to a lack of information regarding the MHD stabilities in the simulator.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Development of an operation trajectory design algorithm for control of multiple 0D parameters using deep reinforcement learning in KSTAR

Seo¹,

Na²,

Kim³

et al. 2022

Nucl. Fusion

Self Cite

View full text Add to dashboard Cite

This work develops an artificially intelligent (AI) tokamak operation design algorithm that provides an adequate operation trajectory to control multiple plasma parameters simultaneously into arbitrary targets. An AI is trained with the reinforcement learning technique in the data-driven tokamak simulator, searching for the best action policy to get a higher reward. By setting the reward function to increase as the achieved βp, q95, and li are close to the given target values, the AI tries to properly determine the plasma current and boundary shape to reach the given targets. After training the AI with various targets and conditions in the simulation environment, we demonstrated that we could successfully achieve the target plasma states with the AI-designed operation trajectory in a real KSTAR experiment. The developed algorithm would replace the human task of searching for an operation setting for given objectives, provide clues for developing advanced operation scenarios, and serve as a basis for the autonomous operation of a fusion reactor.

show abstract

“…Therefore, it is really important building a robust counterfactual generator as well as perfecting the RWM stability forecaster. The implementation of ML-based controllers has been gaining attention in recent years with through the usage of reinforcement learning, for example to determine the optimal control scenario to obtain a constant user-defined β N level in the KSTAR tokamak [54]. The direction we have taken in the present work follows the observation that the disruptive β N is not constant when it comes to RWM stability and the usage of counterfactuals might lay the basis for a control algorithm that takes this variability into account.…”

Section: Ml-informed Safe Scenarios For Potential Real-time Controlmentioning

confidence: 99%

Predicting resistive wall mode stability in NSTX through balanced random forests and counterfactual explanations

Piccione¹,

Berkery²,

Sabbagh³

et al. 2022

Nucl. Fusion

View full text Add to dashboard Cite

Recent progress in the disruption event characterization and forecasting framework has shown that machine learning guided by physics theory can be easily implemented as a supporting tool for fast computations of ideal stability properties of spherical tokamak plasmas. In order to extend that idea, a customized random forest (RF) classifier that takes into account imbalances in the training data is hereby employed to predict resistive wall mode (RWM) stability for a set of high beta discharges from the NSTX spherical tokamak. More specifically, with this approach each tree in the forest is trained on samples that are balanced via a user-defined over/under-sampler. The proposed approach outperforms classical cost-sensitive methods for the problem at hand, in particular when used in conjunction with a random under-sampler, while also resulting in a threefold reduction in the training time. In order to further understand the model’s decisions, a diverse set of counterfactual explanations based on determinantal point processes (DPP) is generated and evaluated. Via the use of DPP, the underlying RF model infers that the presence of hypothetical magnetohydrodynamic activity would have prevented the RWM from concurrently going unstable, which is a counterfactual that is indeed expected by prior physics knowledge. Given that this result emerges from the data-driven RF classifier and the use of counterfactuals without hand-crafted embedding of prior physics intuition, it motivates the usage of counterfactuals to simulate real-time control by generating the β N levels that would have kept the RWM stable for a set of unstable discharges.

show abstract

Feedforward beta control in the KSTAR tokamak by deep reinforcement learning

Cited by 38 publications

References 44 publications

Magnetic control of tokamak plasmas through deep reinforcement learning

Magnetic control of tokamak plasmas through deep reinforcement learning

Development of an operation trajectory design algorithm for control of multiple 0D parameters using deep reinforcement learning in KSTAR

Predicting resistive wall mode stability in NSTX through balanced random forests and counterfactual explanations

Contact Info

Product

Resources

About