Solving the RNA design problem with reinforcement learning

Abstract

We use reinforcement learning to train an agent for computational RNA design: given a target secondary structure, design a sequence that folds to that structure in silico. Our agent uses a novel graph convolutional architecture allowing a single model to be applied to arbitrary target structures of any length. After training it on randomly generated targets, we test it on the Eterna100 benchmark and find it outperforms all previous algorithms. Analysis of its solutions shows it has successfully learned some advanced strategies identified by players of the game Eterna, allowing it to solve some very difficult structures. On the other hand, it has failed to learn other strategies, possibly because they were not required for the targets in the training set. This suggests the possibility that future improvements to the training protocol may yield further gains in performance.

EternaBrain: Automated RNA design through move sets and strategies from an Internet-scale RNA videogame

“…On one hand, this performance is better than other algorithms that had been tested on the Eterna100 at the time of development (54 of 100 was the previous maximum). On the other hand, the EternaBrain-SAP performance is similar to or worse than newer algorithms (SIMARD, sentRNA, the reinforcement learning algorithm of Eastman et al, NEMO) that have been developed concomitantly or after EternaBrain and whose performance on the Eterna100 has been reported in newer papers or preprints [15][16][17][18][19]. Furthermore, none of these methods match the level of the top ten Eterna human players, who can solve all 100 puzzles of the Eterna100 benchmark.…”

“…It is instructive to compare EternaBrain-SAP to the newer generation of RNA design algorithms. Like EternaBrain-SAP, the new methods SentRNA, the Eastman et al method, and LEARNA use artificial neural networks to distill potentially useful information from gameplay and solve 80, 60, and 65 out of 100 Eterna100 puzzles, respectively [15][16][18]. SentRNA seeks to find solutions to RNA secondary design problems in ‘one shot’ rather than through EternaBrain’s iterative moves [18].…”

“…AlphaGo and its successor AlphaGo Zero have further improved their performance using reinforcement learning [14]. While reinforcement learning has shown promise for RNA design, it is not yet human competitive [15][16]. This observation suggests that there remain lessons to be learned from human moves and strategies.…”

“…Furthermore, EternaBrain-SAP performs similarly to newer methods developed concomitantly by us and other groups using complementary algorithmic approaches. In the discussion, we compare EternaBrain-SAP’s performance on the Eterna100 with these more recently-reported methods, including SIMARD [17], SentRNA [18], NEMO [19], antaRNA [20], MCTS-RNA [21], and the reinforcement learning methods of Eastman et al [15] and LEARNA [16], drawing lessons for future efforts in automated RNA design. Additionally, we highlight the likelihood of future progress as other methods [4][22] and newer computational energy functions [23] are developed and tested on the same benchmark as well as on biological RNA structures.…”

EternaBrain: Automated RNA design through move sets and strategies from an Internet-scale RNA videogame

“…At the same time, EternaBrain-SAP performs similarly or poorer than newer methods developed concomitantly by us and other groups. In the discussion, we compare EternaBrain-SAP's performance on the Eterna100 with these more recentlyreported methods, including SIMARD (17), SentRNA (18), NEMO (19), antaRNA (20), MCTS-RNA (21), and the reinforcement learning methods of Eastman et al (15) and LEARNA (16), drawing lessons for future efforts in automated RNA design. Additionally, we highlight the likelihood of future progress as other methods (4,22) and newer computational energy functions (23) are developed and tested on the same benchmark as well as on biological RNA structures.…”