Identification of Animal Behavioral Strategies by Inverse Reinforcement Learning

Abstract

Animals are able to flexibly adapt to new environments by controlling different behavioral patterns. Identification of the behavioral strategy used for this control is important for understanding animals' decision-making, but methods available for quantifying such behavioral strategies have not been fully established. In this study, we developed an inverse reinforcement-learning (IRL) framework to identify an animal's behavioral strategy from behavioral time-series data. As a particular target, we applied this framework to thermotactic behavior in C.elegans. After identifying the behavioral strategy dependent on thermosensory state, we found it comprised mixture of two strategies: directed migration (DM) and isothermal migration (IM). First, the DM is a strategy that the worms efficiently reach to specific temperature, which not only explained observation that the worms migrate toward the cultivated temperature, but also clarifies how the worms control thermosensory state through the 10 migration. Second, the IM is a strategy that the worms track along a constant temperature, which reflects isothermal tracking well observed in previous studies. By further applying our method to starved worm and thermosensory neuron-deficient worms, we identified the neural basis underlying the strategies. Consequently, we interpreted behavioral strategies in terms of control theory. Therefore, this study validates and presents a novel approach that should propel the development of new, more effective experiments to identify behavioral strategies and decision-making in animals. Author SummaryUnderstanding animal decision-making has been a fundamental problem in neuroscience and behavioral 20 ecology. Many studies analyze actions that represent decision-making in behavioral tasks, in which rewards are artificially designed with specific objectives. However, it is impossible to extend this artificially designed experiment to a natural environment, because in a natural environment, the rewards for freely-behaving animals cannot be clearly defined. To this end, we must reverse the current paradigm so that rewards are identified from behavioral data. Here, we propose a new reverse-engineering approach (inverse reinforcement learning) that can estimate a behavioral strategy from time-series data of freely-behaving animals. By applying this technique with thermotaxis in C. elegans, we successfully identified the rewardbased behavioral strategy. 30not peer-reviewed)

Circuit Degeneracy Facilitates Robustness and Flexibility of Navigation Behavior in C.elegans

“…By contrast, the sub-circuits mediating shallow turns and omega turns were less defined (Figures 4F and S7B), suggesting that other classifications of turns might be needed to fully elucidate their underlying circuitry as some possibilities have been previously proposed (Broekmans et al, 2016; Kim et al, 2011). Also, non-rule based classifications of behaviors (Brown et al, 2013; Yamaguchi et al, 2018), especially the description of the state of the animal in shape space (Stephens et al, 2008), are reported to be successful in assessing the impact of cell-ablations (Hums et al, 2016; Yan et al, 2017) and might enable the definition of further sub-circuits.…”

“…Behavioral recordings were performed using a Multi-Worm Tracker (Swierczek et al, 2011; Yamaguchi et al, 2018) with a CMOS sensor Camera Link Camera (8 bits, 4,096 × 3,072 pixels; CSC12M25BMP19-01B, Toshiba-Teli), a lens adaptor (F-TAR2), a Line-Scan Lens (35mm, f/2.8; YF3528, PENTAX), and a PCIe-1433 camera-link frame grabber (781169-01, National Instruments). The camera was mounted at a distance above the assay plate resulting in an image with 33.2 μm per pixel.…”

Quantitative approaches to energy and glucose homeostasis: machine learning and modelling for precision understanding and prediction

“…Once this reward function has been learnt, the model can then be used to reproduce behaviour similar to that of the system being modelled. This has been successfully applied to thermotactic behaviour in Caenorhabditis elegans [ 113 ] (electronic supplementary material, S4.8) and is likely to be applicable to other homeostatic behaviours such as feeding. Applying inverse reinforcement learning to neuronal firing data from modern imaging techniques [ 114 ] could provide a natural interpretation of the inferred reward function and way to integrate results such as the negative valence of AGRP neuronal activation [ 115 ].…”