Animals in groups touch each other, move in paths that cross, and interact in complex ways. Current video tracking methods sometimes switch identities of unmarked individuals during these interactions. These errors propagate and result in random assignments after a few minutes unless manually corrected. We present idTracker, a multitracking algorithm that extracts a characteristic fingerprint from each animal in a video recording of a group. It then uses these fingerprints to identify every individual throughout the video. Tracking by identification prevents propagation of errors, and the correct identities can be maintained indefinitely. idTracker distinguishes animals even when humans cannot, such as for size-matched siblings, and reidentifies animals after they temporarily disappear from view or across different videos. It is robust, easy to use and general. We tested it on fish (Danio rerio and Oryzias latipes), flies (Drosophila melanogaster), ants (Messor structor) and mice (Mus musculus).

No abstract

The striking patterns of collective animal behavior, including ant trails, bird flocks, and fish schools, can result from local interactions among animals without centralized control. Several of these rules of interaction have been proposed, but it has proven difficult to discriminate which ones are implemented in nature. As a method to better discriminate among interaction rules, we propose to follow the slow birth of a rule of interaction during animal development. Specifically, we followed the development of zebrafish, Danio rerio, and found that larvae turn toward each other from 7 days postfertilization and increase the intensity of interactions until 3 weeks. This developmental dataset allows testing the parameter-free predictions of a simple rule in which animals attract each other part of the time, with attraction defined as turning toward another animal chosen at random. This rule makes each individual likely move to a high density of conspecifics, and moving groups naturally emerge. Development of attraction strength corresponds to an increase in the time spent in attraction behavior. Adults were found to follow the same attraction rule, suggesting a potential significance for adults of other species.collective behavior | interaction rule | zebrafish | development | shoaling C ollective animal behavior is studied with increasing detail in natural habitats (1-6) and laboratory conditions (7-14). Local interactions among animals can, in many cases, explain these patterns of collective behavior, and a variety of interaction rules have been proposed (7,8,11,(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27).One of the technical problems in discriminating among possible interaction rules is the difficulty of obtaining high-quality experimental data (25). We reasoned that the ontogeny of attraction behavior offers a unique opportunity to obtain a large highquality dataset. This dataset should constrain the space of possible models to those that can explain interactions every day during development.We turned to zebrafish, Danio rerio, a species in which larvae seem not to attract each other after hatching but that develop shoaling and schooling behavior during the first month of development (12,14,(28)(29)(30)(31)(32)(33). Our choice was based on our previous work in the adult suggesting a simplicity of the rules compared with other species (14).In this work, we follow the formation of attraction behavior during the ontogeny of collective behavior in zebrafish. We used our newly developed tracking system of animals in groups, idTracker (34), in a total of 524 videos for the study of development and 25 videos for adults. We found that zebrafish are very weakly attracted to each other by 7 days postfertilization (dpf), and the attraction gets stronger each day during development. By 9 dpf, larvae are likely found close to each other, and, by 15 dpf, it is common to see animals moving in groups. Analysis and modeling of the developmental dataset point to attraction as turning toward a randomly chosen consp...

A variety of simple models has been proposed to understand the collective motion of animals. These models can be insightful but may lack important elements necessary to predict the motion of each individual in the collective. Adding more detail increases predictability but can make models too complex to be insightful. Here we report that deep attention networks can obtain a model of collective behavior that is simultaneously predictive and insightful thanks to an organization in modules. When using simulated trajectories, the model recovers the ground-truth interaction rule used to generate them, as well as the number of interacting neighbours. For experimental trajectories of large groups of 60-100 zebrafish, Danio rerio, the model obtains that interactions between pairs can approximately be described as repulsive, attractive or as alignment, but only when moving slowly. At high velocities, interactions correspond only to alignment or alignment mixed with repulsion at close distances. The model also shows that each zebrafish decides where to move by aggregating information from the group as a weighted average over neighbours. Weights are higher for neighbours that are close, in a collision path or moving faster in frontal and lateral locations. The network also extracts that the number of interacting individuals is dynamical and typically in the range 8–22, with 1–10 more important ones. Our results suggest that each animal decides by dynamically selecting information from the collective.

Our understanding of collective animal behavior is limited by our ability to track each of the individuals. We describe an algorithm and software, idtracker.ai, that extracts from video all trajectories with correct identities at a high accuracy for collectives of up to 100 individuals. It uses two deep networks, one detecting when animals touch or cross and another one for animal identi cation, trained adaptively to conditions and di culty of the video.Obtaining animal trajectories from a video faces the problem of how to track animals with correct identities after they touch, cross or they are occluded by environmental features. To bypass this problem, we proposed in idTracker the idea of tracking by identi cation of each individual using a set of reference images obtained from the video [1]. idTracker and further developments in animal identi cation algorithms [2-6] can work for small groups of 2-15 individuals. In larger groups, they only work for particular videos with few animal crossings [7] or with few crossings of particular species-speci c features [5].Here we present idtracker.ai, a system to track all individuals in small or large collectives (up to 100 individuals) at a high identi cation accuracy, often of > 99.9%. The method is species-agnostic and we have tested it in small and large collectives of zebra sh, Danio rerio and ies, Drosophila melanogaster. Code, quickstart guide and data used are provided (see Methods), and Supplementary Text describes algorithms and gives pseudocode. A graphical user interface walks users through tracking, exploration and validation (Fig. 1a).Similar to idTracker [1], but with di erent algorithms, idtracker.ai identi es animals using their visual features. In idtracker.ai, animal identi cation is done adapting deep learning [8][9][10] to work in videos of animal collectives thanks to speci c training protocols. In brief, it consists of a series of processing steps summarized in Fig. 1b. After image preprocessing, the rst deep network nds when animals are touching or crossing. Then the system uses the images between these detected to train a second deep network for animal identi cation. The system rst assumes that a single portion of video when animals do not touch or cross has enough images to properly train the identi cation network (Protocol 1). However, animals touch or cross often and this portion is then typically very short, making the system estimate that identi cation quality is too low. If this happens, two extra 1 . CC-BY-NC 4.0 International license not peer-reviewed) is the author/funder. It is made available under a

We investigate the transient times for the onset of control of steady states by timedelayed feedback. The optimization of control by minimising the transient time before control becomes effective is discussed analytically and numerically, and the competing influences of local and global features are elaborated. We derive an algebraic scaling of the transient time and confirm our findings by numerical simulations in dependence on feedback gain and time delay.

A variety of simple models has been proposed to understand the collective motion of animals. These models can be insightful but lack important elements necessary to predict the motion of each individual in the collective. Adding more detail increases predictability but can make models too complex to be insightful. Here we report that deep attention networks can obtain in a data-driven way a model of collective behavior that is simultaneously predictive and insightful thanks to an organization in modules. The model obtains that interactions between two zebrafish, Danio rerio, in a large groups of 60-100, can be approximately be described as repulsive, attractive or as alignment, but only when moving slowly. At high velocities, interactions correspond only to alignment or alignment mixed with repulsion at close distances. The model also shows that each zebrafish decides where to move by aggregating information from the group as a weighted average over neighbours. Weights are higher for neighbours that are close, in a collision path or moving faster in frontal and lateral locations. These weights effectively select 5 relevant neighbours on average, but this number is dynamical, changing between a single neighbour to up to 12, often in less than a second. Our results suggest that each animal in a group decides by dynamically selecting information from the group. Highlights• At 30 days postfertilization, zebrafish, Danio rerio, can move in very cohesive and predictable large groups• Deep attention networks obtain a predictive and understadable model of collective motion• When moving slowly, interations between pairs of zebrafish have clear components of repulsion, attraction and alignment• When moving fast, interactions correspond to alignment and a mixture of alignment and repulsion at close distances • Zebrafish turn left or right depending on a weighted average of interaction information with other fish, with weights higher for close fish, those in a collision path or those moving fast in front or to the sides • Aggregation is dynamical, oscillating between 1 and 12 neighbouring fish, with 5 on average