Quantitative comparison ofDrosophilabehavior annotations by human observers and a machine learning algorithm

Abstract

Automated quantification of behavior is increasingly prevalent in neuroscience research. Human judgments can influence machine-learning-based behavior classification at multiple steps in the process, for both supervised and unsupervised approaches. Such steps include the design of the algorithm for machine learning, the methods used for animal tracking, the choice of training images, and the benchmarking of classification outcomes. However, how these design choices contribute to the interpretation of automated behavioral classifications has not been extensively characterized. Here, we quantify the effects of experimenter choices on the outputs of automated classifiers of Drosophila social behaviors. Drosophila behaviors contain a considerable degree of variability, which was reflected in the confidence levels associated with both human and computer classifications. We found that a diversity of sex combinations and tracking features was important for robust performance of the automated classifiers. In particular, features concerning the relative position of flies contained useful information for training a machine-learning algorithm. These observations shed light on the importance of human influence on tracking algorithms, the selection of training images, and the quality of annotated sample images used to benchmark the performance of a classifier (the 'ground truth'). Evaluation of these factors is necessary for researchers to accurately interpret behavioral data quantified by a machine-learning algorithm and to further improve automated classifications. Significance Statement:Accurate quantification of animal behaviors is fundamental to neuroscience. Here, we quantitatively assess how human choices influence the performance of automated classifiers trained by a machine-learning algorithm. We found that human decisions about the computational tracking method, the training images, and the images used for performance evaluation impact both the classifier outputs and how human observers interpret the results. These factors are sometimes overlooked but are critical, especially because animal behavior is itself inherently variable. Automated quantification of animal behavior is becoming increasingly prevalent: our results provide a model for bridging the gap between traditional human annotations and computer-based annotations. Systematic assessment of human choices is important for developing behavior classifiers that perform robustly in a variety of experimental conditions. 3

Improved accuracy for estrous cycle staging using supervised object detection

“…Since ODES was trained to make decisions in a manner similar to human judgment, it encounters the same problems with clumped cells that humans do. Additionally, since the model's training involves human input, any bias or inconsistency in cell annotations can directly influence the model's learning patterns [25].…”

“…For instance, if the annotations provided by human experts vary due to subjective interpretations of what constitutes a particular cell type or boundary, the model will learn these inconsistencies, potentially reducing its accuracy and generalizability. Additionally, human annotators can have varying levels of confidence and criteria in classification [25]. This variability can lead to a model that is uncertain or inconsistent in its classifications, mirroring the inconsistencies of its training data.…”