Virtual reality for animal navigation with camera-based optical flow tracking

Abstract

Background Virtual reality combined with spherical treadmills is used across species for studying neural circuits underlying navigation.New Method We developed an optical flow-based method for tracking treadmil ball motion in real-time using a single high-resolution camera.Results Tracking accuracy and timing were determined using calibration data. Ball tracking was performed at 500 Hz and integrated with an open source game engine for virtual reality projection. The projection was updated at 120 Hz with a latency with respect to ball motion of 30 ± 8 ms.Comparison with Existing Method(s) Optical flow based tracking of treadmill motion is typically achieved using optical mice. The camera-based optical flow tracking system developed here is based on o↵-the-shelf components and o↵ers control over the image acquisition and processing parameters. This results in flexibility with respect to tracking conditions -such as ball surface texture, lighting conditions, or ball size -as well as camera alignment and calibration.Conclusions A fast system for rotational ball motion tracking suitable for virtual reality animal behavior across di↵erent scales was developed and characterized. 2 underlying behavior [1]. In many implementations, animals navigate through 3 virtual realities on a spherical treadmill -a ball which can be freely rotated 4 around its center of mass [1, 2, 3, 4, 5]. 5 Tracking of ball rotation is typically accomplished using optical mice 6 which are based on low-resolution, high-speed cameras integrated with a light 7 source for measuring displacements when moving across a surface. Movement 8 across the surface results in optical flow -the displacement of features across 9 the camera sensor. Such features can for example be speckle-like reflections 10 from surface roughness; comparing these speckle images between di↵erent 11 frames then allows computing the displacement using hardware-integrated 12 image processing. 13 Optical mice come however with some limitations for measuring ball ro-14 tation. First, a single optical mouse measures displacements only in two 15 directions and therefore two mice are required for tracking all three degrees 16 of freedom of ball motion. Secondly, limited or no control over the onboard 17 processing algorithms as well as camera settings requires careful calibration. 18 In particular, if the mouse sensors can't be placed in direct proximity of 19 the ball surface, accurate alignment of the two sensors as well as calibration 20 with respect to surface properties and lighting conditions is necessary [5]. 21As an approach that overcomes some of these limitations, real-time track-22 ing was developed with a single high-resolution camera for situations where a 23 uniquely patterned ball can be used [6]. In that case, ball orientation was cal-24 culated by matching each recorded frame to a map of the entire ball surface 25 pattern. Using a high-resolution cameras allows control over all recording 26 parameters and o↵ers the freedom to choose a custom algorithms and test 27 its per...

Automated long-term two-photon imaging in head-fixed walkingDrosophila

“…Fly behavior was extracted from ball movements recorded over the entire experiment, also in between imaging trials. Ball tracking 15 provided the angular displacement of the ball along the three axes of rotation, ( b x ( k ), b y ( k ), b z ( k )), at a rate of 300 Hz for time stamp k . We first computed the linear velocity of the fly in mm/sec as: where R b = 3 mm is the radius of the ball and Δ t = 1/300 sec is the time between consecutive frames.…”

An Inexpensive, High-Precision, Modular Spherical Treadmill Setup Optimized for Drosophila Experiments

“…In tethered walking experiments the flies are fixed in space, however their intended locomotion, as if walking on an infinite virtual plane, can be estimated from the rotation of the sphere they are turning. Several methods have been developed for measuring relative rotations of the ball, for example through optical mice sensors or via optical flow calculated with camera-based systems (Lott et al, 2007;Seelig et al, 2010;Vishniakou et al, 2019). Under ideal circumstances, these relative measurements can be calibrated for excellent accuracy, but these systems can be quite sensitive to the uniformity of the lighting and focus of the sensors, etc.…”

Accurate Realtime Motion Estimation Using Optical Flow on an Embedded System

“…Many approaches have been based on optical flow and suggested improvements in computer vision application using motion estimation. Ivan Vishniakou et al [17] developed an optical flow estimation method for tracking treadmill ball motion in real time using a single high-resolution camera. Using calibration data, they determined the tracking accuracy and timing.…”