Virtual reality for animal navigation with camera-based optical flow tracking

Abstract

Background Virtual reality combined with spherical treadmills is used across species for studying neural circuits underlying navigation.New Method We developed an optical flow-based method for tracking treadmil ball motion in real-time using a single high-resolution camera.Results Tracking accuracy and timing were determined using calibration data. Ball tracking was performed at 500 Hz and integrated with an open source game engine for virtual reality projection. The projection was updated at 120 Hz with a latency with respect to ball motion of 30 ± 8 ms.Comparison with Existing Method(s) Optical flow based tracking of treadmill motion is typically achieved using optical mice. The camera-based optical flow tracking system developed here is based on o↵-the-shelf components and o↵ers control over the image acquisition and processing parameters. This results in flexibility with respect to tracking conditions -such as ball surface texture, lighting conditions, or ball size -as well as camera alignment and calibration.Conclusions A fast system for rotational ball motion tracking suitable for virtual reality animal behavior across di↵erent scales was developed and characterized.Virtual reality (VR) is used across species for studying neural circuits 2 underlying behavior [1]. In many implementations, animals navigate through 3 virtual realities on a spherical treadmill -a ball which can be freely rotated 4 around its center of mass [1, 2, 3, 4, 5]. 5Tracking of ball rotation is typically accomplished using optical mice 6 which are based on low-resolution, high-speed cameras integrated with a light 7 source for measuring displacements when moving across a surface. Movement 8 across the surface results in optical flow -the displacement of features across 9 the camera sensor. Such features can for example be speckle-like reflections 10 from surface roughness; comparing these speckle images between di↵erent 11 frames then allows computing the displacement using hardware-integrated 12 image processing. 13Optical mice come however with some limitations for measuring ball ro-14 tation. First, a single optical mouse measures displacements only in two 15 directions and therefore two mice are required for tracking all three degrees 16 of freedom of ball motion. Secondly, limited or no control over the onboard 17 processing algorithms as well as camera settings requires careful calibration. 18In particular, if the mouse sensors can't be placed in direct proximity of 19 the ball surface, accurate alignment of the two sensors as well as calibration 20 with respect to surface properties and lighting conditions is necessary [5]. 21 As an approach that overcomes some of these limitations, real-time track-22 ing was developed with a single high-resolution camera for situations where a 23 uniquely patterned ball can be used [6]. In that case, ball orientation was cal-24 culated by matching each recorded frame to a map of the entire ball surface 25 pattern. Using a high-resolution cameras allows control over all recording 26 parameter...

Virtual reality for animal navigation with camera-based optical flow tracking

“…The copyright holder for this preprint . http://dx.doi.org/10.1101/579813 doi: bioRxiv preprint first posted online Mar. 16, 2019; resolution camera was developed with short latencies suitable for VR exper-324 iments at di↵erent scales.…”

“…The copyright holder for this preprint . http://dx.doi.org/10.1101/579813 doi: bioRxiv preprint first posted online Mar. 16, 2019; to the ROI circle at this point, directed counterclockwise).…”

“…The copyright holder for this preprint . http://dx.doi.org/10.1101/579813 doi: bioRxiv preprint first posted online Mar. 16, 2019; Similarly, the turning in the virtual environment is calculated from the ball displacement as a dot product with ballXYZtoArenaYaw: arena yaw displacement = ballXYZtoArenaYaw · ball displacement.…”

Dynamics of sleep and feeding homeostasis inDrosophilaglia and neurons

“…A side view of the fly was recorded with a camera at 10 Hz with a resolution of 640 × 480 during the whole experiment, which was used later for behavior classification. The VR, implemented as described 27,73 displayed a dark stripe on a bright background using a blue laser (488 nm), which was switched off 14/37 during the night for 12 hours. The VR night and day cycle was synchronized to the day and night cycle used in the fly incubator before the experiment.…”

Optical Flow Based Object Movement Tracking

“…Optical flow has also been combined with other mathematical models such as Markov models as discussed in [9] to achieve better results with respect to better detection and tracking. Optical flow can also be used to track the movement of persons and animals as discussed in [10].…”