Does computer vision matter for action?

Abstract

Controlled experiments indicate that explicit intermediate representations help action.https://doi.org/10.1126/scirobotics.aaw6661Biological vision systems evolved to support action in physical environments [1, 2]. Action is also a driving inspiration for computer vision research. Problems in computer vision are often motivated by their relevance to robotics and their prospective utility for systems that move and act in the physical world. In contrast, a recent stream of research at the intersection of machine learning and robotics demonstrates that models can be trained to map raw visual input directly to action [3][4][5][6]. These models bypass explicit computer vision entirely. They do not incorporate modules that perform recognition, depth estimation, optical flow, or other explicit vision tasks. The underlying assumption is that perceptual capabilities will arise in the model as needed, as a result of training for specific motor tasks. This is a compelling hypothesis that, if taken at face value, appears to obsolete much computer vision research. If any robotic system can be trained directly for the task at hand, with only raw images as input and no explicit vision modules, what is the utility of further perfecting models for semantic segmentation, depth estimation, optical flow, and other vision tasks?We report controlled experiments that assess whether specific vision capabilities are useful in mobile sensorimotor systems that act in complex three-dimensional environments. To conduct these experiments, we use realistic three-dimensional simulations derived from immersive computer games. We instrument the game engines to support controlled execution of specific scenarios that simulate tasks such as driving a car, traversing a trail in rough terrain, and battling opponents in a labyrinth. We then train sensorimotor systems equipped with different vision modules and measure their performance on these tasks.Our baselines are end-to-end pixels-to-actions models that are trained directly for the task at hand. These models do not rely on any explicit computer vision modules and embody the assumption that perceptual capabilities will arise as needed, in the course of learning to perform the requisite sensorimotor task. To these we compare models that receive as additional input the kinds of representations that are studied in computer vision research, such as semantic label maps, depth maps, and optical flow. We can therefore assess whether representations produced in computer vision are useful for sensorimotor challenges. In effect, we ask: What if a given vision task was solved? Would this matter for learning to act in complex three-dimensional environments?Our first finding is that computer vision does matter. When agents are provided with representations studied in computer vision, they achieve higher performance in sensorimotor tasks. The effect is significant and is consistent across simulation platforms and tasks.We then examine in finer granularity how useful specific computer vision capabilities are in th...

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

“…D EPTH is among the most useful intermediate representations for action in physical environments [1]. Despite its utility, monocular depth estimation remains a challenging problem that is heavily underconstrained.…”

ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding

“…Oxford Robotcar [27] was the first real-world large-scale dataset in which adverse visual conditions such as nighttime, rain and snow were significantly represented, but it did not feature semantic annotations. While more recent large-scale sets [2,30] that cover adverse conditions, such as Waymo Open [42] and nuScenes [3], include bounding boxes, they still lack dense pixel-level semantic annotations, which are vital for real-world autonomous agents [63]. BDD100K [55] is the only exception to this rule, with ca.…”

Learning to See before Learning to Act: Visual Pre-training for Manipulation

“…The field of unsupervised learning has explored different ways to learn state representations [27,28,29] for policy learning. Recently, several works [30,31,32] have studied the benefits of combining various mid-level visual representations for reinforcement learning. Different from previous works, we 1) demonstrate real-world results on manipulation while previous works present simulation results on navigation, 2) highlight the importance of using a pre-trained model for exploration, and 3) propose an initialization strategy without the need of data collection at target domain.…”