Time-Lapse Light Field Photography for Perceiving Non-Lambertian Scenes

Abstract

Abstract-Robust robotic perception and manipulation of household objects requires the ability to detect, localize and manipulate a wide variety of objects, which may be mirror reflective like polished metal, glossy like smooth plastic, or transparent like glass; for example, picking a metal fork out of a sink full of running water or screwing a metal nut onto a bolt. Existing perceptual approaches based on photographs only take into account the average intensity of light arriving at each pixel from one direction, which limits their ability to account for these non-Lambertian scenes. To address this problem, we demonstrate time-lapse light field photography with an eye-inhand camera of a manipulator robot. An eye-in-hand robot can capture both the intensity of rays, as in a conventional photograph, as well as the direction of the rays. We present a formal model for robotic light-field photography that fits into a probabilistic robotics framework. Using this model, we can synthesize orthographic photographs, remove specular highlights from those photographs, and perform 3D reconstruction with a monocular camera by finding approximate maximum-likelihood estimates. This information can be used to detect, localize and manipulate non-Lambertian objects in non-Lambertian scenes: our approach enables the Baxter robot to pick a shiny metal fork out of a sink filled with running water 24/25 times, as well as to localize objects well enough to screw a nut onto a quarter inch bolt. The techniques in this paper point the way toward new approaches to robotic perception that leverage a robot's ability to move its camera to infer the state of the external world.

SaccadeCam: Adaptive Visual Attention for Monocular Depth Sensing

“…Saccades, foveation and related ideas have been studied in robotics and active vision for many years [1,3,18,44,12,19,7]. In addition, foveal designs to enable high-quality imaging are also common [45,27,43,13].…”

GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter

“…The use of light field perception in robotics is still relatively new. Oberlin and Tellex [19] proposed a time-lapse light field capturing pipeline for static scenes by mounting a RGB camera on the end-effector of the robot and moving in a designed trajectory. Tsai et al [28] introduced a algorithm for distinguishing refracted and Lambertian features from light field image.…”

Plenoptic Monte Carlo Object Localization for Robot Grasping Under Layered Translucency

“…(1) Each surface point emits light rays ρ in each channel as a Gaussian over (r, g, b) with mean (µ r , µ g , µ b ) and variance (σ 2 r , σ 2 g , σ 2 b ) which means ρ = N (λ; µ c , σ 2 c ), c ∈ {r, g, b}, as similarly assumed by Oberlin and Tellex [21]. Under constant lighting condition we assume every point in the scene shares the same variance for the same color channel which means σ c = σ c , c ∈ {r, g, b} for all points in the scene.…”

“…Light field photography offers considerable potential for robot perception in scenes with translucency. For example, Z Oberlin and Tellex [21] found that a high-resolution camera on the wrist of a robot manipulator can capture light fields for a static scene. By moving the robot end-effector in a designed trajectory, this time lapse approach to light field capture was demonstrated as capable of manipulating transparent and reflective objects.…”