Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems

Eppner, Clemens; Höfer, Sebastian; Jonschkowski, Rico; Martí­n-Martí­n, Roberto; Sieverling, Arne; Wall, Vincent; Brock, Oliver

doi:10.15607/rss.2016.xii.036

Cited by 93 publications

(29 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Visual grounding of referring expressions is closely related to object recognition. In robotics, object recognition is often treated as a classification task, with a predefined set of object category labels (Eppner et al, 2016; Krishnamurthy and Kollar, 2013; Pangercic et al, 2012). These methods restrict themselves to tasks covered by predefined visual concepts and simple language expression templates.…”

Section: Related Workmentioning

confidence: 99%

INGRESS: Interactive visual grounding of referring expressions

Shridhar

Mittal

Hsu

2020

The International Journal of Robotics Research

View full text Add to dashboard Cite

This article presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The key question here is to ground referring expressions: understand expressions about objects and their relationships from image and natural language inputs. INGRESS allows unconstrained object categories and rich language expressions. Further, it asks questions to clarify ambiguous referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural-network model for grounding. The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expressions, and identifies a set of candidate objects. The second stage uses another neural network to examine all pairwise relations between the candidates and infers the most likely referred objects. The same neural networks are used for both grounding and question generation for disambiguation. Experiments show that INGRESS outperformed a state-of-the-art method on the RefCOCO dataset and in robot experiments with humans. The INGRESS source code is available at https://github.com/MohitShridhar/ingress .

show abstract

Section: Related Workmentioning

confidence: 99%

INGRESS: Interactive visual grounding of referring expressions

Shridhar

Mittal

Hsu

2020

The International Journal of Robotics Research

View full text Add to dashboard Cite

show abstract

“…Using fuzzy logic to address the grasping problem is one way to reduce the gap between conventional and intelligent control due to the uncertainties and complex mathematical solutions involved in the task [31]. In a recent edition of the Amazon picking challenge, the authors of [32] described problems with an open-loop control solution during pick and place tasks; therefore, autonomous, stable grasping is an essential aspect of in-hand manipulation. The authors of [33] presented a closed loop control using tactile sensing for a task of removing a book from a bookshelf when maximization of the contact surface is required.…”

Section: Literature Reviewmentioning

confidence: 99%

Estimating the Orientation of Objects from Tactile Sensing Data Using Machine Learning Methods and Visual Frames of Reference

Fonseca

Oliveira

Petriu

2019

Sensors

View full text Add to dashboard Cite

Underactuated hands are useful tools for robotic in-hand manipulation tasks due to their capability to seamlessly adapt to unknown objects. To enable robots using such hands to achieve and maintain stable grasping conditions even under external disturbances while keeping track of an in-hand object’s state requires learning object-tactile sensing data relationships. The human somatosensory system combines visual and tactile sensing information in their “What and Where” subsystem to achieve high levels of manipulation skills. The present paper proposes an approach for estimating the pose of in-hand objects combining tactile sensing data and visual frames of reference like the human “What and Where” subsystem. The system proposed here uses machine learning methods to estimate the orientation of in-hand objects from the data gathered by tactile sensors mounted on the phalanges of underactuated fingers. While tactile sensing provides local information about objects during in-hand manipulation, a vision system generates egocentric and allocentric frames of reference. A dual fuzzy logic controller was developed to achieve and sustain stable grasping conditions autonomously while forces were applied to in-hand objects to expose the system to different object configurations. Two sets of experiments were used to explore the system capabilities. On the first set, external forces changed the orientation of objects while the fuzzy controller kept objects in-hand for tactile and visual data collection for five machine learning estimators. Among these estimators, the ridge regressor achieved an average mean squared error of 0.077 ∘ . On the second set of experiments, one of the underactuated fingers performed open-loop object rotations and data recorded were supplied to the same set of estimators. In this scenario, the Multilayer perceptron (MLP) neural network achieved the lowest mean squared error of 0.067 ∘ .

show abstract

“…Eppner et al (2016) described their winning system for APC 2015. Mechanically, the robot consists of a mobile base and a 7-DOF arm to reach all shelf bins comfortably.…”

Section: Related Workmentioning

confidence: 99%

RGB-D object detection and semantic segmentation for autonomous manipulation in clutter

Schwarz

Milan

Periyasamy

et al. 2017

The International Journal of Robotics Research

144

View full text Add to dashboard Cite

Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic data sets possible. We evaluate our approach on two challenging data sets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task, and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.

show abstract

Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems

Cited by 93 publications

References 43 publications

INGRESS: Interactive visual grounding of referring expressions

INGRESS: Interactive visual grounding of referring expressions

Estimating the Orientation of Objects from Tactile Sensing Data Using Machine Learning Methods and Visual Frames of Reference

RGB-D object detection and semantic segmentation for autonomous manipulation in clutter

Contact Info

Product

Resources

About