Residual learning-based two-stream network for RGB-T object tracking

Chen, Yili; Wan, Minjie; Xu, Yunkai; Zhang, Xiaojie; Chen, Qian; Gu, Guohua

doi:10.1117/1.jei.31.6.063041

Cited by 4 publications

(18 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the teacher policy is not recurrent, ruling out this possibility. The unexpected observation of reorientation without knowledge of shape was made by earlier work in the context of a reorientation system in simulation (7). However, because real-world results were not demonstrated, it remained unclear whether such an observation was an artifact of the simulator or the property of the reorientation problem.…”

Section: Discussionmentioning

confidence: 96%

“…Because the controllers discussed above were trained with a supporting surface, when the supporting surface was removed, the manipulator consistently dropped the object, resulting in failures. Prior work used a specialized training procedure of configuring the object in a good pose at the start of each training episode and a manually designed gravity curriculum (7) to learn in-air (without supporting surface) reorientation controllers. Consequently, it was necessary to train separate controllers for reorientation with a supporting surface and in the air.…”

Section: Toward Object Reorientation In Airmentioning

confidence: 99%

“…In-hand object reorientation, illustrated in Fig. 1, is a specific dexterous manipulation problem where the goal is to manipulate a handheld object from an arbitrary initial orientation to an arbitrary target orientation (1)(2)(3)(4)(5)(6)(7). Object reorientation occupies a special place in manipulation because it is a precursor to flexible tool use.…”

Section: Introductionmentioning

confidence: 99%

“…Because of challenges in perception and control, no prior work has demonstrated a real-world-ready reorientation system. Although controlling directly from perception is hard, given the full low-dimensional representation of relevant state information such as the object's position, velocity, pose, and manipulator's proprioceptive state, it is possible to build a controller using deep reinforcement learning (RL) that successfully reorients diverse objects in simulation (7). RL effectively leverages large amounts of interaction data to find an approximate solution to the computationally challenging optimization problem of solving for reorientation.…”

Section: Introductionmentioning

confidence: 99%

“…If the optimal actions were known in advance, it would be simpler to train a model that predicts these actions from visual inputs (supervised learning). Such a two-stage teacher-student training paradigm, where first a control policy is trained via RL with full state information (teacher) and then a second student policy is trained via supervised learning to mimic the teacher, has been successfully used for several applications (7,(27)(28)(29)(30). We found that the major roadblock in learning a visual policy that works across diverse objects is the slow speed of rendering in simulation, which resulted in training times of more than 20 days with our compute resources.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Visual dexterity: In-hand reorientation of novel and complex object shapes

Chen,

Tippur,

et al. 2023

Sci. Robot.

View full text Add to dashboard Cite

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments, which remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following conditions: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasi-static manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints that make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real time, with the median reorientation time being close to 7 seconds. The controller was trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only used open-source components that cost less than 5000 dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56% of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23°) 75% of the time.

show abstract

Section: Discussionmentioning

confidence: 96%

Section: Toward Object Reorientation In Airmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Visual dexterity: In-hand reorientation of novel and complex object shapes

Chen,

Tippur,

et al. 2023

Sci. Robot.

View full text Add to dashboard Cite

show abstract

Geometric Fabrics: Generalizing Classical Mechanics to Capture the Physics of Behavior

Wyk

Xie

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

An Evaluation of AI Integrated Education in Graduate School of Education Using the CIPP Model

Kim,

Han

2023

KAEIM

View full text Add to dashboard Cite

Fig. 1: Illustration of our tasks. The first column shows the initial robot and object poses and the desired object poses in the dark. The third column shows a subset of objects we manipulate in each domain. The middle column shows the manipulation motions. First row: the card is too flat to be grasped, so the robot must use dragging and re-orientation. Second row: the box is too large to be grasped, so the robot must push it to the bump, tumble it over, and re-orient it. Third row: the wall to the object's right is blocking all feasible grasps. The robot must first lift it up against the wall, drag it to the top, and then finally give a little push to move it to the target location. In the last two tasks, notice how the robot must both overcome and exploit the environmental contact to manipulate the object.

show abstract

Residual learning-based two-stream network for RGB-T object tracking

Cited by 4 publications

References 28 publications

Visual dexterity: In-hand reorientation of novel and complex object shapes

Visual dexterity: In-hand reorientation of novel and complex object shapes

Geometric Fabrics: Generalizing Classical Mechanics to Capture the Physics of Behavior

An Evaluation of AI Integrated Education in Graduate School of Education Using the CIPP Model

Contact Info

Product

Resources

About