MQA: Answering the Question via Robotic Manipulation

Deng, Yuhong; Guo, Di; Guo, Xijuan; Zhang, Naifu; Liu, Huaping; Sun, Fuchun

doi:10.15607/rss.2021.xvii.044

Cited by 12 publications

(9 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deng et al provided a description of an alternative method to retrieve a target object based on the question-and-answer (QA) policy using the manipulation question-answering (MQA) method, in which the robot would employ manipulation actions to change the environment in order to answer a question. Deng et al [ 99 ] have a QA module and a manipulation module in their framework. In the QA module, the visual question-answering task is used.…”

Section: Critical Reviewmentioning

confidence: 99%

“…Furthermore, despite the complexity of a scene, no mutual support nor occlusion exists between them [ 107 ]. The approaches in [ 96 , 97 , 99 ] have different degrees of success; nevertheless, due to the high-dimensional and under-actuated issues and the uncertainty of real-world physics, the complex scenarios involved in this problem remain challenging for autonomous systems. To mitigate these issues, approaches utilizing randomized planning have been proposed.…”

Section: Critical Reviewmentioning

confidence: 99%

“…However, they do not consider the surrounding objects while attempting to reach the target object. In this instance, the value function is learned from predefined features, which limits the framework’s applicability to certain single-shaped objects, as described in [ 96 , 98 , 99 , 100 ]. In addition, those methods presupposed a predefined collection of geometric descriptions of real-world objects and used Cartesian coordinates to express the state.…”

Section: Critical Reviewmentioning

confidence: 99%

See 2 more Smart Citations

Review of Learning-Based Robotic Manipulation in Cluttered Environments

Mohammed

Kwek

Chua

et al. 2022

Sensors

View full text Add to dashboard Cite

Robotic manipulation refers to how robots intelligently interact with the objects in their surroundings, such as grasping and carrying an object from one place to another. Dexterous manipulating skills enable robots to assist humans in accomplishing various tasks that might be too dangerous or difficult to do. This requires robots to intelligently plan and control the actions of their hands and arms. Object manipulation is a vital skill in several robotic tasks. However, it poses a challenge to robotics. The motivation behind this review paper is to review and analyze the most relevant studies on learning-based object manipulation in clutter. Unlike other reviews, this review paper provides valuable insights into the manipulation of objects using deep reinforcement learning (deep RL) in dense clutter. Various studies are examined by surveying existing literature and investigating various aspects, namely, the intended applications, the techniques applied, the challenges faced by researchers, and the recommendations adopted to overcome these obstacles. In this review, we divide deep RL-based robotic manipulation tasks in cluttered environments into three categories, namely, object removal, assembly and rearrangement, and object retrieval and singulation tasks. We then discuss the challenges and potential prospects of object manipulation in clutter. The findings of this review are intended to assist in establishing important guidelines and directions for academics and researchers in the future.

show abstract

Section: Critical Reviewmentioning

confidence: 99%

Section: Critical Reviewmentioning

confidence: 99%

Section: Critical Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Review of Learning-Based Robotic Manipulation in Cluttered Environments

Mohammed

Kwek

Chua

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Zheng et al propose a compositional benchmark framework for Visual-Language Manipulation, VLM where a robot need to conduct a series of expected manipulations with human language instruction and egocentric vision [17]. Deng et al introduce robotic manipulation ability and propose a novel task called Manipulation Question Answering (MQA) [18], where the robot is required to find the answer by actively interacting with the environment via manipulation, rather than simply doing some simple choices for interaction.…”

Section: A Manipulation Task In Cluttered Environmentmentioning

confidence: 99%

“…Answering (MQA) task and build a benchmark MQA dataset [18]. The MQA task poses several new challenges.…”

Section: Deng Et Al Formulated a Novel Manipulation Questionmentioning

confidence: 99%

Scene Graph for Embodied Exploration in Cluttered Scenario

Deng¹,

Qie²,

Guo³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The ability to handle objects in cluttered environment has been long anticipated by robotic community. However, most of works merely focus on manipulation instead of rendering hidden semantic information in cluttered objects. In this work, we introduce the scene graph for embodied exploration in cluttered scenarios to solve this problem. To validate our method in cluttered scenario, we adopt the Manipulation Question Answering (MQA) tasks as our test benchmark, which requires an embodied robot to have the active exploration ability and semantic understanding ability of vision and language. As a general solution framework to the task, we propose an imitation learning method to generate manipulations for exploration. Meanwhile, a VQA model based on dynamic scene graph is adopted to comprehend a series of RGB frames from wrist camera of manipulator along with every step of manipulation is conducted to answer questions in our framework. The experiments on of MQA dataset with different interaction requirements demonstrate that our proposed framework is effective for MQA task a representative of tasks in cluttered scenario.

show abstract

Survey on reinforcement learning for language processing

et al. 2022

View full text Add to dashboard Cite

In recent years some researchers have explored the use of reinforcement learning (RL) algorithms as key components in the solution of various natural language processing (NLP) tasks. For instance, some of these algorithms leveraging deep neural learning have found their way into conversational systems. This paper reviews the state of the art of RL methods for their possible use for different problems of NLP, focusing primarily on conversational systems, mainly due to their growing relevance. We provide detailed descriptions of the problems as well as discussions of why RL is well-suited to solve them. Also, we analyze the advantages and limitations of these methods. Finally, we elaborate on promising research directions in NLP that might benefit from RL.

show abstract

MQA: Answering the Question via Robotic Manipulation

Cited by 12 publications

References 22 publications

Review of Learning-Based Robotic Manipulation in Cluttered Environments

Review of Learning-Based Robotic Manipulation in Cluttered Environments

Scene Graph for Embodied Exploration in Cluttered Scenario

Survey on reinforcement learning for language processing

Contact Info

Product

Resources

About