Toward Affordance Detection and Ranking on Novel Objects for Real-World Robotic Manipulation

Chu, Fu-Jen; Xu, Ruinian; Seguin, Landan; Vela, Patricio A.

doi:10.1109/lra.2019.2930364

Cited by 26 publications

(18 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Action maps have been proposed as environment representations focusing on actions, which embed the action possibility in real space based on the history of human activities [14], [15]. In addition, another approach is to apply an object classification method based on the concept of affordance to associate the actions with objects [16], [17].…”

Section: B Environmental Representation Methodsmentioning

confidence: 99%

“…Therefore, the actions are associated with the object set based on the subsystem-level affordance, which is defined as the affordance considering the robot's subsystem configuration. In this study, we assume that robots can acquire prior knowledge about the affordances and actions, as shown in Tables 1 and 2 with the development of the affordance classification methods [16], [17] and the action understanding methods [21], [22], Then, the actions are associated with the recognized object sets based on the prior knowledge and the robot's subsystem configuration.…”

Section: Association Of Action With Objectmentioning

confidence: 99%

See 1 more Smart Citation

ASTRON: Action-Based Spatio-Temporal Robot Navigation

2021

View full text Add to dashboard Cite

To achieve the tasks provided by a user, it is necessary for robots to have a plan that fully exploits their functionalities in an environment. The objective of this study is to realize robot task planning in real space for effectively use of the robot's functions The plan is formed by deriving a feasible action sequence by interpreting the instructions within the scope of the action possibilities of the robots and the changes in them. In this paper, we first propose an action graph as a novel environmental representation approach to facilitate the understanding of the robot's action possibility in real space. In the action graph, the action possibility is represented by nodes, which describe the spatial position to perform each feasible action, and edges, which describe the feasible actions, based on the subsystem-level affordance and the arrangement of objects in the environment. We also propose an action-based spatio-temporal robot navigation (ASTRON), which focuses on robot navigation tasks. ASTRON enables the robots to determine a feasible action sequence that utilizes their functions by interpreting the instructions based on the action graph. The effectiveness of the proposed method was evaluated through simulations and actual machine experiments in a coffee shop environment. In the actual machine experiments, the proposed method was applied to robots with different subsystem configurations. The experimental results demonstrated that the proposed method could plan the feasible action sequence to complete the tasks by considering the environmental state and the subsystem configurations of the robot.

show abstract

Section: B Environmental Representation Methodsmentioning

confidence: 99%

Section: Association Of Action With Objectmentioning

confidence: 99%

ASTRON: Action-Based Spatio-Temporal Robot Navigation

2021

View full text Add to dashboard Cite

show abstract

“…Similarly, LR has been used to rank affordances of a group of novel objects to assist with manipulation tasks (Chu et al. 2019 ). Combined with deep learning, label ranking has been broadly used in text classification (Liu et al.…”

Section: Related Workmentioning

confidence: 99%

An adaptive decision-making system supported on user preference predictions for human–robot interactive communication

Maroto-Gómez

Castro‐González

Castillo

et al. 2022

User Model User-Adap Inter

View full text Add to dashboard Cite

Adapting to dynamic environments is essential for artificial agents, especially those aiming to communicate with people interactively. In this context, a social robot that adapts its behaviour to different users and proactively suggests their favourite activities may produce a more successful interaction. In this work, we describe how the autonomous decision-making system embedded in our social robot Mini can produce a personalised interactive communication experience by considering the preferences of the user the robot interacts with. We compared the performance of Top Label as Class and Ranking by Pairwise Comparison, two promising algorithms in the area, to find the one that best predicts the user preferences. Although both algorithms provide robust results in preference prediction, we decided to integrate Ranking by Pairwise Comparison since it provides better estimations. The method proposed in this contribution allows the autonomous decision-making system of the robot to work on different modes, balancing activity exploration with the selection of the favourite entertaining activities. The operation of the preference learning system is shown in three real case studies where the decision-making system works differently depending on the user the robot is facing. Then, we conducted a human–robot interaction experiment to investigate whether the robot users perceive the personalised selection of activities more appropriate than selecting the activities at random. The results show how the study participants found the personalised activity selection more appropriate, improving their likeability towards the robot and how intelligent they perceive the system. query Please check the edit made in the article title.

show abstract

“…It consists of perception, goal learning via a deep network, task planning via a task planner and execution. Inspired by previous methods for manipulation task completion via object affordance recognition [11], [12], we formulate goal learning as predicting symbolic goal representation for Planning Domain Definition Language (PDDL), which bridges connectionist goal learning and symbolic task planning in the proposed hybrid system. In addition to incomplete human instructions, we further consider implicit human intent, which has not been explored in existing methods.…”

Section: Introductionmentioning

confidence: 99%

SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following

Xu¹,

Chen²,

Lin³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper investigates robot manipulation based on human instruction with ambiguous requests. The intent is to compensate for imperfect natural language via visual observations. Early symbolic methods, based on manually defined symbols, built modular framework consist of semantic parsing and task planning for producing sequences of actions from natural language requests. Modern connectionist methods employ deep neural networks to automatically learn visual and linguistic features and map to a sequence of low-level actions, in an endto-end fashion. These two approaches are blended to create a hybrid, modular framework: it formulates instruction following as symbolic goal learning via deep neural networks followed by task planning via symbolic planners. Connectionist and symbolic modules are bridged with Planning Domain Definition Language. The vision-and-language learning network predicts its goal representation, which is sent to a planner for producing a task-completing action sequence. For improving the flexibility of natural language, we further incorporate implicit human intents with explicit human instructions. To learn generic features for vision and language, we propose to separately pretrain vision and language encoders on scene graph parsing and semantic textual similarity tasks. Benchmarking evaluates the impacts of different components of, or options for, the vision-and-language learning model and shows the effectiveness of pretraining strategies. Manipulation experiments conducted in the simulator AI2THOR show the robustness of the framework to novel scenarios.

show abstract

Toward Affordance Detection and Ranking on Novel Objects for Real-World Robotic Manipulation

Cited by 26 publications

References 38 publications

ASTRON: Action-Based Spatio-Temporal Robot Navigation

ASTRON: Action-Based Spatio-Temporal Robot Navigation

An adaptive decision-making system supported on user preference predictions for human–robot interactive communication

SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following

Contact Info

Product

Resources

About