Reasoning With Scene Graphs for Robot Planning Under Partial Observability

Amiri, Saeid; Chandan, Kishan; Zhang, Shiqi

doi:10.1109/lra.2022.3157567

Cited by 21 publications

(5 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Scene graph has been widely applied to visionrelated tasks such as image retrieval (Johnson et al, 2015;Wang et al, 2020;Schuster et al, 2015), image generation , VQA (Benyounes et al, 2019;Li et al, 2019;Shi et al, 2019), and robot planning (Amiri et al, 2022) because of their powerful representation of semantic features of scenes. A scene graph was first proposed in (Johnson et al, 2015) as a data structure for describing objects instances in a scene and relationships between objects.…”

Section: Automatic Evaluation Metricsmentioning

confidence: 99%

JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models

Wada,

Kaneda,

Sugiura

2023

Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

Image captioning studies heavily rely on automatic evaluation metrics such as BLEU and METEOR. However, such n-gram-based metrics have been shown to correlate poorly with human evaluation, leading to the proposal of alternative metrics such as SPICE for English; however, no equivalent metrics have been established for other languages. Therefore, in this study, we propose an automatic evaluation metric called JaSPICE, which evaluates Japanese captions based on scene graphs. The proposed method generates a scene graph from dependencies and the predicate-argument structure, and extends the graph using synonyms. We conducted experiments employing 10 image captioning models trained on STAIR Captions and PFN-PIC and constructed the Shichimi dataset, which contains 103,170 human evaluations. The results showed that our metric outperformed the baseline metrics for the correlation coefficient with the human evaluation.

show abstract

Section: Automatic Evaluation Metricsmentioning

confidence: 99%

JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models

Wada,

Kaneda,

Sugiura

2023

Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

“…Recent works show that non-myopic graph-based planning methods yield promising performance across various tasks and domains [11], [12], [21]. Amiri et al [21] proposed a graph-based planner for task completion under partial observability in indoor environments. Choudhury et al [11] introduced an adaptive informative path planning method leveraging partially observable Monte Carlo planning [22] over graphs encoding a UAV's workspace.…”

Section: Related Workmentioning

confidence: 99%

Combining Local and Global Viewpoint Planning for Fruit Coverage

Zaenker

Lehnert

McCool

et al. 2021

2021 European Conference on Mobile Robots (ECMR)

View full text Add to dashboard Cite

Crop monitoring is crucial for maximizing agricultural productivity and efficiency. However, monitoring large and complex structures such as sweet pepper plants presents significant challenges, especially due to frequent occlusions of the fruits. Traditional next-best view planning can lead to unstructured and inefficient coverage of the crops. To address this, we propose a novel view motion planner that builds a graph network of viable view poses and trajectories between nearby poses, thereby considering robot motion constraints. The planner searches the graphs for view sequences with the highest accumulated information gain, allowing for efficient pepper plant monitoring while minimizing occlusions. The generated view poses aim at both sufficiently covering already detected and discovering new fruits. The graph and the corresponding best view pose sequence are computed with a limited horizon and are adaptively updated in fixed time intervals as the system gathers new information. We demonstrate the effectiveness of our approach through simulated and real-world experiments using a robotic arm equipped with an RGB-D camera and mounted on a trolley. As the experimental results show, our planner produces view pose sequences to systematically cover the crops and leads to increased fruit coverage when given a limited time in comparison to a state-of-the-art single next-best view planner.

show abstract

“…b) Scene graphs in object navigation: Although learning a navigation policy directly from sensor input is viable [12], additional inductive biases and knowledge representations improve efficiency in larger scenes. Unlike other representations such as semantic maps, scene graphs scale with the number of objects rather than the size of the scene, making them suitable as a knowledge and memory representation for object navigation in large-scale scenes [46][47][48]. Scene graphs explicitly and compactly store information about objects' geometry, placement, semantics, and relationships, which makes them ideal for reasoning about object relations.…”

Section: Introductionmentioning

confidence: 99%

Task-Driven Graph Attention for Hierarchical Relational Object Navigation

Lingelbach

Hwang

et al. 2023

2023 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Embodied AI agents in large scenes often need to navigate to find objects. In this work, we study a naturally emerging variant of the object navigation task, hierarchical relational object navigation (HRON), where the goal is to find objects specified by logical predicates organized in a hierarchical structure-objects related to furniture and then to rooms-such as finding an apple on top of a table in the kitchen. Solving such a task requires an efficient representation to reason about object relations and correlate the relations in the environment and in the task goal. HRON in large scenes (e.g. homes) is particularly challenging due to its partial observability and long horizon, which invites solutions that can compactly store the past information while effectively exploring the scene. We demonstrate experimentally that scene graphs are the best-suited representation compared to conventional representations such as images or 2D maps. We propose a solution that uses scene graphs as part of its input and integrates graph neural networks as its backbone, with an integrated task-driven attention mechanism, and demonstrate its better scalability and learning efficiency than state-of-the-art baselines.

show abstract

Reasoning With Scene Graphs for Robot Planning Under Partial Observability

Cited by 21 publications

References 34 publications

JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models

JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models

Combining Local and Global Viewpoint Planning for Fruit Coverage

Task-Driven Graph Attention for Hierarchical Relational Object Navigation

Contact Info

Product

Resources

About