The Robotic Vision Scene Understanding Challenge

Hall, D.; Talbot, Ben; Bista, Suman Raj; Zhang, Haoyang; Smith, Robert; Dayoub, Feras; Sünderhauf, Niko

doi:10.48550/arxiv.2009.05246

Cited by 4 publications

(9 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CubeSLAM [18] uses 3D IoU and AP calculated using an IoU threshold of 0.25 for evaluating estimated cuboids. Recently, [13] extends the Probability-based Detection Quality (PDQ) [26] evaluation measure designed for probabilistic object detection to evaluate object-based semantic map. This Object Map Quality (OMQ) evaluation method is used as the metrics in the Scene Understanding Challenge [13].…”

Section: B Evaluation Metricsmentioning

confidence: 99%

“…Recently, [13] extends the Probability-based Detection Quality (PDQ) [26] evaluation measure designed for probabilistic object detection to evaluate object-based semantic map. This Object Map Quality (OMQ) evaluation method is used as the metrics in the Scene Understanding Challenge [13]. Since mAP using 3D IoU and OMQ can both evaluate spatial and semantic quality with a single metric, we choose to use them in evaluating the performance of some of the popular semantic mapping algorithms in this work.…”

Section: B Evaluation Metricsmentioning

confidence: 99%

“…The variations present changes in both lightning and the objects present within each scene. This system has been primarily designed for the Scene Understanding Challenge and provides groundtruth 3D cuboid maps for 30 classes of object : bottle, cup, knife, bowl, wine glass, fork, spoon, banana, apple, orange, cake, potted plant, mouse, keyboard, laptop, cellphone, book, clock, chair, dining table, couch, bed, toilet, monitor, microwave, toaster, refrigerator, oven, sink, and person [13]. However, we were unable to use all of the objects in the larger environments due to GPU memory requirements of some of the methods being evaluated.…”

Section: A Simulation Environmentsmentioning

confidence: 99%

“…1) mean Average Precision (mAP) [22], [29] calculated based on the IoU of cuboids, and 2) Object Map Quality (OMQ) [13]. Calculating the 3D IoU between ground-truth cuboids and those produced by the Semantic SLAM systems is a core component to both measures.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…While some standard real-world datasets exist such as in [8], [9], we leverage advances in simulated environments for robotics tasks (like [10], [11]) to attain repeatable results in environments with controlled variations. We use the BenchBot framework [12] to gain access to ground-truth instance segmentation, depth, and pose data within a set of simulated environments [13]. By selectively utilizing this ground-truth data within the semantic mapping pipeline, we are able to examine which components have…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Evaluating the Impact of Semantic Segmentation and Pose Estimation on Dense Semantic SLAM

Bista,

Hall,

Talbot

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent Semantic SLAM methods combine classical geometry-based estimation with deep learning-based object detection or semantic segmentation. In this paper we evaluate the quality of semantic maps generated by state-of-the-art classand instance-aware dense semantic SLAM algorithms whose codes are publicly available and explore the impacts both semantic segmentation and pose estimation have on the quality of semantic maps. We obtain these results by providing algorithms with ground-truth pose and/or semantic segmentation data available from simulated environments. We establish that semantic segmentation is the largest source of error through our experiments, dropping mAP and OMQ performance by up to 74.3% and 71.3% respectively.

show abstract

Section: B Evaluation Metricsmentioning

confidence: 99%

Section: B Evaluation Metricsmentioning

confidence: 99%

Section: A Simulation Environmentsmentioning

confidence: 99%

Section: Evaluation Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Evaluating the Impact of Semantic Segmentation and Pose Estimation on Dense Semantic SLAM

Bista,

Hall,

Talbot

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

A Deep Dive into Robot Vision - An Integrative Systematic Literature Review Methodologies and Research Endeavor Practices

Sultana,

Alam,

Su’ud

et al. 2024

ACM Comput. Surv.

View full text Add to dashboard Cite

Novel technological swarm and industry 4.0 mold the recent Robot vision research into innovative discovery. To enhance technological paradigm Deep Learning offers magical pace to get into diversified advancement. This research considers most topical, recent, related and state of the art research review revolves around Robot vision, shapes the research into Systematic Literature Survey – SLR. The SLR considers a combo of more than 100 reviews and empirical studies to perform a critical categorical study and shapes findings against Research Questions. The research study contribution is span over multiple categories of Robot vision and tinted along with technical limitations and future research endeavor. Previously multiple researches have been observed to leverage Robotic vision technique. Yet, there is none alike SLR summarizing recent vision techniques for all targeted Robotic fields. This research SLR could be a precious mile stone in Robot vision for each glimpse of Robotics.

show abstract

BenchBot environments for active robotics (BEAR): Simulated data for active scene understanding research

Hall

Talbot

Bista

et al. 2022

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

We present a platform to foster research in active scene understanding, consisting of high-fidelity simulated environments and a simple yet powerful API that controls a mobile robot in simulation and reality. In contrast to static, pre-recorded datasets that focus on the perception aspect of scene understanding, agency is a top priority in our work. We provide three levels of robot agency, allowing users to control a robot at varying levels of difficulty and realism. While the most basic level provides pre-defined trajectories and ground-truth localisation, the more realistic levels allow us to evaluate integrated behaviours comprising perception, navigation, exploration and SLAM. In contrast to existing simulation environments, we focus on robust scene understanding research using our environment interface (BenchBot) that provides a simple API for seamless transition between the simulated environments and real robotic platforms. We believe this scaffolded design is an effective approach to bridge the gap between classical static datasets without any agency and the unique challenges of robotic evaluation in reality. Our BenchBot Environments for Active Robotics (BEAR) consist of 25 indoor environments under day and night lighting conditions, a total of 1443 objects to be identified and mapped, and ground-truth 3D bounding boxes for use in evaluation. BEAR website: https://qcr.github.io/dataset/benchbot-bear-data/ .

show abstract

The Robotic Vision Scene Understanding Challenge

Cited by 4 publications

References 36 publications

Evaluating the Impact of Semantic Segmentation and Pose Estimation on Dense Semantic SLAM

Evaluating the Impact of Semantic Segmentation and Pose Estimation on Dense Semantic SLAM

A Deep Dive into Robot Vision - An Integrative Systematic Literature Review Methodologies and Research Endeavor Practices

BenchBot environments for active robotics (BEAR): Simulated data for active scene understanding research

Contact Info

Product

Resources

About