Probabilistic Object Detection: Definition and Evaluation

Hall, David R.; Dayoub, Feras; Skinner, John; Zhang, Haoyang; Miller, Dimity; Corke, Peter; Carneiro, Gustavo; Angelova, Anelia; Sünderhauf, Niko

doi:10.1109/wacv45572.2020.9093599

Cited by 96 publications

(76 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We must point out that partially these low scores might also originate from the format conversion to evaluate COCO-style bounding boxes using the PDQ. However, no model evaluated in the original PDQ publication exceeds an average spatial score of 45% [ 30 ].…”

Section: Resultsmentioning

confidence: 99%

“…In contrast, the (PDQ Probability-based Detection Quality), introduced by Hall et al [ 30 ], focuses on a probabilistic evaluation what an object detected is and where it is. The main reason for choosing this metric is that the PDQ explicitly penalizes false positives and false negatives and penalizes low confidences scores.…”

Section: Experimental Setupmentioning

confidence: 99%

“…In the fields of robotics or in medical applications, the consequences of an unreliable model for object detection are likely to be significantly more severe than e.g., in a deployment scenario that involves object detection for an online retail store [ 28 , 29 , 30 ]. Similarly, when considering on-board processing on satellites or other edge devices, consequences of large amounts of false positives and false negatives may render a product or an application pointless.…”

Section: Introductionmentioning

confidence: 99%

“…Therefore, we evaluate a large subset of state-of-the-art models performance at different confidence score thresholds, using the standard COCO metrics as well as Probability-based Detection Quality (PDQ) [ 30 ], and propose a method for finding the optimal confidence score for a model as a basis for a fairer model comparison than what is commonly used ( Figure 1 ). In other words, we propose a method to find the optimal operating point of a model that could be used to decide what model to choose and what to expect if the threshold is lowered or increased.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Confidence Score: The Forgotten Dimension of Object Detection Performance Evaluation

Wenkel¹,

Alhazmi

Liiv³

et al. 2021

Sensors

View full text Add to dashboard Cite

When deploying a model for object detection, a confidence score threshold is chosen to filter out false positives and ensure that a predicted bounding box has a certain minimum score. To achieve state-of-the-art performance on benchmark datasets, most neural networks use a rather low threshold as a high number of false positives is not penalized by standard evaluation metrics. However, in scenarios of Artificial Intelligence (AI) applications that require high confidence scores (e.g., due to legal requirements or consequences of incorrect detections are severe) or a certain level of model robustness is required, it is unclear which base model to use since they were mainly optimized for benchmark scores. In this paper, we propose a method to find the optimum performance point of a model as a basis for fairer comparison and deeper insights into the trade-offs caused by selecting a confidence score threshold.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Experimental Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Confidence Score: The Forgotten Dimension of Object Detection Performance Evaluation

Wenkel¹,

Alhazmi

Liiv³

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…We employ the data splits defined in [33], where 5 000 images are used for validation, 5 000 images for testing and the rest for training. Further, to assess the performance of our approach on images taken from a robot-centric point of view, we employ the ACVR Robotic Vision Challenge dataset [34] which contains simulated data from a domestic robot scenario. The dataset contains scenes with cluttered surfaces, and day and night lighting conditions.…”

Section: A Datasetsmentioning

confidence: 99%

SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability

Cornia

Baraldi

Cucchiara

2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

The ability to generate natural language explanations conditioned on the visual perception is a crucial step towards autonomous agents which can explain themselves and communicate with humans. While the research efforts in image and video captioning are giving promising results, this is often done at the expense of the computational requirements of the approaches, limiting their applicability to real contexts. In this paper, we propose a fully-attentive captioning algorithm which can provide state-of-the-art performances on language generation while restricting its computational demands. Our model is inspired by the Transformer model and employs only two Transformer layers in the encoding and decoding stages. Further, it incorporates a novel memory-aware encoding of image regions. Experiments demonstrate that our approach achieves competitive results in terms of caption quality while featuring reduced computational demands. Further, to evaluate its applicability on autonomous agents, we conduct experiments on simulated scenes taken from the perspective of domestic robots.

show abstract

The National Airworthiness Council artificial intelligence working group (NACAIWG) summit proceedings 2022

Parry

Costello

Rupert³

et al. 2023

Systems Engineering

View full text Add to dashboard Cite

During the 2021 summit, a baseline overview of literature pertaining to the field of computer vision (CV) was provided. For the 2022 summit, a sample of literature published since the 2021 summit was provided to introduce participants to updated work in the field of test and evaluation (T&E), CV, algorithmic assurance, and frameworks to provide assurance of algorithms. All papers were collected from the Purdue University Library 3 and Dissertations and Theses Database 4 and cited in references. 5-14 Applicable standards governing LECBuilding off the standards survey conducted for the 2021 summit, a follow-on survey of standards for all modes of transportation (e.g., rail, sea, road, etc.) was conducted in support of the 2022 summit to determine if any notable improvements or advancements had occurred.

show abstract

Probabilistic Object Detection: Definition and Evaluation

Cited by 96 publications

References 26 publications

Confidence Score: The Forgotten Dimension of Object Detection Performance Evaluation

Confidence Score: The Forgotten Dimension of Object Detection Performance Evaluation

SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability

The National Airworthiness Council artificial intelligence working group (NACAIWG) summit proceedings 2022

Contact Info

Product

Resources

About