HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

Luiten, Jonathon; Ošep, Aljoša; Dendorfer, Patrick; Torr, Philip H. S.; Geiger, Andreas; Leal-Taixé, Laura; Leibe, Bastian

doi:10.1007/s11263-020-01375-2

Cited by 457 publications

(238 citation statements)

References 80 publications

Supporting

Mentioning

234

Contrasting

Order By: Relevance

“…Jaccard distance (Volk et al, 2020;Luiten et al, 2021) used both FP and FN instances and is described as:…”

Section: Traditional Metrics or Performance Indicatorsmentioning

confidence: 99%

“…The numerator considers the area of intersection of the two bounding boxes while the denominator is their union. Figure 4 depicts the concept of IoU which serves as a similarity indicator based on object detection (Luiten et al, 2021).…”

Section: Traditional Metrics or Performance Indicatorsmentioning

confidence: 99%

“…Self-developed performance metrics are being employed to improve the ADS, using data from limited field deployments. Several other research studies related to the safety of ADS are already published (e.g., Alkhoury, 2017;Every et al, 2017;Fraade-blanar et al, 2018;Nistér et al, 2019;Wood et al, 2019;Berk et al, 2020;Riedmaier et al, 2020;Weng et al, 2020;Wishart et al, 2020;Bansal et al, 2021;Elli et al, 2021;Huang and Kurniawati, 2021;Luiten et al, 2021;Wang et al, 2021). Researchers across the globe are making a considerable effort to quantify the safety of ADS and consequently improve it.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Literature Review of Performance Metrics of Automated Driving Systems for On-Road Vehicles

Sharath

Mehran

2021

Front. Future Transp.

View full text Add to dashboard Cite

The article presents a review of recent literature on the performance metrics of Automated Driving Systems (ADS). More specifically, performance indicators of environment perception and motion planning modules are reviewed as they are the most complicated ADS modules. The need for the incorporation of the level of threat an obstacle poses in the performance metrics is described. A methodology to quantify the level of threat of an obstacle is presented in this regard. The approach involves simultaneously considering multiple stimulus parameters (that elicit responses from drivers), thereby not ignoring multivariate interactions. Human-likeness of ADS is a desirable characteristic as ADS share road infrastructure with humans. The described method can be used to develop human-like perception and motion planning modules of ADS. In this regard, performance metrics capable of quantifying human-likeness of ADS are also presented. A comparison of different performance metrics is then summarized. ADS operators have an obligation to report any incident (crash/disengagement) to safety regulating authorities. However, precrash events/states are not being reported. The need for the collection of the precrash scenario is described. A desirable modification to the data reporting/collecting is suggested as a framework. The framework describes the precrash sequences to be reported along with the possible ways of utilizing such a valuable dataset (by the safety regulating authorities) to comprehensively assess (and consequently improve) the safety of ADS. The framework proposes to collect and maintain a repository of precrash sequences. Such a repository can be used to 1) comprehensively learn and model the precrash scenarios, 2) learn the characteristics of precrash scenarios and eventually anticipate them, 3) assess the appropriateness of the different performance metrics in precrash scenarios, 4) synthesize a diverse dataset of precrash scenarios, 5) identify the ideal configuration of sensors and algorithms to enhance safety, and 6) monitor the performance of perception and motion planning modules.

show abstract

“…Jaccard distance (Volk et al, 2020;Luiten et al, 2021) used both FP and FN instances and is described as:…”

Section: Traditional Metrics or Performance Indicatorsmentioning

confidence: 99%

Section: Traditional Metrics or Performance Indicatorsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Literature Review of Performance Metrics of Automated Driving Systems for On-Road Vehicles

Sharath

Mehran

2021

Front. Future Transp.

View full text Add to dashboard Cite

show abstract

“…In this section, we introduce the Higher Order Tracking Accuracy (HOTA) evaluation metric developed in detail by (Luiten et al, 2020). This metric was used to assess the performance of our detector/tracker combinations on the multi-object tracking task.…”

Section: Hota Evaluation Metricmentioning

confidence: 99%

Deep Learning-Based Object Tracking via Compressed Domain Residual Frames

Khoury

Samelson

Macq

2021

Front. Signal Process.

View full text Add to dashboard Cite

The extensive rise of high-definition CCTV camera footage has stimulated both the data compression and the data analysis research fields. The increased awareness of citizens to the vulnerability of their private information, creates a third challenge for the video surveillance community that also has to encompass privacy protection. In this paper, we aim to tackle those needs by proposing a deep learning-based object tracking solution via compressed domain residual frames. The goal is to be able to provide a public and privacy-friendly image representation for data analysis. In this work, we explore a scenario where the tracking is achieved directly on a restricted part of the information extracted from the compressed domain. We utilize exclusively the residual frames already generated by the video compression codec to train and test our network. This very compact representation also acts as an information filter, which limits the amount of private information leakage in a video stream. We manage to show that using residual frames for deep learning-based object tracking can be just as effective as using classical decoded frames. More precisely, the use of residual frames is particularly beneficial in simple video surveillance scenarios with non-overlapping and continuous traffic.

show abstract

“…Several papers contribute novel metrics and taxonomies for the evaluation of different tasks, including multi-object tracking (Luiten et al 2021), semantic segmentation (Yan et al 2021), visual place recognition (Zaffar et al 2021), image quality assessment (Ding et al 2021), conditional image generation (Benny et al 2021), and embodied exploration (Ramakrishnan et al 2021).…”

mentioning

confidence: 99%