A survey of imagery techniques for semantic labeling of human-vehicle interactions in persistent surveillance systems

Abstract

Understanding and semantic annotation of Human-Vehicle Interactions (HVI) facilitate fusion of Hard sensor (HS) and Human Intelligence (HUMINT) in a cohesive way. By characterization, classification, and discrimination of HVI patterns pertinent threats may be realized. Various Persistent Surveillance System (PSS) imagery techniques have been proposed in the past decade for identifying human interactions with various objects in the environment. Understanding of such interactions facilitates to discover human intentions and motives. However, without consideration of incidental context, reasoning and analysis of such behavioral activities is a very challenging and difficult task. This paper presents a current survey of related publications in the area of context-based Imagery techniques applied for HVI recognition, in particular, it discusses taxonomy and ontology of HVI and presents a summary of reported robust image processing techniques for spatiotemporal characterization and tracking of human targets in urban environments. The discussed techniques include model-based, shape-based and appearance-based techniques employed for identification and classification of objects. A detailed overview of major past research activities related to HVI in PSS with exploitation of spatiotemporal reasoning techniques applied to semantic labeling of the HVI is also presented.

An ontology of human walk for autonomous systems

“…Other ontologies in the literature that have similarities to the one discussed here are reported, e.g., in [16], [17]. The study [17], aiming to describe the interaction of people with vehicles, may seem close to ours in its scope.…”

“…The study [17], aiming to describe the interaction of people with vehicles, may seem close to ours in its scope. However, the similarity is very low, because the type of interactions in that study significantly differs from the interactions considered in our study.…”

In-vehicle group activity modeling and simulation in sensor-based virtual environment

“…In the past, we have investigated methods for characterization of group activities without significant consideration of inherent physical obstructions [3][4][5][6][7][8][9]. In many FPSS applications, however, the detection, tracking, and characterization of group activities are not that simple.…”