Multi-source Multi-modal Activity Recognition in Aerial Video Surveillance

Abstract

Recognizing activities in wide aerial/overhead imagery remains a challenging problem due in part to low-resolution video and cluttered scenes with a large number of moving objects. In the context of this research, we deal with two unsynchronized data sources collected in real-world operating scenarios: full-motion videos (FMV) and analyst call-outs (ACO) in the form of chat messages (voice-to-text) made by a human watching the streamed FMV from an aerial platform. We present a multi-source multi-modal activity/event recognition system for surveillance applications, consisting of: (1) detecting and tracking multiple dynamic targets from a moving platform, (2) representing FMV target tracks and chat messages as graphs of attributes, (3) associating FMV tracks and chat messages using a probabilistic graph-based matching approach, and (4) detecting spatial-temporal activity boundaries. We also present an activity pattern learning framework which uses the multi-source associated data as training to index a large archive of FMV videos. Finally, we describe a multi-intelligence user interface for querying an index of activities of interest (AOIs) by movement type and geo-location, and for playing-back a summary of associated text (ACO) and activity video segments of targetsof-interest (TOIs) (in both pixel and geo-coordinates). Such tools help the end-user to quickly search, browse, and prepare mission reports from multi-source data.

Dynamic Data-driven Application System (DDDAS) for Video Surveillance User Support

“…To test the subquery evaluation performance of the execution engine and related metadata structures, an evaluation scenario comprised of executing queries simultaneously for a period of duration. The evaluation was done to detect objects, classify type (by size), and discern activities (Hammoud, Sahin, Blasch, et al, 2014). The LVC-DMBS could be applicable to make image collections such as plume detection (Ravela, 2013).…”

Agile battle management efficiency for command, control, communications, computers and intelligence (C4I)

“…The result was an effort in controlled natural language which then could utilize the many developments in natural language processing (NLP) [35] for situation awareness [36]. Subsequent efforts included applications for control of full-motion video platforms [37,38]. Recent efforts are enabled the next generation analysts [39].…”

Multi-Intelligence Analytics for Next Generation Analysts (MIAGA)

“…The technology should augment a human analyst with their associated work domain objectives. As shown in Figure 3, information fusion includes low-level information fusion [LLIF] of Level 1 Object Assessment tracking and classification [20,21]. LLIF can be accessed from video, radar, or text data, among others for intelligence services of explicit decision making.…”