VidCEP: Complex Event Processing Framework to Detect Spatiotemporal Patterns in Video Streams

Abstract

Video data is highly expressive and has traditionally been very difficult for a machine to interpret. Querying event patterns from video streams is challenging due to its unstructured representation. Middleware systems such as Complex Event Processing (CEP) mine patterns from data streams and send notifications to users in a timely fashion. Current CEP systems have inherent limitations to query video streams due to their unstructured data model and lack of expressive query language. In this work, we focus on a CEP framework where users can define high-level expressive queries over videos to detect a range of spatiotemporal event patterns. In this context, we propose-i) VidCEP, an in-memory, on the fly, near real-time complex event matching framework for video streams. The system uses a graphbased event representation for video streams which enables the detection of high-level semantic concepts from video using cascades of Deep Neural Network models, ii) a Video Event Query language (VEQL) to express high-level user queries for video streams in CEP, iii) a complex event matcher to detect spatiotemporal video event patterns by matching expressive user queries over video data. The proposed approach detects spatiotemporal video event patterns with an F-score ranging from 0.66 to 0.89. VidCEP maintains near real-time performance with an average throughput of 70 frames per second for 5 parallel videos with sub-second matching latency.

Design and Development of an Internet of Smart Cameras Solution for Complex Event Detection in COVID-19 Risk Behaviour Recognition

“…As all of the frames are coming from a single source, the coordinate system and camera angle are the same during the complex event detection process. The process consists of object matching, spatial matching, and temporal matching [20]. Pseudocode 1 in Appendix A shows the steps for our proposed event matching algorithm.…”

“…For root relationships, the relationship between temporal relationships was considered before the complex event was finally detected. VEQL proposed by [20] was used for querying events. The output of every detected complex event is a GeoJSON record that is stored in a DynamoDB table.…”

“…The majority of research into complex event detection has attempted to improve the querying of video content to increase the ease of information retrieval for users [14][15][16][17][18][19]. There were also other studies that focused on event pattern detection and event matching, which is the process of matching observed events frames with complex event patterns [20][21][22]. However, most of the research did not consider missing frames and objects for complex event detection in a real camera streaming scenario.…”

“…Studies such as NoScope [44] and Focus [45] attempted to optimise spatiotemporal querying in video streams but did not provide expressive querying for retrieving spatiotemporal patterns. Hence, Yadav and Curry proposed a video complex event framework that detected spatiotemporal patterns in video streams [20]. Additionally, they proposed a query language that processed complex events from spatial and temporal relationships between objects.…”

Investigating response time and accuracy in online classifier learning for multimedia publish-subscribe systems

“…However, event-based systems are more focused on structured (scalar) events [22]. Existing multimedia-based communication technologies have very competitive performance, but all are domain-specific [43,46,68,74]. Monitoring applications related to transport management cannot handle hospital-related events; similarly, ecological surveillance applications will not be able to respond to the events associated with security.…”