Knowledge Graph Driven Approach to Represent Video Streams for Spatiotemporal Event Pattern Matching in Complex Event Processing

Abstract

Complex Event Processing (CEP) is an event processing paradigm to perform real-time analytics over streaming data and match high-level event patterns. Presently, CEP is limited to process structured data stream. Video streams are complicated due to their unstructured data model and limit CEP systems to perform matching over them. This work introduces a graph-based structure for continuous evolving video streams, which enables the CEP system to query complex video event patterns. We propose the Video Event Knowledge Graph (VEKG), a graph-driven representation of video data. VEKG models video objects as nodes and their relationship interaction as edges over time and space. It creates a semantic knowledge representation of video data derived from the detection of high-level semantic concepts from the video using an ensemble of deep learning models. A CEP-based state optimization — VEKG-Time Aggregated Graph (VEKG-TAG) — is proposed over VEKG representation for faster event detection. VEKG-TAG is a spatiotemporal graph aggregation method that provides a summarized view of the VEKG graph over a given time length. We defined a set of nine event pattern rules for two domains (Activity Recognition and Traffic Management), which act as a query and applied over VEKG graphs to discover complex event patterns. To show the efficacy of our approach, we performed extensive experiments over 801 video clips across 10 datasets. The proposed VEKG approach was compared with other state-of-the-art methods and was able to detect complex event patterns over videos with [Formula: see text]-Score ranging from 0.44 to 0.90. In the given experiments, the optimized VEKG-TAG was able to reduce 99% and 93% of VEKG nodes and edges, respectively, with 5.19[Formula: see text] faster search time, achieving sub-second median latency of 4–20[Formula: see text]ms.

Small sample medical event extraction based on data augmentation

“…In formulas ( 6), ( 7) and (8), n represents the head index, O W represents the weight parameter, and the multi-head attention calculation result is obtained by splicing the attention of i times.…”

“…Traditional event extraction, event extraction based on pattern matching refers to the detection and information extraction of events under the guidance of event templates [[6]~ [8]]. This method requires the construction of event templates, which is time-consuming and labor-intensive, and requires strong professional knowledge.…”

UrbanAccess

“…The information cascades onto another service in the DAG. This chain of information forms Video Event Knowledge Graphs (VEKG) [9,11,13] that stores information in the form of nodes and edges. A declarative query language called GNOSIS Event Processing Language (EPL) later queries VEKGs to obtain the final output.…”

Query-driven video event processing for the internet of multimedia things

“…The GNOSIS EPL is built over powerful openCypher 1 graph query language and enables users to query video events in SQL-like declarative syntax. GNOSIS EPL provides a variety of event rules [18] enabling a wide range of video event detection capabilities such as identifying objects, attributes, and complex spatiotemporal relationships. As shown in Figure 2, the Query Manager component stores, parse and instantiate other GNOSIS components (like windows, Video as a Graph Stream.…”