Exploitation of Semantic Keywords for Malicious Event Classification

Abstract

Learning an event classifier is challenging when the scenes are semantically different but visually similar. However, as humans, we typically handle such tasks painlessly by adding our background semantic knowledge. Motivated by this observation, we aim to provide an empirical study about how additional information such as semantic keywords can boost up the discrimination of such events. To demonstrate the validity of this study, we first construct a novel Malicious Crowd Dataset containing crowd images with two events, benign and malicious, which look visually similar. Note that the primary focus of this paper is not to provide the state-ofthe-art performance on this dataset but to show the beneficial aspects of using semantically-driven keyword information. By leveraging crowd-sourcing platforms, such as Amazon Mechanical Turk, we collect semantic keywords associated with images and then subsequently identify a subset of keywords (e.g. police, fire, etc.) unique to specific events. We first show that by using recently introduced attention models, a naïve CNN-based event classifier actually learns to primarily focus on local attributes associated with the discriminant semantic keywords identified by the Turks. We further show that incorporating the keyword-driven information into earlyand late-fusion approaches can significantly enhance malicious event classification.

S-DOD-CNN: Doubly Injecting Spatially-Preserved Object Information for Event Recognition

“…In addition to the label of the event class, bounding box annotations of three rigid objects (police, helmet, car) and two non-rigid objects (fire, smoke) are provided. [12] provides details on how these objects are selected.…”

“…Malicious Crowd Dataset [12,22] is selected as it provides the appropriate components to evaluate the effects of using object information for event recognition. It contains 1133 images and is equally divided into malicious classes and benign classes.…”

“…In order to prove the effectiveness of using a spatiallypreserved object detection feature maps for event recognition, we conducted several experiments on the malicious event classification [12]. We have validated that all four combinations of the novel RoI projection within S-DOD-CNN provide higher accuracy than all the baselines.…”

DOD-CNN: Doubly-injecting Object Information for Event Recognition

“…There are two approaches to exploit the object detection |Rn|=5 first approach is to make use of a separately constructed object detection module and its output for boosting the event recognition. In this approach, the object detection results can either be directly fed into the event recognition module [1,2,3] or be integrated with the event recognition output via a late fusion [4,5,6,7,8,9,10,11]. The second approach is to transfer the object information by sharing the network weights between the object detection and event recognition and co-learning them in a unified architecture.…”

“…We evaluated the proposed approach on the Malicious Crowd Dataset [11]. The experiments demonstrate that utilizing the object detection information in both direct (injecting the feature maps) and indirect (transferring the information via shared weights) ways are effective in enhancing malicious event recognition performance.…”