Protest Activity Detection and Perceived Violence Estimation from Social Media Images

Abstract

We develop a novel visual model which can recognize protesters, describe their activities by visual attributes and estimate the level of perceived violence in an image. Studies of social media and protests use natural language processing to track how individuals use hashtags and links, often with a focus on those items' diffusion. These approaches, however, may not be effective in fully characterizing actual real-world protests (e.g., violent or peaceful) or estimating the demographics of participants (e.g., age, gender, and race) and their emotions. Our system characterizes protests along these dimensions. We have collected geotagged tweets and their images from 2013-2017 and analyzed multiple major protest events in that period. A multi-task convolutional neural network is employed in order to automatically classify the presence of protesters in an image and predict its visual attributes, perceived violence and exhibited emotions. We also release the UCLA Protest Image Dataset, our novel dataset of 40,764 images (11,659 protest images and hard negatives) with various annotations of visual attributes and sentiments. Using this dataset, we train our model and demonstrate its effectiveness. We also present experimental results from various analysis on geotagged image data in several prevalent protest events. Our dataset will be made accessible at

CASM: A Deep-Learning Approach for Identifying Collective Action Events with Text and Image Data from Social Media

“…Deep-learning algorithms have helped make significant advances in many machine learning tasks, especially tasks related to the analysis of images and text, such as image classification (He et al 2016; Simonyan and Zisserman 2015), multiple object detection (Ren et al 2015), automated image captioning (Shin et al 2016), voice recognition (Dahl et al 2012; Hinton et al 2012), machine translation (Bahdanau, Cho, and Bengio 2014; Sutskever, Vinyals, and Le 2014), and parts of speech tagging (Santos and Zadrozny 2014). Deep-learning algorithms are just beginning to be used in social science research, in particular in research using image data (Torres 2018; Won, Steinert-Threlkeld, and Joo 2017). Our work expands on this emerging strand of social science research by using deep learning for image and textual classification.…”

Editors’ Introduction: Visual Politics, Grand Collaborative Programs, and the Opportunity to Think Big

“…Visual analysis conducted in real time and at scale could play an adjudicating role here, establishing certain ground truths about the peaceful (or violent) nature of marches and other collective actions. To move beyond anecdotal and limited eyewitness accounts, computer vision analysis of surveillance cameras, social media posts, or crowd-sourced video could inform ongoing news coverage (see Won et al 2017). Where vandalism, outside agitation, or the use of force by authorities becomes an issue, visual analysis of available footage could help establish who was provoked and who responded-before propagandists have an opportunity to mischaracterize events and cement a false picture of what happened in the public mind.…”

Prototyping a Social Media Flooding Photo Screening System Based on Deep Learning

“…For instance, in the United States dogs appear more in social media photos than other countries. Other visual-based models can also be used, such as violence detection [51,52], face recognition [53], gender and age extraction [54], or skin color analysis [55].…”