Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

Abstract

Interactions among users on social network platforms are usually positive, constructive and insightful. However, sometimes people also get exposed to objectionable content such as hate speech, bullying, and verbal abuse etc. Most social platforms have explicit policy against hate speech because it creates an environment of intimidation and exclusion, and in some cases may promote real-world violence. As users' interactions on today's social networks involve multiple modalities, such as texts, images and videos, in this paper we explore the challenge of automatically identifying hate speech with deep multimodal technologies, extending previous research which mostly focuses on the text signal alone. We present a number of fusion approaches to integrate text and photo signals. We show that augmenting text with image embedding information immediately leads to a boost in performance, while applying additional attention fusion methods brings further improvement.

Implicitly Abusive Language – What does it actually look like and why are we not getting there?

“…Indeed the community is aware of this form of abuse and there have been several attempts for multimodal analysis (Singh et al, 2017;Yang et al, 2019;Gomez et al, 2020). In our work, however, we do not address the aspect of multimodal abuse simply because many datasets only include the textual component of a micropost and the reconstruction of non-textual components of posts can only be reconstructed with greater effort or even not be obtained at all.…”

Exploring Multi-Task Multi-Lingual Learning of Transformer Models for Hate Speech and Offensive Speech Identification in Social Media

“…A large portion of the hateful content that is shared on social media is in the form of memes, which feature multiple modalities like audio, text, images and videos in some cases as well. [54] present different fusion approaches to tackle multi-modal information for hate speech detection. [11] explore multi-modal hate speech consisting of text and image modalities.…”

Edge-Enhanced Graph Convolution Networks for Event Detection with Syntactic Relation

“…Most recent works have focused on leveraging neural networks in this task (Chen et al, 2015;Nguyen and Grishman, 2015;Nguyen et al, 2016;Ghaeini et al, 2016;Feng et al, 2016). The existing approaches can be categorized into two classes: The first class is to improve ED through special learning techniques including adversarial training (Hong et al, 2018), knowledge distillation (Liu et al, 2019; and model pretraining (Yang et al, 2019). The second class is to improve ED by introducing extra resource, such as argument information , document information (Duan et al, 2017;Chen et al, 2018), multi-lingual information (Liu et al, 2018a(Liu et al, , 2019, knowledge base and syntactic information (Sha et al, 2018).…”