Which Emoji Talks Best for My Picture?

Abstract

Emojis have evolved as complementary sources for expressing emotion in social-media platforms where posts are mostly composed of texts and images. In order to increase the expressiveness of the social media posts, users associate relevant emojis with their posts. Incorporating domain knowledge has improved machine understanding of text. In this paper, we investigate whether domain knowledge for emoji can improve the accuracy of emoji recommendation task in case of multimedia posts composed of image and text. Our emoji recommendation can suggest accurate emojis by exploiting both visual and textual content from social media posts as well as domain knowledge from Emojinet. Experimental results using pre-trained image classifiers and pre-trained word embedding models on Twitter dataset show that our results outperform the current state-ofthe-art by 9.6%. We also present a user study evaluation of our recommendation system on a set of images chosen from MSCOCO dataset.

Emojis as Contextual Indicants in Location-Based Social Media Posts

“…Studies with a focus on communication examine how emojis are interpreted, what they convey and what influencing factors need to be taken into account here [11,[23][24][25][26][27]. A technology intended to support the use of emojis is emoji recommendation [28][29][30][31] which intends to suggest suitable emojis to the user when typing messages. Apart from the already mentioned fields, emojis are also utilized for research in marketing, behavioral science, psychology, medicine and education [5].…”

“…It can be assumed that emojis also reflect this because neither words nor emojis are chosen randomly [37]. When being used in combination with images (like on Instagram), emojis provide additional contextual information about what is shown in an image [30]. Additionally, detached from an image, emojis do not only express emotional states or alter the tone of a text; they also serve as decoration, as a replacement for lexical units, and deliver information not explicitly stated in the text [37][38][39][40].…”

Multimodal Emotion Classification

“…We chose fasttext over other conventional word embedding models due to its capability of capturing sub-word information which plays a significant role in social NLP tasks. Also fasttext word embedding model has proved to give greater accuracies compared to other word embedding models over various NLP tasks such as sentiment analysis, emotion detection, sarcasm identification [14], and emoji prediction [12].…”

“…Recent research in NLP Emojinet has improved the accuracies of emoji similarity [32], learning emoji embeddings, emoji sense disambiguation [31]. Emojinet has also improved the accuracies of emoji prediction in case of images [12]. In the case of image processing tasks, knowledge graphs have also improved the task of video captioning [29] and other related tasks [19].…”

“…We chose fasttext over other conventional word embedding models due to its capability of capturing sub-word information which plays a significant role in social NLP tasks. Also fasttext word embedding model has proved to give greater accuracies compared to other word embedding models over various NLP tasks such as sentiment analysis, emotion detection, sarcasm identification [14], and emoji prediction [12]. As we have observed that the emotional intent of the emoji used and the emotion expressed by the user in the post are parallel, we tried to use EmojiNet which lists 21,000 emoji sense definitions and 4,618 emoji sense forms to 2,389 emojis.…”