Are Emoji Processed Like Words? An Eye‐Tracking Study

Abstract

In this study, we investigate the processing of object-denoting emoji in sentences using eye tracking. We hypothesize that (a) such emoji are more difficult to process when used as word replacement; and (b) their processing is subject to ambiguity constraints similarly to what happens with words. We conduct two experiments in which participants have to read sentences in which an emoji either follows or replaces a word. Control stimuli not containing emoji are also tested. In the second experiment, the emoji are presented in two different disambiguating contexts. First fixation duration, total visit duration, and total revisit duration are modeled in the various conditions using linear mixed models. Both our hypotheses are supported. We observe longer total visit time for non-redundant emoji, and higher values for all three measures for ambiguous emoji. We conclude that lexical access may be more difficult for emoji especially when they are used as word replacements and their meaning is not immediately clear. Furthermore, we also conclude that non-redundant emoji are more difficult to integrate in the processing of the sentence than the equivalent words, or emoji used in a redundant way. In turn, our results indicate that emoji may not always be as immediate and easy to process for readers in spite of their popularity.

How do GenZ speakers use and process emoji in chatbot conversations: An eye-tracking study

“…EMOJI PROCESSING: EYE TRACKING STUDIES. Eye tracking studies have proven to be a valuable technique for investigating emoji processing during reading (Barach et al 2021;Boutet et al 2021;Howman & Filik 2020;Paggio & Tse 2022;Robus et al 2020). Eye tracking measures reflect the time course cognitive processing, providing an online measure of how emojis are handled during reading.…”

“…First fixation duration indicates the initial time spent fixating on a target and relates to early stages of lexical access, whereas total fixation duration captures all the fixation time, and therefore all of the processing for a target (Rayner 1998). Paggio & Tse (2022) conducted a reading task in which an emoji either follows or replaces a word and found that emojis were sometimes more difficult to process when used to replace words, with shorter total visit times for unambiguous (such as Airplane '✈' or Pizza '🍕') vs. ambiguous (such as Bowl with Spoon '🥣' or Tent '⛺') emojis. This indicates that lexical access may be more effortful when the meaning of an emoji is not immediately clear for the reader.…”

“…Their interpretation varies greatly: the way users perceive, interpret, and use emojis is influenced by their age, gender, and culture (An et al 2018;Freedman 2018;Guntuku et al 2019;Jones et al 2020;Miller et al 2016;Prada et al 2018;Weiß et al 2020). Despite clear evidence for userspecific interpretation, previous psycholinguistic studies showed that emojis are processed similarly to words during reading (Barach et al 2021;Boutet et al 2021;Howman & Filik 2020;Paggio & Tse 2022;Robus et al 2020;Weissman & Tanner 2018). This leaves an open puzzle as to how both persona-based and lexical aspects of emojis are integrated in processing.…”