EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios

Bhattacharya, Sunit; Kloudová, Věra; Zouhar, Vilém; Bojar, Ondřej

doi:10.48550/arxiv.2204.02905

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this direction, Bhattacharya et al (2022) conducted an experiment on human participants with translation enhanced by image modality, which is parallel to our experiment with language modelling using the same modality. Finally, some researchers (Hladká et al, 2009(Hladká et al, , 2011 have utilized the Games With a Purpose methodology (Von Ahn and Dabbish, 2008) to frame tasks that are difficult for computers but relatively easy for humans as games.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Shannon Game with Images

Zouhar¹,

Bhattacharya²,

Bojar³

2023

Preprint

View full text Add to dashboard Cite

The Shannon game has long been used as a thought experiment in linguistics and NLP, asking participants to guess the next letter in a sentence based on its preceding context. We extend the game by introducing an optional extra modality in the form of image information. To investigate the impact of multimodal information in this game, we use human participants and a language model (LM, GPT-2).We show that the addition of image information improves both self-reported confidence and accuracy for both humans and LM. Certain word classes, such as nouns and determiners, benefit more from the additional modality information. The priming effect in both humans and the LM becomes more apparent as the context size (extra modality information + sentence context) increases. These findings highlight the potential of multimodal information in improving language understanding and modeling.

show abstract

Section: Introductionmentioning

confidence: 99%

Multimodal Shannon Game with Images

Zouhar¹,

Bhattacharya²,

Bojar³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We analyze of the Eyetracked Multi-Modal Translation (EMMT) corpus (Bhattacharya et al, 2022). It consists of observations from a set of psycholinguistic experiments involving translation under multi-modal settings with a number of Czech native speakers with an advanced level of English language proficiency.…”

Section: Introductionmentioning

confidence: 99%

“…The experiment design setup implemented during the collection of data used a combination of sight translation, reading aloud and thinking aloud (Tirkkonen-Condit, 1990) protocols. Bhattacharya et al (2022) intend to compare the behavioural data of the participants when they read out a textual stimulus (sentence) or looked at a combination of text and visual (image) stimulus versus when they actually translated the textual stimulus. The experiment thus followed four stages corresponding to each stimulus: Stage 1 (READ: reading of the source English sentence), Stage 2 (TRANSLATE: translating the English sentence into Czech), Stage 3 (SEE: observing the corresponding image) and Stage 4 (UPDATE: producing the final translation of the English sentence given the image).…”

Section: Introductionmentioning

confidence: 99%

Stroop Effect in Multi-Modal Sight Translation

Bhattacharya¹,

Zouhar²,

Kloudová³

et al. 2022

Preprint

View full text Add to dashboard Cite

This study investigates the human translation process from English to Czech in a multi-modal scenario (images) using reaction times. We make a distinction between ambiguous and unambiguous sentences where in the former, more information would be needed in order to make a proper translation (e.g. gender of the subject). Simultaneously, we also provide visual aid to help in disambiguation, which is necessary for the ambiguous sentences. We confirm that ambiguous sentences take longer to translate and the provision of disambiguating visual aid slows the translation process. When provided with an unrelated visual aid, humans are able to recognize and spend less time on it but still significantly more than in other conditions. These findings are a clear manifestation of the Stroop effect (longer processing times for incongruent combinations).

show abstract

EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios

Cited by 2 publications

References 0 publications

Multimodal Shannon Game with Images

Multimodal Shannon Game with Images

Stroop Effect in Multi-Modal Sight Translation

Contact Info

Product

Resources

About