Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

Zhan, Likan

doi:10.3791/58086

Cited by 2 publications

(1 citation statement)

References 36 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ipak, paradigma čitanja omogućava da se jezična obrada ispita na drugačiji način. U paradigmi slušanja pokreti očiju mjere se slikovnima podražajima, koje valja pažljivo izabrati kako bi odražavali sadržaj jezičnoga podražaja, a dosta se raspravljalo i o valjanosti mjera kojima se jezična obrada tom metodom mjeri(Kaiser, 2013a;Zhan, 2018). Paradigma čitanja jednostavnija je za primjenu, pokreti očiju mjere se izravno na jezičnom podražaju te pruža veći raspon mjera kojima se mogu izmjeriti različiti vidovi jezične obrade.…”

unclassified

Razrješavanje dvosmislenosti anaforične lične zamjenice

Košutar¹

View full text Add to dashboard Cite

Language comprehension is a complex process that requires rapid integration of information at multiple levels (Carreiras and Clifton, 2004; Jelaska, 2021). The automatic, unconscious aspect of this process is called language processing. The ultimate goal of comprehension is to interpret a linguistic structure, i.e. to create a mental representation of the meaning of that structure based on linguistic knowledge, world knowledge and the features of the immediate linguistic and extra-linguistic context (Harley, 2014; Raffaelli, 2015). One of the prerequisites for successful language comprehension is the ability to establish anaphora. Anaphora refers to the relationship between two elements in a sentence or text, where the interpretation of one element (anaphor) depends on the interpretation of the preceding element (antecedent) (Crystal, 2008). Anaphora has been one of the central topics in linguistics for several decades, including theoretical linguistics, experimental linguistics, psycholinguistics, neurolinguistics, and computational linguistics (Huang, 2000; Holler and Suckow, 2016). Research on anaphora is generally experimental and focuses on 3rd person anaphoric pronouns at the sentence level. However, the phenomenon itself is complex, as the reference can be ambiguous, i.e. when pronoun and antecedent agree in morphosyntactic features. Due to typological differences between languages, it is difficult to draw conclusions about what determines pronoun ambiguity resolution (Hemforth et al., 2010). In pro-drop languages, pronouns can be omitted and reference can be made either in the presence or in the absence of the pronoun. Moreover, studies have shown that there are differences in pronoun ambiguity resolution preferences even between speakers of typologically similar languages such as Italian and Spanish (see Filiaci, Sorace and Carreiras, 2014). Although research on this phenomenon is extensive, many questions remain unanswered: How do speakers resolve pronoun ambiguity resolution? What is the cause of language-specific and/or cross-linguistic differences in pronoun ambiguity resolution? What mental processes underlie pronoun ambiguity resolution during sentence processing? What factors determine the final choice of antecedent? By studying pronoun ambiguity resolution, one can gain insight into the nature of language comprehension in general, as this phenomenon can reveal what information triggers sentence processing at different levels and influences the final sentence interpretation. Many studies have attributed an important role to the notion of salience in pronoun ambiguity resolution. The most salient antecedent is more accessible in the speaker’s mind, i.e. it is the focus of attention and it is referred to by the 3rd person pronoun (Arnold and Lao, 2015). Psychological studies of perception confirm that attentional processes in the brain make decisions about which information is crucial for further processing, thus reducing processing effort (Wolfe and Horowitz, 2004). Different theoretical approaches have been developed to explain which factors influence salience and thus contribute to pronoun ambiguity resolution. Formal approaches are based on structural factors and assume that pronoun ambiguity resolution is determined by the correlation between the grammatical role of the antecedent and the type of anaphor (Ariel, 1990; Walker, Joshi and Prince, 1998; Carminati, 2002). The most phonologically reduced and less informative anaphors, such as 3rd person pronoun, are used to refer to the most salient antecedent. Salience is influenced by the grammatical role, according to the scale subject > object > other. In English, a non-pro-drop language, the 3rd person pronoun is the most reduced anaphor and is used to refer to the most salient antecedent, i.e. the subject. In pro-drop languages, the most salient antecedent is referred to in the absence of the 3rd person pronoun, while the less salient antecedent is referred to in the presence of the 3rd person pronoun. Previous studies have mostly considered the complementary distribution in pronoun ambiguity resolution in pro-drop languages as universal. However, some studies question the generalisability of this hypothesis and reveal language-specific preferences (Filiaci, Sorace and Carreiras, 2014). Semantic approaches are based on factors such as implicit causality verb bias and coherence relations (Kehler, 2002; Koornneef and van Berkum, 2006). Implicit causality verbs influence pronoun ambiguity resolution by directing the speaker’s attention to the antecedent that is likely to be the cause of the event or state. Since this depends on individual preferences, the direction (subject- or object-biased) and strength (from weak to strong) of the bias vary. Although the implicit causality bias is considered universal, cross-linguistic differences in the direction and strength of the bias have been found, and not many languages have been studied (see Hartshorne, Sudo and Uruwashi, 2013). However, these theoretical approaches prove insufficient, as they focus on only one type of factor and disregard the potential relationship between factors and their influence on pronoun ambiguity resolution. Formal approaches do not take into account meaning as a relevant factor and assume that pronoun ambiguity resolution occurs during the structural representation of the sentence, thus separating the structural from the semantic level of the sentence. Semantic approaches, on the other hand, pay no attention to the importance of structural factors and ignore the fact that the presence or the absence of the 3rd person pronoun also contributes to the meaning of the sentence. The studies conducted within these approaches have mostly used offline methods. When pronoun ambiguity resolution has been studied using online methods, the relationship between the factors has not been closely examined. The probabilistic approach (Kehler et al., 2008; Kehler and Rohde, 2013a, 2013b) is based on the assumption that pronoun ambiguity resolution is determined by the interaction of two factors – the implicit causality bias (semantic factor) and the pronoun bias (structural factor). Apart from considering both factors, pronoun ambiguity resolution is explained from the perspective of both language processing and the final choice of antecedent. Pronoun ambiguity resolution is seen as an incremental and predictive process based on the likelihood that each of the factors plays a role. Based on implicit causality verbs, speakers can predict already at the beginning of the sentence which verb argument (subject or object) is likely to be re-mentioned. When speakers encounter the pronoun (or the part of the sentence in which the pronoun is omitted), they integrate their prior predictions made based on the implicit causality bias with those made based on the pronoun bias. In pro-drop languages, the absence of the pronoun leads to a bias towards the subject antecedent, while the presence of the pronoun leads to a bias towards the object antecedent. When the biases are inconsistent (i.e., they indicate the antecedent with a different grammatical role), processing effort occurs. Final preferences are largely influenced by the implicit causality verbs and the meaning of the entire sentence but can be modulated in interaction with pronoun bias. The probabilistic approach offers a more comprehensive view of the phenomenon from (a) a linguistic point of view, as it allows us to observe the relationship between the factors that are assumed to contribute to salience in language comprehension, and (b) from a methodological point of view, as it allows us to examine how the relationship between these factors influences pronoun ambiguity resolution during sentence processing and the final interpretation of the sentence. However, the approach was initially developed for English and does not further explain the phenomenon from a cross-linguistic perspective (Rohde and Kehler, 2013b). The results of previous studies in pro-drop languages are rather inconsistent, and a systematic understanding of how the pro-drop feature contributes to pronoun ambiguity resolution is still lacking (Fedele and Kaiser, 2015; Ueno and Kehler, 2016; Mayol, 2018; Zhan, Kehler and Levy, 2020). The methodological inconsistency of previous studies calls for new studies that test the theoretical assumptions of the probabilistic approach using both online and offline methods. There are no comprehensive studies on pronoun dropping in Croatian, nor on the interplay of factors influencing this phenomenon or their effects on sentence comprehension in general. Aims and hypotheses: The aim of this study was to investigate the ambiguity resolution of the 3rd person anaphoric pronoun in Croatian, a pro-drop language, both during sentence processing and in the final choice of antecedent. Pronoun ambiguity resolution was observed within the probabilistic approach by considering the implicit causality verb bias as a semantic factor and the pronoun bias, i.e. the absence or presence of the 3rd person pronoun, as a structural factor. Several research questions were posed and hypotheses were formulated: 1) Is the pronoun ambiguity resolution during sentence processing determined by the interaction of implicit causality bias and pronoun bias in accordance with the assumed direction of these biases? (Experiment 1) 2) When does the effect of implicit causality bias occur in pronoun ambiguity resolution during sentence processing? (Experiment 2) 3) Is the final choice of antecedent determined by the interaction of implicit causality bias and pronoun bias in accordance with the assumed direction of these biases? (Experiment 3) 4) Which of the two factors – implicit causality bias or pronoun bias – has a greater effect on the final choice of antecedent? (Experiment 3) 5) Is the final choice of antecedent in the sentences without implicit causality verbs determined by the pronoun bias in accordance with the assumed direction of that bias? (Experiment 4) H1: The pronoun ambiguity resolution will be determined by the interaction of implicit causality bias and pronoun bias in accordance with the assumed direction of these biases. In sentences with subject-biased implicit causality verbs, the processing time will be significantly faster when the 3rd person pronoun is omitted. In sentences with object-biased implicit causality verbs, the processing time will be significantly faster when the 3rd person pronoun is present. H2.1: Implicit causality verbs will affect pronoun ambiguity resolution already at the beginning of sentence processing. Accordingly, sentences with implicit causality verbs will be processed faster in the earlier stages of processing than sentences with neutral verbs, namely in the part of the subordinate clause containing the 3rd person pronoun and the verb, or only the verb if the 3rd person pronoun is omitted. H2.2: The effect of implicit causality bias will persist in pronoun ambiguity resolution during sentence processing. Accordingly, sentences with implicit causality verbs will be processed faster in the later stages of language processing than sentences with neutral verbs. H3: The final choice of antecedent will be determined by the interaction of implicit causality bias and pronoun bias in accordance with the assumed direction of these biases. In sentences with subject-biased implicit causality verbs, the percentage of choosing the subject antecedent will be significantly lower when the 3rd person pronoun is present. In sentences with object- biased implicit causality verbs, the percentage of choosing the subject antecedent will be significantly higher when the 3rd person pronoun is omitted. H4: The implicit causality bias will have a stronger effect on the final choice of antecedent than the pronoun bias – regardless of whether the 3rd person pronoun is omitted or present in the sentence, there will be a significantly higher percentage of choosing an antecedent in accordance with the verb bias. H5: In the sentences without implicit causality verbs, the final choice of antecedent will be determined by the pronoun bias – the percentage of choosing the subject antecedent will be significantly higher when the 3rd person pronoun is omitted and it will be significantly lower when the 3rd person pronoun is present. Methodology: To answer the research questions, four experiments were conducted. Experiments 1 and 2 were conducted using an online method (eye-tracking reading paradigm), while Experiments 3 and 4 were conducted using an offline method (questionnaire with the forced-choice task). The online method provides insight into the course of automatic, unconscious language processing and the difficulties encountered during this process (Kaiser, 2013). The eye-tracking method is based on the assumption that there is a direct link between what the eyes fixate on and what the brain processes (Staub and Rayner, 2007). It allows us to study the time course of the effects of structural and semantic factors and the interaction of these factors before the final interpretation of the sentence. Offline methods provide insights into the speaker’s behaviour, which is the result of language processing. They allow us to study final preferences or the choice of antecedent. The same experimental stimuli were used in Experiments 1 and 3 and Experiments 2 and 4. Such an experimental design allowed us to compare eye-tracking data with questionnaire data and to draw more valid conclusions about pronoun ambiguity resolution. All relevant factors were controlled, i.e. the direction (subject- vs. object-biased) and the strength of verb bias (strong vs. weak semantic bias), the type of antecedent (proper noun), the conjunction (without causal meaning), the content of the subordinate clause (which was consistent with the implicit causality bias), the length and frequency of the regions of interest in the sentences (which were the same for all experimental stimuli). A pilot study was conducted to determine the direction and strength of verb biases (see Košutar and Matić Škorić, 2022). 86 participants completed a questionnaire containing a sentence continuation task with implicit causality verbs and other verbs. After completing the questionnaire, they selected the antecedent to which they referred in the sentence. Based on the data obtained in the pilot study, a pre-selection of verbs was made for the main study. Verb biases were calculated using the formula from E. C. Ferstl, A. Garnham and C. Manouilidou (2011). Only implicit causality verbs with a strong semantic bias (n = 20) and neutral verbs with a low semantic bias (n = 12) were included in the study. In Experiments 1 and 3, the target stimuli consisted of temporal sentences with the conjunction after. The main clauses contained implicit causality verbs (subject-biased or object- biased) and proper nouns as verb arguments. The content of the subordinate clause was consistent with the implicit causality bias in the main clause. Two factors were manipulated, and each of them had two levels (2 x 2 design) – implicit causality verb (subject- vs. object- biased) and 3rd person pronoun (present vs. absent). In Experiments 2 and 4, the target stimuli were the same sentences as in Experiments 1 and 3 but additionally contained structurally equivalent sentences with neutral verbs. Two factors were manipulated, and each of them had two levels (2 x 2 design) – verb bias (implicit causality verb vs. neutral verb) and 3rd person pronoun (present vs. absent). The experimental stimuli also contained filler sentences. Participants were adult native speakers of Croatian, who had no history of language or neurological deficits. 95 participants took part in Experiments 1 and 2 (eye-tracking) and 126 participants in Experiments 3 and 4 (questionnaire). None of them participated in the pilot study. Only participants without ocular disorders participated in the eye-tracking study. Participants were evenly distributed in terms of gender and had similar levels of education. Experiments 1 and 2 were conducted first (eye-tracking), while Experiments 3 and 4 (questionnaire) took place one month later. Eye movements were recorded with the SMI Hi- Speed View 500 in the Laboratory for Psycholinguistic Research (POLIN) at the Faculty of Education and Rehabilitation Sciences, University of Zagreb. Participants read the sentences on the screen and processing time was measured during reading. The questionnaire data were collected online using the SurveyMonkey platform. Participants were asked to read the sentences and select an antecedent. The eye-tracking data were analysed using the SMI BeGaze programme. First-pass reading time (the sum of all fixations in the region before the regressive or progressive saccade), total reading time (the sum of all fixations in the region, including re-reading the region), and the number of regressions (returns to the region already read) were measured. The first pass captures the early stages of processing, while the total reading time and regressions reflect the late stages of processing. A longer duration of fixations and a greater number of regressions to a region of interest indicate higher processing costs. Measures were calculated in the predefined regions of interest – the main clause with an implicit causality or neutral verb (1), the conjunction (2), the verb in the subordinate clause (3), and the verb complements in the subordinate clause (4). The questionnaire data were coded as 1 (subject antecedent) and 0 (object antecedent). The percentage of selecting the subject antecedent was calculated. Data were analysed using repeated measures ANOVA in SPSS version 25.0. Results: In Experiment 1 (Influence of verb bias and pronoun bias on pronoun ambiguity resolution during language processing), eye-tracking measures revealed a significant interaction between implicit causality verb bias and pronoun bias, in accordance with the assumed direction of these biases (H1 confirmed). Speakers predict an antecedent (subject or object) based on the implicit causality bias already at the beginning of the sentence. In the further processing of the sentence, their predictions are modulated by the pronoun bias (3rd person pronoun absent or present). When the pronoun bias is consistent with the verb bias, sentence processing is facilitated. In the sentences with subject-biased implicit causality verbs, processing time was significantly faster when the 3rd person pronoun was omitted. In the sentences with object-biased implicit causality verbs, processing time was significantly faster when the 3rd person pronoun was present. In experiment 2 (Influence of verb bias on pronoun ambiguity resolution during language processing), eye-tracking measures revealed an early effect of implicit causality bias (H2.1 confirmed). The effect of verb bias was found in the earlier stages of processing in the part of the subordinate clause containing the 3rd person pronoun and the verb, or only the verb if the 3rd person pronoun is omitted. Once the meaning of the implicit causality verb is available, the speaker’s attention directs to the antecedent (subject or object), which is the likely cause of the event or state in the subordinate clause. The effect of implicit causality bias was also found in later stages of sentence processing (H2.2 confirmed). Sentences with implicit causality verbs were processed significantly faster than sentences with neutral verbs in both the early and late stages of sentence processing. In Experiment 3 (Influence of verb bias and pronoun bias on the final choice of antecedent), the results showed a significant interaction between implicit causality bias and pronoun bias in the final choice of antecedent in accordance with the assumed direction of these biases (H3 confirmed). In sentences with subject-biased implicit causality verbs, the percentage of choosing the subject antecedent was significantly lower when the 3rd person pronoun was present. In sentences with object-biased implicit causality verbs, the percentage of choosing the subject antecedent was significantly higher when the 3rd person pronoun was omitted. When the verb bias and the pronoun bias did not coincide (i.e., they referred to the antecedents with different grammatical roles), speakers hesitated and relied significantly more on the verb bias (H4 confirmed). In Experiment 4 (Influence of pronoun bias on the final choice of antecedent), the results showed a significant effect of pronoun bias on the final choice of antecedent in the sentences without implicit causality verbs. In Croatian, the pronoun bias shows a complementary distribution depending on whether the pronoun is omitted in the sentence (H5 confirmed). When the 3rd person pronoun is omitted, speakers choose the subject antecedent more often. When the 3rd person pronoun is present, they more often choose the object antecedent. Conclusion: Despite numerous studies on language comprehension, including language processing as an unconscious process, it is not fully understood how speakers deal with certain parts of complex syntax. One of the insufficiently studied linguistic phenomena is the ambiguity resolution of the 3rd person anaphoric pronoun. This is the first experimental study to observe pronoun ambiguity resolution in terms of structural and semantic factors during sentence processing and in the final choice of antecedent in the less represented pro-drop languages, namely Croatian. The results showed that both structural and semantic factors influence the pronoun ambiguity resolution during sentence processing and in the final choice of antecedent. Implicit causality verbs contribute to the salience of the antecedent from the earliest stages of language processing to the final sentence interpretation. Based on verb bias, speakers can predict the antecedent very early during sentence processing. The verb bias also affects the later stages of sentence processing and influences the final choice of antecedent. However, structural features of the sentence are also important. Speakers are sensitive to pronoun bias, i.e. whether the 3rd person pronoun is omitted or not. When the pronoun is omitted, the preferred antecedent is the subject. When the pronoun is present, the preferred antecedent is the object. The results on the interaction of structural and semantic factors show that the structural and semantic levels of the sentence are closely related and that any change at the structural level leads to changes at the semantic level. These findings confirm the assumptions of functional linguistic approaches, in which syntactic phenomena are interpreted in terms of structural-semantic interaction. This is also the first study to give an insight into implicit causality verbs and their effects on sentence processing in Croatian. The results on the interaction between implicit causality bias and pronoun bias obtained by eye-tracking confirm that sentence processing is an incremental and predictive process in which speakers draw on all available information. The structural and semantic factors that make the antecedent salient influence speaker’s predictions during sentence processing, and more predictable linguistic structures are more mentally accessible and easier to process. This study focused on the Croatian language and showed that pronoun ambiguity resolution is not universal, but is determined by language-specific features and cross-linguistic differences. Thus, this study contributes to the understanding of pronoun ambiguity resolution in the context of Croatian, but also in the international context, as it provides a broader understanding of this phenomenon from a typological perspective. There is a certain overlap in pronoun ambiguity resolution preferences with other pro-drop languages, but also inconsistencies in the strength of pronoun bias. These findings point to the need to consider cross-linguistic differences and their impact on pronoun ambiguity resolution to improve existing theoretical approaches and future research. Finally, the combination of online and offline methods has provided a more comprehensive insight into pronoun ambiguity resolution. Although the results on the interaction between structural and semantic factors obtained with online and offline methods overlap, the offline method has revealed that verb bias plays a greater role in the final choice of antecedent. This shows the importance of combining different methods to better understand the complexity of linguistic phenomena.

show abstract

unclassified