Twitter Universal Dependency Parsing for African-American and Mainstream American English

Abstract

Due to the presence of both Twitterspecific conventions and non-standard and dialectal language, Twitter presents a significant parsing challenge to current dependency parsing tools. We broaden English dependency parsing to handle social media English, particularly social media African-American English (AAE), by developing and annotating a new dataset of 500 tweets, 250 of which are in AAE, within the Universal Dependencies 2.0 framework. We describe our standards for handling Twitter-and AAE-specific features and evaluate a variety of crossdomain strategies for improving parsing with no, or very little, in-domain labeled data, including a new data synthesis approach. We analyze these methods' impact on performance disparities between AAE and Mainstream American English tweets, and assess parsing accuracy for specific AAE lexical and syntactic features. Our annotated data and a parsing model are available at: http://slanglab.cs.umass.edu/ TwitterAAE/.

A Survey of Race, Racism, and Anti-Racism in NLP

“…An equally important consideration, in addition to whom the data describes is who authored the data. For example, Blodgett et al (2018) show that parsing systems trained on White Mainstream American English perform poorly on African American English (AAE). 5 In a more general example, Wikipedia has become a popular data source for many NLP tasks.…”

“…• Annotation schema Returning to Blodgett et al (2018), this work defines new parsing standards for formalisms common in AAE, demonstrating how parsing labels themselves were not designed for racialized language varieties. • Annotation instructions Sap et al (2019) show that annotators are less likely to label tweets using AAE as offensive if they are told the likely language varieties of the tweets.…”

“…While observed race is often appropriate for examining discrimination and some types of disparities, it is impossible to assess potential harms and benefits of NLP systems without assessing their performance over text generated by and directed to people of different races. The corpus from Blodgett et al (2016) does serve as a starting point and forms the basis of most current work assessing performance gaps in NLP models (Sap et al, 2019;Blodgett et al, 2018;Xia et al, 2020;Xu et al, 2019;Groenwold et al, 2020), but even this corpus is explicitly not intended to infer race.…”

Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics

“…Table 2 expresses 22 metrics from the literature as instances of our generalized metrics from Section 3. The presented metrics span a number of NLP tasks, including text classification (Dixon et al, 2018;Garg et al, 2019;Borkan et al, 2019;Prabhakaran et al, 2019), relation extraction (Gaut et al, 2020), text generation (Huang et al, 2020a) and dependency parsing (Blodgett et al, 2018). We arrive at this list by reviewing 146 papers that study bias from the survey of Blodgett et al…”

Universal Dependencies for Urdu Noisy Text

“…They evaluated the performance of two parsers on this dataset and found that their performance lagged significantly in comparison to their performance on the Italian UD Treebank. A new dataset of 500 tweets within the framework of UD 2.0 was developed and annotated by [5], out of which 250 tweets are in African American English. TWEEBANK V2 was developed by [13] by completely labelling TWEEBANK V1 according to UD 2.0 along with additionally sampled tweets, for a total of 3,550 tweets.…”