Temporally-Informed Analysis of Named Entity Recognition

Rijhwani, Shruti; Preoţiuc-Pietro, Daniel

doi:10.18653/v1/2020.acl-main.680

Cited by 38 publications

(40 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to scenario 1, the F1 scores of the models trained on instances selected based on their trend scores are always higher than random sampling F1 scores. In addition, scenario 2, on average, works better than scenario 1, which is consistent with Rijhwani and Preotiuc-Pietro (2020). However, this setting requires the data available from all years from the very beginning.…”

Section: Resultssupporting

confidence: 72%

Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp

Chen¹,

Neves²,

Solorio³

2021

Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media

View full text Add to dashboard Cite

Performance of neural models for named entity recognition degrades over time, becoming stale. This degradation is due to temporal drift, the change in our target variables' statistical properties over time. This issue is especially problematic for social media data, where topics change rapidly. In order to mitigate the problem, data annotation and retraining of models is common. Despite its usefulness, this process is expensive and timeconsuming, which motivates new research on efficient model updating. In this paper, we propose an intuitive approach to measure the potential trendiness of tweets and use this metric to select the most informative instances to use for training. We conduct experiments on three state-of-the-art models on the Temporal Twitter Dataset. Our approach shows larger increases in prediction accuracy with less training data than the alternatives, making it an attractive, practical solution. 11 We release the code at https://github.com/ RiTUAL-UH/trending_NER.

show abstract

Section: Resultssupporting

confidence: 72%

Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp

Chen¹,

Neves²,

Solorio³

2021

Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media

View full text Add to dashboard Cite

show abstract

“…Since we have time stamps for two of our datasets we study these in greater detail. For similar studies of temporal drift, see Lukes and Søgaard (2018); Rijhwani and Preotiuc-Pietro (2020). The training data sizes are comparable (1.63-1.76M), the publisher distributions (AFP, APW, CNA, NYT or XIN) are also similar.…”

Section: A4 Computing Adversarial Splitsmentioning

confidence: 87%

We Need To Talk About Random Splits

Søgaard¹,

Ebert²,

Bastings³

et al. 2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

View full text Add to dashboard Cite

Gorman and Bedrick ( 2019) argued for using random splits rather than standard splits in NLP experiments. We argue that random splits, like standard splits, lead to overly optimistic performance estimates. We can also split data in biased or adversarial ways, e.g., training on short sentences and evaluating on long ones. Biased sampling has been used in domain adaptation to simulate real-world drift; this is known as the covariate shift assumption. In NLP, however, even worst-case splits, maximizing bias, often under-estimate the error observed on new samples of in-domain data, i.e., the data that models should minimally generalize to at test time. This invalidates the covariate shift assumption. Instead of using multiple random splits, future benchmarks should ideally include multiple, independent test sets instead; if infeasible, we argue that multiple biased splits leads to more realistic performance estimates than multiple random splits.

show abstract

“…In addition to research on improving the performance of the NER model, other experimental setups have been proposed for this task. These include domain adaptation, where a model trained on data from a source domain is used to tag data from a different target domain (Guo et al, 2009;Greenberg et al, 2018;Wang et al, 2020), temporal drift, where a model is tested on data from future time intervals (Derczynski et al, 2016;Rijhwani and Preotiuc-Pietro, 2020), cross-lingual modelling where models trained in one language are adapted to other languages (Tsai et al, 2016;Ni et al, 2017;Xie et al, 2018), identifying nested entities Lu and Roth, 2015) or high-precision NER models (Arora et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Identifying Named Entities as they are Typed

Arora

Tsai

Preoţiuc-Pietro³

2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Self Cite

View full text Add to dashboard Cite

Identifying named entities in written text is an essential component of the text processing pipeline used in applications such as text editors to gain a better understanding of the semantics of the text. However, the typical experimental setup for evaluating Named Entity Recognition (NER) systems is not directly applicable to systems that process text in real time as the text is being typed. Evaluation is performed on a sentence level assuming the end-user is willing to wait until the entire sentence is typed for entities to be identified and further linked to identifiers or coreferenced. We introduce a novel experimental setup for NER systems for applications where decisions about named entity boundaries need to be performed in an online fashion. We study how state-of-the-art methods perform under this setup in multiple languages and propose adaptations to these models to suit this new experimental setup. Experimental results show that the best systems that are evaluated on each token after its typed, reach performance within 1-5 F 1 points of systems that are evaluated at the end of the sentence. These show that entity recognition can be performed in this setup and open up the development of other NLP tools in a similar setup.

show abstract

Temporally-Informed Analysis of Named Entity Recognition

Cited by 38 publications

References 38 publications

Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp

Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp

We Need To Talk About Random Splits

Identifying Named Entities as they are Typed

Contact Info

Product

Resources

About