Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition

Abstract

Interpretable rationales for model predictions play a critical role in practical applications. In this study, we develop models possessing interpretable inference process for structured prediction. Specifically, we present a method of instance-based learning that learns similarities between spans. At inference time, each span is assigned a class label based on its similar spans in the training set, where it is easy to understand how much each training instance contributes to the predictions. Through empirical analysis on named entity recognition, we demonstrate that our method enables to build models that have high interpretability without sacrificing performance.

SpanNER: Named Entity Re-/Recognition as Span Prediction

“…Although this is not the first work that formulates NER as a span prediction problem (Jiang et al, 2020;Ouchi et al, 2020;Li et al, 2020;Mengge et al, 2020), we contribute by (1) exploring how different design choices influence the performance of SPANNER and (2) interpreting complementary strengths between SEQLAB and SPANNER with different design choices. In what follows, we first detail span prediction-based NER systems with the vanilla configuration and proposed advanced featurization.…”

“…NER as Different Tasks Although NER is commonly formulated as a sequence labeling task (Chiu and Nichols, 2015;Huang et al, 2015;Ma and Hovy, 2016;Lample et al, 2016;Akbik et al, 2018;Peters et al, 2018;Devlin et al, 2018;Xia et al, 2019;Akbik et al, 2019;Luo et al, 2020;Lin et al, 2020), recently other new forms of frameworks have been explored and have shown impressive results. For example, (Jiang et al, 2020;Ouchi et al, 2020; shift NER from tokenlevel tagging to span-level prediction task while (Li et al, 2020;Mengge et al, 2020) conceptualize it as reading comprehension task. In this work we aim to interpret the complementarity between sequence labeling and span prediction.…”

“…The rapid evolution of neural architectures (Kalchbrenner et al, 2014a;Kim, 2014;Hochreiter and Schmidhuber, 1997) and large pre-trained models (Devlin et al, 2019;Lewis et al, 2020) not only drive the state-of-the-art performance of many NLP tasks (Devlin et al, 2019;Liu and Lapata, 2019) to a new level but also change the way how researchers formulate the task. For example, recent years have seen frequent paradigm shifts for the task of named entity recognition (NER) from token-level tagging, which conceptualize NER as a sequence labeling (SEQLAB) task (Chiu and Nichols, 2015;Huang et al, 2015;Ma and Hovy, 2016;Lample et al, 2016;Akbik et al, 2018;Peters et al, 2018;Devlin et al, 2018;Xia et al, 2019;Luo et al, 2020;Lin et al, 2020;, to span-level prediction (SPANNER) (Li et al, 2020;Mengge et al, 2020;Jiang et al, 2020;Ouchi et al, 2020;, which regards NER either as question answering (Li et al, 2020;Mengge et al, 2020), span classification (Jiang et al, 2020;Ouchi et al, 2020;Yamada et al, 2020), and dependency parsing tasks .…”

Larger-Context Tagging: When and Why Does It Work?

“…Regarding sentence encoders, recurrent neural nets (Huang et al, 2015;Chiu and Nichols, 2015 Lample et al, 2016;Lin et al, 2020) and convolutional neural nets (Strubell et al, 2017;Yang et al, 2018;Fu et al, 2020a) were widely used while transformer were also studied to get sentential representations (Yan et al, 2019;Yu et al, 2020). Some recent works consider the NER as a span classification Jiang et al, 2019;Mengge et al, 2020;Ouchi et al, 2020) task, unlike most works that view it as a sequence labeling task. To capture morphological information, some previous works introduced a character or subword-aware encoders with unsupervised pre-trained knowledge (Peters et al, 2018;Akbik et al, 2018;Devlin et al, 2018;Akbik et al, 2019;Yang et al, 2019;Lan et al, 2019).…”

Instance-Based Neural Dependency Parsing

“…Compared to image recognition, there are much fewer studies on deep metric learning in natural language processing (NLP). As a few exceptions, Wiseman and Stratos (2019) and Ouchi et al (2020) developed neural models that have an instance-based inference process for sequence labeling tasks. They reported that their models have high explainability without sacrificing the prediction accuracy.…”