Empower Sequence Labeling with Task-Aware Neural Language Model

Liu, Liyuan; Shang, Jingbo; Xu, Frank F.; Ren, Xiang; Gui, Huan; Peng, Jian; Han, Jiawei

doi:10.1609/aaai.v32i1.12006

Cited by 136 publications

(24 citation statements)

References 30 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We summarize the related methods for discontinuous NER in the following categories. Sequence-tagging-based methods In the field of NLP, NER is usually considered as a sequence tagging problem [25]- [27]. Based on well-designed features, CRF based models have achieved leading performance [28]- [30].…”

Section: Related Workmentioning

confidence: 99%

TOE: A Grid-Tagging Discontinuous NER Model Enhanced by Embedding Tag/Word Relations and More Fine-Grained Tags

Jiang

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

So far, discontinuous named entity recognition (NER) has received increasing research attention and many related methods have surged such as hypergraph-based methods, span-based methods, and sequence-to-sequence (Seq2Seq) methods, etc. However, these methods more or less suffer from some problems such as decoding ambiguity and efficiency, which limit their performance. Recently, grid-tagging methods, which benefit from the flexible design of tagging systems and model architectures, have shown superiority to adapt for various information extraction tasks. In this paper, we follow the line of such methods and propose a competitive grid-tagging model for discontinuous NER. We call our model TOE because we incorporate two kinds of Tag-Oriented Enhancement mechanisms into a state-of-the-art (SOTA) grid-tagging model that casts the NER problem into word-word relationship prediction. First, we design a Tag Representation Embedding Module (TREM) to force our model to consider not only word-word relationships but also word-tag and tag-tag relationships. Concretely, we construct tag representations and embed them into TREM, so that TREM can treat tag and word representations as queries/keys/values and utilize self-attention to model their relationships. On the other hand, motivated by the Next-Neighboring-Word (NNW) and Tail-Head-Word (THW) tags in the SOTA model, we add two new symmetric tags, namely Previous-Neighboring-Word (PNW) and Head-Tail-Word (HTW), to model more fine-grained word-word relationships and alleviate error propagation from tag prediction. In the experiments of three benchmark datasets, namely CADEC, ShARe13 and ShARe14, our TOE model pushes the SOTA results by about 0.83%, 0.05% and 0.66% in F1, demonstrating its effectiveness.

show abstract

Section: Related Workmentioning

confidence: 99%

TOE: A Grid-Tagging Discontinuous NER Model Enhanced by Embedding Tag/Word Relations and More Fine-Grained Tags

Jiang

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…However, when NER is applied to deep learning, non-linear mapping from input to output can be generated to learn much more complex and intensive features from the data than a linear model. In addition, it is effective in learning useful representations and basic elements from raw data through deep-learning-based models, so excellent performance can be expected [ 29 , 30 , 31 ]. Therefore, in this paper, we successfully perform an entity-extraction task, the second task of knowledge selection, by utilizing the deep-learning-based NER with the neural language model.…”

Section: Related Workmentioning

confidence: 99%

A Knowledge-Grounded Task-Oriented Dialogue System with Hierarchical Structure for Enhancing Knowledge Selection

Lee

Jeong

2023

Sensors

View full text Add to dashboard Cite

For a task-oriented dialogue system to provide appropriate answers to and services for users’ questions, it is necessary for it to be able to utilize knowledge related to the topic of the conversation. Therefore, the system should be able to select the most appropriate knowledge snippet from the knowledge base, where external unstructured knowledge is used to respond to user requests that cannot be solved by the internal knowledge addressed by the database or application programming interface. Therefore, this paper constructs a three-step knowledge-grounded task-oriented dialogue system with knowledge-seeking-turn detection, knowledge selection, and knowledge-grounded generation. In particular, we propose a hierarchical structure of domain-classification, entity-extraction, and snippet-ranking tasks by subdividing the knowledge selection step. Each task is performed through the pre-trained language model with advanced techniques to finally determine the knowledge snippet to be used to generate a response. Furthermore, the domain and entity information obtained because of the previous task is used as knowledge to reduce the search range of candidates, thereby improving the performance and efficiency of knowledge selection and proving it through experiments.

show abstract

“…Generate different type of elements, depending whether they represent particular instance of the specified class, or the whole class itself Deep learning driven approaches [63,64], their combinations with CRF [41,65,66…”

Section: Semantic Analysismentioning

confidence: 99%

Exploring Natural Language Processing in Model-To-Model Transformations

Danėnas

Skersys

2022

IEEE Access

View full text Add to dashboard Cite

In this paper, we explore the possibility to apply natural language processing in visual modelto-model (M2M) transformations. Therefore, we present our research results on information extraction from text labels in process models modeled using Business Process Modeling Notation (BPMN) and use case models depicted in Unified Modeling Language (UML) using the most recent developments in natural language processing (NLP). In this paper, we focus on three relevant tasks, namely, the extraction of verb/noun phrases that would be used to form relations, parsing of conjunctive/disjunctive statements, and the detection of abbreviations and acronyms. Relation extraction was attempted to solve by implementing techniques that combine state-of-the-art NLP language models with formal regular expressions grammarbased structure detection. In this paper, we perform thorough testing of the most recent state-of-the-art NLP tools (CoreNLP, Stanford Stanza, Flair, Spacy, AllenNLP, BERT, ELECTRA), as well as custom BERT-BiLSTM-CRF and ELMo-BiLSTM-CRF implementations, trained with certain data augmentations to improve performance on the most ambiguous cases; these tools are used as a foundation for building tools to extract noun and verb phrases from short text labels generally used in UML and BPMN models. Furthermore, we describe our attempts to improve these extractors by solving the abbreviation/acronym detection problem using machine learning-based detection, as well as process conjunctive and disjunctive statements, due to their relevance to performing advanced text normalization. The obtained results show that the best phrase extraction and conjunctive phrase processing performance was obtained using Stanza based implementation, yet, our trained BERT-BiLSTM-CRF outperformed it for the verb phrase detection task. Our acronym detection approach resulted in the precision of 0.78 and F1-Score of 0.73 which may also be considered quite positive. While this work was inspired by our ongoing research on partial model-to-model transformations, we believe it to be applicable in other areas requiring similar text processing capabilities as well. INDEX TERMSinformation extraction; relation extraction; acronym detection; process models; use-case models; natural language processing; model-to-model transformation This work is inspired by our previous research on NLPenhanced information extraction in model-to-model trans- formations [6, 7]. However, the need for similar solutions was also identified in other areas involving visual modeling, such as business process modeling [8,9,10]. In this paper, we address the issue of relation extraction from graphical

show abstract

Empower Sequence Labeling with Task-Aware Neural Language Model

Cited by 136 publications

References 30 publications

TOE: A Grid-Tagging Discontinuous NER Model Enhanced by Embedding Tag/Word Relations and More Fine-Grained Tags

TOE: A Grid-Tagging Discontinuous NER Model Enhanced by Embedding Tag/Word Relations and More Fine-Grained Tags

A Knowledge-Grounded Task-Oriented Dialogue System with Hierarchical Structure for Enhancing Knowledge Selection

Exploring Natural Language Processing in Model-To-Model Transformations

Contact Info

Product

Resources

About