BERT got a Date: Introducing Transformers to Temporal Tagging

Almasian, Satya; Aumiller, Dennis; Gertz, Michael

doi:10.48550/arxiv.2109.14927

Cited by 3 publications

(4 citation statements)

References 17 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We demonstrated the effectiveness of our proposed method using a variety of pre-train models of varying sizes. Centralised (Ahmed, Lin Jerry, & Srivastava, 2022b), FedAVG (Li et al, 2019) and Embedded averages were used to evaluate BERT model (Almasian et al, 2021).…”

Section: Experimental Results and Analysismentioning

confidence: 99%

“…The active learning process is shown in Figures 1, 2, and 3. The proposed technique first trains a classifier (model) for each labelled dataset using the temporally pre-trained network (Almasian et al, 2021). The pseudo-labels are then applied to the broader pool of unlabeled data to create highly reliable pseudo-labels.…”

Section: Temporal Focus Time Attention Networkmentioning

confidence: 99%

See 1 more Smart Citation

Temporal positional lexicon expansion for federated learning based on hyperpatism detection

2022

View full text Add to dashboard Cite

Internet-based information exchange has resulted in the propagation of false and misleading information, which is highly detrimental to individuals and humankind.Due to the speed and volume of social media news production, supervised artificial intelligence algorithms require many annotated data, which is difficult, costly, and time-consuming. To address this issue, we offer a novel federated semi-supervised framework based on self-ensembling that utilizes the linguistic and stylometric information of annotated news articles and searches for hidden patterns in unlabeled data to denoise labels. Self-ensembling predicts the labels of unlabeled data by using the outcomes of network-in-training from earlier epochs. These cumulative predictions should be a stronger predictor for unknown labels than the output of the most recent training epoch; hence, they may be utilized as a substitute for the labels of unlabeled data. The approach is distinctive in collecting all of the outputs from the neural network's past training periods. It utilizes them as an unsupervised target against which to assess the current output prediction of unlabeled articles. We intend to create a dataset centred on denoising to forward the study. The dataset is mapped using (1) the shifting focus time from published news articles and (2) the semi-supervised method based on coincidence contexts for a neural contrast embedding model for learning low-dimensional continuous vectors that generate a focus time-based query in sequential news articles for temporal comprehension. The model achieved 0.83% F-measure with lexicon expansion semi-supervised learning.

show abstract

Section: Experimental Results and Analysismentioning

confidence: 99%

Section: Temporal Focus Time Attention Networkmentioning

confidence: 99%

Temporal positional lexicon expansion for federated learning based on hyperpatism detection

2022

View full text Add to dashboard Cite

show abstract

“…Recently, there has been an increase in attention to the infusion of temporal information into contextualized embeddings with the goal of improving prediction tasks. However, the focus has primarily been on temporal relation prediction (Liu et al, 2019 ; Guan and Devarakonda, 2020 ) with some recent work on temporal tagging in the general domain (Almasian et al, 2022 ) and prediction of clinical outcomes (Pang et al, 2021 ). As of yet, there are no publications utilizing contextualized embeddings for the task of temporal disambiguation of relative temporal expressions.…”

Section: Methodsmentioning

confidence: 99%

Temporal disambiguation of relative temporal expressions in clinical texts

Olex

McInnes²

2022

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

Temporal expression recognition and normalization (TERN) is the foundation for all higher-level temporal reasoning tasks in natural language processing, such as timeline extraction, so it must be performed well to limit error propagation. Achieving new heights in state-of-the-art performance for TERN in clinical texts requires knowledge of where current systems struggle. In this work, we summarize the results of a detailed error analysis for three top performing state-of-the-art TERN systems that participated in the 2012 i2b2 Clinical Temporal Relation Challenge, and compare our own home-grown system Chrono to identify specific areas in need of improvement. Performance metrics and an error analysis reveal that all systems have reduced performance in normalization of relative temporal expressions, specifically in disambiguating temporal types and in the identification of the correct anchor time. To address the issue of temporal disambiguation we developed and integrated a module into Chrono that utilizes temporally fine-tuned contextual word embeddings to disambiguate relative temporal expressions. Chrono now achieves state-of-the-art performance for temporal disambiguation of relative temporal expressions in clinical text, and is the only TERN system to output dual annotations into both TimeML and SCATE schemes.

show abstract

“…Seq-to-Seq: Previous works have reported sequence to sequence approach using the encoder-decoder architecture as an alternative for the sequence tagging scheme (Almasian, Aumiller, and Gertz 2021;Yan et al 2021). Following these works, we use a transformer-based generative framework to auto-regressively decode the acronyms and long-forms present in input sentence.…”

Section: Baselinesmentioning

confidence: 99%

CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction

Kannen¹,

Sheth²,

Chandra³

et al. 2021

Preprint

View full text Add to dashboard Cite

Acronyms and long-forms are commonly found in research documents, more so in documents from scientific and legal domains. Many acronyms used in such documents are domain-specific, and are very rarely found in normal text corpora. Owing to this, transformer-based NLP models often detect OOV (Out of Vocabulary) for acronym tokens, especially for non-English languages, and their performance suffers while linking acronyms to their long forms during extraction. Moreover, pre-trained transformer models like BERT are not specialized to handle scientific and legal documents. With these points being the overarching motivation behind this work, we propose a novel framework CABACE: Character-Aware BERT for ACronym Extraction, which takes into account character sequences in text, and is adapted to scientific and legal domains by masked language modelling. We further use an objective with an augmented loss function, adding max loss and mask loss terms to the standard cross-entropy loss for training CABACE. We further leverage pseudo labelling and adversarial data generation to improve the generalizability of the framework. Experimental results prove the superiority of the proposed framework in comparison to various baselines. Additionally, we show that the proposed framework is better suited than baseline models for zero-shot generalization to non-English languages, thus reinforcing the effectiveness of our approach. Our team BacKGProp secured the highest scores on the French dataset, second-highest on Danish and Vietnamese, and third-highest in English-Legal dataset on the global leaderboard for the acronym extraction (AE) shared task at SDU AAAI-22.

show abstract

BERT got a Date: Introducing Transformers to Temporal Tagging

Cited by 3 publications

References 17 publications

Temporal positional lexicon expansion for federated learning based on hyperpatism detection

Temporal positional lexicon expansion for federated learning based on hyperpatism detection

Temporal disambiguation of relative temporal expressions in clinical texts

CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction

Contact Info

Product

Resources

About