Transformers in the Real World: A Survey on NLP Applications

Patwardhan, Narendra; Marrone, Stefano; Sansone, Carlo

doi:10.3390/info14040242

Cited by 36 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Self-attention, a groundbreaking mechanism for deep learning, has ushered in transformative advancements in NLP and CV [35]. In NLP, large language models like BERT and GPT-4, built on self-attention, have excelled in language tasks due to their ability to capture context and dependencies in text [36]. In CV, the vision transformer architecture and its variants leverage self-attention to process images by dividing them into patches and applying this mechanism to them [37].…”

Section: Discussionmentioning

confidence: 99%

Cross-attention learning enables real-time nonuniform rotational distortion correction in OCT

Yang,

Zhang,

Zhang

et al. 2023

Preprint

View full text Add to dashboard Cite

Nonuniform rotational distortion (NURD) correction is vital for endoscopic optical coherence tomography (OCT) imaging and its functional extensions, such as angiography and elastography. Current NURD correction methods require time-consuming feature tracking/registration or cross-correlation calculations and thus sacrifice temporal resolution. Here we propose a cross-attention learning method for the NURD correction in OCT. Our method is inspired by the recent success of the self-attention mechanism in natural language processing and computer vision. By leveraging its ability to model long-range dependencies, we can directly obtain the correlation between OCT A-lines at any distance, thus accelerating the NURD correction. We develop an end-to-end stacked cross-attention network and design three types of optimization constraints. We compare our method with two traditional feature-based methods and a CNN-based method on two publicly-available endoscopic OCT datasets. We further verify the NURD correction performance of our method on 3D stent reconstruction using a home-built endoscopic OCT system. Our method achieves a ×3 speedup to real time (26±3 fps), and superior correction performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Cross-attention learning enables real-time nonuniform rotational distortion correction in OCT

Yang,

Zhang,

Zhang

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The advantages of the abstractive approach can remove words that are considered unimportant, while the extractive approach performs summaries based on different phrases in the input data [34]. In the Transformer-based Text-to-Text Transfer Transformer or T5 model, a lot of text summaries are carried out with an abstract approach such as that done by Patwardhan et al [35], Cheng and Yu [36], and Mars [37] which goes through several stages such as tokenization, formation of input-output data, Pretraining and Fine-Tuning, Encoder-Decoder Transformation, and text generation.…”

Section: Related Workmentioning

confidence: 99%

Enhancing Text Summarization with a T5 Model and Bayesian Optimization

Lubis,

Safitri,

Irvan

et al. 2023

RIA

View full text Add to dashboard Cite

At present the habits and interests of individuals in obtaining information by reading large amounts of information have changed at the stage of reading information more concisely, but these changes have challenges such as the nature of the data which is still unstructured making it difficult to summarize text. This study applies a data cleaning process with text processing and manually annotates to divide the data into summary data and text data so that it can be used for the process of implementing the T5 model and Bayesian optimization. In the implementation of Bayesian optimization using the prior distribution and likelihood parameters. In implementing the T5 model there will be several stages such as processing training and test data then Decodification and Post-Processing processes. The results of this study were obtained using the ROUGE evaluation technique which resulted in an increased evaluation value. The T5 model produces a ROUGE 1 value with an average value of 0.42, ROUGE-2 has a value of 0.55 and ROUGE-L has a value of 0.46 while applying Bayesian optimization produces a ROUGE-1 evaluation with an average value of 0.53 ROUGE-2 has a value of 0.55 and ROUGE-L has a value of 0.59.

show abstract

“…Therefore, the rest of the case studies are concerned with machine learning and a deep learning-based efficient approach for fake news detection by means of various state-of-the-art approaches. Deep learning has contributed a lot recently in many fields such as pattern analysis and artificial intelligence [31][32][33], with important applications in fake news detection model development [34][35][36][37][38]. However, deep learning has two major disadvantages: the first disadvantage is the overfitting problem most of the time and the second one is that it takes a lot of time to model the underlying data.…”

Section: Machine Learning For Fake News Detection 41 Overviewmentioning

confidence: 99%

The PolitiFact-Oslo Corpus: A New Dataset for Fake News Analysis and Detection

Põldvere,

Uddin,

Thomas

2023

Information

View full text Add to dashboard Cite

This study presents a new dataset for fake news analysis and detection, namely, the PolitiFact-Oslo Corpus. The corpus contains samples of both fake and real news in English, collected from the fact-checking website PolitiFact.com. It grew out of a need for a more controlled and effective dataset for fake news analysis and detection model development based on recent events. Three features make it uniquely placed for this: (i) the texts have been individually labelled for veracity by experts, (ii) they are complete texts that strictly correspond to the claims in question, and (iii) they are accompanied by important metadata such as text type (e.g., social media, news and blog). In relation to this, we present a pipeline for collecting quality data from major fact-checking websites, a procedure which can be replicated in future corpus building efforts. An exploratory analysis based on sentiment and part-of-speech information reveals interesting differences between fake and real news as well as between text types, thus highlighting the importance of adding contextual information to fake news corpora. Since the main application of the PolitiFact-Oslo Corpus is in automatic fake news detection, we critically examine the applicability of the corpus and another PolitiFact dataset built based on less strict criteria for various deep learning-based efficient approaches, such as Bidirectional Long Short-Term Memory (Bi-LSTM), LSTM fine-tuned transformers such as Bidirectional Encoder Representations from Transformers (BERT) and RoBERTa, and XLNet.

show abstract

Transformers in the Real World: A Survey on NLP Applications

Cited by 36 publications

References 21 publications

Cross-attention learning enables real-time nonuniform rotational distortion correction in OCT

Cross-attention learning enables real-time nonuniform rotational distortion correction in OCT

Enhancing Text Summarization with a T5 Model and Bayesian Optimization

The PolitiFact-Oslo Corpus: A New Dataset for Fake News Analysis and Detection

Contact Info

Product

Resources

About