From Web Crawled Text to Project Descriptions: Automatic Summarizing of Social Innovation Projects

Milošević, Nikola; Marinov, Dimitar; Gök, Abdullah; Nenadić, Goran

doi:10.1007/978-3-030-23281-8_13

Cited by 2 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human evaluations have been always the gold standard to judge the quality of natural language processing (NLP) system's outputs Freitag et al, 2021;. This applies to many sub-tasks including machine translation (MT) (Han et al, 2020;Charlampidou and Gladkoff, 2022), text summarisation (Bhandari et al, 2020;Latif et al, 2009), question answering (Alrdahi et al, 2020), information extraction (Wu et al, 2022;, and prediction (Yang et al, 2009), as well as domain applications such as social media, biomedical and clinical domains knowledge representation (Milošević et al, 2019;Yang et al, 2021;Krauthammer and Nenadic, 2004). Nonetheless, human evaluations have been subject to criticisms and debates about their reliability, particularly when conducted without strictly defined procedures.…”

Section: Introductionmentioning

confidence: 99%

Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce

Gladkoff¹,

Han²,

Nenadić³

2023

Preprint

View full text Add to dashboard Cite

In natural language processing (NLP) we always rely on human judgement as the golden quality evaluation method. However, there has been an ongoing debate on how to better evaluate inter-rater reliability (IRR) levels for certain evaluation tasks, such as translation quality evaluation (TQE), especially when the data samples (observations) are very scarce. In this work, we first introduce the study on how to estimate the confidence interval for the measurement value when only one data (evaluation) point is available. Then, this leads to our example with two human-generated observational scores, for which, we introduce "Student's t-Distribution" method and explain how to use it to measure the IRR score using only these two data points, as well as the confidence intervals (CIs) of the quality evaluation. We give quantitative analysis on how the evaluation confidence can be greatly improved by introducing more observations, even if only one extra observation. We encourage researchers to report their IRR scores in all possible means, e.g. using Student's t-Distribution method whenever possible; thus making the NLP evaluation more meaningful, transparent, and trustworthy. This t-Distribution method can be also used outside of NLP fields to measure IRR level for trustworthy evaluation of experimental investigations, whenever the observational data is scarce.

show abstract

Section: Introductionmentioning

confidence: 99%

Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce

Gladkoff¹,

Han²,

Nenadić³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Considering the promising performance produced by neural word embeddings (Word2vec [23], FastText [24], and Glove [25]) in variety of NLP tasks including hierarchical text categorization [26], multi-class text document classification [27] [28], investigation of gender roles [29], nonrelevant post detection [30], topic modelling [31], automated sarcasm detection [32], synonym extraction [33], automated enrichment of lexicons for misogyny detection [34], sentiment analysis [35], automated text summarization [36], text clustering [37], measuring emotional polarity from debates [38], recommendation system [39], the paper in hand for the very first time provides Word2vec [23], FastText [24], and Glove [25] embeddings for Roman Urdu. These pretrained embeddings can be used to enhance the performance of diverse deep learning based Roman Urdu processing tasks.…”

Section: Introductionmentioning

confidence: 99%

A Precisely Xtreme-Multi Channel Hybrid Approach for Roman Urdu Sentiment Analysis

et al. 2020

View full text Add to dashboard Cite

In order to accelerate the performance of various Natural Language Processing tasks for Roman Urdu, this paper for the very first time provides 3 neural word embeddings prepared using most widely used approaches namely Word2vec, FastText, and Glove. The integrity of generated neural word embeddings is evaluated using intrinsic and extrinsic evaluation approaches. Considering the lack of publicly available benchmark datasets, it provides a first-ever Roman Urdu public dataset which consists of 3241 sentiments annotated against positive, negative, and neutral classes. To provide benchmark baseline performance over the presented dataset for Roman Urdu sentiment analysis, we adapt diverse machine learning (Support Vector Machine, Logistic Regression, Naive Bayes), deep learning (convolutional neural network, recurrent neural network), and hybrid deep learning approaches. Performance impact of generated neural word embeddings based representation is compared with other most widely used bag of words based feature representation approaches using diverse machine and deep learning classifiers. In order to improve the performance of Roman Urdu sentiment analysis, it proposes a novel precisely extreme multi-channel hybrid methodology which makes use of convolutional and recurrent neural networks along with pre-trained neural word embeddings. The proposed hybrid approach outperforms adapted machine learning approaches by the significant figure of 9% and deep learning approaches by the figure of 4% in terms of F1-score. INDEX TERMS Fast-Text, Glove, Pretrain word embeddings for Roman Urdu, Roman Urdu Sentiment Analysis, Word2Vec,

show abstract

From Web Crawled Text to Project Descriptions: Automatic Summarizing of Social Innovation Projects

Cited by 2 publications

References 17 publications

Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce

Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce

A Precisely Xtreme-Multi Channel Hybrid Approach for Roman Urdu Sentiment Analysis

Contact Info

Product

Resources

About