MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection

Matero, Matthew; Soni, Nikita; Balasubramanian, Niranjan; Schwartz, H. Andrew

doi:10.18653/v1/2021.findings-emnlp.253

Cited by 6 publications

(4 citation statements)

References 20 publications

(7 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HaRT. Recent works (Lynn et al, 2020;Matero et al, 2021b; have highlighted the importance of incorporating author context into the message representations through the use of history and multi-level modeling. We use the Human aware Recurrent Transformer model which is built on GPT2 (Radford et al, 2019), to produce message representations that encode the latent representation of the author as well.…”

Section: Task Amentioning

confidence: 99%

Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology

2022

View full text Add to dashboard Cite

Mental distress like depression and anxiety contribute to the largest proportion of the global burden of diseases. Automated diagnosis system of such disorders, empowered by recent innovations in Artificial Intelligence, can pave the way to reduce the sufferings of the affected individuals. Development of such systems requires information-rich and balanced corpora. In this work, we introduce a novel mental distress analysis audio dataset DEPAC, labelled based on established thresholds on depression and anxiety standard screening tools. This large dataset comprises multiple speech tasks per individual, as well as relevant demographic information. Alongside, we present a feature set consisting of hand-curated acoustic and linguistic features, which were found effective in identifying signs of mental illnesses in human speech. Finally, we justify the quality and effectiveness of our proposed audio corpus and feature set in predicting depression severity by comparing the performance of baseline machine learning models built on this dataset with baseline models trained on other well-known depression corpora.

show abstract

Section: Task Amentioning

confidence: 99%

Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology

2022

View full text Add to dashboard Cite

show abstract

“…HaRT. Recent works (Lynn et al, 2020;Matero et al, 2021b;Soni et al, 2022) have highlighted the importance of incorporating author context into the message representations through the use of history and multi-level modeling. We use the Human aware Recurrent Transformer model (Soni et al, 2022) which is built on GPT2 (Radford et al, 2019), to produce message representations that encode the latent representation of the author as well.…”

Section: Task Amentioning

confidence: 99%

WWBP-SQT-lite: Multi-level Models and Difference Embeddings for Moments of Change Identification in Mental Health Forums

Ganesan¹,

Varadarajan²,

Mittal³

et al. 2022

Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology

Self Cite

View full text Add to dashboard Cite

Psychological states unfold dynamically; to understand and measure mental health at scale we need to detect and measure these changes from sequences of online posts. We evaluate two approaches to capturing psychological changes in text: the first relies on computing the difference between the embedding of a message with the one that precedes it, the second relies on a "human-aware" multi-level recurrent transformer (HaRT). The mood changes of timeline posts of users were annotated into three classes, 'ordinary,' 'switching' (positive to negative or vice versa) and 'escalations' (increasing in intensity). For classifying these mood changes, the difference-between-embeddings technique -applied to RoBERTa embeddings -showed the highest overall F1 score (0.61) across the three different classes on the test set. The technique particularly outperformed the HaRT transformer (and other baselines) in the detection of switches (F1 = .33) and escalations (F1 = .61). Consistent with the literature, the language use patterns associated with mental-health related constructs in prior work (including depression, stress, anger and anxiety) predicted both mood switches and escalations.

show abstract

“…Many of the works have focused on data from Twitter, incorporating conversational and interactional context [9,10] in order to better classify the stances of the users in a thread of tweets or simply by taking the tweets in an independent way [11][12][13][14]. The SemEval-2017 task 8 [15] proposes to use the interactional context of Twitter threads, focusing on rumouroriented stance classification, where the objective is to identify support towards a rumour and an entire statement, rather than individual target concepts.…”

Section: Introductionmentioning

confidence: 99%

“…Since then, the conversation has been integrated with graphical models that allow taking into account its dynamics [22][23][24] through the different successive speech turns of the participants. Neural networks [12,[25][26][27] fall into this type of model and can even be pre-trained for the conversation setting [10] to understand better the conversational context to analyse stances in Twitter threads.…”

Section: Introductionmentioning

confidence: 99%

Multilingual Multi-Target Stance Recognition in Online Public Consultations

Barrière¹,

Balahur

2023

Mathematics

View full text Add to dashboard Cite

Machine Learning is an interesting tool for stance recognition in a large-scale context, in terms of data size, but also regarding the topics and themes addressed or the languages employed by the participants. Public consultations of citizens using online participatory democracy platforms offer this kind of setting and are good use cases for automatic stance recognition systems. In this paper, we propose to use three datasets of public consultations, in order to train a model able to classify the stance of a citizen within a text, towards a proposal or a debate question. We studied stance detection in several contexts: using data from an online platform without interactions between users, using multilingual data from online debates that are in one language, and using data from online intra-multilingual debates, which can contain several languages inside the same unique debate discussion. We propose several baselines and methods in order to take advantage of the different available data, by comparing the results of models using out-of-dataset annotations, and binary or ternary annotations from the target dataset. We finally proposed a self-supervised learning method to take advantage of unlabelled data. We annotated both the datasets with ternary stance labels and made them available.

show abstract

MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection

Cited by 6 publications

References 20 publications

Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology

Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology

WWBP-SQT-lite: Multi-level Models and Difference Embeddings for Moments of Change Identification in Mental Health Forums

Multilingual Multi-Target Stance Recognition in Online Public Consultations

Contact Info

Product

Resources

About