The Unreasonable Effectiveness of the Baseline: Discussing SVMs in Legal Text Classification

Clavié, Benjamin; Alphonsus, Marc

doi:10.3233/faia210317

Cited by 8 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This means that for any given word in a text segment, its neighboring words to both the left and right are examined so that the context of the word is well understood. These representations lend themselves to high performance in text classification tasks when compared with traditional approaches using SVMs, for example [52,53]. We used the Simple Transformers software library [54] to deploy LMs.…”

Section: Methodsmentioning

confidence: 99%

Enabling Early Health Care Intervention by Detecting Depression in Users of Web-Based Forums using Language Models: Longitudinal Analysis and Evaluation

Owen¹,

Antypas²,

Hassoulas³

et al. 2023

JMIR AI

View full text Add to dashboard Cite

Background Major depressive disorder is a common mental disorder affecting 5% of adults worldwide. Early contact with health care services is critical for achieving accurate diagnosis and improving patient outcomes. Key symptoms of major depressive disorder (depression hereafter) such as cognitive distortions are observed in verbal communication, which can also manifest in the structure of written language. Thus, the automatic analysis of text outputs may provide opportunities for early intervention in settings where written communication is rich and regular, such as social media and web-based forums. Objective The objective of this study was 2-fold. We sought to gauge the effectiveness of different machine learning approaches to identify users of the mass web-based forum Reddit, who eventually disclose a diagnosis of depression. We then aimed to determine whether the time between a forum post and a depression diagnosis date was a relevant factor in performing this detection. Methods A total of 2 Reddit data sets containing posts belonging to users with and without a history of depression diagnosis were obtained. The intersection of these data sets provided users with an estimated date of depression diagnosis. This derived data set was used as an input for several machine learning classifiers, including transformer-based language models (LMs). Results Bidirectional Encoder Representations from Transformers (BERT) and MentalBERT transformer-based LMs proved the most effective in distinguishing forum users with a known depression diagnosis from those without. They each obtained a mean F1-score of 0.64 across the experimental setups used for binary classification. The results also suggested that the final 12 to 16 weeks (about 3-4 months) of posts before a depressed user’s estimated diagnosis date are the most indicative of their illness, with data before that period not helping the models detect more accurately. Furthermore, in the 4- to 8-week period before the user’s estimated diagnosis date, their posts exhibited more negative sentiment than any other 4-week period in their post history. Conclusions Transformer-based LMs may be used on data from web-based social media forums to identify users at risk for psychiatric conditions such as depression. Language features picked up by these classifiers might predate depression onset by weeks to months, enabling proactive mental health care interventions to support those at risk for this condition.

show abstract

Section: Methodsmentioning

confidence: 99%

Enabling Early Health Care Intervention by Detecting Depression in Users of Web-Based Forums using Language Models: Longitudinal Analysis and Evaluation

Owen¹,

Antypas²,

Hassoulas³

et al. 2023

JMIR AI

View full text Add to dashboard Cite

show abstract

“…The introduction of large PLMs such as BERT (Devlin et al, 2018) has led to new SOTA performance in many NLP domains in recent years. In the legal domain, using domain-specific PLMs for simpler tasks such as text classification has only shown small improvements (Clavi'e and Alphonsus, 2021;Chalkidis et al, 2020). However, bigger gains were achieved for more complex tasks (Zheng et al, 2021).…”

Section: Logical Relations In Amrmentioning

confidence: 99%

Can AMR Assist Legal and Logical Reasoning?

Schrack,

Cui,

López

et al. 2022

Findings of the Association for Computational Linguistics: EMNLP 2022

View full text Add to dashboard Cite

Meaning Representation (AMR) has been shown to be useful for many downstream tasks. In this work, we explore the use of AMR for legal and logical reasoning. Specifically, we investigate if AMR can help capture logical relationships on multiple choice question answering (MCQA) tasks. We propose neural architectures that utilize linearised AMR graphs in combination with pre-trained language models. While these models are not able to outperform text-only baselines, they correctly solve different instances than the text models, suggesting complementary abilities. Error analysis further reveals that AMR parsing quality is the most prominent challenge, especially regarding inputs with multiple sentences. We conduct a theoretical analysis of how logical relations are represented in AMR and conclude it might be helpful in some logical statements but not for others. 1

show abstract

“…2 However, the de facto interpretation of precedent in today's legal NLP landscape focuses only on positive outcomes. Several researchers have shown that a simple model can achieve very high performance for such formulation of the outcome prediction task (Aletras et al, 2016;Chalkidis et al, 2019;Clavié and Alphonsus, 2021;Chalkidis et al, 2021b), a finding that has been replicated for a number of jurisdictions .…”

Section: Introductionmentioning

confidence: 98%

On the Role of Negative Precedent in Legal Outcome Prediction

Valvoda

Cotterell

Teufel

2023

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Every legal case sets a precedent by developing the law in one of the following two ways. It either expands its scope, in which case it sets positive precedent, or it narrows it, in which case it sets negative precedent. Legal outcome prediction, the prediction of positive outcome, is an increasingly popular task in AI. In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction. We discover an asymmetry in existing models’ ability to predict positive and negative outcomes. Where the state-of-the-art outcome prediction model we used predicts positive outcomes at 75.06 F1, it predicts negative outcomes at only 10.09 F1, worse than a random baseline. To address this performance gap, we develop two new models inspired by the dynamics of a court process. Our first model significantly improves positive outcome prediction score to 77.15 F1 and our second model more than doubles the negative outcome prediction performance to 24.01 F1. Despite this improvement, shifting focus to negative outcomes reveals that there is still much room for improvement for outcome prediction models. https://github.com/valvoda/Negative-Precedent-in-Legal-Outcome-Prediction

show abstract

The Unreasonable Effectiveness of the Baseline: Discussing SVMs in Legal Text Classification

Cited by 8 publications

References 11 publications

Enabling Early Health Care Intervention by Detecting Depression in Users of Web-Based Forums using Language Models: Longitudinal Analysis and Evaluation

Enabling Early Health Care Intervention by Detecting Depression in Users of Web-Based Forums using Language Models: Longitudinal Analysis and Evaluation

Can AMR Assist Legal and Logical Reasoning?

On the Role of Negative Precedent in Legal Outcome Prediction

Contact Info

Product

Resources

About