FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

Abstract

We present a benchmark suite of four datasets for evaluating the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Switzerland, and China), five languages (English, German, French, Italian and Chinese) and fairness across five attributes (gender, age, region, language, and legal area). In our experiments, we evaluate pretrained language models using several grouprobust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP.

Systematic Evaluation of Predictive Fairness

“…A similar effect is also observed by Baldini et al (2022). Chalkidis et al (2022) examine the effectiveness of debiasing methods over a multi-lingual benchmark dataset consisting of four subsets of legal documents, covering five languages and various sensitive attributes. They find that methods aim-ing to improve worse-case performance tend to fail in more realistic settings, where both target label and protected attribute distributions vary over time.…”

“…Beyond the standard definitions of fairness, a number of studies have examined the effectiveness of various debiasing methods in additional settings (Gonen and Goldberg, 2019;Meade et al, 2021;Lamba et al, 2021;Baldini et al, 2022;Chalkidis et al, 2022). For example, Meade et al (2021) not only examine the effectiveness of various debiasing methods but also measure the impact of debiasing methods on a model's language modeling ability and downstream task performance.…”

Rawlsian AI fairness loopholes

“…AI/NLP researchers have uniformly adopted a Rawlsian notion of fairness. This is reflected in the by now common practice of citing Rawls when mentioning fairness [3][4][5][6][7]. Fairness plays a central role in the philosophy of John Rawls.…”

“…6 Welfare, like 'benefit', is performance as measured by the go-to performance metric. 7 Rawlsian fairness thus becomes maximizing the performance on data sampled from the group on which performance is currently lowest. Many algorithms have therefore been developed to maximize performance on the groups with the worst performance.…”

“…As[9] note, the first option is not really on the table anyway 6. Such groups are sometimes referred to as categories in social science research[10] 7. Most AI/NLP tasks come with multiple performance metrics, and it is often common practice to average across several metrics 8.…”