Explainable Text Classification in Legal Document Review A Case Study of Explainable Predictive Coding

Abstract

In today's legal environment, lawsuits and regulatory investigations require companies to embark upon increasingly intensive data-focused engagements to identify, collect and analyze large quantities of data. When documents are staged for reviewwhere they are typically assessed for relevancy or privilegethe process can require companies to dedicate an extraordinary level of resources, both with respect to human resources, but also with respect to the use of technology-based techniques to intelligently sift through data. Companies regularly spend millions of dollars producing 'responsive' electronically-stored documents for these types of matters. For several years, attorneys have been using a variety of tools to conduct this exercise, and most recently, they are accepting the use of machine learning techniques like text classification (referred to as predictive coding in the legal industry) to efficiently cull massive volumes of data to identify responsive documents for use in these matters. In recent years, a group of AI and Machine Learning researchers have been actively researching Explainable AI. In an explainable AI system, actions or decisions are human understandable. In typical legal 'document review' scenarios, a document can be identified as responsive, as long as one or more of the text snippets (small passages of text) in a document are deemed responsive. In these scenarios, if predictive coding can be used to locate these responsive snippets, then attorneys could easily evaluate the model's document classification decision. When deployed with defined and explainable results, predictive coding can drastically enhance the overall quality and speed of the document review process by reducing the time it takes to review documents. Moreover, explainable predictive coding provides lawyers with greater confidence in the results of that supervised learning task. The authors of this paper propose the concept of explainable predictive coding and simple explainable predictive coding methods to locate responsive snippets within responsive documents. We also report our preliminary experimental results using the data from an actual legal matter that entailed this type of document review. The purpose of this paper is to demonstrate the feasibility of explainable predictive coding in the context of professional services in the legal space. Keywords-machine learning, text categorization, explainable AI, predictive coding, explainable predictive coding, legal document reviewI.

A comparison of instance-level counterfactual explanation algorithms for behavioral and textual data: SEDC, LIME-C and SHAP-C

“…Text classification is ubiquitous in business and government (Provost and Fawcett 2013). Example applications are automatic identification of spam emails (Attenberg et al 2009), objectionable web content detection (Martens and Provost 2014) and legal document classification (Chhatwal et al 2018), just to name a few.…”

Metafeatures-based Rule-Extraction for Classifiers on Behavioral and Textual Data

“…Moving to the stability of the explanations over different bootstraps, we observe from Table 3 that, overall, the rules with DDMF are also more stable (8 wins versus 1) 13 , and that this difference in stability is statistically significant at a 5% significance level. This may be expected -but whether it is true is an empirical question -to be due to the higher coverage of the DDMF compared to the FG features.…”

“…Textual data are also increasingly available and used. Example text-based applications are automatic identification of spam emails [5], objectionable web content detection [48] and legal document classification [13], just to name a few. Behavioral and textual data are very high-dimensional compared to traditional data, which is primarily structured in a numeric format and is relatively low-dimensional [52,19,50].…”

“…However, for simplicity, we only used fidelity for all data sets 13. The relatively large difference in stability for the Movielens100 data is most likely because of the difference in optimal tree depth (1 for FG and 3 for DDMF).…”

Analysing Predictive Coding Algorithms for Document Review

“…The superiority of their solution over existing methods (Brinker and SVMactive) for the experiment is supported by findings on a series of large-scale real-life legal document collections. By giving responsive snippets justifying the use of predictive coding, the authors believe explainable AI has the potential to considerably enhance the application of text categorization in legal document review problems [3].…”