Cross-Policy Compliance Detection via Question Answering

Abstract

Policy compliance detection is the task of ensuring that a scenario conforms to a policy (e.g. a claim is valid according to government rules or a post in an online platform conforms to community guidelines). This task has been previously instantiated as a form of textual entailment, which results in poor accuracy due to the complexity of the policies. In this paper we propose to address policy compliance detection via decomposing it into question answering, where questions check whether the conditions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label. Despite the initial upfront annotation cost, we demonstrate that this approach results in better accuracy, especially in the cross-policy setup where the policies during testing are unseen in training. In addition, it allows us to use existing question answering models pre-trained on existing large datasets. Finally, it explicitly identifies the information missing from a scenario in case policy compliance cannot be determined. We conduct our experiments using a recent dataset consisting of government policies, which we augment with expert annotations and find that the cost of annotating question answering decomposition is largely offset by improved interannotator agreement and speed.

Policy Compliance Detection via Expression Tree Inference

“…Our dataset is an extension of the QA4PC dataset for cross-policy compliance detection (Saeidi et al, 2021). We describe our dataset annotation process, and we also present our analysis, including key statistics, related to the dataset.…”

“…Reasoning over policies expressed in natural language is an important task in machine comprehension (Zhong et al, 2020). The problem of determining the compliance of scenarios to written policies or other legal frameworks, referred to as policy compliance detection (PCD) (Saeidi et al, 2021) has a wide range of applications, from the legal domain, e.g. statutory law (Holzenberger and Van Durme, 2021) to the issue of content moderation on the social network websites (Pavlopoulos et al, 2017).…”

“…More recent work decomposes the policy into constituent propositions reformulated as questions, and uses a question answering (QA) model to obtain answers to each of these questions with respect to the scenario (Saeidi et al, 2021). Finally, an expression tree in propositional logic is employed to obtain a final answer.…”

“…Finally, an expression tree in propositional logic is employed to obtain a final answer. However, Saeidi et al (2021) assumes expression trees are provided by experts. This limitation makes the approach less straightforward to port to new domains and policies.…”

“…This results in a label prediction that is faithful to the answers to these questions, thus offering improved explainability. The expression tree summarises the constraints presented in the policy (Saeidi et al, 2021) as shown in Figure 1.…”