Unifying Privacy Policy Detection

Abstract

Privacy policies have become a focal point of privacy research. With their goal to reflect the privacy practices of a website, service, or app, they are often the starting point for researchers who analyze the accuracy of claimed data practices, user understanding of practices, or control mechanisms for users. Due to vast differences in structure, presentation, and content, it is often challenging to extract privacy policies from online resources like websites for analysis. In the past, researchers have relied on scrapers tailored to the specific analysis or task, which complicates comparing results across different studies. To unify future research in this field, we developed a toolchain to process website privacy policies and prepare them for research purposes. The core part of this chain is a detector module for English and German, using natural language processing and machine learning to automatically determine whether given texts are privacy or cookie policies. We leverage multiple existing data sets to refine our approach, evaluate it on a recently published longitudinal corpus, and show that it contains a number of misclassified documents. We believe that unifying data preparation for the analysis of privacy policies can help make different studies more comparable and is a step towards more thorough analyses. In addition, we provide insights into common pitfalls that may lead to invalid analyses.

A Bilingual Longitudinal Analysis of Privacy Policies Measuring the Impacts of the GDPR and the CCPA/CPRA

“…For all corpora, we applied the best practices for privacy policy preprocessing identified in our earlier work [34]. Following these, we used the Boilerpipe text extractor with the NumWordsRules-Extractor setting [38] to obtain the plain text of privacy policies from web pages, determined the languages of the texts by applying a majority voting scheme on the results of multiple language detection libraries, and identified non-privacy policies by applying trained classifiers [34] that achieved F1 scores of 99.1 % and 99.8 % for English and German. For each data collection time point, we manually inspected a random sample of 10 % of the downloaded policies and the final output for correctness and found no issues.…”

Evolution of Composition, Readability, and Structure of Privacy Policies over Two Decades

“…Ambiguity is further introduced by the need to have complete information in these policies [13,40]. In addition, many users need help retrieving specific information from a policy [28], and sometimes locate where the policy is mentioned in a website [30,32,34]. Most of these prior works point to critical challenges of privacy policies; however, they focused on analyzing a single snapshot of the policies.…”

Comparing Large-Scale Privacy and Security Notifications

“…A growing body of literature has tackled the problem to automatically find privacy policies on websites and download them for further analysis. Hosseini et al [28] discuss and evaluate different approaches and identify best practices. Recent research has also shown growing interest in (cookie) consent notices.…”

“…Thus, any website collecting such information must have a privacy policy that explains the use of visitors' personal data. To determine if a website had a privacy policy, we followed best practices identified by Hosseini et al [28] and searched for privacy policy specific words in and around HTML link tags. For this, we extended a list of common words for privacy policy links, terms-of-service, and contact pages from a recent study [55] to cover all official EU languages.…”