RelDiff: Enriching Knowledge Graph Relation Representations for Sensitivity Classification

Abstract

The relationships that exist between entities can be a reliable indicator for classifying sensitive information, such as commercially sensitive information. For example, the relation person-IsDirectorOf-company can indicate whether an individual's salary should be considered as sensitive personal information. Representations of such relations are often learned using a knowledge graph to produce embeddings for relation types, generalised across different entity-pairs. However, a relation type may or may not correspond to a sensitivity depending on the entities that participate to the relation. Therefore, generalised relation embeddings are typically insufficient for classifying sensitive information. In this work, we propose a novel method for representing entities and relations within a single embedding to better capture the relationship between the entities. Moreover, we show that our proposed entity-relation-entity embedding approach can significantly improve (McNemar's test, p < 0.05) the effectiveness of sensitivity classification, compared to classification approaches that leverage relation embedding approaches from the literature (0.426 F 1 vs 0.413 F 1 ).

The Role of Latent Semantic Categories and Clustering in Enhancing the Efficiency of Human Sensitivity Review

“…In particular, we first describe: (1) the document collection used in the studies and for training clustering approaches, (2) the specific clustering approaches that we evaluate, (3) selection of the appropriate number of clusters in the collection. Sensitivity Collection: To train the clustering approaches we use a collection (GovSensitivity [16]) of 3801 government documents (502 sensitive) that are annotated at document-level and sentence-level by government sensitivity reviewers for two FOI sensitivities, i.e, "Personal Information" and "International Relations". In the user studies we use passages of the documents instead of the documents itself to reduce the complexity in reviewing large documents.…”

“…We deployed an SVM text classification approach as described in [13] to classify the documents as either sensitive or non-sensitive. To train the classifier, we used a 5-fold cross validation with stratified samples of the GovSensitivity collection as described in [16]. The effectiveness of the learned classifier was 0.733 BAC.…”

“…On a collection of government documents with real sensitivities (GovSensitivity [16]), our user studies show that reviewing documents in semantic clusters can significantly improve the reviewing speed by 15.65% (T-Test, 𝑝 < 0.05). Furthermore, we show that our proposed review prioritisation strategy that leverages document metadata attributes for ranking clusters with finer grained sensitivity proportions can significantly improve the hourly openness by 37.99% and openness as a function of time (area under the curve) by 23.78% (T-Test, 𝑝 < 0.05).…”