Breaking the Closed World Assumption in Text Classification

Abstract

Existing research on multiclass text classification mostly makes the closed world assumption, which focuses on designing accurate classifiers under the assumption that all test classes are known at training time. A more realistic scenario is to expect unseen classes during testing (open world). In this case, the goal is to design a learning system that classifies documents of the known classes into their respective classes and also to reject documents from unknown classes. This problem is called open (world) classification. This paper approaches the problem by reducing the open space risk while balancing the empirical risk. It proposes to use a new learning strategy, called center-based similarity (CBS) space learning (or CBS learning), to provide a novel solution to the problem. Extensive experiments across two datasets show that CBS learning gives promising results on multiclass open text classification compared to state-ofthe-art baselines.

Out-of-Domain Detection for Natural Language Understanding in Dialog Systems

“…The performance of the resulting model (i.e., ER+POG) on the OSQ and IPA dataset is depicted in Table II and Table III, respectively. 4 Note that FPR= FP FP+TN , and TPR= TP TP+FN , where FP, TP, TN, FN is the number of false positives, true positives, true negatives, false negatives, respectively. It can be seen that our model outperforms all the baselines on both datasets significantly, which demonstrates the effectiveness of the generated pseudo OOD utterances.…”

“…Recently, various deep neural network based NLU models are proposed and some of these models have been applied in real-world applications [1]- [3]. Most existing neural NLU modules are built by following a closed-world assumption [4], [5], i.e, the data used in the training and testing phrase are drawn from the same distribution. However, such an assumption is commonly violated in practical systems that are deployed in a dynamic or open environment.…”

A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier

“…The main idea is that a classifier should not cover too much open space with few or no training data, thereby rejecting the unknown images. cbsSVM [10] shares the similar ideas in text classification. However, these methods are all based on SVM, which fails to effectively capture the high-level semantic concept of intents comparing with deep neural networks.…”

“…How do we detect unknown intent without any prior knowledge about it? In [9,10], a m-class classifier should be able to reject examples from unknown class while performing m-class classification tasks. It is because not all test classes are known in the training set, which forms a (m+1)-class classification problem where the (m+1) th class represents the unknown class.…”

Algorithm based on modified angle‐based outlier factor for open‐set classification of text documents

“…Although most of research papers on text classification deal with the problem of closed‐set classification, some methods have been recently proposed to tackle open‐set text classification. Fei and Liu proposed a center‐based similarity method, which is based on a decision threshold (usually 0.5) on a posteriori probability to reject an observation as unrecognized, where the probabilities are estimated from SVM scores using Platt's algorithm . Doan and Kalita proposed an algorithm called nearest class mean, which attempts to find boundary regions for known classes using spheres centered at class centroids, with observations falling outside the sphere boundaries treated as either outliers or indicators of possible new unknown classes.…”