End-user feature labeling: Supervised and semi-supervised approaches based on locally-weighted logistic regression

Abstract

This is the unspecified version of the paper.This version of the publication may differ from the final published version. Permanent repository link AbstractWhen intelligent interfaces, such as intelligent desktop assistants, email classifiers, and recommender systems, customize themselves to a particular end user, such customizations can decrease productivity and increase frustration due to inaccurate predictions-especially in early stages when training data is limited. The end user can improve the learning algorithm by tediously labeling a substantial amount of additional training data, but this takes time and is too ad hoc to target a particular area of inaccuracy. To solve this problem, we propose new supervised and semi-supervised learning algorithms based on locally weighted logistic regression for feature labeling by end users, enabling them to point out which features are important for a class, rather than provide new training instances.We first evaluate our algorithms against other feature labeling algorithms under idealized conditions using feature labels generated by an oracle. In addition, another of our contributions is an evaluation of feature labeling algorithms under real world conditions using feature labels harvested from actual end users in our user study. Our user study is the first statistical user study for feature labeling involving a large number of end users (43 participants), all of whom have no background in machine learning.Our supervised and semi-supervised algorithms were among the best performers when compared to other feature labeling algorithms in the idealized setting and they are also robust to poor quality feature labels provided by ordinary end users in our study. We also perform an analysis to investigate the relative gains of incorporating the different sources of knowledge available in the labeled training set, the feature labels and the unlabeled data. Together, our results strongly suggest that feature labeling by end users is both viable and effective for allowing end users to improve the learning algorithm behind their customized applications.

Learning with rationales for document classification

“…Melville and Sindhwani (2009) presented pooling multinomials to incorporate feature annotations into the training of multinomial naïve Bayes, hence we chose this as a baseline for our approach using multinomial naïve Bayes. We are not aware of any approach specifically developed to incorporate rationales into the training of logistic regression classifier, and the closest work is that of Das et al (2013), which was specifically designed to incorporate feature annotation into the training of locally-weighted logistic regression, and hence we chose it as a baseline for our approach using logistic regression. Zaidan et al (2007) Zaidan et al (2007) presented a method to incorporate rationales into the training of support vector machines.…”

“…To address this issue, many methods have been developed that are classifier-specific. Examples include knowledge-based neural networks (Towell and Shavlik 1994;Girosi and Chan 1995;Towell et al 1990), knowledge-based support vector machines (Fung et al 2002), pooling multinomial naïve Bayes ), incorporating feature annotation into locally-weighted logistic regression (Das et al 2013), incorporating constraints into the training of naïve Bayes (Stumpf et al 2007), and converting rationales and feature annotations into constraints for support vector machines (Small et al 2011;Zaidan et al 2007). Being classifier-specific limits their applicability when one does not know which classifier is best suited for his/her domain and hence would like to test several classifiers, necessitating a simple and generic approach that can be utilized by several off-the-shelf classifiers.…”

“…-We compare our method to Das et al (2013), which was specifically designed for incorporating feature labels into the training of locally-weighted logistic regression, and show that our method has comparable performance. -We propose and evaluate a novel active learning approach specifically tailored for utilizing the rationales provided by the labeler.…”

“…In Sect. 4, we compare our method to three baselines, Melville and Sindhwani (2009), Zaidan et al (2007), and Das et al (2013). In Sect.…”

“…Melville and Sindhwani (2009) incorporated feature annotation into multinomial naïve Bayes by training two multinomial naïve Bayes models, one on labeled instances and the other on labeled features, and used linear pooling to combine the two models. Das et al (2013) utilized locally-weighted logistic regression to incorporate feature labels into logistic regression by locally fitting a logistic function on instances around a small neighborhood of test instances and taking into account the labeled features. We next describe our approach that can readily incorporate rationales into any classifier by modifying the training data, without requiring changes to the training algorithm of a classifier.…”