Improving Automated Email Tagging with Implicit Feedback

Abstract

Tagging email is an important tactic for managing informa tion overload. Machine learning methods can help the user with this task by predicting tags for incoming email mes sages. The natural user interface displays the predicted tags on the email message, and the user doesn't need to do any thing unless those predictions are wrong (in which case, the user can delete the incorrect tags and add the missing tags). From a machine learning perspective, this means that the learning algorithm never receives confirmation that its pre dictions are correct-it only receives feedback when it makes a mistake. This can lead to slower learning, particularly when the predictions were not very confident, and hence, the learn ing algorithm would benefit from positive feedback. One could assume that if the user never changes any tag, then the predictions are correct, but users sometimes forget to correct the tags, presumably because they are focused on the content of the email messages and fail to notice incorrect and missing tags. The aim of this paper is to determine whether implicit feedback can provide useful additional training examples to the email prediction subsystem of TaskTracer, known as EP2 (Email Predictor 2). Our hypothesis is that the more time a user spends working on an email message, the more likely it is that the user will notice tag errors and correct them. If no corrections are made, then perhaps it is safe for the learn ing system to treat the predicted tags as being correct and train accordingly. This paper proposes three algorithms (and two baselines) for incorporating implicit feedback into the EP2 tag predictor. These algorithms are then evaluated us ing email interaction and tag correction events collected from 14 user-study participants as they performed email-directed tasks while using TaskTracer EP2. The results show that im plicit feedback produces important increases in training feed back, and hence, significant reductions in subsequent predic tion errors despite the fact that the implicit feedback is not perfect. We conclude that implicit feedback mechanisms can provide a useful performance boost for email tagging sys tems.

Towards a Reliable Evaluation of Local Interpretation Methods

“…Machine learning technology that automatically discovers patterns and structures in massive amounts of data is becoming more and more widely used in real life [1][2][3]. In particular, deep convolutional neural networks (DCNNs) have demonstrated impressive results in many real scenarios, such as image classification [4,5], speech recognition [6], and recommendation systems [7].…”