Negative Confidence-Aware Weakly Supervised Binary Classification for Effective Review Helpfulness Classification

Abstract

The incompleteness of positive labels and the presence of many unlabelled instances are common problems in binary classification applications such as in review helpfulness classification. Various studies from the classification literature consider all unlabelled instances as negative examples. However, a classification model that learns to classify binary instances with incomplete positive labels while assuming all unlabelled data to be negative examples will often generate a biased classifier. In this work, we propose a novel Negative Confidence-aware Weakly Supervised approach (NCWS), which customises a binary classification loss function by discriminating the unlabelled examples with different negative confidences during the classifier's training. NCWS allows to effectively, unbiasedly identify and separate positive and negative instances after its integration into various binary classifiers from the literature, including SVM, CNN and BERT-based classifiers. We use the review helpfulness classification as a test case for examining the effectiveness of our NCWS approach. We thoroughly evaluate NCWS by using three different datasets, namely one from Yelp (venue reviews), and two from Amazon (Kindle and Electronics reviews). Our results show that NCWS outperforms strong baselines from the literature including an existing SVM-based approach (i.e. SVM-P), the positive and unlabelled learning-based approach (i.e. C-PU) and the positive confidence-based approach (i.e. P-conf) in addressing the classifier's bias problem. Moreover, we further examine the effectiveness of NCWS by using its classified helpful reviews in a state-of-theart review-based venue recommendation model (i.e. DeepCoNN) and demonstrate the benefits of using NCWS in enhancing venue recommendation effectiveness in comparison to the baselines.

Leveraging Review Properties for Effective Recommendation

“…•Prob_Helpful: We classify the reviews with a state-of-the-art review helpfulness classification model [45] and obtain the probability of a given review being helpful.…”

“…Next, in the review property encoding layer, we use the Negative Confidence-aware Weakly Supervised (i.e. NCWS) review helpfulness classifier [45] to generate the 'Polar_Helpful' property scores, which estimates the probability of the reviews being helpful. In particular, we follow [45] in training the NCWS model using reviews from the 'food' and 'nightlife' categories of the Yelp Challenge dataset round 12 7 and on the Kindle reviews from Amazon 8 .…”

“…NCWS) review helpfulness classifier [45] to generate the 'Polar_Helpful' property scores, which estimates the probability of the reviews being helpful. In particular, we follow [45] in training the NCWS model using reviews from the 'food' and 'nightlife' categories of the Yelp Challenge dataset round 12 7 and on the Kindle reviews from Amazon 8 . We use NCWS to predict the 'Polar_Helpful' property scores of reviews in both the Yelp and Amazon datasets.…”

Leveraging Review Properties for Effective Recommendation

“…•Prob_Helpful: We classify the reviews with a state-of-the-art review helpfulness classification model [45] and obtain the probability of a given review to be helpful.…”

“…Next, in the review property encoding layer, we use the Negative Confidence-aware Weakly Supervised (i.e. NCWS) review helpfulness classifier [45] to generate the 'Polar_Helpful' property scores, which estimates the probability of the reviews being helpful. In particular, we follow [45] in training the NCWS model using reviews from the using 'food' and 'nightlife' categories of the Yelp Challenge dataset round 12 7 and on the Kindle reviews from Amazon 8 .…”