The Document Similarity Index based on the Jaccard Distance for Mail Filtering

Temma, Seiya; Sugii, Manabu; Matsuo, Hiroshi

doi:10.1109/itc-cscc.2019.8793419

Cited by 14 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For that purpose, we insert a similarity detector based on the Jaccard distance [55] before the classifier. This is inspired by spam detection techniques that use this distance to seek for characteristic sentences [56]- [58].…”

Section: ) Training Sample Reduction With Similarity Detection Stagementioning

confidence: 99%

Identifying Banking Transaction Descriptions via Support Vector Machine Short-Text Classification Based on a Specialized Labelled Corpus

García-Méndez

Fernández-Gavilanes²,

Juncal-Martínez

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Short texts are omnipresent in real-time news, social network commentaries, etc. Traditional text representation methods have been successfully applied to self-contained documents of medium size. However, information in short texts is often insufficient, due, for example, to the use of mnemonics, which makes them hard to classify. Therefore, the particularities of specific domains must be exploited. In this article we describe a novel system that combines Natural Language Processing techniques with Machine Learning algorithms to classify banking transaction descriptions for personal finance management, a problem that was not previously considered in the literature. We trained and tested that system on a labelled dataset with real customer transactions that will be available to other researchers on request. Motivated by existing solutions in spam detection, we also propose a short text similarity detector to reduce training set size based on the Jaccard distance. Experimental results with a two-stage classifier combining this detector with a SVM indicate a high accuracy in comparison with alternative approaches, taking into account complexity and computing time. Finally, we present a use case with a personal finance application, CoinScrap, which is available at Google Play and App Store. INDEX TERMS Machine learning, natural language processing, banking, personal finance management.

show abstract

Section: ) Training Sample Reduction With Similarity Detection Stagementioning

confidence: 99%

Identifying Banking Transaction Descriptions via Support Vector Machine Short-Text Classification Based on a Specialized Labelled Corpus

García-Méndez

Fernández-Gavilanes²,

Juncal-Martínez

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Regarding approximate word matching in item (c) above, various well‐known algorithms based on different word features may be adopted. For item (c)(i), in determining word similarity based on spelling , our PASS‐ToP implementation chose to use the Jaccard Similarity (coefficient), developed by Paul Jaccard [18], as it is one of the simplest and widely used similarity algorithms in data science and text processing applications, such as mail filtering [41]. For item (c)(ii), in determining word similarity based on pronunciation , our PASS‐ToP implementation chose to use the Soundex phonetic algorithm [25].…”

Section: From Pass To Pass‐top: Towards a Versatile Test Oraclementioning

confidence: 99%

An automated system with a versatile test oracle for assessing student programs

Tang

Poon

2022

Comp Applic In Engineering

View full text Add to dashboard Cite

Automated program assessment systems have been widely adopted in many universities. Many of these systems judge the correctness of student programs by comparing their actual outputs with predefined expected outputs for selected test inputs. A common weakness of such systems is that student programs would be marked as incorrect as long as their outputs deviate from the predefined ones, even if the deviations are only minor, insignificant, and considered acceptable by a human assessor that the programs have satisfied the specifications. This critical weakness caused undue frustration to students and undesirable pedagogical consequences that undermine these systems’ benefits. To address this issue, we developed an improved mechanism for program output comparison to serve as a versatile test oracle that brings the results of automated assessment much closer to those of human assessors. We evaluated the new mechanism in real programming classes using an existing automated program assessment system. We found that the new mechanism achieved zero false‐positive error (did not wrongly accept any incorrect output) and very low (0%–0.02%) false‐negative error (that wrongly rejected correct outputs), with very high accuracy (99.8%–100%) in correctly recognizing outputs deemed acceptable by instructors. This represents a major improvement over an existing assessment mechanism, which had 56.4%–64.1% false‐negative error with an accuracy of 25.4%–40.9%. Moreover, about 67%–96% of students achieved their best results in their first attempt, which could be encouraging to them and reduce their frustration. Furthermore, students generally welcomed the new assessment mechanism and agreed it was beneficial to their learning.

show abstract

“…The data used social media twitter hoax against the corona virus, politics and the environment with the hoax label as 49.4% and 50.6% non-hoax. Implementation of the Jaccard Index has been conducted to determine the similarity index in the classification of ham and spam e-mails [15]. The method used is the Document Similarity Index (DSI) which is calculated from the Jaccard Index and gets 98% precision results for Ham labels and 98% for Spam labels.…”

Section: Previous Workmentioning

confidence: 99%

“…Mail Spam Jaccard Similarity [15] The result of this resarch is 98% precision for ham and spam labels.…”

Section: Decisionmentioning

confidence: 99%

Covid-19 Hoax Detection Using KNN in Jaccard Space

Utami¹,

Iskandar²,

Hidayat³

et al. 2021

Indonesian J. Comput. Cybern. Syst.

View full text Add to dashboard Cite

Social media has become a communication key to spark thinking, dialogue and action around social issues. Hoax is information that added or subtracted from the content of the actual news. The spread of unconfirmed Covid-19 news can cause public concern. The purpose of this research was to modify KNN with Jaccard Space in the classification of hoax news related to Covid-19. The data used from Jabar Saber Hoaks and Jala Hoaks. The classification results with KNN with Jaccard Space and stemming Nazief & Adriani get the highest accuracy than other models in this research. The accuracy of the KNN model on the Jaccard Space with stemming Nazief & Adriani and K = 5 was 75.89%, while for Naïve Bayes was 65.18%.

show abstract

The Document Similarity Index based on the Jaccard Distance for Mail Filtering

Cited by 14 publications

References 0 publications

Identifying Banking Transaction Descriptions via Support Vector Machine Short-Text Classification Based on a Specialized Labelled Corpus

Identifying Banking Transaction Descriptions via Support Vector Machine Short-Text Classification Based on a Specialized Labelled Corpus

An automated system with a versatile test oracle for assessing student programs

Covid-19 Hoax Detection Using KNN in Jaccard Space

Contact Info

Product

Resources

About