Rare Class Problem in Data Mining: Review

Dongre, Snehlata

doi:10.26483/ijarcs.v8i7.4530

Cited by 6 publications

(4 citation statements)

References 20 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RUS attempts to balance the class distribution by randomly removing an instance of the majority class. This generates the problem of losing valuable information [15]. Because of its efficiency, simplicity, and speed, RUS has shown very good performance, and it is used in boosting for these reasons [7,11].…”

Section: Data-level Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A Survey Study on Proposed Solutions for Imbalanced Big Data

Razoqi,

Al-Talib

2024

Iraqi Journal of Science

View full text Add to dashboard Cite

Learning from imbalanced data has been a focus of studies for more than two decades of continuous development. Training data is considered imbalanced when the size of the positive (minority) class is neglected because of the large size of the negative (majority) class, in addition to the problem of deviating distributions of binary tasks. The appearance of big data brings new problems and challenges to the imbalance problem. Big Data announces the challenges with 5V: volume, velocity, veracity, value, and variety. This study relied on dividing the solution to the problem of data imbalance into three levels: data level, algorithm level, and hybrid approaches. First, the standard solutions for this problem that were proposed were mentioned, and in addition, the most important metrics adopted for measuring the classification efficiency of imbalanced data were identified. In this survey study, 27 studies were reviewed during the period 2015–2022, distributed according to the levels of treatment of the imbalance problem. They also reviewed the performance metrics that were used in these studies and the sources of the datasets to which these solutions were applied. The study makes it easier for researchers and scholars to see the solutions to addressing the problem of data imbalance and the hybrid approaches recently used for that, and to take advantage of them in improving the classification process.

show abstract

Section: Data-level Methodsmentioning

confidence: 99%

“…One-class learning techniques can be used at the algorithm level to recognize instances of one class and deny others. This approach optimizes the performance of the learning algorithm on unseen data [15].…”

Section: Algorithm-level Approachesmentioning

confidence: 99%

A Survey Study on Proposed Solutions for Imbalanced Big Data

Razoqi,

Al-Talib

2024

Iraqi Journal of Science

View full text Add to dashboard Cite

show abstract

“…For instance, we determined throughout our research, when applying sampling in machine learning, a balanced ratio (50:50) is not as beneficial for Big Data as it is for smaller datasets, at least in the Medicare fraud detection domain. Studies that focus on class rarity are far less common [2,37]. With regards to the research presented in this paper, we limit our discussion to studies that employ Big Data for studying class imbalance in relation to rarity.…”

Section: Related Workmentioning

confidence: 99%

The effects of class rarity on the evaluation of supervised healthcare fraud detection models

2019

View full text Add to dashboard Cite

The United States healthcare system produces an enormous volume of data with a vast number of financial transactions generated by physicians administering healthcare services. This makes healthcare fraud difficult to detect, especially when there are considerably less fraudulent transactions (documented and readily available) than non-fraudulent. The ability to successfully detect fraudulent activities in healthcare, given such discrepancies, can garner up to $350 billion in recovered monetary losses. In machine learning, when one class has a substantially larger number of instances (majority) compared to the other (minority), this is known as class imbalance. In this paper, we focus specifically on Medicare, utilizing three 'Big Data' Medicare claims datasets with real-world fraudulent physicians. We create a training and test dataset for all three Medicare parts, both separately and combined, to assess fraud detection performance. To emulate class rarity, which indicates particularly severe levels of class imbalance, we generate additional datasets, by removing fraud instances, to determine the effects of rarity on fraud detection performance. Before a machine learning model can be distributed for real-world use, a performance evaluation is necessary to determine the best configuration (e.g. learner, class sampling ratio) and whether the associated error rates are low, indicating good detection rates. With our research, we demonstrate the effects of severe class imbalance and rarity using a training and testing (Train_Test) evaluation method via a hold-out set, and provide our recommendations based on the supervised machine learning results. Additionally, we repeat the same experiments using Cross-Validation, and determine it is a viable substitute for Medicare fraud detection. For machine learning with the severe class imbalance datasets, we found that, as expected, fraud detection performance decreased as the fraudulent instances became more rare. We apply Random Undersampling to both Train_Test and Cross-Validation, for all original and generated datasets, in order to assess potential improvements in fraud detection by reducing the adverse effects of class imbalance and rarity. Overall, our results indicate that the Train_Test method significantly outperforms Cross-Validation.

show abstract

“…The analysis drawn from a comparative study of each reported research solution is shown in Table 4. Dongre and Malik (2017) reported that data adjusting provide the better solution than other techniques. But Dongre and Malik (2017) suggest that a hybrid approach gives the best solution for class imbalance learning.…”

Section: Cost-sensitive Boostingmentioning

confidence: 99%

An Insight into Rare Class Problem: Analysis and Potential Solutions

Maheshwari¹,

Jain²,

Jadon³

2018

Journal of Computer Science

View full text Add to dashboard Cite

The class imbalance problem presents an important challenge to the data mining community, in which the number of examples of one class is more than the others. This problem is characterized by a different distribution of cases between all the classes. In this paper, our goal is to study the various challenges of class imbalance problem and provide a comparative study of the current development of research in learning from imbalanced data. We provide a thorough understanding of the nature of the problem, the methods used for data balancing, the learning objectives and assessment metrics used for getting measurable performance, the stated research solutions and the imbalanced problem in multiple classes. This paper highlights the significant opportunities and challenges in the field and provides potential future research directions in the class imbalance problem.

show abstract

Rare Class Problem in Data Mining: Review

Cited by 6 publications

References 20 publications

A Survey Study on Proposed Solutions for Imbalanced Big Data

A Survey Study on Proposed Solutions for Imbalanced Big Data

The effects of class rarity on the evaluation of supervised healthcare fraud detection models

An Insight into Rare Class Problem: Analysis and Potential Solutions

Contact Info

Product

Resources

About