The effects of class rarity on the evaluation of supervised healthcare fraud detection models

Abstract

The United States healthcare system produces an enormous volume of data with a vast number of financial transactions generated by physicians administering healthcare services. This makes healthcare fraud difficult to detect, especially when there are considerably less fraudulent transactions (documented and readily available) than non-fraudulent. The ability to successfully detect fraudulent activities in healthcare, given such discrepancies, can garner up to $350 billion in recovered monetary losses. In machine learning, when one class has a substantially larger number of instances (majority) compared to the other (minority), this is known as class imbalance. In this paper, we focus specifically on Medicare, utilizing three 'Big Data' Medicare claims datasets with real-world fraudulent physicians. We create a training and test dataset for all three Medicare parts, both separately and combined, to assess fraud detection performance. To emulate class rarity, which indicates particularly severe levels of class imbalance, we generate additional datasets, by removing fraud instances, to determine the effects of rarity on fraud detection performance. Before a machine learning model can be distributed for real-world use, a performance evaluation is necessary to determine the best configuration (e.g. learner, class sampling ratio) and whether the associated error rates are low, indicating good detection rates. With our research, we demonstrate the effects of severe class imbalance and rarity using a training and testing (Train_Test) evaluation method via a hold-out set, and provide our recommendations based on the supervised machine learning results. Additionally, we repeat the same experiments using Cross-Validation, and determine it is a viable substitute for Medicare fraud detection. For machine learning with the severe class imbalance datasets, we found that, as expected, fraud detection performance decreased as the fraudulent instances became more rare. We apply Random Undersampling to both Train_Test and Cross-Validation, for all original and generated datasets, in order to assess potential improvements in fraud detection by reducing the adverse effects of class imbalance and rarity. Overall, our results indicate that the Train_Test method significantly outperforms Cross-Validation.

CatBoost for big data: an interdisciplinary review

“…We feel using these parameters in addition to SMOTE or ADASYN would be redundant, since these algorithms balance the data before it is presented to a classifier. For further studies on application of techniques for addressing classs imbalance see [ 65 ] and [ 6 ].…”

Investigating rarity in web attacks with ensemble learners

“…These extreme imbalance and rarity conditions provide a problem statement as to whether RUS treatments can improve classification performance. In addition to these severe class imbalances, the XSS and SQL Injection web attacks exhibit rarity [53] with a low PCC as indicated in Table 2.…”

CatBoost for Big Data: an Interdisciplinary Review

“…We feel using these parameters in addition to SMOTE or ADASYN would be redundant, since these algorithms balance the data before it is presented to a classifier. For further studies on application of techniques for addressing classs imbalance see [48] and [56]. Yang and Beth report that the data they use in their study is imbalanced with 142 out of 9,666 records in the positive class for one dataset they use, and 109 out of 8,445 records in the positive class for another dataset.…”

“…We feel using these parameters in addition to SMOTE or ADASYN would be redundant, since these algorithms balance the data before it is presented to a classifier. For further studies on application of techniques for addressing classs imbalance see [48] and [56].…”