Rare pattern mining: challenges and future perspectives

Borah, Anindita; Nath, Bhabesh

doi:10.1007/s40747-018-0085-9

Cited by 28 publications

(16 citation statements)

References 139 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unsupervised rare pattern mining in medical imaging has been proved to be beneficial [14], but most of the classic rare pattern mining techniques (Apriori [15], [16] based and FP-Growth [17] based) cannot be directly applied to image data as it is two dimensional with space-related information [18]. There are several methods developed for unsupervised image anomaly detection.…”

Section: Related Workmentioning

confidence: 99%

A Two-Dimensional Sparse Matrix Profile DenseNet for COVID-19 Diagnosis Using Chest CT Images

Liu

Leung

2020

IEEE Access

View full text Add to dashboard Cite

COVID-19 is a newly identified disease, which is very contagious and has been rapidly spreading across different countries around the world, calling for rapid and accurate diagnosis tools. Chest CT imaging has been widely used in clinical practice for disease diagnosis, but image reading is still a time-consuming work. We aim to integrate an image preprocessing technology for anomaly detection with supervised deep learning for chest CT imaging-based COVID-19 diagnosis. In this study, a matrix profile technique was introduced to CT image anomaly detection in two levels. At one-dimensional level, CT images were simply flatted and transformed to a one-dimensional vector so that the matrix profile algorithm could be implemented for them directly. At two-dimensional level,a matrix profile was calculated in a sliding window way for every segment in the image. An anomaly severity score (CT-SS) was calculated, and the difference of the CT-SS between the COVID-19 CT images and Non-COVID-19 CT images was tested. A sparse anomaly mask was calculated and applied to penalize the pixel values of each image. The anomaly weighted images were then used to train standard DenseNet deep learning models to distinguish the COVID-19 CT from Non-COVID-19 CT images. A VGG19 model was used as a baseline model for comparison. Although extra finetuning needs to be done manually, the onedimensional matrix profile method could identify the anomalies successfully. Using the two-dimensional matrix profiling method, CT-SS and anomaly weighted image can be successfully generated for each image. The CT-SS significantly differed among the COVID-19 CT images and Non-COVID-19 CT images (p − value < 0.05). Furthermore, we identified a potential causal association between the number of underlying diseases of a COVID-19 patient and the severity of the disease through statistical mediation analysis. Compared to the raw images, the anomaly weighted images showed generally better performance in training the DenseNet models with different architectures for diagnosing COVID-19, which was validated using two publicly available COVID-19 lung CT image datasets. The metric Area Under the Curve(AUC) on one dataset were 0.7799(weighted)vs. 0.7391(unweighted), 0.7812(weighted) vs. 0.7410(unweighted), 0.7780(weighted) vs. 0.7399(unweighted), 0.7045(weighted) vs. 0.6910(unweighted) for DenseNet121, DenseNet169, DenseNet201, and the baseline model VGG19, respectively. The same trend was observed using another independent dataset. The significant results revealed the critical value of using this existing state-of-the-art algorithm for image anomaly detection. Furthermore, the end-to-end model structure has the potential to work as a rapid tool for clinical imaging-based diagnosis. INDEX TERMS Rare pattern mining, matrix profile, COVID-19 CT images, risk score, DenseNet, mediation analysis I. INTRODUCTION U NSUPERVISED anomaly detection using rare pattern mining is one of the most intuitive medical imaging based disease diagnosis methods [1]. People without medical expe...

show abstract

Section: Related Workmentioning

confidence: 99%

A Two-Dimensional Sparse Matrix Profile DenseNet for COVID-19 Diagnosis Using Chest CT Images

Liu

Leung

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Mining rare pattern from a database is not as popular as frequent ones. However, there are some areas where the rare pattern mining was found to be more important compared to frequent pattern mining [6]. Pattern mining concept was invented by [6] for frequent pattern mining.…”

Section: Related Work a Infrequent Itemset Miningmentioning

confidence: 99%

Conceptual Model of Incremental R-Eclat Algorithm for Infrequent Itemset Mining

Man¹,

Ruslan²,

Jusoh³

et al. 2020

IJETT

View full text Add to dashboard Cite

Mining valuable information from database could be very challenging especially for crucial decision-making. This is because mining association rule may require repetitious scanning of large databases that leads to the use of high memory usage and affects the running time. Few methods and algorithms were introduced by researchers to handle the issues in data mining. Rare Equivalence Class Transformation (R-Eclat) algorithm is one of the rule mining techniques using vertical format data repositories for infrequent pattern mining. The main operation in R-Eclat is intersecting tidset. Since the size of tidsets would affect the memory usage and its running time, more memory and time required for a bigger tidsets. Adopting to R-Eclat algorithm, a new incremental approach is introduced called Incremental Rare Equivalence Class Transformation (IR-Eclat). IR-Eclat specifically design for infrequent pattern mining and it is beneficial for dynamic database as the data is increasing in volume from time to time. In conjunction with big data explosion, the end users are at an advantage for the use of this incremental approach.

show abstract

“…However, extraction of rare patterns or itemsets have recently gained much importance considering its manifold applications in several domains [10]. Substantial quantity of research has already been performed in the area of rare pattern mining that contributes enormous techniques looking for these previously unwanted rare itemsets [9,[12][13][14]16,17]. The rare pattern mining techniques considered for comparative analysis have been described below.…”

Section: Rare Itemset Generationmentioning

confidence: 99%

Comparative evaluation of pattern mining techniques: an empirical study

Borah

Nath

2020

Complex Intell. Syst.

Self Cite

View full text Add to dashboard Cite

Pattern mining has emerged as a compelling field of data mining over the years. Literature has bestowed ample endeavors in this field of research ranging from frequent pattern mining to rare pattern mining. A precise and impartial analysis of the existing pattern mining techniques has therefore become essential to widen the scope of data analysis using the notion of pattern mining. This paper is therefore an attempt to provide a comparative scrutiny of the fundamental algorithms in the field of pattern mining through performance analysis based on several decisive parameters. The paper provides a structural classification of the widely referenced techniques in four pattern mining categories: frequent, maximal frequent, closed frequent and rare. It provides an analytical comparison of these techniques based on computational time and memory consumption using benchmark real and synthetic data sets. The results illustrate that tree based approaches perform exceptionally well over level wise approaches in case of dense data sets for all the categories. However, for sparse data sets, level wise approaches performed better than the former ones. This study has been carried out with an aim to analyze the pros and cons of the well known pattern mining techniques under different categories. Through this empirical study, an endeavor has been made to enable the researchers identify some fruitful and promising research directions in one of the most remarkable area of research, pattern mining.

show abstract

Rare pattern mining: challenges and future perspectives

Cited by 28 publications

References 139 publications

A Two-Dimensional Sparse Matrix Profile DenseNet for COVID-19 Diagnosis Using Chest CT Images

A Two-Dimensional Sparse Matrix Profile DenseNet for COVID-19 Diagnosis Using Chest CT Images

Conceptual Model of Incremental R-Eclat Algorithm for Infrequent Itemset Mining

Comparative evaluation of pattern mining techniques: an empirical study

Contact Info

Product

Resources

About