Assessing the quality of classification models: Performance measures and evaluation procedures

Cichosz, Paweł

doi:10.2478/s13531-011-0022-9

Cited by 11 publications

(12 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recall = TP TP + FN (7) In this way, the arithmetic mean of all F1 score values obtained class-wise (denominated as the macro-averaged F1 score) provides a key figure of merit to compare different indicators + FE + classifier configurations [33,34]. Therefore, Figure 7 shows the results of the macro F1 score, where each boxplot is generated from the 100 iterations performed for each configuration.…”

Section: Discussionmentioning

confidence: 99%

Location-Awareness for Failure Management in Cellular Networks: An Integrated Approach

Fortes

Baena

Villegas

et al. 2021

Sensors

View full text Add to dashboard Cite

Recent years have seen the proliferation of different techniques for outdoor and, especially, indoor positioning. Still being a field in development, localization is expected to be fully pervasive in the next few years. Although the development of such techniques is driven by the commercialization of location-based services (e.g., navigation), its application to support cellular management is considered to be a key approach for improving its resilience and performance. When different approaches have been defined for integrating location information into the failure management activities, they commonly ignore the increase in the dimensionality of the data as well as their integration into the complete flow of networks failure management. Taking this into account, the present work proposes a complete integrated approach for location-aware failure management, covering the gathering of network and positioning data, the generation of metrics, the reduction in the dimensionality of such data, and the application of inference mechanisms. The proposed scheme is then evaluated by system-level simulation in ultra-dense scenarios, showing the capabilities of the approach to increase the reliability of the supported diagnosis process as well as reducing its computational cost.

show abstract

Section: Discussionmentioning

confidence: 99%

Location-Awareness for Failure Management in Cellular Networks: An Integrated Approach

Fortes

Baena

Villegas

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…A 10–fold cross–validation was used. This approach has the advantage of using all available data, while balancing the tradeoff of bias and variance [ 17 , 18 ]. At each fold, ca.…”

Section: Methodsmentioning

confidence: 99%

An improved machine learning pipeline for urinary volatiles disease detection: Diagnosing diabetes

et al. 2018

View full text Add to dashboard Cite

MotivationThe measurement of disease biomarkers in easily–obtained bodily fluids has opened the door to a new type of non–invasive medical diagnostics. New technologies are being developed and fine–tuned in order to make this possibility a reality. One such technology is Field Asymmetric Ion Mobility Spectrometry (FAIMS), which allows the measurement of volatile organic compounds (VOCs) in biological samples such as urine. These VOCs are known to contain a range of information on the relevant person’s metabolism and can in principle be used for disease diagnostic purposes. Key to the effective use of such data are well–developed data processing pipelines, which are necessary to extract the most useful data from the complex underlying biological structure.ResultsIn this study, we present a new data analysis pipeline for FAIMS data, and demonstrate a number of improvements over previously used methods. We evaluate the effect of a series of candidate operational steps during data processing, such as the use of wavelet transforms, principal component analysis (PCA), and classifier ensembles. We also demonstrate the use of FAIMS data in our pipeline to diagnose diabetes on the basis of a simple urine sample using machine learning classifiers. We present results for data generated from a case-control study of 115 urine samples, collected from 72 type II diabetic patients, with 43 healthy volunteers as negative controls. The resulting pipeline combines the steps that resulted in the best classification model performance. These include the use of a two–dimensional discrete wavelet transform, and the Wilcoxon rank–sum test for feature selection. We are able to achieve a best ROC curve AUC of 0.825 (0.747–0.9, 95% CI) for classification of diabetes vs control. We also note that this result is robust to changes in the data pipeline and different analysis runs, with AUC > 0.80 achieved in a range of cases. This is a substantial improvement in performance over previously used data processing methods in this area. Our ability to make strong statements about FAIMS ability to diagnose diabetes is sadly limited, as we found confounding effects from the demographics when including these data in the pipeline. The demographics alone produced a best AUC of 0.87 (0.795–0.94, 95% CI). While the combination of the demographics and FAIMS data resulted in an improvement on the AUC (0.907; 0.848–0.97, 95% CI), it did not prove to be a significant difference. Nevertheless, the pipeline itself shows a significant improvement in performance over more basic methods which have been used with FAIMS data in the past.

show abstract

“…Figure 12 shows a confusion matrix for DGA multiclass classification. The confusion matrix allows us to deduce several other evaluation parameters such as the classification accuracy, classification error, sensitivity, precision and the f1-score [70][71][72][73][74][75].…”

Section: Confusion Matrix (Cm)mentioning

confidence: 99%

“…The classification accuracy gives us a measure of how often the classifier is correct [70][71][72][73][74][75][76]. Equation (7) gives the formula for calculating the classification accuracy.…”

Section: Classification Accuracymentioning

confidence: 99%

A Multinomial DGA Classifier for Incipient Fault Detection in Oil-Impregnated Power Transformers

2021

View full text Add to dashboard Cite

This study investigates the use of machine-learning approaches to interpret Dissolved Gas Analysis (DGA) data to find incipient faults early in oil-impregnated transformers. Transformers are critical pieces of equipment in transmitting and distributing electrical energy. The failure of a single unit disturbs a huge number of consumers and suppresses economic activities in the vicinity. Because of this, it is important that power utility companies accord high priority to condition monitoring of critical assets. The analysis of dissolved gases is a technique popularly used for monitoring the condition of transformers dipped in oil. The interpretation of DGA data is however inconclusive as far as the determination of incipient faults is concerned and depends largely on the expertise of technical personnel. To have a coherent, accurate, and clear interpretation of DGA, this study proposes a novel multinomial classification model christened KosaNet that is based on decision trees. Actual DGA data with 2912 entries was used to compute the performance of KosaNet against other algorithms with multiclass classification ability namely the decision tree, k-NN, Random Forest, Naïve Bayes, and Gradient Boost. Investigative results show that KosaNet demonstrated an improved DGA classification ability particularly when classifying multinomial data.

show abstract

Assessing the quality of classification models: Performance measures and evaluation procedures

Cited by 11 publications

References 7 publications

Location-Awareness for Failure Management in Cellular Networks: An Integrated Approach

Location-Awareness for Failure Management in Cellular Networks: An Integrated Approach

An improved machine learning pipeline for urinary volatiles disease detection: Diagnosing diabetes

A Multinomial DGA Classifier for Incipient Fault Detection in Oil-Impregnated Power Transformers

Contact Info

Product

Resources

About