Accuracy Performance Degradation in Image Classification Models due to Concept Drift

Hashmani, Manzoor Ahmed; Jameel, Syed Muslim; Alhussain, Hitham; Rehman, Mobashar; Budiman, Arif

doi:10.14569/ijacsa.2019.0100552

Cited by 16 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interestingly, through the results, we can determine that ACNNELM is better for handling CD for MNIST and Not-MNIST dataset, whereas we found the promising testing accuracy of CNN in CIFAR10 dataset (color images). Also, in our previous study [32], we have performed several experiments to validate the Concept Drift issue.…”

Section: Results Analysis and Deductionmentioning

confidence: 99%

A Critical Review on Adverse Effects of Concept Drift over Machine Learning Classification Models

Jameel¹,

Hashmani²,

Alhussain³

et al. 2020

IJACSA

Self Cite

View full text Add to dashboard Cite

Big Data (BD) is participating in the current computing revolution in a big way. Industries and organizations are utilizing their insights for Business Intelligence using Machine Learning Models (ML-Models). Deep Learning Models (DL-Models) have been proven to be a better selection than Shallow Learning Models (SL-Models). However, the dynamic characteristics of BD introduce many critical issues for DL-Models, Concept Drift (CD) is one of them. CD issue frequently appears in Online Supervised Learning environments in which data trends change over time. The problem may even worsen in the BD environment due to veracity and variability factors. Due to the CD issue, the accuracy of classification results degrades in ML-Models, which may make ML-Models not applicable. Therefore, ML-Models need to adapt quickly to changes to maintain the accuracy level of the results. In current solutions, a substantial improvement in accuracy and adaptability is needed to make ML-Models robust in a non-stationary environment. In the existing literature, the consolidated information on this issue is not available. Therefore, in this study, we have carried out a systematic critical literature review to discuss the Concept Drift taxonomy and identify the adverse effects and existing approaches to mitigate CD.

show abstract

Section: Results Analysis and Deductionmentioning

confidence: 99%

A Critical Review on Adverse Effects of Concept Drift over Machine Learning Classification Models

Jameel¹,

Hashmani²,

Alhussain³

et al. 2020

IJACSA

Self Cite

View full text Add to dashboard Cite

show abstract

“…We expect that use of the model in other places will require different CTs or corrective equations, although, it is also possible that a more generic model might be developed that is effective across the snapper distributional range. Our model is also linked to a specific time, and so further work is required to guard against inaccuracies due to concept drift (Hashmani et al, 2019). Changes in the environment or camera equipment over time can reduce the accuracy of computer vision models (e.g., Langenkämper et al, 2020).…”

Section: Discussionmentioning

confidence: 99%

Improved Accuracy for Automated Counting of a Fish in Baited Underwater Videos for Stock Assessment

Connolly

Fairclough²,

Jinks

et al. 2021

Front. Mar. Sci.

View full text Add to dashboard Cite

The ongoing need to sustainably manage fishery resources can benefit from fishery-independent monitoring of fish stocks. Camera systems, particularly baited remote underwater video system (BRUVS), are a widely used and repeatable method for monitoring relative abundance, required for building stock assessment models. The potential for BRUVS-based monitoring is restricted, however, by the substantial costs of manual data extraction from videos. Computer vision, in particular deep learning (DL) models, are increasingly being used to automatically detect and count fish at low abundances in videos. One of the advantages of BRUVS is that bait attractants help to reliably detect species in relatively short deployments (e.g., 1 h). The high abundances of fish attracted to BRUVS, however, make computer vision more difficult, because fish often obscure other fish. We build upon existing DL methods for identifying and counting a target fisheries species across a wide range of fish abundances. Using BRUVS imagery targeting a recovering fishery species, Australasian snapper (Chrysophrys auratus), we tested combinations of three further mathematical steps likely to generate accurate, efficient automation: (1) varying confidence thresholds (CTs), (2) on/off use of sequential non-maximum suppression (Seq-NMS), and (3) statistical correction equations. Output from the DL model was more accurate at low abundances of snapper than at higher abundances (>15 fish per frame) where the model over-predicted counts by as much as 50%. The procedure providing the most accurate counts across all fish abundances, with counts either correct or within 1–2 of manual counts (R2 = 88%), used Seq-NMS, a 45% CT, and a cubic polynomial corrective equation. The optimised modelling provides an automated procedure offering an effective and efficient method for accurately identifying and counting snapper in the BRUV footage on which it was tested. Additional evaluation will be required to test and refine the procedure so that automated counts of snapper are accurate in the survey region over time, and to determine the applicability to other regions within the distributional range of this species. For monitoring stocks of fishery species more generally, the specific equations will differ but the procedure demonstrated here could help to increase the usefulness of BRUVS.

show abstract

“…We expect that use of the model in other places will require different CTs or corrective equations, although it is also possible that a more generic model might be developed that is effective across the snapper distributional range. Our model is also linked to a specific time, and so further work is required to guard against inaccuracies due to concept drift (Hashmani et al 2019). Changes in the environment or camera equipment over time can reduce the accuracy of computer vision models (e.g.…”

Section: Discussionmentioning

confidence: 99%

Improved accuracy for automated counting of a fish in baited underwater videos for stock assessment

Connolly

Fairclough

Jinks

et al. 2021

Preprint

View full text Add to dashboard Cite

The ongoing need to sustainably manage fishery resources necessitates fishery-independent monitoring of the status of fish stocks. Camera systems, particularly baited remote underwater video stations (BRUVS), are a widely-used and repeatable method for monitoring relative abundance, required for building stock assessment models. The potential for BRUVS-based monitoring is restricted, however, by the substantial costs of manual data extraction from videos. Computer vision, in particular deep learning models, are increasingly being used to automatically detect and count fish at low abundances in videos. One of the advantages of BRUVS is that bait attractants help to reliably detect species in relatively short deployments (e.g. 1 hr). The high abundances of fish attracted to BRUVS, however, make computer vision more difficult, because fish often occlude other fish. We build upon existing deep learning methods for identifying and counting a target fisheries species across a wide range of fish abundances. Using BRUVS imagery targeting a recovering fishery species, Australian snapper (Chrysophrys auratus), we tested combinations of three further mathematical steps likely to generate accurate, efficient automation: 1) varying confidence thresholds (CTs), 2) on/off use of sequential non-maximum suppression (Seq-NMS), and 3) statistical correction equations. Output from the deep learning model was accurate at very low abundances of snapper; at higher abundances, however, the model over-predicted counts by as much as 50%. The procedure providing the most accurate counts across all fish abundances, with counts either correct or within 1 to 2 of manual counts (R2 = 93.4%), used Seq-NMS, a 55% confidence threshold, and a cubic polynomial corrective equation. The optimised modelling provides an automated procedure offering an effective and efficient method for accurately identifying and counting snapper in BRUV footage. Further testing is required to ensure that automated counts of snapper remain accurate in the survey region over time, and to determine the applicability to other regions within the distributional range of this species. For monitoring stocks of fishery species more generally, the specific equations will differ but the procedure demonstrated here would help to increase the usefulness of BRUVS, while decreasing costs.

show abstract

Accuracy Performance Degradation in Image Classification Models due to Concept Drift

Cited by 16 publications

References 10 publications

A Critical Review on Adverse Effects of Concept Drift over Machine Learning Classification Models

A Critical Review on Adverse Effects of Concept Drift over Machine Learning Classification Models

Improved Accuracy for Automated Counting of a Fish in Baited Underwater Videos for Stock Assessment

Improved accuracy for automated counting of a fish in baited underwater videos for stock assessment

Contact Info

Product

Resources

About