Handling Concept Drifts in Regression Problems – the Error Intersection Approach

Baier, Lucas; Kühl, Niklas; Satzger, Gerhard; Hofmann, Marcel; Mohr, Marisa

doi:10.30844/wi_2020_c1-baier

Cited by 14 publications

(6 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sudden or reoccurring concept drift presumably requires a different approach such as switching between two different prediction models, e.g. one model for normal situations and one for extreme situations [45] or training a prediction model for summer and winter respectively. This work systematically tests different adaptation strategies for handling incremental concept drift and evaluates the strategies based on their prediction performance in hindsight.…”

Section: Discussionmentioning

confidence: 99%

Switching Scheme: A Novel Approach for Handling Incremental Concept Drift in Real-World Data Sets

Baier¹,

Kellner²,

Kühl³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning models nowadays play a crucial role for many applications in business and industry. However, models only start adding value as soon as they are deployed into production. One challenge of deployed models is the effect of changing data over time, which is often described with the term concept drift. Due to their nature, concept drifts can severely affect the prediction performance of a machine learning system.In this work, we analyze the effects of concept drift in the context of a real-world data set. For efficient concept drift handling, we introduce the switching scheme which combines the two principles of retraining and updating of a machine learning model. Furthermore, we systematically analyze existing regular adaptation as well as triggered adaptation strategies. The switching scheme is instantiated on New York City taxi data, which is heavily influenced by changing demand patterns over time. We can show that the switching scheme outperforms all other baselines and delivers promising prediction results.

show abstract

Section: Discussionmentioning

confidence: 99%

Switching Scheme: A Novel Approach for Handling Incremental Concept Drift in Real-World Data Sets

Baier¹,

Kellner²,

Kühl³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, the use of error detection mechanisms for model weights can be combined with error correction mechanism or with downloading an uncorrupted version of the weights ( 24 ). If an error in the weights causes abnormal distortions in the feature space, it can increase predictive uncertainty and require a delegation of control to a human or switching to another model or model branch ( 27 , 28 ).…”

Section: Architecting Resilient Mlops-based Medical Diagnostic Systemmentioning

confidence: 99%

“…An expert can be reinforced by an AI explanation algorithm, while a large model is used with auxiliary semantic information (i.e., Zero-shot learning) ( 26 ). Switching to a simpler model can also be viewed as a graceful degradation, as a simple model is generally less sensitive to disturbances in the data, but produces more coarse or abstract predictions ( 27 , 28 ).…”

Section: Architecting Resilient Mlops-based Medical Diagnostic Systemmentioning

confidence: 99%

“…out-of-distribution data(23), and faults(24); AI Explanation mechanism(25); Zero-shot learning (26); Switch to simpler model(27) or Reduce prediction granularity(28) Active Learning (29); Continual Learning(30); Few-Shot Learning(31).Adversarial attackData encryption and sanitization(32); Gradient Masking(33); and Robustness Optimization(10) Fault injection Fault masking(34); Explicit redundancy(35); andError detection and correction (24)10.3389/fpubh.2024.1342937 Frontiers in Public Health 05 frontiersin.org 2.3 Resilienсe-aware MLOps architecture for medical diagnostic systems…”

mentioning

confidence: 99%

See 1 more Smart Citation

Resilience-aware MLOps for AI-based medical diagnostic system

Moskalenko,

Kharchenko

2024

Front. Public Health

View full text Add to dashboard Cite

BackgroundThe healthcare sector demands a higher degree of responsibility, trustworthiness, and accountability when implementing Artificial Intelligence (AI) systems. Machine learning operations (MLOps) for AI-based medical diagnostic systems are primarily focused on aspects such as data quality and confidentiality, bias reduction, model deployment, performance monitoring, and continuous improvement. However, so far, MLOps techniques do not take into account the need to provide resilience to disturbances such as adversarial attacks, including fault injections, and drift, including out-of-distribution. This article is concerned with the MLOps methodology that incorporates the steps necessary to increase the resilience of an AI-based medical diagnostic system against various kinds of disruptive influences.MethodsPost-hoc resilience optimization, post-hoc predictive uncertainty calibration, uncertainty monitoring, and graceful degradation are incorporated as additional stages in MLOps. To optimize the resilience of the AI based medical diagnostic system, additional components in the form of adapters and meta-adapters are utilized. These components are fine-tuned during meta-training based on the results of adaptation to synthetic disturbances. Furthermore, an additional model is introduced for post-hoc calibration of predictive uncertainty. This model is trained using both in-distribution and out-of-distribution data to refine predictive confidence during the inference mode.ResultsThe structure of resilience-aware MLOps for medical diagnostic systems has been proposed. Experimentally confirmed increase of robustness and speed of adaptation for medical image recognition system during several intervals of the system’s life cycle due to the use of resilience optimization and uncertainty calibration stages. The experiments were performed on the DermaMNIST dataset, BloodMNIST and PathMNIST. ResNet-18 as a representative of convolutional networks and MedViT-T as a representative of visual transformers are considered. It is worth noting that transformers exhibited lower resilience than convolutional networks, although this observation may be attributed to potential imperfections in the architecture of adapters and meta-adapters.СonclusionThe main novelty of the suggested resilience-aware MLOps methodology and structure lie in the separating possibilities and activities on creating a basic model for normal operating conditions and ensuring its resilience and trustworthiness. This is significant for the medical applications as the developer of the basic model should devote more time to comprehending medical field and the diagnostic task at hand, rather than specializing in system resilience. Resilience optimization increases robustness to disturbances and speed of adaptation. Calibrated confidences ensure the recognition of a portion of unabsorbed disturbances to mitigate their impact, thereby enhancing trustworthiness.

show abstract

“…Elena Ikonomovska et al [8], present a regression tree model for data stream processing using concept drifts detection techniques. Lucas Baier et al [9] elaborate a strategy that uses simple machine learning models when concept drifts are detected and a general complex model otherwise.…”

Section: Introductionmentioning

confidence: 99%

Concept Drift Detection to Improve Time Series Forecasting of Wind Energy Generation

Cabello-López

Cañizares-Juan

Carranza-García

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Most of the current data sources generate large amounts of data over time. Renewable energy generation is one example of such data sources. Machine learning is often applied to forecast time series. Since data flows are usually large, trends in data may change and learned patterns might not be optimal in the most recent data. In this paper, we analyse wind energy generation data extracted from the Sistema de Información del Operador del Sistema (ESIOS) of the Spanish power grid. We perform a study to evaluate detecting concept drifts to retrain models and thus improve the quality of forecasting. To this end, we compare the performance of a linear regression model when it is retrained randomly and when a concept drift is detected, respectively. Our experiments show that a concept drift approach improves forecasting between a 7.88% and a 33.97% depending on the concept drift technique applied.

show abstract

Handling Concept Drifts in Regression Problems – the Error Intersection Approach

Cited by 14 publications

References 38 publications

Switching Scheme: A Novel Approach for Handling Incremental Concept Drift in Real-World Data Sets

Switching Scheme: A Novel Approach for Handling Incremental Concept Drift in Real-World Data Sets

Resilience-aware MLOps for AI-based medical diagnostic system

Concept Drift Detection to Improve Time Series Forecasting of Wind Energy Generation

Contact Info

Product

Resources

About