LAOps: Learning Analytics with Privacy-aware MLOps

Niemelä, Pia; Silverajan, Bilhanan; Nurminen, Mikko; Hukkanen, Jenni; Järvinen, Hannu-Matti

doi:10.5220/0011113300003182

Cited by 4 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An analysis of Tables 4,5 shows that MedViT-T exhibits similar behavior to ResNet-18 on the same tests and MLOps configurations. 4, the margin of error does not exceed 0.9% at a 95% confidence level.…”

Section: Resultsmentioning

confidence: 85%

“…Machine learning operations have gained momentum in healthcare due to their potential to streamline the development, deployment, and maintenance of machine learning models ( 4 ). Several studies have delved into the unique requirements and challenges that healthcare poses to the MLOps methodology, such as patient data confidentiality ( 5 ), bias reduction ( 6 ), and compliance with health regulations like HIPAA ( 7 ). However, most existing MLOps frameworks are designed to ensure efficient operation rather than resilience to the various disturbances that healthcare environments may present.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Resilience-aware MLOps for AI-based medical diagnostic system

Moskalenko,

Kharchenko

2024

Front. Public Health

View full text Add to dashboard Cite

BackgroundThe healthcare sector demands a higher degree of responsibility, trustworthiness, and accountability when implementing Artificial Intelligence (AI) systems. Machine learning operations (MLOps) for AI-based medical diagnostic systems are primarily focused on aspects such as data quality and confidentiality, bias reduction, model deployment, performance monitoring, and continuous improvement. However, so far, MLOps techniques do not take into account the need to provide resilience to disturbances such as adversarial attacks, including fault injections, and drift, including out-of-distribution. This article is concerned with the MLOps methodology that incorporates the steps necessary to increase the resilience of an AI-based medical diagnostic system against various kinds of disruptive influences.MethodsPost-hoc resilience optimization, post-hoc predictive uncertainty calibration, uncertainty monitoring, and graceful degradation are incorporated as additional stages in MLOps. To optimize the resilience of the AI based medical diagnostic system, additional components in the form of adapters and meta-adapters are utilized. These components are fine-tuned during meta-training based on the results of adaptation to synthetic disturbances. Furthermore, an additional model is introduced for post-hoc calibration of predictive uncertainty. This model is trained using both in-distribution and out-of-distribution data to refine predictive confidence during the inference mode.ResultsThe structure of resilience-aware MLOps for medical diagnostic systems has been proposed. Experimentally confirmed increase of robustness and speed of adaptation for medical image recognition system during several intervals of the system’s life cycle due to the use of resilience optimization and uncertainty calibration stages. The experiments were performed on the DermaMNIST dataset, BloodMNIST and PathMNIST. ResNet-18 as a representative of convolutional networks and MedViT-T as a representative of visual transformers are considered. It is worth noting that transformers exhibited lower resilience than convolutional networks, although this observation may be attributed to potential imperfections in the architecture of adapters and meta-adapters.СonclusionThe main novelty of the suggested resilience-aware MLOps methodology and structure lie in the separating possibilities and activities on creating a basic model for normal operating conditions and ensuring its resilience and trustworthiness. This is significant for the medical applications as the developer of the basic model should devote more time to comprehending medical field and the diagnostic task at hand, rather than specializing in system resilience. Resilience optimization increases robustness to disturbances and speed of adaptation. Calibrated confidences ensure the recognition of a portion of unabsorbed disturbances to mitigate their impact, thereby enhancing trustworthiness.

show abstract

Section: Resultsmentioning

confidence: 85%

Section: Introductionmentioning

confidence: 99%

Resilience-aware MLOps for AI-based medical diagnostic system

Moskalenko,

Kharchenko

2024

Front. Public Health

View full text Add to dashboard Cite

show abstract

“…LA might be used to track how students use online course materials [44]. LA may automatically reorder the materials according to what students prioritized and gained the most from.…”

Section: Embedded Analyticsmentioning

confidence: 99%

A Systematic Review of the Role of Learning Analytics in Supporting Personalized Learning

Khor,

2023

Education Sciences

View full text Add to dashboard Cite

Personalized learning is becoming more important in today’s diverse classrooms. It is a strategy that tailors instruction to each student’s abilities and interests. The benefits of personalized learning include students’ enhanced motivation and academic success. The average teacher-to-student ratio in classes is 1:15.3, making it challenging for teachers to identify each student’s areas of strength (or weakness). Learning analytics (LA) has the potential to fill the need for personalized learning environments as it has recently revolutionized education by making it possible to gather and analyze vast volumes of student data to enhance the learning process. The convergence of these two fields has, therefore, become an important area for research. The purpose of this study is to conduct a systematic review to understand the ways in which LA can support personalized learning as well as the challenges involved. A total of 40 articles were included in the final review of this study, and the findings demonstrated that LA could support personalized instruction at the individual, group, and structural levels with or without teacher intervention. It can do so by (1) gathering feedback on students’ development, skill level, learning preferences, and emotions; (2) classifying students; (3) building feedback loops with continuously personalized resources; (4) predicting performance; and (5) offering real-time insights and visualizations of classroom dynamics. As revealed in the findings, the prominent challenges of LA in supporting personalized learning were the accuracy of insights, opportunity costs, and concerns of fairness and privacy. The study could serve as the basis for future research on personalizing learning with LA.

show abstract

“…MLOps contains practices both from Machine Learning and DevOps processes. Where MLOps is also applicable to this study's proposed cybersecurity system reference architecture, an application of privacy aware MLOps for learning analytics is discussed in (Niemelä et al, 2022). A simplified version of MLOps is presented in (Niemelä et al, 2022), and its pipeline consists of data science (build, experiment, and evaluate the model), ML model, and production parts (deploy the model).…”

Section: Designing a Reference Architecture For Machine Learning-supp...mentioning

confidence: 99%

“…Where MLOps is also applicable to this study's proposed cybersecurity system reference architecture, an application of privacy aware MLOps for learning analytics is discussed in (Niemelä et al, 2022). A simplified version of MLOps is presented in (Niemelä et al, 2022), and its pipeline consists of data science (build, experiment, and evaluate the model), ML model, and production parts (deploy the model). The idea of MLOps is having a high level of communication among parts of the system, reproducible results, and reusable tools.…”

Section: Designing a Reference Architecture For Machine Learning-supp...mentioning

confidence: 99%

Cybersecurity design of data-intensive systems

Avcı

View full text Add to dashboard Cite

Research method has a critical role within the empirical studies. In order to converge valid and reliable outcomes, clear cut research methodologies should be applied and reported in the selected primary studies. The types of the research methods can be listed as "Case Study" (in depth investigation with a real-life context), "Experiment" (scientific procedure to test a hypothesis) and "Short Example". It can be derived from Table 5.5 that there is not a tendency towards a research method, considering the fact that the gap between the percentages of the methodologies is not wide. Nevertheless, experimentation is used more often comparing to case studies and short examples to evaluate the system architectures.

show abstract

LAOps: Learning Analytics with Privacy-aware MLOps

Cited by 4 publications

References 0 publications

Resilience-aware MLOps for AI-based medical diagnostic system

Resilience-aware MLOps for AI-based medical diagnostic system

A Systematic Review of the Role of Learning Analytics in Supporting Personalized Learning

Cybersecurity design of data-intensive systems

Contact Info

Product

Resources

About