Data cleansing mechanisms and approaches for big data analytics: a systematic study

Hosseinzadeh, Mehdi; Azhir, Elham; Ahmed, Omed Hassan; Ghafour, Marwan Yassin; Ahmed, Sarkar Hasan; Rahmani, Amir Masoud; Vo, Bay

doi:10.1007/s12652-021-03590-2

Cited by 8 publications

(6 citation statements)

References 38 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach therefore combines a sample-based, an expert-based, and a rule-based approach. This way, the best possible data preparation quality can be achieved [79].…”

Section: Data Preparation Phasementioning

confidence: 99%

SMARTEN—A Sample-Based Approach towards Privacy-Friendly Data Refinement

Stach

Behringer

Bräcker

et al. 2022

JCP

View full text Add to dashboard Cite

Two factors are crucial for the effective operation of modern-day smart services: Initially, IoT-enabled technologies have to capture and combine huge amounts of data on data subjects. Then, all these data have to be processed exhaustively by means of techniques from the area of big data analytics. With regard to the latter, thorough data refinement in terms of data cleansing and data transformation is the decisive cornerstone. Studies show that data refinement reaches its full potential only by involving domain experts in the process. However, this means that these experts need full insight into the data in order to be able to identify and resolve any issues therein, e.g., by correcting or removing inaccurate, incorrect, or irrelevant data records. In particular for sensitive data (e.g., private data or confidential data), this poses a problem, since these data are thereby disclosed to third parties such as domain experts. To this end, we introduce SMARTEN, a sample-based approach towards privacy-friendly data refinement to smarten up big data analytics and smart services. SMARTEN applies a revised data refinement process that fully involves domain experts in data pre-processing but does not expose any sensitive data to them or any other third-party. To achieve this, domain experts obtain a representative sample of the entire data set that meets all privacy policies and confidentiality guidelines. Based on this sample, domain experts define data cleaning and transformation steps. Subsequently, these steps are converted into executable data refinement rules and applied to the entire data set. Domain experts can request further samples and define further rules until the data quality required for the intended use case is reached. Evaluation results confirm that our approach is effective in terms of both data quality and data privacy.

show abstract

“…Our approach therefore combines a sample-based, an expert-based, and a rule-based approach. This way, the best possible data preparation quality can be achieved [79].…”

Section: Data Preparation Phasementioning

confidence: 99%

SMARTEN—A Sample-Based Approach towards Privacy-Friendly Data Refinement

Stach

Behringer

Bräcker

et al. 2022

JCP

View full text Add to dashboard Cite

show abstract

“…The demand for extensive data, often referred to as “data hungriness” ( 62 ), poses medico-legal challenges, as single institutions may lack sufficient data for reliable predictions ( 16 ). Finally, data cleansing can enhance data usability in the context of intelligence, but it must be implemented carefully to avoid introducing another source of errors ( 59 , 63 ). This is the hypothetical responsibility of trainers and programmers, with potential shared responsibility on the part of the purchasing company, considering the limitations in the assessment related to the concepts of bias and error.…”

Section: Medical Malpractice Liability Assessmentmentioning

confidence: 99%

AI and professional liability assessment in healthcare. A revolution in legal medicine?

Terranova,

Cestonaro,

Fava

et al. 2024

Front. Med.

View full text Add to dashboard Cite

The adoption of advanced artificial intelligence (AI) systems in healthcare is transforming the healthcare-delivery landscape. Artificial intelligence may enhance patient safety and improve healthcare outcomes, but it presents notable ethical and legal dilemmas. Moreover, as AI streamlines the analysis of the multitude of factors relevant to malpractice claims, including informed consent, adherence to standards of care, and causation, the evaluation of professional liability might also benefit from its use. Beginning with an analysis of the basic steps in assessing professional liability, this article examines the potential new medical-legal issues that an expert witness may encounter when analyzing malpractice cases and the potential integration of AI in this context. These changes related to the use of integrated AI, will necessitate efforts on the part of judges, experts, and clinicians, and may require new legislative regulations. A new expert witness will be likely necessary in the evaluation of professional liability cases. On the one hand, artificial intelligence will support the expert witness; however, on the other hand, it will introduce specific elements into the activities of healthcare workers. These elements will necessitate an expert witness with a specialized cultural background. Examining the steps of professional liability assessment indicates that the likely path for AI in legal medicine involves its role as a collaborative and integrated tool. The combination of AI with human judgment in these assessments can enhance comprehensiveness and fairness. However, it is imperative to adopt a cautious and balanced approach to prevent complete automation in this field.

show abstract

“…In this dataset we cleanse ip addresses that do not give a ping response (Hosseinzadeh et al, 2021).…”

Section: Cleansingmentioning

confidence: 99%

Global Network Cyberattack Classification Using Naive Bayes Method Time Range 2020 – 2023

Sandi Mutia,

Irawan,

Juliane

2024

Astonjadro

View full text Add to dashboard Cite

This study focuses on developing a classification model for cyberattacks on global networks during the time span of 2020 to 2023 using the Naive Bayes method. The main objective of the study is to analyze and classify the frequent severity of cyber, which helps in improving network security and reducing vulnerabilities. The Naive Bayes method was chosen for its efficiency in handling large datasets and its ability to make predictions based on probabilities. Collecting cyberattack data from a variety of reliable and up-to-date sources, the study covers attacks such as ransomware, phishing, DDoS, and other malware. The classification process includes data pre-processing, feature extraction, and finally the application of Naive Bayes algorithms to identify patterns in such attacks. The classification results are then evaluated using the Apply Model and Performance validation methods to assess the effectiveness of the model. The results of this study show that Naive Bayes is able to accurately classify cyberattacks, providing a useful tool for cybersecurity professionals to understand attack trends and respond proactively. The study also suggests areas for further research, including the integration of the Naive Bayes model with other artificial intelligence systems for improved cyberattack detection. The study provides new insights into the application of the Naive Bayes method in cybersecurity and paves the way for improved data-driven cyber defense strategies.

show abstract

Data cleansing mechanisms and approaches for big data analytics: a systematic study

Cited by 8 publications

References 38 publications

SMARTEN—A Sample-Based Approach towards Privacy-Friendly Data Refinement

SMARTEN—A Sample-Based Approach towards Privacy-Friendly Data Refinement

AI and professional liability assessment in healthcare. A revolution in legal medicine?

Global Network Cyberattack Classification Using Naive Bayes Method Time Range 2020 – 2023

Contact Info

Product

Resources

About