Data Integration in Big Data Environment

Arputhamary, B.; Arockiam, L.

doi:10.9756/bijdm.8001

Cited by 13 publications

(9 citation statements)

References 18 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to implement a Big Data ecosystem, the first steps are the data extraction and data loading. Here, the ETL stages come to mind, Figure 2 describes the three applied tasks in a traditional data warehousing process (Arputhamary and Arockiam, 2015). In Hadoop, ETL process becomes Extract, Load and Transform (ELT) targeting processing time reduction; however, in practice, during data loading step it was clear the existence of useless and redundant fields and transformation process improves if data is cleaned first, thus, we propose: Extract, Cleaning, Load and Transform (ECLT).…”

Section: Etl Processmentioning

confidence: 99%

“…Comparing our attained results with other researches, the main difference is probably the nature of the data, since they propose to discard data at row level (Aye, 2011), and we consider important all the security log rows; for that reason, our data cleaning proposal is based only in a vertical dimension. This intuitive solution is easily implementable and gives good results, however, this proposal does not fully comply some main characteristics to be considered as a suitable solution (Arputhamary and Arockiam, 2015): Reliability, Maintainability, Freshness, Recoverability, Scalability, Availability, Traceability, Auditability and Maintenance. For instance, Scalability since the code lines for data cleaning must be created in house according the company's requirements, automatic Recoverability due to the need of the human intervention to restore the data cleaning script.…”

Section: Intuitive Proposal For Data Cleaningmentioning

confidence: 99%

“…Security data can be considered as Big Data, since it complies with five Vs, the defining properties of Big Data (Arputhamary and Arockiam, 2015):…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Data Cleaning Technique for Security Big Data Ecosystem

Martinez-Mosquera

Luján-Mora

2017

Proceedings of the 2nd International Conference on Internet of Things, Big Data and Security

View full text Add to dashboard Cite

The information networks growth have given rise to an ever-multiplying number of security threats; it is the reason some information networks currently have incorporated a Computer Security Incident Response Team (CSIRT) responsible for monitoring all the events that occur in the network, especially those affecting data security. We can imagine thousands or even millions of events occurring every day and handling such amount of information requires a robust infrastructure. Commercially, there are many available solutions to process this kind of information, however, they are either expensive, or cannot cope with such volume. Furthermore, and most importantly, security information is by nature confidential and sensitive thus, companies should opt to process it internally. Taking as case study a university's CSIRT responsible for 10,000 users, we propose a security Big Data ecosystem to process a high data volume and guarantee the confidentiality. It was noted during implementation that one of the first challenges was the cleaning phase after data extraction, where it was observed that some data could be safely ignored without affecting result's quality, and thus reducing storage size requirements. For this cleaning phase, we propose an intuitive technique and a comparative proposal based on the Fellegi-Sunter theory.

show abstract

Section: Etl Processmentioning

confidence: 99%

Section: Intuitive Proposal For Data Cleaningmentioning

confidence: 99%

See 1 more Smart Citation

Data Cleaning Technique for Security Big Data Ecosystem

Martinez-Mosquera

Luján-Mora

2017

Proceedings of the 2nd International Conference on Internet of Things, Big Data and Security

View full text Add to dashboard Cite

show abstract

“…Another element to take into consideration is the fact that the value of data increases exponentially when it is linked and fused with other data. Hence addressing the data integration challenge is critical to realizing the promise of Big Data [5], and unfortunately existing data warehousing techniques are inefficient to handle such integration [6]. Indeed, traditional data warehouses integrate structured, transactional data that is contained within relational databases.…”

Section: Introductionmentioning

confidence: 99%

Survey: When semantics meet crowdsourcing to enhance big data variety

Dhayne

Kilany

Sokhn

2018

2018 IEEE Middle East and North Africa Communications Conference (MENACOMM)

View full text Add to dashboard Cite

With the rapid growth of collected data and the variety of its content, the need for efficient integration at a Big Data level becomes crucial. Semantic technologies, as a means of integration and coordination of heterogeneous systems, may help big data to manage terminology and relationships to link various data from different data sources. However, and due to the difficulty of integration and analytics of some datasets with high-precision, automated processes cannot reach a high level of accuracy without the human cognitive ability. Crowdsourcing platforms have the potential to integrate (entity matching, entity resolution) and analyze (sentiment analysis, image recognition) heterogeneous data sources when in some cases these integration tasks may prove to be problematic for computers. In this survey, we explore and compare empirical research studies that rely on merging semantic and crowdsourcing technologies. And, in the light of this comparison, we propose a high-level integration workflow, which shows how merging these technologies can enhance the big data integration process and tackle the data analysis challenges.

show abstract

“…More recent studies have confirmed that maintenance is a major cost issue, with a ratio between maintenance costs and added-value higher than 25% in some sectors (Sophie et al, 2014). In fact, data quality practices -including maintenance reports -have a considerable impact on maintenance tasks, risks and business performance since poor data quality results in losses across a number of fronts (Arputhamary & Arockiam., 2015), and reciprocally, high data quality fosters enhanced business activities and decisionmaking.…”

Section: Introductionmentioning

confidence: 99%

Data quality assessment of maintenance reporting procedures

Madhikermi

Kubler

Robert

et al. 2016

Expert Systems with Applications

View full text Add to dashboard Cite

Today's largest and fastest growing companies' assets are no longer physical, but rather digital (software, algorithms. . . ). This is all the more true in the manufacturing, and particularly in the maintenance sector where quality of enterprise maintenance services are closely linked to the quality of maintenance data reporting procedures. If quality of the reported data is too low, it can results in wrong decision-making and loss of money. Furthermore, various maintenance experts are involved and directly concerned about the quality of enterprises' daily maintenance data reporting (e.g., maintenance planners, plant managers. . . ), each one having specific needs and responsibilities. To address this Multi-Criteria Decision Making (MCDM) problem, and since data quality is hardly considered in existing expert maintenance systems, this paper develops a Maintenance Reporting Quality Assessment (MRQA) dashboard that enables any company stakeholder to easily -and in real-time -assess/rank company branch offices in terms of maintenance reporting quality. From a theoretical standpoint, AHP is used to integrate various data quality dimensions as well as expert preferences. A use case describes how the proposed MRQA dashboard is being used by a Finnish multinational equipment manufacturer to assess and enhance reporting practices in a specific or a group of branch offices.

show abstract

Data Integration in Big Data Environment

Abstract: Abstract---Data Integration is the process of transferring the data in source format into the destination format. Many

Cited by 13 publications

References 18 publications

Data Cleaning Technique for Security Big Data Ecosystem

Data Cleaning Technique for Security Big Data Ecosystem

Survey: When semantics meet crowdsourcing to enhance big data variety

Data quality assessment of maintenance reporting procedures

Contact Info

Product

Resources

About