Bridging Data Silos Using Big Data Integration

Abstract

With cloud computing, cheap storage and technology advancements, an enterprise uses multiple applications to operate business functions. Applications are not limited to just transactions, customer service, sales, finance but they also include security, application logs, marketing, engineering, operations, HR and many more. Each business vertical uses multiple applications which generate a huge amount of data. On top of that, social media, IoT sensors, SaaS solutions, and mobile applications record exponential growth in data volume. In almost all enterprises, data silos exist through these applications. These applications can produce structured, semi-structured, or unstructured data at different velocity and in different volume. Having all data sources integrated and generating timely insights helps in overall decision making. With recent development in Big Data Integration, data silos can be managed better and it can generate tremendous value for enterprises. Big data integration offers flexibility, speed and scalability for integrating large data sources. It also offers tools to generate analytical insights which can help stakeholders to make effective decisions. This paper presents the overview on data silos, challenges with data silos and how big data integration can help to stun them.

A multimodal clinical data resource for personalized risk assessment of sudden unexpected death in epilepsy

“…It may be difficult, time-consuming, and costly to clean and reorganize data, and studies' conclusions may be affected by hidden biases. Only 20% of data scientists' time is spent building models, analyzing, visualizing, and analyzing the data, while most of their time (80%) is spent cleaning and preparing data (Patel, 2019 ; Li et al, 2021 ). However, having high-quality data is not sufficient to say the system is data-driven.…”

A Big Data Reference Architecture for Emergency Management

“…The need for cooperation is widely recognized in emergency management [140], since responses require a great diversity of skills and resources. Big data integration and Extract, Transform, and Load (ETL) technologies can be crucial for breaking down and bridging data silos [141]. Moreover, the proposed reference architecture can help in organizing and classifying existing experiences and sharing best practices.…”

Can Big Data guide prognosis and clinical decisions in epilepsy?

“…There may also be hidden biases that affect conclusions of studies, and cleaning and reorganization of data may be difficult, time‐consuming, and expensive. It is estimated that data scientists spend 80% of their time obtaining, cleaning, and preparing data, and only 20% of their time building models, analyzing, visualizing, and drawing conclusions from that data 64,65 . Accessibility is important; it should be joinable (in a form that can be joined to other clinical data when necessary) and shareable (a data‐sharing culture within the hospital ecosystem so that data can be joined) 66 .…”

“…It is estimated that data scientists spend 80% of their time obtaining, cleaning, and preparing data, and only 20% of their time building models, analyzing, visualizing, and drawing conclusions from that data. 64,65 Accessibility is important; it should be joinable (in a form that can be joined to other clinical data when necessary) and shareable (a data-sharing culture within the hospital ecosystem so that data can be joined). 66 If clinicians/researchers do not have a coherent, accurate picture of patient flow, diagnostic processes, and complete longitudinal data acquisition processes of patients, it is hard to analyze and improve processes and care.…”