Approaching the (Big) Data Science Engineering Process

PharmacoDB 2.0 : Improving scalability and transparency ofin vitropharmacogenomics analysis

“…Once the highly curated and standardized PSets are released via ORCESTRA, they need to be preprocessed into tables which match the PharmacoDB Entity Relationship Diagram (ERD) before being loaded into the database. To ensure that the data ingestion standards in PharmacoDB adhere to FAIR data principles, the Pharmaco-Data Ingestion (PharmacoDI) project was initiated to create an Extract Transform Load (ETL) pipeline which adheres to modern data engineering best practices (28,29).…”

Towards a Concept for Building a Big Data Architecture with Microservices

“…While initially being referred to as a synonym for large amounts of data that cannot be easily handled by relational databases and technologies of that time, today Big Data covers a variety of advanced data characteristics, technologies, paradigms and methods [1]. During this time, the concept has undergone significant changes that dramatically changed the term from a hype topic [7] to the foundation of most of the data-driven and data-intensive projects known today [4]. Despite that long lasting maturation and a highly active research community [8], no distinct and universally applied definition was found that precisely describes the nature and elements of that term [1].…”

“…Apart from the pure lack of experts and qualified staff [13], the comprehensive planning, engineering and integration of architectures represents a cumbersome task [1]. Many practitioners and researchers noted this problem and attempted to reduce the prevailing complexity through the design and developments of promising solutions, such as reference architectures [14], decision support systems [4], automation approaches [15] or the application of new technologies [6]. Especially in times at which highly decentralized or loosely coupled environments are sought more than ever, as in the case of very large business application scenarios, the use of Big Data in combination with the latter remains desirable.…”

“…Consequently, the performance of conventional methods and technologies are growing insufficient, thus motivating the invention of new techniques and the need of modern data architectures, as they are denoted by the terms Big Data and Big Data architecture [2]. Organizations that implement aforementioned technologies can experience a significant i ncrease i n p erformance [ 3] d ue t o m ore e fficient de cision ma king, re duced expenses and a more result-oriented portfolio of services while improving customer relations [4]. However, implementing the respective applications is still a challenging task [4].…”

“…Organizations that implement aforementioned technologies can experience a significant i ncrease i n p erformance [ 3] d ue t o m ore e fficient de cision ma king, re duced expenses and a more result-oriented portfolio of services while improving customer relations [4]. However, implementing the respective applications is still a challenging task [4]. Contributing to this problem, the newly implemented algorithms and solutions might evolve over time, thus rendering the effort pointless, if the Big Data architecture is not adapted accordingly.…”