Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture

Freymann, Andreas; Maier, Florian; Schaefer, Kristian; Böhnel, Tom

doi:10.5220/0009388602490256

Cited by 5 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from the pure lack of experts and qualified staff [13], the comprehensive planning, engineering and integration of architectures represents a cumbersome task [1]. Many practitioners and researchers noted this problem and attempted to reduce the prevailing complexity through the design and developments of promising solutions, such as reference architectures [14], decision support systems [4], automation approaches [15] or the application of new technologies [6]. Especially in times at which highly decentralized or loosely coupled environments are sought more than ever, as in the case of very large business application scenarios, the use of Big Data in combination with the latter remains desirable.…”

Section: Big Datamentioning

confidence: 99%

“…Both describe how they build their architecture and they also described that the main challenge is to keep the data integrity. Freymann et al described how modular services such as microservices can be used to tackle the six fundamental challenges: Volume, Velocity, Variety, Complexity, Veracity and Value [6]. They found that the modular architecture should have the properties described in Table 2.…”

Section: Microservices In Big Datamentioning

confidence: 99%

“…This results in a rising demand for designs, which consist of building blocks that can be modified and replaced independently from each other, contrasting isolated implementations and applications [5]. Implementing microservices as the building blocks of Big Data architecture appears to be a feasible solution due to high degree of modularization [6]. Therefore, the research question is:…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards a Concept for Building a Big Data Architecture with Microservices

et al. 2021

View full text Add to dashboard Cite

Microservices and Big Data are renowned hot topics in computer science that have gained a lot of hype. While the use of microservices is an approach that is used in modern software development to increase flexibility, Big Data allows organizations to turn today’s information deluge into valuable insights. Many of those Big Data architectures have rather monolithic elements. However, a new trend arises in which monolithic architectures are replaced with more modularized ones, such as microservices. This transformation provides the benefits from microservices such as modularity, evolutionary design and extensibility while maintaining the old monolithic product’s functionality. This is also valid for Big Data architectures. To facilitate the success of this transformation, there are certain beneficial factors. In this paper, those aspects will be presented and the transformation of an exemplary Big Data architecture with somewhat monolithic elements into a microservice favoured one is outlined.

show abstract

Section: Big Datamentioning

confidence: 99%

Section: Microservices In Big Datamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards a Concept for Building a Big Data Architecture with Microservices

et al. 2021

View full text Add to dashboard Cite

show abstract

“…As a result, those microservice architectures were not only mentioned as a promising approach in the course of the interviews, but also gain interest from academics and practitioners [73]. These services are described as autonomous and isolated pieces of software, which means that they are designed, developed, tested and deployed independently and have only one small purpose each [63], [71].…”

Section: Facilitating the Modifiability Of Bda Solutionsmentioning

confidence: 99%

“…For instance, if new data sources arise, the only task of the development department is to integrate the capabilities to acquire data from that specific source. While more and more functionalities can be provided over time [73], the use of the system has value as soon as a first chain of logically coupled microservices is integrated [71]. As a consequence, businesses in rapidly evolving dynamic areas might profit from integrating microservice architectures to get an edge over their competitors when the conditions for business success are tied to the fast processing and integration of new data respectively new data sources.…”

Section: Facilitating the Modifiability Of Bda Solutionsmentioning

confidence: 99%

Discussing Relations Between Dynamic Business Environments and Big Data Analytics

Staegemann

Volk

Daase

et al. 2020

CSIMQ

View full text Add to dashboard Cite

Big data attracts researchers and practitioners around the globe in their desire to effectively manage the data deluge resulting from the ongoing evolution of the information systems domain. Consequently, many decision makers attempt to harness the potentials arising with the use of those modern technologies in a multitude of application scenarios. As a result, big data has gained an important role for many businesses. However, as of today, the developed solutions are oftentimes perceived as completed products, without considering that the application in highly dynamic environments might benefit from a deviation of this approach. Relevant data sources as well as the questions that are supposed to be answered by their analysis may change rapidly and so do subsequently the requirements regarding the functionalities of the system. To our knowledge, while big data itself is a prominent topic, fields of application that are likely to evolve in a short period of time and the resulting consequences were not specifically investigated until now. Therefore, this research aims to overcome this paucity by clarifying the relation between dynamic business environments and big data analytics (BDA), sensitizing researchers and practitioners for future big data engineering activities. Apart from a thorough literature review, expert interviews are conducted that evaluate the made inferences regarding dynamic and stable influencing factors, the influence of dynamic environments on BDA applications as well as possible countermeasures. The ascertained insights are condensed into a proposal for decision making, facilitating the alignment of BDA and business needs in dynamic business environments.

show abstract