V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows

Ghit, Bogdan; Capotă, Mihai; Hegeman, Tim; Hidders, Jan; Epema, Dick; Iosup, Alexandru

doi:10.1109/ccgrid.2014.97

Cited by 6 publications

(12 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each of these studies required the development of new systems for measurement and analysis, including Multi-Probe [62] and BTWorld [63], which are both global-scale monitors for BT-ecosystems, the former also focusing on collecting Internet-tracing data (not possible anymore under GDPR laws), the latter focusing on efficient collection of aggregate-data. In 2014, while trying to analyze the full BTWorld-dataset, we developed a novel big data analytics pipeline [67]; the process allowed us to discover the phenomenon of vicissitude [38] (see Section 2.5).…”

Section: The Design Of P2p Systemsmentioning

confidence: 99%

“…Ecosystems are super-distributed [1]: they are recursively distributed, with their constituents often being distributed (eco)systems; yet, FRs and NFRs in distributed systems are not known to be directly composable across ecosystems. Various dynamic phenomena appear in distributed ecosystems, seemingly unique situations that do not fit the patterns expected from current theory and practice; for example, vicissitude [38] is a class of phenomena where several known bottlenecks appear seemingly at random in various parts of the system, performance variability is common in clouds [39], datacenter networks [40], and big data operations [41], and ecosystem owners spar with each other (e.g., in Jan 2019, Apple denied Facebook and Google access to its APIs, Unity changed their Terms-of-Service and thus locked out small developers like SpatialOS).…”

Section: New Challenges In Mcs Designmentioning

confidence: 99%

“…An evolving reference architecture for the datacenter ecosystem: (top) Our understanding of the big data ecosystem, 2011-2016. Reproduced and adapted from our previous work[38]. (bottom) Our understanding of the entire datacenter ecosystem, 2016-ongoing.…”

mentioning

confidence: 99%

See 2 more Smart Citations

The AtLarge Vision on the Design of Distributed Systems and Ecosystems

Iosup

Versluis

Trivedi

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

Self Cite

View full text Add to dashboard Cite

High-quality designs of distributed systems and services are essential for our digital economy and society. Threatening to slow down the stream of working designs, we identify the mounting pressure of scale and complexity of (eco-)system, of ill-defined and wicked problems, and of unclear processes, methods, and tools. We envision design itself as a core research topic in distributed systems, to understand and improve the science and practice of distributed (eco-)system design. Toward this vision, we propose the ATLARGE design framework, accompanied by a set of 8 core design principles. We also propose 10 key challenges, which we hope the community can address in the following 5 years. In our experience so far, the proposed framework and principles are practical, and lead to pragmatic and innovative designs for large-scale distributed systems. arXiv:1902.05416v1 [cs.DC] 14 Feb 2019 2. Why Focus on MCS Design?We argue in this section for the timely and important need to focus on MCS design. Not only is (good) design needed (Section 2.1), but we identify an increasing need for good design (Section 2.2) and designers (Section 2.3).We also analyze what good design needs to address, that is, complex challenges from system design (Section 2.4) and from MCS design (Section 2.5).3. We anonymize the venue, but consider it relevant because its held year is after 2014, the venue is a conference, and its ranking is A in CORE18 and green in MSAR14. For comparison, ICDCS has these rankings too.4. We anonymize the university, but consider the course relevant because it is large, it took place after 2014, and the university is ranked in the top-150 (in computer science) in both the THE and the QS 2018 World University Rankings (out of nearly 1,000 universities), and in Webometrics of July 18 (out of over 28,000).

show abstract

Section: The Design Of P2p Systemsmentioning

confidence: 99%

Section: New Challenges In Mcs Designmentioning

confidence: 99%

See 1 more Smart Citation

The AtLarge Vision on the Design of Distributed Systems and Ecosystems

Iosup

Versluis

Trivedi

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Chains and workflows of MapReduce jobs are useful [52], but could be difficult to manage and troubleshoot. Vicissitudeworkflows of MapReduce jobs lead to diverse challenges, by stressing different system resources at different or even the same time [39]. Workloads can be dominated by a few MapReduce jobs, used periodically or in bursts [13].…”

Section: Mapreduce-based Data-intensive Batch Processingmentioning

confidence: 99%

Self-awareness of Cloud Applications

Iosup

Zhu

Merchant

et al. 2017

Self-Aware Computing Systems

View full text Add to dashboard Cite

Note: This is an extended survey. A much shorter, revised version of this material will be available in print, as part of a Springer book on "Self-Aware Computing". The book is due to appear in 2017.Abstract Cloud applications today deliver an increasingly larger portion of the Information and Communication Technology (ICT) services. To address the scale, growth, and reliability of cloud applications, self-aware management and scheduling are becoming commonplace. How are they used in practice? In this chapter, we propose a conceptual framework for analyzing state-of-the-art self-awareness approaches used in the context of cloud applications. We map important applications corresponding to popular and emerging application domains to this conceptual framework, and compare the practical characteristics, benefits, and drawbacks of self-awareness approaches. Last, we propose a roadmap for addressing open challenges in self-aware cloud and datacenter applications.

show abstract

“…This property signies a combination between the large volume of data and the complexity of processing workow, which prevent to gather useful insights in data [18].…”

Section: Big Data Applicationsmentioning

confidence: 99%

Analysis of power consumption in heterogeneous virtual machine environments

et al. 2016

View full text Add to dashboard Cite

Reduction of energy consumption in Cloud computing datacenters today is a hot a research topic, as these consume large amounts of energy. Furthermore, most of the energy is used ineciently because of the improper usage of computational resources such as CPU, storage, and network. A good balance between the computing resources and performed workload is mandatory. In the context of data-intensive applications, a signicant portion of energy is consumed just to keep alive virtual machines or to move data around without performing useful computation. Moreover, heterogeneity of resources increases the degree of diculty, when try to achieve energy eciency. Power consumption optimization requires identication of those inefciencies in the underlying system and applications. Based on the relation between server load and energy consumption, we study the eciency of data-intensive applications, and the penalties, in terms of power consumption, that are introduced by dierent degrees of

show abstract

V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows

Cited by 6 publications

References 9 publications

The AtLarge Vision on the Design of Distributed Systems and Ecosystems

The AtLarge Vision on the Design of Distributed Systems and Ecosystems

Self-awareness of Cloud Applications

Analysis of power consumption in heterogeneous virtual machine environments

Contact Info

Product

Resources

About