BigSift: automated debugging of big data analytics in data-intensive scalable computing

Gulzar, Muhammad Ali; Wang, Siman; Kim, Miryung

doi:10.1145/3236024.3264586

Cited by 14 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gulzar et al [15] propose BigDebug for interactive debugging of big data software written in Spark. Similarly Gulzar et al [16] propose BigSift for automatic identification of the root cause of a debugging error. More details about these previous works can be found in Section 6.…”

Section: Rq1: Big Data Topicsmentioning

confidence: 99%

“…BigDebug supports interactive debugging through simulated breakpoints and on demand watchpoints to allow for inspection of a program without pausing its entire computation and retrieval of select intermediate data. Gulzar et al [16] propose BigSift for automatic identification of the root cause of an error. BigSift uses delta debugging and data provenance to identify the root cause of an error in a Spark program.…”

Section: Relatedmentioning

confidence: 99%

See 1 more Smart Citation

Going big: a large-scale study on what big data developers ask

Bagherzadeh

Khatchadourian

2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

129

View full text Add to dashboard Cite

Software developers are increasingly required to write big data code. However, they find big data software development challenging. To help these developers it is necessary to understand big data topics that they are interested in and the difficulty of finding answers for questions in these topics. In this work, we conduct a large-scale study on Stackoverflow to understand the interest and difficulties of big data developers. To conduct the study, we develop a set of big data tags to extract big data posts from Stackoverflow; use topic modeling to group these posts into big data topics; group similar topics into categories to construct a topic hierarchy; analyze popularity and difficulty of topics and their correlations; and discuss implications of our findings for practice, research and education of big data software development and investigate their coincidence with the findings of previous work. CCS CONCEPTS • General and reference → Empirical studies; • Theory of computation → Concurrency.

show abstract

Section: Rq1: Big Data Topicsmentioning

confidence: 99%

Section: Relatedmentioning

confidence: 99%

Going big: a large-scale study on what big data developers ask

Bagherzadeh

Khatchadourian

2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

129

View full text Add to dashboard Cite

show abstract

“…Software testing State-of-the-art techniques for software testing [25,34], statistical debugging [40,60], and bug localization [3,4,29] are often application-specific and/or require a user-defined test suite. Some approaches require the instrumentation of binaries or source code in the form of predicates that can be observed during computational runs [40,60].…”

Section: Related Workmentioning

confidence: 99%

BugDoc

Lourenço

Freire

Simon³

et al. 2022

The VLDB Journal

View full text Add to dashboard Cite

Applications in domains ranging from large-scale simulations in astrophysics and biology to enterprise analytics rely on computational pipelines. A pipeline consists of modules and their associated parameters, data inputs, and outputs, which are orchestrated to produce a set of results. If some modules derive unexpected outputs, the pipeline can crash or lead to incorrect results. Debugging these pipelines is difficult since there are many potential sources of errors including: bugs in the code, input data, software updates, and improper parameter settings. We present BugDoc, a system that automatically infers the root causes and derive succinct explanations of failures for black-box pipelines. BugDoc does so by using provenance from previous runs of a given pipeline to derive hypotheses for the errors, and then iteratively runs new pipeline configurations to test these hypotheses. Besides identifying issues associated with computational modules in a pipeline, we also propose methods for: "opportunistic group testing" to identify portions of data inputs that might be responsible for failed executions (what we call ), helping users narrow down the cause of failure; and "selective instrumentation" to determine nodes in pipelines that should be instrumented to improve efficiency and reduce the number of iterations to test. Through a case study of deployed workflows at a software company and an experimental evaluation using synthetic pipelines, we assess the effectiveness of BugDoc and show that it requires fewer iterations to derive root causes and/or achieves higher quality results than previous approaches.

show abstract

“…Moreover, printing runtime values of arguments may fill the log with extra information which will not help the developer to find the bug (e.g., information about not faulty execution). Alternatively, the developer could use more advanced techniques, such as data provenance [11], to detect which of the records caused the error. However, such a technique also requires various replays of the execution untill debugging can happen.…”

Section: Application Bugs By Example: Poll Analysis Applicationmentioning

confidence: 99%