Going big: a large-scale study on what big data developers ask

Abstract

Software developers are increasingly required to write big data code. However, they find big data software development challenging. To help these developers it is necessary to understand big data topics that they are interested in and the difficulty of finding answers for questions in these topics. In this work, we conduct a large-scale study on Stackoverflow to understand the interest and difficulties of big data developers. To conduct the study, we develop a set of big data tags to extract big data posts from Stackoverflow; use topic modeling to group these posts into big data topics; group similar topics into categories to construct a topic hierarchy; analyze popularity and difficulty of topics and their correlations; and discuss implications of our findings for practice, research and education of big data software development and investigate their coincidence with the findings of previous work. CCS CONCEPTS • General and reference → Empirical studies; • Theory of computation → Concurrency.

Challenges in Docker Development

“…Finally, we refined by keeping tags that are significantly relevant to Docker and excludeding others. We used two heuristics and from previous work [4], [8] and [42] to measure the significance and relevance of each tag in the Docker tag set .…”

“…It provides 93,183 answer posts where 32,632 answer posts are considered as accepted answer posts. To reduce the noise, only accepted answers are taken into consideration by following earlier approaches [4], [8] and [33]. Finally, the data set contains 113,922 SoF questions and answers of which 81,290 (71.4%) are questions and 32,632 (28.6%) are accepted answers.…”

“…Preprocessing of Docker Posts. In this step, we removed code snippets marked with <code></code>, HTML tags, such as paragraph <p></p>, urls <a></a>, stop words such as 'a', 'how', 'can' by using NLTK stopwords corpus [26] for preprocessing the set of Docker posts to minimize the noise by following previous approach [42], [12] and [8]. Afterwards, bigram models were developed by using Gensim [18] since bigram models enhance the quality of text processing as reported by Tan et al [37].…”

“…An immense amount of knowledge is created within these communities, including question and answers on Docker technology, on configuration, troubleshooting, and various experiences. In the past, various works have turned to this vast repository of knowledge, using data mining techniques to understand the software developers' perspectives on various matters, such as DevOps [43], Machine Learning (ML) [5,11], Big Data [8], and software security [42].…”

An Empirical Study on the Use and Misuse of Java 8 Streams

“…Traditional big data systems, for which MapReduce is a popular backbone [3], minimize the complexity of writing massively distributed programs by facilitating processing on multiple nodes using succinct functional-like constructs. This makes writing parallel code easier, as writing such code can be difficult due to possible data races, thread interference, and contention [1,4,28].…”