Towards methods for systematic research on big data

Comparing Data Science Project Management Methodologies via a Controlled Experiment

“…For example, compared to software development, data science projects have an increased focus on data, what data is needed and the availability, quality and timeliness of the data [1,3,21]. This suggests that the factors driving the adoption of a more mature project methodology within a data science context might be different from the factors identified in other domains.…”

“…Researchers have begun to address the need for a team-based data science process methodology via case studies to understand effective practices and success criteria [1,15,17]. However, the need for more guidance is recognized; e.g., a recent Gartner Consulting report advocates for more careful management of analysis processes, though a specific methodology is not identified [18].…”

“…Data science projects typically have a goal to identify correlations and causal relationships, classify and predict events, identify patterns and anomalies, and infer probabilities, interest and sentiment [1]. Big Data is a related field, often thought of as a subset of data science, in that data science applies to large and small data sets and covers the end-to-end process of collecting, analyzing and communicating the results of the analysis.…”

Predicting data science sociotechnical execution challenges by categorizing data science projects

“…Typically, a data science project is described as a project that uses statistical and machine‐learning techniques on large volumes of unstructured and/or structured data generated by systems, people, sensors, or digital traces of information from people. This work is done in a distributed computing environment with a goal to identify correlations and causal relationships, classify and predict events, identify patterns and anomalies, and infer probabilities, interest, and sentiment (Das, Cui, Campbell, Agrawal, & Ramnath, ). Big Data is often thought of as a subset of data science, where the amount of data requires the use of special tools and algorithms.…”

“…As an example of other characteristics that might be important, two recent case studies (Das et al, , Saltz & Shamshurin, ) have implicitly suggested that a different way to characterize a project might be via the type of problem that a data science team is trying to solve. Beyond that, one might try to apply Bystrom and Jarvelin's () classification (clarity and difficulty of the project), but one could argue that most data science projects are complex (i.e., more difficult) and ambiguous (i.e., less clear).…”