Agile big data analytics: AnalyticsOps for data science

Grady, Nancy W.; Payne, Jerry A.; Parker, Huntley

doi:10.1109/bigdata.2017.8258187

Cited by 30 publications

(14 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Grady et al [8] note the similarity of data science projects with software development before the adoption of agile methodologies. Such similarities can also be seen in a study revealing difficulties related to processes in data science projects [18].…”

Section: Background and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Demystifying Data Science Projects: A Look on the People and Process of Data Science Today

Aho

Sievi-Korte

Kilamo

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Processes and practices used in data science projects have been reshaping especially over the last decade. These are different from their software engineering counterparts. However, to a large extent, data science relies on software, and, once taken to use, the results of a data science project are often embedded in software context. Hence, seeking synergy between software engineering and data science might open promising avenues. However, while there are various studies on data science workflows and data science project teams, there have been no attempts to combine these two very interlinked aspects. Furthermore, existing studies usually focus on practices within one company. Our study will fill these gaps with a multi-company case study, concentrating both on the roles found in data science project teams as well as the process. In this paper, we have studied a number of practicing data scientists to understand a typical process flow for a data science project. In addition, we studied the involved roles and the teamwork that would take place in the data context. Our analysis revealed three main elements of data science projects: Experimentation, Development Approach, and Multidisciplinary team(work). These key concepts are further broken down to 13 different sub-themes in total. The found themes pinpoint critical elements and challenges found in data science projects, which are still often done in an ad-hoc fashion. Finally, we compare the results with modern software development to analyse how good a match there is.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

“…Today's data science projects exhibit some problems that could be tackled with more mature project management methodologies [8,9]. These include minimal focus on identifying result quality and problems in estimating budget and scheduling in advance [18].…”

Section: Introductionmentioning

confidence: 99%

Demystifying Data Science Projects: A Look on the People and Process of Data Science Today

Aho

Sievi-Korte

Kilamo

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Londhe and Rao [7] presented various software frameworks available for BDA and discussed some widely used data mining algorithms. Grady et al [8] illustrated the implications of an agile process for the cleansing, transformation, and analytics of data in BDA. Vatrapu et al [9] demonstrated the suitability and effectiveness of BDA for conceptualizing, formalizing and analyzing of big social data from content-driven social media platforms e.g.…”

Section: Related Work a Big Data Analyticsmentioning

confidence: 99%

Big Data Analytics and Mining for Effective Visualization and Trends Forecasting of Crime Data

et al. 2019

View full text Add to dashboard Cite

Big data analytics (BDA) is a systematic approach for analyzing and identifying different patterns, relations, and trends within a large volume of data. In this paper, we apply BDA to criminal data where exploratory data analysis is conducted for visualization and trends prediction. Several the state-of-theart data mining and deep learning techniques are used. Following statistical analysis and visualization, some interesting facts and patterns are discovered from criminal data in San Francisco, Chicago, and Philadelphia. The predictive results show that the Prophet model and Keras stateful LSTM perform better than neural network models, where the optimal size of the training data is found to be three years. These promising outcomes will benefit for police departments and law enforcement organizations to better understand crime issues and provide insights that will enable them to track activities, predict the likelihood of incidents, effectively deploy resources and optimize the decision making process. INDEX TERMS Big data analytics (BDA), data mining, data visualization, neural network, time series forecasting.

show abstract

“…As a consequence, the collected data is large volumes of data and it is also in the unstructured data formats. The collected data is stored into different databases [9,10,17].…”

Section: Crime Data Analysismentioning

confidence: 99%

Crime Data Analysis, Visualization and Prediction Using LSTM

Manengadan¹,

Nandanan²,

Subash³

2021

IJDSA

View full text Add to dashboard Cite

Crimes are common social problems that can even affect the quality of life, even the economic growth of a country. Big Data Analytics (BDA) is used for analyzing and identifying different crime patterns, their relations, and the trends within a large amount of crime data. Here, BDA is applied to criminal data in which, data analysis is conducted for the purpose of visualization. Big data analytics and visualization techniques were utilized to analyze crime big data within the different parts of India. Here, we have taken all the states of Indian for analysis, visualization and prediction. The series of operations performed are data collection, data pre-processing, visualization and trends prediction, in which LSTM model is used. The data includes different cases of crimes with in different years and the crimes such as crime against women and children in which, kidnap, murder, rape. The predictive results show that the LSTM perform better than neural network models. Hence, the generated outcomes will benefit for police and law enforcement organizations to clearly understand crime issues and that will help them to track activities, predict the similar incidents, and optimize the decision making process.

show abstract

Agile big data analytics: AnalyticsOps for data science

Cited by 30 publications

References 7 publications

Demystifying Data Science Projects: A Look on the People and Process of Data Science Today

Demystifying Data Science Projects: A Look on the People and Process of Data Science Today

Big Data Analytics and Mining for Effective Visualization and Trends Forecasting of Crime Data

Crime Data Analysis, Visualization and Prediction Using LSTM

Contact Info

Product

Resources

About