Successful Data Science Projects: Lessons Learned from Kaggle Competition

Abstract

Abstract:The workflow from data understanding to deployment of an analytical model of a data science project begins at framing the problem at hand, a task that is typically business-oriented and requires humanto-human interaction. However, the next three steps: data understanding, feature extraction, and model building that come next in the pipeline are the key to successful data science projects. Failing to fully understand the requirements of each of these three steps can negatively affect the performance of the proposed system. Hence, the current study tries to answer the following question "What are the requirements of a successful data science project?" To answer this question, we will use the solution that we built to measure the relevance of local search results of small online e-businesses and submitted to Kaggle data science platform to shed light on why our solution did not achieve a top position among other competitors. Evaluation of the design that we submitted to the competition is going to be carried out in the spirit of the three winning submissions. Our results revealed that well-performed data preprocessing, well-defined features, and model ensembling are critical for building successful data science projects. Such a clarification provides insight into specific aspects of model design to help others including Kagglers avoid possible mistakes while approaching their data science projects.

Online Data Preprocessing: A Case Study Approach

“…If any of the data preprocessing phases is not correctly handled, machine-learning algorithms will not run or can give misleading results [2]. Having in mind the significance of data preprocessing to data science [3], this study discusses data preprocessing and how we applied it to Flight MH370 social data.…”