Big Data analytics for prediction: parallel processing of the big learning base with the possibility of improving the final result of the prediction

Abstract

Purpose This paper aims to solve the problems of big data analytics for prediction including volume, veracity and velocity by improving the prediction result to an acceptable level and in the shortest possible time. Design/methodology/approach This paper is divided into two parts. The first one is to improve the result of the prediction. In this part, two ideas are proposed: the double pruning enhanced random forest algorithm and extracting a shared learning base from the stratified random sampling method to obtain a representative learning base of all original data. The second part proposes to design a distributed architecture supported by new technologies solutions, which in turn works in a coherent and efficient way with the sampling strategy under the supervision of the Map-Reduce algorithm. Findings The representative learning base obtained by the integration of two learning bases, the partial base and the shared base, presents an excellent representation of the original data set and gives very good results of the Big Data predictive analytics. Furthermore, these results were supported by the improved random forests supervised learning method, which played a key role in this context. Originality/value All companies are concerned, especially those with large amounts of information and want to screen them to improve their knowledge for the customer and optimize their campaigns.

Dynamic Distributed and Parallel Machine Learning algorithms for big data mining processing

“…Test 5. Presents the work presented by Djafri et al (2018), where he classified big data (KDD Cup 2012) using representative learning base and the classical random forests (CRF), as well as using the random forest classifier which he improved (IRF), but in a completely different method from the method proposed in this work. The results obtained are shown in Table 6:…”

“…Table 8 shows the final classification result of the original dataset (KDD Cup 2012) and the representative learning base (RLB) using Classical Random Forests (CRF) and Improved Random Forests (IRF) (Djafri et al , 2018).…”

“…RF (Random Forests) seem insensitive to over-fitting, this method generally does not require a lot of parameter optimization efforts. Random forests therefore avoid one of the main pitfalls of Big Data approaches in machine learning, which are taken for granted because we have talked about them in detail in our work (Djafri et al , 2018).…”

Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams

“…For such an amount of data, the design of the system handling the communication between the data source and the different parts of the analytic pipeline can create a major bottleneck. In addition to needing the right choice of software/hardware architecture, routine optimisation and partial processing (only selecting the fraction of data representative of the whole) are common workarounds [43]. • When we cannot process any faster, we try to process many things at once: A single computational node, even when using multiple cores (parallel computation) can only process as fast as its hardware enables; even considering the right data subset to process and the optimal algorithmic implementation, this threshold would constitute a theoretical upper limit.…”

Artificial intelligence in retail: The AI-enabled value chain

“…While some companies use data to create a competitive advantage, many businesses fall short of gaining real insights from their data. This can mainly be ascribed to big data requiring powerful technologies, computer processing power, skilled personnel and predictive models to crunch enormous amounts of data (Djafri et al, 2018; Gupta, 2018).…”