A Roadmap Towards Big Data Opportunities, Emerging Issues and Hadoop as a Solution

Qayyum, Rida

doi:10.5815/ijeme.2020.04.02

Cited by 10 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At present, there are two methods for storing knowledge maps containing relational databases and graph databases. However, the amount of data continues to grow as information technology advances, and relational databases cause data redundancy, prompting some people to use graph databases [27]. A graph database is a new NoSQL database that uses a computer graph theory, which analyzes the internal relationships of data more intuitively and efciently.…”

Section: Knowledge Map Storagementioning

confidence: 99%

A Personalized Learning Path for French Study in Colleges Based on a Big Data Knowledge Map

Xiao

2023

Scientific Programming

View full text Add to dashboard Cite

The education industry is gradually improving with the rapid development of information technology. The learners use networks and computers to alter the traditional instructional framework based on educational information technology and achieve personalized learning. This teaching method emphasizes each learner’s identity and autonomy. However, due to the huge number of learning resources available on the Internet, students lack relevant courses, clear learning tasks, and the connection between various knowledge points, resulting in an unsatisfactory effect on the learning process. Knowledge maps for different learner types are created using historical learners’ conceptual knowledge and the segmentation and correlation technique of big data knowledge maps. Using a big data method in this process will automatically generate a set of weak conceptual learning pathways. For this problem, in the era of big data, people put forward the concept of knowledge map and used the algorithm based on the big data knowledge map to study the personalized learning path for college French. The content, structure, and relationship of college French knowledge points can be accurately expressed using this method, which is preferred by college administrators and teachers. This paper aims to investigate the personalized learning path for college French using a big data knowledge map, starting with the characteristics of a college French field of study. This study provides technical support in the establishment of a big data knowledge map based on a learning path recommendation framework. So, after the performance of several commonly used learning path recommendation algorithms, three French students have been selected at random for learning path planning. The results show that personalized learning path planning can be realized based on a knowledge map pre-repair relationship and objective attributes. In the analysis, not only the proposed technique is compared with the conventional optimization approach, but also a comparison study on the benefits of several learning effect prediction models is also performed. The results of this study suggest that this algorithm has a high learning efficiency and that the effective implementation of recommendations produced using our proposed strategy has a significant advantage.

show abstract

Section: Knowledge Map Storagementioning

confidence: 99%

A Personalized Learning Path for French Study in Colleges Based on a Big Data Knowledge Map

Xiao

2023

Scientific Programming

View full text Add to dashboard Cite

show abstract

“…In industry, the input data size of a Spark SQL application is typically from hundreds of Giga bytes to Tera bytes, even Peta bytes [49]. In such a case, it is very inconvenient, if feasible, to apply the above approaches to find the optimal configurations for Spark SQL applications.…”

Section: Motivationmentioning

confidence: 99%

“…Big data query systems such as Hive [58], Presto [50], and Spark SQL [4] have been widely deployed in industry to mine valued information from massive data efficiently [49]. As a higher level library on top of Apache Spark [68], Spark SQL not only inherits Spark's excellent big data processing capabilities, but also provides support for query-like large-scale data analysis, such as OnLine Analytical Processing (OLAP) [19,39].…”

Section: Introductionmentioning

confidence: 99%

LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications

Xin¹,

Hwang²,

Yu³

2022

Preprint

View full text Add to dashboard Cite

Spark SQL has been widely deployed in industry but it is challenging to tune its performance. Recent studies try to employ machine learning (ML) to solve this problem. They however suffer from two drawbacks. First, it takes a long time (high overhead) to collect training samples. Second, the optimal configuration for one input data size of the same application might not be optimal for others.To address these issues, we propose a novel Bayesian Optimization (BO) based approach named LOCAT to automatically tune the configurations of Spark SQL applications online. LOCAT innovates three techniques. The first technique, named QCSA, eliminates the configuration-insensitive queries by Query Configuration Sensitivity Analysis (QCSA) when collecting training samples. The second technique, dubbed DAGP, is a Datasize-Aware Gaussian Process (DAGP) which models the performance of an application as a distribution of functions of configuration parameters as well as input data size. The third technique, called IICP, Identifies Important Configuration Parameters (IICP) with respect to performance and only tunes the important parameters. As such, LOCAT can tune the configurations of a Spark SQL application with low overhead and adapt to different input data sizes.We employ Spark SQL applications from benchmark suites𝑇 𝑃𝐶− 𝐷𝑆, 𝑇 𝑃𝐶 − 𝐻 , and 𝐻𝑖𝐵𝑒𝑛𝑐ℎ running on two significantly different clusters, a four-node ARM cluster and an eight-node x86 cluster, to evaluate LOCAT. The experimental results on the ARM cluster show that LOCAT accelerates the optimization procedures of Tuneful [22], DAC [66], GBO-RL [36], and QTune [37] by factors of 6.4×, 7.0×, 4.1×, and 9.7× on average, respectively. On the x86 cluster, LOCAT reduces the optimization time of Tuneful, DAC, GBO-RL, and QTune by factors of 6.4×, 6.3×, 4.0×, and 9.2× on average, respectively. Moreover, LOCAT improves the performance of the applications on

show abstract

“…The computation of the delivery function across a peer-to-peer network is shared to cloud clones. The cloud clone provides greater access to system services than mobile devices [35], which protect privacy. This scheme offers a strong load balancing, low overhead and cost connectivity with increased privacy level in LB-ISA.…”

Section: Related Workmentioning

confidence: 99%

A Critical Survey on Privacy Prevailing in Mobile Cloud Computing: Challenges, State of the Art Methods and Future Directions

Qayyum¹

2020

IJWMT

View full text Add to dashboard Cite

A Roadmap Towards Big Data Opportunities, Emerging Issues and Hadoop as a Solution

Cited by 10 publications

References 24 publications

A Personalized Learning Path for French Study in Colleges Based on a Big Data Knowledge Map

A Personalized Learning Path for French Study in Colleges Based on a Big Data Knowledge Map

LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications

A Critical Survey on Privacy Prevailing in Mobile Cloud Computing: Challenges, State of the Art Methods and Future Directions

Contact Info

Product

Resources

About