Multivariate modeling and two-level scheduling of analytic queries

Liu, Zhuo; Nath, Asoke; Ding, Xiaoning; Fu, Huansong; Khan, Md. Muhib; Yu, Weikuan

doi:10.1016/j.parco.2019.01.006

Cited by 3 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [25] authors proposed a query processing approach by utilizing two-level scheduling and multivariate modeling. The semantic extraction-based query modeling was implemented by constructing a DAG based structure to examine the connectivity between the queries.…”

Section: Schemes Of Query Schedulingmentioning

confidence: 99%

See 1 more Smart Citation

SparkGrid: Blockchain Assisted Secure Query Scheduling and Dynamic Risk Assessment for Live Migration of Services in Apache Spark based Grid Environment

G.M.

Nalini

2022

Preprint

View full text Add to dashboard Cite

Grid computing is an emerging technology that enabled the heterogeneous collection of data and provisioning of services to the users. Due to the high amount of incoming heterogeneous request, grid computing needs an efficient scheduling to reduce execution time and satisfy Service Level Agreement (SLA) and Quality of Service (QoS) requirements. For that purpose, we proposed SprakGrid method to reduce execution time and satisfying SLA, QoS requirements. The proposed work includes four consecutive phases which are explained as follows, in first we perform user authentication in order to ensure the legitimacy of the users using Elliptic Curve based Chaos Theory (ECCT) algorithm which generate secret key and stored it into the blockchain. In second we perform query scheduling for resource discovery using Soft Actor Critic (SAC) algorithm by considering 3P’s parameters which is performed by spark environment that schedules optimal resources based on the service request. In third, we perform risk assessment and request dropping, in which the risk nodes of workers are evaluated by master node. To address the resource wastage by attacker, this research evaluates the risk value in a dynamic manner using Shannon entropy. Based on the risk assessment the requestsare classified into two classes such as normal and malicious. In fourth we perform service live migration, in which the malicious requests are dropped and normal request are migrated from source node to target node using Multi-Constraints based Emperor Penguin Optimization (MC-EPO). Finally, simulation is performed by GridSim and the simulation results demonstrate that the proposed SparkGrid method achieves superior performance compared to other state-of-the art methods.

show abstract

Section: Schemes Of Query Schedulingmentioning

confidence: 99%

“…where 𝑄 ∈ (0, 𝑛 × 𝑀), ∝∈ [0,4], 𝑃(0,2 × 𝑁), 𝛿 ∈ [1,2],𝑁 = 2 𝑖 and 𝑛 = 2 𝑖 with integers 1 and 1 [25]. The initial value of P and Q are exchanged using ECC based key generation.…”

Section: Userauthenticationmentioning

confidence: 99%

SparkGrid: Blockchain Assisted Secure Query Scheduling and Dynamic Risk Assessment for Live Migration of Services in Apache Spark based Grid Environment

G.M.

Nalini

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It also schedules the map and reduce tasks according to the processing capacity of nodes in the heterogeneous Hadoop cluster. Liu et al [23] proposed that the available resources in the Hadoop MapReduce cluster are partitioned among multiple organizations who collectively fund the cluster based on computing needs. MapReduce adopts a job-level scheduling policy for a balanced distribution of tasks and effective utilization of resources.…”

Section: Related Workmentioning

confidence: 99%

HybSMRP: a hybrid scheduling algorithm in Hadoop MapReduce framework

et al. 2019

View full text Add to dashboard Cite

IntroductionDistributed and parallel processing is one of the best intelligent ways to store and compute big data [1]. Most definitions defined big data as characterized by the 3Vs: the extreme volume of data, the wide variety of data types and the velocity at which the data must be processed. MapReduce [2] is a programming model for big data processing. MapReduce programs are intrinsically parallel [3,4]. MapReduce executes the programs in two phases, map and reduce, so that each phase is defined by a function called mapper and reducer. A MapReduce framework consists of a master and multiple slaves. The master is responsible for the management of the framework, including user interaction, job queue organization and task scheduling. Each slave has a fixed number of map and reduce slots to perform tasks. The job scheduler located in the master assigns tasks according to the number of free task slots AbstractDue to the advent of new technologies, devices, and communication tools such as social networking sites, the amount of data produced by mankind is growing rapidly every year. Big data is a collection of large datasets that cannot be processed using traditional computing techniques. MapReduce has been introduced to solve largedata computational problems. It is specifically designed to run on commodity hardware, and it depends on dividing and conquering principles. Nowadays, the focus of researchers has shifted towards Hadoop MapReduce. One of the most outstanding characteristics of MapReduce is data locality-aware scheduling. Data locality-aware scheduler is a further efficient solution to optimize one or a set of performance metrics such as data locality, energy consumption and job completion time. Similar to all situations, time and scheduling are the most important aspects of the MapReduce framework. Therefore, many scheduling algorithms have been proposed in the past decades. The main ideas of these algorithms are increasing data locality rate and decreasing the response and completion time. In this paper, a new hybrid scheduling algorithm has been proposed, which uses dynamic priority and localization ID techniques and focuses on increasing data locality rate and decreasing completion time. The proposed algorithm was evaluated and compared with Hadoop default schedulers (FIFO, Fair), by running concurrent workloads consisting of Wordcount and Terasort benchmarks. The experimental results show that the proposed algorithm is faster than FIFO and Fair scheduling, achieves higher data locality rate and avoids wasting resources.

show abstract

Parallel anomaly detection algorithm for cybersecurity on the highspeed train control system

Wang¹,

Hei²,

Ma³

et al. 2022

MBE

View full text Add to dashboard Cite

<abstract> <p>With the rapid development of the high-speed train industry, the high-speed train control system has now been exposed to a complicated network environment full of dangers. This paper provides a speculative parallel data detection algorithm to rapidly detect the potential threats and ensure data transmission security in the railway network. At first, the structure of the high-speed train control data received by the railway control center was analyzed and divided tentatively into small chunks to eliminate the inside dependencies. Then the traditional threat detection algorithm based on deterministic finite automaton was reformed by the speculative parallel optimization so that the inline relationship's influences that affected the data detection order could be avoided. At last, the speculative parallel detection algorithm would inspect the divided data chunks on a distributed platform. With the help of both the speculative parallel technique and the distributed platform, the detection deficiency for train control data was improved significantly. The results showed that the proposed algorithm exhibited better performance and scalability when compared with the traditional, non-parallel detection method, and massive train control data could be inspected and processed promptly. Now it has been proved by practical use that the proposed algorithm was stable and reliable. Our local train control center was able to quickly detect the anomaly and make a fast response during the train control data transmission by adopting the proposed algorithm.</p> </abstract>

show abstract

Multivariate modeling and two-level scheduling of analytic queries

Cited by 3 publications

References 32 publications

SparkGrid: Blockchain Assisted Secure Query Scheduling and Dynamic Risk Assessment for Live Migration of Services in Apache Spark based Grid Environment

SparkGrid: Blockchain Assisted Secure Query Scheduling and Dynamic Risk Assessment for Live Migration of Services in Apache Spark based Grid Environment

HybSMRP: a hybrid scheduling algorithm in Hadoop MapReduce framework

Parallel anomaly detection algorithm for cybersecurity on the highspeed train control system

Contact Info

Product

Resources

About