An Optimized Iterative Semantic Compression Algorithm And Parallel Processing for Large Scale Data

Jin, R.; Chen, Gang; Tung, Anthony K. H.; Shou, Lidan; Ooi, Beng Chin

doi:10.3837/tiis.2018.06.018

Cited by 2 publications

(1 citation statement)

References 32 publications

(62 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apache Spark is an efficient and stretchable clustering computing system, which inherits MapReduce's linear scalability and fault tolerance on the Hadoop platform. However, Spark extends the MapReduce model in many ways and utilizes the RDD model for computing large-scale data in parallel [26]. The RDD model is the core component of Spark.…”

Section: The Spark Cloud Platformmentioning

confidence: 99%

A Hybrid Mechanism of Particle Swarm Optimization and Differential Evolution Algorithms based on Spark

2019

KSII TIIS

View full text Add to dashboard Cite

With the onset of the big data age, data is growing exponentially, and the issue of how to optimize large-scale data processing is especially significant. Large-scale global optimization (LSGO) is a research topic with great interest in academia and industry. Spark is a popular cloud computing framework that can cluster large-scale data, and it can effectively support the functions of iterative calculation through resilient distributed datasets (RDD). In this paper, we propose a hybrid mechanism of particle swarm optimization (PSO) and differential evolution (DE) algorithms based on Spark (SparkPSODE). The SparkPSODE algorithm is a parallel algorithm, in which the RDD and island models are employed. The island model is used to divide the global population into several subpopulations, which are applied to reduce the computational time by corresponding to RDD's partitions. To preserve population diversity and avoid premature convergence, the evolutionary strategy of DE is integrated into SparkPSODE. Finally, SparkPSODE is conducted on a set of benchmark problems on LSGO and show that, in comparison with several algorithms, the proposed SparkPSODE algorithm obtains better optimization performance through experimental results.

show abstract

Section: The Spark Cloud Platformmentioning

confidence: 99%

A Hybrid Mechanism of Particle Swarm Optimization and Differential Evolution Algorithms based on Spark

2019

KSII TIIS

View full text Add to dashboard Cite

show abstract

The Segmentation of Road Scenes Based on Improved ESPNet Model

Jin

Han

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

Image segmentation is an important research in image processing and machine vision in which automated driving can be seen the main application scene of image segmentation algorithms. Due to the many constraints of power supply and communication in in-vehicle systems, the vast majority of current image segmentation algorithms are implemented based on the deep learning model. Despite the ultrahigh segmentation accuracy, the problem of mesh artifacts and segmentation being too severe is obvious, and the high cost, computational, and power consumption devices required are difficult to apply in real-world scenarios. It is the focus of this paper to construct a road scene segmentation model with simple structure and no need of large computing power under the premise of certain accuracy. In this paper, the ESPNet (Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation) model is introduced in detail. On this basis, an improved ESPNet model is proposed based on ESPNet. Firstly, the network structure of the ESPNet model is optimized, and then, the model is optimized by using a small amount of weakly labeled and unlabeled scene sample data. Finally, the new model is applied to video image segmentation based on dash cam. It is verified on Cityscape, PASCAL VOC 2012, and other datasets that the algorithm proposed in this paper is faster, and the amount of parameters required is less than 1% of other algorithms, so it is suitable for mobile terminals.

show abstract

An Optimized Iterative Semantic Compression Algorithm And Parallel Processing for Large Scale Data

Cited by 2 publications

References 32 publications

A Hybrid Mechanism of Particle Swarm Optimization and Differential Evolution Algorithms based on Spark

A Hybrid Mechanism of Particle Swarm Optimization and Differential Evolution Algorithms based on Spark

The Segmentation of Road Scenes Based on Improved ESPNet Model

Contact Info

Product

Resources

About