Upgrading a high performance computing environment for massive data processing

Abstract

High-performance computing (HPC) and massive data processing (Big Data) are two trends that are beginning to converge. In that process, aspects of hardware architectures, systems support and programming paradigms are being revisited from both perspectives. This paper presents our experience on this path of convergence with the proposal of a framework that addresses some of the programming issues derived from such integration. Our contribution is the development of an integrated environment that integretes (i) COMPSs, a programming framework for the development and execution of parallel applications for distributed infrastructures; (ii) Lemonade, a data mining and analysis tool; and (iii) HDFS, the most widely used distributed file system for Big Data systems. To validate our framework, we used Lemonade to create COMPSs applications that access data through HDFS, and compared them with equivalent applications built with Spark, a popular Big Data framework. The results show that the HDFS integration benefits COMPSs by simplifying data access and by rearranging data transfer, reducing execution time. The integration with Lemonade facilitates COMPSs's use and may help its popularization in the Data Science community, by providing efficient algorithm implementations for experts from the data domain that want to develop applications with a higher level abstraction.

English Language Features in Linguistics by High-Performance Computing

“…Observations show that ants leave a secretion during their movement, and ants behind them can make a biased path choice based on the secretion left by the ants in front. is constitutes a positive feedback mechanism for learning information, and ants seek the shortest path to food through this information exchange [18,19].…”

Application of Neural Networks in Distribution of the Load in Cluster-Based Web Systems

“…The load balancing methods [10] are used to distribute tasks and loads between nodes in the system. These methods help to improve resource utilization, shorten response times, and enable flexible scalability, which is an essential part of cloud computing [11,12].…”

DDF Library: Enabling functional programming in a task-based model

“…In that way, different resources will be used to execute, in parallel, the different tasks. COMPSs is able to transfer, transparently, files that are used as input parameter for a task; it also supports shared disks or HDFS (by using a connector [27]) to speed up the read step. As an example, Fig.…”

“…Recent frameworks, such as COMPSs and Spark, implement different schedulers to explore data locality and other policies transparently to users. In addition, distributed storage systems (e.g., HDFS, Cassandra, Hive, among others) that are supported in many frameworks like Spark (natively) or COMPSs (through an API [27]) can help the schedulers by increasing the possibilities of improving data locality when reading files. Frameworks that use a conventional file system typically adopt as a rule that input files will be located on the master computer and will be transferred over the network when required by a task.…”