Implementation of cost effective hierarchical Hadoop cluster–a case study for education

Abstract

To equip the younger generation of the province with computing skills and provide them access to a wide variety of modern educational resources such as multimedia based on educational content in government schools and colleges to set a strong foundation at an early stage of their education. Educational Media help to empower educational institutions by altering the way of using Information Communication Technology (ICT). To put this into practice, several challenges need to be addressed. It requires a scalable technological architecture and algorithms to form a cluster in which effective resource sharing like CPU, Memory among multiple nodes, tools to monitor, assess and evaluate data under hierarchical Hadoop cluster is needed. By analyzing text, audios, videos information, Periodic reports will be generated to assist the students, teachers and Government. In this case, a software framework is required to process big data stored in hierarchical nodes. As the architecture of Hadoop, an open source software framework doesn't support processing the data stored in hierarchical nodes. This case study proposes the Hierarchical Hadoop cluster to alter the way of using ICT. The proposed work helps in monitoring, reporting the usage of ICT and also acts a help desk to address the issues of the educational institutions. This establishes a novel communication media by generating reports on text, audio, video information based on analysis.

An improved hadoop load rebalancer

“…The queue with the lowest load is chosen, from which the oldest remaining job is chosen, When a Task Tracker slot becomes free. In general, this has the effect of enforcing cluster capacity sharing among users, rather than among jobs [14].…”

“…Currently rebalancing is done on demand. The administrator has to issue a command to ensure the load is balanced across the nodes [12][13][14][15][16]. We propose a method to make use of the replicated data to maintain the load balance with minimal cost.…”