A study of big data computing platforms : performance, fairness and energy consumption

Abstract

Today, population aging is a trend spreading across the world. It is estimated that the number of people aged 60 years or over, is increasing rapidly. Although the elderly has wisdom and wealth gathered from their life experience, they often require long-term assistance from others. Ambient assisted living (AAL) systems open up a new opportunity to address the needs of aged by utilizing information and communication techniques. Multiple sensors and actuators coexist in ambient assisted living systems and are deployed everywhere surrounding elderly. A huge volume of sensed data is collected continuously from elderly, storage of these massive raw data and efficient processing on them to infer knowledge from the data and also guide the actuators to meet the needs of elderly is a problem we face. In the thesis, we mainly focus on the efficient large-scale data processing, we investigate whether and how we can build an effective large-scale data processing system to effectively excavate knowledge, wisdom and skills from elderly people to impact the development of the entire society.A lot of large-scale big data analytics systems have thereby emerged to process the massive data effectively. Efficient job scheduling and resource management for these data analytic frameworks are nontrivial. Modern job schedulers and resource coordinators in data processing frameworks often need to consider multiple objectives simultaneously due to various system operators requirements on data analytics. Currently, resource efficiency (throughput), job latency (per-job performance), fairness (isolation guarantee) and energy consumption are important concerns for the job scheduler in modern large-scale multiple-tenant environments. Resource efficiency is de facto the very important factor for the big data analytic framework. Job latency reflects the waiting time of the application. Fairness is a key building block of any multiple-tenant computing system that allows resource sharing effectively. Energy usage of the data center has reached 3% of the global electricity consumption while generating 200 million metric tons of CO2 in 2014. In order to reduce carbon emission and financial burden on the electricity, a lot of data centers have been re-designed and powered with multiple energy sources, including renewable (green) energy from non-polluting sources and brown energy from traditional polluting sources.Improving the resource efficiency and reducing the per-job latency are the common sense on those large-scale data processing frameworks and fruitful studies are proposed in these directions. In i I would like to thank my friends and colleagues in IGS-LILY and PDSL. I will always remember all the great memories we created together in the part four years.I gratefully acknowledge the scholarship provided by LILY, IGS(Interdisciplinary Graduate School) and Nanyang Technological University for my doctorial program.Last but not least, I want to thank my family and friends for their unwavering support throughput my PhD.