Abstract-Clouds provide the abstraction of nearly-unlimited computing resources through the elastic use of federated resource pools (virtualized datacenters). They are being increasingly considered for HPC applications, which have traditionally targeted grids and supercomputing clusters. However, maximizing energy efficiency and utilization of cloud datacenter resources, avoiding undesired thermal hotspots (due to overheating of over-utilized computing equipment), and ensuring quality of service guarantees for HPC applications are all conflicting objectives, which require joint consideration of multiple pairwise tradeoffs. The novel concept of heat imbalance, which captures the unevenness in heat generation and extraction, at different regions inside a HPC cloud datacenter is introduced. This thermal awareness enables proactive datacenter management through prediction of future temperature trends as opposed to the state-of-the-art reactive management based on current temperature measurements. VMAP, an innovative proactive thermal-aware virtual machine consolidation technique is proposed to maximize computing resource utilization, to minimize datacenter energy consumption for computing, and to improve the efficiency of heat extraction. The effectiveness of the proposed technique is verified through experimental evaluations with HPC workload traces under singleas well as federated-datacenter scenarios (in the machine rooms at Rutgers University and University of Florida).