Resource optimization is cost effective process in cloud. The efficiency of load balancing completely depends on how the infrastructure is utilizing. As per the current study, the resource optimization techniques are very costly and taking more convergence time to execute the task and load distribution among different virtual machines (VM). The objective of this paper is to develop a hybrid optimization algorithm to find the best virtual machine based on their fitness values and schedule different task to the fittest VM so that each task should get complete on time, and system can utilize the VM as well. The proposed algorithm is hybrid version of genetic (GA), ant-colony (Aco), and particle-swarm (Pso) algorithms, which is implemented and tested in amazon web service and compared with existing algorithms based on VM utilization, completion time, and cost. The proposed hybrid system genetic-aco-pso based algorithm (GAP) perform utmost while comparing with the existing systems.