In many high level vision applications such as tracking and surveillance, background estimation is a fundamental step. In the past, background estimation was usually based on low level hand-crafted features such as raw color components, gradients, or local binary patterns. These existing algorithms observe performance degradation in the presence of various challenges such as dynamic backgrounds, photo-metric variations, camera jitter, and shadows. To handle these challenges for the purpose of accurate background estimation, we propose a unified method based on Generative Adversarial Network (GAN) and image inpainting. It is an unsupervised visual feature learning hybrid GAN based on context prediction. It is followed by a semantic inpainting network for texture optimization. We also propose a solution of arbitrary region inpainting by using center region inpainting and Poisson blending. The proposed algorithm is compared with the existing algorithms for background estimation on SBM.net dataset and for foreground segmentation on CDnet 2014 dataset. The proposed algorithm has outperformed the compared methods with significant margin.