Impulse noise is often introduced to images when captured through image sensors due to sharp and sudden disturbances in the image signal, analog-to-digital converter errors, sensor temperature, etc., severely degrading their visual quality. Therefore, it is essential to develop an effective method to remove image noise. We propose a novel image denoising method for "salt-and-pepper" (SP) noise, using cascaded filtering based on overlapped adaptive Gaussian smoothing (OAGS) and the convolutional refinement networks (CRNs). First, the noisy input image can be preliminarily denoised by OAGS, where the noisy pixels are removed and recovered. The CRNs refine the result by restoring fine details for the denoised image. Through extensive experimental results, we demonstrate the proposed method substantially outperforms other state-of-the-art denoising methods, especially for high-density SP noise.