Recently, for the recovery of images' processing history, passive forensics of possible manipulations has attracted wide interest. In particular, due to highly non-linearity, median filtering (MF) usually serves as an effective tool of counter forensic techniques for other image operations. Therefore, the importance of median filtering detection is selfevident. In this paper, through analysing the pixel differences of images, we found the indications to study the complex correlations introduced by median filtering and adopt two sets of describing features to measure them. More Specifically, we utilize a linear prediction model for the differences of image that is computed along a specific direction and estimate the prediction coefficients to construct a linear descriptor L. Besides, we make use of the histogram of rotation invariant local binary pattern (LBP) to form a nonlinear descriptor N . According to our observation, we also propose an enhanced feature EF to further improve the detection performance. Based on these, we present a novel median filtering detection scheme incorporating both the linear and nonlinear descriptors. Extensive experiments are carried out, which demonstrate that our proposed scheme gains favorable performance comparing to state-of-the-art methods, especially for low resolution images and JPEG compressed images, and shows resistance to noise attack.