A fast algorithm is proposed to calculate the difference field RCS (d-RCS) of the electromagnetic scattering from an object above rough surface. The electric field integral equation (EFIE) of the difference induced field on the rough surface and the induced field on the target are derived, and is solved by an iterative solver. The characteristic basis functions (CBFs) are used to calculate the induced field on the rough surface, which is part of the right-hand side of the system. Since the coupling matrices between the object and rough surface and the non-self interaction matrices of the rough surface are rank deficient, it is accelerated by the adaptive cross approximation (ACA) algorithm. Through numerical experiments, it is concluded that the proposed method is efficient in analyzing the electromagnetic scattering from an object above rough surface.