In building change detection task, factors such as phenological changes, illumination changes, and registration errors will cause unchanged areas in remote sensing images to have obvious differences in pixels, which will lead to pseudochanges in results. Existing methods focus on the change information of multi-temporal remote sensing images, ignoring the exploration of pseudo-change problems. Therefore, FODA (Feature-Output space Dual-Alignment) method is proposed to reduce the negative effect of the pseudo-change problem by paying attention to the relationship between unchanged areas of multi-temporal images. On the one hand, FODA narrows the distance between the features of the unchanged areas in the feature space, increasing its feature extraction ability of pseudochanged areas. On the other hand, given the spatial context of image scene implicit in the output space, the ability to recognize pseudo-changes of the FODA is improved through an adversarial learning procedure. Due to its simplicity and effectiveness, FODA achieves 88.73% and 82.75% F1 scores on the LEVIR-CD dataset and WHU-CD dataset respectively. Compared with stateof-the-art methods, FODA can effectively reduce the problem of pseudo-changes and significantly improve the effect of change detection even only based on a simple backbone model.