PolSAR provides important support for built-up areas (BA) information analysis, due to the ability of weather independent imaging and sensitivity to targets scattering and geometric characteristics. However, PolSAR BA with large orientation angles are usually misdetected as vegetation, and labeled BA samples with special orientations are difficult to obtain. Furthermore, the labeled BA samples and trained models can hardly work well in cross-domain PolSAR imagery BA analysis. This paper presents a PolSAR BA extraction method based on eigenvalue statistical components (ESC) and PU-Learning (PUL), and it helps to realize cross-domain BA extraction by combining subspace alignment (SA). Firstly, the roll invariance of coherency-matrix eigenvalues and building orientation effects are analyzed. Then, by adopting eigenvalue-Wishart unsupervised classification, regional statistical information and rotation invariant property are comprehensively utilized in ESC. Finally, the BA can be extracted by combining PUL classifier with only positive samples at same distinguishable orientation. Combined with SA, the novel ESC-PUL-SA domain adaptation facilitates robust unsupervised cross-domain PolSAR BA analysis, reducing the differences caused by sensors and imaging scenes. The ESC-PUL BA extraction on seven PolSAR imageries showed that the accuracies reach 92-98% with only a few positive samples (less than 0.65%). The ESC-PUL-SA performance was further validated by 14 unsupervised cross-domain BA analysis units among ten datasets, including Radarsat-2, Gaofen-3, AirSAR and UAVSAR images. With randomly selected positive samples from source domain, the proposed ESC-PUL-SA achieved accuracies of all cross-domain BA extraction range from 89.64-95.53%.