Prostate cancer aggressiveness and metastatic potential are influenced by gene expression, genomic aberrations, and cellular morphology. These processes are in turn dependent in part on the 3D structure of chromosomes, packaged inside the nucleus. Using chromosome conformation capture (Hi-C), we conducted a systematic genome architecture comparison on a cohort of cell lines that model prostate cancer progression, ranging from normal epithelium to bone metastasis. Here, we describe how chromatin compartmentalization identity (A- open vs. B-closed) changes with progression: specifically, we find that 48 gene clusters switch from the B to the A compartment, including androgen receptor, WNT5A, and CDK14. These switches could prelude transcription activation and are accompanied by changes in the structure, size, and boundaries of the topologically associating domains (TADs). Further, compartmentalization changes in chromosome 21 are exacerbated with progression and may explain, in part, the genesis of the TMPRSS2-ERG translocation: one of the main drivers of prostate cancer. These results suggest that discrete, 3D genome structure changes play a deleterious role in prostate cancer progression.