Prostate cancer (PCa) with a high incidence worldwide is a serious threat to men's health. Despite the continuous development of treatment strategies for PCa in recent years, the long-term prognosis of patients is still poor. Hence, the discovery and development of novel, secure, and efficient therapeutic approaches hold significant clinical significance. Although sorafenib (SOR) displays potential as a therapeutic option for PCa, its clinical efficacy is hindered by drug resistance, limited water solubility, and rapid metabolism. Therefore, we proposed to prepare nanoparticles (named SOR@8P4 NPs) utilizing the phenylalanine-based poly(ester amide) polymer (8P4) as the drug carrier to enhance the solubility and drug stability of SOR and improve the therapeutic targeting and bioavailability. SOR@8P4 NPs had high stability and showed acid-responsive drug release at the acidic tumor microenvironment. Additionally, SOR@8P4 NPs demonstrated more remarkable anticancer, antimetastatic, and antiproliferative abilities in vitro, compared with those of free drugs. SOR@8P4 NPs showed high tumor targeting and significantly inhibited tumor growth in vivo. In summary, the drug delivery system of SOR@8P4 NPs provides new ideas for the clinical treatment of PCa.