Breast cancer overtakes lung cancer as the largest cancer in the world in 2021. Studies have found that the advancement of breast cancer is tightly associated with the estrogen receptor (ER). Thus, ERα antagonist has become an essential medicine in treating breast cancer. Considering the heterogeneity and adaptability of cancers, the development of suitable ERα antagonists has become an essential research target in the field of pharmaceutical sciences. In order to screen suitable active compounds as candidate medicines, pharmaceutical researchers build quantitative activity-structure relationship models of compounds by collecting molecular descriptors of compounds and their bioactivity data when acting on ERα. Then, an activity-structure relationship model was built quantitatively to predict which compound molecules are more suitable for the new medicine. To achieve this, this paper combines the machine learning algorithm with the particle swarm algorithm to develop a compound optimization model for the new anti-breast cancer medicine candidates. The optimization model is used to obtain a range of molecular descriptors which can provide better biological activation for ERα antagonist and better pharmacokinetic properties of the compound. The results of experiments in this paper show that the model performs better in the determination of the range of the major molecular descriptors, which can lead to better biological activity of the candidate compound for ERα antagonist and better pharmacokinetic properties.