In the design of offshore engineering foundations, a critical consideration involves determining the peak shear strength of marine soft clay sediment. To enhance the accuracy of estimating this value, a database containing 729 direct shear tests on marine soft clay sediment was established. Employing a machine learning approach, the Particle Swarm Optimization algorithm (PSO) was integrated with the Adaptive Boosting Algorithm (ADA) and Back Propagation Artificial Neural Network (BPANN). This novel methodology represents the initial effort to employ such a model for predicting the peak shear strength of the soil. To validate the proposed approach, four conventional machine learning algorithms were also developed as references, including PSO-optimized BPANN, Support Vector Machine (SVM), BPANN, and ADA-BPANN. The study results show that the PSO-BPANN model, which has undergone optimization via Particle Swarm Optimization (PSO), has prediction accuracy and efficiency in determining the peak shear performance of marine soft clay sediments that surpass that offered by traditional machine learning models. Additionally, a sensitivity analysis conducted with this innovative model highlights the notable impact of factors such as normal stress, initial soil density, the number of drying–wetting cycles, and average soil particle size on the peak shear strength of this type of sediment, while the impact of initial soil moisture content and temperature is comparatively minor. Finally, an analytical formula derived from the novel algorithm allows for precise estimation of the peak shear strength of marine soft clay sediment, catering to individuals lacking a background in machine learning.