One of the most commonly occurring bacteria, Bacillus subtilis, can produce a wide variety of secondary metabolites. In this study, the antimicrobial effect of B. subtilis KSRLAB3 against Vibrio alginolyticus was optimized using the Plackett–Burman design (PBD) method, response surface methodology (RSM), and genetic algorithm (GA). Initially, the effects of carbon source, nitrogen source, NaCl concentration, pH, temperature, and incubation time on antimicrobial effects were studied. Among the carbon and nitrogen sources investigated, mannose and peptone elicited maximum antimicrobial effect. Using PBD, the most significant variables that influence the antimicrobial effect were identified, including incubation time, peptone concentration, and temperature. The optimum conditions required for attaining maximum antimicrobial effect was identified using the RSM‐GA hybrid method, and the optimum condition includes 49.999 h of incubation time, 4.39 g/L of peptone concentration, and 27.629°C of incubation temperature. The confirmatory experiments performed around the optimum condition showed a zone of inhibition of 35 ± 0.52 mm. Methanolic extract also proved the presence of antibacterial lipopeptide surfactin. Therefore, the RSM‐GA hybrid method was successfully used in this study to model the antimicrobial effect of B. subtilis KSRLAB3 against V. alginolyticus. The effective inhibition of V. alginolyticus can be investigated further for the development of antifouling coatings.