Fosmidomycin (FOS) is a naturally occurring compound active against the enzyme DXR in the MEP pathway, and using it as a template for lead structure design is an effective strategy in developing new active compounds. In this work, by replacing the hydroxamate unit of FOS with pyrazole, isoxazole and the related heterocycles that also have the metal ion binding affinity, while retaining the monophosphonic acid in FOS or replacing it with a bisphosphonic acid group, heterocycle-containing mono- and bisphosphonic acid compounds as FOS analogs were designed. The key steps involved in the facile synthesis of these FOS analogs include the Michael addition of diethyl vinylphosphonate or tetraethyl vinylidenebisphosphonate to β-dicarbonyl compounds and the subsequent cyclic condensation with hydrazine or hydroxylamine. Two additional isoxazolinone-bearing FOS analogs were synthesized by the Michaelis-Becker reaction with diethyl phosphite as a key step. The bioactivity evaluation on model plants demonstrated that several compounds have better herbicidal activities compared to FOS, with the most active compound showing a 3.7-fold inhibitory activity on Arabidopsis thaliana, while on the root and stalk of Brassica napus L. and Echinochloa crus-galli in pre-emergence inhibitory activity test, the activities of this compound were found to be 3.2- and 14.3-, 5.4- and 9.4-fold, respectively, and in post-emergency activity test on Amaranthus retroflexus and Echinochloa crus-galli, 2.2- and 2.0-fold inhibition activities were displayed. Despite the significant herbicidal activity, this compound exhibited a DXR inhibitory activity lower than that of FOS but comparable to that of other non-hydroxamate DXR inhibitors, and the dimethylallyl pyrophosphate rescue assay gave no statistical significance, suggesting a different target might be involved in the inhibiting process. This work demonstrates that using bioisosteric replacement can be considered a valuable strategy for discovering new FOS analogs that may have high herbicidal activity.