Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), a new coronavirus causing Coronavirus Disease 2019 (COVID-19), is a major topic of global human health concern. The Delta and Omicron variants have caused alarming responses worldwide due to their high transmission rates and a number of mutations. During a one-year follow-up (from June 2020 to June 2021), we included 114 patients with SARS-CoV-2 infection to study the long-term dynamics and the correlative factors of neutralizing antibodies (NAbs) in convalescent patients. The blood samples were collected at two detection time points (at 6 and 12 months after discharge). We evaluated the NAbs response of discharged patients by performing a micro-neutralization assay using a SARS-CoV-2 wild type. In addition, a total of 62 serum samples from discharged COVID-19 patients with Alpha, Beta, Delta, and Omicron variants of infection were enrolled to perform cross-neutralization tests using the original SARS-CoV-2 strain and VOCs variants (including Alpha, Beta, Gamma, Delta, and Omicron variants) and to assess the ability of NAbs against the SARS-CoV-2 variants. NAbs seroconversion occurred in 91.46% of patients (n = 82) in the first timepoint and in 89.29% of patients (n = 84) in the second detection point, and three kinds of NAbs kinetics curves were perceived. The NAbs levels in young patients had higher values than those in elder patients. The kinetics of disease duration was accompanied by an opposite trend in NAbs levels. Despite a declining NAbs response, NAbs activity was still detectable in a substantial proportion of recovered patients one year after discharge. Compared to the wild strain, the Omicron strain could lead to a 23.44-, 3.42-, 8.03-, and 2.57-fold reduction in neutralization capacity in “SAlpha”, “SBeta”, “SDelta”, and “SOmicron”, respectively, and the NAbs levels against the Omicron strain were significantly lower than those of the Beta and Delta variants. Remarkably, the NAbs activity of convalescent serum with Omicron strain infection was most obviously detectable against six SARS-CoV-2 strains in our study. The role of the vaccination history in NAbs levels further confirmed the previous study that reported vaccine-induced NAbs as the convincing protection mechanism against SARS-CoV-2. In conclusion, our findings highlighted the dynamics of the long-term immune responses after the disappearance of symptoms and revealed that NAbs levels varied among all types of convalescent patients with COVID-19 and that NAbs remained detectable for one year, which is reassuring in terms of protection against reinfection. Moreover, a moderate correlation between the duration of disease and Nabs titers was observed, whereas age was negatively correlated with Nabs titers. On the other hand, compared with other VOCs, the Omicron variant was able to escape the defenses of the immune system more significantly, and the convalescent serum infected with the Omicron variant played a critical part in protection against different SARS-CoV-2 variants. Recovery serum from individuals vaccinated with inactivated vaccine preceding infection with the Omicron strain had a high efficacy against the original strain and the VOCs variants, whereas the convalescent serum of persons vaccinated by inactivated vaccine prior to infection with the Delta variant was only potent against the wild-type strain.