A Hybrid Approach for Soil Total Nitrogen Anomaly Detection Integrating Machine Learning and Spatial Statistics

Abstract

Soil total nitrogen is one of the most important basic indicators for fertiliser decision making, but tens of millions of soil total nitrogen sampling data have been accumulated, forming a huge database. In this large database, there is a large amount of anomalous data, which can interfere with data analysis, affect the construction of spatial interpolation and prediction models, and then affect the accuracy of nutrient management decisions. The traditional method of identifying soil total nitrogen anomalies based on boxplots suffers from the problems of not being able to identify local anomalies, which can easily lead to misclassification of soil total nitrogen data anomalies, and the detection efficiency is not high. We propose a method to identify soil total nitrogen outliers by combining the Isolation Forest algorithm and local spatial autocorrelation analysis, which can simultaneously detect global and local outliers from large amounts of data and combine organic matter as an auxiliary indicator in the spatial analysis to help judge local outliers. Finally, the results of global and local anomalies were combined to provide a comprehensive assessment of the soil nitrogen data, avoiding the misjudgement or omission of judgement that can occur when using a single method. Using 25,930 soil test data from Yunnan Province in 2009 as an example, we compared and analysed the typical boxplot method and the unsupervised OneClassSVM method and evaluated the performance of each method in terms of correct detection rate, false positive rate and false negative rate. The results show that the proposed method has a correct detection rate (TR) of 99.97%, a false positive rate (FPR) of 8.06% and a false negative rate (FNR) of 0.01% on the data, which shows high validity and accuracy; it is also comparable to the independent isolated forests (FNR = 4.76%), boxplot (FNR = 3.90%) and OneClassSVM (FNR = 4.77%), and the false negative rate is reduced by 4.75%, 3.89% and 4.76%, respectively.

Rainfall Runoff and Nitrogen Loss Characteristics on the Miyun Reservoir Slope

“…Furthermore, the soils have weak fertilizer and water sorption capacity, which makes them particularly prone to soil erosion and nutrient loss, as well as eutrophication of the surrounding water bodies, which poses significant ecological risks. Soil nitrogen is the most important indicator of soil nutrient levels [5] and also the main factor causing eutrophication/pollution of water bodies [6]. However, there have been few studies on soil nitrogen loss from slopes in the Miyun Reservoir area, especially the temporal and spatial effects of surface and subsurface runoff on soil nitrogen loss processes and mechanisms.…”