Disentangling data dependency using cross-validation strategies to evaluate prediction quality of cattle grazing activities using machine learning algorithms and wearable sensor data

Abstract

Wearable sensors have been explored as an alternative for real-time monitoring of cattle feeding behavior in grazing systems. To evaluate the performance of predictive models such as machine learning (ML) techniques, data cross-validation (CV) approaches are often employed. However, due to data dependencies and confounding effects, poorly performed validation strategies may significantly inflate the prediction quality. In this context, our objective was to evaluate the effect of different CV strategies on the prediction of grazing activities in cattle using wearable sensor (accelerometer) data and ML algorithms. Six Nellore bulls (average live weight of 345 ± 21 kg) had their behavior visually classified as grazing or not-grazing for a period of 15 days. Elastic Net Generalized Linear Model (GLM), Random Forest (RF), and Artificial Neural Network (ANN) were employed to predict grazing activity (grazing or not-grazing) using 3-axis accelerometer data. For each analytical method, three CV strategies were evaluated: holdout, leave-one-animal-out (LOAO), and leave-one-day-out (LODO). Algorithms were trained using similar dataset sizes (holdout: n = 57,862; LOAO: n = 56,786; LODO: n = 56,672). Overall, GLM delivered the worst prediction accuracy (53%) compared to the ML techniques (65% for both RF and ANN), and ANN performed slightly better than RF for LOAO (73%) and LODO (64%) across CV strategies. The holdout yielded the highest nominal accuracy values for all three ML approaches (GLM: 59%, RF: 76%, and ANN: 74%), followed by LODO (GLM: 49%, RF: 61%, and ANN: 63%) and LOAO (GLM: 52%, RF: 57%, and ANN: 57%). With a larger dataset (i.e., more animals and grazing management scenarios), it is expected that accuracy could be increased. Most importantly, the greater prediction accuracy observed for holdout CV may simply indicate a lack of data independence and the presence of carry-over effects from animals and grazing management. Our results suggest that generalizing predictive models to unknown (not used for training) animals or grazing management may incur poor prediction quality. The results highlight the need for using management knowledge to define the validation strategy that is closer to the real-life situation, i.e., the intended application of the predictive model.

The Combined Use of Automated Milking System and Sensor Data to Improve Detection of Mild Lameness in Dairy Cattle

“…In the first approach, an animal-based split [ 62 ] was chosen, meaning that all observations on an individual cow were either placed within the training dataset or test dataset. The dataset was randomly split by animal into a training dataset containing 80% of the records and a test dataset consisting of the remaining 20%.…”

Predicting dry matter intake in Canadian Holstein dairy cattle using milk mid-infrared reflectance spectroscopy and other commonly available predictors via artificial neural networks

“…Therefore, validation B assessed how the model would perform when predicting weekly average DMI for new cows. Data dependencies between calibration and test sets, which were reported when animals from the same herds are included in both calibration and test data sets, may inflate prediction performance (e.g., Wang and Bovenhuis, 2019;Coelho Ribeiro et al, 2021). However, this is not always the case, as in study by Lahart et al (2019), where the average accuracy of DMI prediction in test sets using only MIRS and MIRS combined with MY, F%, P%, BW, stage of lactation and parity, using within herd and across-herd crossvalidations, were 0.69, 0.87, 0.55, and 0.80, respectively.…”

Research on Students’ Mental Health Based on Data Mining Algorithms

“…And, the significance test based on p value faces a crisis of duplication, that is, the irreproducibility of research results. Strict and systematic use of machine learning cross-validation technology [ 44 ] can provide great potential for realizing the reproducibility of psychological research. Technology-based machine learning can construct learning models from massive amounts of data, more accurately identify the underlying laws of the data, and have stronger generalization capabilities.…”