Impact of dimensionality reduction on machine learning-based crop health prediction

Abstract

Abstract Accurately predicting fungal disease dynamics poses a major challenge for data-driven sustainable agriculture. This study evaluates the impact of dimensionality reduction (DR) on the predictive performance of classical machine learning (ML) and deep learning (DL) models for classification (fungal disease occurrence) and regression (fungal disease severity) tasks using multidimensional meteorological and phytosanitary data collected from carrot, lettuce, and onion farms in southern Quebec (Canada). We systematically benchmark traditional ML models - Decision Tree (DT), Random Forest (RF), and k-Nearest Neighbours (k-NN), a basic non-temporal deep learner - Multilayer Perceptron (MLP), recurrent DL models - Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), as well as state-of-the-art Mamba state-space model (SSM) and pretrained tabular foundation model TabPFN, each of them combined with linear (PCA and MDS), nonlinear (KernelPCA and Isomap), and domain-informed (BOTCAST) DR methods. TabPFN provides the best overall performance in classification and RF in regression, while LSTM and GRU networks rank among top-performing models for both tasks. In classification, DR methods combine well with TabPFN and the recurrent approaches, with the BOTCAST + TabPFN, PCA + TabPFN, and Isomap + GRU pairings, yielding the best overall classification results for carrot, lettuce, and onion data, respectively. In regression, nonlinear DR methods perform best when coupled with ensemble learners (i.e. the best overall results were obtained by the KernelPCA + RF pairing for both lettuce and onion data), while the domain-informed BOTCAST embedding yields excellent results when combined with TabPFN (i.e. the best overall result was obtained by the BOTCAST + TabPFN pairing for carrot data). Overall, the success of predictions largely depends on a good alignment of the selected dimensionality reduction strategy with the downstream machine learning or deep learning model. Our results provide practical guidance for decision-support systems aiming at forecasting fungal disease occurrence and severity in agricultural plants.