A Spatio-Temporal Convolutional Neural Network Model for Rice Nutrient Level Diagnosis at Rice Panicle Initiation Stage

Nitrogen (N) and potassium (K) are the key mineral nutrient elements during rice growth. Accurate diagnosis of N and K status is very important for the rational fertilizing application at a specific rice growth stage. The rice canopy images were collected by an unmanned aerial vehicle (UAV) carrying a RGB camera at different growth states of two rice varieties, Huanghuazhan (HHZ, an indica cultivar) and Xiushui134 (XS134, a japonica cultivar) in 2021. We proposed a spatio-temporal convolutional network model based on a convolutional neural network (CNN) embedded with the attention mechanism and a long short-term memory network (LSTM) to diagnose the rice nutrient level at the early-panicle initiation stage (EPIS) after the rice plants were treated by 16 combination fertilizers of four different N and K dosages. Different pretrained neural networks were used to extract features from time-series rice canopy images to classify sixteen rice nutrient levels. Compared with VGG16, AlexNet, GoogleNet, DenseNet, and inceptionV3, ResNet101 combined with LSTM obtained the highest average accuracy of 83.81% on the dataset of HHZ. Compared with the traditional CNN model trained from single-time images, the ResNet101-SE-LSTM based on the ResNet101-LSTM model enhanced with Squeeze-and-Excitation (SE) block had great improvement on the accuracy of nutrient level diagnosis. It obtained a promising generalization evaluated by a cross-dataset validation, with the average accuracy of 85.88% and 88.38% on the test datasets of HHZ and XS134, respectively. Our proposed model involves with the dynamic information of different rice growth stages and can efficiently diagnose different rice nutrient status at EPIS, which are helpful for making a practical decision for the rational fertilization at the panicle initiation stage.