Simultaneously Predicting Multiple Plant Traits from Multiple Sensors using Deformable CNN Regression

Pranav Vishnu Raja, Alex Olenskyj, Hamid Kamangir, Mason Earles

Keywords: multimodal data fusion, yield prediction, plant breeding, controlled environment agriculture, lettuce, multi-trait modeling

TL;DR: State-of-the-art prediction for lettuce traits via deformable CNN regression

Abstract: Trait measurement is a critical for the plant breeding and agricultural production pipeline. Typically, a suite of plant traits is measured using laborious manual measurements and then used to train and/or validate higher throughput trait estimation techniques. Here, we design a relatively simple convolutional neural network (CNN) model that accepts multiple sensor inputs and predicts multiple continuous trait outputs – i.e. a multi-input, multi-output CNN (MIMO-CNN). Further, we introduce deformable convolutional layers into this network architecture (MIMO-DCNN) to enable the model to adaptively adjust its receptive field, model complex variable geometric transformations in the data, and fine-tune the continuous trait outputs. We examine how the MIMO-CNN and MIMO-DCNN models perform on a multi-input (i.e. RGB + depth images), multi-trait output lettuce dataset from the 2021 Autonomous Greenhouse Challenge. Ablation studies were conducted to examine the effect of using single versus multiple inputs, and single versus multiple outputs. The MIMO-DCNN model resulted in a normalized mean squared error (NMSE) of 0.068; a substantial improvement over the top 2021 leaderboard score of 0.081. Open-source code is provided.