Sampling Training Data for Continual Learning Between Robots and the Cloud

Sandeep Chinchali, Evgenya Pergament, Manabu Nakanoya, Eyal Cidon, Edward Zhang, Dinesh Bharadia, Marco Pavone, Sachin Katti

Abstract: Today’s robotic fleets are increasingly measuring high-volume video and LIDAR sensory streams, which can be mined for valuable training data to continually improve robotic perception models. However, re-training perception models on growing volumes of rich sensory data in central servers (or the “cloud”) incurs significant systems costs for network transfer, cloud storage, human annotation, and cloud computing resources. Hence, we introduce HarvestNet, an intelligent sampling algorithm that resides on-board a robot and reduces system bottlenecks by only storing rare, useful events to continually improve and re-train perception models. HarvestNet significantly improves the accuracy of perception models on our novel dataset of road construction sites, field testing of self-driving cars, and face recognition, while reducing cloud storage, dataset annotation time, and cloud compute time by between 65.7−81.3 $$\%$$ . Further, it is between 1.05−2.58 $$\times $$ more accurate than baseline algorithms and runs on low-power deep learning hardware.