Evaluating Predictive Deep Learning Models

Patrick Ribu Gorton, Kai Olav Ellefsen

Abstract: Predicting the future using deep learning models is a research field of increasing interest. However, there is a lack of established evaluation methods for assessing their predictive abilities. Images and videos are targeted towards human observers, and since humans have individual perceptions of the world, evaluation of videos should take subjectivity into account. In this paper, we present a framework for evaluating predictive models using subjective data. The methodology is based on a mixed methods research design, and is applied in an experiment to measure the realism and accuracy of predictions of a visual traffic environment. Our method is shown to be uncorrelated with the predominant approach for evaluating predictive models, which is a frame-wise comparison between predictions and ground truth. These findings emphasise the importance of using subjective data in the assessment of predictive abilities of models and open up a new direction for evaluating predictive deep learning models.

External IDs:doi:10.1007/978-3-030-71711-7_12