Using Time Series Clustering to Inform Multimodal CNN Architectures

Joshua Sylvester, Matthew Lee, Daniel Canon Ellis, Mitchell A. Thornton, Eric C. Larson

Abstract: Multimodal machine learning, in the context of deep learning, allows a neural network to process various sources of data and combine information from each data source. However, there are an exponential number of ways in which modalities can be combined for processing which can result in large architecture design searches to inform the most optimal manner of combining data streams. To mitigate this problem, we present a way to inform the creation of multimodal machine learning convolutional neural network architectures in the domain of time series datasets. Specifically, we propose the use of time series clustering as a method for informing the creation of a model's multimodal architecture. We investigate two different approaches to this method (a Euclidean-and Granger-based approach) and demonstrate effectiveness with multiple time series datasets. We find that our proposed methods can improve a model's predictive capabilities while decreasing the training time required for the model to converge. Moreover, our method eliminates the need for a costly architecture search.