LeViS: A Vision Transformer for Fast Combinatorial Optimization of Imaging Techniques

Ryan Mei, Weiduo Zhu, Zaee Shah, Vidhi Lalchand, Mina Konakovic Lukovic

Keywords: channel selection, feature selection, experimental design, feature, channel, hyperspectral, multichannel, multi-channel, optimal sampling

Abstract: The performance of a vision model depends greatly on the types of data on which it is trained. For applications from classifying objects to predicting the weather, models can be trained on a variety of image sensors and sampling patterns. Vision systems often face practical constraints (e.g.\ acquisition time, compute resources, energy, and memory consumption) that limit the number of sensors and samples. In such cases, obtaining and training the model on a subset of available modalities is beneficial. Although many iterative combinatorial optimization algorithms can find optimal sets of modalities, they are drastically slowed by the need to train the model to evaluate each proposed set. We introduce LeViS (Learned Vision System), a vision transformer that proficiently classifies images collected with any sampling pattern or sensor set \textit{without retraining}. The predictive power of a set can of techniques can thus be quickly determined by evaluating the performance of LeViS on the set. We use LeViS with a variety of optimization algorithms (genetic algorithms, beam search, simulated annealing, sequential) to rapidly find optimal sets of satellite wavelength channels to classify local climate zones using So2Sat, and optimal pixel sampling patterns to classify handwritten digits and charaters using MNIST. Evaluating sets with LeViS instead of training new models enables optimization algorithms to find optimal sampling patterns and sensor sets up to 6800x faster.

Primary Area: optimization

Submission Number: 7786