High-Dimensional Tensor Regression With Oracle Properties

Wenbin Wang, Yu Shi, Ziping Zhao

Abstract: The emergence of multi-dimensional data presents significant challenges for traditional regression models based on matrices or vectors, particularly in capturing multi-directional correlations. In response, tensor regression has been proposed as a powerful framework for modeling linear relationships among multi-dimensional variables. In this paper, we introduce a high-dimensional tensor-response tensor regression model under low-dimensional structural assumptions, such as sparsity and low-rankness. Assuming the underlying tensor lies within an unknown low-dimensional subspace, we consider a least squares estimation framework with non-convex penalties. Theoretically, we derive general risk bounds for the resulting estimators and demonstrate that they achieve the oracle statistical rates under mild technical conditions. To compute the proposed estimators efficiently, we introduce an accelerated proximal gradient algorithm demonstrating rapid convergence in practice. Extensive experiments on synthetic and real-world datasets validate the effectiveness of the proposed regression model and showcase the practical utility of the theoretical findings.

Lay Summary:

Modern data often come in the form of tensors, which are multi-dimensional arrays extending beyond simple tables or lists. Traditional statistical methods, which work with simpler data formats like vectors or matrices, struggle to accurately capture the complex relationships across these multiple dimensions.

To overcome these limitations, our research introduces a new regression method specifically designed for tensor data. This method efficiently identifies underlying simple patterns, like sparsity (where many elements are zero or irrelevant) or low-rank structures (where the data can be simplified into fewer meaningful dimensions). By assuming such underlying simplicity, our method effectively estimates the relationships between multiple high-dimensional datasets.

We provide a rigorous theoretical analysis showing that our approach accurately recovers these relationships, even when the datasets are large and complicated. To make our method practical, we also develop a fast and effective algorithm to solve the regression model efficiently.

Through comprehensive experiments using both artificial and real-world data, we demonstrate that our method works very well in practice, providing accurate predictions and meaningful insights. This approach helps researchers and practitioners analyze complex data easily, effectively, and with greater confidence.

Primary Area: General Machine Learning->Evaluation

Keywords: tensor regression, non-convex optimization, oracle properties, high-dimensional statistics, sparse, low rank

Submission Number: 6182