Enhancing Deep Learning Models through Tensorization: A Comprehensive Survey and Framework

Abstract

In modern machine learning, data are often simplified to 2-dimensional matrices for ease of application in linear algebra-based algorithms despite being inherently high-dimensional. However, applying multiway analysis through multilinear algebra to these multidimensional datasets provides more expressive models, reduces parameters, and accelerates processing, defying the expected dimensionality curse. This paper surveys the theoretical background necessary for understanding these methods, outlines the process of tensorizing matrix-form datasets, and reviews current methods and applications of multiway analysis in compressing deep learning models. It includes a framework for tensorization, a case study on Blind Source Separation, and a comprehensive review of tensorized machine learning and deep learning applications, concluding with key insights and future research directions.