Journal of Systems Engineering and Electronics ›› 2024, Vol. 35 ›› Issue (3): 509-531.doi: 10.23919/JSEE.2023.000159


Low rank optimization for efficient deep learning: making a balance between compact architecture and fast training

Xinwei OU(), Zhangxin CHEN(), Ce ZHU(), Yipeng LIU()   

  • Received:2022-09-21 Accepted:2023-07-21 Online:2024-06-18 Published:2024-06-19
  • Contact: Zhangxin CHEN, Yipeng LIU;;;
  • About author:
    OU Xinwei was born in 2000. She received her B.S. degree in electronic information engineering from Xidian University, Xi’an, China, in 2022. She is working towards her M.S. degree with University of Electronic Science and Technology of China, Chengdu, China. Her research interests include tensors for efficient deep learning. E-mail:

    CHEN Zhangxin was born in 1978. He received his M.S. degrees and Ph.D. degrees from University of Electronic Science and Technology of China, both in communication and information system, in 2003 and 2009, respectively. From 2012, he has been an associate professor at the Department of Electronic Engineering, University of Electronic Science and Technology of China. His research interests focus on signal processing in distributed radar system and airborne radar system. E-mail:

    ZHU Ce was born in 1961. He received his B.S. degree in communication engineering from Sichuan University, Chengdu, China, in 1989, and M.E. and Ph.D. degrees from Southeast University, Nanjing, China, in 1992 and 1994, respectively, all in electronic and information engineering. He has been with the University of Electronic Science and Technology of China, Chengdu, China, as a professor since 2012. His research interests include video coding and communications, video analysis and processing, three-dimensional video, and visual perception and applications. E-mail:

    LIU Yipeng was born in 1983. He received his B.S. degree in biomedical engineering and Ph.D. degree in information and communication engineering from the University of Electronic Science and Technology of China (UESTC), Chengdu, China, in 2006 and 2011, respectively. Since 2014, he has been an associate professor with UESTC, Chengdu, China. His research interest is tensor for data processing.E-mail:
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (62171088;U19A2052;62020106011), and the Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China (ZYGX2021YGLH215;ZYGX2022YGRH005).


Deep neural networks (DNNs) have achieved great success in many data processing applications. However, high computational complexity and storage cost make deep learning difficult to be used on resource-constrained devices, and it is not environmental-friendly with much power cost. In this paper, we focus on low-rank optimization for efficient deep learning techniques. In the space domain, DNNs are compressed by low rank approximation of the network parameters, which directly reduces the storage requirement with a smaller number of network parameters. In the time domain, the network parameters can be trained in a few subspaces, which enables efficient training for fast convergence. The model compression in the spatial domain is summarized into three categories as pre-train, pre-set, and compression-aware methods, respectively. With a series of integrable techniques discussed, such as sparse pruning, quantization, and entropy coding, we can ensemble them in an integration framework with lower computational complexity and storage. In addition to summary of recent technical advances, we have two findings for motivating future works. One is that the effective rank, derived from the Shannon entropy of the normalized singular values, outperforms other conventional sparse measures such as the $ \ell_1 $ norm for network compression. The other is a spatial and temporal balance for tensorized neural networks. For accelerating the training of tensorized neural networks, it is crucial to leverage redundancy for both model compression and subspace training.

Key words: model compression, subspace training, effective rank, low rank tensor optimization, efficient deep learning