Leveraging tensor methods in neural architecture search for the automatic development of lightweight convolutional neural networks

Most state-of-the-art Convolutional Neural Networks (CNNs) are bulky and cannot be deployed on resource-constrained edge devices. In order to leverage the exceptional generalizability of CNNs on edge-devices, they need to be made efficient in terms of memory usage, model size, and power consumption, while maintaining acceptable performance. Neural architecture search (NAS) is a recent approach for developing efficient, edge-deployable CNNs. On the other hand, CNNs used for classification, albeit developed using NAS, often contain large fully-connected (FC) layers with thousands of parameters, contributing to the bulkiness of CNNs. Recent works have shown that FC layers can be compressed, with minimal loss in performance, if any, using tensor processing methods. In this work, for the first time in literature, we leverage tensor methods in the NAS framework to discover efficient CNNs. Specifically, we employ tensor contraction layers (TCLs) to compress fully connected layers in the NAS framework and control the trade-off between compressibility and classification performance by handcrafting the ranks of TCLs. Additionally, we modify the NAS procedure to incorporate automatic TCL rank search in an end-to-end fashion, without human intervention. Our numerical studies on a wide variety of datasets including CIFAR-10, CIFAR-100, and Imagenette (a subset of ImageNet) demonstrate the superior performance of the proposed method in the automatic discovery of CNNs, whose model sizes are many-fold smaller than other cutting-edge mobile CNNs, while maintaining similar classification performance.