A 25.1-TOPS/W Sparsity-Aware Hybrid CNN-GCN Deep Learning SoC for Mobile Augmented Reality

Augmented reality (AR) has been applied to various mobile applications. Modern AR algorithms include neural networks, such as convolutional neural networks (CNNs) and graph convolutional networks (GCNs). The high computational complexity of these networks poses challenges for real-time operation on energy-constrained devices. This article presents the first energy-efficient hybrid CNN-GCN system-on-chip (SoC) for mobile AR. A CNN engine exploits the channel-wise structured feature sparsity to eliminate redundant computations and data movements. By utilizing the proposed channel-sparse encoding scheme on a specialized processing element (PE) architecture, up to 8 $\times$ higher throughput and 6.1 $\times$ higher energy efficiency can be achieved. A reconfigurable convolution PE (CPE) array is deployed for efficient CNN inference. A GCN engine is designed to implement skeleton-based action recognition and gesture recognition. Up to 71% of total operations and 39% of memory footprint can be reduced by leveraging the data and graph properties. A RISC-V MCU is integrated for system control and network deployment. The proposed SoC is implemented in a 28-nm CMOS technology with a core area of 8.28 mm $^2$ . By exploiting various sparsity levels across network layers, the chip achieves an up to 3.277-TOPS peak performance and a 25.1-TOPS/W energy efficiency for sparse CNN inference, 2.0 $\times$ higher energy-efficient than prior arts. It also achieves an up to 72 actions/s recognition throughput, 18 $\times$ faster than the state of the art.