VideoTime³: A 40-uJ/frame 38 FPS Video Understanding Accelerator With Real-Time DiffFrame Temporal Redundancy Reduction and Temporal Modeling

VideoTime(3) is an accelerator for state-of-the-art video understanding with deep learning on the edge. Different from prior work, it highlights a real-time DiffFrame convolution achieving $2.2\times $ DRAM access reduction compared to conventional convolution, a sorter-free architecture for efficient sparse output stationary dataflow, temporal modeling capability achieving high accuracy on video understanding applications, and optimized data buffering to remove DRAM traffic overhead for temporal modeling and reduce 55%-79% input activation DRAM traffic in depth-wise layers. The chip consumes 40 uJ/frame with 38 frames/s at 0.6 V in 28-nm CMOS.