On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition
CoRR(2023)
摘要
Recently, the cascaded two-pass architecture has emerged as a strong
contender for on-device automatic speech recognition (ASR). A cascade of causal
and shallow non-causal encoders coupled with a shared decoder enables operation
in both streaming and look-ahead modes. In this paper, we propose shallow
cascaded model by combining various model compression techniques such as
knowledge distillation, shared decoder, and tied-and-reduced transducer network
in order to reduce the model footprint. The shared decoder is changed into a
tied-and-reduced network. The cascaded two-pass model is further compressed
using knowledge distillation using a Kullback-Leibler divergence loss on the
model posteriors. We demonstrate a 50% reduction in the size of a 41 M
parameter cascaded teacher model with no noticeable degradation in ASR accuracy
and a 30% reduction in latency
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要