Exploiting Sequential Low-Rank Factorization For Multilingual Dnns

DNNs have shown remarkable performance in multilingual scenarios; however, these models are often too large in size that adaptation to a target language with relatively small amount of data cannot be well accomplished. In our previous work, we utilized Low-Rank Factorization (LRF) using singular value decomposition for multilingual DNNs to learn compact models which can be adapted more successfully. In this paper, we address two problems associated with that LRF scheme and we propose a compellingly simple methodology to overcome them. First, factorizing all layers results in a huge drop in performance and consequently a long recovery process is required which is not practically efficient. Secondly, LRF can be viewed as a regularization by which some noise is added to weight layers; however, factorizing all layers together equates to adding too much noise which results in bad performance. To mitigate these problems, we propose to apply LRF sequentially. We demonstrate that the lost information after factorizing one layer is small and can be rapidly retrieved; hence, sequential factorization is more efficient. Moreover, the sequential LRF adds only a small amount of noise sequentially which is a better regularization. Our experiments