An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancement
CoRR(2024)
摘要
Transformer architecture has enabled recent progress in speech enhancement.
Since Transformers are position-agostic, positional encoding is the de facto
standard component used to enable Transformers to distinguish the order of
elements in a sequence. However, it remains unclear how positional encoding
exactly impacts speech enhancement based on Transformer architectures. In this
paper, we perform a comprehensive empirical study evaluating five positional
encoding methods, i.e., Sinusoidal and learned absolute position embedding
(APE), T5-RPE, KERPLE, as well as the Transformer without positional encoding
(No-Pos), across both causal and noncausal configurations. We conduct extensive
speech enhancement experiments, involving spectral mapping and masking methods.
Our findings establish that positional encoding is not quite helpful for the
models in a causal configuration, which indicates that causal attention may
implicitly incorporate position information. In a noncausal configuration, the
models significantly benefit from the use of positional encoding. In addition,
we find that among the four position embeddings, relative position embeddings
outperform APEs.
更多查看译文
关键词
speech enhancement,Transformer,position encoding
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要