iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
arxiv(2023)
摘要
The recent boom of linear forecasting models questions the ongoing passion
for architectural modifications of Transformer-based forecasters. These
forecasters leverage Transformers to model the global dependencies over
temporal tokens of time series, with each token formed by multiple variates of
the same timestamp. However, Transformers are challenged in forecasting series
with larger lookback windows due to performance degradation and computation
explosion. Besides, the embedding for each temporal token fuses multiple
variates that represent potential delayed events and distinct physical
measurements, which may fail in learning variate-centric representations and
result in meaningless attention maps. In this work, we reflect on the competent
duties of Transformer components and repurpose the Transformer architecture
without any modification to the basic components. We propose iTransformer that
simply applies the attention and feed-forward network on the inverted
dimensions. Specifically, the time points of individual series are embedded
into variate tokens which are utilized by the attention mechanism to capture
multivariate correlations; meanwhile, the feed-forward network is applied for
each variate token to learn nonlinear representations. The iTransformer model
achieves state-of-the-art on challenging real-world datasets, which further
empowers the Transformer family with promoted performance, generalization
ability across different variates, and better utilization of arbitrary lookback
windows, making it a nice alternative as the fundamental backbone of time
series forecasting. Code is available at this repository:
https://github.com/thuml/iTransformer.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要