Transformers_And_LLM_Are_What_You_Dont_Need

The best repository showing why transformers don’t work in time series forecasting

Table of Contents

• PhD and MSc Theses

• Videos

• Papers

• Articles

Videos

1. Problems in the current research on forecasting with transformers, foundational models, etc. by Christof Bergmeir

Theses

1. Cotton Price Long-Term Time Series Forecasting: A look at Transformers Suitability

Papers

1. Are Transformers Effective for Time Series Forecasting? by Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu (The Chinese University of Hong Kong, International Digital Economy Academy (IDEA), 2022) code 🔥🔥🔥🔥🔥
2. LLMs and foundational models for time series forecasting: They are not (yet) as good as you may hope by Christoph Bergmeir (2023) 🔥🔥🔥🔥🔥
3. Transformers Are What You Do Not Need by Valeriy Manokhin (2023) 🔥🔥🔥🔥🔥
4. Deep Learning is What You Do Not Need by Valeriy Manokhin (2022) 🔥🔥🔥🔥🔥
5. Why do Transformers suck at Time Series Forecasting by Devansh (2023)
6. Frequency-domain MLPs are More Effective Learners in Time Series Forecasting by Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Defu Lian, Ning An, Longbing Cao, Zhendong Niu (Bejing Institute of Technology, Tongji University, University of Oxford, Universuty of Technology Sydney, University of Macau, HeFei University of Technology, Macquarie University) (2023) 🔥🔥🔥🔥🔥
7. Forecasting CPI inflation under economic policy and geo-political uncertainties by Shovon Sengupta, Tanujit Chakraborty, Sunny Kumar Singh (Fidelity Investments, Sorbonne University, BITS Pilani Hyderabad). (2024) 🔥🔥🔥🔥🔥
8. Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping by Zhe Li, Shiyi Qi, Yiduo Li, Zenglin Xu (Harbin Institute of Technology, Shenzhen, 2023) code
9. SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction by Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, Qiang Xu (The Chinese University of Hong Kong,2022) code
10. WINNET:TIME SERIES FORECASTING WITH A WINDOW-ENHANCED PERIOD EXTRACTING AND INTERACTING by Wenjie Ou, Dongyue Guo, Zheng Zhang, Zhishuo Zhao, Yi Lin (Sichuan University, China, 2023)
11. A Multi-Scale Decomposition MLP-Mixer for Time Series Analysis by Shuhan Zhong, Sizhe Song, Guanyao Li, Weipeng Zhuo, Yang Liu, S.-H. Gary Chan, The Hong Kong University of Science and Technology Hong Kong, 2023) code 🔥🔥🔥🔥🔥
12. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis by (Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, Mingsheng Longj, , Tsinghua University, 2023) code 🔥🔥🔥🔥🔥
13. MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal and Channel Mixing code 🔥🔥🔥🔥🔥
14. Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift by Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, Jaegul Choo (Kaist AI, Vuno, Naver Corp, ETRI, ICLR 2022) code project page 🔥🔥🔥🔥🔥
15. WINNet: Wavelet-inspired Invertible Network for Image Denoising by Wenjie Ou, Dongyue Guo, Zheng Zhang, Zhishuo Zhao, Yi Lin (College of Computer Science, Sichuan University, China) code 🔥🔥🔥🔥🔥
16. Mlinear: Rethink the Linear Model for Time-series Forecasting Wei Li, Xiangxu Meng, Chuhao Chen and Jianing Chen (Harbin Engineering University, 2023) 🔥🔥🔥🔥🔥
17. Minimalist Traffic Prediction: Linear Layer Is All You Need by Wenying Duan, Hong Rao, Wei Huang, Xiaoxi He (Nanchang, University, Universify of Macau, 2023)
18. Frequency-domain MLPs are More Effective Learners in Time Series Forecasting by Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Defu Lian, Ning An, Longbing Cao, Zhendong Niu (Beijing Institute of Technology, Tongji University, University of Oxford University of Technology Sydney, University of Macau, USTC, HeFei University of Technology, Macquarie University, 2023) code 🔥🔥🔥🔥🔥
19. AN END-TO-END TIME SERIES MODEL FOR SIMULTANEOUS IMPUTATION AND FORECAST by Trang H. Tran, Lam M. Nguyen, Kyongmin Yeo, Nam Nguyen, Dzung Phan, Roman Vaculin Jayant Kalagnanam (School of Operations Research and Information Engineering, Cornell University; IBM Research, Thomas J. Watson Research Center, Yorktown Heights, NY, USA, 2023) 🔥🔥🔥🔥🔥
20. Long-term Forecasting with TiDE: Time-series Dense Encoder by Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, Rose Yu (Google Cloud, University of California, San Diego, 2023)
21. TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting by Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, Jayant Kalagnanam (IBM Research, 2023) code code
22. Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors by Yong Liu, Chenyu Li, Jianmin Wang, Mingsheng Long (Tsinghua University, 2023) code 🔥🔥🔥🔥🔥
23. Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective Jiaxi Hu, Yuehong Hu, Wei Chen, Ming Jin, Shirui Pan, Qingsong Wen, Yuxuan Liang (2024) 🔥🔥🔥🔥🔥
24. When and How: Learning Identifiable Latent States for Nonstationary Time Series Forecasting (2024) 🔥🔥🔥🔥🔥
25. Deep Coupling Network For Multivariate Time Series Forecasting (2024)
26. Linear Dynamics-embedded Neural Network for Long-Sequence Modeling by Tongyi Liang and Han-Xiong Li (City University of Hong Kong, 2024).
27. PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the perspective of partial differential equations (2024)
28. CATS: Enhancing Multivariate Time Series Forecasting by Constructing Auxiliary Time Series as Exogenous Variables (2024) 🔥🔥🔥🔥🔥
29. Is Mamba Effective for Time Series Forecasting? code (2024) 🔥🔥🔥🔥🔥
30. STG-Mamba: Spatial-Temporal Graph Learning via Selective State Space Model (2024)
31. TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting code (2024)🔥🔥🔥🔥🔥
32. FITS: Modeling Time Series with 10k Parameters code (2023)
33. TSLANet: Rethinking Transformers for Time Series Representation Learning code (2024) 🔥🔥🔥🔥🔥
34. WFTNet: Exploiting Global and Local Periodicity in Long-term Time Series Forecasting code (2024) 🔥🔥🔥🔥🔥
35. SiMBA: Simplified Mamba-based Architecture for Vision and Multivariate Time series code (2024) 🔥🔥🔥🔥🔥
36. SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion code (2024) 🔥🔥🔥🔥🔥
37. Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting code (2024) 🔥🔥🔥🔥🔥
38. SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters (2024) 🔥🔥🔥🔥🔥
39. Boosting MLPs with a Coarsening Strategy for Long-Term Time Series Forecasting (2024) 🔥🔥🔥🔥🔥
40. Multi-Scale Dilated Convolution Network for Long-Term Time Series Forecasting (2024)
41. ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis code (ICLR 2024 Spotlight)
42. Adaptive Extraction Network for Multivariate Long Sequence Time-Series Forecasting (2024) 🔥🔥🔥🔥🔥
43. Interpretable Multivariate Time Series Forecasting Using Neural Fourier Transform (2024) 🔥🔥🔥🔥🔥
44. PERIODICITY DECOUPLING FRAMEWORK FOR LONG- TERM SERIES FORECASTING code (2024) 🔥🔥🔥🔥🔥
45. Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models 🔥🔥🔥🔥🔥 (2024)
46. Time Evidence Fusion Network: Multi-source View in Long-Term Time Series Forecasting code (2024)
47. ATFNet: Adaptive Time-Frequency Ensembled Network for Long-term Time Series Forecasting code (2024) 🔥🔥🔥🔥
48. C-Mamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting (2024) 🔥🔥🔥🔥
49. The Power of Minimalism in Long Sequence Time-series Forecasting
50. WindowMixer: Intra-Window and Inter-Window Modeling for Time Series Forecasting
51. xLSTMTime : Long-term Time Series Forecasting With xLSTM code (2024)
52. Not All Frequencies Are Created Equal:Towards a Dynamic Fusion of Frequencies in Time-Series Forecasting (2024) 🔥🔥🔥🔥
53. FMamba: Mamba based on Fast-attention for Multivariate Time-series Forecasting (2024)
54. Long Input Sequence Network for Long Time Series Forecasting (2024)
55. Time-series Forecasting with Tri-Decomposition Linear-based Modelling and Series-wise Metrics (2024) 🔥🔥🔥🔥
56. An Evaluation of Standard Statistical Models and LLMs on Time Series Forecasting (2024) LLM 🔥🔥🔥🔥
57. Macroeconomic Forecasting with Large Language Models (2024) LLM 🔥🔥🔥🔥
58. Language Models Still Struggle to Zero-shot Reason about Time Series (2024) LLM 🔥🔥🔥🔥
59. KAN4TSF: Are KAN and KAN-based models Effective for Time Series Forecasting? (2024)
60. Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting (2024)
61. Transformers are Expressive, But Are They Expressive Enough for Regression? (2024) paper showing transformers cant approximate smooth functions
62. MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with 0.1K Parameters
63. MMFNet: Multi-Scale Frequency Masking Neural Network for Multivariate Time Series Forecasting
64. Neural Fourier Modelling: A Highly Compact Approach to Time-Series Analysis code
65. CMMamba: channel mixing Mamba for time series forecasting
66. EffiCANet: Efficient Time Series Forecasting with Convolutional Attention
67. Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
68. CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns code
69. Are Language Models Actually Useful for Time Series Forecasting?
70. SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion code
71. FTLinear: MLP based on Fourier Transform for Multivariate Time-series Forecasting
72. WPMixer: Efficient Multi-Resolution Mixing for Long-Term Time Series Forecasting code
73. Zero Shot Time Series Forecasting Using Kolmogorov Arnold Networks

Articles

1. [TimeGPT vs TiDE: Is Zero-Shot Inference the Future of Forecasting or Just Hype?](https://arxiv.org/abs/2205.13504 by Luís Roque and Rafael Guedes. (2024)🔥🔥🔥🔥🔥
2. TimeGPT-1, discussion on Hacker News (2023) 🔥🔥🔥🔥🔥
3. TimeGPT : The first Generative Pretrained Transformer for Time-Series Forecasting