Time series forecasting is vital to varied real-world purposes, from demand forecasting to pandemic unfold prediction. In multivariate time series forecasting (forecasting a number of variants on the identical time), one can break up current strategies into two classes: univariate fashions and multivariate fashions. Univariate fashions concentrate on inter-series interactions or temporal patterns that embody developments and seasonal patterns on a time series with a single variable. Examples of such developments and seasonal patterns is likely to be the way in which mortgage charges improve on account of inflation, and the way visitors peaks throughout rush hour. In addition to inter-series patterns, multivariate fashions course of intra-series options, often called cross-variate data, which is very helpful when one series is a complicated indicator of one other series. For instance, an increase in physique weight might trigger a rise in blood strain, and rising the value of a product might result in a lower in gross sales. Multivariate fashions have just lately change into fashionable options for multivariate forecasting as practitioners consider their functionality of dealing with cross-variate data might result in higher efficiency.
In latest years, deep studying Transformer-based architectures have change into a well-liked alternative for multivariate forecasting fashions on account of their superior efficiency on sequence duties. However, superior multivariate fashions carry out surprisingly worse than easy univariate linear fashions on commonly-used long-term forecasting benchmarks, resembling Electricity Transformer Temperature (ETT), Electricity, Traffic, and Weather. These outcomes increase two questions:
- Does cross-variate data profit time series forecasting?
- When cross-variate data is just not useful, can multivariate fashions nonetheless carry out in addition to univariate fashions?
In “TSMixer: An All-MLP Architecture for Time Series Forecasting”, we analyze some great benefits of univariate linear fashions and reveal their effectiveness. Insights from this evaluation lead us to develop Time-Series Mixer (TSMixer), a complicated multivariate mannequin that leverages linear mannequin traits and performs nicely on long-term forecasting benchmarks. To the very best of our data, TSMixer is the primary multivariate mannequin that performs in addition to state-of-the-art univariate fashions on long-term forecasting benchmarks, the place we present that cross-variate data is much less useful. To display the significance of cross-variate data, we consider a tougher real-world software, M5. Finally, empirical outcomes present that TSMixer outperforms state-of-the-art fashions, resembling PatchTST, Fedformer, Autoformer, DeepAR and TFT.
TSMixer architecture
A key distinction between linear fashions and Transformers is how they seize temporal patterns. On one hand, linear fashions apply fastened and time-step-dependent weights to seize static temporal patterns, and are unable to course of cross-variate data. On the opposite hand, Transformers use consideration mechanisms that apply dynamic and data-dependent weights at every time step, capturing dynamic temporal patterns and enabling them to course of cross-variate data.
In our evaluation, we present that beneath widespread assumptions of temporal patterns, linear fashions have naïve options to completely recuperate the time series or place bounds on the error, which implies they’re nice options for studying static temporal patterns of univariate time series extra successfully. In distinction, it’s non-trivial to search out related options for consideration mechanisms, because the weights utilized to every time step are dynamic. Consequently, we develop a brand new architecture by changing Transformer consideration layers with linear layers. The ensuing TSMixer mannequin, which is analogous to the pc imaginative and prescient MLP-Mixer methodology, alternates between purposes of the multi-layer perceptron in several instructions, which we name time-mixing and feature-mixing, respectively. The TSMixer architecture effectively captures each temporal patterns and cross-variate data, as proven within the determine under. The residual designs be certain that TSMixer retains the capability of temporal linear fashions whereas nonetheless having the ability to exploit cross-variate data.
 |
| Transformer block and TSMixer block architectures. TSMixer replaces the multi-head consideration layer with time-mixing, a linear mannequin utilized on the time dimension. |
 |
| Comparison between data-dependent (consideration mechanisms) and time-step-dependent (linear fashions). This is an instance of forecasting the following time step by studying the weights of the earlier three time steps. |
Evaluation on long-term forecasting benchmarks
We consider TSMixer utilizing seven fashionable long-term forecasting datasets (ETTm1, ETTm2, ETTh1, ETTh2, Electricity, Traffic, and Weather), the place latest analysis has proven that univariate linear fashions outperform superior multivariate fashions with giant margins. We evaluate TSMixer with state-of-the-art multivariate fashions (TFT, FEDformer, Autoformer, Informer), and univariate fashions, together with linear fashions and PatchTST. The determine under reveals the typical enchancment of imply squared error (MSE) by TSMixer in contrast with others. The common is calculated throughout datasets and a number of forecasting horizons. We display that TSMixer considerably outperforms different multivariate fashions and performs on par with state-of-the-art univariate fashions. These outcomes present that multivariate fashions are able to performing in addition to univariate fashions.
 |
| The common MSE enchancment of TSMixer in contrast with different baselines. The crimson bars present multivariate strategies and the blue bars present univariate strategies. TSMixer achieves important enchancment over different multivariate fashions and achieves comparable outcomes to univariate fashions. |
Ablation examine
We carried out an ablation examine to check TSMixer with TMix-Only, a TSMixer variant that consists of time mixing layers solely. The outcomes present that TMix-Only performs virtually the identical as TSMixer, which implies the extra characteristic mixing layers don’t enhance the efficiency and confirms that cross-variate data is much less useful on fashionable benchmarks. The outcomes validate the superior univariate mannequin efficiency proven in earlier analysis. However, current long-term forecasting benchmarks will not be nicely consultant of the necessity for cross-variate data in some real-world purposes the place time series could also be intermittent or sparse, therefore temporal patterns is probably not enough for forecasting. Therefore, it might be inappropriate to guage multivariate forecasting fashions solely on these benchmarks.
Evaluation on M5: Effectiveness of cross-variate data
To additional display the good thing about multivariate fashions, we consider TSMixer on the difficult M5 benchmark, a large-scale retail dataset containing essential cross-variate interactions. M5 comprises the knowledge of 30,490 merchandise collected over 5 years. Each product description contains time series information, like day by day gross sales, promote value, promotional occasion data, and static (non-time-series) options, resembling retailer location and product class. The aim is to forecast the day by day gross sales of every product for the following 28 days, evaluated utilizing the weighted root imply sq. scaled error (WRMSSE) from the M5 competitors. The difficult nature of retail makes it tougher to forecast solely utilizing univariate fashions that target temporal patterns, so multivariate fashions with cross-variate data and even auxiliary options are extra important.
First, we evaluate TSMixer to different strategies solely contemplating the historic information, resembling day by day gross sales and historic promote costs. The outcomes present that multivariate fashions outperforms univariate fashions considerably, indicating the usefulness of cross-variate data. And amongst all in contrast strategies, TSMixer successfully leverages the cross-variate data and achieves the very best efficiency.
Additionally, to leverage extra data, resembling static options (e.g., retailer location, product class) and future time series (e.g., a promotional occasion scheduled in coming days) supplied in M5, we suggest a precept design to increase TSMixer. The prolonged TSMixer aligns several types of options into the identical size, after which applies a number of mixing layers to the concatenated options to make predictions. The prolonged TSMixer architecture outperforms fashions fashionable in industrial purposes, together with DeepAR and TFT, showcasing its sturdy potential for real-world influence.
 |
| The architecture of the prolonged TSMixer. In the primary stage (align stage), it aligns the several types of options into the identical size earlier than concatenating them. In the second stage (mixing stage) it applies a number of mixing layers conditioned with static options. |
 |
| The WRMSSE on M5. The first three strategies (blue) are univariate fashions. The center three strategies (orange) are multivariate fashions that contemplate solely historic options. The final three strategies (crimson) are multivariate fashions that contemplate historic, future, and static options. |
Conclusion
We current TSMixer, a complicated multivariate mannequin that leverages linear mannequin traits and performs in addition to state-of-the-art univariate fashions on long-term forecasting benchmarks. TSMixer creates new potentialities for the event of time series forecasting architectures by offering insights into the significance of cross-variate and auxiliary data in real-world eventualities. The empirical outcomes spotlight the necessity to contemplate extra real looking benchmarks for multivariate forecasting fashions in future analysis. We hope that this work will encourage additional exploration within the area of time series forecasting, and result in the event of extra highly effective and efficient fashions that may be utilized to real-world purposes.
Acknowledgements
This analysis was performed by Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O. Arik, and Tomas Pfister.