3 months ago

Long-term series forecasting with Query Selector -- efficient model of sparse attention

Jacek Klimek Jakub Klimek Witold Kraskiewicz Mateusz Topolewski

Abstract

Various modifications of TRANSFORMER were recently used to solve time-series forecasting problem. We propose Query Selector - an efficient, deterministic algorithm for sparse attention matrix. Experiments show it achieves state-of-the art results on ETT, Helpdesk and BPI'12 datasets.

Code Repositories

moraieu/query-selector

Official

pytorch

Mentioned in GitHub

moraieu/query-selector-classification

pytorch

Mentioned in GitHub

Benchmarks

Benchmark	Methodology	Metrics
multivariate-time-series-forecasting-on	QuerySelector	Accuracy: 0.743
multivariate-time-series-forecasting-on-bpi	QuerySelector	Accuracy: 0.79
time-series-forecasting-on-etth1-168-1	Transformer	MAE: 0.6325 MSE: 0.7146
time-series-forecasting-on-etth1-168-1	QuerySelector	MAE: 0.6088 MSE: 0.6835
time-series-forecasting-on-etth1-168-2	Transformer	MAE: 0.2539 MSE: 0.1049
time-series-forecasting-on-etth1-168-2	QuerySelector	MAE: 0.2371 MSE: 0.0935
time-series-forecasting-on-etth1-24-1	Transformer	MAE: 0.4788 MSE: 0.4496
time-series-forecasting-on-etth1-24-1	QuerySelector	MAE: 0.4627 MSE: 0.4226
time-series-forecasting-on-etth1-24-2	Transformer	MAE: 0.183 MSE: 0.0548
time-series-forecasting-on-etth1-24-2	QuerySelector	MAE: 0.1616 MSE: 0.0436
time-series-forecasting-on-etth1-336-1	Transformer	MAE: 0.7041 MSE: 0.8321
time-series-forecasting-on-etth1-336-1	QuerySelector	MAE: 0.7039 MSE: 0.8503
time-series-forecasting-on-etth1-336-2	Transformer	MAE: 0.3201 MSE: 0.1541
time-series-forecasting-on-etth1-336-2	QuerySelector	MAE: 0.2844 MSE: 0.1267
time-series-forecasting-on-etth1-48-1	Transformer	MAE: 0.4968 MSE: 0.4668
time-series-forecasting-on-etth1-48-1	QuerySelector	MAE: 0.4878 MSE: 0.4581
time-series-forecasting-on-etth1-48-2	Transformer	MAE: 0.2144 MSE: 0.074
time-series-forecasting-on-etth1-48-2	QuerySelector	MAE: 0.2118 MSE: 0.0721
time-series-forecasting-on-etth1-720-1	Transformer	MAE: 0.8399 MSE: 1.108
time-series-forecasting-on-etth1-720-1	QuerySelector	MAE: 0.8428 MSE: 1.115
time-series-forecasting-on-etth1-720-2	Transformer	MAE: 0.4213 MSE: 0.2501
time-series-forecasting-on-etth1-720-2	QuerySelector	MAE: 0.373 MSE: 0.2136
time-series-forecasting-on-etth2-168-1	QuerySelector	MAE: 1.0125 MSE: 1.7385
time-series-forecasting-on-etth2-168-1	Transformer	MAE: 0.9726 MSE: 1.6225
time-series-forecasting-on-etth2-168-2	QuerySelector	MAE: 0.3322 MSE: 0.1753
time-series-forecasting-on-etth2-168-2	Transformer	MAE: 0.3547 MSE: 0.1974
time-series-forecasting-on-etth2-24-1	QuerySelector	MAE: 0.4864 MSE: 0.4124
time-series-forecasting-on-etth2-24-1	Transformer	MAE: 0.5013 MSE: 0.4237
time-series-forecasting-on-etth2-24-2	QuerySelector	MAE: 0.2239 MSE: 0.0843
time-series-forecasting-on-etth2-24-2	Transformer	MAE: 0.2479 MSE: 0.0999
time-series-forecasting-on-etth2-336-1	Transformer	MAE: 1.2189 MSE: 2.6617
time-series-forecasting-on-etth2-336-1	QuerySelector	MAE: 1.1859 MSE: 2.3168
time-series-forecasting-on-etth2-336-2	Transformer	MAE: 0.3805 MSE: 0.2191
time-series-forecasting-on-etth2-336-2	QuerySelector	MAE: 0.371 MSE: 0.2088
time-series-forecasting-on-etth2-48-1	Transformer	MAE: 0.9488 MSE: 1.522
time-series-forecasting-on-etth2-48-1	QuerySelector	MAE: 0.9317 MSE: 1.4074
time-series-forecasting-on-etth2-48-2	QuerySelector	MAE: 0.2622 MSE: 0.1117
time-series-forecasting-on-etth2-48-2	Transformer	MAE: 0.2763 MSE: 0.1218
time-series-forecasting-on-etth2-720-1	QuerySelector	MAE: 1.3084 MSE: 3.0664
time-series-forecasting-on-etth2-720-1	Transformer	MAE: 1.3668 MSE: 3.1805
time-series-forecasting-on-etth2-720-2	Transformer	MAE: 0.434 MSE: 0.2853
time-series-forecasting-on-etth2-720-2	QuerySelector	MAE: 0.413 MSE: 0.2585

Build AI with AI

From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.

AI Co-coding

Ready-to-use GPUs

Best Pricing

Get Started

Hyper Newsletters

Subscribe to our latest updates

We will deliver the latest updates of the week to your inbox at nine o'clock every Monday morning

Command Palette