WEATHER-5K Benchmark

Paper: WEATHER-5K: A Large-scale Global Station Weather Dataset Towards Comprehensive Time-series Forecasting Benchmark

Introduction

The WEATHER-5K dataset is a large-scale time series forecasting dataset containing weather data from 5,672 weather stations worldwide. It is a valuable resource for researchers and developers in the field of time-series forecasting, providing a comprehensive evaluation of various methods and models. WEATHER-5K dataset consists of a comprehensive collection of data from 5,672 weather stations worldwide, spanning a 10-year period with one-hour intervals. It includes multiple crucial weather elements (temperature, dewpint temperature, wind speed, wind rate, sea level pressure), providing a more reliable and interpretable resource for forecasting.

🚩News (2024.06) We release the WEATHER-5K as a comprehensive benchmark, allowing for a thorough evaluation of time-series forecasting methods and facilitates advancements in this field.

Leaderboard of WEATHER-5K benchmark

Until now, we have bnchmarked the following models in this repo:

• iTransformer - iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [ICLR 2024] [Code].

• Corrformer - nterpretable weather forecasting for worldwide stations with a unified deep model [NMI 2023] [Code].

• PatchTST - A Time Series is Worth 64 Words: Long-term Forecasting with Transformers [ICLR 2023] [Code].

• DLinear - Are Transformers Effective for Time Series Forecasting? [AAAI 2023] [Code].

• FEDformer - FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting [ICML 2022] [Code].

• Pyraformer - Pyraformer: Low-complexity Pyramidal Attention for Long-range Time Series Modeling and Forecasting [ICLR 2022] [Code].

• Autoformer - Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting [NeurIPS 2021] [Code].

• Informer - Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting [AAAI 2021] [Code].

Benchmarking results

The results are reported at 4 different prediction lengths: 24, 72, 120, and 168, where the input length is 48.

Baselines	Lead Time	Temperature		Dewpoint		Wind Speed		Wind Direction		Sea Level Pressure		Overall
		MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE
🥇 1st Pyraformer	24	1.75	6.92	1.83	7.88	1.30	3.58	61.8	6930.2	1.90	9.72	13.7	1391.7
	72	2.47	13.03	2.67	15.39	1.52	4.97	72.0	8222.4	3.76	33.67	16.5	1657.9
	120	2.77	16.04	3.00	18.95	1.59	5.37	75.1	8610.7	4.43	43.91	17.4	1739.0
	168	2.95	17.95	3.20	21.06	1.61	5.56	76.4	8773.5	4.77	49.97	17.8	1773.6
🥈 2nd iTransformer	24	1.82	7.49	1.93	8.80	1.32	3.77	63.2	7358.8	1.99	10.84	14.1	1478.0
	72	2.60	14.46	2.84	17.5	1.52	4.96	73.2	8713.3	4.14	40.65	16.9	1758.2
	120	2.97	18.36	3.24	22.16	1.59	5.42	76.4	9192.2	4.95	54.67	17.8	1858.6
	168	3.18	20.64	3.48	24.89	1.64	5.67	78.0	9441.1	5.36	62.31	18.3	1910.9
🥉 3rd Informer	24	1.88	7.51	1.94	8.30	1.30	3.62	60.7	6906.9	2.01	10.56	13.6	1387.4
	72	2.75	14.84	2.86	17.24	1.53	4.86	71.5	8251.4	4.24	39.24	16.4	1631.4
	120	3.11	18.21	3.25	21.50	1.60	5.38	75.7	8504.5	5.15	54.31	18.3	1720.4
	168	3.24	20.24	3.43	24.89	1.63	5.65	76.2	8718.4	5.26	58.42	18.1	1764.4
Autoformer	24	1.93	8.64	2.06	9.57	1.42	3.97	66.5	7710.0	2.26	12.78	15.2	1553.4
	72	2.72	15.14	2.97	18.38	1.54	5.14	75.4	9111.5	4.25	42.34	17.8	1846.7
	120	3.21	20.27	3.34	23.12	1.58	5.73	79.2	9143.5	4.83	48.88	18.1	1868.3
	168	3.43	21.71	3.56	22.55	1.64	5.95	79.8	9435.8	5.32	61.85	18.5	1885.7
FEDformer	24	1.98	8.45	2.02	9.25	1.36	3.91	66.0	7384.1	2.13	11.43	14.7	1483.4
	72	2.87	16.50	3.01	18.70	1.59	5.31	76.2	8824.8	4.15	37.60	17.6	1780.6
	120	3.19	20.29	3.36	23.10	1.66	5.71	79.0	9143.3	4.81	48.86	18.4	1848.3
	168	3.35	22.12	3.54	25.21	1.68	5.88	79.7	9189.2	5.01	53.39	18.7	1859.2
Dlinear	24	2.71	13.82	2.47	12.36	1.44	4.34	66.6	8234.5	3.09	21.34	15.3	1657.3
	72	3.55	23.05	3.48	22.85	1.62	5.37	75.0	9250.8	4.64	45.83	17.7	1869.6
	120	3.90	27.60	3.89	27.72	1.67	5.70	77.3	9510.6	5.19	56.22	18.4	1925.6
	168	4.11	30.38	4.11	30.58	1.69	5.88	78.4	9630.0	5.48	61.73	18.8	1951.7
PatchTST	24	2.05	9.26	2.16	10.58	1.40	4.20	66.2	7765.8	2.19	12.54	14.8	1560.5
	72	2.82	16.60	3.06	19.96	1.60	5.39	75.2	9067.8	4.28	42.46	17.4	1830.5
	120	3.15	20.32	3.43	24.39	1.66	5.79	77.8	9452.6	5.09	57.29	18.2	1912.1
	168	3.33	22.54	3.63	26.94	1.69	6.00	79.0	9638.1	5.51	65.3	18.6	1951.7
Corrformer	24	1.99	8.21	2.09	9.47	1.38	3.83	66.7	7832.3	2.19	12.39	14.9	1584.4
	72	2.74	15.16	2.99	18.40	1.56	4.91	75.6	9111.7	4.27	42.36	17.8	1846.7
	120	3.06	18.63	3.34	22.48	1.61	5.56	78.0	9477.4	5.08	57.13	18.1	1915.8
	168	3.09	18.69	3.36	22.53	1.63	5.69	78.9	9636.0	5.34	61.83	18.4	1938.8
Mamba	24	1.98	8.59	2.01	9.52	1.37	4.02	66.0	7709.5	2.21	12.73	14.7	1548.9
	72	2.79	16.00	2.90	18.11	1.55	5.11	75.1	8863.9	4.29	41.88	17.3	1789.0
	120	3.03	18.47	3.18	21.02	1.58	5.28	76.7	8931.2	4.93	52.56	17.9	1805.7
	168	3.16	19.88	3.32	22.53	1.59	5.35	77.4	8958.8	5.21	57.37	18.1	1812.8

**Compared models of this leaderboard.** ☑ means that their codes have already been included in this repo.

Usage

1. Install Python 3.8. For convenience, execute the following command.

pip install -r requirements.txt

2. Prepare Data. You can obtain the well pre-processed datasets from [OneDrive], Then place and unzip the downloaded data in the folder./WEATHER-5K.

3. Train and evaluate model. We provide the experiment scripts for all benchmarks under the folder ./scripts/. You can reproduce the experiment results as the following examples:

# Global Station Weather Forecasting
bash ./scripts/weather-5k/iTransformer.sh

4. Develop your own model.

• Add the model file to the folder ./models. You can follow the ./models/Transformer.py.
• Include the newly added model in the Exp_Basic.model_dict of ./exp/exp_basic.py.
• Create the corresponding scripts under the folder ./scripts.

Citation

If you find WEATHER-5K is useful, please cite our paper.

@misc{han2024weather5k,
    title={WEATHER-5K: A Large-scale Global Station Weather Dataset Towards Comprehensive Time-series Forecasting Benchmark},
    author={Tao Han and Song Guo and Zhenghao Chen and Wanghan Xu and Lei Bai},
    year={2024},
    eprint={2406.14399},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

Contact

If you have any questions or suggestions, feel free to contact:

• Tao Han (hantao10200@gmail.com)

Or describe it in Issues.

Acknowledgement

This library is constructed based on the Time-Series-Library. We sincerely thank the contributors for their contributions.