Add time series forecasting support #611

Lopa10ko · 2024-03-06T15:56:56Z

Summary

Add support for time series forecasting functionality from FEDOT framework.

Context

closes #610

shchur · 2024-03-25T09:43:23Z

frameworks/FEDOT/exec_ts.py

+    training_params.update({k: v for k, v in config.framework_params.items() if not k.startswith('_')})
+    n_jobs = training_params["n_jobs"]
+
+    log.info('Running FEDOT with a maximum time of %ss on %s cores, optimizing %s.',


Nit: f-strings are a bit more readable and are compatible with all Python versions used by AMLB

f"Running FEDOT with a maximum time of {config.max_runtime_seconds}s on {n_jobs} cores, optimizing {scoring_metric}"

shchur · 2024-03-25T09:47:07Z

frameworks/FEDOT/exec_ts.py

+    predictions = []
+
+    for label in train_df[id_column].unique():
+        train_sub_df = train_df[train_df[id_column] == label].drop(columns=[id_column], axis=1)


This can be very inefficient for large dataframes with many time series (scales as O(N^2)). A better option would be to use groupby, e.g.

for label, ts in train_df.groupby(id_column, sort=False): train_series = ts[dataset.target].to_numpy()

thanks, tried that: imo using zip may be inappropriate here, so I would stick with a janky bad-asymptotic solution to ensure that the labels are matching
e6f19e7

shchur · 2024-03-25T09:48:10Z

frameworks/FEDOT/exec_ts.py

+        train_sub_df = train_df[train_df[id_column] == label].drop(columns=[id_column], axis=1)
+        train_series = np.array(train_sub_df[dataset.target])
+        train_input = InputData(
+            idx=train_sub_df.index.to_numpy(),


This code uses the range index [0, 1, 2, ..., n-1] instead of the timestamps of the time series. Should this be train_sub_df[timestamp_column] instead, or does FEDOT ignore the timestamps?

FEDOT ignores timestamps, so that idx should be fine

shchur · 2024-03-25T09:51:11Z

frameworks/FEDOT/exec_ts.py

+        )
+
+        test_sub_df = test_df[test_df[id_column] == label].drop(columns=[id_column], axis=1)
+        test_series = np.array(test_sub_df[dataset.target])


This contains the future values of the time series, I doubt that these values need to be fed to the model at predictions time.

For example, if training data contains time steps [1, 2, 3, ..., T] and the goal is to predict [T+1, ..., T+H], then based on your current code

train_series contains timesteps [1, 2, 3, ..., T]

test_series contains timesteps [T+1, ..., T+H]

My guess is that we need to pass train_series as input to both fit() and predict()

in essence predict and forecast are the same thing, but we could pass test_input with features from train_series and target from test_series to get the prediction horizon in a predict

changed to a clear forecast method
a357726

shchur · 2024-03-25T09:51:36Z

frameworks/FEDOT/exec_ts.py

+            timeout=runtime_min,
+            metric=scoring_metric,
+            seed=config.seed,
+            max_pipeline_fit_time=runtime_min / 10,


Why is / 10 necessary here?

generally speaking, this is a small safety measure to ensure that the training time of one pipeline is exactly within the total timeout. the classification and regression #563 uses the same empirical approach. it should be patched in the future

Would it make sense to split the time limit evenly across the series? Right now it seems that 10% of the total time limit is given to each series, which may lead to overruns if >10 series are available.

shchur · 2024-03-25T09:53:15Z

frameworks/FEDOT/exec_ts.py

+                  training_duration=training_duration,
+                  predict_duration=predict_duration)


Nit: I think it would be cleaner to remove the line training_duration, predict_duration = 0, 0 above and just return training_duration=training.duration, predict_duration=predict.duration here.

since we train the FEDOT model for each series (for each label), training.duration value isn't cumulative and will only reflect the time spent on the last iteration

My bad, I didn't realize this was happening inside the loop over individual series

shchur · 2024-03-25T09:55:57Z

frameworks/FEDOT/exec_ts.py

+        models_count += fedot.current_pipeline.length
+
+    save_artifacts(fedot, config)
+    return result(output_file=config.output_predictions_file,


It's necessary to return dataset.repeated_item_id and dataset.repeated_abs_seasonal_error as optional_columns in the result for MASE computation to work correctly (see https://github.com/openml/automlbenchmark/blob/master/frameworks/AutoGluon/exec_ts.py#L63C1-L67C1).
This is a rather ugly hack that is necessary to make history-dependent metrics like MASE compatible with the AMLB results API.

Lopa10ko · 2024-03-25T12:54:53Z

@shchu, thanks for the initial review :)
I intend to continue exploring the benchmarks and testing some things locally in order to ensure that all FEDOT-related functionality is working correctly. This may take some time, so when I'm finished, I will undraft this pull request.

Lopa10ko · 2024-07-11T13:37:08Z

@shchur sorry it took ages to finally return to this PR.
would be great to have a further review

PGijsbers · 2024-09-17T12:55:08Z

@shchur would you be available for another look?:)

shchur · 2024-09-20T06:47:32Z

Hi @Lopa10ko @PGijsbers, I will try to have a look at the PR today

shchur

Two small comments that probably need to be addressed, but otherwise looks good to me.

shchur · 2024-09-20T08:02:03Z

frameworks/FEDOT/exec_ts.py

+            timeout=runtime_min,
+            metric=scoring_metric,
+            seed=config.seed,
+            max_pipeline_fit_time=runtime_min / 10,


Would it make sense to split the time limit evenly across the series? Right now it seems that 10% of the total time limit is given to each series, which may lead to overruns if >10 series are available.

shchur · 2024-09-20T08:04:44Z

frameworks/FEDOT/exec_ts.py

+
+        fedot = Fedot(
+            problem=TaskTypesEnum.ts_forecasting.value,
+            task_params=task.task_params,


As far as I understand, Fedot currently only supports point forecasting but AMLB may also include probabilistic forecasting tasks (see https://github.com/openml/automlbenchmark/blob/master/amlb/results.py#L767-L792). Probably it would make sense to raise an exception if someone tries to evaluate FEDOT on such a probabilistic forecasting task.

is there a way to distinguish a probabilistic forecasting task based on the benchmark run config?

the get_fedot_metrics function already emits logs in case of unsupported metrics (like mql, wql, and sql)

I think that filtering by the mql, wql, sql metrics is the simplest way to accomplish this.

Another option is to repeat the point forecast for each of the quantile levels and raise a warning.

Lopa10ko · 2024-09-27T11:28:41Z

@shchur thanks for a review,
@PGijsbers if you have any suggestions for improvements, I would be happy to incorporate them

PGijsbers · 2024-10-05T16:10:31Z

In principle it looks fine. Does FEDOT support provided memory constraints? If so, I would appreciate it if you add that to the integration script (config.max_mem_size_mb).

Lopa10ko · 2024-10-05T17:18:51Z

In principle it looks fine. Does FEDOT support provided memory constraints? If so, I would appreciate it if you add that to the integration script (config.max_mem_size_mb).

@PGijsbers sorry, FEDOT does't have the capability to handle memory limitations.

PGijsbers · 2024-12-06T08:37:20Z

Sorry for the delay, I thought this was merged.

Lopa10ko and others added 12 commits February 19, 2024 01:45

feat: decompose fedot mapping for metrics and task type

ae06625

feat: add minimal support for ts forecasting

017b3ef

fix: change idx and timestamp

7ec5ef6

fix: create predictions from long format

385391b

fix: invoke save only one time

2218bd7

fix: use explicit dictiomary method instead of bitwise merge

4c459ff

fix: sum training and predict durations across set of series

dc02093

fix: sum up models_count across all series

4a84442

fix: change id_column relation to dataset instead of config

be5bf84

fix: remove test ts from test input

5d4b2c9

feat: add naive prediction on pipeline fail

956b3d5

Merge branch 'openml:master' into fedot-support-ts

35984b9

shchur self-assigned this Mar 13, 2024

shchur reviewed Mar 25, 2024

View reviewed changes

Lopa10ko and others added 5 commits March 25, 2024 13:02

Merge branch 'openml:master' into fedot-support-ts

11c13ce

fix: change format strings to f-strings

d583cb2

feat: refactor long dataframe iteration

e6f19e7

feat: change predict to forecast

a357726

fix: add optional_columns to result

c400961

Lopa10ko added 2 commits April 24, 2024 13:52

Merge branch 'openml:master' into fedot-support-ts

68e7f7b

Merge branch 'openml:master' into fedot-support-ts

2219055

Lopa10ko marked this pull request as ready for review May 16, 2024 13:26

Lopa10ko requested a review from shchur July 11, 2024 13:37

Merge branch 'openml:master' into fedot-support-ts

9555b59

Merge branch 'openml:master' into fedot-support-ts

d438060

shchur approved these changes Sep 20, 2024

View reviewed changes

Lopa10ko added 2 commits September 24, 2024 16:53

feat: distribute timeout equally among series, add version log

20dac85

feat: copy point forecast for all quantile levels

db10d89

PGijsbers merged commit cdc660d into openml:master Dec 6, 2024
4 of 10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add time series forecasting support #611

Add time series forecasting support #611

Lopa10ko commented Mar 6, 2024 •

edited

Loading

shchur Mar 25, 2024

Lopa10ko Mar 25, 2024

shchur Mar 25, 2024

Lopa10ko Mar 25, 2024

shchur Mar 25, 2024

Lopa10ko Mar 25, 2024

shchur Mar 25, 2024

Lopa10ko Mar 25, 2024

shchur Mar 25, 2024

Lopa10ko Mar 25, 2024 •

edited

Loading

shchur Sep 20, 2024

Lopa10ko Sep 24, 2024

shchur Mar 25, 2024

Lopa10ko Mar 25, 2024

shchur Mar 25, 2024

shchur Mar 25, 2024

Lopa10ko Mar 25, 2024

Lopa10ko commented Mar 25, 2024

Lopa10ko commented Jul 11, 2024

PGijsbers commented Sep 17, 2024

shchur commented Sep 20, 2024

shchur left a comment

shchur Sep 20, 2024

shchur Sep 20, 2024

Lopa10ko Sep 24, 2024

shchur Sep 25, 2024

Lopa10ko Sep 26, 2024

Lopa10ko commented Sep 27, 2024

PGijsbers commented Oct 5, 2024

Lopa10ko commented Oct 5, 2024

PGijsbers commented Dec 6, 2024

		training_duration=training_duration,
		predict_duration=predict_duration)

Add time series forecasting support #611

Add time series forecasting support #611

Conversation

Lopa10ko commented Mar 6, 2024 • edited Loading

Summary

Context

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lopa10ko Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lopa10ko commented Mar 25, 2024

Lopa10ko commented Jul 11, 2024

PGijsbers commented Sep 17, 2024

shchur commented Sep 20, 2024

shchur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lopa10ko commented Sep 27, 2024

PGijsbers commented Oct 5, 2024

Lopa10ko commented Oct 5, 2024

PGijsbers commented Dec 6, 2024

Lopa10ko commented Mar 6, 2024 •

edited

Loading

Lopa10ko Mar 25, 2024 •

edited

Loading