Combining ModelGapFiller & AutoML

[darleybarreto]

Hi folks, I'm trying to combine ModelGapFiller and AutoML, my script is basically the one bellow

task_parameters = tasks.TsForecastingParams(forecast_length=forecast_len)
task = tasks.Task(tasks.TaskTypesEnum.ts_forecasting, task_parameters)

model = Fedot(
    problem='ts_forecasting',
    task_params=task_parameters,
    timeout=0.5)

_input = InputData(idx=np.arange(0, len(original_array)),
                    features=original_array,
                    target=original_array,
                    task=task,
                    data_type=DataTypesEnum.ts)

pipeline = model.fit(_input)

gapfiller = ModelGapFiller(gap_value=gap_value,
                                    pipeline=pipeline)

array_with_gaps = filling_gaps(original_array) # a function to add gap_value

output = gapfiller.forward_inverse_filling(array_with_gaps)

The original_array is the original data itself (without the value representing a missing unit). Here's the stacktrace

  File "...", line XXX, in gapfiller.forward_inverse_filling(array_with_gaps)
  File "fedot/utilities/ts_gapfilling.py", line 208, in forward_inverse_filling
    weights_list, predicted_list = direction_function(output_data,
  File "fedot/utilities/ts_gapfilling.py", line 275, in _forward
    predicted_values = self.__pipeline_fit_predict(timeseries_train_part,
  File "fedot/utilities/ts_gapfilling.py", line 335, in __pipeline_fit_predict
    self.pipeline.fit_from_scratch(input_data)
  File "fedot/core/pipelines/pipeline.py", line 59, in fit_from_scratch
    self.fit(input_data, use_fitted=False)
  File "fedot/core/pipelines/pipeline.py", line 171, in fit
    train_predicted = self._fit(input_data=copied_input_data,
  File "fedot/core/pipelines/pipeline.py", line 139, in _fit
    train_predicted = self.root_node.fit(input_data=input_data)
  File "fedot/core/pipelines/node.py", line 250, in fit
    return super().fit(input_data=secondary_input)
  File "fedot/core/pipelines/node.py", line 108, in fit
    self.fitted_operation, operation_predict = self.operation.fit(data=input_data,
  File "fedot/core/operations/operation.py", line 89, in fit
    fitted_operation = self._eval_strategy.fit(train_data=data)
  File "fedot/core/operations/evaluation/evaluation_interfaces.py", line 178, in fit
    operation_implementation.fit(train_data.features, train_data.target)
  File "sklearn/ensemble/_forest.py", line 304, in fit
    X, y = self._validate_data(X, y, multi_output=True,
  File "sklearn/base.py", line 433, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "sklearn/utils/validation.py", line 871, in check_X_y
    X = check_array(X, accept_sparse=accept_sparse,
  File "sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "sklearn/utils/validation.py", line 694, in check_array
    raise ValueError(ValueError: Expected 2D array, got 1D array instead: array=[].

I tracked down the error, at some point between

fedot/core/pipelines/pipeline.py(139)_fit()
-> train_predicted = self.root_node.fit(input_data=input_data)

and

fedot/core/pipelines/node.py(250)fit()
-> return super().fit(input_data=secondary_input)

the feature in secondary_input becomes array([], dtype=float64), which is the problem.
Would you know what I am doing wrong here? Is it feeding the raw data to the Fedot object?

[Dreamlone]

Hi, Darley

Thank you for your interest in FEDOT's gap-filling functionality. However, there are some points I did not got from your post. Please help me to figure it out (and then I can help you):

1. Please provide the imports
2. What is the function filling_gaps, it is not declared anywhere, is it your custom function, which adds gaps to the data?
3. Do I understand correctly that original_array is the array with gaps on which you run the Fedot object? That's array can be like [1, 5, -100, 4, 3], where -100 is a gap. Am I right?

[darleybarreto]

Hi Mikhail, I'm sorry about my poorly worded and bonkers question. Let me clarify your questions succinctly

Please provide the imports

The imports related to any TS work are

from fedot.api.main import Fedot
from fedot.core.repository import tasks
from fedot.core.data.data import InputData
from fedot.utilities.ts_gapfilling import ModelGapFiller
from fedot.core.repository.dataset_types import DataTypesEnum

What is the function filling_gaps, it is not declared anywhere, is it your custom function, which adds gaps to the data?

This code is a simplification of what I'm actually doing. I am performing some processing with pandas. filling_gaps here is a custom funcion and represents such processing. The bottom line is that this function would add some values in the 1D array. That is, if the original_array has shape (100,), this means that I have 100 time units, right? filling_gaps will add more time units with gap_value, so depending on my requirements, this could be more 20 values or any other number. So the final array_with_gaps will have shape (120,).

Do I understand correctly that original_array is the array with gaps on which you run the Fedot object? That's array can be like [1, 5, -100, 4, 3], where -100 is a gap. Am I right?

No, that would be array_with_gaps, whereas original_array is [1, 5, 4, 3] in this case.

[Dreamlone]

The length of the prediction can be equal to the average length of the gap in your row. But it really doesn't matter that much. The main thing is that the size of the moving_window hyperparameter in the lagged operation should be equal to the smallest interval amongst the gaps intervals

[darleybarreto]

Yeah, I copied the wrong name, I meant that moving_window hehe. In the AutoML pipeline, can I set the window_size (which is the moving_window, right?) ?

[Dreamlone]

Yes, you are absolutely right
You can use
node_lagged.custom_params = {'window_size': 10}
construction if you want to set 10 for window_size for example

[darleybarreto]

In the usual automl flow:

model = Fedot(
        problem='ts_forecasting',
        task_params=_input.task.task_params,
        timeout=0.5)

pipeline = model.fit(_input)

Than I have to change the this parameter after computing the pipeline, right?

[Dreamlone]

Than I have to change the this parameter after computing the pipeline, right?

Yes, that should work correctly

[darleybarreto]

Does this make sense to you?

Hyperparameters tuning started
Tuner metric is None, mean_squared_error was set as default
Hyperparameters optimization start
  0%|                                                                         | 0/1000 [00:00<?, ?trial/s, best loss=?]Warning: window size of lagged transformation was changed from 372 to 7
  0%|                                                | 1/1000 [00:00<14:31,  1.15trial/s, best loss: 11.67540994760749]Warning: window size of lagged transformation was changed from 20 to 7
  0%|                                               | 2/1000 [00:03<26:55,  1.62s/trial, best loss: 10.542602734874103]Warning: window size of lagged transformation was changed from 92 to 7
  0%|▏                                              | 3/1000 [00:03<21:04,  1.27s/trial, best loss: 10.542602734874103]Warning: window size of lagged transformation was changed from 86 to 7
  0%|▏                                              | 4/1000 [00:05<24:30,  1.48s/trial, best loss: 10.542602734874103]Warning: window size of lagged transformation was changed from 300 to 7
  0%|▏                                               | 5/1000 [00:07<26:55,  1.62s/trial, best loss: 9.555136600209773]Warning: window size of lagged transformation was changed from 329 to 7
  1%|▎                                               | 6/1000 [00:08<24:28,  1.48s/trial, best loss: 9.555136600209773]Warning: window size of lagged transformation was changed from 277 to 7
  1%|▎                                               | 7/1000 [00:10<28:00,  1.69s/trial, best loss: 9.555136600209773]Warning: window size of lagged transformation was changed from 128 to 7
  1%|▍                                               | 8/1000 [00:12<25:01,  1.51s/trial, best loss: 9.555136600209773]Warning: window size of lagged transformation was changed from 291 to 7
  1%|▍                                               | 9/1000 [00:14<27:31,  1.67s/trial, best loss: 8.254603634956059]Warning: window size of lagged transformation was changed from 209 to 7
  1%|▍                                              | 10/1000 [00:16<26:49,  1.63s/trial, best loss: 8.254603634956059]
Warning: window size of lagged transformation was changed from 291 to 7
Hyperparameters optimization finished
Return tuned pipeline due to the fact that obtained metric 8.255 equal or smaller than initial (+ 5% deviation) 8.335
Model composition finished
Warning: window size of lagged transformation was changed from 291 to 11
Warning: window size of lagged transformation was changed from 291 to -5

And when I checked the custom_params I see {'window_size': 290.78213219674}. The array has size 25 and the array with gaps filled 29. Even though I can change the window_size to whatever, I find this value too big.

[Dreamlone]

Do I understand correctly that you are using fedot version 0.4?
If so, you are probably can see this problem (about custom_params) described in this closed issue. It has been corrected, and now it's not in the master branch.

As for the phrase "Warning: window size of lagged transformation was changed from 291 to -5":
The lagged operation has a built-in function to correct incorrect window_size values, for example when the length of a time series is 100 elements, the forecasting horizon is 10 and window_size is 150, the algorithm will automatically correct the window_size value to an acceptable one: 100 - 10 (forecasting horizon) - 10 (constant threshold) = 80. A lagged feature table (trajectory matrix) with 10 rows and 80 columns will be generated.
But if your time series is 100 elements and your forecast horizon is 110, you won't have a chance to prepare a model. The algorithm will correct the window_size value to 100 - 110 - 10 = -20. This number doesn't make sense, it just means that you're trying to make a prediction for more elements than you have in your time series for fit
We will add a more complete description of such an error later

[darleybarreto]

Thank you so much for the explanation!

[darleybarreto]

I tried running the gapfill.py file, but firstly it gave this error:

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
----> 1 pipeline = model.fit(_input)
...
Exception: The timeout argument should be None or a positive value. Given value: 0

I replaced timeout=0.02 with timeout=0.1, which works fine. Then when the execution reaches output = gapfiller.forward_inverse_filling(array_with_gaps) I have the same problem as before

Warning: window size of lagged transformation was changed from 308 to -2
...
ValueError: Expected 2D array, got 1D array instead:
array=[].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

I'm using the 0.4.0 version.

[Dreamlone]

I've checked your code. There are no errors here:

Code

import matplotlib.pyplot as plt
import random
import numpy as np
from fedot.utilities.ts_gapfilling import SimpleGapFiller

import torch
import tensorflow as tf

SEED = 10
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
tf.random.set_seed(SEED)


forecast_len = 5
original_array = [27,  7, 33, 73, 95,  9, 31, 92, 48, 50, 64, 12, 30, 69, 91, 88, 60, 66, 76, 54, 34, 69, 45, 60]
array_with_gaps = [27, 7, 33, 73, 95, -100, 9, -100, 31, 92, 48, 50, 64, 12, 30, 69, 91, 88, 60, 66, -100, 76, 54, 34, 69, -100, 45, 60]

simple_gapfill = SimpleGapFiller(gap_value=-100)

# @@@@@@ Error in the line below  @@@@@@
without_gap = simple_gapfill.local_poly_approximation(array_with_gaps, degree=2, n_neighbors=10)

plt.plot(array_with_gaps)
plt.plot(without_gap)
plt.show()

So, you can obtain the following results:

So the problem does not arise at the stage of filling in the gaps

[darleybarreto]

Yeah, sorry, that is a overly simplified version of what I'm doing. Let me run it and provide you a synthetic version of the data itself.

[darleybarreto]

I can't seem to hit this error anymore :(

[Dreamlone]

Would it be possible to log only the errors?

Yes, sure. You can set verbose level parameter for it:

Code

_input = InputData(idx=np.arange(0, len(without_gap)),
                    features=without_gap,
                    target=without_gap,
                    task=task,
                    data_type=DataTypesEnum.ts)

model = Fedot(
    problem='ts_forecasting',
    task_params=_input.task.task_params,
    timeout=0.5, verbose_level=-1)
model.fit(_input)

[darleybarreto]

I was thinking about stop printing everything, like The pipeline saved in the path ... and Hyperparameters optimization start progress bar stuff.

[darleybarreto]

I've seen something similar the following code in some examples

input = InputData(idx=list(range(0, len(array))),
                   features=array,
                   target=array,
                   task=task, data_type=DataTypesEnum.ts)

This can cause an error because idx is a list and cannot be indexed by a numpy array or another list here. Unfortunately I lost the stacktrace and I couldn't reproduce the error :/
Perhaps casting the idx to numpy array to make sure it won't break.

[Dreamlone]

Thank you very much, we will fix such places in the framework
Yes, you're right, converting to a numpy array will help avoid errors

Or, if you like, you could fix these bugs via pull request

[darleybarreto]

Right now I can't push to github repos, but I will fill a PR ASAP.

[Dreamlone]

You can make a fork of the repository, and then suggest changes

Please check this page. It tells you in detail how you can become a FEDOT contributor

[darleybarreto]

Yeah, but I mean, for other reasons I can't push anything to github.com :/

[darleybarreto]

There's another problem going on at pipeline = model.fit(_input) in gapfill.py

  File "fedot/api/main.py", line 112, in fit
    self.current_pipeline, self.best_models, self.history = self.helper.obtain_model(**self.composer_dict)
  File "fedot/api/api_utils/composer.py", line 87, in obtain_model
    self.current_model, self.best_models, self.history = self.compose_fedot_model(
  File "fedot/api/api_utils/composer.py", line 202, in compose_fedot_model
    pipeline_gp_composed = gp_composer.compose_pipeline(data=api_params['train_data'])
  File "fedot/core/composer/gp_composer/gp_composer.py", line 137, in compose_pipeline
    best_pipeline = self.optimiser.optimise(objective_function_for_pipeline,
  File "fedot/core/optimisers/gp_comp/param_free_gp_optimiser.py", line 130, in optimise
    self.requirements.pop_size = self.next_population_size(new_population)
  File "fedot/core/optimisers/gp_comp/param_free_gp_optimiser.py", line 221, in next_population_size
    fitness_improved, complexity_decreased = improvements_checker(offspring)
  File "fedot/core/optimisers/gp_comp/param_free_gp_optimiser.py", line 211, in _check_so_improvements
    best_in_offspring = self.get_best_individual(offspring, equivalents_from_current_pop=False)
  File "fedot/core/optimisers/gp_comp/gp_optimiser.py", line 306, in get_best_individual
    best_ind = min(individuals, key=lambda ind: ind.fitness)
  TypeError: '>' not supported between instances of 'float' and 'NoneType'

I've found this before, but forgot about it. Unfortunately I haven't had any luck in getting this back. If I'm not mistaken, the None appears here at some point.

[Dreamlone]

Yes, thank you very much, we are aware of this bug (check issue438)
We are trying hard to fix it

[nicl-nno]

This bug should be fixed in new version (0.4.1). Please check.