How to use y_func in SlidingWindow?

[kiview]

As someone without good expertise in numpy, I am lacking an example of how to use y_func in SlidingWindow, as well as an explanation of what is the default behavior.

What I want to achieve is calling something like scipy.stats.mode on each slice of y values, in order to calculate the corresponding y for a window.

I tried to debug the behavior by writing my own function, but it seemed my custom function received the whole numpy array (not a window) as input parameter.

Also, what is the difference between SlidingWindow and TSUnwindowedDataset?

[oguiza]

Hi @kiview,

I've created a quick gist to try to explain the difference.

Please, let me know if you have any other questions.

[bellerofonte]

Hi @oguiza, I took a look at your gist and I have to say that imho this code looks much more compact and clear.
I've changed your new code to make it solve classification problem without any issues.

[oguiza]

Hi @oguiza, I took a look at your gist and I have to say that imho this code looks much more compact and clear. I've changed your new code to make it solve classification problem without any issues.

Great!. I'm glad to hear that. I still need to decide what to do with TSUnwindowedDataset. But I also find the new approach easier to use. And in most cases, it saves RAM memory as it doesn't copy the data (it uses views).

[bellerofonte]

@oguiza, I've tried to search word TSUnwindowedDataset here in issues and discussions. As far as I can understand, it is not a kind of popular functionality now.

[oguiza]

I don't think it's widely used.
The main use is when you have a large dataset and don't want or can't apply a sliding window on your data, as that may greatly increase its size.
But the new function I've created would avoid that issue if used correctly. By that I mean you should try to use views of your data instead of copies. You can do that if you pass slices or one integer to x_vars and y_vars. If you pass an array or a list a copy will be created, and you might have an OOM error if your data is large.
I've been using this approach for a few months now and it seems to work very well on large datasets.

[bellerofonte]

yep, this approach is pretty logic. imho TSUnwindowedDataset should be marked as deprecated in favor of apply_sliding_window