Soft-DTW as loss function for time series forecasting

[Samcro5C]

Hello,
first of all thanks for the amazing library and its documentation.

For my use-case I am trying to optimize a time series model such that the prediction curve (multi-step output) has a similar shape as the observation curve. Thus the rmse or mae is not the most critical value and I was wondering if a soft-dtw may be used as loss function
(see https://rtavenar.github.io/ml4ts_ensai/contents/align/align4ml.html#sec-forecasting)
I found some implementations (e.g., https://github.com/keonlee9420/Soft-DTW-Loss), however, it was not as straigth-forward to apply this Loss and after getting rid of the errors (due to unmatching tensor shapes), the results were really bad and it seems the model is mostly predicting 0s.

So my question is: has anyone ever successfully applied other loss functions for multi-step time series forecasting? If yes, could you share a code snippet for such a loss function?

Best regards,
sam

[oguiza]

has anyone ever successfully applied other loss functions for multi-step time series forecasting? If yes, could you share a code snippet for such a loss function?

No, I have not used other loss functions.
However, there may be something that might be useful. If you are interested in the shape more than in the actual values, you could normalize the target (ys) between 0 and 1. And use a MSELoss. What do you think?

[Samcro5C]

Thanks yor the reply, I will try this out. This can be done by simply adding Normalize() to the tfms list right? E.g.
TSForecaster(X, y, tfms=[None, Normalize()], ...)
right? However, I am not sure how this is going to make a big difference because in the end it is only scaling the output between 0 and 1, not penalizing the model if it is mostly predicting smooth curves of avg values.

[oguiza]

No, there's no functionality in tsai that would allow you to do a target normalization out of the box. You'd need to write it.
What I'm proposing is something simpler. Assuming y has 2 dimensions, it'd be something like this:

y = np.random.rand(8, 100) * 35 + np.random.rand(8, 100) * 10
y = (y - np.min(y, axis=1, keepdims=True)) / (np.max(y, axis=1, keepdims=True) - np.min(y, axis=1, keepdims=True))
y.min(1), y.max(1)

And you pass the y to the dataloader.

[Samcro5C]

Thanks, I also noticed, that the Normalize() function was not intended for this kind of normalization. I will try out to normalize the output like you proposed and see how it works.
However I also found a way to apply the DILATE loss function proposed by Le Guen et al 2019, as they shared their code on github.

class DILATELoss(nn.Module):
    def __init__(self, alpha=0.5, gamma=0.001, device=None):
        super(DILATELoss, self).__init__()
        store_attr()
        if device is None:
            self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
 
    def forward(self, inputs, targets):        
        loss, _, _ = dilate_loss(inputs.unsqueeze(-1), targets.unsqueeze(-1), 
                                 alpha=self.alpha, gamma=self.gamma, device=self.device)
        return loss

Thanks again for the fast response and the support!