RF (PT) meaning of losses with as_error

[albertz]

When you enable calculate_exp_loss, it will not calculate the exp loss for all the losses, but currently only those where as_error=False.

This decision was somewhat arbitrary. The exp loss often does not make sense for all the losses. Esp I often have the frame-error-rate (FER) with as_error=True, and there the exp loss does not make sense, so I thought the selection of losses with as_error=False is reasonable.

However, the actual meaning of as_error is slightly different: The main difference of as_error=True vs as_error=False (despite maybe reporting, although that difference was mostly removed) is that losses with as_error=True will not be used for the training optimizer.

And now I have some train config where I use as_error=True exactly for this. I calculate some CE, but I don't want to have it being used for the optimizer, thus I set as_error=True. However, I actually want to get the exp loss for this (the PPL).

Maybe the actual problem is also the ambiguous definition of as_error. Maybe I should use scale=0 instead of as_error=True? But if that is not the main aspect about as_error, then what is it actually? Then there is no point at all anymore for as_error, and we should remove (deprecate) it? as_error anyway is somewhat historical.

I think that is actually the main aspect in the issue here: The ambiguous definition of as_error. What should this be used for? Maybe we should just deprecate/remove it.

(The calculate_exp_loss is maybe also ambiguous, but it's not really so problematic, if we just calculate it for all losses, like we do right now. But this is a somewhat separate issue.)