[Question] What is the purpose of "always" checkpointing mode?

[pritamdamania87]

The user guide mentions the following:

Usually, checkpointing at the last micro-batch may not be useful because the saved memory will be reconstructed immediately. That’s why we choose 'except_last' as the default option.

It seems like using the mode "always" might not have any benefit, are there some cases where it makes sense to use "always" instead of "exception_last"?

[sublee]

I agree that "always" might have no benefit for most cases. However, for a special case chunks=1, it is useful to compare performance with and without checkpointing, because every chunk is the last chunk if there is only one chunk.