Batch inference issue and left padding

[bshao001]

Hi there,

Thanks for the project firstly. I saw a method called get_padding_mask in tf_utils.py file. It is combined with attention mask. It is designed to resolve the padding issue in batch inference or training only?

I see that with very a few padding in the left can still make good predictions/generations, but if use a large batch for inference, which cause a long left padding, the predictions get very incorrect. Do you have any suggestions for that?

Looking forward to your response. Thanks.

[bshao001]

I was thinking to perform left padding at training time as well. But in most cases, we may not need batch inference, which does not need padding at all, which can be messed up if we trained that way.

[akanyaani]

Hi @bshao001,

Padding mask also works for inference, so you can do the inference using the right padding.
With left padding, this implementation will not give you the correct result

[bshao001]

Thanks for your quick response. I will give it a try when the model trained with a much larger dataset is ready.