Bias Correction

[A-els]

Hello,

The original BERT TF was pre-trained with bias correction set to False in the Adam optimizer. Is this also the case for how AraBERT was pre-trained?

Thanks.

[WissamAntoun]

I guess yes, We also didn't notice the bias correction was not applied as in google-research/bert#153.
The thing is we used the LAMB optimiser that was used in ALBERT which also doesn't have bias correction https://github.com/aub-mind/arabert/blob/master/arabert/lamb_optimizer.py#L98

[A-els]

Thank you!