sg.sample_sequence returns context after pre-trained model

[bytes-commerce]

First ofd all, thanks for providing this amazing repository providing a possibility for tf2!
Secondly, I were using the Readme to pre-train my model and eventually using sequence_generator.py to pass some context to the model.

However, the response is always 1:1 the same as the context but the capital letters are being replaced with ??s. The question now is, what am I doing wrong? Have I maybe forgotten a thing? Is there maybe a edge case leading to this point that could be prevented?

Please let me know any additional information you might need! Thanks a lot!

[jzl0166]

same problem

[jspangl3r]

also getting weird output like this.

[vedranbajic]

First of all, thank you for sharing your code! Helped me a lot starting with gpt2.
I really do not know if this is relevant but I just debugged sample.py.

output will only append zeros:
tf.Tensor([[ 3 13727 5825 0 0 0 0 0 ...]], shape=(1, 515), dtype=int32)

If my sequence length is 512 — I will get 512 zeros (+3 above zero numbers because of my context).
My output is just the words I have provided as context because the rest is 0.

edit 1:
logits is always nan in my case resulting in 0.

edit 2:
self.embedding_weights is nan. Maybe somethings wrong with the initializer?