IndexError: Out of range: piece id is out of range.

[Carterbouley]

When running sample_sequence, I get this error:

IndexError: Out of range: piece id is out of range.

Any idea on what exactly I am doing wrong?

pre processing and training went fine

[AbhishekSanthanam]

Facing the same error

[tkahn]

I'm having the same issue.
Just like @Carterbouley, I ran the preprocessing and the training and everything went fine.
But when I start the sample sequence I get this error (after 30 seconds or so):

run sequence_generator.py
Model weights loaded into memory
Traceback (most recent call last):

  File "G:\Repos\gpt-2-tensorflow2.0\sequence_generator.py", line 28, in <module>
    seq_gen()

  File "G:\Python\Anaconda3\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)

  File "G:\Python\Anaconda3\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)

  File "G:\Python\Anaconda3\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)

  File "G:\Python\Anaconda3\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)

  File "G:\Repos\gpt-2-tensorflow2.0\sequence_generator.py", line 18, in seq_gen
    generated_seq = sg.sample_sequence(context,

  File "G:\Repos\gpt-2-tensorflow2.0\sample.py", line 117, in sample_sequence
    generated_seq = self.sp.decode_ids(pred[1:])

  File "G:\Python\Anaconda3\lib\site-packages\sentencepiece\__init__.py", line 174, in DecodeIdsWithCheck
    return _sentencepiece.SentencePieceProcessor_DecodeIdsWithCheck(self, ids)

IndexError: Out of range: piece id is out of range.

As you can see from the traceback, I'm running the script in an Anaconda environment (Anaconda Navigator 2.1.1) using Spyder 5.1.5.

Python version 3.8.12.
Tensorflow version 2.7 (currently running on CPU because of insufficient amount of VRAM (11GB) on GPU).
Sentencepiece version 0..1.96
Windows 11

[AbuUbaida]

Just like @tkahn . After training and evaluating an epoch when generating/scoring the predictions it throws the exception:

[/usr/local/lib/python3.7/dist-packages/sentencepiece/__init__.py](https://localhost:8080/#) in _func(v, n)
   1036   def _func(v, n):
   1037     if type(n) is int and (n < 0 or n >= v.piece_size()):
-> 1038       raise IndexError('piece id is out of range.')
   1039     return func(v, n)
   1040 

IndexError: piece id is out of range.

Anybody could figure it out?