How to set breakpoint in TF uniform replay buffer?

[someoneordinary]

Hello,
I'm currently dealing with tf_agents.replay_buffers.TFUniformReplayBuffer.
As the comments of source codes suggest, each block of the replay buffer is consisted of several episodes:

The TFUniformReplayBuffer stores episodes in `B == batch_size` blocks of
size `L == max_length`, with total frame capacity `C == L * B`.  
Storage looks like:

block1 ep1 frame1
           frame2
       ...
       ep2 frame1
           frame2
       ...
       <L frames total>
block2 ep1 frame1
           frame2
       ...
       ep2 frame1
           frame2
       ...
       <L frames total>
...
blockB ep1 frame1
           frame2
       ...
       ep2 frame1
           frame2
       ...
       <L frames total>

What I need to do is to manually write into a replay buffer, frame by frame. Each frame is an instance of tf_agents.trajectories.Trajectory.

I wonder whether there exists something serving as a breakpoint in TFUniformReplayBuffer to seperate each episode in a block? So that when I read data from the replay buffer with num_step>1, I won't get a mixed sequence composed of frames from different episodes, say, the last frame of episode1 and the first frame of episode2. And, if there do exist such thing, how can I set it manually?

Thanks

[sguada]

There is no good way to separate episodes in TFUniformReplaybuffer I would recommend switching to the ReverbReplayBuffer that handles episodes well.

See example usage https://github.com/tensorflow/agents/blob/master/tf_agents/examples/sac/haarnoja18/sac_train_eval.py

[zhengzl18]

Thx a lot for your help! I will take a shot.