Support checkpointing in PaddlePaddle iterator

[szkarpinski]

Category:

New feature (non-breaking change which adds functionality)

Description:

Checkpointing support was already implemented in BaseIterator in #5061, but wasn't tested for frameworks other than Pytorch. In this PR I add tests for PaddlePaddle Iterator and fix a problem with ES checkpointing. This work is very similar to #5282

When an iterator is created and there's no data in the external source, DALI reports "no data in pipeline" error. The problem is that if we checkpoint after the last iteration and restore from such checkpoint, there's no data in the external source but it's not an error. In this PR I silence this error if we're restoring from checkpoint. Exactly the same change was made in #5213 and #5282

Additional information:

Affected modules and functionalities:

PaddlePaddle Iterator

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-3751

[szkarpinski]

@klecki @banasraf Thanks for the approvals, I'll merge #5298 first and then this one

[szkarpinski]

!build

[dali-automaton]

CI MESSAGE: [12664006]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [12664006]: BUILD FAILED

[szkarpinski]

!build

[dali-automaton]

CI MESSAGE: [12676712]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [12676712]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [12676712]: BUILD PASSED

[szkarpinski]

@banasraf @klecki I've successfully rebased on #5298, but with one non-trivial change. It turned out that self.counter_ was reset to 0 by Paddle iterator, overwriting the value restored in _DaliBaseIterator.__init__. I removed this line.

When Paddle iterator was added (#1371), there was no _DaliBaseIterator, so the iterator needed to handle the counter on its own. When _DaliBaseIterator was created (#1831), self.counter_ = 0 was duplicated. It was redundant in Paddle, but didn't cause problems until now.

Can I ask for re-approval for that?