fix: upload non-conflicting files for sharded checkpointing [MD-298]

[azhou-determined]

Ticket

Description

make sharded checkpoint uploads with store_path check for file conflicts across workers before upload.

Test Plan

run a distributed core API example and write sharded checkpoints across all workers:

import logging
import pathlib

import determined as det


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format=det.LOG_FORMAT)
    with det.core.init(
        distributed=det.core.DistributedContext.from_torch_distributed()
    ) as core_context:
        for i in range(100):
            if i % 10 != 0:
                continue
            with core_context.checkpoint.store_path(
                metadata={"steps_completed": i}, shard=True
            ) as (ckpt_path, uuid):
                pathlib.Path(ckpt_path / f"test_file-{i}").touch()
                logging.info(
                    f"Saving {ckpt_path / 'test_file'} to checkpoint {uuid} at {ckpt_path}"
                )

name: core-api-test
entrypoint: python3 -m determined.launch.torch_distributed python3 core_api.py

searcher:
   name: single
   metric: x
   max_length: 1

resources:
   slots_per_trial: 2
max_restarts: 0

Checklist

• Changes have been manually QA'd
• New features have been approved by the corresponding PM
• User-facing API changes have the "User-facing API Change" label
• Release notes have been added as a separate file under docs/release-notes/
  See Release Note for details.
• Licenses have been included for new code which was copied and/or modified from any external code

[netlify]

✅ Deploy Preview for determined-ui ready!

Name	Link
🔨 Latest commit	93f1863
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/669080630b56410008c86ad4
😎 Deploy Preview	https://deploy-preview-9598--determined-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.02%. Comparing base (7fab87b) to head (93f1863).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #9598   +/-   ##
=======================================
  Coverage   53.01%   53.02%           
=======================================
  Files        1255     1255           
  Lines      152884   152960   +76     
  Branches     3233     3234    +1     
=======================================
+ Hits        81053    81106   +53     
- Misses      71680    71703   +23     
  Partials      151      151           

Flag	Coverage Δ
backend	44.19% <ø> (-0.05%)	⬇️
harness	72.85% <100.00%> (+0.08%)	⬆️
web	51.37% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
harness/determined/common/storage/base.py	93.38% <100.00%> (ø)
harness/determined/common/storage/cloud.py	100.00% <100.00%> (ø)
harness/determined/common/storage/shared.py	82.22% <100.00%> (ø)
harness/determined/core/_checkpoint.py	95.45% <100.00%> (+0.99%)	⬆️
harness/tests/core/test_checkpoint.py	100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes

[ioga]

lgtm