fix: unable to find output_dir in multi-GPU during resume_from_checkp…

…oint check (#352)

* fix: output_dir doesn't exist during resume_from_checkpoint

Signed-off-by: Abhishek <maurya.abhishek@ibm.com>

* fix: fmt

Signed-off-by: Abhishek <maurya.abhishek@ibm.com>

---------

Signed-off-by: Abhishek <maurya.abhishek@ibm.com>
Signed-off-by: Anh Uong <anh.uong@ibm.com>

build/accelerate_launch.py

@@ -98,6 +98,8 @@ def main():
     #
     ##########
     output_dir = job_config.get("output_dir")
+    if not os.path.exists(output_dir):
+        os.makedirs(output_dir)
     try:
         # checkpoints outputted to tempdir, only final checkpoint copied to output dir
         launch_command(args)