Skip to content

Commit

Permalink
Use -m flag to run gsutil cp in parallel for longer copies
Browse files Browse the repository at this point in the history
Got this helpful message when running a copy command that timed out:

```
NOTE: You are performing a sequence of gsutil operations that may
run significantly faster if you instead use gsutil -m cp ... Please
see the -m section under "gsutil help options" for further information
about when gsutil -m can be advantageous.
```
PiperOrigin-RevId: 699205081
  • Loading branch information
Zach Howell authored and copybara-github committed Nov 22, 2024
1 parent 5a13c0f commit 7bf06f7
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions perfkitbenchmarker/providers/gcp/gcs.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,9 @@ def Copy(self, src_url, dst_url, recursive=False, timeout=None):
timeout: The timeout for the copy command.
"""
cmd = ['gsutil', 'cp']
if recursive or timeout is not None:
# -m runs in parallel, which is faster.
cmd = ['gsutil', '-m', 'cp']
if recursive:
cmd += ['-r']
cmd += [src_url, dst_url]
Expand Down

0 comments on commit 7bf06f7

Please sign in to comment.