Skip to content

Commit

Permalink
1593 populate existing asset storage image (#1619)
Browse files Browse the repository at this point in the history
* black and flake 8 updates

* add ordering on qs

* black changes

* back changes

* fix bad conflict resolution change

* cleanup comments - switch back to select related

* increase limit and chunk since 20k chunked 2k ran in a minute
  • Loading branch information
jkueloc authored Feb 24, 2022
1 parent 65c337d commit 4823bf2
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions concordia/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@ def populate_storage_image_values(asset_qs=None):
asset_qs = (
Asset.objects.filter(storage_image__isnull=True)
.order_by("id")
.select_related("item__project__campaign")[:20000]
.select_related("item__project__campaign")[:50000]
)

logger.debug("Start storage image chunking")
Expand All @@ -374,7 +374,7 @@ def populate_storage_image_values(asset_qs=None):
# We'll process assets in chunks using an iterator to avoid saving objects
# which will never be used again in memory. We will build the S3 relative key for
# each existing asset and pass them to bulk_update() to be saved in a single query.
for asset_chunk in chunked(asset_qs.iterator(), 2000):
for asset_chunk in chunked(asset_qs.iterator(), 3000):

for asset in asset_chunk:
asset.storage_image = "/".join(
Expand Down

0 comments on commit 4823bf2

Please sign in to comment.