Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1593 populate existing asset storage image #1619

Merged
merged 10 commits into from
Feb 24, 2022
Merged

1593 populate existing asset storage image #1619

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
da71a6f
black and flake 8 updates
jkueloc Feb 8, 2022
4537c2e
add ordering on qs
jkueloc Feb 9, 2022
25df693
black changes
jkueloc Feb 23, 2022
2a7e717
back changes
jkueloc Feb 23, 2022
64e0fea
Merge branch 'main' into 1593-populate-existing-asset-storage-image
jkueloc Feb 23, 2022
c76f65f
fix bad conflict resolution change
jkueloc Feb 23, 2022
33f7efa
cleanup comments - switch back to select related
jkueloc Feb 23, 2022
a7c69c9
Merge branch 'main' into 1593-populate-existing-asset-storage-image
jkueloc Feb 23, 2022
f96a38e
increase limit and chunk since 20k chunked 2k ran in a minute
jkueloc Feb 24, 2022
994e18b
Merge branch 'main' into 1593-populate-existing-asset-storage-image
jkueloc Feb 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions concordia/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@ def populate_storage_image_values(asset_qs=None):
asset_qs = (
Asset.objects.filter(storage_image__isnull=True)
.order_by("id")
.select_related("item__project__campaign")[:20000]
.select_related("item__project__campaign")[:50000]
)

logger.debug("Start storage image chunking")
Expand All @@ -374,7 +374,7 @@ def populate_storage_image_values(asset_qs=None):
# We'll process assets in chunks using an iterator to avoid saving objects
# which will never be used again in memory. We will build the S3 relative key for
# each existing asset and pass them to bulk_update() to be saved in a single query.
for asset_chunk in chunked(asset_qs.iterator(), 2000):
for asset_chunk in chunked(asset_qs.iterator(), 3000):

for asset in asset_chunk:
asset.storage_image = "/".join(
Expand Down