Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Efficient attachment downloading for mailbox export #8069

Open
4 tasks done
Tracked by #1292
charlag opened this issue Dec 2, 2024 · 0 comments · Fixed by #8156
Open
4 tasks done
Tracked by #1292

Efficient attachment downloading for mailbox export #8069

charlag opened this issue Dec 2, 2024 · 0 comments · Fixed by #8156
Assignees
@paw-hub @BijinDev
Labels
feature: mailbox export goal: email for business

Comments

@charlag
Copy link
Contributor

charlag commented Dec 2, 2024 •
edited by paw-hub
Loading

We should optimize attachment data downloading for mailbox export.

Naive approach of downloading each blob one by one leads to N+2 requests.

There are two possible optimizations we can apply:

  • For attachments in our own archive utilize archive-scoped token. This gets rid of per-blob token requests.
  • For attachments in other archives utilize a token that works for multiple instances.
  • Download multiple blobs of the same attachment together if the size permits it
  • Download multiple blobs of different attachments if they belong to the same archive and the size permits it

Context

Downloading blobs requires requesting a blob access token first. The tokens can be scoped to individual instances or to the whole archives.

Archives are associated with a group. Sent attachments and externally-received attachments will belong to the mail group archives, while confidentially-received attachments do belong to the archive associated with the sender.

At the moment when downloading raw blobs (like for attachments) client does request tokens for each instance.

Estimated: 24h
Time Taken: 7h + 4h + 5h
@charlag charlag mentioned this issue Dec 2, 2024
Open
12 tasks
paw-hub added a commit that referenced this issue Dec 17, 2024
@paw-hub @BijinDev
Try to split requests by archive
6b16bf4 
Closes #8069

Co-authored-by: BijinDev <bir@tutao.de>
@paw-hub paw-hub mentioned this issue Dec 17, 2024
Merged
@paw-hub paw-hub linked a pull request Dec 17, 2024 that will close this issue
Efficient mail exporting #8156
Merged
BijinDev added a commit that referenced this issue Dec 18, 2024
@BijinDev @charlag
Use archive tokens when downloading blobs when possible
d103a59 
close : #8069

Co-authored-by: ivk <ivk@tutao.de>
BijinDev added a commit that referenced this issue Dec 19, 2024
@BijinDev @charlag
Use archive tokens when downloading blobs when possible
3a271e6 
close : #8069

Co-authored-by: ivk <ivk@tutao.de>
BijinDev added a commit that referenced this issue Dec 19, 2024
@paw-hub @charlag @BijinDev
Optimize blob downloading
c9a2d5e 
Use archive tokens when downloading blobs when possible

Try to split requests by archive

Closes #8069

Co-authored-by: ivk <ivk@tutao.de>
Co-authored-by: BijinDev <bir@tutao.de>
BijinDev added a commit that referenced this issue Dec 19, 2024
@BijinDev @charlag
Request multiple blobs of the same instance in one request
f93d7c1 
#8069

Co-authored-by: ivk <ivk@tutao.de>
BijinDev added a commit that referenced this issue Dec 19, 2024
@BijinDev @charlag
Request multiple blobs of different instances in same request
67197cc 
#8069

Co-authored-by: ivk <ivk@tutao.de>
paw-hub added a commit that referenced this issue Dec 20, 2024
@paw-hub @charlag @BijinDev
Optimize blob downloading
4d7086c 
Use archive tokens when downloading blobs when possible

Try to split requests by archive

Closes #8069

Co-authored-by: ivk <ivk@tutao.de>
Co-authored-by: BijinDev <bir@tutao.de>
paw-hub pushed a commit that referenced this issue Dec 20, 2024
@BijinDev @charlag @paw-hub
Request multiple blobs of the same instance in one request
5c02afc 
#8069

Co-authored-by: ivk <ivk@tutao.de>
paw-hub pushed a commit that referenced this issue Dec 20, 2024
@BijinDev @charlag @paw-hub
Request multiple blobs of different instances in same request
153329f 
#8069

Co-authored-by: ivk <ivk@tutao.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@paw-hub paw-hub

@BijinDev BijinDev

Labels
feature: mailbox export goal: email for business
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Efficient mail exporting tutao/tutanota
3 participants
@charlag @paw-hub @BijinDev