Bypass server upload large file is slow because of generating index file

[wwwjn]

According to #3370 and test result, bypass server upload need a lot of time to calculate the index file.

[epicfaace]

Ashwin made a POC here: #4212

But we depend on fixing pauldmccarthy/indexed_gzip#102 and mxmlnkn/ratarmount#95 before it can work.

[epicfaace]

Ashwin: get pauldmccarthy/indexed_gzip#103 to work, Ashwin will also publish a fork of indexed_gzip (indexed_gzip_fileobj_fork_epicfaace)

Jiani: fixing up #4212. Maybe try reverting to the simpler peek solution in ac5ed66. Once Ashwin publishes indexed_gzip fork, change it to use the indexed_gzip fork

import indexed_gzip_fileobj_fork_epicfaace as indexed_gzip

[wwwjn]

POC to fix this problem is here: Using peek() instead of read() in SQLiteIndexedTar for gz files

However, the current Indexed_gzip implementation will not call peek(), instead it calls read(). Simple script to reproduce this: epicfaace/test-repro#12

[wwwjn]

There still some bugs in indexed_gzip, which blocks this issue: pauldmccarthy/indexed_gzip#106

[wwwjn]

Update: Related issue for indexed_gizp github repo: pauldmccarthy/indexed_gzip#107

[leilenah]

will sync with ashwin

[epicfaace]

Potential solution:

1. Skip calling peek() from ratarmount (ac5ed66#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbeR46). Assume it's always a gzip file.
2. Fix issues around building indexes (on this line of code ac5ed66#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbeR1439-R1441). 2 options:
   1. Fix For unseekable files, can't build full index by reading the entire file pauldmccarthy/indexed_gzip#107 so that if you read an entire unseekable file, it should build an index for it.
   2. Maybe we can just change ratarmount here ac5ed66#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbeR1439-R1441 to just call build_full_index.

[wwwjn]

Potential solution:

1. Skip calling peek() from ratarmount (ac5ed66#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbeR46). Assume it's always a gzip file.

2. Fix issues around building indexes (on this line of code ac5ed66#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbeR1439-R1441). 2 options:

   1. Fix For unseekable files, can't build full index by reading the entire file pauldmccarthy/indexed_gzip#107 so that if you read an entire unseekable file, it should build an index for it.
   2. Maybe we can just change ratarmount here ac5ed66#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbeR1439-R1441 to just call build_full_index.

Did more tests based on this POC. And test the above potential solution. However, it does not works for me.

For step2 & Option 1, I found an interesting observation: By reading through the entire unseekable file to build index, this only works for small files (smaller than 10 MB). In other words, the solution in this commit only works for small files. For those large files, it will raise ZRAN_READ_FAIL error when reading through the file (Line 753 or line 1439).

For step 2 & Option 2, build_full_index is still buggy for me.

[wwwjn]

Solution:

1. Skip peek() from ratarmount
2. Fix issues around building indexes. Use build full_index() to build index from GzipStream().

• Here is a test script which shows that indexed_gzip works well with GzipStream().(https://github.com/pauldmccarthy/indexed_gzip/pull/110/files#diff-d28f4af2ce24a9bf1c9fd3722ab5a0f506109a450ff6b5b1f141e676c31fc62d)
• Here is a test script to fix bugs and make ratarmount works with unseekable file.(https://github.com/codalab/codalab-worksheets/pull/4347/files#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbeR1345-R1348)

[wwwjn]

Fix this, test on largest bundle uploading to dev (158GB).

[wwwjn]

1. pass the test and deploy
2. upload a large bundle locally, stress test locally (test a lot of small files)

[percyliang]

Can we check that we can upload a 150+GB file and close this?