What is the fastest way to create large Zip?

[jobisoft]

I am a Thunderbird WebExtension developer and I am using your library to import/export messages as a zip file.

For import, I use the FS() method, because I do not have to load the entire file into memory, just to know what is inside, which is a huge speed boost. I assume that is the fastest way?

For export, I currently also use the FS() method:

    let zipFs = new fs.FS();
    ...
    zipFs.addUint8Array(filename, mboxBytes, { useWebWorkers: false }); // webWorkers cause a CSP violation 
    ...
    let zipDir = zipFs.addDirectory(...);
    ...
    zipDir.addUint8Array(filename, mboxBytes, { useWebWorkers: false }); // webWorkers cause a CSP violation 
    ...
    let blob = await zipFs.exportBlob({ level });

Using addUint8Array is faster than using addBlob() (probably because of the extra overhead to create the blob).

Is this the fastest way? Is using FS() for writes the best option? Asking because the zipFs.exportBlob() also needs a considerable amount of time.

I also would be interested if you have experiences with streaming the zip. What I can see is that downloading large zips to the local filesystem needs a lot of free system memory to hold the entire zip. The WebExtension downloads API however can download data in chunks: large files are not downloaded entirely and then written to the local filesystem, but written as soon as the data chunk is downloaded (there is a "parts" file in your download folder while downloading).

It would be great if we could send chunks of the zip as we add files to it, so exporting 5 GB of messages would not need 5 GB of free system memory. Would you have any pointers for me regarding this topic?

[gildas-lormeau]

The best way to improve CPU performance is to use web workers. In the event of a CSP violation, you should be able to use z-worker.js instead of the embedded script. You'll have to call zip.configure to pass the path to the script to zip.js, like this:

zip.configure({
  deflate: ["./path/to/z-worker.js"],
  inflate: ["./path/to/z-worker.js"]
});
...

I can confirm that FS#export* methods should always be the most efficient in terms of performance. I also confirm that Uint8Array are faster but are limited to 4GB (or maybe less) of data.

Internally, zip.js relies entirely on Web Streams (e.g. ReadableStream and WritableStream). If there are APIs available to take advantage of this, I recommend you use them. They should help to consume as little RAM as possible.

Otherwise, you can also write your own Reader and/or Writer classes to process data in blocks. Here is a test that shows how to write such custom classes: https://github.com/gildas-lormeau/zip.js/blob/master/tests/all/test-custom-io.js.

[jobisoft]

Thanks for your swift response! I hope you do not mind me asking follow-up questions. :-)

I tried to use configure:

import { fs, configure } from "./modules/zip.js/index.js";

// Define paths to overcome CSP violations.
// See: https://github.com/gildas-lormeau/zip.js/issues/449#issuecomment-1704312167
configure({
  deflate: ["./modules/zip.js/lib/z-worker.js"],
  inflate: ["./modules/zip.js/lib/z-worker.js"]
});

But calling

zipFs.addUint8Array(filename, mboxBytes, { useWebWorkers: true });

still causes a CSP violation:

Content-Security-Policy: The page’s settings blocked the loading of a resource at blob:moz-extension://46cc048d-e9ae-47b2-890f-557f64f9c6b6/08253163-b1fc-4649-b151-649eeef5f210 (“script-src”).

Did I implement your hint correctly? It could very well be that this is a limitation of the WebExtension framework itself, but before digging into that rabbit hole, I wanted to make sure I used your hint correctly.

I also confirm that Uint8Array are faster but are limited to 4GB (or maybe less) of data.

Is that a limitation of the UInt8Array itself, or of your addUint8Array function? When creating Blobs, I learned that it accepts a sequence of UInt8Arrays, but each sequence element array may not be larger then 2GB, otherwise new Blob() will throw. But honoring that, I was able to create really large blobs, for example a 10GB test file.

So if the "file" I want to add to a zip is larger than 4GB, I should be able to split it up into multiple UInt8Arrays not larger than 2GB, create a blob and then use your addBlob() to add it? I will try that with code looking like that:

  let bytes = await getMboxBytes(exportItem); // returns UInt8Array
  let buffer = bytes.buffer;
  // The Blob constructor accepts a sequence and each element may not exceed 2GB,
  // split the data into smaller chunks.
  let pos = 0;
  let chunk = 1024 * 1024 * 1024;
  let sequence = [];
  while (pos + chunk <= bytes.byteLength) {
    sequence.push(new Uint8Array(buffer, pos, chunk));
    pos += chunk;
  }
  sequence.push(new Uint8Array(buffer, pos));
  let blob = new Blob(sequence, { type: "text/plain" });

I will analyze the streaming pointers, thanks for those!

[gildas-lormeau]

You have to use the z-worker file located in the /dist/ folder, not the /lib/ folder. To circumvent the CSP issue, you can add the path to the z-worker.js file in the web_accessible_resources array in the manifest.json file.

Regarding the max. size of Uint8Arrayinstances, this is an "arbitrary" limitation of JS engines, AFAIK.

[jobisoft]

After browsing your source, I found this solution to be working:

configure({
  useWebWorkers: true,
  workerScripts: {
    deflate: [browser.runtime.getURL("./modules/zip.js/dist/z-worker.js")],
    inflate: [browser.runtime.getURL("./modules/zip.js/dist/z-worker.js")],
  }
});

Adding web_accessible_resources does not seem necessary.

My benchmarks indicate, that adding the UintArrays is slower with webWorkers enabled, but exporting the blob is faster:

WebWorkers	true	false
Adding UInt8Arrays	00:01:18	00:01:16
Exporting to blob	00:00:17	00:00:20
Total	00:01:35	00:01:36

Do you have any insides on this?

[gildas-lormeau]

Sorry, it's my fault, you have to set the option bufferedWrite to true when calling export methods, e.g. await fs.exportBlob({ bufferedWrite: true }). I thought this was the default value. BTW, you don't need to set useWebWorkers to true. That's the default value. You can also set keepOrder to false to optimize web workers usage.

For the record, I attached a demo extension if you want to do some perf tests.

On my machine (Macbook Pro M2), I get these results:

bufferedWrite: true, keepOrder: false => 1237 ms
bufferedWrite: true, keepOrder: true => 1447 ms
bufferedWrite: false => 3805 ms

extension-demo-zip.js.zip

[gildas-lormeau]

I've published the version 2.7.29 where bufferedWrite is set to true by default (keepOrder is still to trueby default though). You just need to update zip.js from NPM.

[jobisoft]

Thanks, but I am still digesting all these options and from my own tests, bufferedWrite:true is a lot slower than bufferedWrite:false, a real big lot.

I will need a few days before I can get back to this. I will provide a test extension for you to try out.

[gildas-lormeau]

Do you get similar results compared to mine when running my tests? (cf. the logs in the previous message)

[gildas-lormeau]

BTW, adding Uint8Array or Blob instances with the FS API (with add* methods) takes approx. 0ms because they are just copied by reference. Therefore, I don't think the time taken when Adding UInt8Arrays (in the table of your previous message) is related to zip.js.

[jobisoft]

BTW, adding Uint8Array or Blob instances with the FS API (with add* methods) takes approx. 0ms because they are just copied by reference. Therefore, I don't think the time taken when Adding UInt8Arrays (in the table of your previous message) is related to zip.js.

That is probably true, that benchmark step also includes generating the data, before adding it. I will clean up my code and make a few more tests and compare it with your example. But I probably need till the end of the week.

[gildas-lormeau]

No problem.

However, note that depending on how the data you want to zip is generated (e.g. the time taken when Adding UInt8Arrays), a solution to optimize this step might be to write your own Reader class.