How to stream a ZIP file containing multiple JSON files?

[kk-min]

Hello, thank you for this library. I have a particular use case where I have a zip file that contains different JSON files (labeled in order) that may need to be read. I have attached a sample zip file with 2 of such JSON files inside.

I am wondering if the following can be done via a chain:

1. Unzip and just stream the files in order based on their name (e.g. term_bank_0 -> term_bank_1 -> ... etc.)
2. If I know the JSON file index before hand (e.g. I want term_bank_74), is it possible to specify this and do unzipping + streaming only for that file?

For the above two, I actually only need the object with a particular key value. I have the following code that does this (e.g. get the index 9 object in the JSON file) with an unzipped file:

import chain from 'stream-chain';
import parser from 'stream-json';
import s from 'stream-json/streamers/StreamArray.js';
import fs from 'fs';

const { streamArray } = s;

function getWordPipeline(key) {
  const pipeline = new chain([
    fs.createReadStream('./term_bank_0.json'),
    parser(),
    streamArray(),
    data => {
      if (data.key === key) {
        return data.value;
      }
    }
  ])
  return pipeline;
}

const pipeline = getWordPipeline(9);
let wordArray = [];
for await (const value of pipeline) {
  wordArray.push(value);
}
console.log(wordArray)

So I am wondering how I can do this on a zipped file scenario. Thank you!
term_bank.zip

[kk-min]

I think I've got it figured out, posting my solution here in case anyone comes across similar needs. I confused the gzip format with the zip archive format, and it seems they are slightly different. The zlib library in Node is for dealing with the gzip compression/decompression, and hence cannot be used for zip archives. Use another library such as yauzl or yauzl-promise for decompressing zip archives.

Here I used yauzl-promise:

async function getPipeline(zipPath, fileName) {
  const zip = await yauzl.open(zipPath);
  try {
    for await (const entry of zip) {
      if (entry.filename === fileName) {
        const readStream = await entry.openReadStream();
        console.log(fileName, 'selected')
        const pipeline = new chain([
          readStream,
          parser(),
          streamArray(),
          data => {
            // Use data
          }
        ]);
        return pipeline;
      }
    }
  } catch (err) {
    console.error('Error processing ZIP file:', err);
  }
  return null;
}

// const pipeline = await getPipeline(zipPath, fileName);
// pipeline.on( ...

I also wondered if the zip itself can be streamed, but it turns out there are complications with the ZIP archive format and it may result in corrupted data according to the yauzl lib documentation.

[uhop]

Unless I am dealing with gzip/brotli, supported by Node directly out of the box, my first reaction is to use *nix pipes. I write a utility that reads from stdin and do the rest in bash or a shell of your choice. Non-verified example:

unzip -p data.zip datum1.json | node my-json-utility.js

If your utility assumes that a JSON file is read from process.stdin, it should work properly: no intermediate unpacking of huge files, everything is on the fly. If you write results, e.g., a JSON file to process.stdout, you can continue using *nix utilities to finish up the processing, e.g., pack it up in a tar file, or send it to a different computer using a network pipe.