Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gzip all plain-text files #102

Open
niemasd opened this issue May 24, 2021 · 2 comments
Open

Gzip all plain-text files #102

niemasd opened this issue May 24, 2021 · 2 comments
Labels

Comments

@niemasd
Copy link
Contributor

niemasd commented May 24, 2021

It would be good if we compressed all plain-text files using gzip (any compression would probably be fine, but I like gzip because it's ready-to-go on many systems and tools, and it probably works well-enough). Any commands that require an unzipped file can simply be rewritten to use a FIFO instead using process subtitution, e.g. the following:

my_program data.txt

can be written as follows to use a FIFO:

my_program <(zcat data.txt.gz)
@niemasd
Copy link
Contributor Author

niemasd commented May 24, 2021

Also, pro-tip: you can use pigz to compress files using multithreading to speed things up, e.g. the following (-9 does level 9 compression, and -p 8 uses 8 threads; you can use however many you want):

pigz -9 -p 8 data.txt

@rob-knight
Copy link
Collaborator

This would be useful to do in general.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants