Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use faster zstd compression settings #455

Open
skwerlman opened this issue Oct 31, 2024 · 2 comments
Open

Use faster zstd compression settings #455

skwerlman opened this issue Oct 31, 2024 · 2 comments

Comments

@skwerlman
Copy link

Currently, DKMS uses compression level 19 when ZSTD is selected as the compression method. However, this does not provide a significant benefit in terms of compressed size when compared to lower levels, especially when considering the time it takes to compress them:

As an example, here are the compression stats for nvidia-dkms on Arch Linux for compression levels -19 and -1.

with -19:

 ╭─sk@a5 in ~ 
 ╰─λ zstd -l /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia*.zst  
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0    58.1 KiB       289 KiB  4.976  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-drm.ko.zst
     1      0    49.7 MiB      96.8 MiB  1.947  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia.ko.zst
     1      0     685 KiB      2.72 MiB  4.072  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-modeset.ko.zst
     1      0    2.23 KiB      8.54 KiB  3.824  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-peermem.ko.zst
     1      0     839 KiB      6.22 MiB  7.599  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-uvm.ko.zst
----------------------------------------------------------------- 
     5      0    51.2 MiB       106 MiB  2.068  XXH64  5 files

with -1:

 ╭─sk@a5 in ~ 
 ╰─λ zstd -l /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia*.zst
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0    73.4 KiB       289 KiB  3.940  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-drm.ko.zst
     1      0    53.5 MiB      96.8 MiB  1.807  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia.ko.zst
     1      0     825 KiB      2.72 MiB  3.383  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-modeset.ko.zst
     1      0    2.46 KiB      8.54 KiB  3.477  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-peermem.ko.zst
     1      0    1.05 MiB      6.22 MiB  5.954  XXH64  /var/lib/dkms/nvidia/560.35.03/6.11.5-zen1-1.1-zen/x86_64/module/nvidia-uvm.ko.zst
----------------------------------------------------------------- 
     5      0    55.5 MiB       106 MiB  1.911  XXH64  5 files

IMO, a saving of only 4.2MiB is not really worth it when the compression takes under a second for -1 but nearly 2 minutes(!) for -19.1

This situation could be improved with either a lower compression level or by making the compression settings configurable (with some documentation of the compatibility concerns noted in #386)

Footnotes

  1. Times are for my personal computer, with a threadripper 2950x @ 3.5GHz with PBO enabled, 64GB of 2933MHz ECC memory, and an nvme drive.

@scaronni
Copy link
Collaborator

scaronni commented Nov 4, 2024

I think we could remove the parameter completely and just use the default (-3). Probably we can also do the same for xz and gzip.

@scaronni
Copy link
Collaborator

scaronni commented Nov 4, 2024

Care to make a merge request? Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants