Add support for much faster hash algorithms (e.g. xxHash)

[rico-chet]

Currently, SCons only supports hash algorithms which are provided by the hashlib, but those are rather slow (in the ballpark of 1-2 GB/s).

There are much faster alternatives available, e.g. xxHash which runs at the speed of RAM (an order of magnitude faster than MD5, up to ~30 times, according to xxHash developers). xxHash is readily available as a C and Python3 library via Debian repositories, so e.g. Ubuntu/Debian users can install it with ease. The hash is not cryptographic, but it's acceptable according to the comments I found in SCons code.

[bdbaddog]

I've moved this to a discussion.

If you've read the docs, SCons thus far hasn't required any other packages besides what's in the python distribution to run.
Thus the current limitation.

Feel free to develop this in the scons-contrib repo if you'd like to see if you can get this working with SCons.

[rico-chet]

I get that SCons shouldn't impose such an exotic library as a dependency on its users, but what about it being a soft dependency, i.e. only be used when present? It would correspond to a suggested package in Debian parlance.

In the code I saw, there was a certain level of coupling to the hashlib which I didn't immediately grasped, so I didn't pursue it, in favor of another topics (which are done now). I'll see if I can find some time to bring it on.

[mwichmann]

We'd need to get more clever... what if several devs share a build tree or a cachedir, and one of them installs xxhash in their environment. Now everything she writes will be using xxhash signatures, and the other devs will have their builds break because their run of scons won't recognize those. At the least, we'd want to switch to a multihash scheme so scons could recognize the kind of hash and emit a sane message if it's one it can't deal with at that time.

[rico-chet]

Wouldn't such issues arise today when state is shared between FIPS-compliant (MD5) and FIPS-incompliant (SHA*) environments? Or even when a group member explicily chooses the hash algorithm (e.g. because they want to use hardware-accelerated SHA on their new machine)?

[rico-chet]

A rough sketch can be found here https://github.com/rico-chet/scons/commits/add-xxhash-support.
It prioritizes xxhash when available, saving half a second per GB of input+output data (without a cache dir, on my machine).

[rico-chet]

This might start with providing boilerplate code which users can integrate into their SConstruct files to boost their builds. It's not easy though because unlike documented, SCons.Util.hashes.set_hash_format() does not support hashlib_used value other than hashlib.

E.g. this should work (but doesn't):

import SCons.Util as scons_util
import xxhash

scons_util.set_hash_format('xxh128', xxhash)

https://scons.org/doc/latest/HTML/scons-api/SCons.Util#SCons.Util.hashes.set_hash_format

[mwichmann]

you have to get it into ALLOWED_HASH_FORMATS somehow