MDencode is a GO file signature and C++ command line file digital signature encoder with modular floors.
This program is written in GO and uses an interface for programmable formatters.
MDencode allows for the creation of sqlite3 db file signatures and the first random file signatures and signature decryption.
This program can generate a file signature at the file or block level and format and write the output in multiple formats.
It can also write to an sqlite db. The mdencode sqlite db files contain a table schema and can be exported into multiple formats.
MDencode also contains an mdzip and mdunzip utility for encoding and decoding files with modular floors and signatures.
- Versioned Cryptographic File Signatures
- Optional File Crypto Signatures
- Optional N-Bit Sized File Block Modular Floors
- Optional N-Byte Block Crypto Signatures
- SQLite3 DB Crypto Signature Files
- Plugin Formatters
- Multiple Signatures for Files and File Blocks
- The "First" Random File Signature or Block signatures
- An example of a parallel signature modulus scan with an encoder and decoder (mdzip and mdunzip)
This is my block chopper.
This program uses a modular floor paired with a package of digital signatures. This makes the digital signature more unique. A modular floor consists of a polynomial which is the modular exponent plus n times the modulus plus the remainder. It is possible to use a modular scan to find matching byte blocks associated with the digital signature package. A digital signature package or group can consist of 1 or more block and file signatures. A modular floor is generaly computed at the block level for a file byte block and utilizes a big integer.
The modulus scan polynomial is calculated as follows.
modremainder = fileblockbigint % modulus
fileblockbigint = modulus * n + modremainder where n equals 0 to m
fileblockbigint = modulus ^ (exponent p) + modulus * n + modremainder where p is an optional power and n equals 0 to m
This is the modular floor equation for two forms. One is the exponential floor while the other contains a multiple of the modular floor plus the remainder. A modulus should be considered part of the file signature subset. A modulus is a digital signature.
This is an example of the modular floor using the C++ test program.
The program mdtest is using a random 14 byte array and then running a modulus scan to find the corresponding file block associated with a digital signature.
It uses a 32 bit modulus and calculates modulus exponent floor or ceiling and then creates a fast64 hash. A modulus scan splits up the parallel search on 32 threads.
It calculated and found the 14 byte block (13 bytes compressed) in 4 hours on a Ryzen 3950x. It compresses the 14 byte block to 13 bytes and decompresses it to 14 bytes.
13/14 Modulus Encoding
- 8 bytes for the 64-bit fasthash 64 - (It can use any number of 64 bit hashes)
- 4 bytes for the 32-bit modulus
- 1 byte for the modulus exponent
C++ Test program
$GOPATH/github.com/singularian/mdencode/code/testdecode_cpp/mdtest
./mdtest --mod=32 --threads=32 --hl 4 --hex=000000001211211111111122FFFC
hash values 4
Start Time Sun Nov 15 17:52:53 2020
Block Size 14
Random Byteblock 000000001211211111111122FFFC
Random Byteblock Hex 00 00 00 00 12 11 21 11 11 11 11 22 FF FC
Random Byteblock Int 0 0 0 0 18 17 33 17 17 17 17 34 255 252
Random Byteblock Bigint 85318574045349531549692
Modulus Size 32
Modulus Bigint 4294967296
Modulus Remainder 287506428
Modulus 2 ^ Exponent 76
Modulus Mod ^ Exponent 2
Block Signatures fast64 5986481724077706591
Thread Count 32
Logging false
Found Match
Elapsed Time (s) 14601.1 = 4.05 hours
Modulus Scan thread 17 and Random byteblock match
0 0 0 0 18 17 33 17 17 17 17 34 255 252
0 0 0 0 18 17 33 17 17 17 17 34 255 252
Second modulus scan decompression example with a larger block running on a ryzen 3950x.
./mdtest --mod=32 --threads=32 --hl 4 --hex=0000000016412161123F1822FFFC --log=true
hash values 4
Start Time Tue Nov 17 21:44:08 2020
Block Size 14
Random Byteblock 0000000016412161123F1822FFFC
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Random Byteblock Hex 00 00 00 00 16 41 21 61 12 3F 18 22 FF FC
Random Byteblock Int 0 0 0 0 22 65 33 97 18 63 24 34 255 252
Random Byteblock Bigint 105093506211661505298428
Modulus Size 32
Modulus Bigint 4294967296
Modulus Remainder 404946940
Modulus 2 ^ Exponent 76
Modulus Mod ^ Exponent 2
Block Signatures fast64 10504795572753995326
Thread Count 32
Logging true
Running decode modscan
Found Match
Elapsed Time (s) 43036 = 12 hours
Modulus Scan thread 31 and Random byteblock match
0 0 0 0 22 65 33 97 18 63 24 34 255 252
0 0 0 0 22 65 33 97 18 63 24 34 255 252
Another modulus scan decompression example with a larger byte block
./mdtest --mod=32 --threads=32 --hl 4 --hex=0000002020412161123F1822FFFC --log=true
hash values 4
Start Time Sat Nov 21 19:59:59 2020
Block Size 14
Random Byteblock 0000002020412161123F1822FFFC
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Random Byteblock Hex 00 00 00 20 20 41 21 61 12 3F 18 22 FF FC
Random Byteblock Int 0 0 0 32 32 65 33 97 18 63 24 34 255 252
Random Byteblock Bigint 38837943398708491548033020
Modulus Size 32
Modulus Bigint 4294967296
Modulus Remainder 404946940
Modulus 2 ^ Exponent 85
Modulus Mod ^ Exponent 2
Block Signatures fast64 2256604700901092545
Thread Count 32
Logging true
Running decode modscan
Found Match
Elapsed Time (s) 220182.811267 - 61.16 hours
Modulus Scan thread 31 and Random byteblock match
0 0 0 32 32 65 33 97 18 63 24 34 255 252
0 0 0 32 32 65 33 97 18 63 24 34 255 252
This is the Sip47 modulus scan decompression example running on a ryzen 3950x. It could use a 4 to 7 bit bitstream to encode the modulus exponent.
./mdtest --mod=32 --hl 30 --randbh --log --hex=0066BDF2F888F000FF00 --block=10
Start Time Sun Oct 31 19:06:06 2021
Block Size 10
Random Byteblock 0066BDF2F888F000FF00
1 2 3 4 5 6 7 8 9 10
Random Byteblock Hex 00 66 BD F2 F8 88 F0 00 FF 00
Random Byteblock Int 0 102 189 242 248 136 240 0 255 0
Random Byteblock Bigint 1895255171002932461312
Modulus Size 32
Modulus Bigint 4294967295
Modulus Remainder 2918447087
Modulus 2 ^ Exponent 70
Modulus Mod ^ Exponent 2
Block Signatures sip40 5246BACF79
Blockkeylist sip40 keys 8E197EC7727CECD973118AEE2E72BE65
Thread Count 32
Logging true
Hash Block Vector
Number Hash ID Hash Name Blocksize Blockkeysize
1 30 sip40 5 16
Total 5 16
Running decode modscan
Found Match
Elapsed Time (s) 2448.636829 = 40.8 minutes
Modulus Scan thread 15 and Random byteblock match
0 102 189 242 248 136 240 0 255 0
0 102 189 242 248 136 240 0 255 0
There are currently two test modulus scan programs. These include
- GO test modulus scan
$GOPATH/github.com/singularian/mdencode/code/testdecode_go/mdtestuses GO routines to run the modulus scan for a signature on a byte block - C++ test modulus scan
$GOPATH/github.com/singularian/mdencode/code/testdecode_cpp/mdtestuses C++ threads to run the modulus scan
MDZip and MDUnzip also use a parallel modulus scan with the GO or C++ core.
These are examples of a parallel modulus scan for mdzip C++ with Siphash 40.
- Siphash 40 MDzip With No Increment Key
- Siphash 40 MDzip With Decrement Key
- Siphash 40 MDzip With Increment Key
- Siphash 47 MDzip With No Increment Key
mdzip is the C++ and GO command line utility can compress a file into a md binary byte block files. mdzip takes an optional input key for the 32 byte Highway Hash 64 and Highway Hash 128 and Highway Hash 256. It can also take input keylist parameters for other signature keys.
mdunzip will decompress the input signature binary block file to an output file.
mdunzip has a context hash list for each decode thread. mdzip can use multiple block signatures and mdunzip will use the specified signatures to unzip the file.
Proccessing power limits the size of the input block.
MDZip Features
- User Specified Modulus Size
- User Specified Block Size
- File Hash Signatures
- Block Hash Signatures
- Optional Signature Keys
- 32 Byte Highway Hash Signature Keys which change the signature
MDUnzip Features
- Multithreaded Parallel Modulus Scan
- Decrypts an MDZip file to an output file
- File Signature Post Validation
- Uses the MDZip Signature Keys to change the output file signature Modulus Scan
These are some Asciinema Video Examples
MDZip Example
MDZip Example 2
MDZip Highway Hash Key Example
These are a brief list of collision management in mdzip and mdunzip.
- Using File Level Signatures to validate Block Level Signatures.
- Using File Level Signatures to differentiate between File Block Signature Collisions.
- Signature Switching. If one signature or signature group has a collsion switch to another signature or signature group.
- Change the signature key. An example is Highway Hash with a 256 bit key which can be changed to different keys.
- Use Optional Block Groups as a wrapper around signature blocks with collisions or add additional signatures.
- Use Optional File Block Collision Numbers. Collision numbers can designate the correct match.
- Use Block Signature Pairs with Modulus / Modular Floors.
- Using Modular floors to skip numbers
This is a guide for building MDencode on Linux.
This is a Docker Hub Docker File for MDencode
docker pull annatarsol/mdencode
docker run -it annatarsol/mdencode/latest /bin/bash
- BTC:
32L3gyJdkiFwgJ72rsSzbCBjTjVFxJLHPm - ETH:
0x137eB09cF2e199dA2Aa79607f96719be1491a312 - LTC:
MJh8XpUQ5wm8x1HCoQ7dewNqhrLfZZc31x
MDEncode was developed by Scott Ross on a Ryzen 2700x and Ryzen 3950x CPU.