diff --git a/README.md b/README.md index 0ffa5b52..bcc31b77 100644 --- a/README.md +++ b/README.md @@ -5,13 +5,24 @@ - Upstream for the Dæmon branch: https://github.com/DaemonEngine/crunch - Bug tracker for the Dæmon branch: https://github.com/DaemonEngine/crunch/issues +![Crunch](crunch.png) + +Target & compiler|Build status|Target & compiler|Build status +-|-|-|- +Linux amd64 GCC|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Linux%20amd64%20Clang)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master)|Windows amd64 MSVC|[![Build status](https://ci.appveyor.com/api/projects/status/github/DaemonEngine/crunch?banch=master&svg=true)](https://ci.appveyor.com/project/DolceTriade/crunch/history) +Linux amd64 Clang|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Linux%20amd64%20Clang)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master)|Windows i686 MSVC|[![Build status](https://ci.appveyor.com/api/projects/status/github/DaemonEngine/crunch?banch=master&svg=true)](https://ci.appveyor.com/project/DolceTriade/crunch/history) +Linux i686 GCC|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Linux%20i686%20GCC)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master)|Windows amd64 MinGW|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Windows%20amd64%20MinGW)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master) +Linux arm64 GCC|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Linux%20arm64%20GCC)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master)|Windows i686 MinGW|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Windows%20i686%20MinGW)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master) +Linux armhf GCC|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Linux%20armhf%20GCC)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master)|macOS amd64 AppleClang|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20macOS%20amd64%20AppleClang)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master) +Web Asm.js Emscripten|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20Web%20Asm.js%20Emscripten)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master)|macOS arm64 AppleClang|[![Build Status](https://dev.azure.com/UnvanquishedDevelopment/crunch/_apis/build/status%2FDaemonEngine.crunch?branchName=master&jobName=Job&configuration=Job%20macOS%20arm64%20AppleClang)](https://dev.azure.com/UnvanquishedDevelopment/crunch/_build/latest?definitionId=3&branchName=master) + ## Dæmon crunch tool -ℹ️ The [Dæmon engine](https://github.com/DaemonEngine/Daemon) uses the [`master`](https://github.com/DaemonEngine/crunch/tree/master) branch, which currently tracks the [Unity fork](https://github.com/Unity-Technologies/crunch/tree/unity), with a few additional minor fixes. The Dæmon engine is the open source game engine powering the [Unvanquished game](https://unvanquished.net). The produced CRN files are openable by the Unity game engine. +ℹ️ The [Dæmon engine](https://github.com/DaemonEngine/Daemon) uses the [`master`](https://github.com/DaemonEngine/crunch/tree/master) branch, which tracks the [Unity fork](https://github.com/Unity-Technologies/crunch/tree/unity), with additional fixes and features. The Dæmon engine is the open source game engine powering the [Unvanquished game](https://unvanquished.net). The produced `.crn` files are compatible with both the Daemon game engine and the Unity game engine. Crunch is brought to you by: -- **2014-2023**: Dæmon Developers and contributors +- **2014-2024**: Dæmon Developers and contributors https://github.com/DaemonEngine/crunch - **2017-2018**: Alexander Suvorov and Unity Software Inc. https://github.com/Unity-Technologies/crunch/tree/unity @@ -27,6 +38,17 @@ The Dæmon branch is known to be used by: - The [Xonotic game](https://xonotic.org), - Some games using the Unity game engine. +This branch provides many improvements over the original crunch: + +- ✅️ Unity crunch format (runs many time faster and produces smaller files), +- ✅️ Unity crunch metadata (the header is compatible with Unity), +- ✅️ Improved image compatibility (1-bit PNG images are now supported), +- ✅️ Added features and command line options (topmip renormalization and more), +- ✅️ Network file system compatibility, +- ✅️ Optional header-only checksumming, +- ✅️ Multisystem and multiplatform (runs almost everywhere), +- ✅️ CMake toolchain (with many useful build options). + ## Unity crunch format This repository merged improvements done by Unity Technologies, it produces @@ -53,6 +75,10 @@ Since [February 11th 2023](faf5127b8c69dfd1ae554f4f3bf8168048b48d9f), this branc Files with this value set to `1` are expected to use the new format. Files with this value set to `0` may or may not use the new format. When Unity updated the tool and modified the format in an incompatible way, no bit was modified to detect if a CRN file was using the old or the new format. Unity has not updated that field in their public repository of Crunch so there may be files using new format with this value set to `0` in the wild. +## Improved image compatibility + +The `stb_image` library was updated from version 1.18 to version 2.30, increasing the amount of image format variants that can be converted, like 1-bit PNG formats. + ## Added features and command line options In addition to the original `crunch` features and command line options this branch brings: @@ -61,28 +87,40 @@ In addition to the original `crunch` features and command line options this bran - `-noNormalDetection`: do not attempt to detect normal map to avoid selecting formats thought for normal maps when it's known an image is not a normal map. It may prevent the tool to use heavier and less-supported `DXT5_AGBR` format when `DXT1` is good enough. - `-h` or `--help`: print the command line built-in help. +## Network file system compatibility + +The original `crunch` tool used I/O functions not working on some network file systems like NFS or SSHFS. The limitations is now gone and `crunch` can now process image files stored over the network. + +## Optional header-only checksumming + +For applications distributing their files in containers already providing a checksum mechanism for the whole contained file, it's now possible to only validate the CRN header checksum instead of the whole file checksum to not checksum the whole file twice. + ## Multisystem and multiplatform portability Unlike upstream branches from Binomial LLC and Unity Software Inc. this branch focuses on keeping the code buildable outside of Windows and Visual Studio, and adds a CMake build option alongside the legacy Makefile. This `crunch` tool and the related `crnlib` library are known to build with and and run on: -- Compilers: GCC, MSYS2/Mingw, MSVC, Clang, Apple Clang, ICC, ICX. +- Compilers: GCC, MSYS2/MinGW, MSVC, Clang, Apple Clang (and others). - Systems: Linux, Windows, macOS, FreeBSD. - Architectures: amd64, arm64, i686, armhf. +## CMake toolchain + +CMake replaced the old Makefile, the provided CMake configuration provides many useful build options and toolchain files for MinGW cross-compilation. + ## How to build ```sh git clone https://github.com/DaemonEngine/crunch.git cd crunch -cmake -H. -Bbuild +cmake -S. -Bbuild cmake --build build --parallel $(nproc) ``` Where `$(nproc)` is the amount of cores of your computer. -You'll then find a `crunch` binary in `build/` folder. +You'll then find a `crunch` binary in the `build/` folder. Some CMake build options are availables (explore more with `ccmake`). @@ -94,7 +132,7 @@ http://opensource.org/licenses/Zlib Richard Geldreich removed copyright on all his work on Crunch in order to put it in public domain on 2020-09-15 with commit [`crunch@57353fa`](https://github.com/BinomialLLC/crunch/commit/57353fa9ac0908893215bc30ba106adfb80c4c95) but this repository also contains commits by Alexander Suvorov from Unity Technologies and from Dæmon Developers and contributors. Portions of this software make use of public domain code originally -written by Igor Pavlov (LZMA), RYG (crn_ryg_dxt*), and Sean Barrett (stb_image.c). +written by Igor Pavlov (LZMA), RYG (`crn_ryg_dxt*`), and Sean Barrett (`stb_image*.h`). If you use this software in a product, an acknowledgment in the product documentation would be highly appreciated but is not required. @@ -122,33 +160,33 @@ between 100-250 megatexels/sec. The current library supports PC (Win32/x64) and Xbox 360. Fast random access to individual mipmap levels is supported. -crnlib can also generates standard .DDS files at specified quality +crnlib can also generates standard DDS files at specified quality setting, which results in files that are much more compressible by LZMA/Deflate/etc. compared to files generated by standard DXTn texture tools (see below). This feature allows easy integration into any engine -or graphics library that already supports .DDS files. +or graphics library that already supports DDS files. -The .CRN file format supports the following core DXTn texture formats: +The CRN file format supports the following core DXTn texture formats: DXT1 (but not DXT1A), DXT5, DXT5A, and DXN/3DC It also supports several popular swizzled variants (several are also supported by AMD's Compressonator): DXT5_XGBR, DXT5_xGxR, DXT5_AGBR, and DXT5_CCxY (experimental luma-chroma YCoCg). -## Recommended Software +## Recommended software AMD's [Compressonator tool](https://github.com/GPUOpen-Tools/Compressonator) -is recommended to view the .DDS files created by the crunch tool and the included example projects. +is recommended to view the DDS files created by the crunch tool and the included example projects. -Note: Some of the swizzled DXTn .DDS output formats (such as DXT5_xGBR) +Note: Some of the swizzled DXTn DDS output formats (such as DXT5_xGBR) read/written by the crunch tool or examples deviate from the DX9 DDS -standard, so DXSDK tools such as DXTEX.EXE won't load them at all or +standard, so DXSDK tools such as `DXTEX.EXE` won't load them at all or they won't be properly displayed. -## Compression Algorithm Details +## Compression algorithm details -The compression process employed in creating both .CRN and -clustered .DDS files utilizes a very high quality, scalable DXTn +The compression process employed in creating both CRN and +clustered DDS files utilizes a very high quality, scalable DXTn endpoint optimizer capable of processing any number of pixels (instead of the typical hard coded 16), optional adaptive switching between several macroblock sizes/configurations (currently any combination of @@ -160,70 +198,70 @@ passes are performed between the clusterization and VQ steps to optimize quality, and several steps use a brute force refinement approach to improve quality. The majority of compression steps are multithreaded. -The .CRN format currently utilizes canonical Huffman coding for speed +The CRN format currently utilizes canonical Huffman coding for speed (similar to Deflate but with much larger tables), but the next major version will also utilize adaptive binary arithmetic coding and higher order context modeling using already developed tech from the my LZHAM compression library. -## Supported File Formats +## Supported file formats crnlib supports two compressed texture file formats. The first -format (clustered .DDS) is simple to integrate into an existing project +format (clustered DDS) is simple to integrate into an existing project (typically, no code changes are required), but it doesn't offer the highest quality/compression ratio that crnlib is capable of. Integrating -the second, higher quality custom format (.CRN) requires a few +the second, higher quality custom format (CRN) requires a few typically straightforward engine modifications to integrate the -.CRN->DXTn transcoder header file library into your tools/engine. +CRN→DXTn transcoder header file library into your tools/engine. -### .DDS +### DDS -crnlib can compress textures to standard DX9-style .DDS files using +crnlib can compress textures to standard DX9-style `.dds` files using clustered DXTn compression, which is a subset of the approach used to -create .CRN files.(For completeness, crnlib also supports vanilla, block +create CRN files.(For completeness, crnlib also supports vanilla, block by block DXTn compression too, but that's not very interesting.) -Clustered DXTn compressed .DDS files are much more compressible than +Clustered DXTn compressed DDS files are much more compressible than files created by other libraries/tools. Apart from increased -compressibility, the .DDS files generated by this process are completely +compressibility, the DDS files generated by this process are completely standard so they should be fairly easy to add to a project with little to no code changes. -To actually benefit from clustered DXTn .DDS files, your engine needs to -further losslessly compress the .DDS data generated by crnlib using a +To actually benefit from clustered DXTn DDS files, your engine needs to +further losslessly compress the DDS data generated by crnlib using a lossless codec such as zlib, lzo, LZMA, LZHAM, etc. Most likely, your engine does this already. (If not, you definitely should because DXTn compressed textures generally contain a large amount of highly redundant data.) -Clustered .DDS files are intended to be the simplest/fastest way to +Clustered DDS files are intended to be the simplest/fastest way to integrate crnlib's tech into a project. -### .CRN +### CRN -The second, better, option is to compress your textures to .CRN files -using crnlib. To read the resulting .CRN data, you must add the .CRN +The second, better, option is to compress your textures to `.crn` files +using crnlib. To read the resulting CRN data, you must add the CRN transcoder library (located in the included single file, stand-alone -header file library inc/crn_decomp.h) into your application. .CRN files +header file library inc/crn_decomp.h) into your application. CRN files provide noticeably higher quality at the same effective bitrate compared -to clustered DXTn compressed .DDS files. Also, .CRN files don't require +to clustered DXTn compressed DDS files. Also, CRN files don't require further lossless compression because they're already highly compressed. -.CRN files are a bit more difficult/risky to integrate into a project, but -the resulting compression ratio and quality is superior vs. clustered .DDS files. +CRN files are a bit more difficult/risky to integrate into a project, but +the resulting compression ratio and quality is superior vs. clustered DDS files. -### .KTX +### KTX -crnlib and crunch can read/write the .KTX file format in various pixel formats. +crnlib and crunch can read/write the `.ktx` file format in various pixel formats. Rate distortion optimization (clustered DXTc compression) is not yet supported -when writing .KTX files. +when writing KTX files. -The .KTX file format is just like .DDS, except it's a fairly well specified +The KTX file format is just like DDS, except it's a fairly well specified standard created by the Khronos Group. Unfortunately, almost all of the tools I've -found that support .KTX are fairly (to very) buggy, or are limited to only a handful -of pixel formats, so there's no guarantee that the .KTX files written by crnlib can +found that support KTX are fairly (to very) buggy, or are limited to only a handful +of pixel formats, so there's no guarantee that the KTX files written by crnlib can be reliably read by other tools. -## Building the Examples +## Building the examples This release contains the source code and projects for three simple example projects: @@ -237,39 +275,41 @@ way hasn't been tested much. ### example1 Demonstrates how to use crnlib's high-level C-helper -compression/decompression/transcoding functions in inc/crnlib.h. It's a +compression/decompression/transcoding functions in `inc/crnlib.h`. It's a fairly complete example of crnlib's functionality. ### example2 -Shows how to transcodec .CRN files to .DDS using **only** -the functionality in inc/crn_decomp.h. It does not link against against -crnlib.lib or depend on it in any way. (Note: The complete source code, -approx. 4800 lines, to the CRN transcoder is included in inc/crn_decomp.h.) -example2 is intended to show how simple it is to integrate CRN textures +Shows how to transcodec CRN files to DDS using **only** +the functionality in `inc/crn_decomp.h`. It does not link against against +`crnlib.lib` or depend on it in any way. (Note: The complete source code, +approx. 4800 lines, to the CRN transcoder is included in `inc/crn_decomp.h`.) + +`example2` is intended to show how simple it is to integrate CRN textures into your application. ### example3 + Shows how to use the regular, low-level DXTn block compressor -functions in inc/crnlib.h. This functionality is included for +functions in `inc/crnlib.h`. This functionality is included for completeness. (Your engine or toolchain most likely already has its own DXTn compressor. crnlib's compressor is typically very competitive or superior to most available closed and open source CPU-based compressors.) -## Creating Compressed Textures from the Command Line (crunch.exe) +## Creating compressed textures from the command line The simplest way to create compressed textures using crnlib is to -integrate the bin\crunch.exe or bin\crunch_x64.exe) command line tool +integrate the `crunch` (or `crunch.exe`) command line tool into your texture build toolchain or export process. It can write DXTn -compressed 2D/cubemap textures to regular DXTn compressed .DDS, -clustered (or reduced entropy) DXTn compressed .DDS, or .CRN files. It +compressed 2D/cubemap textures to regular DXTn compressed DDS, +clustered (or reduced entropy) DXTn compressed DDS, or CRN files. It can also transcode or decompress files to several standard image -formats, such as TGA or BMP. Run crunch.exe with no options for help. +formats, such as TGA or BMP. Run `crunch --help` for help. -The .CRN files created by crunch.exe can be efficiently transcoded to +The `.crn` files created by `crunch` can be efficiently transcoded to DXTn using the included CRN transcoding library, located in full source -form under inc/crn_decomp.h. +form under `inc/crn_decomp.h`. Here are a few example crunch.exe command lines: @@ -330,11 +370,11 @@ void *crn_compress( const crn_comp_params &comp_params, float *pActual_bitrate = NULL); ``` -You can also transcode/uncompress .DDS/.CRN files to raw 32bpp images +You can also transcode/uncompress DDS/CRN files to raw 32bpp images using `crn_decompress_crn_to_dds()` and `crn_decompress_dds_to_images()`. Internally, crnlib just uses inc/crn_decomp.h to transcode textures to -DXTn. If you only need to transcode .CRN format files to raw DXTn bits +DXTn. If you only need to transcode CRN format files to raw DXTn bits at runtime (and not compress), you don't actually need to compile or link against crnlib at all. Just include inc/crn_decomp.h, which contains a completely self-contained CRN transcoder in the "crnd" @@ -346,31 +386,31 @@ GPU you're using. (See example2.) Important note: When compiling under native client, be sure to define the `PLATFORM_NACL` macro before including the `inc/crn_decomp.h` header file library. -## Known Issues/Bugs +## Known issues/Bugs * crnlib currently assumes you'll be further losslessly compressing its -output .DDS files using LZMA. However, some engines use weaker codecs +output DDS files using LZMA. However, some engines use weaker codecs such as LZO, zlib, or custom codecs, so crnlib's bitrate measurements will be inaccurate. It should be easy to allow the caller to plug-in custom lossless compressors for bitrate measurement. * Compressing to a desired bitrate can be time consuming, especially when -processing large (2k or 4k) images to the .CRN format. There are several -high-level optimizations employed when compressing to clustered DXTn .DDS -files using multiple trials, but not so for .CRN. +processing large (2k or 4k) images to the CRN format. There are several +high-level optimizations employed when compressing to clustered DXTn DDS +files using multiple trials, but not so for CRN. -* The .CRN compressor does not currently use 3 color (transparent) DXT1 +* The CRN compressor does not currently use 3 color (transparent) DXT1 blocks at all, only 4 color blocks. So it doesn't support DXT1A transparency, and its output quality suffers a little due to this limitation. (Note that the clustered DXTn compressor used when -writing clustered .DDS files does *not* have this limitation.) +writing clustered DDS files does _not_ have this limitation.) * Clustered DXT5/DXT5A compressor is able to group DXT5A blocks into clusters only if they use absolute (black/white) selector indices. This -hurts performance at very low bitrates, because too many bits are +hurts performance at very low bIt will produce the `crunch.js` and `crunch.wasm` files in the `build-emscripten` directory.itrates, because too many bits are effectively given to alpha. -* DXT3 is not supported when writing .CRN or clustered DXTn DDS files. +* DXT3 is not supported when writing CRN or clustered DXTn DDS files. (DXT3 is supported by crnlib's when compressing to regular DXTn DDS files.) You'll get DXT5 files if you request DXT3. However, DXT3 is supported by the regular DXTn block compressor. (DXT3's 4bpp fixed alpha @@ -380,15 +420,20 @@ sucks verses DXT5 alpha blocks, so I don't see this as a bug deal.) hasn't been tuned for max. quality yet. * Clustered (or rate distortion optimized) DXTc compression is only -supported when writing to .DDS, not .KTX. Also, only plain block by block -compression is supported when writing to ETC1, and .CRN does not support ETC1. +supported when writing to DDS, not KTX. Also, only plain block by block +compression is supported when writing to ETC1, and CRN does not support ETC1. -## Compile to Javascript with Emscripten +## Compile to Javascript and WebAssembly with Emscripten Download and install Emscripten: - http://kripken.github.io/emscripten-site/docs/getting_started/downloads.html -From the root directory, run: -```c - emcc -O3 emscripten/crunch_lib.cpp -I./inc -s EXPORTED_FUNCTIONS="['_malloc', '_free', '_crn_get_width', '_crn_get_height', '_crn_get_levels', '_crn_get_dxt_format', '_crn_get_bytes_per_block', '_crn_get_uncompressed_size', '_crn_decompress']" -s NO_EXIT_RUNTIME=1 -s NO_FILESYSTEM=1 -s ELIMINATE_DUPLICATE_FUNCTIONS=1 -s ALLOW_MEMORY_GROWTH=1 --memory-init-file 0 -o crunch.js +- https://emscripten.org/docs/getting_started/downloads.html + +From the repository directory, run: + +```sh +cmake -Semscripten -Bbuild-emscripten +cmake --build build-emscripten ``` + +You’ll then find a `crunch.js` script and a `crunch.wasm` binary in the `build-emscripten/` folder. \ No newline at end of file