refactored entire codebase.. v2.0.0

.editorconfig

@@ -15,3 +15,9 @@ indent_size = 4
 
 [*.md]
 trim_trailing_whitespace = false
+
+[CascLib/**]
+trim_trailing_whitespace = false
+insert_final_newline = unset
+indent_style = unset
+end_of_line = unset

.gitmodules

@@ -1,6 +1,6 @@
 [submodule "CascLib"]
  path = CascLib
- url = https://github.com/ladislav-zezula/CascLib.git
+ url = https://github.com/Talv/CascLib.git
 [submodule "cxxopts"]
  path = cxxopts
  url = https://github.com/jarro2783/cxxopts.git

CHANGELOG.md

@@ -0,0 +1,25 @@
+# Change Log
+
+## [2.0.0] - 2019-07-26
+
+* Refactored entire codebase..
+* `CascLib` upgraded to `1.20`:
+ * Enumerating MNDX storage is now much faster (SC2 & Storm data)
+ * The library now outputs filepaths with backslashes, instead of forwardslashes as it did previously in case of SC2/Storm.
+ * Applied [patch](https://github.com/Talv/CascLib/commit/b2646e578b43641a46df5725d951b093a7cefce0) to preserve original case sensitivity of filenames.
+* The core functionality remains intact, however some of the existing options/arguments have been renamed and/or reorganized. Hopefully for the better.
+
+## [1.4.0] - 2019-07-22
+
+* Introduced compatibility with Windows
+
+## [1.3.0] - 2019-05-26
+
+* `CascLib` upgraded to [ef66d7bb46f0bb4dd782d3b68eb7dcc358d52a13](https://github.com/ladislav-zezula/CascLib/commit/ef66d7bb46f0bb4dd782d3b68eb7dcc358d52a13).
+
+## [1.1.0] - 2019-05-26
+
+* Forked [storm-extract#216812d7f91ab2ca72b04f2561c587c754273489](https://github.com/nydus/storm-extract/tree/216812d7f91ab2ca72b04f2561c587c754273489).
+* Rebranded to `stormex`.
+* Removed NodeJS bindings, and everything related.
+* Enhanced cli app with regex pattern filters on a filelist.

README.md

@@ -1,7 +1,6 @@
 # stormex
 
-Command-line application to list and extract files from the [CASC](https://wowdev.wiki/CASC) (Content
-Addressable Storage Container) used in Blizzard games.
+Command-line application to enumerate and extract files from [CASC](https://wowdev.wiki/CASC) (Content Addressable Storage Container) used in Blizzard games.
 
 Tested on:
 
@@ -40,59 +39,89 @@ MSBuild STORMEXTRACT.sln /p:Configuration=Release
 ## Usage
 
 ```
-Usage: stormex <PATH> [options]
-
-This program can list and optionally extract files from a CASC storage container.
-
- -h, --help Display this help
+Command-line application to enumerate and extract files from CASC (Content Addressable Storage Container) used in Blizzard games.
+
+Regex pattern is expected to follow ECMAScript syntax
+
+Usage:
+ stormex [OPTION...] [STORAGE]
+
+ Common options:
+ -h, --help Print help.
+ -v, --verbose Verbose output.
+ -q, --quiet Supresses output entirely.
+ --version Print version.
+
+ Base options:
+ -S, --storage [PATH] Path to directory with CASC.
+
+ List options:
+ -l, --list List files inside CASC.
+ -d, --details Show details about each file - such as its size.
+
+ Filter options:
+ -s, --search [SEARCH...] Search for files using a substring.
+ --smart-case Searches case insensitively if the pattern is
+ all lowercase. Search case sensitively
+ otherwise. (default: true)
+ -i, --in-regex [PATTERN...] Include files matching regex.
+ -I, --in-iregex [PATTERN...] Include files matching regex case
+ insensitively.
+ -e, --ex-regex [PATTERN...] Exclude files matching regex.
+ -E, --ex-iregex [PATTERN...] Exclude files matching regex case
+ insensitively.
+
+ Extract options:
+ -x, --extract-all Extract all files matching search filters.
+ -X, --extract-file [FILE...] Extract file(s) matching exactly.
+ -o, --outdir [PATH] Output directory for extracted files.
+ (default: ./)
+ -p, --stdout Pipe content of a file(s) to stdout instead
+ writing it to the filesystem.
+ -P, --progress Notify about progress during extraction.
+ -n, --dry-run Simulate extraction process without writing
+ any data to the filesystem.
+```
 
-Arguments:
- <PATH> Path to game installation folder
+### Examples
 
-Options:
- General:
- -v, --verbose Prints more information
- -q, --quiet Prints nothing, nada, zip
+#### List content
 
- Common:
- -s, --search <STRING> Restrict results to full paths matching STRING
- --ignore-case Case-insensitive pattern
- --include <PATTERN> Include files matching ECMAScript regex PATTERN
- --exclude <PATTERN> Exclude files matching ECMAScript regex PATTERN
+List based on a search phrase
 
- Extract:
- -x, --extract Extract the files found
- -o, --out <PATH> The folder where the files are extracted (extract only)
+```sh
+stormex '/mnt/s1/BnetGameLib/StarCraft II' -s 'buildid' -l
 ```
 
-### Examples
-
-#### List content of CASC based on regex pattern
+List all files with details and sort by filesize.
 
 ```sh
-stormex '/mnt/s1/BnetGameLib/StarCraft II' --include '/BuildId.txt$'`
+stormex '/mnt/s1/BnetGameLib/StarCraft II' -ld | sort -h
 ```
 
-#### Extract content of CASC based on inclusion and exclusion patterns
+#### Extract files based on inclusion and exclusion patterns
 
 ```sh
-stormex '/mnt/s1/BnetGameLib/StarCraft II' --ignore-case -v \
- --include '\/(DocumentInfo|Objects|Regions|Triggers)$' \
- --include '\.(fx|xml|txt|json|galaxy|SC2Style|SC2Hotkeys|SC2Lib|TriggerLib|SC2Interface|SC2Locale|SC2Components|SC2Layout)$' \
- --exclude '(dede|eses|esmx|frfr|itit|kokr|plpl|ptbr|ruru|zhcn|zhtw)\.sc2data' \
- --exclude '(PreloadAssetDB|TextureReductionValues)\.txt$' \
+stormex '/mnt/s1/BnetGameLib/StarCraft II' \
+ -I '\/(DocumentInfo|Objects|Regions|Triggers)$' \
+ -I '\.(fx|xml|txt|json|galaxy|SC2Style|SC2Hotkeys|SC2Lib|TriggerLib|SC2Interface|SC2Locale|SC2Components|SC2Layout)$' \
+ -E '(dede|eses|esmx|frfr|itit|kokr|plpl|ptbr|ruru|zhcn|zhtw)\.sc2data' \
+ -E '(PreloadAssetDB|TextureReductionValues)\.txt$' \
  -x -o './out'
 ```
 
-## Credits
+#### Extract to stdout
 
-The library absolutely, unequivocably, could not be possible without
-[ladislav-zezula's CascLib](https://github.com/ladislav-zezula/CascLib)
-library. Many thanks to [ladislav-zezula](https://github.com/ladislav-zezula).
+Extract specific file to `stdout` and pipe the stream to another program. For example convert dds to png and display it with `imagick`.
 
-~~Most of the program was canibalized from
-[Kanma's CASCExtractor](https://github.com/Kanma/CASCExtractor/) with the
-purpose of customizing it for Heroes of the Storm and integration into
-[NodeJS](https://www.nodejs.org).~~
+```sh
+stormex -S '/mnt/s1/BnetGameLib/StarCraft II' -X 'mods/core.sc2mod/base.sc2data/EditorData/Images/HeroesEditor_Logo.tga' -p | display tga:
+```
+
+```sh
+stormex -S '/mnt/s1/BnetGameLib/StarCraft II' -X 'mods/core.sc2mod/base.sc2data/EditorData/Images/EditorLogo.dds' -p | magick dds: png: | display png:
+```
+
+## Credits
 
-Most of the program was canibalized from [storm-extract](https://github.com/nydus/storm-extract) with the purpose of reducing code complexity by removing NodeJS stuff. And transitioning it into more streamlined cli app.
+* Powered by [CascLib](https://github.com/ladislav-zezula/CascLib)

include/storage.hpp

@@ -11,9 +11,41 @@
 #include "common.hpp"
 #include "util.hpp"
 
-struct StorageSearchResult
+// Based on CASC_FIND_DATA
+struct STORAGE_SEARCH_RESULT
 {
- size_t fileSize;
+ // Full name of the found file. In case when this is CKey/EKey,
+ // this will be just string representation of the key stored in 'FileKey'
+ std::string filename;
+
+ // Content key. This is present if the CASC_FEATURE_ROOT_CKEY is present
+ BYTE CKey[MD5_HASH_SIZE];
+
+ // Encoded key. This is always present.
+ BYTE EKey[MD5_HASH_SIZE];
+
+ // Tag mask. Only valid if the storage supports tags, otherwise 0
+ ULONGLONG TagBitMask;
+
+ // File data ID. Only valid if the storage supports file data IDs, otherwise CASC_INVALID_ID
+ DWORD dwFileDataId;
+
+ // Size of the file, as retrieved from CKey entry or EKey entry
+ DWORD dwFileSize;
+
+ // Locale flags. Only valid if the storage supports locale flags, otherwise CASC_INVALID_ID
+ DWORD dwLocaleFlags;
+
+ // Content flags. Only valid if the storage supports content flags, otherwise CASC_INVALID_ID
+ DWORD dwContentFlags;
+
+ // Hints as for which open method is suitable
+ DWORD bFileAvailable:1; // If true the file is available locally
+ DWORD bCanOpenByName:1;
+ DWORD bCanOpenByDataId:1;
+ DWORD bCanOpenByCKey:1;
+ DWORD bCanOpenByEKey:1;
+ CASC_NAME_TYPE NameType;
 };
 
 /**
@@ -44,7 +76,7 @@ class StorageExplorer {
 
  // TODO: CascGetStorageInfo
 
- std::vector<std::string> enumerateFiles();
+ bool enumerateFiles(std::vector<STORAGE_SEARCH_RESULT*>& searchResults);
 
  /**
  * @brief extract data of given file to location specified under filesystem

src/storage.cc

@@ -1,4 +1,3 @@
-#include <set>
 #include "storage.hpp"
 
 int StorageExplorer::openStorage(std::string src)
@@ -20,23 +19,30 @@ bool StorageExplorer::closeStorage()
  return CascCloseStorage(m_hStorage);
 }
 
-std::vector<std::string> StorageExplorer::enumerateFiles()
+bool StorageExplorer::enumerateFiles(std::vector<STORAGE_SEARCH_RESULT*>& searchResults)
 {
- std::vector<std::string> ret;
-
  CASC_FIND_DATA findData;
  HANDLE handle = CascFindFirstFile(m_hStorage, "*", &findData, NULL);
 
+ if (handle == INVALID_HANDLE_VALUE) {
+ PLOG_FATAL << "CascFindFirstFile E(" << GetLastError() << ")";
+ return false;
+ }
+
  do {
- findData.dwFileSize;
- findData.FileKey;
- findData.szFileName;
- ret.push_back(findData.szFileName);
+ if (!findData.bFileAvailable) continue;
+
+ STORAGE_SEARCH_RESULT *record = new STORAGE_SEARCH_RESULT();
+ record->filename = findData.szFileName;
+ memcpy(record->CKey, findData.CKey, sizeof(record->CKey));
+ memcpy(record->EKey, findData.EKey, sizeof(record->EKey));
+ record->dwFileSize = findData.dwFileSize;
+ searchResults.push_back(record);
  } while (CascFindNextFile(handle, &findData));
 
  CascFindClose(handle);
 
- return ret;
+ return true;
 }
 
 size_t StorageExplorer::extractFileToPath(const std::string& storedFilename, const std::string& targetFilename)