21711: Refactors load* and store* opcodes and parameters to be more f…

…lexible and consistent, as well as corresponding API, MAJOR (#287)

docs/index.html

@@ -296,6 +296,7 @@ <h1>Amalgam Language Reference</h1>
 	<a href="#operators">Language Operators</a><br>
 	<a href="#constructs">Typical Constructs</a><br>
 	<a href="#systemCommands">System Commands</a><br>
+	<a href="#fileIO">File I/O</a><br>
 	<p></p>
 	<b>Keywords</b> <input type="button" style="font-size:0.7em;" onclick='toggleSort()' value='toggle sort'/>
 	<div id='commandToc'></div>
@@ -401,6 +402,21 @@ <h2>System Commands</h2>
 	Additionally, the argument vector passed in on the command line is passed in as the variable <em>argv</em>, with any arguments consumed by the interpreter removed.  This includes the standard 0th argument which is the Amalgam script being run.  The interpreter path and name are passed in as the variable <em>interpreter</em>.
 	<hr />
 
+	<a name="fileIO"></a>
+	<h2>File I/O</h2>
+	These parameters apply to load and store opcodes and API calls:<br />
+<div class='td1'><span class="parameter">include_rand_seeds<span></div><div class='td2'>If true, attempts to include random seeds when storing and loading.</div><br />
+<div class='td1'><span class="parameter">escape_resource_name<span></div><div class='td2'>If true, will escape any characters in the resource or file name that are not universally supported across platforms.</div><br />
+<div class='td1'><span class="parameter">escape_contained_resource_names<span></div><div class='td2'>If true, then for any contained entities and their resource or file paths that extend the base file path, it will escape any characters in the names that are not universally supported across platforms.  This only applies to file formats where the entities are not flattened into one file.</div><br />
+<div class='td1'><span class="parameter">transactional<span></div><div class='td2'>If true, attempts to load and store files in a transactional manner, such that any interruption will not corrupt the file.  Not applicable to all file types.</div><br />
+<div class='td1'><span class="parameter">pretty_print<span></div><div class='td2'>If true, then any code stored will be pretty printed.</div><br />
+<div class='td1'><span class="parameter">sort_keys<span></div><div class='td2'>If true, then any associative arrays will be sorted by keys.</div><br />
+<div class='td1'><span class="parameter">flatten<span></div><div class='td2'>If true, then will attempt to flatten all contained entities into one executable object and thus one file.</div><br />
+<div class='td1'><span class="parameter">parallel_create<span></div><div class='td2'>If true, will attempt use concurrency to store and load entities in parallel.</div><br />
+<div class='td1'><span class="parameter">execute_on_load<span></div><div class='td2'>If true, will execute the code upon load, which is required when entities are stored using flatten in order to create all of the entity structures.</div><br />
+
+	File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string.  Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.
+
 	<h2>Metadata Files</h2>
 	When attempting to load an asset, whether a .amlg file or another type, the interpreter will look for a file of the same name but with the extension .madm.  The .mdam extension stands for metadata of Amalgam.  This file consists of simple code of an associative array where the data within is immediate values representing the metadata.<br />
 	<div class='td1'><span class="parameter">rand_seed<span></div><div class='td2'>The random seed to apply to the entity.</div><br />

docs/language.js

@@ -15,7 +15,7 @@ var data = [
 		"output" : "*",
 		"permissions" : "r",
 		"new value" : "new",
-		"description" : "Executes system command specified by command.  See system commands in later table.",
+		"description" : "Executes system command specified by command.  See the system commands table for further information.",
 		"example" : "(system \"exit\")"
 	},
 
@@ -1343,42 +1343,34 @@ var data = [
 	},
 
 	{
-		"parameter" : "load string file_path [bool escape_filename] [string file_type]",
+		"parameter" : "load string resource_path [string resource_type] [assoc params]",
 		"output" : "*",
 		"permissions" : "r",
-		"description" : "Loads the data specified by the resource in string.  Attempts to load the file type and parse it into appropriate data and evaluate to the corresponding code. The parameter escape_filename defaults to false, but if it is true, it will agressively escape filenames using only alphanumeric characters and the underscore, using underscore as an escape character.  If file_type is specified and not null, it will use the file_type specified instead of the extension of the file_path.  File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string.  Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.",
+		"description" : "Loads the data specified by the resource in string.  Attempts to load the file type and parse it into appropriate data and evaluate to the corresponding code. The parameter escape_filename defaults to false, but if it is true, it will agressively escape filenames using only alphanumeric characters and the underscore, using underscore as an escape character.  If resource_type is specified and not null, it will use the resource_type specified instead of the extension of the resource_path.  File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string.  Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.",
 		"example" : "(print (load \"my_directory/MyModule.amlg\"))"
 	},
 
 	{
-		"parameter" : "load_entity string file_path [id entity] [bool escape_filename] [bool escape_contained_filenames] [string file_type]",
+		"parameter" : "load_entity string resource_path [id entity] [string resource_type] [bool persistent] [assoc params]",
 		"output" : "id",
 		"permissions" : "r",
-		"description" : "Loads an entity specified by the resource in string.  Attempts to load the file type and parse it into appropriate data and store it in the entity specified by id, following the same id creation rules as create_entities, except that if no id is specified, it may default to a name based on the resource if available.  The parameter escape_filename defaults to false, but if it is true, it will agressively escape filenames using only alphanumeric characters and the underscore, using underscore as an escape character.  If escape_contained_filenames is true, which is its default, it will also escape contained entity filenames. If file_type is specified and not null, it will use the file_type specified instead of the extension of the file_path.  File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string.  Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.",
+		"description" : "Loads an entity specified by the resource in string.  Attempts to load the file type and parse it into appropriate data and store it in the entity specified by id, following the same id creation rules as create_entities, except that if no id is specified, it may default to a name based on the resource if available.  If persistent is true, default is false, then any modifications to the entity or any entity contained within it will be written out to the resource, so that the memory and persistent storage are synchronized.  Options for the file I/O are specified as key-value pairs in params.  See File I/O for the file types and related params.",
 		"example" : "(load_entity \"my_directory/MyModule.amlg\" \"MyModule\")"
 	},
 
 	{
-		"parameter" : "load_persistent_entity string file_path [id entity] [bool escape_filename]",
-		"output" : "id",
-		"permissions" : "r",
-		"description" : "Loads an entity specified by the resource in string.  Attempts to load the file type and parse it into appropriate data and store it in the entity specified by id, following the same id creation rules as create_entities. Any modifications to the entity or any entity contained within it will be written out to the resource, so that the memory and persistent storage are synchronized.  The parameter escape_filename defaults to false, but if it is true, it will agressively escape filenames using only alphanumeric characters and the underscore, using underscore as an escape character.  This command will escape contained filenames.  The file type of a persisted entity must match the extension of the file of the main entity.  File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string.  Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.\n\n<b>WARNING:</b> Loading the same file as a persistent entity in more than one place will overwrite the file each time either entity is altered, but changes will not be propogated between the entities.",
-		"example" : "(load_persistent_entity \"my_directory/MyModule.amlg\" \"MyModule\")"
-	},
-
-	{
-		"parameter" : "store string file_path * node [bool escape_filename] [string file_type] [assoc params]",
+		"parameter" : "store string resource_path * node [string resource_type] [assoc params]",
 		"output" : "bool",
 		"permissions" : "r",
-		"description" : "Stores the code specified by * to the resource in string. Returns true if successful, false if not. The parameter escape_filename defaults to false, but if it is true, it will agressively escape filenames using only alphanumeric characters and the underscore, using underscore as an escape character.   If file_type is specified and not null, it will use the file_type specified instead of the extension of the file_path.  File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string.  Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.  If params is specified, it is an assoc that contains key-value pairs describing the format.  The key \"sort_keys\" can be used to specify a boolean value, if true, then it will sort the keys, otherwise the default behavior is to emit the keys based on memory layout.",
+		"description" : "Stores the code specified by * to the resource in string. Returns true if successful, false if not. If resource_type is specified and not null, it will use the resource_type specified instead of the extension of the resource_path.    Options for the file I/O are specified as key-value pairs in params.  See File I/O for the file types and related params.",
 		"example" : "(store \"my_directory/MyData.amlg\" (list 1 2 3))"
 	},
 
 	{
-		"parameter" : "store_entity string file_path id entity [bool escape_filename] [bool escape_contained_filenames] [string file_type] [assoc params]",
+		"parameter" : "store_entity string resource_path id entity [string resource_type] [bool persistent] [assoc params]",
 		"output" : "bool",
 		"permissions" : "r",
-		"description" : "Stores the entity specified by the id to the resource in string. Returns true if successful, false if not. The parameter escape_filename defaults to false, but if it is true, it will agressively escape filenames using only alphanumeric characters and the underscore, using underscore as an escape character.  If escape_contained_filenames is true, which is its default, it will also escape contained entity filenames.  If file_type is specified and not null, it will use the file_type specified instead of the extension of the file_path.  File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string.  Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.  If params is specified, it is an assoc that contains key-value pairs describing the format.  The key \"sort_keys\" can be used to specify a boolean value, if true, then it will sort the keys, otherwise the default behavior is to emit the keys based on memory layout.  The key \"include_rand_seeds\" can be used when storing caml files to indicate whether random seeds will be stored, which defaults to true.  The key \"parallel_create\" can be used when storing caml files to indicate whether creating entities will be performed in parallel when the caml is loaded, which defaults to false.",
+		"description" : "Stores the entity specified by the id to the resource in string. Returns true if successful, false if not. If resource_type is specified and not null, it will use the resource_type specified instead of the extension of the resource_path.  If persistent is true, default is false, then any modifications to the entity or any entity contained within it will be written out to the resource, so that the memory and persistent storage are synchronized.  Options for the file I/O are specified as key-value pairs in params.  See File I/O for the file types and related params.",
 		"example" : "(store_entity \"my_directory/MyData.amlg\" \"MyData\")"
 	},
 

src/3rd_party/rapidyaml/rapidyaml-0.7.2.hpp

@@ -20201,7 +20201,7 @@ RYML_EXPORT int version_patch();
 #endif /* C4_YML_EXPORT_HPP_ */
 
 
-#if defined(C4_MSVC) || defined(C4_MINGW)
+#if defined(C4_MSVC) || defined(C4_MINGW) || defined(_MSC_VER)
 //included above:
 //#include <malloc.h>
 #else

src/Amalgam/Amalgam.h

@@ -25,36 +25,34 @@ extern "C"
 	};
 
 	//loads the entity specified into handle
-	AMALGAM_EXPORT LoadEntityStatus LoadEntity(char *handle, char *path, bool persistent, bool load_contained_entities,
-		bool escape_filename, bool escape_contained_filenames, char *write_log_filename, char *print_log_filename);
-
-	//loads the entity specified into handle
-	//TODO 19512: deprecated - legacy method to support wrappers that can't call LoadEntity returning a LoadEntityStatus yet
-	AMALGAM_EXPORT bool LoadEntityLegacy(char *handle, char *path, bool persistent, bool load_contained_entities, char *write_log_filename, char *print_log_filename);
+	AMALGAM_EXPORT LoadEntityStatus LoadEntity(char *handle, char *path, char *file_type,
+		bool persistent, char *json_file_params, char *write_log_filename, char *print_log_filename);
 
 	//verifies the entity specified by path. Uses LoadEntityStatus to return any errors and version
 	AMALGAM_EXPORT LoadEntityStatus VerifyEntity(char *path);
 
 	//clones the entity in handle to clone_handle
-	//if persistent, then path represents the location it will be persisted to
-	AMALGAM_EXPORT bool CloneEntity(char *handle, char *clone_handle, char *path, bool persistent, char *write_log_filename, char *print_log_filename);
+	//if persistent, then path, file_type, and json_file_params represent where and how it will be stored
+	AMALGAM_EXPORT bool CloneEntity(char *handle, char *clone_handle, char *path,
+		char *file_type, bool persistent, char *json_file_params,
+		char *write_log_filename, char *print_log_filename);
 
 	//stores the entity specified by handle into path
-	AMALGAM_EXPORT void   StoreEntity(char *handle, char *path, bool update_persistence_location = false, bool store_contained_entities = true);
+	AMALGAM_EXPORT void StoreEntity(char *handle, char *path, char *file_type, bool persistent, char *json_file_params);
 
 	//executes label on handle
-	AMALGAM_EXPORT void   ExecuteEntity(char *handle, char *label);
+	AMALGAM_EXPORT void ExecuteEntity(char *handle, char *label);
 
 	//destroys the entity specified by handle
-	AMALGAM_EXPORT void   DestroyEntity(char *handle);
+	AMALGAM_EXPORT void DestroyEntity(char *handle);
 
 	//sets the random seed for the entity specified by handle
-	AMALGAM_EXPORT bool   SetRandomSeed(char *handle, char *rand_seed);
+	AMALGAM_EXPORT bool SetRandomSeed(char *handle, char *rand_seed);
 
 	//sets num_entities to the number of entities and allocates an array of string pointers for the handles loaded
 	AMALGAM_EXPORT char **GetEntities(uint64_t *num_entities);
 
-	AMALGAM_EXPORT void   SetJSONToLabel(char *handle, char *label, char *json);
+	AMALGAM_EXPORT void SetJSONToLabel(char *handle, char *label, char *json);
 
 	AMALGAM_EXPORT wchar_t *GetJSONPtrFromLabelWide(char *handle, char *label);
 	AMALGAM_EXPORT char *GetJSONPtrFromLabel(char *handle, char *label);

src/Amalgam/AmalgamAPI.cpp

@@ -85,50 +85,46 @@ extern "C"
 	// api methods
 	// ************************************
 
-	LoadEntityStatus LoadEntity(char *handle, char *path, bool persistent, bool load_contained_entities,
-		bool escape_filename, bool escape_contained_filenames, char *write_log_filename, char *print_log_filename)
+	LoadEntityStatus LoadEntity(char *handle, char *path, char *file_type,
+		bool persistent, char *json_file_params, char *write_log_filename, char *print_log_filename)
 	{
 		std::string h(handle);
 		std::string p(path);
+		std::string ft(file_type);
+		std::string_view params(json_file_params);
 		std::string wlfname(write_log_filename);
 		std::string plfname(print_log_filename);
-		auto status = entint.LoadEntity(h, p, persistent, load_contained_entities, escape_filename, escape_contained_filenames, wlfname, plfname);
+		auto status = entint.LoadEntity(h, p, ft, persistent, params, wlfname, plfname);
 		return ConvertLoadStatusToCStatus(status);
 	}
 
-	bool LoadEntityLegacy(char *handle, char *path, bool persistent, bool load_contained_entities, char *write_log_filename, char *print_log_filename)
-	{
-		auto status = LoadEntity(handle, path, persistent, load_contained_entities, false, false, write_log_filename, print_log_filename);
-
-		delete[] status.message;
-		delete[] status.version;
-
-		return status.loaded;
-	}
-
 	LoadEntityStatus VerifyEntity(char *path)
 	{
 		std::string p(path);
 		auto status = entint.VerifyEntity(p);
 		return ConvertLoadStatusToCStatus(status);
 	}
 
-	bool CloneEntity(char *handle, char *clone_handle, char *path, bool persistent, char *write_log_filename, char *print_log_filename)
+	bool CloneEntity(char *handle, char *clone_handle, char *path,
+		char *file_type, bool persistent, char *json_file_params, char *write_log_filename, char *print_log_filename)
 	{
 		std::string h(handle);
 		std::string ch(clone_handle);
 		std::string p(path);
+		std::string ft(file_type);
+		std::string_view params(json_file_params);
 		std::string wlfname(write_log_filename);
 		std::string plfname(print_log_filename);
-		return entint.CloneEntity(h, ch, p, persistent, wlfname, plfname);
+		return entint.CloneEntity(h, ch, p, ft, persistent, params, wlfname, plfname);
 	}
 
-	void StoreEntity(char *handle, char *path, bool update_persistence_location, bool store_contained_entities)
+	void StoreEntity(char *handle, char *path, char *file_type, bool persistent, char *json_file_params)
 	{
 		std::string h(handle);
 		std::string p(path);
-
-		entint.StoreEntity(h, p, update_persistence_location, store_contained_entities);
+		std::string ft(file_type);
+		std::string_view params(json_file_params);
+		entint.StoreEntity(h, p, ft, persistent, params);
 	}
 
 	void SetJSONToLabel(char *handle, char *label, char *json)

src/Amalgam/AmalgamMain.cpp

@@ -235,9 +235,9 @@ PLATFORM_MAIN_CONSOLE
 	{
 		//run the standard amlg command line interface
 		EntityExternalInterface::LoadEntityStatus status;
+		AssetManager::AssetParameters asset_params(amlg_file_to_run, "", true);
 		std::string file_type = "";
-		Entity *entity = asset_manager.LoadEntityFromResourcePath(amlg_file_to_run, file_type,
-			false, true, false, true, random_seed, nullptr, status);
+		Entity *entity = asset_manager.LoadEntityFromResource(asset_params, false, random_seed, nullptr, status);
 
 		if(!status.loaded)
 			return 1;