Adding loopTool to analysis tools (watertap-org#1098)

* merged LoopTool from Analysis

* updated diff test

* add_diff_fixed_names

* Update loopTool.rst

* Update index.rst

* fixed tests

* Fixing file pathing, using abspath(__file__)

* fixed covecod for diff_spec, and pathing in test files

* switched to useing parallel manager to get mpi config

PR#1102 adds managment for num of subprocess for ray io when running on cluster

* simplify mpi check

* Remove setup_multi_processing

not really needed can be done in place

* Update docs/technical_reference/tools/loopTool.rst

Co-authored-by: Adam Atia <aatia@keylogic.com>

* Update docs/technical_reference/tools/loopTool.rst

Co-authored-by: Adam Atia <aatia@keylogic.com>

* Update docs/technical_reference/tools/loopTool.rst

Co-authored-by: Adam Atia <aatia@keylogic.com>

* Update docs/technical_reference/tools/loopTool.rst

Co-authored-by: Adam Atia <aatia@keylogic.com>

* Update docs/technical_reference/tools/loopTool.rst

Co-authored-by: Adam Atia <aatia@keylogic.com>

* Update docs/technical_reference/tools/loopTool.rst

Co-authored-by: Adam Atia <aatia@keylogic.com>

* Update docs/technical_reference/tools/loopTool.rst

Co-authored-by: Adam Atia <aatia@keylogic.com>

* update #1 to match new async setup

* updarted tests to reader and small fixes throughout

* added additonal tests

* Update test_data_merging_tool.py

* moved doc to howto as it seems better place for it

* Update index.rst

* Removed legacy code, added additional tests, added MPI test

* Update mpi4py-test.yml

* mpi fix try #1

* Fixing MPI test

* Fixed MPI rank

* Clean up and changes to MPI rank

* update doc and added outputs and tests

* cleaned up sim_csaes and added key tests for yaml config file

* Update mpi4py-test.yml

* Update docs/how_to_guides/how_to_use_loopTool_to_explore_flowsheets.rst

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* Update watertap/tools/analysis_tools/loop_tool/loop_tool.py

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* Update watertap/tools/analysis_tools/loop_tool/loop_tool.py

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* Update docs/how_to_guides/how_to_use_loopTool_to_explore_flowsheets.rst

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* Update watertap/tools/analysis_tools/loop_tool/loop_tool.py

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* Update watertap/tools/analysis_tools/loop_tool/loop_tool.py

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* Update watertap/tools/analysis_tools/loop_tool/loop_tool.py

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* Update watertap/tools/analysis_tools/loop_tool/tests/test_loop_tool.py

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>

* updates to loopt tool

* Update test_loop_tool.py

* Update test_loop_tool.py

* Update loop_tool.py

---------

Co-authored-by: Kinshuk Panda <kinshukpanda@gmail.com>
Co-authored-by: bknueven <30801372+bknueven@users.noreply.github.com>
Co-authored-by: Adam Atia <aatia@keylogic.com>

.github/workflows/mpi4py-test.yml

@@ -64,7 +64,7 @@ jobs:
       run: conda info
     - name: Test parallel pytest w/ MPI
       run: |
-        mpiexec -n 2 coverage run --parallel-mode -m mpi4py -m pytest watertap/tools/parameter_sweep/tests/test*parameter_sweep.py --no-cov
+        mpiexec -n 2 coverage run --parallel-mode -m mpi4py -m pytest watertap/tools/parameter_sweep/tests/test*parameter_sweep.py watertap/tools/analysis_tools/loop_tool/tests/test*loop_tool.py --no-cov
         # single report
         coverage combine
         # convert to XML

docs/how_to_guides/index.rst

@@ -19,6 +19,7 @@ How To Guides
    how_to_use_parameter_sweep
    how_to_use_parameter_sweep_monte_carlo
    how_to_run_differential_parameter_sweep
+   how_to_use_loopTool_to_explore_flowsheets
    how_to_install_electrolyte_database
    how_to_use_EDB
    how_to_use_ui_api

docs/technical_reference/tools/index.rst

@@ -5,3 +5,4 @@ Tools for Flowsheet Analysis
    :maxdepth: 2
 
    parameter_sweep
+

watertap/tools/analysis_tools/loop_tool/data_merging_tool.py

@@ -0,0 +1,128 @@
+#################################################################################
+# WaterTAP Copyright (c) 2020-2023, The Regents of the University of California,
+# through Lawrence Berkeley National Laboratory, Oak Ridge National Laboratory,
+# National Renewable Energy Laboratory, and National Energy Technology
+# Laboratory (subject to receipt of any required approvals from the U.S. Dept.
+# of Energy). All rights reserved.
+#
+# Please see the files COPYRIGHT.md and LICENSE.md for full copyright and license
+# information, respectively. These files are also available online at the URL
+# "https://github.com/watertap-org/watertap/"
+#################################################################################
+
+
+import h5py
+import numpy as np
+import time
+import os
+import datetime
+from datetime import datetime
+
+import logging
+
+__author__ = "Alexander V. Dudchenko (SLAC)"
+
+
+_log = logging.getLogger(__name__)
+
+
+def merge_data_into_file(
+    file_name,
+    backup_file_name,
+    directory,
+    expected_solved_values=None,
+    min_solve_values=None,
+    force_rerun=None,
+):
+    """
+    This function checks if there is a sim file, and a back up file.
+    if there is a sim file, it backs it up, and checks if full solution set already exsts and copies it over,
+    other wise it creates a new group in actual file, and adds new solved data set to it
+    """
+    run_sweep = True
+    if os.path.isfile(file_name) == False:
+        create_h5_file(file_name)
+    h5file = h5py.File(file_name, "a")
+
+    # check if there is a back up
+    if isinstance(backup_file_name, str):
+        f_old_solutions = h5py.File(backup_file_name, "r")
+        solved_values = sum(
+            np.array(
+                f_old_solutions[directory]["solve_successful"]["solve_successful"][()]
+            )
+        )
+    else:
+        solved_values = None
+    if force_rerun:
+        _log.info("Forcing a rerun")
+        run_sweep = False
+    elif force_rerun == False:
+        _log.info("Forced to not rerun")
+        run_sweep = False
+    elif force_rerun == None and solved_values is not None:
+        if min_solve_values is not None:
+            if min_solve_values <= solved_values:
+                run_sweep = False
+        elif expected_solved_values == solved_values:
+            run_sweep = False
+        _log.info(
+            "Found {} solved values, expected {} solved values , min {} solved values, re-running == {}".format(
+                solved_values, expected_solved_values, min_solve_values, run_sweep
+            )
+        )
+    if run_sweep:
+        if directory not in h5file:
+            h5file.create_group(directory)
+    elif backup_file_name is not None and os.path.isfile(backup_file_name):
+        if directory not in h5file:
+            h5file.copy(f_old_solutions[directory], directory)
+        else:
+            _log.warning(
+                "Solution already {} exist in file, not copying over back up data".format(
+                    directory
+                )
+            )
+        f_old_solutions.close()
+    h5file.close()
+    return run_sweep
+
+
+def create_backup_file(file_name, backup_name, h5_dir):
+    """used to created file and back up file
+    file_name - orignal h5 file
+    backup_name - backup name for h5 file if exists
+    h5_dir - directory to check in h5 file, if not there creates fresh file, otherwise
+    renames existing file"""
+
+    if backup_name is None and os.path.isfile(file_name):
+        h5file = h5py.File(file_name, "r")
+
+        if h5_dir in h5file:
+            # need to close file before renaming it
+            h5file.close()
+            date = datetime.now().strftime("%d_%m-%H_%M_%S")
+            backup_name = file_name + "_{}_{}".format(date, ".bak")
+            if os.path.isfile(backup_name) == False:
+                os.rename(file_name, backup_name)
+        else:
+            # close it in case we did not rename
+            h5file.close()
+    return backup_name
+
+
+def create_h5_file(file_name):
+    """used to created h5file, retry in case disk is busy"""
+    if os.path.isfile(file_name) == False:
+        file_created = False
+        for i in range(60):
+            try:
+                f = h5py.File(file_name, "w")
+                f.close()
+                file_created = True
+                break
+            except:
+                _log.warning("Could note create h5 file {}".format(file_name))
+                time.sleep(0.01)  # Waiting to see if file is free to create again
+        if file_created == False:
+            raise OSError("Could not create file {}".format(file_name))